512 lines
14 KiB
Markdown
512 lines
14 KiB
Markdown
# S3/MinIO Database Configuration
|
|
|
|
## Overview
|
|
|
|
The MOTM application now supports database-backed S3/MinIO configuration with a three-tier priority system for maximum flexibility and security.
|
|
|
|
## Configuration Priority
|
|
|
|
The application loads S3 configuration in the following order (highest priority first):
|
|
|
|
### 1. Environment Variables (Highest Priority) 🔐
|
|
- **Use Case**: Kubernetes/container deployments, CI/CD pipelines
|
|
- **Credentials**: ALWAYS from environment variables (never stored in database or files)
|
|
- **When**: Detected if `S3_ENABLED` or `S3_ACCESS_KEY_ID` environment variables are set
|
|
- **Security**: Most secure option - credentials from Kubernetes secrets
|
|
|
|
### 2. Database Settings (Medium Priority) 💾
|
|
- **Use Case**: Admin-configurable via web UI
|
|
- **Storage**: PostgreSQL/MySQL/SQLite database table `s3_settings`
|
|
- **Credentials**: NOT stored in database - only configuration settings
|
|
- **When**: Used if no environment variables are set
|
|
- **Benefits**:
|
|
- Configure without redeploying
|
|
- No code changes needed
|
|
- Settings persist across container restarts
|
|
- Admin UI for easy management
|
|
|
|
### 3. JSON File (Lowest Priority) 📄
|
|
- **Use Case**: Local development only
|
|
- **Storage**: `s3_config.json` file
|
|
- **Credentials**: Stored in JSON file (local dev only)
|
|
- **When**: Used if no environment variables or database settings exist
|
|
- **Benefits**: Simple local development setup
|
|
|
|
## Database Schema
|
|
|
|
### S3Settings Table
|
|
|
|
```sql
|
|
CREATE TABLE s3_settings (
|
|
id INTEGER PRIMARY KEY,
|
|
userid VARCHAR(50) DEFAULT 'admin',
|
|
enabled BOOLEAN DEFAULT FALSE,
|
|
storage_provider VARCHAR(20) DEFAULT 'aws', -- 'aws' or 'minio'
|
|
endpoint VARCHAR(255) DEFAULT '',
|
|
region VARCHAR(50) DEFAULT 'us-east-1',
|
|
bucket_name VARCHAR(255) DEFAULT '',
|
|
bucket_prefix VARCHAR(255) DEFAULT 'assets/',
|
|
use_signed_urls BOOLEAN DEFAULT TRUE,
|
|
signed_url_expiry INTEGER DEFAULT 3600,
|
|
fallback_to_static BOOLEAN DEFAULT TRUE,
|
|
use_ssl BOOLEAN DEFAULT TRUE,
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
```
|
|
|
|
### Fields
|
|
|
|
- **enabled**: Enable/disable S3 storage globally
|
|
- **storage_provider**: `aws` for AWS S3, `minio` for MinIO/S3-compatible storage
|
|
- **endpoint**: MinIO endpoint URL (e.g., `https://minio.example.com` or `http://minio.default.svc.cluster.local:9000`)
|
|
- **region**: AWS region (required by boto3, MinIO ignores this)
|
|
- **bucket_name**: S3/MinIO bucket name
|
|
- **bucket_prefix**: Folder/prefix for assets (e.g., `assets/`, `production/`, `logos/`)
|
|
- **use_signed_urls**: Generate signed URLs for private buckets
|
|
- **signed_url_expiry**: Signed URL expiry time in seconds (default: 1 hour)
|
|
- **fallback_to_static**: Fall back to local static files if S3 is unavailable
|
|
- **use_ssl**: Use SSL/TLS for connections
|
|
|
|
### Security Note
|
|
**🔒 Credentials (`aws_access_key_id`, `aws_secret_access_key`) are NEVER stored in the database for security reasons.**
|
|
|
|
## Admin Configuration
|
|
|
|
### Web UI
|
|
|
|
1. Navigate to **Admin Dashboard** → **S3 Configuration**
|
|
2. Configure settings via the form:
|
|
- Enable/disable S3
|
|
- Select storage provider (AWS S3 or MinIO)
|
|
- Enter bucket details
|
|
- Configure URL settings
|
|
3. **Test Connection** to verify settings
|
|
4. **Save Configuration** to database
|
|
|
|
### What Gets Saved
|
|
|
|
✅ **Saved to Database:**
|
|
- Enable/disable flag
|
|
- Storage provider
|
|
- Endpoint URL
|
|
- Region
|
|
- Bucket name and prefix
|
|
- URL configuration
|
|
- SSL settings
|
|
|
|
❌ **NOT Saved to Database:**
|
|
- Access Key ID
|
|
- Secret Access Key
|
|
|
|
### Credentials Management
|
|
|
|
#### Local Development
|
|
Credentials entered in the web UI are saved to `s3_config.json` file for convenience.
|
|
|
|
#### Production/Kubernetes
|
|
Credentials **MUST** be provided via environment variables:
|
|
```bash
|
|
S3_ACCESS_KEY_ID=your-access-key
|
|
S3_SECRET_ACCESS_KEY=your-secret-key
|
|
```
|
|
|
|
Typically configured in Helm chart secrets or Kubernetes secrets.
|
|
|
|
## Deployment Scenarios
|
|
|
|
### Scenario 1: Production Kubernetes with External Secret
|
|
|
|
```yaml
|
|
# values-production.yaml
|
|
secrets:
|
|
useExternalSecret: true
|
|
externalSecretName: "motm-credentials"
|
|
|
|
s3:
|
|
enabled: false # Controlled via database, not Helm values
|
|
```
|
|
|
|
**How it works:**
|
|
1. Admin enables S3 via web UI → settings saved to database
|
|
2. Credentials loaded from external Kubernetes secret via env vars
|
|
3. Configuration loaded from database
|
|
4. Application uses S3 without redeployment
|
|
|
|
### Scenario 2: Development with MinIO
|
|
|
|
```yaml
|
|
# values-development.yaml
|
|
s3:
|
|
enabled: true
|
|
storageProvider: "minio"
|
|
endpoint: "http://minio.default.svc.cluster.local:9000"
|
|
bucket: "motm-dev"
|
|
```
|
|
|
|
**How it works:**
|
|
1. Environment variables set from Helm chart
|
|
2. Environment variables override database settings
|
|
3. Good for consistent dev environment
|
|
|
|
### Scenario 3: Local Development
|
|
|
|
**Option A: Web UI Configuration**
|
|
1. Start application locally
|
|
2. Configure S3 via web UI
|
|
3. Settings saved to database
|
|
4. Credentials saved to `s3_config.json`
|
|
|
|
**Option B: JSON File**
|
|
1. Edit `s3_config.json` directly
|
|
2. Settings loaded from file
|
|
3. No database needed
|
|
|
|
## Migration Guide
|
|
|
|
### From JSON File to Database
|
|
|
|
**Step 1: Current JSON Config**
|
|
Your existing `s3_config.json`:
|
|
```json
|
|
{
|
|
"enable_s3": true,
|
|
"storage_provider": "aws",
|
|
"bucket_name": "motm-assets",
|
|
"aws_region": "us-east-1",
|
|
...
|
|
}
|
|
```
|
|
|
|
**Step 2: Web UI Import**
|
|
1. Navigate to S3 Configuration page
|
|
2. Current settings from JSON will pre-populate the form
|
|
3. Click "Save Configuration"
|
|
4. Settings now saved to database
|
|
|
|
**Step 3: Verify**
|
|
- Settings persist after restart
|
|
- Can delete `s3_config.json` (backup first!)
|
|
- Configuration now in database
|
|
|
|
### From Environment Variables to Database
|
|
|
|
If you're currently using environment variables for configuration (not just credentials):
|
|
|
|
**Step 1: Current Setup**
|
|
```yaml
|
|
# deployment.yaml
|
|
env:
|
|
- name: S3_ENABLED
|
|
value: "true"
|
|
- name: S3_BUCKET
|
|
value: "motm-assets"
|
|
# etc...
|
|
```
|
|
|
|
**Step 2: Move to Database**
|
|
1. Keep only credential environment variables:
|
|
```yaml
|
|
env:
|
|
- name: S3_ACCESS_KEY_ID
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: motm-credentials
|
|
key: s3-access-key
|
|
- name: S3_SECRET_ACCESS_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: motm-credentials
|
|
key: s3-secret-key
|
|
```
|
|
|
|
2. Configure settings via web UI
|
|
3. Remove configuration env vars from deployment
|
|
|
|
**Step 3: Benefits**
|
|
- Change bucket/region without redeployment
|
|
- Zero-downtime configuration updates
|
|
- Credentials still secure in Kubernetes secrets
|
|
|
|
## API Usage
|
|
|
|
### Loading Configuration
|
|
|
|
```python
|
|
from s3_config import s3_config_manager
|
|
|
|
# Get current config (respects priority: env > db > file)
|
|
config = s3_config_manager.get_config_dict()
|
|
|
|
# Check if S3 is enabled
|
|
if config['enable_s3']:
|
|
bucket = config['bucket_name']
|
|
provider = config['storage_provider']
|
|
```
|
|
|
|
### Saving Configuration
|
|
|
|
```python
|
|
from s3_config import s3_config_manager
|
|
|
|
config_data = {
|
|
'enable_s3': True,
|
|
'storage_provider': 'minio',
|
|
'minio_endpoint': 'https://minio.example.com',
|
|
'aws_region': 'us-east-1',
|
|
'bucket_name': 'motm-assets',
|
|
'bucket_prefix': 'production/',
|
|
'use_signed_urls': True,
|
|
'signed_url_expiry': 3600,
|
|
'fallback_to_static': True,
|
|
'minio_use_ssl': True,
|
|
# Credentials (not saved to database)
|
|
'aws_access_key_id': 'optional-for-local-dev',
|
|
'aws_secret_access_key': 'optional-for-local-dev'
|
|
}
|
|
|
|
# Save to database (or file as fallback)
|
|
success = s3_config_manager.save_config(config_data)
|
|
```
|
|
|
|
### Direct Database Access
|
|
|
|
```python
|
|
from database import get_db_session, S3Settings
|
|
|
|
session = get_db_session()
|
|
try:
|
|
# Get settings
|
|
settings = session.query(S3Settings).filter_by(userid='admin').first()
|
|
|
|
if settings:
|
|
print(f"S3 Enabled: {settings.enabled}")
|
|
print(f"Provider: {settings.storage_provider}")
|
|
print(f"Bucket: {settings.bucket_name}")
|
|
|
|
# Update settings
|
|
settings.enabled = True
|
|
settings.bucket_name = 'new-bucket'
|
|
session.commit()
|
|
|
|
finally:
|
|
session.close()
|
|
```
|
|
|
|
## Environment Variables Reference
|
|
|
|
### Required for Production
|
|
|
|
| Variable | Description | Example |
|
|
|----------|-------------|---------|
|
|
| `S3_ACCESS_KEY_ID` | S3/MinIO access key | `AKIAIOSFODNN7EXAMPLE` |
|
|
| `S3_SECRET_ACCESS_KEY` | S3/MinIO secret key | `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` |
|
|
|
|
### Optional (Override Database Settings)
|
|
|
|
| Variable | Description | Default |
|
|
|----------|-------------|---------|
|
|
| `S3_ENABLED` | Enable S3 | `false` |
|
|
| `S3_STORAGE_PROVIDER` | Storage provider | `aws` |
|
|
| `S3_ENDPOINT` | Custom endpoint | `` |
|
|
| `S3_REGION` | AWS region | `us-east-1` |
|
|
| `S3_BUCKET` | Bucket name | `` |
|
|
| `S3_BUCKET_PREFIX` | Object prefix | `assets/` |
|
|
| `S3_USE_SIGNED_URLS` | Use signed URLs | `true` |
|
|
| `S3_SIGNED_URL_EXPIRY` | Expiry in seconds | `3600` |
|
|
| `S3_FALLBACK_TO_STATIC` | Fallback to static | `true` |
|
|
| `S3_USE_SSL` | Use SSL/TLS | `true` |
|
|
|
|
## Troubleshooting
|
|
|
|
### Settings Not Loading from Database
|
|
|
|
**Symptom:** Web UI configuration not being used
|
|
|
|
**Solutions:**
|
|
1. Check if environment variables are set (they override database)
|
|
```bash
|
|
env | grep S3_
|
|
```
|
|
2. Verify database table exists:
|
|
```sql
|
|
SELECT * FROM s3_settings WHERE userid='admin';
|
|
```
|
|
3. Check application logs for database connection errors
|
|
|
|
### Credentials Not Working
|
|
|
|
**Symptom:** "Access Denied" or authentication errors
|
|
|
|
**Solutions:**
|
|
1. **Kubernetes**: Verify secret exists and is mounted:
|
|
```bash
|
|
kubectl get secret motm-credentials
|
|
kubectl describe pod motm-app-xxx
|
|
```
|
|
2. **Local**: Check `s3_config.json` has credentials
|
|
3. **Environment**: Verify env vars are set:
|
|
```bash
|
|
echo $S3_ACCESS_KEY_ID
|
|
echo $S3_SECRET_ACCESS_KEY
|
|
```
|
|
|
|
### Database Migration Issues
|
|
|
|
**Symptom:** Table `s3_settings` doesn't exist
|
|
|
|
**Solution:**
|
|
```python
|
|
# Run database initialization
|
|
from database import init_database
|
|
init_database()
|
|
|
|
# Or via CLI
|
|
python -c "from database import init_database; init_database()"
|
|
```
|
|
|
|
### Configuration Not Persisting
|
|
|
|
**Symptom:** Settings reset after restart
|
|
|
|
**Causes & Solutions:**
|
|
1. **Using environment variables**: Env vars always override database
|
|
- Solution: Remove config env vars, keep only credential env vars
|
|
2. **Database not writable**: Check permissions
|
|
3. **Using ephemeral database**: In containers, use persistent volume
|
|
|
|
## Security Best Practices
|
|
|
|
### ✅ DO
|
|
|
|
- ✅ Use Kubernetes secrets for credentials in production
|
|
- ✅ Store only configuration in database
|
|
- ✅ Use environment variables for credentials
|
|
- ✅ Enable SSL/TLS for S3 connections
|
|
- ✅ Use signed URLs for private buckets
|
|
- ✅ Rotate credentials regularly
|
|
- ✅ Use IAM roles when possible (AWS)
|
|
- ✅ Restrict bucket permissions to minimum required
|
|
|
|
### ❌ DON'T
|
|
|
|
- ❌ Store credentials in database
|
|
- ❌ Commit `s3_config.json` to version control
|
|
- ❌ Share credentials in application logs
|
|
- ❌ Use root/admin credentials
|
|
- ❌ Disable SSL in production
|
|
- ❌ Make buckets public unless necessary
|
|
- ❌ Hard-code credentials in code
|
|
|
|
## Examples
|
|
|
|
### Example 1: AWS S3 Production
|
|
|
|
**Database Settings** (via Web UI):
|
|
- Enable S3: ✓
|
|
- Provider: AWS S3
|
|
- Region: us-east-1
|
|
- Bucket: motm-prod-assets
|
|
- Prefix: assets/
|
|
- Signed URLs: ✓
|
|
|
|
**Kubernetes Secret:**
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: motm-credentials
|
|
stringData:
|
|
s3-access-key: AKIAIOSFODNN7EXAMPLE
|
|
s3-secret-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
|
|
```
|
|
|
|
### Example 2: MinIO In-Cluster
|
|
|
|
**Database Settings:**
|
|
- Enable S3: ✓
|
|
- Provider: MinIO
|
|
- Endpoint: http://minio.default.svc.cluster.local:9000
|
|
- Region: us-east-1 (ignored by MinIO)
|
|
- Bucket: motm-dev
|
|
- Prefix: dev/
|
|
- Signed URLs: ✗ (public bucket)
|
|
- Use SSL: ✗ (internal HTTP service)
|
|
|
|
**Kubernetes Secret:**
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: motm-credentials
|
|
stringData:
|
|
s3-access-key: minio-access-key
|
|
s3-secret-key: minio-secret-key
|
|
```
|
|
|
|
### Example 3: Digital Ocean Spaces
|
|
|
|
**Database Settings:**
|
|
- Enable S3: ✓
|
|
- Provider: MinIO (S3-compatible)
|
|
- Endpoint: https://nyc3.digitaloceanspaces.com
|
|
- Region: nyc3
|
|
- Bucket: motm-assets
|
|
- Prefix: production/
|
|
- Signed URLs: ✓
|
|
- Use SSL: ✓
|
|
|
|
## Summary
|
|
|
|
### Key Benefits
|
|
|
|
1. **Flexibility**: Three-tier priority system adapts to any deployment scenario
|
|
2. **Security**: Credentials never in database, always from secure sources
|
|
3. **Convenience**: Admin UI for configuration without redeployment
|
|
4. **Compatibility**: Backward compatible with existing file-based configuration
|
|
5. **Production-Ready**: Kubernetes-native with secret management
|
|
|
|
### Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────┐
|
|
│ Application Startup │
|
|
└──────────────┬──────────────────────────┘
|
|
│
|
|
▼
|
|
┌────────────────┐
|
|
│ Load S3 Config │
|
|
└────────┬───────┘
|
|
│
|
|
▼
|
|
┌───────────────────────┐
|
|
│ 1. Check Env Vars? │
|
|
│ (S3_ENABLED set) │───YES──→ Use Environment Variables
|
|
└──────────┬────────────┘
|
|
│ NO
|
|
▼
|
|
┌───────────────────────┐
|
|
│ 2. Check Database? │
|
|
│ (s3_settings table) │───YES──→ Use Database Settings
|
|
└──────────┬────────────┘ + Env Var Credentials
|
|
│ NO
|
|
▼
|
|
┌───────────────────────┐
|
|
│ 3. Check JSON File? │
|
|
│ (s3_config.json) │───YES──→ Use JSON File
|
|
└──────────┬────────────┘
|
|
│ NO
|
|
▼
|
|
┌───────────────────────┐
|
|
│ Use Defaults │
|
|
│ (S3 Disabled) │
|
|
└───────────────────────┘
|
|
```
|
|
|
|
## Conclusion
|
|
|
|
The database-backed S3 configuration provides a robust, secure, and flexible solution for managing object storage settings across different deployment scenarios while maintaining the highest security standards for credential management.
|
|
|
|
For questions or issues, refer to the troubleshooting section or check application logs for detailed error messages.
|
|
|