Jonny Ervine 6a50c9fe90 Add voting deadline function

2025-10-13 22:30:15 +08:00

14 KiB

Raw Blame History

S3/MinIO Database Configuration

Overview

The MOTM application now supports database-backed S3/MinIO configuration with a three-tier priority system for maximum flexibility and security.

Configuration Priority

The application loads S3 configuration in the following order (highest priority first):

1. Environment Variables (Highest Priority) 🔐

Use Case: Kubernetes/container deployments, CI/CD pipelines
Credentials: ALWAYS from environment variables (never stored in database or files)
When: Detected if S3_ENABLED or S3_ACCESS_KEY_ID environment variables are set
Security: Most secure option - credentials from Kubernetes secrets

2. Database Settings (Medium Priority) 💾

Use Case: Admin-configurable via web UI
Storage: PostgreSQL/MySQL/SQLite database table s3_settings
Credentials: NOT stored in database - only configuration settings
When: Used if no environment variables are set
Benefits:
- Configure without redeploying
- No code changes needed
- Settings persist across container restarts
- Admin UI for easy management

3. JSON File (Lowest Priority) 📄

Use Case: Local development only
Storage: s3_config.json file
Credentials: Stored in JSON file (local dev only)
When: Used if no environment variables or database settings exist
Benefits: Simple local development setup

Database Schema

S3Settings Table

CREATE TABLE s3_settings (
    id INTEGER PRIMARY KEY,
    userid VARCHAR(50) DEFAULT 'admin',
    enabled BOOLEAN DEFAULT FALSE,
    storage_provider VARCHAR(20) DEFAULT 'aws',  -- 'aws' or 'minio'
    endpoint VARCHAR(255) DEFAULT '',
    region VARCHAR(50) DEFAULT 'us-east-1',
    bucket_name VARCHAR(255) DEFAULT '',
    bucket_prefix VARCHAR(255) DEFAULT 'assets/',
    use_signed_urls BOOLEAN DEFAULT TRUE,
    signed_url_expiry INTEGER DEFAULT 3600,
    fallback_to_static BOOLEAN DEFAULT TRUE,
    use_ssl BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Fields

enabled: Enable/disable S3 storage globally
storage_provider: aws for AWS S3, minio for MinIO/S3-compatible storage
endpoint: MinIO endpoint URL (e.g., https://minio.example.com or http://minio.default.svc.cluster.local:9000)
region: AWS region (required by boto3, MinIO ignores this)
bucket_name: S3/MinIO bucket name
bucket_prefix: Folder/prefix for assets (e.g., assets/, production/, logos/)
use_signed_urls: Generate signed URLs for private buckets
signed_url_expiry: Signed URL expiry time in seconds (default: 1 hour)
fallback_to_static: Fall back to local static files if S3 is unavailable
use_ssl: Use SSL/TLS for connections

Security Note

🔒 Credentials (aws_access_key_id, aws_secret_access_key) are NEVER stored in the database for security reasons.

Admin Configuration

Web UI

Navigate to Admin Dashboard → S3 Configuration
Configure settings via the form:
- Enable/disable S3
- Select storage provider (AWS S3 or MinIO)
- Enter bucket details
- Configure URL settings
Test Connection to verify settings
Save Configuration to database

What Gets Saved

✅ Saved to Database:

Enable/disable flag
Storage provider
Endpoint URL
Region
Bucket name and prefix
URL configuration
SSL settings

❌ NOT Saved to Database:

Access Key ID
Secret Access Key

Credentials Management

Local Development

Credentials entered in the web UI are saved to s3_config.json file for convenience.

Production/Kubernetes

Credentials MUST be provided via environment variables:

S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key

Typically configured in Helm chart secrets or Kubernetes secrets.

Deployment Scenarios

Scenario 1: Production Kubernetes with External Secret

# values-production.yaml
secrets:
  useExternalSecret: true
  externalSecretName: "motm-credentials"

s3:
  enabled: false  # Controlled via database, not Helm values

How it works:

Admin enables S3 via web UI → settings saved to database
Credentials loaded from external Kubernetes secret via env vars
Configuration loaded from database
Application uses S3 without redeployment

Scenario 2: Development with MinIO

# values-development.yaml
s3:
  enabled: true
  storageProvider: "minio"
  endpoint: "http://minio.default.svc.cluster.local:9000"
  bucket: "motm-dev"

How it works:

Environment variables set from Helm chart
Environment variables override database settings
Good for consistent dev environment

Scenario 3: Local Development

Option A: Web UI Configuration

Start application locally
Configure S3 via web UI
Settings saved to database
Credentials saved to s3_config.json

Option B: JSON File

Edit s3_config.json directly
Settings loaded from file
No database needed

Migration Guide

From JSON File to Database

Step 1: Current JSON Config Your existing s3_config.json:

{
  "enable_s3": true,
  "storage_provider": "aws",
  "bucket_name": "motm-assets",
  "aws_region": "us-east-1",
  ...
}

Step 2: Web UI Import

Navigate to S3 Configuration page
Current settings from JSON will pre-populate the form
Click "Save Configuration"
Settings now saved to database

Step 3: Verify

Settings persist after restart
Can delete s3_config.json (backup first!)
Configuration now in database

From Environment Variables to Database

If you're currently using environment variables for configuration (not just credentials):

Step 1: Current Setup

# deployment.yaml
env:
  - name: S3_ENABLED
    value: "true"
  - name: S3_BUCKET
    value: "motm-assets"
  # etc...

Step 2: Move to Database

Keep only credential environment variables:

env:
  - name: S3_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: motm-credentials
        key: s3-access-key
  - name: S3_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: motm-credentials
        key: s3-secret-key

Configure settings via web UI
Remove configuration env vars from deployment

Step 3: Benefits

Change bucket/region without redeployment
Zero-downtime configuration updates
Credentials still secure in Kubernetes secrets

API Usage

Loading Configuration

from s3_config import s3_config_manager

# Get current config (respects priority: env > db > file)
config = s3_config_manager.get_config_dict()

# Check if S3 is enabled
if config['enable_s3']:
    bucket = config['bucket_name']
    provider = config['storage_provider']

Saving Configuration

from s3_config import s3_config_manager

config_data = {
    'enable_s3': True,
    'storage_provider': 'minio',
    'minio_endpoint': 'https://minio.example.com',
    'aws_region': 'us-east-1',
    'bucket_name': 'motm-assets',
    'bucket_prefix': 'production/',
    'use_signed_urls': True,
    'signed_url_expiry': 3600,
    'fallback_to_static': True,
    'minio_use_ssl': True,
    # Credentials (not saved to database)
    'aws_access_key_id': 'optional-for-local-dev',
    'aws_secret_access_key': 'optional-for-local-dev'
}

# Save to database (or file as fallback)
success = s3_config_manager.save_config(config_data)

Direct Database Access

from database import get_db_session, S3Settings

session = get_db_session()
try:
    # Get settings
    settings = session.query(S3Settings).filter_by(userid='admin').first()
    
    if settings:
        print(f"S3 Enabled: {settings.enabled}")
        print(f"Provider: {settings.storage_provider}")
        print(f"Bucket: {settings.bucket_name}")
    
    # Update settings
    settings.enabled = True
    settings.bucket_name = 'new-bucket'
    session.commit()
    
finally:
    session.close()

Environment Variables Reference

Required for Production

Variable	Description	Example
`S3_ACCESS_KEY_ID`	S3/MinIO access key	`AKIAIOSFODNN7EXAMPLE`
`S3_SECRET_ACCESS_KEY`	S3/MinIO secret key	`wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`

Optional (Override Database Settings)

Variable	Description	Default
`S3_ENABLED`	Enable S3	`false`
`S3_STORAGE_PROVIDER`	Storage provider	`aws`
`S3_ENDPOINT`	Custom endpoint	``
`S3_REGION`	AWS region	`us-east-1`
`S3_BUCKET`	Bucket name	``
`S3_BUCKET_PREFIX`	Object prefix	`assets/`
`S3_USE_SIGNED_URLS`	Use signed URLs	`true`
`S3_SIGNED_URL_EXPIRY`	Expiry in seconds	`3600`
`S3_FALLBACK_TO_STATIC`	Fallback to static	`true`
`S3_USE_SSL`	Use SSL/TLS	`true`

Troubleshooting

Settings Not Loading from Database

Symptom: Web UI configuration not being used

Solutions:

Check if environment variables are set (they override database)
```
env | grep S3_
```

Verify database table exists:

SELECT * FROM s3_settings WHERE userid='admin';

Check application logs for database connection errors

Credentials Not Working

Symptom: "Access Denied" or authentication errors

Solutions:

Kubernetes: Verify secret exists and is mounted:

kubectl get secret motm-credentials
kubectl describe pod motm-app-xxx

Local: Check s3_config.json has credentials

Environment: Verify env vars are set:

echo $S3_ACCESS_KEY_ID
echo $S3_SECRET_ACCESS_KEY

Database Migration Issues

Symptom: Table s3_settings doesn't exist

Solution:

# Run database initialization
from database import init_database
init_database()

# Or via CLI
python -c "from database import init_database; init_database()"

Configuration Not Persisting

Symptom: Settings reset after restart

Causes & Solutions:

Using environment variables: Env vars always override database
- Solution: Remove config env vars, keep only credential env vars
Database not writable: Check permissions
Using ephemeral database: In containers, use persistent volume

Security Best Practices

✅ DO

✅ Use Kubernetes secrets for credentials in production
✅ Store only configuration in database
✅ Use environment variables for credentials
✅ Enable SSL/TLS for S3 connections
✅ Use signed URLs for private buckets
✅ Rotate credentials regularly
✅ Use IAM roles when possible (AWS)
✅ Restrict bucket permissions to minimum required

❌ DON'T

❌ Store credentials in database
❌ Commit s3_config.json to version control
❌ Share credentials in application logs
❌ Use root/admin credentials
❌ Disable SSL in production
❌ Make buckets public unless necessary
❌ Hard-code credentials in code

Examples

Example 1: AWS S3 Production

Database Settings (via Web UI):

Enable S3: ✓
Provider: AWS S3
Region: us-east-1
Bucket: motm-prod-assets
Prefix: assets/
Signed URLs: ✓

Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: motm-credentials
stringData:
  s3-access-key: AKIAIOSFODNN7EXAMPLE
  s3-secret-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Example 2: MinIO In-Cluster

Database Settings:

Enable S3: ✓
Provider: MinIO
Endpoint: http://minio.default.svc.cluster.local:9000
Region: us-east-1 (ignored by MinIO)
Bucket: motm-dev
Prefix: dev/
Signed URLs: ✗ (public bucket)
Use SSL: ✗ (internal HTTP service)