gcp-hockey-results/motm_app/HELM_SECRETS_GUNICORN_UPDATES.md

Helm Secrets and Gunicorn Updates

Summary of Changes

This document describes the improvements made to support external Kubernetes secrets and production-ready gunicorn deployment.

1. External Secret Support

Overview

The Helm chart now supports referencing an external Kubernetes secret instead of creating a managed one. This allows you to:

• Use existing secrets from secret management tools (e.g., External Secrets Operator, Sealed Secrets)
• Share secrets across multiple deployments
• Follow security best practices by not storing secrets in Helm values

Configuration

values.yaml

secrets:
  # Use an existing external secret instead of creating one
  useExternalSecret: false  # Set to true to use external secret
  externalSecretName: ""    # Name of your existing secret
  
  # Secret key names (consistent across both managed and external secrets)
  dbPasswordKey: "db-password"
  s3AccessKeyKey: "s3-access-key"
  s3SecretKeyKey: "s3-secret-key"
  
  # Values for managed secret (only used when useExternalSecret is false)
  dbPassword: ""
  s3AccessKey: ""
  s3SecretKey: ""

Using an External Secret

Example 1: Basic external secret

# values.yaml or values-production.yaml
secrets:
  useExternalSecret: true
  externalSecretName: "motm-app-credentials"
  # The rest of the secret values are ignored when useExternalSecret is true

Example 2: Your external secret should have these keys:

apiVersion: v1
kind: Secret
metadata:
  name: motm-app-credentials
type: Opaque
data:
  db-password: <base64-encoded-password>
  s3-access-key: <base64-encoded-access-key>
  s3-secret-key: <base64-encoded-secret-key>

Example 3: Using with External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: motm-app-credentials
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: motm-app-credentials
  data:
    - secretKey: db-password
      remoteRef:
        key: motm/database
        property: password
    - secretKey: s3-access-key
      remoteRef:
        key: motm/s3
        property: access_key
    - secretKey: s3-secret-key
      remoteRef:
        key: motm/s3
        property: secret_key

Files Modified

• helm-chart/motm-app/values.yaml - Added external secret configuration
• helm-chart/motm-app/templates/secret.yaml - Made conditional based on useExternalSecret
• helm-chart/motm-app/templates/deployment.yaml - Updated to reference external or managed secret

2. S3 Environment Variable Support

Overview

The S3 configuration now prioritizes environment variables over the JSON configuration file, allowing seamless deployment in Kubernetes without managing config files.

Behavior

1. In Kubernetes (when S3_ENABLED or S3_ACCESS_KEY_ID env vars are set): Reads from environment variables
2. Locally (no env vars): Falls back to s3_config.json file

Environment Variables

The following environment variables are now supported:

• S3_ENABLED - Enable/disable S3 (true/false)
• S3_ACCESS_KEY_ID - S3/MinIO access key (from secret)
• S3_SECRET_ACCESS_KEY - S3/MinIO secret key (from secret)
• S3_STORAGE_PROVIDER - Storage provider (aws or minio, default: aws)
• S3_REGION - AWS region (default: us-east-1)
• S3_BUCKET - S3/MinIO bucket name
• S3_BUCKET_PREFIX - Key prefix/folder (default: assets/)
• S3_ENDPOINT - Custom endpoint for MinIO or S3-compatible storage
• S3_USE_SIGNED_URLS - Use signed URLs (default: true)
• S3_SIGNED_URL_EXPIRY - Signed URL expiry in seconds (default: 3600)
• S3_FALLBACK_TO_STATIC - Fallback to static files on error (default: true)
• S3_USE_SSL - Use SSL/TLS for connections (default: true)

Files Modified

• s3_config.py - Added _load_from_env() method and prioritized env vars
• helm-chart/motm-app/templates/deployment.yaml - Added S3_ENABLED environment variable

3. Gunicorn Production Server

Overview

The container now uses gunicorn instead of Flask's development server for production-ready deployment.

Benefits

• Production-ready: Gunicorn is a WSGI HTTP server designed for production
• Better performance: Multi-worker support for handling concurrent requests
• Stability: Auto-restart workers after handling requests to prevent memory leaks
• Proper process management: Better signal handling and graceful shutdowns

Configuration

The gunicorn configuration is defined in gunicorn.conf.py:

• Workers: (CPU cores * 2) + 1 for optimal performance
• Timeout: 30 seconds
• Max requests: 1000 per worker (with jitter) to prevent memory leaks
• Logging: Access and error logs to stdout/stderr
• Preload: App is preloaded for better performance

Files Modified

• Containerfile - Updated startup script to use gunicorn --config gunicorn.conf.py main:app

Note on main.py

The main.py file still contains:

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000, debug=True)

This is intentional and allows:

• Local development: Run with python main.py for development
• Production: Container uses gunicorn, which imports app from main.py directly

S3/MinIO Configuration Examples

AWS S3 Configuration

# values-production.yaml
s3:
  enabled: true
  storageProvider: "aws"
  endpoint: ""  # Leave empty for AWS
  region: "us-east-1"
  bucket: "motm-assets-prod"
  bucketPrefix: "assets/"
  useSignedUrls: true
  signedUrlExpiry: 3600
  fallbackToStatic: true
  useSSL: true

MinIO Configuration (Self-Hosted)

# values.yaml or values-production.yaml
s3:
  enabled: true
  storageProvider: "minio"
  endpoint: "https://minio.yourdomain.com"
  region: "us-east-1"  # Required for boto3, but MinIO ignores it
  bucket: "motm-assets"
  bucketPrefix: "assets/"
  useSignedUrls: false  # Use public URLs if bucket is public
  signedUrlExpiry: 3600
  fallbackToStatic: true
  useSSL: true

MinIO Configuration (In-Cluster)

# values-development.yaml
s3:
  enabled: true
  storageProvider: "minio"
  endpoint: "http://minio.default.svc.cluster.local:9000"
  region: "us-east-1"
  bucket: "motm-assets-dev"
  bucketPrefix: "dev/"
  useSignedUrls: false
  signedUrlExpiry: 3600
  fallbackToStatic: true
  useSSL: false  # HTTP for internal service

Digital Ocean Spaces / S3-Compatible Storage

s3:
  enabled: true
  storageProvider: "minio"  # Use minio provider for S3-compatible services
  endpoint: "https://nyc3.digitaloceanspaces.com"
  region: "nyc3"
  bucket: "motm-assets"
  bucketPrefix: "production/"
  useSignedUrls: true
  signedUrlExpiry: 3600
  fallbackToStatic: true
  useSSL: true

Deployment Examples

Example 1: Using Managed Secret

helm upgrade --install motm-app ./helm-chart/motm-app \
  -f values-production.yaml \
  --set secrets.useExternalSecret=false \
  --set secrets.dbPassword="your-db-password" \
  --set secrets.s3AccessKey="your-s3-key" \
  --set secrets.s3SecretKey="your-s3-secret"

Example 2: Using External Secret

# First create your external secret
kubectl create secret generic motm-app-credentials \
  --from-literal=db-password="your-db-password" \
  --from-literal=s3-access-key="your-s3-key" \
  --from-literal=s3-secret-key="your-s3-secret"

# Then deploy with external secret reference
helm upgrade --install motm-app ./helm-chart/motm-app \
  -f values-production.yaml \
  --set secrets.useExternalSecret=true \
  --set secrets.externalSecretName="motm-app-credentials"

Example 3: Custom Secret Key Names

If your external secret uses different key names:

helm upgrade --install motm-app ./helm-chart/motm-app \
  -f values-production.yaml \
  --set secrets.useExternalSecret=true \
  --set secrets.externalSecretName="my-secret" \
  --set secrets.dbPasswordKey="database-password" \
  --set secrets.s3AccessKeyKey="aws-access-key" \
  --set secrets.s3SecretKeyKey="aws-secret-key"

Backward Compatibility

All changes maintain backward compatibility:

1. Database Configuration:

   • Still reads from database_config.ini for local deployments
   • Environment variables take precedence in containers

2. S3 Configuration:

   • Still reads from s3_config.json when no env vars are set
   • Admin interface still allows configuration via web UI

3. Helm Chart:

   • Default behavior (useExternalSecret: false) creates managed secret
   • Existing deployments continue to work without changes

Security Best Practices

1. Never commit secrets to version control
2. Use external secrets for production deployments
3. Rotate credentials regularly
4. Use RBAC to restrict access to secrets in Kubernetes
5. Enable audit logging for secret access
6. Use secret management tools (Vault, AWS Secrets Manager, etc.) with External Secrets Operator

Testing

Test Database Connection

kubectl exec -it deployment/motm-app -- env | grep DB_
# Should show DB_HOST, DB_PORT, DB_NAME, DB_USER
# DB_PASSWORD won't show (secret)

Test S3 Configuration

kubectl exec -it deployment/motm-app -- env | grep S3_
# Should show S3_ENABLED, S3_REGION, S3_BUCKET, etc.

Test Gunicorn

kubectl logs -f deployment/motm-app
# Should show: "Starting application with gunicorn..."
# And gunicorn worker logs

Verify Secret is Used

kubectl get pods -l app=motm-app -o jsonpath='{.items[0].spec.containers[0].env}' | jq
# Check that secretKeyRef points to your external secret

Troubleshooting

Secret Not Found

Error: secret "motm-app-credentials" not found

Solution: Ensure the external secret exists before deploying:

kubectl get secret motm-app-credentials

Wrong Secret Keys

Error: couldn't find key db-password in Secret

Solution: Verify your secret has the correct keys:

kubectl get secret motm-app-credentials -o jsonpath='{.data}' | jq 'keys'

S3 Not Using Environment Variables

Symptom: S3 still reading from s3_config.json

Solution: Ensure S3_ENABLED=true is set in the deployment environment variables

Gunicorn Not Starting

Symptom: Container keeps restarting

Solution: Check logs for import errors:

kubectl logs deployment/motm-app

Migration Guide

From Managed to External Secret

1. Extract current secrets:

kubectl get secret motm-app-secrets -o yaml > current-secret.yaml

2. Create external secret with same data but new name:

kubectl create secret generic motm-app-credentials \
  --from-literal=db-password="$(kubectl get secret motm-app-secrets -o jsonpath='{.data.db-password}' | base64 -d)" \
  --from-literal=s3-access-key="$(kubectl get secret motm-app-secrets -o jsonpath='{.data.s3-access-key}' | base64 -d)" \
  --from-literal=s3-secret-key="$(kubectl get secret motm-app-secrets -o jsonpath='{.data.s3-secret-key}' | base64 -d)"

3. Update Helm values:

secrets:
  useExternalSecret: true
  externalSecretName: "motm-app-credentials"

4. Upgrade deployment:

helm upgrade motm-app ./helm-chart/motm-app -f values-production.yaml

5. Verify and clean up old secret:

kubectl delete secret motm-app-secrets

Summary

These updates provide:

• ✅ Flexible secret management (external or managed)
• ✅ Production-ready deployment with gunicorn
• ✅ Environment-based S3 configuration
• ✅ Backward compatibility with existing deployments
• ✅ Security best practices
• ✅ Easy migration path