gcp-hockey-results/motm_app/PRODUCTION_DIAGNOSTIC_GUIDE.md

Production Database Diagnostic Guide

Issue

Getting "Admin settings not found" error when activating MOTM vote, even after saving settings.

Root Cause Analysis

The code is looking for the motm_url_suffix column in the admin_settings table. The error suggests one of these issues in production:

1. Wrong table name: Production might be using motmadminsettings instead of admin_settings
2. Missing column: The motm_url_suffix column might not exist
3. NULL value: The motm_url_suffix might be NULL or empty
4. No rows: The table might be empty

Diagnostic Steps

Step 1: Run the Diagnostic Script

Run this in your production Kubernetes pod:

# Find your pod
kubectl get pods -n motm-app -l app.kubernetes.io/name=motm-app

# Run the diagnostic script
kubectl exec -it <POD_NAME> -n motm-app -- python check_production_db.py

Step 2: Analyze the Output

The script will show:

• Which tables exist (admin_settings vs motmadminsettings)
• What data is in the table
• Whether the motm_url_suffix column has a value
• The exact query that's failing

Step 3: Based on the Results

Scenario A: Table is motmadminsettings (Old Name)

If the output shows motmadminsettings exists but admin_settings doesn't:

Solution: Your production database still has the old table name. You need to either:

1. Rename the table (recommended):

   kubectl exec -it <POD_NAME> -n motm-app -- python -c "
   from db_config import db_config
   from sqlalchemy import text
   engine = db_config.engine
   conn = engine.connect()
   conn.execute(text('ALTER TABLE motmadminsettings RENAME TO admin_settings'))
   conn.commit()
   conn.close()
   print('Table renamed successfully')
   "
   

2. Or update the code to use motmadminsettings (not recommended, but works as a quick fix)

Scenario B: motm_url_suffix is NULL or Empty

If the output shows the column exists but is NULL or empty:

Solution: The column needs to be populated. Run this:

kubectl exec -it <POD_NAME> -n motm-app -- python -c "
from db_config import db_config
from sqlalchemy import text
import random
import string

# Generate a random URL suffix
def randomUrlSuffix(length=8):
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

engine = db_config.engine
conn = engine.connect()
urlSuffix = randomUrlSuffix(8)
conn.execute(text('UPDATE admin_settings SET motm_url_suffix = :suffix WHERE userid = \\'admin\\''), {'suffix': urlSuffix})
conn.commit()
conn.close()
print(f'URL suffix set to: {urlSuffix}')
"

Scenario C: Table is Empty

If the output shows 0 rows:

Solution: You need to initialize the admin settings. Run this:

kubectl exec -it <POD_NAME> -n motm-app -- python -c "
from db_config import db_config
from sqlalchemy import text
import random
import string

def randomUrlSuffix(length=8):
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

engine = db_config.engine
conn = engine.connect()
urlSuffix = randomUrlSuffix(8)

# Insert default admin settings
conn.execute(text('''
    INSERT INTO admin_settings (userid, motm_url_suffix, next_fixture, prev_fixture)
    VALUES ('admin', :suffix, 1, 0)
    ON CONFLICT (userid) DO UPDATE SET motm_url_suffix = :suffix
'''), {'suffix': urlSuffix})

conn.commit()
conn.close()
print(f'Admin settings initialized with URL suffix: {urlSuffix}')
"

Scenario D: Column Doesn't Exist

If the output shows the column is missing:

Solution: Add the missing column:

kubectl exec -it <POD_NAME> -n motm-app -- python -c "
from db_config import db_config
from sqlalchemy import text

engine = db_config.engine
conn = engine.connect()

# Add the column if it doesn't exist
try:
    conn.execute(text('ALTER TABLE admin_settings ADD COLUMN motm_url_suffix VARCHAR(50)'))
    conn.commit()
    print('Column added successfully')
except Exception as e:
    if 'already exists' in str(e):
        print('Column already exists')
    else:
        print(f'Error: {e}')

conn.close()
"

Quick Fix Script

If you want to try a comprehensive fix that handles all scenarios:

kubectl exec -it <POD_NAME> -n motm-app -- python << 'EOF'
from db_config import db_config
from sqlalchemy import text
import random
import string

def randomUrlSuffix(length=8):
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

engine = db_config.engine
conn = engine.connect()

# Step 1: Check if table exists
try:
    result = conn.execute(text("SELECT 1 FROM admin_settings LIMIT 1"))
    print("✓ admin_settings table exists")
except:
    print("✗ admin_settings table does not exist")
    print("Please check if you're using the old table name 'motmadminsettings'")
    conn.close()
    exit(1)

# Step 2: Check if motm_url_suffix column exists
try:
    result = conn.execute(text("SELECT motm_url_suffix FROM admin_settings LIMIT 1"))
    print("✓ motm_url_suffix column exists")
except:
    print("✗ motm_url_suffix column does not exist, adding it...")
    conn.execute(text("ALTER TABLE admin_settings ADD COLUMN motm_url_suffix VARCHAR(50)"))
    conn.commit()
    print("✓ Column added")

# Step 3: Check if URL suffix has a value
result = conn.execute(text("SELECT motm_url_suffix FROM admin_settings WHERE userid = 'admin'"))
row = result.fetchone()
if row and row[0]:
    print(f"✓ URL suffix exists: {row[0]}")
else:
    print("✗ URL suffix is NULL or empty, setting it...")
    urlSuffix = randomUrlSuffix(8)
    conn.execute(text("UPDATE admin_settings SET motm_url_suffix = :suffix WHERE userid = 'admin'"), {'suffix': urlSuffix})
    conn.commit()
    print(f"✓ URL suffix set to: {urlSuffix}")

conn.close()
print("\n✓ All checks passed!")
EOF

After Running the Fix

1. Restart the application to clear any cached connections:

   kubectl rollout restart deployment/motm-app -n motm-app
   

2. Test the MOTM admin page:

   • Go to https://motm.ervine.cloud/admin/motm
   • Fill in the match details
   • Click "Activate MotM Vote"
   • It should work without errors

Prevention

To prevent this issue in the future:

1. Run database migrations before deploying new code
2. Use the migration script (add_voting_deadline.py) to ensure all columns exist
3. Test in staging before deploying to production
4. Monitor logs for SQL errors

Support

If you continue to have issues after running these diagnostics:

1. Save the output from check_production_db.py
2. Check the application logs: kubectl logs -n motm-app -l app.kubernetes.io/name=motm-app --tail=100
3. Verify the database connection is working
4. Check if there are any database permission issues