gcp-hockey-results/motm_app/PRODUCTION_DIAGNOSTIC_GUIDE.md

# Production Database Diagnostic Guide

## Issue
Getting "Admin settings not found" error when activating MOTM vote, even after saving settings.

## Root Cause Analysis

The code is looking for the `motm_url_suffix` column in the `admin_settings` table. The error suggests one of these issues in production:

1. **Wrong table name**: Production might be using `motmadminsettings` instead of `admin_settings`
2. **Missing column**: The `motm_url_suffix` column might not exist
3. **NULL value**: The `motm_url_suffix` might be NULL or empty
4. **No rows**: The table might be empty

## Diagnostic Steps

### Step 1: Run the Diagnostic Script

Run this in your production Kubernetes pod:

```bash
# Find your pod
kubectl get pods -n motm-app -l app.kubernetes.io/name=motm-app

# Run the diagnostic script
kubectl exec -it <POD_NAME> -n motm-app -- python check_production_db.py
```

### Step 2: Analyze the Output

The script will show:
- Which tables exist (`admin_settings` vs `motmadminsettings`)
- What data is in the table
- Whether the `motm_url_suffix` column has a value
- The exact query that's failing

### Step 3: Based on the Results

#### Scenario A: Table is `motmadminsettings` (Old Name)

If the output shows `motmadminsettings` exists but `admin_settings` doesn't:

**Solution**: Your production database still has the old table name. You need to either:

1. **Rename the table** (recommended):
   ```bash
   kubectl exec -it <POD_NAME> -n motm-app -- python -c "
   from db_config import db_config
   from sqlalchemy import text
   engine = db_config.engine
   conn = engine.connect()
   conn.execute(text('ALTER TABLE motmadminsettings RENAME TO admin_settings'))
   conn.commit()
   conn.close()
   print('Table renamed successfully')
   "
   ```

2. **Or update the code** to use `motmadminsettings` (not recommended, but works as a quick fix)

#### Scenario B: `motm_url_suffix` is NULL or Empty

If the output shows the column exists but is NULL or empty:

**Solution**: The column needs to be populated. Run this:

```bash
kubectl exec -it <POD_NAME> -n motm-app -- python -c "
from db_config import db_config
from sqlalchemy import text
import random
import string

# Generate a random URL suffix
def randomUrlSuffix(length=8):
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

engine = db_config.engine
conn = engine.connect()
urlSuffix = randomUrlSuffix(8)
conn.execute(text('UPDATE admin_settings SET motm_url_suffix = :suffix WHERE userid = \\'admin\\''), {'suffix': urlSuffix})
conn.commit()
conn.close()
print(f'URL suffix set to: {urlSuffix}')
"
```

#### Scenario C: Table is Empty

If the output shows 0 rows:

**Solution**: You need to initialize the admin settings. Run this:

```bash
kubectl exec -it <POD_NAME> -n motm-app -- python -c "
from db_config import db_config
from sqlalchemy import text
import random
import string

def randomUrlSuffix(length=8):
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

engine = db_config.engine
conn = engine.connect()
urlSuffix = randomUrlSuffix(8)

# Insert default admin settings
conn.execute(text('''
    INSERT INTO admin_settings (userid, motm_url_suffix, next_fixture, prev_fixture)
    VALUES ('admin', :suffix, 1, 0)
    ON CONFLICT (userid) DO UPDATE SET motm_url_suffix = :suffix
'''), {'suffix': urlSuffix})

conn.commit()
conn.close()
print(f'Admin settings initialized with URL suffix: {urlSuffix}')
"
```

#### Scenario D: Column Doesn't Exist

If the output shows the column is missing:

**Solution**: Add the missing column:

```bash
kubectl exec -it <POD_NAME> -n motm-app -- python -c "
from db_config import db_config
from sqlalchemy import text

engine = db_config.engine
conn = engine.connect()

# Add the column if it doesn't exist
try:
    conn.execute(text('ALTER TABLE admin_settings ADD COLUMN motm_url_suffix VARCHAR(50)'))
    conn.commit()
    print('Column added successfully')
except Exception as e:
    if 'already exists' in str(e):
        print('Column already exists')
    else:
        print(f'Error: {e}')

conn.close()
"
```

## Quick Fix Script

If you want to try a comprehensive fix that handles all scenarios:

```bash
kubectl exec -it <POD_NAME> -n motm-app -- python << 'EOF'
from db_config import db_config
from sqlalchemy import text
import random
import string

def randomUrlSuffix(length=8):
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

engine = db_config.engine
conn = engine.connect()

# Step 1: Check if table exists
try:
    result = conn.execute(text("SELECT 1 FROM admin_settings LIMIT 1"))
    print("✓ admin_settings table exists")
except:
    print("✗ admin_settings table does not exist")
    print("Please check if you're using the old table name 'motmadminsettings'")
    conn.close()
    exit(1)

# Step 2: Check if motm_url_suffix column exists
try:
    result = conn.execute(text("SELECT motm_url_suffix FROM admin_settings LIMIT 1"))
    print("✓ motm_url_suffix column exists")
except:
    print("✗ motm_url_suffix column does not exist, adding it...")
    conn.execute(text("ALTER TABLE admin_settings ADD COLUMN motm_url_suffix VARCHAR(50)"))
    conn.commit()
    print("✓ Column added")

# Step 3: Check if URL suffix has a value
result = conn.execute(text("SELECT motm_url_suffix FROM admin_settings WHERE userid = 'admin'"))
row = result.fetchone()
if row and row[0]:
    print(f"✓ URL suffix exists: {row[0]}")
else:
    print("✗ URL suffix is NULL or empty, setting it...")
    urlSuffix = randomUrlSuffix(8)
    conn.execute(text("UPDATE admin_settings SET motm_url_suffix = :suffix WHERE userid = 'admin'"), {'suffix': urlSuffix})
    conn.commit()
    print(f"✓ URL suffix set to: {urlSuffix}")

conn.close()
print("\n✓ All checks passed!")
EOF
```

## After Running the Fix

1. **Restart the application** to clear any cached connections:
   ```bash
   kubectl rollout restart deployment/motm-app -n motm-app
   ```

2. **Test the MOTM admin page**:
   - Go to https://motm.ervine.cloud/admin/motm
   - Fill in the match details
   - Click "Activate MotM Vote"
   - It should work without errors

## Prevention

To prevent this issue in the future:

1. **Run database migrations** before deploying new code
2. **Use the migration script** (`add_voting_deadline.py`) to ensure all columns exist
3. **Test in staging** before deploying to production
4. **Monitor logs** for SQL errors

## Support

If you continue to have issues after running these diagnostics:

1. Save the output from `check_production_db.py`
2. Check the application logs: `kubectl logs -n motm-app -l app.kubernetes.io/name=motm-app --tail=100`
3. Verify the database connection is working
4. Check if there are any database permission issues