Add comprehensive production deployment manual guide

PRODUCTION_DEPLOYMENT.md

@@ -0,0 +1,602 @@
+# SmoothSchedule - Production Deployment Manual
+
+Complete step-by-step guide for manually deploying SmoothSchedule from scratch on a production server. This guide is useful when you need to reset the entire production deployment or troubleshoot deployment issues.
+
+## Table of Contents
+
+1. [Prerequisites](#prerequisites)
+2. [Complete Fresh Deployment](#complete-fresh-deployment)
+3. [Docker Build & Startup](#docker-build--startup)
+4. [Database Initialization](#database-initialization)
+5. [Static Files & Migrations](#static-files--migrations)
+6. [Verification Steps](#verification-steps)
+7. [Troubleshooting](#troubleshooting)
+
+---
+
+## Prerequisites
+
+### Required on Production Server
+
+```bash
+# Check these are installed and running
+docker --version              # Docker 20.10+
+docker compose --version      # Docker Compose 2.0+
+```
+
+### Required Locally (Before Deployment)
+
+1. All code changes committed to git (`main` branch)
+2. All missing files added to git and pushed
+3. Production environment variables backed up locally
+4. `.envs/.production/` directory exists with proper credentials
+
+---
+
+## Complete Fresh Deployment
+
+### Step 1: Save Production Environment Variables Locally
+
+**On your local development machine:**
+
+```bash
+# Backup current production environment variables
+scp poduck@smoothschedule.com:~/smoothschedule/smoothschedule/.envs/.production/.django \
+    /tmp/production_django_env_backup
+
+scp poduck@smoothschedule.com:~/smoothschedule/smoothschedule/.envs/.production/.postgres \
+    /tmp/production_postgres_env_backup
+```
+
+### Step 2: Prepare & Commit All Code Changes Locally
+
+**On your local development machine:**
+
+```bash
+cd /home/poduck/Desktop/smoothschedule2
+
+# Check for any missing files
+git status
+
+# Add all changes and new files
+git add -A
+
+# Review staged changes
+git diff --cached
+
+# Commit changes
+git commit -m "Production deployment: Add missing files and config updates"
+
+# Push to main branch
+git push origin main
+```
+
+### Step 3: Bring Down Production Containers
+
+**SSH into production server:**
+
+```bash
+ssh poduck@smoothschedule.com
+
+cd ~/smoothschedule/smoothschedule
+
+# Stop all containers (preserves volumes)
+docker compose -f docker-compose.production.yml down
+
+# Or to completely reset and remove volumes (careful!):
+docker compose -f docker-compose.production.yml down -v
+```
+
+### Step 4: Remove Production Codebase
+
+**On production server:**
+
+```bash
+cd ~
+
+# Remove everything
+rm -rf smoothschedule
+```
+
+### Step 5: Pull Fresh Code from Git
+
+**On production server:**
+
+```bash
+cd ~
+
+# Clone the repository
+git clone https://git.talova.net/poduck/smoothschedule.git smoothschedule
+
+cd smoothschedule
+```
+
+**Note:** If git authentication fails, configure credentials on the server:
+
+```bash
+git config --global user.email "poduck@smoothschedule.com"
+git config --global user.name "Poduck"
+
+# Or store credentials
+git credential approve
+# Then paste: protocol=https
+#            host=git.talova.net
+#            username=poduck
+#            password=chaff/starry
+```
+
+### Step 6: Restore Production Environment Variables
+
+**On production server:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# Create the .envs/.production directory if it doesn't exist
+mkdir -p .envs/.production
+
+# Restore from backups on local machine or recreate them
+# Option A: Use SCP to copy from local machine
+scp /tmp/production_django_env_backup poduck@smoothschedule.com:~/smoothschedule/smoothschedule/.envs/.production/.django
+scp /tmp/production_postgres_env_backup poduck@smoothschedule.com:~/smoothschedule/smoothschedule/.envs/.production/.postgres
+
+# Option B: Manually recreate the files on the production server
+# (see Environment Variables section below)
+```
+
+---
+
+## Docker Build & Startup
+
+### Step 7: Build Docker Images
+
+**On production server:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# Build all production Docker images
+docker compose -f docker-compose.production.yml build
+```
+
+**Expected output:**
+```
+✓ Successfully built:
+  - smoothschedule_production_django
+  - smoothschedule_production_celeryworker
+  - smoothschedule_production_celerybeat
+  - smoothschedule_production_flower
+  - smoothschedule_production_postgres
+  - smoothschedule_production_traefik
+  - smoothschedule-awscli
+```
+
+### Step 8: Start Containers
+
+**On production server:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# Start all containers in detached mode
+docker compose -f docker-compose.production.yml up -d
+
+# Wait for services to stabilize (10-15 seconds)
+sleep 15
+
+# Check container status
+docker compose -f docker-compose.production.yml ps
+```
+
+**Expected status:** All containers should be `Up` or `Healthy`
+
+---
+
+## Database Initialization
+
+### Step 9: Run Database Migrations
+
+**On production server:**
+
+The application uses django-tenants for multi-tenant support. Migrations must be run in this order:
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# 1. Migrate the public/shared schema (all shared apps)
+docker compose -f docker-compose.production.yml exec -T django \
+  python manage.py migrate_schemas --shared
+
+# Expected output should show multiple migrations applied to "public" schema
+```
+
+**If you get "service django is not running":**
+
+The Django container may not be fully started. Wait a bit longer:
+
+```bash
+sleep 30  # Wait for Django to initialize
+docker compose -f docker-compose.production.yml exec -T django \
+  python manage.py migrate_schemas --shared
+```
+
+---
+
+## Static Files & Migrations
+
+### Step 10: Collect Static Files
+
+**On production server:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+docker compose -f docker-compose.production.yml exec -T django \
+  python manage.py collectstatic --noinput
+```
+
+This collects all static files (CSS, JS, images) and uploads them to:
+- **Local filesystem:** `smoothschedule/staticfiles/`
+- **DigitalOcean Spaces/S3:** Configured via `DJANGO_AWS_*` environment variables
+
+### Step 11: Create Superuser (First Time Only)
+
+**On production server, if this is a fresh installation:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+docker compose -f docker-compose.production.yml exec django \
+  python manage.py createsuperuser
+
+# Follow the prompts to create the admin user
+```
+
+### Step 12: Create Initial Tenant (First Time Only)
+
+**On production server, if you need a demo tenant:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+docker compose -f docker-compose.production.yml exec django python manage.py shell
+```
+
+Then in the Django shell:
+
+```python
+from core.models import Tenant, Domain
+from django.utils.text import slugify
+
+# Create a tenant
+tenant = Tenant.objects.create(
+    name="Demo Company",
+    slug="demo",
+    is_free_trial=False,
+    is_temporary=False,
+)
+
+# Create a domain for the tenant
+Domain.objects.create(
+    domain="demo.smoothschedule.com",
+    tenant=tenant,
+)
+
+print(f"Created tenant: {tenant.name} with domain: demo.smoothschedule.com")
+exit()
+```
+
+---
+
+## Verification Steps
+
+### Step 13: Verify All Services Are Running
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# Check container status
+docker compose -f docker-compose.production.yml ps
+
+# View logs for any errors
+docker compose -f docker-compose.production.yml logs --tail=50
+
+# Check specific service logs
+docker compose -f docker-compose.production.yml logs django --tail=30
+docker compose -f docker-compose.production.yml logs postgres --tail=30
+```
+
+### Step 14: Test API Endpoints
+
+**From your local machine:**
+
+```bash
+# Test the backend API
+curl -s "https://smoothschedule.com/api/health/" | jq
+
+# Test tenant access
+curl -s "https://demo.smoothschedule.com/api/resources/" | jq
+
+# Test platform admin
+curl -s "https://platform.smoothschedule.com/api/admin/businesses/" | jq
+```
+
+### Step 15: Verify Nginx Frontend
+
+The frontend should be accessible at:
+- **Main site:** `https://smoothschedule.com`
+- **Platform dashboard:** `https://platform.smoothschedule.com`
+- **Tenant subdomain:** `https://demo.smoothschedule.com`
+
+---
+
+## Environment Variables Reference
+
+### Django Configuration (`.envs/.production/.django`)
+
+```bash
+# Security & Secrets
+DJANGO_SECRET_KEY=your-secret-key-here
+DJANGO_DEBUG=False
+DJANGO_ALLOWED_HOSTS=smoothschedule.com,platform.smoothschedule.com,*.smoothschedule.com
+
+# Admin & URLs
+DJANGO_ADMIN_URL=<random-slug>/
+FRONTEND_URL=https://platform.smoothschedule.com
+PLATFORM_BASE_URL=https://platform.smoothschedule.com
+
+# Celery & Redis
+REDIS_URL=redis://redis:6379/0
+CELERY_BROKER_URL=redis://redis:6379/0
+
+# AWS/DigitalOcean Spaces (for media storage)
+DJANGO_AWS_ACCESS_KEY_ID=your-access-key
+DJANGO_AWS_SECRET_ACCESS_KEY=your-secret-key
+DJANGO_AWS_STORAGE_BUCKET_NAME=smoothschedule
+DJANGO_AWS_S3_ENDPOINT_URL=https://nyc3.digitaloceanspaces.com
+DJANGO_AWS_S3_REGION_NAME=nyc3
+DJANGO_AWS_S3_CUSTOM_DOMAIN=smoothschedule.nyc3.digitaloceanspaces.com
+
+# Email Configuration (optional)
+MAILGUN_API_KEY=your-mailgun-key
+MAILGUN_DOMAIN=mg.smoothschedule.com
+
+# SSL/Security
+DJANGO_SECURE_SSL_REDIRECT=True
+DJANGO_SECURE_HSTS_SECONDS=60
+DJANGO_SECURE_HSTS_INCLUDE_SUBDOMAINS=True
+DJANGO_SECURE_HSTS_PRELOAD=True
+DJANGO_SESSION_COOKIE_SECURE=True
+DJANGO_CSRF_COOKIE_SECURE=True
+
+# Sentry (optional - error tracking)
+SENTRY_DSN=
+SENTRY_ENVIRONMENT=production
+SENTRY_TRACES_SAMPLE_RATE=0.1
+```
+
+### PostgreSQL Configuration (`.envs/.production/.postgres`)
+
+```bash
+POSTGRES_HOST=postgres
+POSTGRES_PORT=5432
+POSTGRES_DB=smoothschedule
+POSTGRES_USER=<random-username>
+POSTGRES_PASSWORD=<random-secure-password>
+```
+
+---
+
+## Troubleshooting
+
+### Django Container Won't Start
+
+**Symptom:** `docker compose ps` shows Django as exited
+
+**Solution:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# View full error logs
+docker compose -f docker-compose.production.yml logs django --tail=100
+
+# Check if database is accessible
+docker compose -f docker-compose.production.yml logs postgres --tail=20
+
+# Restart Django after fixing the issue
+docker compose -f docker-compose.production.yml restart django
+```
+
+### Migration Fails with "role does not exist"
+
+**Symptom:** Error when running `migrate_schemas --shared`:
+```
+FATAL: role "postgres" does not exist
+```
+
+**Solution:** The database volume is corrupted. Reset and restart:
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# Stop and remove all volumes
+docker compose -f docker-compose.production.yml down -v
+
+# Restart (this will recreate the database)
+docker compose -f docker-compose.production.yml up -d
+
+# Wait for database to initialize (30 seconds)
+sleep 30
+
+# Try migrations again
+docker compose -f docker-compose.production.yml exec -T django \
+  python manage.py migrate_schemas --shared
+```
+
+### Cannot Connect to Production Server
+
+**Check SSH access:**
+
+```bash
+ssh poduck@smoothschedule.com "echo 'Connection successful'"
+```
+
+**If that fails:**
+- Verify your SSH key is authorized on the server
+- Check firewall rules allow SSH (port 22)
+- Verify DNS resolves `smoothschedule.com` to the correct IP
+
+### Traefik Certificate Issues
+
+**Symptom:** SSL certificate errors, HTTP redirects failing
+
+**Solution:**
+
+```bash
+cd ~/smoothschedule/smoothschedule
+
+# Check Traefik logs
+docker compose -f docker-compose.production.yml logs traefik --tail=50
+
+# Restart Traefik to request new certificates
+docker compose -f docker-compose.production.yml restart traefik
+
+# Wait for Let's Encrypt to issue certificates (5-10 minutes)
+# Check via: https://smoothschedule.com (should show valid cert)
+```
+
+### Frontend Not Loading
+
+**Verify nginx container is running:**
+
+```bash
+# Check if nginx image was built
+docker images | grep smoothschedule
+
+# If missing, rebuild
+docker compose -f docker-compose.production.yml build nginx
+
+# Restart the service
+docker compose -f docker-compose.production.yml restart
+
+# Check logs
+docker logs $(docker ps -q --filter "name=nginx")
+```
+
+---
+
+## Quick Reference Commands
+
+```bash
+# SSH to production server
+ssh poduck@smoothschedule.com
+
+# Navigate to project
+cd ~/smoothschedule/smoothschedule
+
+# View all container logs
+docker compose -f docker-compose.production.yml logs -f
+
+# View Django logs only
+docker compose -f docker-compose.production.yml logs django -f --tail=100
+
+# Stop all containers
+docker compose -f docker-compose.production.yml down
+
+# Start all containers
+docker compose -f docker-compose.production.yml up -d
+
+# Restart a specific service
+docker compose -f docker-compose.production.yml restart django
+
+# Run a Django management command
+docker compose -f docker-compose.production.yml exec -T django python manage.py <command>
+
+# Access Django shell
+docker compose -f docker-compose.production.yml exec django python manage.py shell
+
+# Create a database backup
+docker compose -f docker-compose.production.yml exec -T postgres \
+  pg_dump -U $POSTGRES_USER smoothschedule > backup.sql
+
+# Execute arbitrary SQL
+docker compose -f docker-compose.production.yml exec -T postgres \
+  psql -U $POSTGRES_USER smoothschedule -c "SELECT version();"
+
+# Check Docker resource usage
+docker stats
+```
+
+---
+
+## Advanced: Rollback Procedure
+
+If a deployment fails catastrophically:
+
+```bash
+# 1. Stop everything
+cd ~/smoothschedule/smoothschedule
+docker compose -f docker-compose.production.yml down
+
+# 2. Restore from database backup (if available)
+docker compose -f docker-compose.production.yml exec -T postgres \
+  psql -U $POSTGRES_USER smoothschedule < backup.sql
+
+# 3. Checkout previous git commit
+cd ~/smoothschedule
+git checkout <previous-commit-hash>
+
+# 4. Rebuild and restart
+cd smoothschedule
+docker compose -f docker-compose.production.yml build
+docker compose -f docker-compose.production.yml up -d
+```
+
+---
+
+## Monitoring Checklist
+
+After deployment, monitor these for 24 hours:
+
+- [ ] No error logs in `docker compose logs django`
+- [ ] API endpoints respond with 200 status
+- [ ] SSL certificates are valid (https://smoothschedule.com)
+- [ ] Frontend loads without console errors
+- [ ] Tenant subdomains work correctly
+- [ ] Static files are being served (CSS, JS load)
+- [ ] Background tasks execute (check Celery worker logs)
+- [ ] No database connection errors
+
+---
+
+## Important Notes
+
+1. **Always backup before major changes:**
+   - Database: `pg_dump`
+   - Environment variables: Copy `.envs/.production/` locally
+   - Code: Git commits
+
+2. **Never modify code on production directly:**
+   - Change code locally → Git push → Production git pull
+   - Only edit `.env` files directly on production
+
+3. **Multi-Tenancy Considerations:**
+   - Each tenant gets a separate PostgreSQL schema
+   - Migrations apply to "public" schema (shared) first
+   - Tenant migrations happen automatically on first request to new tenant
+
+4. **Docker Compose File:**
+   - Always use `-f docker-compose.production.yml` for production
+   - Never use `-f docker-compose.local.yml` on production
+
+5. **Persistence:**
+   - Database: `production_postgres_data` volume
+   - Redis: `production_redis_data` volume
+   - Traefik certs: `production_traefik` volume
+   - These are preserved when using `down` (not `down -v`)
+
+---
+
+**Last Updated:** December 2025
+**Deployment Version:** Manual v1.0