gochat/docs/rolling_upgrade.md

GoChat Rolling Upgrade Strategy

Reference: Chatwoot deployment uses zero-downtime upgrade pattern

Overview

GoChat follows a blue-green deployment strategy for production upgrades, ensuring zero downtime during version transitions.

Upgrade Process

Step 1: Pre-flight Checks

1. Verify new Docker image is built and pushed: docker pull gochat/gochat:${NEW_VERSION}
2. Run database migrations on a staging environment first
3. Verify backward compatibility of migrations (new code must work with old schema)
4. Check feature flags — new features should be disabled by default

Step 2: Database Migration

# Run migrations BEFORE deploying new code
# Migrations must be backward-compatible
docker compose -f docker-compose.prod.yml exec gochat /app/gochat migrate up

Step 3: Blue-Green Deployment (Docker Compose)

# 1. Deploy new version as "green" alongside "blue" (current)
docker compose -f docker-compose.prod.yml up -d --no-deps gochat-green

# 2. Wait for health check to pass
curl -f http://gochat-green:3000/health

# 3. Switch traffic (update nginx/upstream config)
# nginx: switch upstream from blue to green

# 4. Drain old connections on blue
# Wait 30s for in-flight requests to complete

# 5. Stop blue
docker compose -f docker-compose.prod.yml stop gochat

Step 4: K8s Rolling Update

# Kubernetes handles rolling updates automatically via Deployment strategy
kubectl set image deployment/gochat gochat=gochat/gochat:${NEW_VERSION}
kubectl rollout status deployment/gochat

K8s config (in Helm chart):

• maxSurge: 1 (one extra pod during rollout)
• maxUnavailable: 0 (no pods allowed to be unavailable)
• readinessProbe: /ready endpoint
• livenessProbe: /live endpoint

Step 5: Verification

1. Smoke test: hit /health, /ready endpoints
2. Check logs for errors: docker compose logs gochat --since 5m
3. Verify metrics: Prometheus dashboard should show normal traffic
4. Monitor for 15 minutes before finalizing

Step 6: Rollback (if needed)

# Docker Compose rollback
docker compose -f docker-compose.prod.yml exec gochat /app/gochat migrate down ${N}
docker compose -f docker-compose.prod.yml up -d --no-deps gochat-${OLD_VERSION}

# K8s rollback
kubectl rollout undo deployment/gochat

Migration Compatibility Rules

• Migrations MUST be additive only in production (add columns, never remove)
• Column removals require a 2-phase migration: soft-remove then hard-remove
• New columns should have defaults or be nullable
• Renames require a 3-phase migration: add new → copy data → remove old

Worker Upgrade

Workers drain naturally: set a shutdown deadline, let in-flight jobs finish, then stop. New workers pick up queued jobs from Redis.