gochat/docs/ops/02-production-operations.md

Production operations runbooks

All commands run from the repository root. Set COMPOSE='docker compose -f deploy/docker/docker-compose.prod.yml' and load the approved production environment before use. Never paste secrets into tickets or terminal transcripts.

Install and preflight

• Owner: release engineer; SRE on-call approves the target host.
• Prerequisites: Docker Compose 2.26+, immutable GoChat/Connector digests, external backup mount, untracked backup passphrase and Alertmanager HTTPS webhook files. For external PostgreSQL, set one complete GOCHAT_DATABASE_DSN with exactly one sslmode=verify-ca|verify-full. When using a private CA or client certificate, keep the DSN paths fixed at /run/secrets/external-db-ca.crt, /run/secrets/external-db-client.crt, and /run/secrets/external-db-client.key; set the three corresponding untracked host-file variables plus GOCHAT_DATABASE_TLS_GID, assign that group to the files, and grant group read permission (0640 for the key). Compose mounts the files read-only and adds the group to every database client.
• Commands: deploy/docker/preflight.sh .env; then $COMPOSE config --quiet; then $COMPOSE --profile ops run --rm migrate; finally $COMPOSE up -d --wait.
• Success / failure: success means preflight exits 0, every non-profile service is healthy/running, and rendered gochat, migrate, backup, restore, and postgres-exporter configuration uses the same database DSN; any mutable image, missing secret file, DSN mismatch, migration error, or unhealthy service is failure.
• Rollback: $COMPOSE down leaves named data volumes intact; if migration ran, follow 01-rolling-upgrade.md and restore the preflight backup instead of running schema down.
• Drill cadence: before every release and quarterly on a clean host.

TLS and reverse proxy

• Owner: SRE on-call.
• Prerequisites: approved DNS, certificate and key in the secret manager; only 127.0.0.1:${GOCHAT_PORT} is exposed by Compose.
• Commands: configure the platform proxy to terminate TLS, proxy HTTP/WebSocket traffic to 127.0.0.1:${GOCHAT_PORT}, set X-Forwarded-Proto https, and run curl --fail --proto '=https' --tlsv1.2 https://$GOCHAT_HOST/health plus a WebSocket handshake through the public hostname.
• Success / failure: success is a trusted certificate, HTTP-to-HTTPS redirect, healthy JSON and a 101 WebSocket response; direct public access to ports 3000, 9090, 9093 or 24224 is failure.
• Rollback: restore the previous proxy configuration and certificate, reload the proxy, then repeat the probes.
• Drill cadence: on certificate/proxy changes and monthly certificate-expiry review.

Scale out and in

• Owner: SRE on-call; application owner approves capacity.
• Prerequisites: external PostgreSQL/Redis/Meilisearch endpoints, shared attachment storage for multi-host web replicas, and a reverse proxy target per host. A single Compose host may scale workers directly.
• Commands: worker scale-out: $COMPOSE up -d --scale worker=2 worker; confirm both with $COMPOSE ps worker. For web scale-out, provision another preflighted host with the same immutable digest and shared dependencies, add https://<new-host>/ready to the load balancer, then drain/remove the old target. Scale in only after gochat_background_jobs_total and in-flight requests are stable.
• Success / failure: success is all replicas present, readiness 200, no duplicate/lost jobs and stable queue age; any readiness failure, storage mismatch or rising backlog is failure.
• Rollback: remove the new load-balancer target and restore the previous worker replica count.
• Drill cadence: semi-annually and before forecast traffic peaks.

Capacity thresholds

• Owner: SRE on-call.
• Prerequisites: Prometheus targets are up and at least seven days of representative data exist.
• Commands: review http_request_duration_seconds, gochat_go_memory_alloc_bytes, redis_memory_used_bytes / redis_memory_max_bytes, pg_stat_activity_count / pg_settings_max_connections, gochat_background_jobs_total, Fluentd buffer use and filesystem free space. Increase capacity before sustained 70%; page at the repository rules' 80%, 400 MB, 25 critical jobs, five-minute critical queue age and 15-minute running-job age thresholds.
• Success / failure: success is 30% headroom and no firing capacity alert; missing series or sustained warning thresholds are failure.
• Rollback: revert the last resource/replica change if latency, restarts or errors worsen, and return traffic to the prior capacity.
• Drill cadence: weekly review; threshold calibration quarterly.

Secret and key rotation

• Owner: security on-call with SRE operator.
• Prerequisites: two-person approval, new values in the secret manager, verified backup and recorded old image digest; never print either value.
• Commands: for JWT, set the new GOCHAT_JWT_SECRET, put the old value in GOCHAT_JWT_PREVIOUS_SECRETS, run preflight and roll web/worker; after the maximum token lifetime, clear the previous value and roll again. Rotate Connector/Alertmanager credentials by updating their secret files/variables and recreating only affected services. Rotate the backup passphrase only after creating and restoring one bundle with the new passphrase; retain the old key until old bundles expire.
• Success / failure: new credentials work, old JWTs work only during the bounded overlap, revoked credentials fail, logs contain no secret and backup restore succeeds; otherwise failure.
• Rollback: restore the previous secret version and immutable image, recreate affected services, and invalidate the failed new credential.
• Drill cadence: quarterly and immediately after suspected exposure.

Incident response

• Owner: primary SRE on-call; incident commander for critical incidents.
• Prerequisites: alert payload, dashboard access, immutable deployment record and this repository checkout.
• Commands: acknowledge the alert; record UTC start time; run $COMPOSE ps, curl -fsS http://127.0.0.1:9090/api/v1/alerts, and $COMPOSE logs --since 15m; classify dependency, application, queue or capacity failure; mitigate with the matching runbook; record each command and result without secrets.
• Success / failure: success is customer impact stopped, alert resolved and evidence preserved; recurring/reopened alerts or unknown data integrity are failure and require escalation.
• Rollback: revert the most recent change or isolate the failed dependency; never run ad-hoc schema down.
• Drill cadence: quarterly game day and after each Sev-1/Sev-2 retrospective.

On-call escalation

• Owner: primary SRE on-call.
• Prerequisites: current contact rota and severity policy in the paging system.
• Commands: acknowledge warning within 15 minutes and critical within 5; page secondary SRE after 5 unacknowledged minutes; page application/data owner for 15 minutes of unresolved impact; appoint incident commander and notify product owner at 30 minutes or on confirmed data/security impact.
• Success / failure: success is a named owner, incident channel, next update time and acknowledged handoff; no acknowledgement inside the window is failure.
• Rollback: if escalation was false, resolve the page with reason and tune only after evidence review.
• Drill cadence: monthly paging test and quarterly rota failover.

Log search and retention

• Owner: SRE on-call; security approves retention.
• Prerequisites: fluentd running, fluentd_logs and fluentd_buffer below 70% capacity.
• Commands: search recent compressed output with $COMPOSE exec -T fluentd sh -c 'zgrep -h "request_id_or_error" /fluentd/log/gochat*.gz'; preserve incident files before cleanup; delete only flushed files older than the approved 14-day baseline with $COMPOSE exec -T fluentd sh -c 'find /fluentd/log -type f -name "gochat*.gz" -mtime +14 -delete'.
• Success / failure: success is searchable JSON containing service and environment, with disk below 70%; missing incident logs, full buffers or deletion inside retention are failure.
• Rollback: restore preserved logs from the incident archive and stop cleanup scheduling while retention is investigated.
• Drill cadence: weekly capacity/retention check and quarterly search drill.

Backup and restore

• Owner: database SRE.
• Prerequisites: dedicated local/off-site mounts, passphrase file, Connector online backup and clean isolated restore project.
• Commands: follow 01-rolling-upgrade.md daily backup and clean-environment restore commands; verify gochat_backup_last_success_timestamp_seconds is fresh in Prometheus.
• Success / failure: success is verified encryption/checksums, off-site copy, RPO at most 24 hours and isolated RTO at most 4 hours; any missing metric, checksum/decrypt error or non-empty restore target is failure.
• Rollback: do not overwrite production; discard the isolated target, restore mount/key access, and retry from the last verified bundle.
• Drill cadence: backup daily; non-author restore rehearsal quarterly.

Upgrade and rollback

• Owner: release engineer with database SRE.
• Prerequisites: approved immutable old/new digests, green release gate, verified backup/restore and maintenance window.
• Commands: execute 01-rolling-upgrade.md preflight, one-shot migration, post-check and application rollout exactly as written.
• Success / failure: success is one migration run, clean schema version, ready web/worker and unchanged reconciliation counts; any dirty migration, lock timeout or failed health check is failure.
• Rollback: redeploy the recorded old application digest; if schema is unusable, stop writes and restore the verified bundle into a clean target. Never run production migrate down.
• Drill cadence: every release in staging; quarterly old-digest rollback drill.

Planned shutdown

• Owner: incident commander or release engineer.
• Prerequisites: approved change window, customer notice, verified backup and rollback owner.
• Commands: remove web from the load balancer; wait for active requests; run $COMPOSE stop -t 35 gochat worker shangwutong; stop monitoring last with $COMPOSE stop prometheus alertmanager fluentd; leave data services running unless full host maintenance requires them stopped.
• Success / failure: success is graceful exit inside timeout, no running jobs left locked and no data-volume deletion; forced kill or new accepted traffic is failure.
• Rollback: $COMPOSE up -d --wait, restore the load-balancer target only after readiness succeeds.
• Drill cadence: semi-annually.

Disaster failover

• Owner: incident commander and database SRE.
• Prerequisites: declared disaster, last verified off-site bundle, alternate host/network/DNS, secret-manager access and immutable image digests.
• Commands: fence the failed site; restore to a clean alternate environment via 01-rolling-upgrade.md; validate schema/accounts/attachments/Connector; start monitoring, dependencies, web/worker; lower DNS/load-balancer TTL and shift traffic only after /ready and smoke pass.
• Success / failure: success is a single writable site, RPO/RTO recorded, alerts/logs working and customer smoke passing; split brain, checksum mismatch or failed reconciliation is failure.
• Rollback: stop the alternate writers and return traffic only if the original site is proven authoritative and reconciled; otherwise keep it fenced.
• Drill cadence: semi-annual full failover and quarterly tabletop.