gochat/docs/ops/02-production-operations.md

# Production operations runbooks

All commands run from the repository root. Set `COMPOSE='docker compose -f deploy/docker/docker-compose.prod.yml'` and load the approved production environment before use. Never paste secrets into tickets or terminal transcripts.

## Install and preflight

- **Owner:** release engineer; SRE on-call approves the target host.
- **Prerequisites:** Docker Compose 2.26+, immutable GoChat/Connector digests, external backup mount, untracked backup passphrase and Alertmanager HTTPS webhook files. Create the required `deploy/docker/data/` subdirectories before startup. For external PostgreSQL, set one complete `GOCHAT_DATABASE_DSN` with exactly one `sslmode`: use `disable` only on a trusted private network, or use `verify-ca`/`verify-full` for TLS. Redis may use unauthenticated `redis://` only on a trusted private Docker network; use authenticated `rediss://` when traffic crosses an untrusted network. When PostgreSQL uses a private CA or client certificate, keep the DSN paths fixed at `/run/secrets/external-db-ca.crt`, `/run/secrets/external-db-client.crt`, and `/run/secrets/external-db-client.key`; set the three corresponding untracked host-file variables plus `GOCHAT_DATABASE_TLS_GID`, assign that group to the files, and grant group read permission (`0640` for the key). Compose mounts the files read-only and adds the group to every database client.
- **Commands:** `deploy/docker/preflight.sh .env`; then `$COMPOSE config --quiet`; then `$COMPOSE --profile ops run --rm migrate`; finally `$COMPOSE up -d --wait`.
- **Success / failure:** success means preflight exits 0, every non-profile service is healthy/running, and rendered `gochat`, `migrate`, `backup`, `restore`, and `postgres-exporter` configuration uses the same database DSN; any mutable image, missing secret file, DSN mismatch, migration error, or unhealthy service is failure.
- **Rollback:** `$COMPOSE down` leaves `deploy/docker/data/` intact; if migration ran, follow `01-rolling-upgrade.md` and restore the preflight backup instead of running schema down.
- **Drill cadence:** before every release and quarterly on a clean host.

## TLS and reverse proxy

- **Owner:** SRE on-call.
- **Prerequisites:** approved DNS, certificate and key in the secret manager; only `127.0.0.1:${GOCHAT_PORT}` is exposed by Compose.
- **Commands:** configure the platform proxy to terminate TLS, proxy HTTP/WebSocket traffic to `127.0.0.1:${GOCHAT_PORT}`, set `X-Forwarded-Proto https`, and run `curl --fail --proto '=https' --tlsv1.2 https://$GOCHAT_HOST/health` plus a WebSocket handshake through the public hostname.
- **Success / failure:** success is a trusted certificate, HTTP-to-HTTPS redirect, healthy JSON and a `101` WebSocket response; direct public access to ports 3000, 9090, 9093 or 24224 is failure.
- **Rollback:** restore the previous proxy configuration and certificate, reload the proxy, then repeat the probes.
- **Drill cadence:** on certificate/proxy changes and monthly certificate-expiry review.

## Scale out and in

- **Owner:** SRE on-call; application owner approves capacity.
- **Prerequisites:** external PostgreSQL/Redis/Meilisearch endpoints, shared attachment storage for multi-host web replicas, and a reverse proxy target per host. A single Compose host may scale workers directly.
- **Commands:** worker scale-out: `$COMPOSE up -d --scale worker=2 worker`; confirm both with `$COMPOSE ps worker`. For web scale-out, provision another preflighted host with the same immutable digest and shared dependencies, add `https://<new-host>/ready` to the load balancer, then drain/remove the old target. Scale in only after `gochat_background_jobs_total` and in-flight requests are stable.
- **Success / failure:** success is all replicas present, readiness 200, no duplicate/lost jobs and stable queue age; any readiness failure, storage mismatch or rising backlog is failure.
- **Rollback:** remove the new load-balancer target and restore the previous worker replica count.
- **Drill cadence:** semi-annually and before forecast traffic peaks.

## Capacity thresholds

- **Owner:** SRE on-call.
- **Prerequisites:** Prometheus targets are up and at least seven days of representative data exist.
- **Commands:** review `http_request_duration_seconds`, `gochat_go_memory_alloc_bytes`, `redis_memory_used_bytes / redis_memory_max_bytes`, `pg_stat_activity_count / pg_settings_max_connections`, `gochat_background_jobs_total`, Fluentd buffer use and filesystem free space. Increase capacity before sustained 70%; page at the repository rules' 80%, 400 MB, 25 critical jobs, five-minute critical queue age and 15-minute running-job age thresholds.
- **Success / failure:** success is 30% headroom and no firing capacity alert; missing series or sustained warning thresholds are failure.
- **Rollback:** revert the last resource/replica change if latency, restarts or errors worsen, and return traffic to the prior capacity.
- **Drill cadence:** weekly review; threshold calibration quarterly.

## Secret and key rotation

- **Owner:** security on-call with SRE operator.
- **Prerequisites:** two-person approval, new values in the secret manager, verified backup and recorded old image digest; never print either value.
- **Commands:** for JWT, set the new `GOCHAT_JWT_SECRET`, put the old value in `GOCHAT_JWT_PREVIOUS_SECRETS`, run preflight and roll web/worker; after the maximum token lifetime, clear the previous value and roll again. Rotate Connector/Alertmanager credentials by updating their secret files/variables and recreating only affected services. Rotate the backup passphrase only after creating and restoring one bundle with the new passphrase; retain the old key until old bundles expire.
- **Success / failure:** new credentials work, old JWTs work only during the bounded overlap, revoked credentials fail, logs contain no secret and backup restore succeeds; otherwise failure.
- **Rollback:** restore the previous secret version and immutable image, recreate affected services, and invalidate the failed new credential.
- **Drill cadence:** quarterly and immediately after suspected exposure.

## Incident response

- **Owner:** primary SRE on-call; incident commander for critical incidents.
- **Prerequisites:** alert payload, dashboard access, immutable deployment record and this repository checkout.
- **Commands:** acknowledge the alert; record UTC start time; run `$COMPOSE ps`, `curl -fsS http://127.0.0.1:9090/api/v1/alerts`, and `$COMPOSE logs --since 15m`; classify dependency, application, queue or capacity failure; mitigate with the matching runbook; record each command and result without secrets.
- **Success / failure:** success is customer impact stopped, alert resolved and evidence preserved; recurring/reopened alerts or unknown data integrity are failure and require escalation.
- **Rollback:** revert the most recent change or isolate the failed dependency; never run ad-hoc schema down.
- **Drill cadence:** quarterly game day and after each Sev-1/Sev-2 retrospective.

## On-call escalation

- **Owner:** primary SRE on-call.
- **Prerequisites:** current contact rota and severity policy in the paging system.
- **Commands:** acknowledge warning within 15 minutes and critical within 5; page secondary SRE after 5 unacknowledged minutes; page application/data owner for 15 minutes of unresolved impact; appoint incident commander and notify product owner at 30 minutes or on confirmed data/security impact.
- **Success / failure:** success is a named owner, incident channel, next update time and acknowledged handoff; no acknowledgement inside the window is failure.
- **Rollback:** if escalation was false, resolve the page with reason and tune only after evidence review.
- **Drill cadence:** monthly paging test and quarterly rota failover.

## Log search and retention

- **Owner:** SRE on-call; security approves retention.
- **Prerequisites:** `fluentd` running, `fluentd_logs` and `fluentd_buffer` below 70% capacity.
- **Commands:** search recent compressed output with `$COMPOSE exec -T fluentd sh -c 'zgrep -h "request_id_or_error" /fluentd/log/gochat*.gz'`; preserve incident files before cleanup; delete only flushed files older than the approved 14-day baseline with `$COMPOSE exec -T fluentd sh -c 'find /fluentd/log -type f -name "gochat*.gz" -mtime +14 -delete'`.
- **Success / failure:** success is searchable JSON containing `service` and `environment`, with disk below 70%; missing incident logs, full buffers or deletion inside retention are failure.
- **Rollback:** restore preserved logs from the incident archive and stop cleanup scheduling while retention is investigated.
- **Drill cadence:** weekly capacity/retention check and quarterly search drill.

## Backup and restore

- **Owner:** database SRE.
- **Prerequisites:** dedicated local/off-site mounts, passphrase file, Connector online backup and clean isolated restore project.
- **Commands:** follow `01-rolling-upgrade.md` daily backup and clean-environment restore commands; verify `gochat_backup_last_success_timestamp_seconds` is fresh in Prometheus.
- **Success / failure:** success is verified encryption/checksums, off-site copy, RPO at most 24 hours and isolated RTO at most 4 hours; any missing metric, checksum/decrypt error or non-empty restore target is failure.
- **Rollback:** do not overwrite production; discard the isolated target, restore mount/key access, and retry from the last verified bundle.
- **Drill cadence:** backup daily; non-author restore rehearsal quarterly.

## Upgrade and rollback

- **Owner:** release engineer with database SRE.
- **Prerequisites:** approved immutable old/new digests, green release gate, verified backup/restore and maintenance window.
- **Commands:** execute `01-rolling-upgrade.md` preflight, one-shot migration, post-check and application rollout exactly as written.
- **Success / failure:** success is one migration run, clean schema version, ready web/worker and unchanged reconciliation counts; any dirty migration, lock timeout or failed health check is failure.
- **Rollback:** redeploy the recorded old application digest; if schema is unusable, stop writes and restore the verified bundle into a clean target. Never run production `migrate down`.
- **Drill cadence:** every release in staging; quarterly old-digest rollback drill.

## Planned shutdown

- **Owner:** incident commander or release engineer.
- **Prerequisites:** approved change window, customer notice, verified backup and rollback owner.
- **Commands:** remove web from the load balancer; wait for active requests; run `$COMPOSE stop -t 35 gochat worker shangwutong`; stop monitoring last with `$COMPOSE stop prometheus alertmanager fluentd`; leave data services running unless full host maintenance requires them stopped.
- **Success / failure:** success is graceful exit inside timeout, no running jobs left locked and no data-volume deletion; forced kill or new accepted traffic is failure.
- **Rollback:** `$COMPOSE up -d --wait`, restore the load-balancer target only after readiness succeeds.
- **Drill cadence:** semi-annually.

## Disaster failover

- **Owner:** incident commander and database SRE.
- **Prerequisites:** declared disaster, last verified off-site bundle, alternate host/network/DNS, secret-manager access and immutable image digests.
- **Commands:** fence the failed site; restore to a clean alternate environment via `01-rolling-upgrade.md`; validate schema/accounts/attachments/Connector; start monitoring, dependencies, web/worker; lower DNS/load-balancer TTL and shift traffic only after `/ready` and smoke pass.
- **Success / failure:** success is a single writable site, RPO/RTO recorded, alerts/logs working and customer smoke passing; split brain, checksum mismatch or failed reconciliation is failure.
- **Rollback:** stop the alternate writers and return traffic only if the original site is proven authoritative and reconciled; otherwise keep it fenced.
- **Drill cadence:** semi-annual full failover and quarterly tabletop.