rogee/gochat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HH-445: deploy production observability and runbooks (#96)

Browse Source 
* HH-445: deploy production observability and runbooks

* fix(ops): share production database DSN

* fix(HH-445): enforce database TLS gate

* fix(HH-445): preserve production serve command

* fix(prod): require external database dependencies

* fix(prod): unify database host rejection gates

* test(prod): enforce exact database TLS runbook contract

---------

Co-authored-by: Rogee <rogee@ipao.vip>
This commit is contained in:
Rogee
2026-08-22 19:39:57 +08:00
committed by GitHub
co-authored by rogee
parent 61376a57fd
commit fb83285617
28 changed files with 1622 additions and 220 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+2
View File
@@ -120,6 +120,8 @@ M01-M12 模块的 Chatwoot 功能梳理文档，基于 Chatwoot 源码深度阅
| 文档 | 说明 |
|------|------|
| [ops/01-rolling-upgrade.md](ops/01-rolling-upgrade.md) | 滚动升级策略 — 蓝绿部署、数据库迁移、健康检查 |
| [ops/02-production-operations.md](ops/02-production-operations.md) | 生产预检、TLS、容量、轮换、事故、日志、停服与灾备 runbooks |
| [ops/03-observability-drill.md](ops/03-observability-drill.md) | 告警/恢复/日志链路演练与非作者验收证据模板 |
---
docs/ops/01-rolling-upgrade.md
+16
View File
@@ -1,5 +1,11 @@
# Production backup, restore, and upgrade runbook
- **Owner:** release engineer and database SRE; a non-author SRE executes quarterly restore drills.
- **Prerequisites:** immutable old/new image digests, green release gate, isolated restore target, external off-site mount and secret-manager access.
- **Success / failure:** backup RPO is at most 24 hours, restore RTO at most 4 hours, migration is clean and all reconciliation/health checks pass; any mismatch or dirty migration fails the run.
- **Rollback:** redeploy the recorded old application digest; restore the verified bundle into a clean target if the schema is unusable. Never run production schema down.
- **Drill cadence:** backup daily, staging upgrade/rollback every release, clean restore quarterly.
Production web and worker processes never migrate on startup. The Compose file
also requires `GOCHAT_IMAGE_REF` to be an immutable `image@sha256:digest`; tags such
as `latest` are not an acceptable rollback record.
@@ -35,6 +41,10 @@ export GOCHAT_CONNECTOR_BACKUP_NAME="connector-$(date -u +%Y%m%dT%H%M%SZ).db"
Backup/restore are audited one-shot containers and run as root only to read or
rebuild Docker volumes; web and worker remain non-root.
For an external database using certificate files, keep the three DSN paths
fixed as documented in `.env.example`. Set the host files' group to
`GOCHAT_DATABASE_TLS_GID` and grant that group read permission (`0640` for the
private key); Compose adds this supplemental group to all six database clients.
`GOCHAT_BACKUP_OFFSITE_DIR` must be an existing external mount point provisioned
outside this Compose project. Set its approved source and filesystem type to the
exact values reported by `findmnt -M "$GOCHAT_BACKUP_OFFSITE_DIR"`; the preflight
@@ -47,6 +57,7 @@ encrypted bundle containing PostgreSQL, attachments, and that Connector copy:
```bash
docker compose -f deploy/docker/docker-compose.prod.yml exec shangwutong \
shangwutong backup --output "/backup/$GOCHAT_CONNECTOR_BACKUP_NAME"
deploy/docker/preflight.sh
docker compose -f deploy/docker/docker-compose.prod.yml --profile ops run --rm backup
```
@@ -62,6 +73,7 @@ Do not point this procedure at the live project.
```bash
export COMPOSE_PROJECT_NAME=gochat-restore-$(date +%Y%m%d)
export GOCHAT_RESTORE_BUNDLE='gochat-<timestamp>.tar.enc'
deploy/docker/preflight.sh
docker compose -f deploy/docker/docker-compose.prod.yml --profile ops run --rm restore
```
@@ -108,6 +120,7 @@ migration runners, but the deployment pipeline must contain only this one step.
```bash
export GOCHAT_IMAGE_REF="$NEW_IMAGE"
deploy/docker/preflight.sh
time docker compose -f deploy/docker/docker-compose.prod.yml --profile ops \
up --abort-on-container-exit --exit-code-from migrate migrate
docker compose -f deploy/docker/docker-compose.prod.yml logs migrate
@@ -115,6 +128,9 @@ docker compose -f deploy/docker/docker-compose.prod.yml up -d --no-deps gochat w
curl -fsS http://127.0.0.1:3000/health
```
Do not override the database client entrypoint or use `docker run` for these
operations; that bypasses the shared production DSN and certificate gate.
Post-migration checks:
```sql
docs/ops/02-production-operations.md
+111
View File
@@ -0,0 +1,111 @@
# Production operations runbooks
All commands run from the repository root. Set `COMPOSE='docker compose -f deploy/docker/docker-compose.prod.yml'` and load the approved production environment before use. Never paste secrets into tickets or terminal transcripts.
## Install and preflight
- **Owner:** release engineer; SRE on-call approves the target host.
- **Prerequisites:** Docker Compose 2.26+, immutable GoChat/Connector digests, external backup mount, untracked backup passphrase and Alertmanager HTTPS webhook files. For external PostgreSQL, set one complete `GOCHAT_DATABASE_DSN` with exactly one `sslmode=verify-ca|verify-full`. When using a private CA or client certificate, keep the DSN paths fixed at `/run/secrets/external-db-ca.crt`, `/run/secrets/external-db-client.crt`, and `/run/secrets/external-db-client.key`; set the three corresponding untracked host-file variables plus `GOCHAT_DATABASE_TLS_GID`, assign that group to the files, and grant group read permission (`0640` for the key). Compose mounts the files read-only and adds the group to every database client.
- **Commands:** `deploy/docker/preflight.sh .env`; then `$COMPOSE config --quiet`; then `$COMPOSE --profile ops run --rm migrate`; finally `$COMPOSE up -d --wait`.
- **Success / failure:** success means preflight exits 0, every non-profile service is healthy/running, and rendered `gochat`, `migrate`, `backup`, `restore`, and `postgres-exporter` configuration uses the same database DSN; any mutable image, missing secret file, DSN mismatch, migration error, or unhealthy service is failure.
- **Rollback:** `$COMPOSE down` leaves named data volumes intact; if migration ran, follow `01-rolling-upgrade.md` and restore the preflight backup instead of running schema down.
- **Drill cadence:** before every release and quarterly on a clean host.
## TLS and reverse proxy
- **Owner:** SRE on-call.
- **Prerequisites:** approved DNS, certificate and key in the secret manager; only `127.0.0.1:${GOCHAT_PORT}` is exposed by Compose.
- **Commands:** configure the platform proxy to terminate TLS, proxy HTTP/WebSocket traffic to `127.0.0.1:${GOCHAT_PORT}`, set `X-Forwarded-Proto https`, and run `curl --fail --proto '=https' --tlsv1.2 https://$GOCHAT_HOST/health` plus a WebSocket handshake through the public hostname.
- **Success / failure:** success is a trusted certificate, HTTP-to-HTTPS redirect, healthy JSON and a `101` WebSocket response; direct public access to ports 3000, 9090, 9093 or 24224 is failure.
- **Rollback:** restore the previous proxy configuration and certificate, reload the proxy, then repeat the probes.
- **Drill cadence:** on certificate/proxy changes and monthly certificate-expiry review.
## Scale out and in
- **Owner:** SRE on-call; application owner approves capacity.
- **Prerequisites:** external PostgreSQL/Redis/Meilisearch endpoints, shared attachment storage for multi-host web replicas, and a reverse proxy target per host. A single Compose host may scale workers directly.
- **Commands:** worker scale-out: `$COMPOSE up -d --scale worker=2 worker`; confirm both with `$COMPOSE ps worker`. For web scale-out, provision another preflighted host with the same immutable digest and shared dependencies, add `https://<new-host>/ready` to the load balancer, then drain/remove the old target. Scale in only after `gochat_background_jobs_total` and in-flight requests are stable.
- **Success / failure:** success is all replicas present, readiness 200, no duplicate/lost jobs and stable queue age; any readiness failure, storage mismatch or rising backlog is failure.
- **Rollback:** remove the new load-balancer target and restore the previous worker replica count.
- **Drill cadence:** semi-annually and before forecast traffic peaks.
## Capacity thresholds
- **Owner:** SRE on-call.
- **Prerequisites:** Prometheus targets are up and at least seven days of representative data exist.
- **Commands:** review `http_request_duration_seconds`, `gochat_go_memory_alloc_bytes`, `redis_memory_used_bytes / redis_memory_max_bytes`, `pg_stat_activity_count / pg_settings_max_connections`, `gochat_background_jobs_total`, Fluentd buffer use and filesystem free space. Increase capacity before sustained 70%; page at the repository rules' 80%, 400 MB, 25 critical jobs, five-minute critical queue age and 15-minute running-job age thresholds.
- **Success / failure:** success is 30% headroom and no firing capacity alert; missing series or sustained warning thresholds are failure.
- **Rollback:** revert the last resource/replica change if latency, restarts or errors worsen, and return traffic to the prior capacity.
- **Drill cadence:** weekly review; threshold calibration quarterly.
## Secret and key rotation
- **Owner:** security on-call with SRE operator.
- **Prerequisites:** two-person approval, new values in the secret manager, verified backup and recorded old image digest; never print either value.
- **Commands:** for JWT, set the new `GOCHAT_JWT_SECRET`, put the old value in `GOCHAT_JWT_PREVIOUS_SECRETS`, run preflight and roll web/worker; after the maximum token lifetime, clear the previous value and roll again. Rotate Connector/Alertmanager credentials by updating their secret files/variables and recreating only affected services. Rotate the backup passphrase only after creating and restoring one bundle with the new passphrase; retain the old key until old bundles expire.
- **Success / failure:** new credentials work, old JWTs work only during the bounded overlap, revoked credentials fail, logs contain no secret and backup restore succeeds; otherwise failure.
- **Rollback:** restore the previous secret version and immutable image, recreate affected services, and invalidate the failed new credential.
- **Drill cadence:** quarterly and immediately after suspected exposure.
## Incident response
- **Owner:** primary SRE on-call; incident commander for critical incidents.
- **Prerequisites:** alert payload, dashboard access, immutable deployment record and this repository checkout.
- **Commands:** acknowledge the alert; record UTC start time; run `$COMPOSE ps`, `curl -fsS http://127.0.0.1:9090/api/v1/alerts`, and `$COMPOSE logs --since 15m`; classify dependency, application, queue or capacity failure; mitigate with the matching runbook; record each command and result without secrets.
- **Success / failure:** success is customer impact stopped, alert resolved and evidence preserved; recurring/reopened alerts or unknown data integrity are failure and require escalation.
- **Rollback:** revert the most recent change or isolate the failed dependency; never run ad-hoc schema down.
- **Drill cadence:** quarterly game day and after each Sev-1/Sev-2 retrospective.
## On-call escalation
- **Owner:** primary SRE on-call.
- **Prerequisites:** current contact rota and severity policy in the paging system.
- **Commands:** acknowledge warning within 15 minutes and critical within 5; page secondary SRE after 5 unacknowledged minutes; page application/data owner for 15 minutes of unresolved impact; appoint incident commander and notify product owner at 30 minutes or on confirmed data/security impact.
- **Success / failure:** success is a named owner, incident channel, next update time and acknowledged handoff; no acknowledgement inside the window is failure.
- **Rollback:** if escalation was false, resolve the page with reason and tune only after evidence review.
- **Drill cadence:** monthly paging test and quarterly rota failover.
## Log search and retention
- **Owner:** SRE on-call; security approves retention.
- **Prerequisites:** `fluentd` running, `fluentd_logs` and `fluentd_buffer` below 70% capacity.
- **Commands:** search recent compressed output with `$COMPOSE exec -T fluentd sh -c 'zgrep -h "request_id_or_error" /fluentd/log/gochat*.gz'`; preserve incident files before cleanup; delete only flushed files older than the approved 14-day baseline with `$COMPOSE exec -T fluentd sh -c 'find /fluentd/log -type f -name "gochat*.gz" -mtime +14 -delete'`.
- **Success / failure:** success is searchable JSON containing `service` and `environment`, with disk below 70%; missing incident logs, full buffers or deletion inside retention are failure.
- **Rollback:** restore preserved logs from the incident archive and stop cleanup scheduling while retention is investigated.
- **Drill cadence:** weekly capacity/retention check and quarterly search drill.
## Backup and restore
- **Owner:** database SRE.
- **Prerequisites:** dedicated local/off-site mounts, passphrase file, Connector online backup and clean isolated restore project.
- **Commands:** follow `01-rolling-upgrade.md` daily backup and clean-environment restore commands; verify `gochat_backup_last_success_timestamp_seconds` is fresh in Prometheus.
- **Success / failure:** success is verified encryption/checksums, off-site copy, RPO at most 24 hours and isolated RTO at most 4 hours; any missing metric, checksum/decrypt error or non-empty restore target is failure.
- **Rollback:** do not overwrite production; discard the isolated target, restore mount/key access, and retry from the last verified bundle.
- **Drill cadence:** backup daily; non-author restore rehearsal quarterly.
## Upgrade and rollback
- **Owner:** release engineer with database SRE.
- **Prerequisites:** approved immutable old/new digests, green release gate, verified backup/restore and maintenance window.
- **Commands:** execute `01-rolling-upgrade.md` preflight, one-shot migration, post-check and application rollout exactly as written.
- **Success / failure:** success is one migration run, clean schema version, ready web/worker and unchanged reconciliation counts; any dirty migration, lock timeout or failed health check is failure.
- **Rollback:** redeploy the recorded old application digest; if schema is unusable, stop writes and restore the verified bundle into a clean target. Never run production `migrate down`.
- **Drill cadence:** every release in staging; quarterly old-digest rollback drill.
## Planned shutdown
- **Owner:** incident commander or release engineer.
- **Prerequisites:** approved change window, customer notice, verified backup and rollback owner.
- **Commands:** remove web from the load balancer; wait for active requests; run `$COMPOSE stop -t 35 gochat worker shangwutong`; stop monitoring last with `$COMPOSE stop prometheus alertmanager fluentd`; leave data services running unless full host maintenance requires them stopped.
- **Success / failure:** success is graceful exit inside timeout, no running jobs left locked and no data-volume deletion; forced kill or new accepted traffic is failure.
- **Rollback:** `$COMPOSE up -d --wait`, restore the load-balancer target only after readiness succeeds.
- **Drill cadence:** semi-annually.
## Disaster failover
- **Owner:** incident commander and database SRE.
- **Prerequisites:** declared disaster, last verified off-site bundle, alternate host/network/DNS, secret-manager access and immutable image digests.
- **Commands:** fence the failed site; restore to a clean alternate environment via `01-rolling-upgrade.md`; validate schema/accounts/attachments/Connector; start monitoring, dependencies, web/worker; lower DNS/load-balancer TTL and shift traffic only after `/ready` and smoke pass.
- **Success / failure:** success is a single writable site, RPO/RTO recorded, alerts/logs working and customer smoke passing; split brain, checksum mismatch or failed reconciliation is failure.
- **Rollback:** stop the alternate writers and return traffic only if the original site is proven authoritative and reconciled; otherwise keep it fenced.
- **Drill cadence:** semi-annual full failover and quarterly tabletop.
docs/ops/03-observability-drill.md
+100
View File
@@ -0,0 +1,100 @@
# Observability alert and recovery drill
- **Owner:** a non-author SRE on-call performs and signs the drill; the change author may observe only.
- **Prerequisites:** isolated staging project, approved paging receiver, no production traffic, current immutable images, successful backup, and `PROMETHEUS_PORT`/`ALERTMANAGER_PORT` reachable only on localhost.
- **Success / failure:** every scenario must produce a firing notification, a resolved notification and a matching searchable Fluentd record. Missing any of the three is failure.
- **Rollback:** each scenario includes recovery; if recovery fails, run `$COMPOSE up -d --wait` and restore the latest verified staging bundle.
- **Drill cadence:** before first production release, quarterly, and after alert-route changes.
Set `COMPOSE='docker compose -f deploy/docker/docker-compose.prod.yml'`. Record UTC timestamps, Alertmanager payload IDs and redacted log excerpts; never record environment files, URLs carrying tokens or secret values.
## Baseline
```bash
$COMPOSE up -d --wait
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/-/ready
curl -fsS http://127.0.0.1:${ALERTMANAGER_PORT:-9093}/-/ready
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/api/v1/targets
```
Pass when every configured target is healthy. Fix baseline failures before injecting faults.
## Database readiness failure and recovery
Use the external PostgreSQL provider's approved staging-only fault mechanism to
block GoChat's staging clients while preserving operator access. Wait for
`GoChatDatabaseReadinessFailed`, record its notification ID, restore access,
then run `$COMPOSE up -d --wait gochat` and record the resolved notification.
Include the provider change/audit ID in the evidence; production Compose does
not run a local PostgreSQL service.
Search logs with `$COMPOSE exec -T fluentd sh -c 'zgrep -h -E "database|unhealthy" /fluentd/log/gochat*.gz'` and record a redacted excerpt.
## Redis readiness failure and recovery
Use the external Redis provider's approved staging-only fault mechanism to
block GoChat's staging clients while preserving operator access. Wait for
`GoChatRedisReadinessFailed`, record its notification ID, restore access, then
run `$COMPOSE up -d --wait gochat worker` and record the resolved notification.
Include the provider change/audit ID in the evidence; production Compose does
not run a local Redis service.
Search logs with `$COMPOSE exec -T fluentd sh -c 'zgrep -h -E "redis|unhealthy" /fluentd/log/gochat*.gz'`.
## HTTP 5xx rate and recovery
The unknown dependency selector intentionally returns 503 and is recorded by the normal HTTP metrics middleware.
```bash
end=$((SECONDS + 360))
while ((SECONDS < end)); do
curl -sS 'http://127.0.0.1:3000/health?check=drill-unknown' >/dev/null || true
sleep 0.2
done
# Wait for GoChatHighErrorRate, stop injection, then wait for its resolved notification.
```
Search logs for `drill-unknown` or the matching request IDs. Failure to resolve after the five-minute rate window plus route delay fails the drill.
## Process and worker failure recovery
```bash
$COMPOSE stop gochat worker
# Record GoChatAppDown and GoChatWorkerDown notifications.
$COMPOSE up -d --wait gochat worker
# Record both resolved notifications.
```
Confirm Fluentd contains the shutdown and startup records for both services.
## Stale backup and recovery
Use the existing backup service so the test writes the same shared textfile volume as production:
```bash
$COMPOSE --profile ops run --rm --entrypoint sh backup -c \
'printf "%s\n" "gochat_backup_last_success_timestamp_seconds 0" "gochat_backup_rpo_target_seconds 1" > /metrics/gochat_backup.prom'
# Wait for GoChatBackupStale and record its notification ID.
$COMPOSE --profile ops run --rm backup
# Wait for the resolved notification and confirm the new metric timestamp.
```
Search Fluentd for the backup service's `backup=... created_at=...` record without copying bundle paths or secrets into the evidence.
## Evidence record
Commit a completed copy of this table or attach it to HH-445. The executor must not be the implementation author.
| Field | Recorded value |
|---|---|
| Date / staging revision | |
| Non-author executor | |
| Baseline targets healthy | |
| DB firing / resolved IDs | |
| Redis firing / resolved IDs | |
| 5xx firing / resolved IDs | |
| process + worker firing / resolved IDs | |
| backup stale firing / resolved IDs | |
| redacted Fluentd evidence references | |
| RPO / RTO observed | |
| Result and follow-ups | |