rogee/gochat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HH-441: harden durable storage and recovery (#92)

Browse Source 
* HH-441: harden durable storage and recovery

* HH-441: clear recovery review blockers

* HH-441: enforce offsite backup failure domain

---------

Co-authored-by: Rogee <rogee@ipao.vip>
This commit is contained in:
Rogee
2026-08-22 02:16:02 +08:00
committed by GitHub
co-authored by rogee
parent 77c91662d2
commit 6d6d80dd86
20 changed files with 966 additions and 202 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ops/01-rolling-upgrade.md
+130 -47
View File
@@ -1,65 +1,148 @@
# GoChat Rolling Upgrade Strategy
# Reference: Chatwoot deployment uses zero-downtime upgrade pattern
# Production backup, restore, and upgrade runbook
## Overview
Production web and worker processes never migrate on startup. The Compose file
also requires `GOCHAT_IMAGE_REF` to be an immutable `image@sha256:digest`; tags such
as `latest` are not an acceptable rollback record.
GoChat follows a blue-green deployment strategy for production upgrades,
ensuring zero downtime during version transitions.
The bundled `gochat_storage` volume is shared and durable for replicas on one
Docker host. Multi-host replicas must provision that volume with a shared
volume driver/filesystem; never use separate node-local volumes.
## Upgrade Process
## Recovery objectives
### Step 1: Pre-flight Checks
1. Verify new Docker image is built and pushed: `docker pull gochat/gochat:${NEW_VERSION}`
2. Run database migrations on a staging environment first
3. Verify backward compatibility of migrations (new code must work with old schema)
4. Check feature flags — new features should be disabled by default
- RPO: 24 hours. Run the encrypted backup at least daily and alert if the latest
off-site bundle is older than 24 hours.
- RTO: 4 hours. Rehearse a clean-environment restore quarterly and record the
script's `rpo_seconds`, `rto_seconds`, image digest, migration version, account
count, and attachment count.
- Local and off-site backup directories must be different mounts/failure
domains. The bundle is AES-256 encrypted; keep the passphrase file in the
secret manager, never beside either backup copy.
## Daily encrypted backup
Set these operator-owned paths before any Compose command:
### Step 2: Database Migration
```bash
# Run migrations BEFORE deploying new code
# Migrations must be backward-compatible
docker compose -f docker-compose.prod.yml exec gochat /app/gochat migrate up
export GOCHAT_IMAGE_REF='ghcr.io/gochat/gochat@sha256:<digest>'
export GOCHAT_BACKUP_DIR='/mnt/backup-local/gochat'
export GOCHAT_BACKUP_OFFSITE_DIR='/mnt/gochat-offsite'
export GOCHAT_BACKUP_OFFSITE_SOURCE='backup.example.com:/gochat'
export GOCHAT_BACKUP_OFFSITE_FSTYPE='nfs4'
export GOCHAT_BACKUP_PASSPHRASE_FILE='/run/secrets/gochat-backup-passphrase'
export GOCHAT_CONNECTOR_BACKUP_NAME="connector-$(date -u +%Y%m%dT%H%M%SZ).db"
```
### Step 3: Blue-Green Deployment (Docker Compose)
Backup/restore are audited one-shot containers and run as root only to read or
rebuild Docker volumes; web and worker remain non-root.
`GOCHAT_BACKUP_OFFSITE_DIR` must be an existing external mount point provisioned
outside this Compose project. Set its approved source and filesystem type to the
exact values reported by `findmnt -M "$GOCHAT_BACKUP_OFFSITE_DIR"`; the preflight
rejects `/`, in-memory filesystems, unapproved mount metadata, and the local
backup device.
Create a consistent online Connector backup, then create and verify the single
encrypted bundle containing PostgreSQL, attachments, and that Connector copy:
```bash
# 1. Deploy new version as "green" alongside "blue" (current)
docker compose -f docker-compose.prod.yml up -d --no-deps gochat-green
# 2. Wait for health check to pass
curl -f http://gochat-green:3000/health
# 3. Switch traffic (update nginx/upstream config)
# nginx: switch upstream from blue to green
# 4. Drain old connections on blue
# Wait 30s for in-flight requests to complete
# 5. Stop blue
docker compose -f docker-compose.prod.yml stop gochat
docker compose -f deploy/docker/docker-compose.prod.yml exec shangwutong \
shangwutong backup --output "/backup/$GOCHAT_CONNECTOR_BACKUP_NAME"
docker compose -f deploy/docker/docker-compose.prod.yml --profile ops run --rm backup
```
### Step 4: Verification
1. Smoke test: hit /health endpoint
2. Check logs for errors: `docker compose logs gochat --since 5m`
3. Verify metrics: Prometheus dashboard should show normal traffic
4. Monitor for 15 minutes before finalizing
Schedule those two commands daily. Preserve their final `backup=... offsite=...`
line as audit evidence. A failed dump, archive checksum, decrypt/list check, or
off-site copy exits non-zero and must alert.
## Clean-environment restore rehearsal
Use an isolated host/project with empty volumes and an empty PostgreSQL database.
Do not point this procedure at the live project.
### Step 5: Rollback (if needed)
```bash
# Docker Compose rollback
docker compose -f docker-compose.prod.yml exec gochat /app/gochat migrate down ${N}
docker compose -f docker-compose.prod.yml up -d --no-deps gochat-${OLD_VERSION}
export COMPOSE_PROJECT_NAME=gochat-restore-$(date +%Y%m%d)
export GOCHAT_RESTORE_BUNDLE='gochat-<timestamp>.tar.enc'
docker compose -f deploy/docker/docker-compose.prod.yml --profile ops run --rm restore
```
## Migration Compatibility Rules
The restore refuses a non-empty database, attachment directory, or Connector DB.
It verifies the encrypted bundle and internal checksums before restoring, then
prints the RPO/RTO, image version, migration version, accounts, and attachments.
Afterward start web/worker with the recorded digest and verify `/health`, one
attachment URL, and Connector `readyz`.
- Migrations MUST be additive only in production (add columns, never remove)
- Column removals require a 2-phase migration: soft-remove then hard-remove
- New columns should have defaults or be nullable
- Renames require a 3-phase migration: add new → copy data → remove old
## Upgrade preflight
## Worker Upgrade
1. Record the running digest as `OLD_IMAGE`; pull and record `NEW_IMAGE` by
digest. Never derive rollback state from a mutable tag.
2. Complete the backup above and restore it in the isolated environment.
3. Stop writes or enter the maintenance window. Migration 000079 takes
`SHARE ROW EXCLUSIVE` locks; migration 000048 takes `ACCESS EXCLUSIVE` on the
rollup table. The migration job uses a 5-second lock timeout and 15-minute
statement timeout, so contention fails instead of waiting indefinitely.
4. Capture these pre-migration checks:
Workers drain naturally: set a shutdown deadline, let in-flight jobs finish,
then stop. New workers pick up queued jobs from Redis.
```sql
SELECT count(*) AS rollups FROM reporting_events_rollups;
SELECT count(*) AS swt_source_rows FROM conversations
WHERE COALESCE(custom_attributes, '{}'::jsonb) ?| ARRAY['swt_source_url','swt_source_search_term'];
SELECT account_id, (config->>'assistant_id')::bigint, count(*)
FROM agent_bots
WHERE bot_type = 'captain' AND config->>'assistant_id' ~ '^[0-9]+$'
GROUP BY 1, 2 HAVING count(*) > 1;
```
While migrating, observe lock waits with:
```sql
SELECT pid, wait_event_type, wait_event, clock_timestamp() - query_start AS elapsed, query
FROM pg_stat_activity
WHERE datname = current_database() AND state <> 'idle';
```
## One-shot migration and application rollout
Run exactly one named migration service before changing web/worker. Its output is
the audit log; `golang-migrate` is idempotent and PostgreSQL serializes competing
migration runners, but the deployment pipeline must contain only this one step.
```bash
export GOCHAT_IMAGE_REF="$NEW_IMAGE"
time docker compose -f deploy/docker/docker-compose.prod.yml --profile ops \
up --abort-on-container-exit --exit-code-from migrate migrate
docker compose -f deploy/docker/docker-compose.prod.yml logs migrate
docker compose -f deploy/docker/docker-compose.prod.yml up -d --no-deps gochat worker
curl -fsS http://127.0.0.1:3000/health
```
Post-migration checks:
```sql
SELECT version, dirty FROM schema_migrations;
SELECT count(*) AS rollups FROM reporting_events_rollups;
SELECT count(*) AS duplicate_bot_bindings FROM (
SELECT account_id, captain_assistant_id FROM agent_bots
WHERE captain_assistant_id IS NOT NULL GROUP BY 1, 2 HAVING count(*) > 1
) duplicates;
SELECT count(*) AS orphaned_inbox_bindings FROM agent_bot_inboxes b
LEFT JOIN agent_bots a ON a.id = b.agent_bot_id WHERE a.id IS NULL;
```
Migration 000048 preserves every legacy rollup row, 000076 retains legacy
Connector attributes for old-image compatibility, and 000079 repairs references
before deduplication. Any count mismatch or dirty version stops the rollout.
## Rollback
Rollback the application only, using the recorded digest:
```bash
export GOCHAT_IMAGE_REF="$OLD_IMAGE"
docker compose -f deploy/docker/docker-compose.prod.yml up -d --no-deps gochat worker
curl -fsS http://127.0.0.1:3000/health
```
Production `migrate down` and negative `migrate steps` are blocked. Never run a
destructive schema down during an application rollback. If the new schema itself
is unusable, stop web/worker and restore the verified pre-upgrade bundle into a
clean database and volumes.