gochat/docs/ops/03-observability-drill.md

Observability alert and recovery drill

• Owner: a non-author SRE on-call performs and signs the drill; the change author may observe only.
• Prerequisites: isolated staging project, approved paging receiver, no production traffic, current immutable images, successful backup, and PROMETHEUS_PORT/ALERTMANAGER_PORT reachable only on localhost.
• Success / failure: every scenario must produce a firing notification, a resolved notification and a matching searchable Fluentd record. Missing any of the three is failure.
• Rollback: each scenario includes recovery; if recovery fails, run $COMPOSE up -d --wait and restore the latest verified staging bundle.
• Drill cadence: before first production release, quarterly, and after alert-route changes.

Set COMPOSE='docker compose -f deploy/docker/docker-compose.prod.yml'. Record UTC timestamps, Alertmanager payload IDs and redacted log excerpts; never record environment files, URLs carrying tokens or secret values.

Baseline

$COMPOSE up -d --wait
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/-/ready
curl -fsS http://127.0.0.1:${ALERTMANAGER_PORT:-9093}/-/ready
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/api/v1/targets

Pass when every configured target is healthy. Fix baseline failures before injecting faults.

Database readiness failure and recovery

Use the external PostgreSQL provider's approved staging-only fault mechanism to block GoChat's staging clients while preserving operator access. Wait for GoChatDatabaseReadinessFailed, record its notification ID, restore access, then run $COMPOSE up -d --wait gochat and record the resolved notification. Include the provider change/audit ID in the evidence; production Compose does not run a local PostgreSQL service.

Search logs with $COMPOSE exec -T fluentd sh -c 'zgrep -h -E "database|unhealthy" /fluentd/log/gochat*.gz' and record a redacted excerpt.

Redis readiness failure and recovery

Use the external Redis provider's approved staging-only fault mechanism to block GoChat's staging clients while preserving operator access. Wait for GoChatRedisReadinessFailed, record its notification ID, restore access, then run $COMPOSE up -d --wait gochat worker and record the resolved notification. Include the provider change/audit ID in the evidence; production Compose does not run a local Redis service.

Search logs with $COMPOSE exec -T fluentd sh -c 'zgrep -h -E "redis|unhealthy" /fluentd/log/gochat*.gz'.

HTTP 5xx rate and recovery

The unknown dependency selector intentionally returns 503 and is recorded by the normal HTTP metrics middleware.

end=$((SECONDS + 360))
while ((SECONDS < end)); do
  curl -sS 'http://127.0.0.1:3000/health?check=drill-unknown' >/dev/null || true
  sleep 0.2
done
# Wait for GoChatHighErrorRate, stop injection, then wait for its resolved notification.

Search logs for drill-unknown or the matching request IDs. Failure to resolve after the five-minute rate window plus route delay fails the drill.

Process and worker failure recovery

$COMPOSE stop gochat worker
# Record GoChatAppDown and GoChatWorkerDown notifications.
$COMPOSE up -d --wait gochat worker
# Record both resolved notifications.

Confirm Fluentd contains the shutdown and startup records for both services.

Stale backup and recovery

Use the existing backup service so the test writes the same shared textfile volume as production:

$COMPOSE --profile ops run --rm --entrypoint sh backup -c \
  'printf "%s\n" "gochat_backup_last_success_timestamp_seconds 0" "gochat_backup_rpo_target_seconds 1" > /metrics/gochat_backup.prom'
# Wait for GoChatBackupStale and record its notification ID.
$COMPOSE --profile ops run --rm backup
# Wait for the resolved notification and confirm the new metric timestamp.

Search Fluentd for the backup service's backup=... created_at=... record without copying bundle paths or secrets into the evidence.

Evidence record

Commit a completed copy of this table or attach it to HH-445. The executor must not be the implementation author.

Field	Recorded value
Date / staging revision
Non-author executor
Baseline targets healthy
DB firing / resolved IDs
Redis firing / resolved IDs
5xx firing / resolved IDs
process + worker firing / resolved IDs
backup stale firing / resolved IDs
redacted Fluentd evidence references
RPO / RTO observed
Result and follow-ups