Rogeeandrogee
fb83285617
* HH-445: deploy production observability and runbooks * fix(ops): share production database DSN * fix(HH-445): enforce database TLS gate * fix(HH-445): preserve production serve command * fix(prod): require external database dependencies * fix(prod): unify database host rejection gates * test(prod): enforce exact database TLS runbook contract --------- Co-authored-by: Rogee <rogee@ipao.vip>
101 lines
4.4 KiB
Markdown
101 lines
4.4 KiB
Markdown
# Observability alert and recovery drill
|
|
|
|
- **Owner:** a non-author SRE on-call performs and signs the drill; the change author may observe only.
|
|
- **Prerequisites:** isolated staging project, approved paging receiver, no production traffic, current immutable images, successful backup, and `PROMETHEUS_PORT`/`ALERTMANAGER_PORT` reachable only on localhost.
|
|
- **Success / failure:** every scenario must produce a firing notification, a resolved notification and a matching searchable Fluentd record. Missing any of the three is failure.
|
|
- **Rollback:** each scenario includes recovery; if recovery fails, run `$COMPOSE up -d --wait` and restore the latest verified staging bundle.
|
|
- **Drill cadence:** before first production release, quarterly, and after alert-route changes.
|
|
|
|
Set `COMPOSE='docker compose -f deploy/docker/docker-compose.prod.yml'`. Record UTC timestamps, Alertmanager payload IDs and redacted log excerpts; never record environment files, URLs carrying tokens or secret values.
|
|
|
|
## Baseline
|
|
|
|
```bash
|
|
$COMPOSE up -d --wait
|
|
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/-/ready
|
|
curl -fsS http://127.0.0.1:${ALERTMANAGER_PORT:-9093}/-/ready
|
|
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/api/v1/targets
|
|
```
|
|
|
|
Pass when every configured target is healthy. Fix baseline failures before injecting faults.
|
|
|
|
## Database readiness failure and recovery
|
|
|
|
Use the external PostgreSQL provider's approved staging-only fault mechanism to
|
|
block GoChat's staging clients while preserving operator access. Wait for
|
|
`GoChatDatabaseReadinessFailed`, record its notification ID, restore access,
|
|
then run `$COMPOSE up -d --wait gochat` and record the resolved notification.
|
|
Include the provider change/audit ID in the evidence; production Compose does
|
|
not run a local PostgreSQL service.
|
|
|
|
Search logs with `$COMPOSE exec -T fluentd sh -c 'zgrep -h -E "database|unhealthy" /fluentd/log/gochat*.gz'` and record a redacted excerpt.
|
|
|
|
## Redis readiness failure and recovery
|
|
|
|
Use the external Redis provider's approved staging-only fault mechanism to
|
|
block GoChat's staging clients while preserving operator access. Wait for
|
|
`GoChatRedisReadinessFailed`, record its notification ID, restore access, then
|
|
run `$COMPOSE up -d --wait gochat worker` and record the resolved notification.
|
|
Include the provider change/audit ID in the evidence; production Compose does
|
|
not run a local Redis service.
|
|
|
|
Search logs with `$COMPOSE exec -T fluentd sh -c 'zgrep -h -E "redis|unhealthy" /fluentd/log/gochat*.gz'`.
|
|
|
|
## HTTP 5xx rate and recovery
|
|
|
|
The unknown dependency selector intentionally returns 503 and is recorded by the normal HTTP metrics middleware.
|
|
|
|
```bash
|
|
end=$((SECONDS + 360))
|
|
while ((SECONDS < end)); do
|
|
curl -sS 'http://127.0.0.1:3000/health?check=drill-unknown' >/dev/null || true
|
|
sleep 0.2
|
|
done
|
|
# Wait for GoChatHighErrorRate, stop injection, then wait for its resolved notification.
|
|
```
|
|
|
|
Search logs for `drill-unknown` or the matching request IDs. Failure to resolve after the five-minute rate window plus route delay fails the drill.
|
|
|
|
## Process and worker failure recovery
|
|
|
|
```bash
|
|
$COMPOSE stop gochat worker
|
|
# Record GoChatAppDown and GoChatWorkerDown notifications.
|
|
$COMPOSE up -d --wait gochat worker
|
|
# Record both resolved notifications.
|
|
```
|
|
|
|
Confirm Fluentd contains the shutdown and startup records for both services.
|
|
|
|
## Stale backup and recovery
|
|
|
|
Use the existing backup service so the test writes the same shared textfile volume as production:
|
|
|
|
```bash
|
|
$COMPOSE --profile ops run --rm --entrypoint sh backup -c \
|
|
'printf "%s\n" "gochat_backup_last_success_timestamp_seconds 0" "gochat_backup_rpo_target_seconds 1" > /metrics/gochat_backup.prom'
|
|
# Wait for GoChatBackupStale and record its notification ID.
|
|
$COMPOSE --profile ops run --rm backup
|
|
# Wait for the resolved notification and confirm the new metric timestamp.
|
|
```
|
|
|
|
Search Fluentd for the backup service's `backup=... created_at=...` record without copying bundle paths or secrets into the evidence.
|
|
|
|
## Evidence record
|
|
|
|
Commit a completed copy of this table or attach it to HH-445. The executor must not be the implementation author.
|
|
|
|
| Field | Recorded value |
|
|
|---|---|
|
|
| Date / staging revision | |
|
|
| Non-author executor | |
|
|
| Baseline targets healthy | |
|
|
| DB firing / resolved IDs | |
|
|
| Redis firing / resolved IDs | |
|
|
| 5xx firing / resolved IDs | |
|
|
| process + worker firing / resolved IDs | |
|
|
| backup stale firing / resolved IDs | |
|
|
| redacted Fluentd evidence references | |
|
|
| RPO / RTO observed | |
|
|
| Result and follow-ups | |
|