# Observability alert and recovery drill

- **Owner:** a non-author SRE on-call performs and signs the drill; the change author may observe only.
- **Prerequisites:** isolated staging project, approved paging receiver, no production traffic, current immutable images, successful backup, and `PROMETHEUS_PORT`/`ALERTMANAGER_PORT` reachable only on localhost.
- **Success / failure:** every scenario must produce a firing notification, a resolved notification and a matching searchable Fluentd record. Missing any of the three is failure.
- **Rollback:** each scenario includes recovery; if recovery fails, run `$COMPOSE up -d --wait` and restore the latest verified staging bundle.
- **Drill cadence:** before first production release, quarterly, and after alert-route changes.

Set `COMPOSE='docker compose -f deploy/docker/docker-compose.prod.yml'`. Record UTC timestamps, Alertmanager payload IDs and redacted log excerpts; never record environment files, URLs carrying tokens or secret values.

## Baseline

```bash
$COMPOSE up -d --wait
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/-/ready
curl -fsS http://127.0.0.1:${ALERTMANAGER_PORT:-9093}/-/ready
curl -fsS http://127.0.0.1:${PROMETHEUS_PORT:-9090}/api/v1/targets
```

Pass when every configured target is healthy. Fix baseline failures before injecting faults.

## Database readiness failure and recovery

Use the external PostgreSQL provider's approved staging-only fault mechanism to
block GoChat's staging clients while preserving operator access. Wait for
`GoChatDatabaseReadinessFailed`, record its notification ID, restore access,
then run `$COMPOSE up -d --wait gochat` and record the resolved notification.
Include the provider change/audit ID in the evidence; production Compose does
not run a local PostgreSQL service.

Search logs with `$COMPOSE exec -T fluentd sh -c 'zgrep -h -E "database|unhealthy" /fluentd/log/gochat*.gz'` and record a redacted excerpt.

## Redis readiness failure and recovery

Use the external Redis provider's approved staging-only fault mechanism to
block GoChat's staging clients while preserving operator access. Wait for
`GoChatRedisReadinessFailed`, record its notification ID, restore access, then
run `$COMPOSE up -d --wait gochat worker` and record the resolved notification.
Include the provider change/audit ID in the evidence; production Compose does
not run a local Redis service.

Search logs with `$COMPOSE exec -T fluentd sh -c 'zgrep -h -E "redis|unhealthy" /fluentd/log/gochat*.gz'`.

## HTTP 5xx rate and recovery

The unknown dependency selector intentionally returns 503 and is recorded by the normal HTTP metrics middleware.

```bash
end=$((SECONDS + 360))
while ((SECONDS < end)); do
  curl -sS 'http://127.0.0.1:3000/health?check=drill-unknown' >/dev/null || true
  sleep 0.2
done
# Wait for GoChatHighErrorRate, stop injection, then wait for its resolved notification.
```

Search logs for `drill-unknown` or the matching request IDs. Failure to resolve after the five-minute rate window plus route delay fails the drill.

## Process and worker failure recovery

```bash
$COMPOSE stop gochat worker
# Record GoChatAppDown and GoChatWorkerDown notifications.
$COMPOSE up -d --wait gochat worker
# Record both resolved notifications.
```

Confirm Fluentd contains the shutdown and startup records for both services.

## Stale backup and recovery

Use the existing backup service so the test writes the same shared textfile volume as production:

```bash
$COMPOSE --profile ops run --rm --entrypoint sh backup -c \
  'printf "%s\n" "gochat_backup_last_success_timestamp_seconds 0" "gochat_backup_rpo_target_seconds 1" > /metrics/gochat_backup.prom'
# Wait for GoChatBackupStale and record its notification ID.
$COMPOSE --profile ops run --rm backup
# Wait for the resolved notification and confirm the new metric timestamp.
```

Search Fluentd for the backup service's `backup=... created_at=...` record without copying bundle paths or secrets into the evidence.

## Evidence record

Commit a completed copy of this table or attach it to HH-445. The executor must not be the implementation author.

| Field | Recorded value |
|---|---|
| Date / staging revision | |
| Non-author executor | |
| Baseline targets healthy | |
| DB firing / resolved IDs | |
| Redis firing / resolved IDs | |
| 5xx firing / resolved IDs | |
| process + worker firing / resolved IDs | |
| backup stale firing / resolved IDs | |
| redacted Fluentd evidence references | |
| RPO / RTO observed | |
| Result and follow-ups | |