rogee/gochat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(worker): add durable background jobs

Browse Source
This commit is contained in:
rogee
2026-06-05 15:38:03 +08:00
parent 52b6497d3a
commit 2cb02c8a91
8 changed files with 732 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md
+25 -7
View File
@@ -17,9 +17,9 @@ Build GoChat as a Go backend that can directly reuse the frontend from `referenc
## Current Baseline
- Plan freeze checkpoint: 2026-06-05 docs-only tracker landing after `13cb750 feat(captain): align task payload persistence`.
- Latest implementation checkpoint: this checkpoint, prepared as `test(parity): add enterprise browser smoke coverage`.
- Latest implementation checkpoint: this checkpoint, prepared as `feat(worker): add durable background jobs`.
- Latest documentation checkpoint before this freeze: `3263ed9 docs: land copilot task execution plan`; this document now carries the active follow-up plan directly.
- Worktree status at this implementation checkpoint: B11.1a aligns Captain assistant CRUD/tools/inbox bindings; B11.1b aligns Captain scenarios and custom tools; B11.1c aligns Captain documents, assistant responses, bulk actions, and custom-tool test payloads; B11.2 aligns Copilot thread/message create/list/get/delete payloads, account/user scoping, and no-LLM fallback persistence; B11.3a aligns Captain preferences show/update payloads and account-level model/feature storage; B11.3b aligns Captain playground request/response payloads, account scoping, v2 history handling, and no-LLM fallback; B11.3c adds the fakeable Captain document sync backend gate with disabled, failed, and fake-success states; B11.3d aligns Captain task request/response payloads, no-provider disabled states, follow-up context, suggestion persistence, and Copilot message tool-call key validation; B11.3e aligns Captain stream DTOs/disabled SSE fallbacks and Copilot push-event payload shapes; B12.1 adds the reusable GoChat server/seed entrypoint plus a Meilisearch-first reused Chatwoot frontend smoke harness and report; B12.2a adds API smoke assertions for auth/profile, inbox, conversation/messages, contact/company, widget config/message, and public CSAT; B12.2b adds a zero-dependency Chrome DevTools browser smoke that loads the reused Chatwoot login and dashboard entrypoints through Vite and checks browser auth/dashboard API requests; B12.3a adds enterprise API smoke assertions for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot; B12.3b adds reused-frontend enterprise browser route navigation for SLA, CSAT, automation, macros, audit logs, custom roles, capacity, Captain, and Copilot request coverage. Next active implementation slice is Phase 5 job parity.
- Worktree status at this implementation checkpoint: B11.1a aligns Captain assistant CRUD/tools/inbox bindings; B11.1b aligns Captain scenarios and custom tools; B11.1c aligns Captain documents, assistant responses, bulk actions, and custom-tool test payloads; B11.2 aligns Copilot thread/message create/list/get/delete payloads, account/user scoping, and no-LLM fallback persistence; B11.3a aligns Captain preferences show/update payloads and account-level model/feature storage; B11.3b aligns Captain playground request/response payloads, account scoping, v2 history handling, and no-LLM fallback; B11.3c adds the fakeable Captain document sync backend gate with disabled, failed, and fake-success states; B11.3d aligns Captain task request/response payloads, no-provider disabled states, follow-up context, suggestion persistence, and Copilot message tool-call key validation; B11.3e aligns Captain stream DTOs/disabled SSE fallbacks and Copilot push-event payload shapes; B12.1 adds the reusable GoChat server/seed entrypoint plus a Meilisearch-first reused Chatwoot frontend smoke harness and report; B12.2a adds API smoke assertions for auth/profile, inbox, conversation/messages, contact/company, widget config/message, and public CSAT; B12.2b adds a zero-dependency Chrome DevTools browser smoke that loads the reused Chatwoot login and dashboard entrypoints through Vite and checks browser auth/dashboard API requests; B12.3a adds enterprise API smoke assertions for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot; B12.3b adds reused-frontend enterprise browser route navigation for SLA, CSAT, automation, macros, audit logs, custom roles, capacity, Captain, and Copilot request coverage; P5.1 adds the PostgreSQL-backed durable `background_jobs` model/migration plus WorkerPool enqueue, schedule, retry/backoff, dead-letter, idempotency, stale-lock recovery, and focused tests. Next active implementation slice is P5.2/P5.4 job integration.
- `go test ./...` passes.
- Route dump succeeds with `TOTAL: 830` after adding the Chatwoot-compatible applied-SLA index route.
- Route parity artifacts now exist under `docs/parity/` and are generated by `cmd/route_parity`.
@@ -78,7 +78,7 @@ Open work after the current checkpoint:
| Phase 2 | Route and controller parity audit | Doing | Ruby/Bundler unavailable, so Chatwoot route extraction currently uses static `routes.rb` fallback |
| Phase 3 | Data and serializer parity | Doing | JSON fixture coverage is partial and still endpoint-family based |
| Phase 4 | Enterprise feature completion | Doing | B7, B8, B9, B10, and B11 are in Review; B12 reused frontend smoke is the next broad verification gate |
| Phase 5 | Background jobs and integrations | Doing | durable worker choice and job parity are the next active slice after B12 harness review |
| Phase 5 | Background jobs and integrations | Doing | P5.1 durable worker core is in Review; dispatcher/automation/search/SLA/Captain job integration remains open |
| Phase 6 | Core placeholder burn-down | Doing | account/contact/conversation/message/inbox placeholder groups remain broad |
| Phase 7 | Verification harness | Review | B12.1 boot/readiness, B12.2a API assertions, B12.2b browser smoke harness, B12.3a enterprise API assertions, and B12.3b enterprise browser route navigation exist; optional live Meilisearch/full-browser runs remain environment-dependent |
@@ -101,6 +101,7 @@ This ledger records the committed parity checkpoints that future slices should b
| Commit | Scope | Verification summary | Follow-up state |
| --- | --- | --- | --- |
| `feat(worker): add durable background jobs` | Advances P5.1 with a persistent `background_jobs` model and migration plus `internal/worker.WorkerPool` enqueue, scheduled claim, handler registry, retry/backoff, dead-letter, idempotency-key reuse, queue filtering, PostgreSQL `SKIP LOCKED` claim support, stale lock recovery, and graceful start/stop loop. The model is registered in app auto-migration and test DB defaults. | `go test ./internal/worker -count=1`; `go test ./...`; `git diff --check`; full verification recorded in the P5.1 section. | Move P5.1 to Review; continue P5.2 dispatcher async routing and P5.4/P5.5 automation/CSAT job integration on top of the durable core. |
| `test(parity): add enterprise browser smoke coverage` | Advances B12.3b with `scripts/parity_frontend_smoke.sh --enterprise-browser-smoke`. The browser driver now supports `--enterprise`, creates route-specific reused Chatwoot dashboard shells, and asserts browser-origin requests for SLA reports, CSAT reports, automation rules, macros, audit logs, custom roles, agent capacity list/edit, Captain settings/assistants, and Copilot thread requests. The smoke seed now enables the Chatwoot enterprise feature flags required by those route guards. | `bash -n scripts/parity_frontend_smoke.sh`; `node --check scripts/parity_frontend_browser_smoke.mjs`; `scripts/parity_frontend_smoke.sh --print`; `scripts/parity_frontend_smoke.sh --check`; `go test ./cmd/gochat -count=1`; full verification recorded in the B12.3b section. Live enterprise browser smoke requires running GoChat and Chatwoot Vite. | Move B12 verification harness to Review; continue Phase 5 job parity and convert any optional live-smoke failures into named slices. |
| `test(parity): add enterprise api smoke coverage` | Advances B12.3a with `scripts/parity_frontend_smoke.sh --enterprise-smoke`. The command composes the core API smoke with enterprise API assertions for SLA reports/metrics/download, public and account CSAT reports/metrics/download, automation-rule create/list, macro create/execute, audit-log list, custom-role list, agent-capacity list/users, Captain preferences/assistants, and Copilot thread/message payloads. | `bash -n scripts/parity_frontend_smoke.sh`; `scripts/parity_frontend_smoke.sh --print`; `scripts/parity_frontend_smoke.sh --check`; focused verification recorded in the B12.3a section. Live enterprise smoke requires running GoChat with PostgreSQL/Redis/Meilisearch. | Continue B12.3b with reused-frontend browser navigation for the same enterprise screens. |
| `test(parity): add browser smoke harness` | Advances B12.2b with `scripts/parity_frontend_smoke.sh --browser-smoke` and `scripts/parity_frontend_browser_smoke.mjs`. The browser smoke creates temporary Vite-served HTML shells for the reused Chatwoot `v3app` login and `dashboard` entrypoints, drives login through headless Chrome via the DevTools protocol without adding Node dependencies, confirms the Chatwoot auth cookie, and checks dashboard validation/conversation requests. | `bash -n scripts/parity_frontend_smoke.sh`; `node --check scripts/parity_frontend_browser_smoke.mjs`; `scripts/parity_frontend_smoke.sh --print`; `scripts/parity_frontend_smoke.sh --check`; full verification recorded in the B12.2b section. Live browser smoke requires running GoChat and Chatwoot Vite. | Continue B12.3 with enterprise screen browser/API smoke for SLA, CSAT reports, automation/macros, audit/custom roles, capacity, Captain, and Copilot. |
@@ -1578,7 +1579,7 @@ Reference job families to keep in scope:
Known hotspots:
- `internal/worker/worker.go` is still mostly placeholder.
- `internal/worker/worker.go` now has the P5.1 durable core, but no feature-specific producers/handlers are wired yet.
- `internal/automation/action_service.go` has pending webhook/email transcript work.
- `internal/automation/csat_survey_listener.go` now handles CSAT enablement, survey rules, generic send idempotency, and message-updated response building; durable queueing and external WhatsApp/Twilio template delivery remain follow-ups.
- `internal/auth/webhook_registry.go` has pending signature verification for Facebook/WhatsApp.
@@ -1586,8 +1587,8 @@ Known hotspots:
Checklist:
- [ ] Add `background_jobs` persistence with job type, payload, queue, status, attempt counters, scheduled/locked timestamps, idempotency key, last error, and completion timestamps.
- [ ] Replace `internal/worker/worker.go` placeholder with enqueue, schedule, perform, retry/backoff, dead-letter, and graceful shutdown behavior.
- [x] Add `background_jobs` persistence with job type, payload, queue, status, attempt counters, scheduled/locked timestamps, idempotency key, last error, and completion timestamps.
- [x] Replace `internal/worker/worker.go` placeholder with enqueue, schedule, perform, retry/backoff, dead-letter, and graceful shutdown behavior.
- [ ] Map Chatwoot jobs/listeners to Go worker responsibilities and record out-of-scope migration-only jobs explicitly.
- [ ] Implement durable job dispatch for automation, CSAT, notifications, webhooks, SLA processing, Captain/Copilot, contact export, macros, and search indexing.
- [ ] Add retry and failure logging for external calls.
@@ -1602,7 +1603,7 @@ Tracking table:
| ID | Task | Reference source | Current Go hotspot | Done boundary | Status |
| --- | --- | --- | --- | --- | --- |
| P5.1 | Implement durable worker core and job model. | `reference/chatwoot/app/jobs/application_job.rb`, `mutex_application_job.rb` | `internal/worker/worker.go` | Job table, repository, enqueue API, worker loop, retry/backoff, scheduled jobs, mutex/idempotency keys, dead-letter state, and restart pickup tests exist. | Next |
| P5.1 | Implement durable worker core and job model. | `reference/chatwoot/app/jobs/application_job.rb`, `mutex_application_job.rb` | `internal/worker/worker.go`, `internal/model/background_job.go`, `migrations/000026_add_background_jobs.*.sql` | Job table, worker persistence API, enqueue API, worker loop, retry/backoff, scheduled jobs, mutex/idempotency keys, dead-letter state, and restart pickup tests exist. | Review by `feat(worker): add durable background jobs` |
| P5.2 | Route async dispatcher events through durable jobs. | `event_dispatcher_job.rb`, Chatwoot async dispatcher listeners | `internal/dispatch/dispatcher.go`, `internal/channel/dispatcher.go` | Heavy listeners can enqueue durable jobs without changing sync listener behavior; tests cover sync vs async routing and replay. | Todo |
| P5.3 | Move Meilisearch indexing and reindex fan-out into retryable jobs. | Meilisearch plan plus Chatwoot callbacks/jobs that index searchable records | search services, contact/company/conversation indexing hooks | Create/update/delete indexing survives handler success, retries on Meilisearch failure, and optional live Meilisearch gate remains green. | Todo |
| P5.4 | Queue automation webhook and transcript delivery. | `webhook_job.rb`, automation action execution services | `internal/automation/action_delivery.go`, `internal/automation/action_service.go` | Existing timeout/retry fakeable delivery is invoked by durable jobs; logs preserve attempt metadata and idempotency. | Todo |
@@ -1616,6 +1617,22 @@ Tracking table:
| P5.12 | Queue conversation maintenance jobs. | `conversations/resolution_job.rb`, `reopen_snoozed_conversations_job.rb`, `update_message_status_job.rb`, `bulk_actions_job.rb` | conversation service/handlers | Auto-resolution, snooze reopen, status updates, and bulk actions are scheduled/retryable with idempotent tests. | Todo |
| P5.13 | Replace placeholder analytics/report builders that need background aggregation. | reporting jobs/services and report controllers | `internal/service/analytics_service.go`, reporting services | Frontend-visible reports no longer use placeholder values; any expensive aggregation is scheduled or cached with freshness rules. | Todo |
P5.1 current checkpoint:
- Added `background_jobs` as the shared durable job table with queue, job type, JSON payload, status, priority, attempt counters, scheduled/locked timestamps, idempotency key, last error, and completion/dead-letter timestamps.
- Replaced the `WorkerPool` stub with enqueue/schedule APIs, handler registration, `ProcessOne`, start/stop worker loops, PostgreSQL `FOR UPDATE SKIP LOCKED` claiming, retry/backoff, dead-letter state, and stale-lock recovery for worker restart pickup.
- Kept older `NewWorkerPool()` construction as a no-op-compatible path while adding `NewWorkerPoolWithOptions(db, ...)` for durable wiring.
- Added focused tests proving enqueue/idempotency, due job completion, retry to dead-letter, queue/schedule filtering, and stale running job requeue.
- Remaining Phase 5 work must now wire real producers/handlers: dispatcher async events, Meilisearch indexing, automation webhooks/transcripts, delayed actions/macros, CSAT templates, SLA scans, contact exports, provider webhooks, outbound delivery, Captain/Copilot, and conversation maintenance.
P5.1 verification:
```bash
env GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go test ./internal/worker -count=1
env TMPDIR=/home/rogee/Projects/gochat/.tmp/test-tmp GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go test ./...
git diff --check
```
## Phase 6: Core Product Placeholder Burn-down
Status: doing.
@@ -1807,3 +1824,4 @@ Verification milestone gates:
- 2026-06-05: B12.2b browser smoke harness checkpoint prepared as `test(parity): add browser smoke harness`; `scripts/parity_frontend_smoke.sh --browser-smoke` now runs the live API smoke, verifies Chatwoot Vite readiness, and drives the reused Chatwoot login/dashboard entrypoints through headless Chrome DevTools without adding Playwright or frontend adapters. The smoke captures browser requests and console output in `.tmp/frontend-smoke/browser-smoke-report.json`, while default `--check` keeps live browser execution optional because it needs PostgreSQL, Redis, Meilisearch, GoChat, Vite, and Chrome. Verification: `bash -n`, `node --check`, `--print`, `--check`, full `go test ./...`, and `git diff --check`. Next slice is B12.3 enterprise screen smoke.
- 2026-06-05: B12.3a enterprise API smoke checkpoint prepared as `test(parity): add enterprise api smoke coverage`; `scripts/parity_frontend_smoke.sh --enterprise-smoke` now composes the core API smoke with enterprise checks for SLA reports/download, CSAT public submit plus reports/download, automation/macros, audit/custom roles, capacity, Captain preferences/assistants, and Copilot thread/message payloads. Default verification keeps the live enterprise run optional because it needs PostgreSQL, Redis, Meilisearch, and GoChat. Verification: `bash -n`, `--print`, and `--check` passed locally. Next slice is B12.3b reused-frontend enterprise browser navigation.
- 2026-06-05: B12.3b enterprise browser smoke checkpoint prepared as `test(parity): add enterprise browser smoke coverage`; `scripts/parity_frontend_smoke.sh --enterprise-browser-smoke` now runs enterprise API smoke, verifies Chatwoot Vite, and drives reused Chatwoot enterprise dashboard routes through the Chrome DevTools browser harness. Browser assertions cover SLA, CSAT, automation, macros, audit logs, custom roles, agent capacity, Captain settings/assistants, and Copilot thread request paths. Seed now enables the corresponding enterprise feature flags for route guards. Verification: `bash -n`, `node --check`, `--print`, `--check`, `go test ./cmd/gochat`, full `go test ./...`, and `git diff --check` passed locally. Next slice is Phase 5 job parity.
- 2026-06-05: P5.1 durable worker checkpoint prepared as `feat(worker): add durable background jobs`; the `background_jobs` table/model and WorkerPool core now cover enqueueing, scheduled jobs, queue filtering, idempotency keys, PostgreSQL `SKIP LOCKED` claims, handler execution, retry/backoff, dead-letter state, stale-lock recovery, and graceful start/stop. Focused worker tests cover idempotent enqueue, completion, retries, schedule/queue gating, and restart pickup. Next slice is P5.2/P5.4 integration of dispatcher and automation jobs onto the durable core.