diff --git a/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md b/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md index 55c7823a..4932c75f 100644 --- a/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md +++ b/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md @@ -16,10 +16,10 @@ Build GoChat as a Go backend that can directly reuse the frontend from `referenc ## Current Baseline -- Plan freeze checkpoint: 2026-06-05 docs-only tracker landing after `13cb750 feat(captain): align task payload persistence`. -- Latest implementation checkpoint: this checkpoint, prepared as `feat(conversations): queue bulk actions`. -- Latest documentation checkpoint before this freeze: `3263ed9 docs: land copilot task execution plan`; this document now carries the active follow-up plan directly. -- Worktree status at this implementation checkpoint: B11.1a aligns Captain assistant CRUD/tools/inbox bindings; B11.1b aligns Captain scenarios and custom tools; B11.1c aligns Captain documents, assistant responses, bulk actions, and custom-tool test payloads; B11.2 aligns Copilot thread/message create/list/get/delete payloads, account/user scoping, and no-LLM fallback persistence; B11.3a aligns Captain preferences show/update payloads and account-level model/feature storage; B11.3b aligns Captain playground request/response payloads, account scoping, v2 history handling, and no-LLM fallback; B11.3c adds the fakeable Captain document sync backend gate with disabled, failed, and fake-success states; B11.3d aligns Captain task request/response payloads, no-provider disabled states, follow-up context, suggestion persistence, and Copilot message tool-call key validation; B11.3e aligns Captain stream DTOs/disabled SSE fallbacks and Copilot push-event payload shapes; B12.1 adds the reusable GoChat server/seed entrypoint plus a Meilisearch-first reused Chatwoot frontend smoke harness and report; B12.2a adds API smoke assertions for auth/profile, inbox, conversation/messages, contact/company, widget config/message, and public CSAT; B12.2b adds a zero-dependency Chrome DevTools browser smoke that loads the reused Chatwoot login and dashboard entrypoints through Vite and checks browser auth/dashboard API requests; B12.3a adds enterprise API smoke assertions for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot; B12.3b adds reused-frontend enterprise browser route navigation for SLA, CSAT, automation, macros, audit logs, custom roles, capacity, Captain, and Copilot request coverage; P5.1 adds the PostgreSQL-backed durable `background_jobs` model/migration plus WorkerPool enqueue, schedule, retry/backoff, dead-letter, idempotency, stale-lock recovery, and focused tests; P5.2 wires `channel.Dispatcher` and `dispatch.EventDispatcher` async paths into durable event jobs with worker replay tests; P5.3 queues Meilisearch write-side index/delete jobs for conversations, messages, contacts, companies, and articles while keeping search reads Meilisearch-first; P5.4 queues automation webhook and email transcript side effects as durable jobs while preserving fakeable delivery boundaries; P5.5 queues Chatwoot-style macro execute fan-out through durable `automation:macro_execution` jobs; P5.6 queues resolve-triggered CSAT survey sends and WhatsApp/Twilio CSAT template creation through durable jobs; P5.7 queues Chatwoot enterprise SLA account scans and applied-SLA evaluation jobs through the durable worker; P5.8 queues Chatwoot-style contact export artifact generation through durable `contact:export` jobs; P5.11 queues Captain document sync requests through durable `captain:document_sync` jobs; P5.12 queues scheduled item fan-out, one-off campaigns, snoozed conversation reopening, account auto-resolution, widget/public message status updates, and account conversation bulk actions through durable jobs. Next active implementation slice is remaining Captain/Copilot jobs, provider webhook/outbound job fan-out, inbound webhook job deferral, and analytics aggregation. +- Current tracking checkpoint: 2026-06-05 after `cda7d02 feat(conversations): queue bulk actions`. +- Latest implementation checkpoint: `cda7d02 feat(conversations): queue bulk actions`. +- Latest documentation-only checkpoint: this checkpoint, prepared as `docs: land active parity tracking board`; this document is now the active follow-up plan and supersedes `.hermes/plans/*`. +- Worktree status at this implementation checkpoint: B11.1a aligns Captain assistant CRUD/tools/inbox bindings; B11.1b aligns Captain scenarios and custom tools; B11.1c aligns Captain documents, assistant responses, bulk actions, and custom-tool test payloads; B11.2 aligns Copilot thread/message create/list/get/delete payloads, account/user scoping, and no-LLM fallback persistence; B11.3a aligns Captain preferences show/update payloads and account-level model/feature storage; B11.3b aligns Captain playground request/response payloads, account scoping, v2 history handling, and no-LLM fallback; B11.3c adds the fakeable Captain document sync backend gate with disabled, failed, and fake-success states; B11.3d aligns Captain task request/response payloads, no-provider disabled states, follow-up context, suggestion persistence, and Copilot message tool-call key validation; B11.3e aligns Captain stream DTOs/disabled SSE fallbacks and Copilot push-event payload shapes; B12.1 adds the reusable GoChat server/seed entrypoint plus a Meilisearch-first reused Chatwoot frontend smoke harness and report; B12.2a adds API smoke assertions for auth/profile, inbox, conversation/messages, contact/company, widget config/message, and public CSAT; B12.2b adds a zero-dependency Chrome DevTools browser smoke that loads the reused Chatwoot login and dashboard entrypoints through Vite and checks browser auth/dashboard API requests; B12.3a adds enterprise API smoke assertions for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot; B12.3b adds reused-frontend enterprise browser route navigation for SLA, CSAT, automation, macros, audit logs, custom roles, capacity, Captain, and Copilot request coverage; P5.1 adds the PostgreSQL-backed durable `background_jobs` model/migration plus WorkerPool enqueue, schedule, retry/backoff, dead-letter, idempotency, stale-lock recovery, and focused tests; P5.2 wires `channel.Dispatcher` and `dispatch.EventDispatcher` async paths into durable event jobs with worker replay tests; P5.3 queues Meilisearch write-side index/delete jobs for conversations, messages, contacts, companies, and articles while keeping search reads Meilisearch-first; P5.4 queues automation webhook and email transcript side effects as durable jobs while preserving fakeable delivery boundaries; P5.5 queues Chatwoot-style macro execute fan-out through durable `automation:macro_execution` jobs; P5.6 queues resolve-triggered CSAT survey sends and WhatsApp/Twilio CSAT template creation through durable jobs; P5.7 queues Chatwoot enterprise SLA account scans and applied-SLA evaluation jobs through the durable worker; P5.8 queues Chatwoot-style contact export artifact generation through durable `contact:export` jobs; P5.11 queues Captain document sync requests through durable `captain:document_sync` jobs; P5.12 queues scheduled item fan-out, one-off campaigns, snoozed conversation reopening, account auto-resolution, widget/public message status updates, and account conversation bulk actions through durable jobs. Next active implementation slice is P5.10 outbound message delivery, followed by P5.9 inbound webhook deferral, remaining P5.11 Captain/Copilot jobs, and P5.13 analytics aggregation. - `go test ./...` passes. - Route dump succeeds with `TOTAL: 830` after adding the Chatwoot-compatible applied-SLA index route. - Route parity artifacts now exist under `docs/parity/` and are generated by `cmd/route_parity`. @@ -45,16 +45,18 @@ Next ordered checkpoints: | Order | Slice | Required outcome | Primary verification | | --- | --- | --- | --- | -| 1 | B12 | Reused Chatwoot frontend smoke runs repeatably against GoChat. | Checked smoke command plus gap report under `docs/parity/`. | -| 2 | Phase 5 job parity | Any async behavior deferred from B7-B11 has a durable worker decision, fakeable delivery boundary, and replay/idempotency tests. | Focused worker/job tests plus a Phase 5 gap ledger update. | -| 3 | Phase 2/3 audit pass | Route/controller/serializer drift found by B12 or new reference inspection is captured as named slices, not free-form TODOs. | Regenerated route parity artifacts and fixture-backed serializer tests. | -| 4 | Phase 6 placeholder burn-down | Remaining account/contact/conversation/message/inbox placeholder handlers are either real Chatwoot-compatible flows or explicitly tracked as unsupported reference gaps. | `rg` placeholder audit, route smoke, and endpoint-family tests. | +| 1 | P5.10 outbound jobs | Queue Chatwoot `SendReplyJob`-style outbound provider delivery and provider delivery-status updates. | Focused message delivery worker tests, provider fake tests, and `go test ./...`. | +| 2 | P5.9 inbound jobs | Move provider webhook persistence/dispatch that Chatwoot defers into durable jobs while keeping HTTP acknowledgement behavior compatible. | Provider webhook handler tests plus worker replay/idempotency tests. | +| 3 | P5.11 remaining Captain/Copilot jobs | Queue Captain crawl/response/embedding and Copilot response work behind fakeable provider gates. | Captain/Copilot worker tests and disabled/failure-state fixtures. | +| 4 | P5.13 analytics aggregation | Replace frontend-visible placeholder report values with scheduled or cached real aggregations. | Report service/handler fixtures and freshness/idempotency tests. | +| 5 | Phase 2/3 audit pass | Route/controller/serializer drift found by B12 or new reference inspection is captured as named slices, not free-form TODOs. | Regenerated route parity artifacts and fixture-backed serializer tests. | +| 6 | Phase 6 placeholder burn-down | Remaining account/contact/conversation/message/inbox placeholder handlers are either real Chatwoot-compatible flows or explicitly tracked as unsupported reference gaps. | `rg` placeholder audit, route smoke, and endpoint-family tests. | ## Handoff Contract This checkpoint is intended to make the development plan complete enough to track without reading Hermes notes first. -- The next active implementation slice is B12. B11.3e has deterministic Captain stream disabled/error SSE behavior and Copilot push payload tests; provider-specific realtime delivery depth remains a Phase 5 integration concern if it needs durable dispatch infrastructure. +- The next active implementation slice is Phase 5 remaining job parity, starting with P5.10 outbound message delivery. B12 has repeatable API/browser/enterprise smoke harnesses in Review; optional live failures should be converted into named slices instead of blocking Phase 5 job work. - The Hermes search plan is fully represented by Phase 1/B6. Future search changes must be Meilisearch-first and must not reintroduce production DB fallback. - The Hermes automation/macro/CSAT plan is fully represented by B8/B9 and Phase 5. Durable delayed execution and channel-specific template delivery remain visible Phase 5 work, not hidden Done work. - Enterprise scope is fixed: SLA, Audit, CustomRole, AgentCapacity, Captain/Copilot, CSAT, InboxLimit, automation, macros, assignment policies, and related limits/workflows are in scope; SSO/SAML/LDAP/OIDC are out of scope. @@ -64,8 +66,8 @@ Open work after the current checkpoint: | Area | Next concrete action | Tracking location | Done boundary | | --- | --- | --- | --- | +| Phase 5 jobs | Finish outbound delivery, inbound webhook deferral, remaining Captain/Copilot jobs, and analytics aggregation on top of the committed durable worker. | `Phase 5: Background Jobs And Integrations` | Worker tests prove enqueue, retry, idempotency, and fakeable external effects. | | B12 smoke | Run optional live API/browser/enterprise smoke in a full PostgreSQL/Redis/Meilisearch/Vite environment and convert failures into named slices. | `B12 reused frontend verification breakdown` | `docs/parity/frontend_smoke_report.md` records checked pass/fail results and maps failures to slices. | -| Phase 5 jobs | Decide durable worker mechanism and wire deferred SLA/automation/macro/export/template delivery jobs behind idempotent boundaries. | `Phase 5: Background Jobs And Integrations` | Worker tests prove enqueue, retry, idempotency, and fakeable external effects. | | Phase 2/3 drift | Expand tracked route/serializer fixtures when B12 exposes frontend-critical gaps. | `Phase 2`, `Phase 3`, `docs/parity/` | Route parity remains 0 missing for tracked frontend routes; serializers have reference fixtures. | | Phase 6 placeholders | Re-run placeholder audit and burn down any frontend-reachable stub. | `Phase 6: Core Product Placeholder Burn-down` | Stub list has no reused-frontend critical path without a named owner. | @@ -77,11 +79,25 @@ Open work after the current checkpoint: | Phase 1 | Meilisearch search engine | Review | B6 payload parity, optional live gate, and DB-fallback hardening are implemented; an actual live Meilisearch run is optional and environment-dependent | | Phase 2 | Route and controller parity audit | Doing | Ruby/Bundler unavailable, so Chatwoot route extraction currently uses static `routes.rb` fallback | | Phase 3 | Data and serializer parity | Doing | JSON fixture coverage is partial and still endpoint-family based | -| Phase 4 | Enterprise feature completion | Doing | B7, B8, B9, B10, and B11 are in Review; B12 reused frontend smoke is the next broad verification gate | +| Phase 4 | Enterprise feature completion | Doing | B7, B8, B9, B10, and B11 are in Review; B12 reused frontend smoke harnesses exist and optional live runs can expose follow-up slices | | Phase 5 | Background jobs and integrations | Doing | P5.1/P5.2/P5.3/P5.4/P5.5/P5.6/P5.7/P5.8/P5.11/P5.12 durable worker, event dispatch, search indexing, automation delivery, macro, CSAT survey/template, SLA scan, contact export, Captain document sync, conversation maintenance, message status update, and account bulk-action cores are in Review; remaining Captain/Copilot/provider/outbound job integration remains open | | Phase 6 | Core placeholder burn-down | Doing | account/contact/conversation/message/inbox placeholder groups remain broad | | Phase 7 | Verification harness | Review | B12.1 boot/readiness, B12.2a API assertions, B12.2b browser smoke harness, B12.3a enterprise API assertions, and B12.3b enterprise browser route navigation exist; optional live Meilisearch/full-browser runs remain environment-dependent | +## Active Remaining Work Board + +This table is the shortest authoritative handoff view. If an older lower section still says a completed checkpoint is `Todo`, prefer this board and update the older row in the same commit that touches that area. + +| Priority | Workstream | Current state | Next checkpoint | Commit close rule | +| --- | --- | --- | --- | --- | +| 1 | P5.10 outbound message delivery | Not started after durable worker foundations. Existing outgoing message creation still needs Chatwoot `SendReplyJob` separation and duplicate-send review. | Queue outbound provider delivery and delivery-status updates through durable jobs. | Commit code, tests, P5.10 table row, commit ledger row, and progress log together. | +| 2 | P5.9 inbound webhook deferral | Provider ingress routes verify, persist, and dispatch synchronously. | Preserve provider HTTP ack behavior while moving Chatwoot-deferred persistence/dispatch into retryable jobs. | Provider HTTP tests plus worker replay/idempotency tests must pass. | +| 3 | P5.11 Captain/Copilot jobs | Captain document sync is queued; crawl/response/embedding/Copilot response jobs remain open. | Add fakeable durable jobs for remaining Captain/Copilot async work and disabled-provider states. | Focused Captain/Copilot worker tests plus full `go test ./...`. | +| 4 | P5.13 reports/analytics | Frontend smoke harness exists; `analytics_service` still has placeholder aggregation paths. | Replace frontend-visible report placeholders with real scheduled/cached aggregations. | Report fixtures verify values, cache/freshness behavior, and no hidden placeholder JSON. | +| 5 | Phase 2/3 drift | Tracked route parity is 0 missing for the current critical set; serializer fixtures remain partial. | Expand route/serializer fixtures when smoke or reference inspection exposes drift. | Regenerate parity artifacts and add endpoint-family fixture tests. | +| 6 | Phase 6 placeholder audit | Widget/public/webhook critical placeholders are burned down; account/contact/conversation/message/inbox audit remains broad. | Run a fresh placeholder audit and assign every frontend-reachable stub to a tracked owner. | `rg` audit result is recorded and no reused-frontend blocker is ownerless. | +| 7 | B12 optional live smoke | API/browser/enterprise smoke commands are checked in; live runs need PostgreSQL, Redis, Meilisearch, Vite, and Chrome. | Run full live smoke when environment is available and map failures to the board. | `docs/parity/frontend_smoke_report.md` records pass/fail and linked owners. | + ## Tracking Artifacts | Artifact | Purpose | Update rule | @@ -101,6 +117,7 @@ This ledger records the committed parity checkpoints that future slices should b | Commit | Scope | Verification summary | Follow-up state | | --- | --- | --- | --- | +| `docs: land active parity tracking board` | Documentation-only checkpoint that reconciles the current state after `cda7d02`. Adds the front-loaded Active Remaining Work Board, makes Phase 5 the active slice, records P5.10/P5.9/P5.11/P5.13 as the next implementation order, and updates stale B12/M6/M7 tracker rows to Review/Doing. | `git diff --check`; documentation-only change. | Start P5.10 outbound message delivery from the new active board. | | `feat(conversations): queue bulk actions` | Advances P5.12 with a durable Chatwoot `BulkActionsJob` equivalent for account conversation bulk actions. `POST /bulk_actions` now accepts frontend Chatwoot payloads without `action_name`, including `fields`, nested `labels.add/remove`, `snoozed_until`, and display-ID `ids`; when a WorkerPool is configured it enqueues `conversation:bulk_action` on the `medium` queue and returns empty `200 OK`. Worker replay scopes by account and display IDs, merges labels, applies status/team/assignee/snooze updates, and leaves same-display conversations in other accounts untouched. | `go test ./internal/service -run 'ConversationMaintenanceJobs.*BulkAction\|ConversationMaintenanceJobs' -count=1`; `go test ./internal/handler/api/v1 -run 'BulkActionHandler' -count=1`; `go test ./internal/service ./internal/handler/api/v1 ./internal/worker ./internal/app -count=1`; `go test ./...`; `git diff --check`; full verification recorded in the P5.12 section. | Move P5.12 account bulk actions to Review; continue remaining Captain/Copilot jobs, provider webhook/outbound delivery, inbound webhook job deferral, and analytics aggregation. | | `feat(conversations): queue message status updates` | Advances P5.12 with a durable Chatwoot `Conversations::UpdateMessageStatusJob` equivalent. Widget and public inbox `update_last_seen` paths now enqueue `conversation:update_message_status` on the `deferred` queue when a WorkerPool is configured. Worker replay validates `read`/`delivered`, skips missing conversations, and updates only non-incoming messages created at or before the viewing timestamp while preserving already-read and post-cutoff messages. | `go test ./internal/service -run 'ConversationMaintenanceJobs\|WidgetService_UpdateLastSeenQueuesMessageStatusJob' -count=1`; `go test ./internal/service ./internal/worker ./internal/app -count=1`; `go test ./...`; `git diff --check`; full verification recorded in the P5.12 section. | Move P5.12 update-message-status depth to Review; continue durable bulk-action fan-out, remaining Captain/Copilot jobs, provider webhook/outbound delivery, and analytics aggregation. | | `feat(captain): queue document syncs` | Advances P5.11 with a durable Chatwoot `Captain::Documents::PerformSyncJob` equivalent. `POST /captain/documents/:id/sync` still returns `202` after marking the account-scoped document as syncing, but now enqueues `captain:document_sync` on the `low` queue when a WorkerPool is configured. Worker replay calls the existing fakeable `CaptainDocumentSyncBackend`, persists synced/failed/disabled states through `SyncDocumentByAccount`, and lets missing documents or malformed payloads retry through `background_jobs`. | `go test ./internal/service -run 'CaptainDocumentService' -count=1`; `go test ./internal/handler/api/v1 -run 'CaptainDocument\|CaptainResource' -count=1`; `go test ./internal/service ./internal/handler/api/v1 ./internal/worker ./internal/app -count=1`; `go test ./...`; `git diff --check`; full verification recorded in the P5.11 section. | Move P5.11 document sync core to Review; continue Captain crawl/response-building/embedding/Copilot response jobs, provider webhook/outbound delivery, update-message-status/bulk-action jobs, and analytics aggregation. | @@ -467,10 +484,10 @@ Upcoming enterprise task boards: | B10 | B10.4 | Review remaining InboxLimit/account-limit create-path enforcement outside capacity policies. | Chatwoot enterprise account/inbox limit policies. | Inbox/channel create tests for over-limit behavior and frontend-readable errors. | Done by `feat(inboxes): enforce chatwoot inbox limits` | | B11 | B11.1 | Align Captain assistant CRUD, inbox bindings, responses, documents, scenarios, and custom tools payloads. | Captain controllers/services/frontend clients under `reference/chatwoot`. | Handler/service fixtures for every Captain dashboard client path. | Review; assistant CRUD/tools/inbox binding, scenarios, custom tools, documents, assistant responses, bulk actions, and custom-tool test payloads are landed | | B11 | B11.2 | Align Copilot threads, messages, tasks, preferences, playground/tool-call behavior, and disabled-state feature gates. | Copilot controllers/services/frontend clients under `reference/chatwoot`. | Copilot handler/service tests for persistence, disabled LLM state, and frontend payloads. | Review; thread/message payloads, account/user scoping, assistant scope, and no-LLM fallback are landed; tasks/preferences/tool-call/playground depth remains in B11.3 follow-up | -| B11 | B11.3 | Add document sync/embedding/LLM job boundaries where external dependencies are required and finish remaining Copilot task/preference/tool-call/streaming depth. | Captain/Copilot jobs, document services, Copilot controllers/services/frontend clients. | Worker tests or explicit feature-gated fallback tests plus Copilot task/preference/tool-call fixtures. | Doing; B11.3a Captain preferences show/update payloads are landed, while document sync/indexing and remaining Copilot task/tool-call depth remain active | +| B11 | B11.3 | Add document sync/embedding/LLM job boundaries where external dependencies are required and finish remaining Copilot task/preference/tool-call/streaming depth. | Captain/Copilot jobs, document services, Copilot controllers/services/frontend clients. | Worker tests or explicit feature-gated fallback tests plus Copilot task/preference/tool-call fixtures. | Review; B11.3a preferences, B11.3b playground, B11.3c document sync gate, B11.3d task/tool-call persistence, and B11.3e streaming fallback landed. Remaining durable crawl/response/embedding/Copilot jobs are P5.11 | | B12 | B12.1 | Add a repeatable command to run the reused Chatwoot frontend against GoChat. | `reference/chatwoot` frontend boot/auth/API clients. | Smoke command documented and runnable locally. | Done by `test(parity): add frontend smoke harness` | | B12 | B12.2 | Cover login, inbox list/settings, conversation list/detail/message send, contact/company views, widget init/message, public CSAT, SLA/CSAT reports, and enterprise admin screens. | Dashboard route usage and frontend stores/API modules. | Smoke report checked into `docs/parity/` with pass/fail gaps. | Review; B12.2a API assertion command and B12.2b browser smoke harness landed; live run is environment-dependent | -| B12 | B12.3 | Add enterprise smoke coverage for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot. | Enterprise dashboard clients and `reference/chatwoot` enterprise screens. | Enterprise smoke command records pass/fail and names failed route/serializer owners. | Doing; B12.3a API assertion command landed, browser screen navigation remains open | +| B12 | B12.3 | Add enterprise smoke coverage for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot. | Enterprise dashboard clients and `reference/chatwoot` enterprise screens. | Enterprise smoke command records pass/fail and names failed route/serializer owners. | Review; B12.3a enterprise API assertions and B12.3b enterprise browser navigation landed. Optional live execution is environment-dependent | B8 CSAT execution breakdown: @@ -905,9 +922,9 @@ B11.3c-B11.3e follow-up contracts: | Slice | Landing rule | Status gate | | --- | --- | --- | -| B11.3c document sync/indexing | Document sync must expose frontend-visible `syncing/synced/failed` states and call a fakeable Meilisearch/embedding boundary. Missing provider config must be a raw disabled/failure state, not a placeholder success. | B11 remains `Doing` until disabled, failed, and fake-success sync tests are present. | -| B11.3d Copilot tasks/tool-calls | Rewrite, summarize, reply suggestion, label suggestion, and follow-up task routes must accept Chatwoot request payloads, persist enough task/tool-call output for reloads, and serialize raw frontend payloads. | Done by `feat(captain): align task payload persistence`; B11 remains `Doing` for B11.3e streaming/realtime fallback. | -| B11.3e streaming/realtime | Streaming can be deferred only with a deterministic non-streaming response or explicit disabled state. Realtime push payloads should use the same thread/message serializer already landed in B11.2. | B11 moves to `Review` only after every streaming/realtime endpoint has a tested frontend-compatible fallback. | +| B11.3c document sync/indexing | Document sync must expose frontend-visible `syncing/synced/failed` states and call a fakeable Meilisearch/embedding boundary. Missing provider config must be a raw disabled/failure state, not a placeholder success. | Done by `feat(captain): gate document sync backend`; remaining crawl/embedding jobs are tracked in P5.11. | +| B11.3d Copilot tasks/tool-calls | Rewrite, summarize, reply suggestion, label suggestion, and follow-up task routes must accept Chatwoot request payloads, persist enough task/tool-call output for reloads, and serialize raw frontend payloads. | Done by `feat(captain): align task payload persistence`. | +| B11.3e streaming/realtime | Streaming can be deferred only with a deterministic non-streaming response or explicit disabled state. Realtime push payloads should use the same thread/message serializer already landed in B11.2. | Done by `feat(captain): align streaming fallbacks`; B11 is in Review. | B11.3c current checkpoint: @@ -1036,7 +1053,7 @@ B12 smoke harness contract: | Seed data | Create or document deterministic seed data for admin login, account, inbox, contact, conversation, CSAT, SLA, macro, automation, audit, and custom-role screens. | Seed command or fixture reference in `docs/parity/`. | | Core smoke paths | Cover login/current-user, inbox list/settings, conversation list/detail/message send, contact/company views, widget config/message, and public CSAT. | `docs/parity/frontend_smoke_report.md` with pass/fail status and failed API calls. | | Enterprise smoke paths | Cover SLA reports, CSAT reports/download, automation rules, macros, audit logs, custom roles, capacity settings, Captain, and Copilot as their slices land. | Same smoke report links each failure to the owning B-slice. | -| Exit rule | A failing smoke does not block code commits if the failure is named, scoped, and tracked; hidden failures block moving B12 out of `Doing`. | B12 moves to `Review` only with a repeatable command and checked report. | +| Exit rule | A failing optional live smoke does not block code commits if the failure is named, scoped, and tracked; hidden failures block moving B12 to `Done`. | B12 stays in `Review` until live pass/fail results are recorded or every environment-dependent gap has an owner. | B12.1 current checkpoint: @@ -1044,7 +1061,7 @@ B12.1 current checkpoint: - `scripts/parity_frontend_smoke.sh --check` verifies local command prerequisites, validates `cmd/gochat`, and writes `docs/parity/frontend_smoke_report.md` without requiring live PostgreSQL/Redis/Meilisearch/frontend boot. - `scripts/parity_frontend_smoke.sh --print` prints the exact GoChat and reused Chatwoot frontend commands. - `scripts/parity_frontend_smoke.sh --boot-only` starts GoChat plus `reference/chatwoot` Vite and verifies backend `/health` plus frontend HTTP readiness. The default backend command is Meilisearch-first via `GOCHAT_SEARCH_ENGINE=meilisearch`; DB fallback is only opt-in through `GOCHAT_SMOKE_SEARCH_ENGINE=db` for local debugging. -- The current report is a readiness report, not a real browser pass. B12.2 must add login/current-user/inbox/conversation/CRM/widget/CSAT browser or API assertions and update the smoke matrix with pass/fail results. +- B12.1 itself is a readiness checkpoint. B12.2 and B12.3 below add the repeatable API/browser assertions and enterprise route coverage; optional live execution still depends on the full local service stack. B12.1 verification: @@ -1062,7 +1079,7 @@ B12.2a current checkpoint: - `scripts/parity_frontend_smoke.sh --api-smoke` now runs against a live GoChat backend. It seeds deterministic data, logs in with `/auth/sign_in`, captures DeviseTokenAuth-compatible headers, and exercises the frontend-critical API paths without frontend adapters. - Covered API assertions: `/auth/validate_token`, `/api/v1/profile`, inbox list, conversation list/messages/message create, contact show, company show, widget config, widget message create, and public CSAT show. - `cmd/gochat seed` now emits `company_id`, `conversation_display_id`, and `csat_message_id`; it also creates a deterministic company relation and an `input_csat` message so B12 API smoke can verify CRM and public CSAT surfaces. -- This is still not a browser-navigation pass. B12.2b must drive the reused Vite frontend and capture failed API calls or UI blockers in the report; B12.3 must add enterprise screens. +- This is the core API assertion checkpoint. B12.2b below adds reused Vite browser navigation, and B12.3 adds enterprise API/browser coverage. B12.2a verification: @@ -1197,7 +1214,7 @@ This is the ordered queue for the next implementation slices. Do not skip the ro | Q5 | Review Meilisearch document shape and endpoint payloads against Chatwoot frontend consumers. | Phase 1 and Phase 3 | Search remains Meilisearch-first and payload mismatches are fixed or tracked. | Review | | Q6 | Implement provider-specific webhook ingress for Chatwoot public webhook paths. | Phase 6 | Generic webhook placeholder no longer masks provider gaps; Telegram, LINE, SMS/Twilio, WhatsApp, Instagram/Twitter/TikTok routes resolve and verify like Chatwoot where supported. | Done | | Q7 | Burn down enterprise gaps in this order: SLA, assignment policy and capacity, CSAT, automation/macros, Audit, CustomRole, InboxLimit, Captain/Copilot. | Phase 4 and Phase 5 | Each feature passes route, persistence, auth, side-effect, response, and test checks. | Doing | -| Q8 | Add frontend smoke harness using the reused Chatwoot frontend once core API flows boot end-to-end. | Phase 7 | Login, inbox list, conversation list/detail, message send, contact view, and widget init run without frontend adapters. | Todo | +| Q8 | Add frontend smoke harness using the reused Chatwoot frontend once core API flows boot end-to-end. | Phase 7 | Login, inbox list, conversation list/detail, message send, contact view, widget init/message, public CSAT, and key enterprise routes have repeatable API/browser smoke commands. | Review | ## Current Decision Ledger @@ -1223,8 +1240,8 @@ These milestones are the tracking spine for the remaining Chatwoot frontend reus | M3 | Serializer parity for frontend API families. | Fixture tests cover auth/profile, accounts/users, inboxes, conversations/messages, contacts/companies, notifications, reports, widget/public, and search. | Doing | | M4 | Core handler placeholder burn-down. | Account/contact/conversation/message/inbox/webhook handlers are repository-backed and no frontend-critical route returns placeholder JSON. | Doing | | M5 | Paid feature parity excluding SSO family. | SLA, Audit, CustomRole, AgentCapacity, Captain/Copilot, CSAT, InboxLimit, automation/macros, and assignment policies pass route, persistence, auth, side-effect, serializer, and tests. | Doing | -| M6 | Durable jobs and external integrations. | Search indexing, CSAT send, automation actions, notifications, webhooks, and external deliveries are queued, retryable, logged, and idempotent. | Todo | -| M7 | Reused Chatwoot frontend smoke validation. | Chatwoot frontend boots against GoChat for login, inbox list, conversation detail, message send, contact view, widget init/message, public CSAT, and key enterprise screens without adapters. | Todo | +| M6 | Durable jobs and external integrations. | Search indexing, CSAT send, automation actions, notifications, webhooks, and external deliveries are queued, retryable, logged, and idempotent. | Doing | +| M7 | Reused Chatwoot frontend smoke validation. | Chatwoot frontend boots against GoChat for login, inbox list, conversation detail, message send, contact view, widget init/message, public CSAT, and key enterprise screens without adapters. | Review | ## Slice Backlog @@ -1243,7 +1260,7 @@ Work proceeds top-down unless a failing test or frontend blocker forces a narrow | B9 | Automation/macros durable side effects. | Chatwoot automation/macro services and jobs. | Action execution, logs, webhook/email transcript retry tests. | Review | | B10 | Audit, CustomRole, InboxLimit. | Chatwoot enterprise admin behavior and policies. | Authorization, audit emission, limits enforcement, admin payload fixtures. | Review | | B11 | Captain/Copilot deep behavior. | Chatwoot Captain/Copilot controllers, services, frontend clients. | Assistant/tool/document/scenario/copilot thread/task tests and feature gates. | Review | -| B12 | Frontend smoke harness. | `reference/chatwoot` frontend. | Repeatable smoke command and checked gap report. | Doing; B12.1 harness landed, B12.2 browser/API assertions pending | +| B12 | Frontend smoke harness. | `reference/chatwoot` frontend. | Repeatable smoke command and checked gap report. | Review; B12.1 boot harness, B12.2 API/browser assertions, and B12.3 enterprise API/browser assertions landed. Optional live runs remain environment-dependent | Remaining slice landing plan: @@ -1255,7 +1272,7 @@ Remaining slice landing plan: | B9 | Done: B9.1 automation rule CRUD/listener/log/external-action parity and B9.2 macro frontend CRUD/execute side effects. | Delayed actions, durable queued worker scheduling, and deeper macro attachment/file parity remain named B9.3/B9.4 follow-ups. | Review after `feat(macros): align chatwoot macro payloads`; move to Done only after durable worker/attachment gaps are implemented or formally split out. | | B10 | Done: audit list payload, audit writer boundary for representative mutating core resources, CustomRole permission-key parity, AccountUser permission resolution, admin gates, delete nullification, and account-level InboxLimit enforcement in inbox/channel creation paths. | Frontend smoke coverage for enterprise settings remains B12, not a hidden B10 blocker. | Review after B10.4; move to Done only after reused frontend smoke confirms audit/custom-role/limit settings flows or any smoke gaps are split into owned follow-ups. | | B11 | Done through B11.3e: Captain Assistant CRUD, inbox binding, scenarios, documents, responses, custom tools, Copilot threads/messages, Captain preferences, playground, document sync gates, task/tool-call payload persistence, stream disabled-state SSE, and Copilot push payload fixtures. | Reused frontend smoke remains B12; provider-specific durable realtime/LLM delivery can move to Phase 5 if smoke exposes it. | Review after B11.3e; move to Done only after B12 proves reused frontend Captain/Copilot screens or names any remaining provider/deployment follow-ups. | -| B12 | B12.1 boot/readiness harness landed: GoChat `serve`/`seed`, Meilisearch-first smoke script, and checked report. | Add B12.2 browser/API assertions for auth/profile/inbox/conversation/contact/widget/CSAT, then B12.3 enterprise screens. | Done only after checked smoke paths have pass/fail results and every failure is mapped to an owning slice. | +| B12 | B12.1 boot/readiness, B12.2 core API/browser assertions, and B12.3 enterprise API/browser route assertions landed. | Optional live smoke runs in a full PostgreSQL/Redis/Meilisearch/Vite/Chrome environment must map every failure to an owning slice. | Move to Done only after live-smoke pass/fail results are recorded and every failure is owned. | Per-slice documentation rule: @@ -1575,7 +1592,7 @@ Excluded tracking table: ## Phase 5: Background Jobs And Integrations -Status: doing; this is the next active implementation slice after B12 smoke harness review. +Status: doing; this is the current active implementation slice. Default implementation decision for tracking: use a PostgreSQL-backed durable job table first, with Redis used only for optional locks/rate limiting where existing code already depends on it. This keeps job state inspectable in the Go backend, avoids adding a new hard runtime beyond the already-required database, and still allows Chatwoot-style retry, scheduling, idempotency, and observability. If a specific reference behavior requires Redis timing semantics, record that under the owning P5 task before implementation. @@ -1590,20 +1607,20 @@ Reference job families to keep in scope: Known hotspots: -- `internal/worker/worker.go` now has the P5.1 durable core, but no feature-specific producers/handlers are wired yet. -- `internal/automation/action_service.go` has pending webhook/email transcript work. -- `internal/automation/csat_survey_listener.go` now handles CSAT enablement, survey rules, generic send idempotency, and message-updated response building; durable queueing and external WhatsApp/Twilio template delivery remain follow-ups. -- `internal/auth/webhook_registry.go` has pending signature verification for Facebook/WhatsApp. -- `internal/service/analytics_service.go` has placeholder analytics paths. +- `internal/worker/worker.go` has the committed P5.1 durable core and is already wired to dispatcher, search indexing, automation delivery, macro fan-out, CSAT survey/template sends, SLA scans, contact exports, Captain document sync, and conversation maintenance jobs. +- `internal/service/message_service.go`, `internal/channel/registry.go`, and provider outbound listeners remain the next P5.10 focus for Chatwoot `SendReplyJob` parity and duplicate-send prevention. +- `internal/handler/webhook/*` and provider services have synchronous verified ingress/persistence; P5.9 must move Chatwoot-deferred webhook work behind durable jobs without changing provider acknowledgement semantics. +- `internal/service/captain_*` and `internal/service/copilot_*` have document sync job coverage only; P5.11 still needs crawl/response/embedding/Copilot response jobs. +- `internal/service/analytics_service.go` has placeholder analytics/report paths and is the owner for P5.13 aggregation work. Checklist: - [x] Add `background_jobs` persistence with job type, payload, queue, status, attempt counters, scheduled/locked timestamps, idempotency key, last error, and completion timestamps. - [x] Replace `internal/worker/worker.go` placeholder with enqueue, schedule, perform, retry/backoff, dead-letter, and graceful shutdown behavior. -- [ ] Map Chatwoot jobs/listeners to Go worker responsibilities and record out-of-scope migration-only jobs explicitly. -- [ ] Implement durable job dispatch for automation, CSAT, notifications, webhooks, SLA processing, Captain/Copilot, contact export, macros, and search indexing. -- [ ] Add retry and failure logging for external calls. -- [ ] Add tests for enqueueing, uniqueness/idempotency, delayed execution, retries, worker restart pickup, and fakeable external side effects. +- [x] Map the committed Chatwoot job/listener families to Go worker responsibilities in the tracking table. +- [ ] Finish durable job dispatch for outbound provider delivery/status, inbound provider webhook deferral, remaining Captain/Copilot jobs, delayed automation actions, and report aggregation. +- [ ] Add retry and failure logging for the remaining external provider calls. +- [ ] Add tests for the remaining enqueueing, uniqueness/idempotency, delayed execution, retries, worker restart pickup, and fakeable external side effects. Acceptance: @@ -1634,7 +1651,7 @@ P5.1 current checkpoint: - Replaced the `WorkerPool` stub with enqueue/schedule APIs, handler registration, `ProcessOne`, start/stop worker loops, PostgreSQL `FOR UPDATE SKIP LOCKED` claiming, retry/backoff, dead-letter state, and stale-lock recovery for worker restart pickup. - Kept older `NewWorkerPool()` construction as a no-op-compatible path while adding `NewWorkerPoolWithOptions(db, ...)` for durable wiring. - Added focused tests proving enqueue/idempotency, due job completion, retry to dead-letter, queue/schedule filtering, and stale running job requeue. -- Remaining Phase 5 work must now wire real producers/handlers: dispatcher async events, Meilisearch indexing, automation webhooks/transcripts, delayed actions/macros, CSAT templates, SLA scans, contact exports, provider webhooks, outbound delivery, Captain/Copilot, and conversation maintenance. +- Remaining Phase 5 work: delayed automation scheduled-item execution, inbound provider webhook deferral, outbound provider delivery/status jobs, Captain crawl/response/embedding/Copilot response jobs, and analytics/report aggregation. P5.1 verification: @@ -1895,6 +1912,7 @@ Verification milestone gates: ## Progress Log +- 2026-06-05: Active parity tracker checkpoint prepared as `docs: land active parity tracking board`; the plan now front-loads the current authoritative remaining-work board, points the next implementation slice at P5.10 outbound message delivery, records P5.9 inbound job deferral, P5.11 remaining Captain/Copilot jobs, P5.13 analytics aggregation, Phase 2/3 drift, Phase 6 placeholder audit, and B12 optional live smoke as tracked owners, and reconciles stale B12/M6/M7 status rows. Documentation-only checkpoint; `git diff --check` passed. - 2026-06-04: Baseline stabilized and committed as `42cdab8 chore: stabilize chatwoot parity baseline`; `go test ./...` passed and route dump reported `TOTAL: 704`. - 2026-06-04: Phase 1 search foundation added: Meilisearch config/env defaults, `SearchEngine` contract, Meilisearch HTTP wrapper with bootstrap/settings, DB fallback adapter, document builders, reindex command, and no-live-Meilisearch tests. Verified `go test ./...` in unsandboxed mode because miniredis/httptest need local sockets; route dump still reports `TOTAL: 704`. - 2026-06-04: Phase 1 indexing hooks wired for conversations, messages, contacts, companies, and articles. Create/update/delete paths now call the service-layer `SearchIndexer` boundary, bootstrap injects the Meilisearch-backed search service, and unit tests cover each entity hook path.