diff --git a/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md b/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md index b0c2e78e..b3ae8456 100644 --- a/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md +++ b/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md @@ -16,9 +16,9 @@ Build GoChat as a Go backend that can directly reuse the frontend from `referenc ## Current Baseline -- Latest implementation checkpoint: `a16c23c feat(search): align chatwoot search payloads`. -- Latest documentation checkpoint before this update: `178bc36 docs: record channel route cleanup checkpoint`. -- Worktree status at this planning checkpoint: clean after the matching implementation checkpoint; next active slice is B6 live Meilisearch gate and DB-fallback hardening. +- Latest implementation checkpoint: `f08c743 test(search): add meilisearch live gate`. +- Latest documentation checkpoint before this update: `f36512e docs: record search payload checkpoint`. +- Worktree status at this planning checkpoint: clean after the matching implementation checkpoint; next active slice is either running the optional live Meilisearch gate or starting B7 SLA/assignment capacity. - `go test ./...` passes. - Route dump succeeds with `TOTAL: 829` after adding Chatwoot-compatible nested AgentCapacityPolicy users and inbox-limit routes. - Route parity artifacts now exist under `docs/parity/` and are generated by `cmd/route_parity`. @@ -32,7 +32,7 @@ Build GoChat as a Go backend that can directly reuse the frontend from `referenc | Phase | Name | Status | Blocking gaps | | --- | --- | --- | --- | | Phase 0 | Test and route baseline | Done | none | -| Phase 1 | Meilisearch search engine | Doing | B6 payload parity has a mocked Meilisearch checkpoint; live integration and DB-fallback hardening still need verification | +| Phase 1 | Meilisearch search engine | Review | B6 payload parity, optional live gate, and DB-fallback hardening are implemented; an actual live Meilisearch run is optional and environment-dependent | | Phase 2 | Route and controller parity audit | Doing | Ruby/Bundler unavailable, so Chatwoot route extraction currently uses static `routes.rb` fallback | | Phase 3 | Data and serializer parity | Doing | JSON fixture coverage is partial and still endpoint-family based | | Phase 4 | Enterprise feature completion | Doing | excluded SSO family must stay out of scope; all other enterprise features remain included | @@ -94,21 +94,20 @@ This ledger records the committed parity checkpoints that future slices should b | `b197e54 feat(capacity): align chatwoot inbox capacity limits` | Completed B5.5c AgentCapacityPolicy/InboxCapacityLimit API/data parity: policy CRUD now returns raw Chatwoot payloads with Unix timestamps, `assigned_agent_count`, and `inbox_capacity_limits`; `assignment_logic` is optional/defaulted; nested policy users and inbox limits are registered; `InboxCapacityLimit` validates account scope, duplicate inbox assignment, and non-negative limits; account users can be assigned/unassigned to capacity policies. | `go test ./internal/service -run AgentCapacity -count=1`; `go test ./internal/handler/api/v1 -run AgentCapacity -count=1`; `go test ./internal/router -count=1`; `go test ./internal/handler/api/v1 -count=1`; sandboxed `go test ./...` failed on local socket restrictions; escalated `go test ./...` passed; route dump regenerated with `TOTAL: 829`; route parity is `267 exact, 7 parameter-compatible, 0 missing`; `git diff --check`. | Continue final B5.5 review for remaining channel-specific route response cleanup, then move to B6 Meilisearch live-shape review. | | `f04a03b feat(channels): align channel route inbox payloads` | Completed the final B5.5 channel route-response cleanup: Email, Twilio SMS, and LINE dedicated channel create/get/update/list routes now return Chatwoot frontend-compatible raw inbox payloads or `{ payload: [...] }` lists instead of `{ channel }`, `{ channels }`, `{ channel, inbox }`, or success-message envelopes. Delete routes now return empty `200 OK`, and channel creation binds the dedicated channel ID/config back onto the inbox before serialization. | `go test ./internal/handler/api/v1 -run 'Test(Email\|Twilio\|LINE)Channel' -count=1`; `go test ./internal/service -run Inbox -count=1`; `go test ./internal/handler/api/v1 -count=1`; `go test ./...`; `git diff --check`. No route changes; route dump remains `TOTAL: 829`. | B5 inbox/channel API parity moves to Done for current frontend-critical scope; next active slice is B6 Meilisearch live-shape review. | | `a16c23c feat(search): align chatwoot search payloads` | Advanced B6 search live-shape parity: global and entity search endpoints now return Chatwoot frontend `payload` envelopes, search filters accept Chatwoot `since`, `until`, and `from=contact:id/agent:id` params, message sender IDs are indexed/filterable in Meilisearch and DB fallback, SearchController default page size is aligned to 15, and Meilisearch hit maps are serialized into frontend message/contact/conversation/article shapes. | `go test ./internal/handler/api/v1 -run SearchHandler -count=1`; `go test ./internal/search -run 'SearchFilter\|Meili\|Engine' -count=1`; `go test ./internal/repository -run Search -count=1`; `go test ./internal/handler/api/v1 -count=1`; `go test ./internal/search -count=1`; `go test ./internal/repository -count=1`; sandboxed `go test ./...` failed on local socket restrictions; escalated `go test ./...` passed; `git diff --check`. No route changes; route dump remains `TOTAL: 829`. | B6.1 and B6.3 are Done; B6.2 is in Review with mocked Meilisearch coverage. Continue B6.4 live integration gate and B6.5 DB-fallback hardening before moving B6 to Review/Done. | +| `f08c743 test(search): add meilisearch live gate` | Completed B6 gate/hardening work: added an env-gated live Meilisearch test that bootstraps isolated indexes, indexes message/contact documents, verifies account-scoped searches and sender filters, rejects `search.engine=db` in release mode, and prevents `cmd/reindex_search` from silently running against the DB fallback. | `go test ./internal/config -count=1`; `go test ./cmd/reindex_search -count=1`; `go test ./internal/search -count=1`; `go test ./...`; `git diff --check`. Live gate is skipped unless `GOCHAT_LIVE_MEILI_HOST` is set. No route changes; route dump remains `TOTAL: 829`. | B6 moves to Review. Optional next verification is running `GOCHAT_LIVE_MEILI_HOST=http://localhost:7700 GOCHAT_LIVE_MEILI_API_KEY=... go test ./internal/search -run TestLiveMeiliSearchEngineIndexesAndSearchesChatwootShapes -count=1`; otherwise continue B7 SLA/assignment capacity. | ## Next Slice Contract -Completed implementation slice: B5.1-B5.5 now cover inbox serializer shape, Chatwoot frontend create/update binding, working-hours persistence, out-of-office behavior, inbox member assignment payload/mutation semantics, channel-specific config depth, AgentCapacityPolicy/InboxCapacityLimit API data contracts, and dedicated Email/Twilio/LINE channel route response shapes. B6 now has its first search payload checkpoint in `a16c23c`: Chatwoot search response envelopes, frontend query params, Meilisearch sender filters, and mocked hit serialization are covered. B3 and B4 remain in review for deeper side effects and browser validation. +Completed implementation slice: B5.1-B5.5 now cover inbox serializer shape, Chatwoot frontend create/update binding, working-hours persistence, out-of-office behavior, inbox member assignment payload/mutation semantics, channel-specific config depth, AgentCapacityPolicy/InboxCapacityLimit API data contracts, and dedicated Email/Twilio/LINE channel route response shapes. B6 is now in Review after `a16c23c` and `f08c743`: Chatwoot search response envelopes, frontend query params, Meilisearch sender filters, mocked hit serialization, env-gated live Meilisearch validation, release-mode DB fallback rejection, and reindex Meilisearch-only guard are covered. B3 and B4 remain in review for deeper side effects and browser validation. -Next implementation slice: finish B6.4 live Meilisearch gate and B6.5 DB-fallback hardening, then move to B7 SLA/assignment capacity unless a reused-frontend search smoke exposes a blocker. +Next implementation slice: start B7 SLA/assignment capacity unless an operator wants to run the optional live Meilisearch gate first. | Step | Required result | Reference source | Verification | | --- | --- | --- | --- | -| N1 | Keep B6 search payload checkpoint as the current baseline. | `SearchController`, `search.js`, `conversationSearch.js`, search Jbuilder views. | Done by `a16c23c`; response envelopes, Chatwoot params, sender filters, and mocked Meilisearch hit shape are covered. | -| N2 | Add the live Meilisearch integration gate. | `cmd/reindex_search`, `internal/search/engine_meili.go`, local Meilisearch env. | Env-gated test or documented command starts from indexed data, runs global/messages/contacts searches, and records result shape. | -| N3 | Harden and document DB fallback as development-only. | Search config defaults, `NewSearchEngine`, bootstrap wiring. | Config tests keep `meilisearch` as default; docs call DB fallback non-production and explain how to fail/alert when Meilisearch is unavailable. | -| N4 | Decide B6 close status after live gate. | B6 task board and Phase 1 checklist. | Mark B6 `Review` or `Done` only after live gate and fallback hardening are recorded; otherwise keep B6 `Doing`. | -| N5 | Start B7 SLA and assignment capacity. | Chatwoot enterprise SLA policies, assignment policies, capacity enforcement. | First B7 checkpoint must include routes/data/service tests before deeper timer/report behavior. | -| N6 | Update this tracker after every implementation checkpoint. | This document. | `git diff --check`; `go test ./...` for Go changes. | +| N1 | Keep B6 search payload/gate checkpoint as the current baseline. | `SearchController`, `search.js`, `conversationSearch.js`, search Jbuilder views, `cmd/reindex_search`. | Done by `a16c23c` and `f08c743`; response envelopes, Chatwoot params, sender filters, mocked hit shape, live gate, and fallback guards are covered. | +| N2 | Optionally run the live Meilisearch gate when a local Meilisearch instance is available. | `internal/search/engine_meili_live_test.go`. | `GOCHAT_LIVE_MEILI_HOST=http://localhost:7700 GOCHAT_LIVE_MEILI_API_KEY=... go test ./internal/search -run TestLiveMeiliSearchEngineIndexesAndSearchesChatwootShapes -count=1`. | +| N3 | Start B7 SLA and assignment capacity. | Chatwoot enterprise SLA policies, assignment policies, capacity enforcement. | First B7 checkpoint must include routes/data/service tests before deeper timer/report behavior. | +| N4 | Update this tracker after every implementation checkpoint. | This document. | `git diff --check`; `go test ./...` for Go changes. | Current B2 profile checkpoint: @@ -264,8 +263,8 @@ Active B6 task board: | B6.1 | Inventory Chatwoot search frontend request/response consumers and current Go search routes. | `reference/chatwoot/app/javascript/dashboard/api/search.js`, `conversationSearch.js`, `SearchController`, `internal/handler/api/v1/search_handler.go`. | Done | Matrix below records global, contacts, conversations, messages, and articles params/payload fields. | | B6.2 | Compare Meilisearch document fields and filters against frontend payload needs. | `internal/search/engine.go`, `engine_meili.go`, document builders, Chatwoot search views/entities. | Review | `a16c23c` adds mocked Meilisearch sender filter and hit-shape tests; live Meilisearch gate remains B6.4. | | B6.3 | Tighten endpoint serializers for `/search`, `/search/contacts`, `/search/conversations`, `/search/messages`, and `/search/articles`. | `reference/chatwoot/app/controllers/api/v1/accounts/search_controller.rb`, dashboard search API specs. | Done | `SearchHandler` tests assert Chatwoot `payload` envelopes and Meilisearch hit serialization. | -| B6.4 | Add optional live Meilisearch integration gate. | Local Meilisearch flow in Phase 1 and `cmd/reindex_search`. | Todo | Env-gated test or documented command runs reindex plus representative searches against a live Meilisearch instance. | -| B6.5 | Document DB fallback as development-only and verify production config remains Meilisearch-first. | User decision ledger and search config. | Review | Existing config defaults to `meilisearch`; next step is a focused config/doc warning around DB fallback usage. | +| B6.4 | Add optional live Meilisearch integration gate. | Local Meilisearch flow in Phase 1 and `cmd/reindex_search`. | Done | `f08c743` adds `TestLiveMeiliSearchEngineIndexesAndSearchesChatwootShapes`, skipped unless `GOCHAT_LIVE_MEILI_HOST` is set. | +| B6.5 | Document DB fallback as development-only and verify production config remains Meilisearch-first. | User decision ledger and search config. | Done | `f08c743` rejects DB fallback in release mode and prevents `reindex_search` from using DB fallback. | B6 request and payload matrix after `a16c23c`: @@ -283,7 +282,8 @@ B6 current checkpoint: - `search.ParseSearchFilter` now understands Chatwoot dashboard `since`/`until` Unix seconds and `from=contact:id`/`from=agent:id` sender filters while preserving legacy `date_from`/`date_to` and `sender_id` inputs. - Meilisearch documents now store `sender_id`, index settings mark it filterable, and DB fallback applies the same sender ID filter for local tests/dev mode. - Search handler pagination defaults to Chatwoot's 15 items when `per_page` is not supplied. -- Remaining B6 risk: live Meilisearch reindex/search has not been run in this checkpoint, and DB fallback needs an explicit production-safety warning/test before B6 can leave Doing. +- The optional live gate is implemented but not run by default. It requires `GOCHAT_LIVE_MEILI_HOST` and validates isolated live indexes, account-scoped contact/message search, and message sender filters. +- DB fallback remains available for explicit local development, but release config validation and `reindex_search` now refuse it so production parity stays Meilisearch-first. Active B4 task board: @@ -313,7 +313,7 @@ This is the ordered queue for the next implementation slices. Do not skip the ro | Q2 | Patch route aliases discovered by Q1, especially Chatwoot widget/public paths such as `/api/v1/widget/...` versus existing `/widget/...`. | Phase 2 | Missing tracked routes return to zero or are explicitly documented with implementation tasks. | Done | | Q3 | Add route boot regression coverage for wildcard conflict groups before expanding more Rails-style resources. | Phase 2 | Router tests cover nested dynamic resources that previously risked Gin conflicts. | Done | | Q4 | Start serializer parity fixtures for auth/session, conversations/messages, contacts/companies, inboxes, notifications, and search. | Phase 3 | Each area has at least one reference fixture and Go response test. | Doing | -| Q5 | Review Meilisearch document shape and endpoint payloads against Chatwoot frontend consumers. | Phase 1 and Phase 3 | Search remains Meilisearch-first and payload mismatches are fixed or tracked. | Doing | +| Q5 | Review Meilisearch document shape and endpoint payloads against Chatwoot frontend consumers. | Phase 1 and Phase 3 | Search remains Meilisearch-first and payload mismatches are fixed or tracked. | Review | | Q6 | Implement provider-specific webhook ingress for Chatwoot public webhook paths. | Phase 6 | Generic webhook placeholder no longer masks provider gaps; Telegram, LINE, SMS/Twilio, WhatsApp, Instagram/Twitter/TikTok routes resolve and verify like Chatwoot where supported. | Done | | Q7 | Burn down enterprise gaps in this order: SLA, assignment policy and capacity, CSAT, automation/macros, Audit, CustomRole, InboxLimit, Captain/Copilot. | Phase 4 and Phase 5 | Each feature passes route, persistence, auth, side-effect, response, and test checks. | Todo | | Q8 | Add frontend smoke harness using the reused Chatwoot frontend once core API flows boot end-to-end. | Phase 7 | Login, inbox list, conversation list/detail, message send, contact view, and widget init run without frontend adapters. | Todo | @@ -337,7 +337,7 @@ These milestones are the tracking spine for the remaining Chatwoot frontend reus | Milestone | Scope | Exit gate | Status | | --- | --- | --- | --- | | M0 | Test, route, and documentation baseline. | Clean worktree, `go test ./...`, route dump/parity artifacts current. | Done | -| M1 | Meilisearch-first search foundation. | Config, engine, indexing hooks, reindex command, mocked tests, and live-shape review tracked. | Doing | +| M1 | Meilisearch-first search foundation. | Config, engine, indexing hooks, reindex command, mocked tests, and live-shape review tracked. | Review | | M2 | Route parity expansion for frontend-critical routes. | Tracked route set has zero missing routes and every new route group has router boot coverage. | Doing | | M3 | Serializer parity for frontend API families. | Fixture tests cover auth/profile, accounts/users, inboxes, conversations/messages, contacts/companies, notifications, reports, widget/public, and search. | Doing | | M4 | Core handler placeholder burn-down. | Account/contact/conversation/message/inbox/webhook handlers are repository-backed and no frontend-critical route returns placeholder JSON. | Doing | @@ -356,7 +356,7 @@ Work proceeds top-down unless a failing test or frontend blocker forces a narrow | B3 | Conversation/message serializer and behavior fixtures. | Chatwoot conversation/message controllers, entities, jobs. | Fixture tests for list/show/create/update/private notes/attachments/status/assignment. | Review | | B4 | Contact/company behavior fixtures. | Chatwoot contact/company controllers, merge/import/export/notes/labels. | Fixture tests for CRUD/search/merge/relation/import-export shells and frontend CRM API smoke. | Review | | B5 | Inbox/channel behavior fixtures. | Chatwoot inbox/channel controllers and channel models. | Fixture tests for inbox CRUD, settings, business hours, members, avatar, channel config. | Done | -| B6 | Meilisearch live-shape review. | Chatwoot frontend search usage and search controllers. | Meilisearch-backed response fixtures plus optional live integration gate. | Doing | +| B6 | Meilisearch live-shape review. | Chatwoot frontend search usage and search controllers. | Meilisearch-backed response fixtures plus optional live integration gate. | Review | | B7 | SLA and assignment capacity. | Chatwoot enterprise SLA and assignment policy behavior. | Unit/integration tests for SLA state, breach, assignment capacity, policy selection. | Todo | | B8 | CSAT account-side completion. | Chatwoot CSAT survey responses, reports, downloads, listeners. | Metrics/list/review/download/send idempotency tests. | Todo | | B9 | Automation/macros durable side effects. | Chatwoot automation/macro services and jobs. | Action execution, logs, webhook/email transcript retry tests. | Todo | @@ -443,11 +443,11 @@ Tracking table: | P1.1 | Add `SearchConfig` with engine, host, API key, index prefix, env binding, and defaults. | `internal/config/config.go`, `configs/config.yaml`, config tests | `.hermes/plans/2025-05-24-global-search-meilisearch.md` | Done | | P1.2 | Add stable engine contract for search, indexing, deletes, batch indexing, and close. | `internal/search/engine.go` | Chatwoot global/entity search behavior | Done | | P1.3 | Implement Meilisearch engine wrapper, index naming, bootstrap, sortable/filterable/searchable settings. | `internal/search/engine_meili.go` | Chatwoot search models/services | Review | -| P1.4 | Keep existing DB search as explicit dev fallback only. Production config must prefer Meilisearch. | `internal/search/engine_db.go`, `internal/search/search_service.go` | User decision on Meilisearch | Done | +| P1.4 | Keep existing DB search as explicit dev fallback only. Production config must prefer Meilisearch. | `internal/search/engine_db.go`, `internal/search/search_service.go`, `internal/config/validator.go`, `cmd/reindex_search` | User decision on Meilisearch | Done | | P1.5 | Define documents and serializers for conversations, messages, contacts, companies, articles, and help-center content. | `internal/search/engine.go` | `reference/chatwoot` models/serializers | Review | | P1.6 | Wire create/update/delete hooks from entity services into async or synchronous indexing boundary. | `internal/service/*`, `internal/search/search_service.go`, `internal/app/bootstrap.go` | Chatwoot callbacks/jobs | Done | | P1.7 | Add batch reindex command and account/entity filters. | `cmd/reindex_search` | Chatwoot reindex/search tasks | Done | -| P1.8 | Add mocked engine tests and service integration tests without requiring live Meilisearch. | `internal/search/engine_test.go`, `internal/config/config_test.go` | Existing test style | Done | +| P1.8 | Add mocked engine tests and service integration tests without requiring live Meilisearch. | `internal/search/engine_test.go`, `internal/search/engine_meili_live_test.go`, `internal/config/config_test.go` | Existing test style | Done | | P1.9 | Document Meilisearch env vars and local startup flow. | this doc, ops docs if needed | Hermes plan | Done | Meilisearch local flow: @@ -876,3 +876,4 @@ Verification milestone gates: - 2026-06-05: B5.5c AgentCapacityPolicy/InboxCapacityLimit checkpoint committed as `b197e54 feat(capacity): align chatwoot inbox capacity limits`; policy CRUD now returns raw Chatwoot serializers, nested policy users and inbox limits are implemented, `account_users.agent_capacity_policy_id` is persisted, duplicate/wrong-account/non-negative limit checks are covered, route dump is `TOTAL: 829`, tracked route parity is `267 exact, 7 parameter-compatible, 0 missing`, focused service/handler/router tests passed, sandboxed full tests failed only on socket restrictions, escalated full `go test ./...` passed, and `git diff --check` passed. - 2026-06-05: B5.5 final channel route-response cleanup committed as `f04a03b feat(channels): align channel route inbox payloads`; Email, Twilio SMS, and LINE dedicated channel routes now return raw Chatwoot-compatible inbox payloads for create/get/update, `{ payload: [...] }` for list, and empty `200 OK` for delete. `InboxService.BindChannel` keeps dedicated channel IDs/configs reflected in inbox serialization. Focused Email/Twilio/LINE channel tests, service inbox tests, handler package tests, full `go test ./...`, and `git diff --check` passed. Route dump unchanged at `TOTAL: 829`; next active slice is B6 Meilisearch live-shape review. - 2026-06-05: B6 search payload checkpoint committed as `a16c23c feat(search): align chatwoot search payloads`; global/entity search endpoints now return Chatwoot `payload` envelopes, default to 15 results, accept `since`/`until`/`from` frontend params, serialize model and Meilisearch-hit data into frontend-ready conversation/contact/message/article shapes, and filter message sender IDs through both Meilisearch and DB fallback. Focused handler/search/repository tests, handler/search/repository package tests, escalated full `go test ./...`, and `git diff --check` passed. Route dump unchanged at `TOTAL: 829`; B6.4 live Meilisearch gate and B6.5 fallback hardening remain active. +- 2026-06-05: B6 live-gate/fallback checkpoint committed as `f08c743 test(search): add meilisearch live gate`; added an env-gated live Meilisearch test for isolated bootstrap/index/search of contacts and messages, kept it skipped unless `GOCHAT_LIVE_MEILI_HOST` is set, rejected `search.engine=db` in release validation, and made `cmd/reindex_search` fail fast if configured with DB fallback. Focused config/reindex/search tests, full `go test ./...`, and `git diff --check` passed. B6 moves to Review; next implementation slice is B7 SLA/assignment capacity unless the optional live gate is explicitly run first.