diff --git a/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md b/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md index ce325f5f..b0c2e78e 100644 --- a/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md +++ b/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md @@ -16,9 +16,9 @@ Build GoChat as a Go backend that can directly reuse the frontend from `referenc ## Current Baseline -- Latest implementation checkpoint: `f04a03b feat(channels): align channel route inbox payloads`. -- Latest documentation checkpoint before this update: `10c7df0 docs: record capacity parity checkpoint`. -- Worktree status at this planning checkpoint: clean after the matching implementation checkpoint; next active slice is B6 Meilisearch live-shape review. +- Latest implementation checkpoint: `a16c23c feat(search): align chatwoot search payloads`. +- Latest documentation checkpoint before this update: `178bc36 docs: record channel route cleanup checkpoint`. +- Worktree status at this planning checkpoint: clean after the matching implementation checkpoint; next active slice is B6 live Meilisearch gate and DB-fallback hardening. - `go test ./...` passes. - Route dump succeeds with `TOTAL: 829` after adding Chatwoot-compatible nested AgentCapacityPolicy users and inbox-limit routes. - Route parity artifacts now exist under `docs/parity/` and are generated by `cmd/route_parity`. @@ -32,7 +32,7 @@ Build GoChat as a Go backend that can directly reuse the frontend from `referenc | Phase | Name | Status | Blocking gaps | | --- | --- | --- | --- | | Phase 0 | Test and route baseline | Done | none | -| Phase 1 | Meilisearch search engine | Doing | B6 live-shape review is the next active slice; document/payload parity and optional live integration still need verification | +| Phase 1 | Meilisearch search engine | Doing | B6 payload parity has a mocked Meilisearch checkpoint; live integration and DB-fallback hardening still need verification | | Phase 2 | Route and controller parity audit | Doing | Ruby/Bundler unavailable, so Chatwoot route extraction currently uses static `routes.rb` fallback | | Phase 3 | Data and serializer parity | Doing | JSON fixture coverage is partial and still endpoint-family based | | Phase 4 | Enterprise feature completion | Doing | excluded SSO family must stay out of scope; all other enterprise features remain included | @@ -93,22 +93,22 @@ This ledger records the committed parity checkpoints that future slices should b | `8fd40ec docs: record channel config checkpoint` | Recorded the B5.5 channel config checkpoint and kept the route count at `TOTAL: 821` with tracked route parity still at 262 exact and 0 missing. | Documentation-only checkpoint after the B5.5 channel config implementation. | Next active slice is B5.5c AgentCapacityPolicy/InboxCapacityLimit route, serializer, and service parity. | | `b197e54 feat(capacity): align chatwoot inbox capacity limits` | Completed B5.5c AgentCapacityPolicy/InboxCapacityLimit API/data parity: policy CRUD now returns raw Chatwoot payloads with Unix timestamps, `assigned_agent_count`, and `inbox_capacity_limits`; `assignment_logic` is optional/defaulted; nested policy users and inbox limits are registered; `InboxCapacityLimit` validates account scope, duplicate inbox assignment, and non-negative limits; account users can be assigned/unassigned to capacity policies. | `go test ./internal/service -run AgentCapacity -count=1`; `go test ./internal/handler/api/v1 -run AgentCapacity -count=1`; `go test ./internal/router -count=1`; `go test ./internal/handler/api/v1 -count=1`; sandboxed `go test ./...` failed on local socket restrictions; escalated `go test ./...` passed; route dump regenerated with `TOTAL: 829`; route parity is `267 exact, 7 parameter-compatible, 0 missing`; `git diff --check`. | Continue final B5.5 review for remaining channel-specific route response cleanup, then move to B6 Meilisearch live-shape review. | | `f04a03b feat(channels): align channel route inbox payloads` | Completed the final B5.5 channel route-response cleanup: Email, Twilio SMS, and LINE dedicated channel create/get/update/list routes now return Chatwoot frontend-compatible raw inbox payloads or `{ payload: [...] }` lists instead of `{ channel }`, `{ channels }`, `{ channel, inbox }`, or success-message envelopes. Delete routes now return empty `200 OK`, and channel creation binds the dedicated channel ID/config back onto the inbox before serialization. | `go test ./internal/handler/api/v1 -run 'Test(Email\|Twilio\|LINE)Channel' -count=1`; `go test ./internal/service -run Inbox -count=1`; `go test ./internal/handler/api/v1 -count=1`; `go test ./...`; `git diff --check`. No route changes; route dump remains `TOTAL: 829`. | B5 inbox/channel API parity moves to Done for current frontend-critical scope; next active slice is B6 Meilisearch live-shape review. | +| `a16c23c feat(search): align chatwoot search payloads` | Advanced B6 search live-shape parity: global and entity search endpoints now return Chatwoot frontend `payload` envelopes, search filters accept Chatwoot `since`, `until`, and `from=contact:id/agent:id` params, message sender IDs are indexed/filterable in Meilisearch and DB fallback, SearchController default page size is aligned to 15, and Meilisearch hit maps are serialized into frontend message/contact/conversation/article shapes. | `go test ./internal/handler/api/v1 -run SearchHandler -count=1`; `go test ./internal/search -run 'SearchFilter\|Meili\|Engine' -count=1`; `go test ./internal/repository -run Search -count=1`; `go test ./internal/handler/api/v1 -count=1`; `go test ./internal/search -count=1`; `go test ./internal/repository -count=1`; sandboxed `go test ./...` failed on local socket restrictions; escalated `go test ./...` passed; `git diff --check`. No route changes; route dump remains `TOTAL: 829`. | B6.1 and B6.3 are Done; B6.2 is in Review with mocked Meilisearch coverage. Continue B6.4 live integration gate and B6.5 DB-fallback hardening before moving B6 to Review/Done. | ## Next Slice Contract -Completed implementation slice: B5.1-B5.5 now cover inbox serializer shape, Chatwoot frontend create/update binding, working-hours persistence, out-of-office behavior, inbox member assignment payload/mutation semantics, channel-specific config depth, AgentCapacityPolicy/InboxCapacityLimit API data contracts, and dedicated Email/Twilio/LINE channel route response shapes. B4 remains in review after contact/company CRUD payloads, relations, labels, merge, import/export, notes, nested conversations, Meilisearch CRM search, and API smoke coverage landed. +Completed implementation slice: B5.1-B5.5 now cover inbox serializer shape, Chatwoot frontend create/update binding, working-hours persistence, out-of-office behavior, inbox member assignment payload/mutation semantics, channel-specific config depth, AgentCapacityPolicy/InboxCapacityLimit API data contracts, and dedicated Email/Twilio/LINE channel route response shapes. B6 now has its first search payload checkpoint in `a16c23c`: Chatwoot search response envelopes, frontend query params, Meilisearch sender filters, and mocked hit serialization are covered. B3 and B4 remain in review for deeper side effects and browser validation. -Next implementation slice: B6 Meilisearch live-shape review. Keep B3 and B4 in review for delivery/storage side-effect parity and real browser frontend validation. +Next implementation slice: finish B6.4 live Meilisearch gate and B6.5 DB-fallback hardening, then move to B7 SLA/assignment capacity unless a reused-frontend search smoke exposes a blocker. | Step | Required result | Reference source | Verification | | --- | --- | --- | --- | -| N1 | Align inbox list/show/create/update/delete/avatar/reset-secret serializer shapes. | `InboxesController`, `api/v1/models/_inbox.json.jbuilder`, dashboard inbox store. | Done by `0e83e8d`; list returns `{ payload: [...] }`, item responses are raw inbox payloads, channel slugs serialize as `Channel::*`, and route dump is `TOTAL: 821`. | -| N2 | Accept Chatwoot frontend inbox create/update request bodies, including multipart FormData and `channel[...]` nested params. | `dashboard/store/modules/inboxes.js#buildInboxData`, `InboxesController#permitted_params`. | Done by `ee93546`; handler tests cover website multipart create, API JSON create, Telegram no-name create, and settings update request bodies. | -| N3 | Persist inbox working hours and out-of-office behavior with Chatwoot defaults. | `OutOfOffisable`, `Inbox#OFFISABLE_ATTRS`, settings pages. | Done by `945782e`; update tests prove weekly schedule persistence and service tests prove out-of-office behavior. | -| N4 | Align inbox member payloads and membership mutation semantics. | `InboxMembersController`, `Inbox#add_members/remove_members`, dashboard `inboxMembers.js`. | Done by `82167f5`; account-scoped and nested member routes return `{ payload: [agent] }`, create is add-only/idempotent, update is diff-based, and delete returns empty `200 OK`. | -| N5 | Land AgentCapacityPolicy/InboxCapacityLimit Chatwoot API parity. | `reference/chatwoot/config/routes.rb:123-127`, enterprise agent capacity policy and inbox limit controllers/views/models. | Done by `b197e54`; policy CRUD returns raw Chatwoot payloads, nested inbox-limit and user routes are registered, duplicate/negative/wrong-account limits and `assigned_agent_count` are covered, route dump is `TOTAL: 829`, and tracked route parity has 0 missing routes. | -| N6 | Align dedicated Email, Twilio SMS, and LINE channel response envelopes for reused frontend compatibility. | `reference/chatwoot/app/controllers/api/v1/accounts/channels/twilio_channels_controller.rb`, dashboard channel clients and generic inbox serializer. | Done by `f04a03b`; create/get/update return raw inbox payloads, list returns `{ payload: [...] }`, delete returns empty `200 OK`, and focused channel handler tests cover all three dedicated handlers. | -| N7 | Update this tracker after every implementation checkpoint. | This document. | `git diff --check`; `go test ./...` for Go changes. | +| N1 | Keep B6 search payload checkpoint as the current baseline. | `SearchController`, `search.js`, `conversationSearch.js`, search Jbuilder views. | Done by `a16c23c`; response envelopes, Chatwoot params, sender filters, and mocked Meilisearch hit shape are covered. | +| N2 | Add the live Meilisearch integration gate. | `cmd/reindex_search`, `internal/search/engine_meili.go`, local Meilisearch env. | Env-gated test or documented command starts from indexed data, runs global/messages/contacts searches, and records result shape. | +| N3 | Harden and document DB fallback as development-only. | Search config defaults, `NewSearchEngine`, bootstrap wiring. | Config tests keep `meilisearch` as default; docs call DB fallback non-production and explain how to fail/alert when Meilisearch is unavailable. | +| N4 | Decide B6 close status after live gate. | B6 task board and Phase 1 checklist. | Mark B6 `Review` or `Done` only after live gate and fallback hardening are recorded; otherwise keep B6 `Doing`. | +| N5 | Start B7 SLA and assignment capacity. | Chatwoot enterprise SLA policies, assignment policies, capacity enforcement. | First B7 checkpoint must include routes/data/service tests before deeper timer/report behavior. | +| N6 | Update this tracker after every implementation checkpoint. | This document. | `git diff --check`; `go test ./...` for Go changes. | Current B2 profile checkpoint: @@ -261,11 +261,29 @@ Active B6 task board: | ID | Task | Reference source | Status | Exit gate | | --- | --- | --- | --- | --- | -| B6.1 | Inventory Chatwoot search frontend request/response consumers and current Go search routes. | `reference/chatwoot/app/javascript/dashboard/api/search.js`, `conversationSearch.js`, `SearchController`, `internal/handler/api/v1/search_handler.go`. | Todo | A checked matrix lists global, contacts, conversations, messages, and articles search params and payload fields. | -| B6.2 | Compare Meilisearch document fields and filters against frontend payload needs. | `internal/search/engine.go`, `engine_meili.go`, document builders, Chatwoot search views/entities. | Todo | Mocked Meilisearch tests prove account scoping, type filters, pagination, and result data shape without DB-only assumptions. | -| B6.3 | Tighten endpoint serializers for `/search`, `/search/contacts`, `/search/conversations`, `/search/messages`, and `/search/articles`. | `reference/chatwoot/app/controllers/api/v1/accounts/search_controller.rb`, dashboard search API specs. | Todo | Handler fixture tests assert Chatwoot-compatible `payload/meta` or documented search result shapes for each endpoint. | +| B6.1 | Inventory Chatwoot search frontend request/response consumers and current Go search routes. | `reference/chatwoot/app/javascript/dashboard/api/search.js`, `conversationSearch.js`, `SearchController`, `internal/handler/api/v1/search_handler.go`. | Done | Matrix below records global, contacts, conversations, messages, and articles params/payload fields. | +| B6.2 | Compare Meilisearch document fields and filters against frontend payload needs. | `internal/search/engine.go`, `engine_meili.go`, document builders, Chatwoot search views/entities. | Review | `a16c23c` adds mocked Meilisearch sender filter and hit-shape tests; live Meilisearch gate remains B6.4. | +| B6.3 | Tighten endpoint serializers for `/search`, `/search/contacts`, `/search/conversations`, `/search/messages`, and `/search/articles`. | `reference/chatwoot/app/controllers/api/v1/accounts/search_controller.rb`, dashboard search API specs. | Done | `SearchHandler` tests assert Chatwoot `payload` envelopes and Meilisearch hit serialization. | | B6.4 | Add optional live Meilisearch integration gate. | Local Meilisearch flow in Phase 1 and `cmd/reindex_search`. | Todo | Env-gated test or documented command runs reindex plus representative searches against a live Meilisearch instance. | -| B6.5 | Document DB fallback as development-only and verify production config remains Meilisearch-first. | User decision ledger and search config. | Todo | Config tests and docs make `meilisearch` the default and warn on DB fallback. | +| B6.5 | Document DB fallback as development-only and verify production config remains Meilisearch-first. | User decision ledger and search config. | Review | Existing config defaults to `meilisearch`; next step is a focused config/doc warning around DB fallback usage. | + +B6 request and payload matrix after `a16c23c`: + +| Endpoint | Frontend params now accepted | Chatwoot payload shape now returned | Covered by | +| --- | --- | --- | --- | +| `GET /api/v1/accounts/:account_id/search` | `q`, `page`, default 15 items, existing local filters, plus `since`, `until`, `from`, `inbox_id`. | `{ payload: { conversations, contacts, messages, articles } }`. | `TestSearchHandler_GlobalSearch_Success`, `TestParseSearchFilter_ChatwootSearchParams`. | +| `GET /api/v1/accounts/:account_id/search/conversations` | `q`, `page`, status/priority/assignee/team/inbox/labels/date filters. | `{ payload: { conversations: [...] } }`. | `TestSearchHandler_SearchConversations_Success`. | +| `GET /api/v1/accounts/:account_id/search/messages` | `q`, `page`, `message_type`, `sender_type`, `from`, `sender_id`, `content_type`, `private`, `inbox_id`, `since`, `until`. | `{ payload: { messages: [...] } }` with numeric Chatwoot `message_type` for model and Meilisearch hit data. | `TestSearchHandler_SearchMessages_Success`, `TestSearchHandler_SearchMessages_MeiliHitPayloadShape`, `TestMeiliSearchEngine_SearchSendsMessageSenderIDFilter`. | +| `GET /api/v1/accounts/:account_id/search/contacts` | `q`, `page`, `contact_source`. | `{ payload: { contacts: [...] } }`. | `TestSearchHandler_SearchContacts_Success`. | +| `GET /api/v1/accounts/:account_id/search/articles` | `q`, `page`, `portal_id`, `article_status`, `locale`. | `{ payload: { articles: [...] } }`. | `TestSearchHandler_SearchArticles_Success`. | + +B6 current checkpoint: + +- Search endpoints no longer use the local `{ success, data, meta }` response envelope for frontend paths. +- `search.ParseSearchFilter` now understands Chatwoot dashboard `since`/`until` Unix seconds and `from=contact:id`/`from=agent:id` sender filters while preserving legacy `date_from`/`date_to` and `sender_id` inputs. +- Meilisearch documents now store `sender_id`, index settings mark it filterable, and DB fallback applies the same sender ID filter for local tests/dev mode. +- Search handler pagination defaults to Chatwoot's 15 items when `per_page` is not supplied. +- Remaining B6 risk: live Meilisearch reindex/search has not been run in this checkpoint, and DB fallback needs an explicit production-safety warning/test before B6 can leave Doing. Active B4 task board: @@ -857,3 +875,4 @@ Verification milestone gates: - 2026-06-05: B5.5c planning contract added for AgentCapacityPolicy/InboxCapacityLimit parity; the next slice now tracks exact Chatwoot route sources, raw policy and inbox-limit serializers, optional `assignment_logic`, nested `users` and `inbox_limits` routes, duplicate/wrong-account/non-negative validation gates, and route artifact regeneration rules before implementation starts. - 2026-06-05: B5.5c AgentCapacityPolicy/InboxCapacityLimit checkpoint committed as `b197e54 feat(capacity): align chatwoot inbox capacity limits`; policy CRUD now returns raw Chatwoot serializers, nested policy users and inbox limits are implemented, `account_users.agent_capacity_policy_id` is persisted, duplicate/wrong-account/non-negative limit checks are covered, route dump is `TOTAL: 829`, tracked route parity is `267 exact, 7 parameter-compatible, 0 missing`, focused service/handler/router tests passed, sandboxed full tests failed only on socket restrictions, escalated full `go test ./...` passed, and `git diff --check` passed. - 2026-06-05: B5.5 final channel route-response cleanup committed as `f04a03b feat(channels): align channel route inbox payloads`; Email, Twilio SMS, and LINE dedicated channel routes now return raw Chatwoot-compatible inbox payloads for create/get/update, `{ payload: [...] }` for list, and empty `200 OK` for delete. `InboxService.BindChannel` keeps dedicated channel IDs/configs reflected in inbox serialization. Focused Email/Twilio/LINE channel tests, service inbox tests, handler package tests, full `go test ./...`, and `git diff --check` passed. Route dump unchanged at `TOTAL: 829`; next active slice is B6 Meilisearch live-shape review. +- 2026-06-05: B6 search payload checkpoint committed as `a16c23c feat(search): align chatwoot search payloads`; global/entity search endpoints now return Chatwoot `payload` envelopes, default to 15 results, accept `since`/`until`/`from` frontend params, serialize model and Meilisearch-hit data into frontend-ready conversation/contact/message/article shapes, and filter message sender IDs through both Meilisearch and DB fallback. Focused handler/search/repository tests, handler/search/repository package tests, escalated full `go test ./...`, and `git diff --check` passed. Route dump unchanged at `TOTAL: 829`; B6.4 live Meilisearch gate and B6.5 fallback hardening remain active.