gochat/docs/CHATWOOT_PARITY_DEVELOPMENT_PLAN.md

# Chatwoot Parity Development Plan

Updated: 2026-06-04

## Goal

Build GoChat as a Go backend that can directly reuse the frontend from `reference/chatwoot`. The backend API, data contracts, side effects, permissions, and runtime behavior must match the local `reference/chatwoot` repository first. Existing docs are secondary when they conflict with the reference implementation.

## Confirmed Decisions

- Frontend: reuse Chatwoot frontend directly. Backend compatibility is mandatory.
- Baseline: `reference/chatwoot` is the source of truth for routes, controllers, models, serializers, jobs, and service behavior.
- Immediate order: keep `go test ./...` green first, then deepen Chatwoot behavior parity.
- Search: final implementation must use Meilisearch. DB/LIKE search is not acceptable as the final engine.
- Enterprise scope: exclude SSO/SAML/LDAP/OIDC. Include the remaining paid features already present in planning and code: SLA, Audit, CustomRole, AgentCapacity, Captain/Copilot, CSAT, InboxLimit, automation, macros, assignment policies, and related limits/workflows.

## Current Baseline

- `go test ./...` passes.
- Route dump succeeds with `TOTAL: 704`.
- Handler test stability fixes are committed into the baseline before feature parity work continues.
- `.codegraph/` is generated indexing output and is not part of tracked product code.

## Phase 0: Test And Route Baseline

Status: done.

Checklist:

- [x] Fix `internal/handler/api/v1` route param mismatches around `:account_id`, `:id`, and resource IDs.
- [x] Fix malformed test routes that conflict with Gin wildcard rules.
- [x] Add defensive handling for zero-value services in handler edge tests.
- [x] Align handler error responses where tests encode the expected Chatwoot-compatible status category.
- [x] Keep platform routes bootable and dumpable.
- [x] Verify `go test ./...`.
- [x] Verify `go run ./cmd/dump_routes`.

Verification commands:

```bash
env GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go test ./...
env GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go run ./cmd/dump_routes
```

## Phase 1: Meilisearch Search Engine

Status: next.

Source material:

- `.hermes/plans/2025-05-24-global-search-meilisearch.md`
- `reference/chatwoot` search controllers, models, and indexing behavior
- Current local packages under `internal/search`, `internal/handler/api/v1/search_handler.go`, and entity repositories

Checklist:

- [ ] Define a stable `SearchEngine` interface for Meilisearch-backed search and indexing.
- [ ] Add search config for engine, host, API key, and index prefix.
- [ ] Implement Meilisearch client wrapper with index bootstrapping and settings.
- [ ] Define per-entity documents for conversations, messages, contacts, companies, articles, and help-center content as required by Chatwoot frontend behavior.
- [ ] Wire create/update/delete hooks from services into indexing.
- [ ] Add batch reindex command for existing data.
- [ ] Keep DB search only as explicit development fallback, not as final production mode.
- [ ] Add tests with a mocked search engine and integration hooks that can run without a live Meilisearch instance.
- [ ] Document required Meilisearch environment variables and local startup flow.

Acceptance:

- Global search endpoint and entity search endpoints return Chatwoot-compatible payloads.
- Search results are account-scoped.
- Search indexing survives entity updates and deletes.
- `go test ./...` stays green.

Tracking table:

| ID | Task | Target files | Reference source | Status |
| --- | --- | --- | --- | --- |
| P1.1 | Add `SearchConfig` with engine, host, API key, index prefix, env binding, and defaults. | `internal/config/config.go`, `configs/config.yaml`, config tests | `.hermes/plans/2025-05-24-global-search-meilisearch.md` | Done |
| P1.2 | Add stable engine contract for search, indexing, deletes, batch indexing, and close. | `internal/search/engine.go` | Chatwoot global/entity search behavior | Done |
| P1.3 | Implement Meilisearch engine wrapper, index naming, bootstrap, sortable/filterable/searchable settings. | `internal/search/engine_meili.go` | Chatwoot search models/services | Review |
| P1.4 | Keep existing DB search as explicit dev fallback only. Production config must prefer Meilisearch. | `internal/search/engine_db.go`, `internal/search/search_service.go` | User decision on Meilisearch | Done |
| P1.5 | Define documents and serializers for conversations, messages, contacts, companies, articles, and help-center content. | `internal/search/engine.go` | `reference/chatwoot` models/serializers | Review |
| P1.6 | Wire create/update/delete hooks from entity services into async or synchronous indexing boundary. | `internal/service/*`, `internal/search/search_service.go`, `internal/app/bootstrap.go` | Chatwoot callbacks/jobs | Done |
| P1.7 | Add batch reindex command and account/entity filters. | `cmd/reindex_search` | Chatwoot reindex/search tasks | Done |
| P1.8 | Add mocked engine tests and service integration tests without requiring live Meilisearch. | `internal/search/engine_test.go`, `internal/config/config_test.go` | Existing test style | Done |
| P1.9 | Document Meilisearch env vars and local startup flow. | this doc, ops docs if needed | Hermes plan | Done |

Meilisearch local flow:

```bash
docker run --rm -p 7700:7700 -e MEILI_MASTER_KEY=gochat_dev getmeili/meilisearch:latest
GOCHAT_SEARCH_ENGINE=meilisearch GOCHAT_SEARCH_HOST=http://localhost:7700 GOCHAT_SEARCH_API_KEY=gochat_dev go run ./cmd/reindex_search -types all
```

Search environment variables:

| Variable | Default | Notes |
| --- | --- | --- |
| `GOCHAT_SEARCH_ENGINE` | `meilisearch` | Use `db` only for explicit local fallback. |
| `GOCHAT_SEARCH_HOST` | `http://localhost:7700` | Meilisearch endpoint. |
| `GOCHAT_SEARCH_API_KEY` | empty | Set to Meilisearch master/search key when enabled. |
| `GOCHAT_SEARCH_INDEX_PREFIX` | `gochat_` | Prefixes indexes such as `gochat_conversations`. |
| `GOCHAT_SEARCH_TIMEOUT_SECONDS` | `5` | HTTP timeout for search/index requests. |

## Phase 2: Route And Controller Parity Audit

Status: planned.

Checklist:

- [ ] Dump Chatwoot routes from `reference/chatwoot`.
- [ ] Dump GoChat routes with `cmd/dump_routes`.
- [ ] Build a tracked route parity table covering method, path, controller, auth scope, request params, and response serializer.
- [ ] Prioritize frontend-critical routes used by the Chatwoot web app.
- [ ] Patch route names and wildcard params where Gin constraints require different internal names, while preserving external URLs.
- [ ] Add regression tests for route groups that previously conflicted.

Acceptance:

- Route gap report is generated and checked in.
- All currently implemented routes boot without panic.
- Missing frontend-critical routes have tickets or implementation tasks.

Tracking table:

| ID | Task | Artifact | Status |
| --- | --- | --- | --- |
| P2.1 | Generate Chatwoot route dump from `reference/chatwoot`. | `docs/parity/chatwoot_routes.*` | Todo |
| P2.2 | Generate GoChat route dump with `cmd/dump_routes`. | `docs/parity/gochat_routes.*` | Todo |
| P2.3 | Produce route parity table: method, path, controller/handler, auth, request params, serializer, status. | `docs/parity/route_parity.md` | Todo |
| P2.4 | Mark frontend-critical gaps from Chatwoot web app route usage. | `docs/parity/route_parity.md` | Todo |
| P2.5 | Convert existing placeholders/stubs into tracked feature tasks instead of hidden debt. | this doc and gap report | Todo |
| P2.6 | Add route boot regression tests for Gin wildcard/param conflicts. | router and handler tests | Todo |

## Phase 3: Data And Serializer Parity

Status: planned.

Checklist:

- [ ] Compare key Chatwoot serializers/entities with Go response payloads.
- [ ] Align account, user, inbox, conversation, message, contact, company, team, label, canned response, campaign, help center, and notification payload shapes.
- [ ] Verify timestamps, IDs, enum strings, nested objects, pagination metadata, and error envelopes.
- [ ] Add fixture-driven tests for payload compatibility.

Acceptance:

- Chatwoot frontend can consume the payloads without adapter code.
- Serializer deviations are documented only where GoChat intentionally differs.

Frontend-critical API groups to audit first:

| ID | Area | Scope | Status |
| --- | --- | --- | --- |
| P3.1 | Auth/session/profile | Login, logout, current user, profile, availability, notification settings. | Todo |
| P3.2 | Accounts/users/teams | Account settings, users, agents, teams, invitations, roles, permissions. | Todo |
| P3.3 | Inboxes/channels | Inbox CRUD, assignable agents, avatars, channel config, business hours, widget config. | Todo |
| P3.4 | Conversations/messages | List filters, status changes, assignment, labels, private notes, attachments, drafts, typing/read events. | Todo |
| P3.5 | Contacts/companies | CRUD, merge, labels, notes, custom attributes, import/export, conversations relation. | Todo |
| P3.6 | Labels/custom attributes/custom filters | Create/update/list behavior and exact response shapes. | Todo |
| P3.7 | Notifications/reports/help center/campaigns | Frontend-visible payloads and pagination/error envelopes. | Todo |

Serializer comparison rules:

- Compare against `reference/chatwoot` serializers/entities before changing Go responses.
- Prefer fixture-driven tests for exact JSON shape, enum strings, pagination metadata, and error envelopes.
- Preserve Chatwoot field names even if Go internal naming differs.

## Phase 4: Enterprise Feature Completion

Status: planned.

Excluded:

- SSO
- SAML
- LDAP
- OIDC

Included checklist:

- [ ] SLA policies and SLA event tracking.
- [ ] Audit logs and admin-readable audit endpoints.
- [ ] Custom roles and permission checks.
- [ ] Agent capacity and assignment limits.
- [ ] Assignment policies and auto-assignment compatibility.
- [ ] Captain/Copilot assistant, custom tools, scenarios, documents, responses, and inbox bindings.
- [ ] CSAT survey response flow, metrics, filters, and review notes.
- [ ] Inbox limits and account/inbox usage enforcement.
- [ ] Automation rules, macros, execution logs, and action side effects.

Acceptance:

- Each included feature has routes, persistence, authorization, tests, and Chatwoot-compatible response behavior.
- Unsupported SSO family features are explicitly disabled or omitted without breaking frontend navigation for enabled features.

Enterprise tracking table:

| ID | Feature | Existing Go surface | Required next work | Status |
| --- | --- | --- | --- | --- |
| P4.1 | SLA policies/events | `internal/model/sla_policy.go`, `internal/model/sla_event.go`, `internal/service/sla_policy_service.go`, `internal/service/sla_event_service.go`, `internal/handler/api/v1/sla_policy_handler.go` | Compare with Chatwoot SLA behavior; wire lifecycle events and breach tracking; add frontend payload tests. | Todo |
| P4.2 | Audit logs | `internal/model/audit.go`, `internal/service/audit_service.go`, `internal/repository/audit_repo.go`, `internal/handler/api/v1/audit_handler.go` | Ensure every mutating enterprise/core action emits audit events and filters match Chatwoot. | Todo |
| P4.3 | Custom roles/permissions | `internal/model/custom_role.go`, `internal/service/custom_role_service.go`, `internal/middleware/role_check.go`, `internal/handler/api/v1/custom_role_handler.go` | Align permission keys, inherited roles, authorization failures, and admin UX payloads. | Todo |
| P4.4 | Agent capacity | `internal/model/agent_capacity_policy.go`, `internal/service/agent_capacity_policy_service.go`, `internal/handler/api/v1/agent_capacity_handler.go`, `internal/autoassignment/*` | Enforce capacity in assignment path and auto-assignment; add limit tests. | Todo |
| P4.5 | Inbox limits | `internal/model/inbox_limit.go`, `internal/service/inbox_limit_service.go`, `internal/repository/inbox_limit_repo.go`, `internal/handler/api/v1/inbox_limit_handler.go` | Enforce account/inbox usage limits and frontend-compatible responses. | Todo |
| P4.6 | Captain/Copilot | `internal/model/captain_models.go`, `internal/model/copilot_models.go`, `internal/service/captain_*`, `internal/service/copilot_*`, `internal/handler/api/v1/captain_*`, `internal/handler/api/v1/copilot_handler.go` | Complete assistant, tools, scenarios, documents, responses, inbox bindings, suggestions, and streaming compatibility. | Todo |
| P4.7 | CSAT | `internal/csat/*`, `internal/automation/csat_survey_*`, `internal/handler/api/v1/csat_*`, `internal/service/csat_metrics_service.go` | Wire resolve-triggered survey send, public update flow, metrics, review notes, filters. | Todo |
| P4.8 | Automation and macros | `internal/automation/*`, `internal/handler/api/v1/automation_rule_handler.go`, `internal/handler/api/v1/macro_handler.go` | Finish action side effects, execution logs, webhook/email transcript delivery, and rule trigger coverage. | Todo |
| P4.9 | Assignment policies | `internal/autoassignment/*`, `internal/automation/agent_bot_rule_listener.go` | Match Chatwoot assignment policy behavior and availability/capacity rules. | Todo |

Excluded tracking table:

| Feature | Decision | Required handling |
| --- | --- | --- |
| SSO | Excluded | Do not prioritize implementation. Existing surfaces should not block core frontend use. |
| SAML | Excluded | Do not expand beyond already present code unless needed to disable safely. |
| LDAP | Excluded | Omit from parity scope. |
| OIDC | Excluded | Omit from parity scope. |

## Phase 5: Background Jobs And Integrations

Status: planned.

Known hotspots:

- `internal/worker/worker.go` is still mostly placeholder.
- `internal/automation/action_service.go` has pending webhook/email transcript work.
- `internal/automation/csat_survey_listener.go` has pending CSAT enable/send behavior.
- `internal/auth/webhook_registry.go` has pending signature verification for Facebook/WhatsApp.
- `internal/service/analytics_service.go` has placeholder analytics paths.

Checklist:

- [ ] Map Chatwoot jobs/listeners to Go worker responsibilities.
- [ ] Implement durable job dispatch for automation, CSAT, notifications, webhooks, and search indexing.
- [ ] Add retry and failure logging for external calls.
- [ ] Add tests for job enqueueing and idempotency.

Acceptance:

- User-visible side effects do not depend on synchronous handler-only execution.
- Failed background work is observable and retryable.

Tracking table:

| ID | Task | Current hotspot | Status |
| --- | --- | --- | --- |
| P5.1 | Choose and wire durable job runner compatible with current Go stack. | `internal/worker/worker.go`, `internal/channel/dispatcher.go` | Todo |
| P5.2 | Move search indexing into retryable jobs where Chatwoot uses callbacks/jobs. | search services and worker | Todo |
| P5.3 | Implement automation webhook delivery with retry, timeout, and logs. | `internal/automation/action_service.go` | Todo |
| P5.4 | Implement email transcript delivery once email infrastructure is available. | `internal/automation/action_service.go` | Todo |
| P5.5 | Complete CSAT survey send listener and idempotency. | `internal/automation/csat_survey_listener.go`, `internal/csat/listener.go` | Todo |
| P5.6 | Complete webhook signature verification for Facebook/WhatsApp/Twilio/provider paths. | `internal/auth/webhook_registry.go`, channel providers | Todo |
| P5.7 | Replace placeholder analytics with real report builders/queries. | `internal/service/analytics_service.go`, reporting services | Todo |

## Phase 6: Core Product Placeholder Burn-down

Status: planned.

Purpose: route parity is not enough; existing handlers that return placeholder JSON must be converted into real Chatwoot-compatible behavior before the frontend can be reused directly.

Tracking table:

| ID | Area | Known references | Required next work | Status |
| --- | --- | --- | --- | --- |
| P6.1 | Account APIs | `docs/ROUTE_GAP_ANALYSIS.md`, account handlers | Replace placeholder responses with repository-backed behavior and serializer tests. | Todo |
| P6.2 | Contact APIs | `docs/ROUTE_GAP_ANALYSIS.md`, contact handlers/services | Implement show/create/update/list/search relations, labels, notes, merge. | Todo |
| P6.3 | Conversation APIs | `docs/ROUTE_GAP_ANALYSIS.md`, conversation handlers/services | Implement frontend-critical filters, assignment, status, snooze, merge, bulk actions. | Todo |
| P6.4 | Message APIs | `docs/ROUTE_GAP_ANALYSIS.md`, message handlers/services | Implement create/list/delete, private notes, attachments, source attribution, events. | Todo |
| P6.5 | Inbox APIs | `docs/ROUTE_GAP_ANALYSIS.md`, inbox handlers/services | Implement CRUD, assignable agents, avatar, campaigns, channel settings, reset secret. | Todo |
| P6.6 | Widget APIs | `docs/ROUTE_GAP_ANALYSIS.md`, widget/channel provider code | Implement init, contact, conversations, messages, cable token with Chatwoot widget contracts. | Todo |
| P6.7 | Webhook ingress | `internal/router/router.go`, `internal/handler/webhook/*`, channel providers | Replace generic placeholder with provider-specific verified ingestion and dispatch. | Todo |

## Phase 7: Verification Harness

Status: planned.

Checklist:

- [ ] Add repeatable command to compare Chatwoot and GoChat route dumps.
- [ ] Add fixture-based JSON parity tests for major frontend endpoints.
- [ ] Add Meilisearch mock tests and optional integration test mode gated by env vars.
- [ ] Add frontend smoke path using reused Chatwoot frontend once backend boot flow is ready.
- [ ] Keep a generated gap report in `docs/parity/` with date, reference commit, Go commit, and pass/fail summary.

Required verification before each feature commit:

```bash
env GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go test ./...
env GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go run ./cmd/dump_routes
```

## Task Status Legend

- `Todo`: not started or only skeleton exists.
- `Doing`: implementation is underway in the current working tree.
- `Review`: code is implemented and needs parity/test verification.
- `Done`: committed, tested, and linked to the matching `reference/chatwoot` behavior.
- `Blocked`: needs a decision, missing dependency, or reference behavior cannot yet be reproduced.

## Ongoing Tracking Rules

- Every parity task should name the matching file or behavior in `reference/chatwoot`.
- Every route change should keep `cmd/dump_routes` passing.
- Every phase must keep `go test ./...` green before moving on.
- Test-only route fixes should be limited to malformed tests. Production route changes must preserve external Chatwoot-compatible URLs.
- New search work must target Meilisearch first.
- Update this document in the same commit as each completed parity slice: change task status, add verification command output summary, and link the touched `reference/chatwoot` behavior.
- Do not count a feature as done because a model/handler exists; it is done only when route behavior, persistence, authorization, side effects, response shape, and tests are covered.
- Keep `.codegraph/` and other generated local analysis artifacts out of product commits unless explicitly requested.

## Progress Log

- 2026-06-04: Baseline stabilized and committed as `42cdab8 chore: stabilize chatwoot parity baseline`; `go test ./...` passed and route dump reported `TOTAL: 704`.
- 2026-06-04: Phase 1 search foundation added: Meilisearch config/env defaults, `SearchEngine` contract, Meilisearch HTTP wrapper with bootstrap/settings, DB fallback adapter, document builders, reindex command, and no-live-Meilisearch tests. Verified `go test ./...` in unsandboxed mode because miniredis/httptest need local sockets; route dump still reports `TOTAL: 704`.
- 2026-06-04: Phase 1 indexing hooks wired for conversations, messages, contacts, companies, and articles. Create/update/delete paths now call the service-layer `SearchIndexer` boundary, bootstrap injects the Meilisearch-backed search service, and unit tests cover each entity hook path.