feat(worker): add durable background jobs
This commit is contained in:
@@ -17,9 +17,9 @@ Build GoChat as a Go backend that can directly reuse the frontend from `referenc
|
||||
## Current Baseline
|
||||
|
||||
- Plan freeze checkpoint: 2026-06-05 docs-only tracker landing after `13cb750 feat(captain): align task payload persistence`.
|
||||
- Latest implementation checkpoint: this checkpoint, prepared as `test(parity): add enterprise browser smoke coverage`.
|
||||
- Latest implementation checkpoint: this checkpoint, prepared as `feat(worker): add durable background jobs`.
|
||||
- Latest documentation checkpoint before this freeze: `3263ed9 docs: land copilot task execution plan`; this document now carries the active follow-up plan directly.
|
||||
- Worktree status at this implementation checkpoint: B11.1a aligns Captain assistant CRUD/tools/inbox bindings; B11.1b aligns Captain scenarios and custom tools; B11.1c aligns Captain documents, assistant responses, bulk actions, and custom-tool test payloads; B11.2 aligns Copilot thread/message create/list/get/delete payloads, account/user scoping, and no-LLM fallback persistence; B11.3a aligns Captain preferences show/update payloads and account-level model/feature storage; B11.3b aligns Captain playground request/response payloads, account scoping, v2 history handling, and no-LLM fallback; B11.3c adds the fakeable Captain document sync backend gate with disabled, failed, and fake-success states; B11.3d aligns Captain task request/response payloads, no-provider disabled states, follow-up context, suggestion persistence, and Copilot message tool-call key validation; B11.3e aligns Captain stream DTOs/disabled SSE fallbacks and Copilot push-event payload shapes; B12.1 adds the reusable GoChat server/seed entrypoint plus a Meilisearch-first reused Chatwoot frontend smoke harness and report; B12.2a adds API smoke assertions for auth/profile, inbox, conversation/messages, contact/company, widget config/message, and public CSAT; B12.2b adds a zero-dependency Chrome DevTools browser smoke that loads the reused Chatwoot login and dashboard entrypoints through Vite and checks browser auth/dashboard API requests; B12.3a adds enterprise API smoke assertions for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot; B12.3b adds reused-frontend enterprise browser route navigation for SLA, CSAT, automation, macros, audit logs, custom roles, capacity, Captain, and Copilot request coverage. Next active implementation slice is Phase 5 job parity.
|
||||
- Worktree status at this implementation checkpoint: B11.1a aligns Captain assistant CRUD/tools/inbox bindings; B11.1b aligns Captain scenarios and custom tools; B11.1c aligns Captain documents, assistant responses, bulk actions, and custom-tool test payloads; B11.2 aligns Copilot thread/message create/list/get/delete payloads, account/user scoping, and no-LLM fallback persistence; B11.3a aligns Captain preferences show/update payloads and account-level model/feature storage; B11.3b aligns Captain playground request/response payloads, account scoping, v2 history handling, and no-LLM fallback; B11.3c adds the fakeable Captain document sync backend gate with disabled, failed, and fake-success states; B11.3d aligns Captain task request/response payloads, no-provider disabled states, follow-up context, suggestion persistence, and Copilot message tool-call key validation; B11.3e aligns Captain stream DTOs/disabled SSE fallbacks and Copilot push-event payload shapes; B12.1 adds the reusable GoChat server/seed entrypoint plus a Meilisearch-first reused Chatwoot frontend smoke harness and report; B12.2a adds API smoke assertions for auth/profile, inbox, conversation/messages, contact/company, widget config/message, and public CSAT; B12.2b adds a zero-dependency Chrome DevTools browser smoke that loads the reused Chatwoot login and dashboard entrypoints through Vite and checks browser auth/dashboard API requests; B12.3a adds enterprise API smoke assertions for SLA reports/download, CSAT reports/download, automation/macros, audit/custom roles, capacity, Captain, and Copilot; B12.3b adds reused-frontend enterprise browser route navigation for SLA, CSAT, automation, macros, audit logs, custom roles, capacity, Captain, and Copilot request coverage; P5.1 adds the PostgreSQL-backed durable `background_jobs` model/migration plus WorkerPool enqueue, schedule, retry/backoff, dead-letter, idempotency, stale-lock recovery, and focused tests. Next active implementation slice is P5.2/P5.4 job integration.
|
||||
- `go test ./...` passes.
|
||||
- Route dump succeeds with `TOTAL: 830` after adding the Chatwoot-compatible applied-SLA index route.
|
||||
- Route parity artifacts now exist under `docs/parity/` and are generated by `cmd/route_parity`.
|
||||
@@ -78,7 +78,7 @@ Open work after the current checkpoint:
|
||||
| Phase 2 | Route and controller parity audit | Doing | Ruby/Bundler unavailable, so Chatwoot route extraction currently uses static `routes.rb` fallback |
|
||||
| Phase 3 | Data and serializer parity | Doing | JSON fixture coverage is partial and still endpoint-family based |
|
||||
| Phase 4 | Enterprise feature completion | Doing | B7, B8, B9, B10, and B11 are in Review; B12 reused frontend smoke is the next broad verification gate |
|
||||
| Phase 5 | Background jobs and integrations | Doing | durable worker choice and job parity are the next active slice after B12 harness review |
|
||||
| Phase 5 | Background jobs and integrations | Doing | P5.1 durable worker core is in Review; dispatcher/automation/search/SLA/Captain job integration remains open |
|
||||
| Phase 6 | Core placeholder burn-down | Doing | account/contact/conversation/message/inbox placeholder groups remain broad |
|
||||
| Phase 7 | Verification harness | Review | B12.1 boot/readiness, B12.2a API assertions, B12.2b browser smoke harness, B12.3a enterprise API assertions, and B12.3b enterprise browser route navigation exist; optional live Meilisearch/full-browser runs remain environment-dependent |
|
||||
|
||||
@@ -101,6 +101,7 @@ This ledger records the committed parity checkpoints that future slices should b
|
||||
|
||||
| Commit | Scope | Verification summary | Follow-up state |
|
||||
| --- | --- | --- | --- |
|
||||
| `feat(worker): add durable background jobs` | Advances P5.1 with a persistent `background_jobs` model and migration plus `internal/worker.WorkerPool` enqueue, scheduled claim, handler registry, retry/backoff, dead-letter, idempotency-key reuse, queue filtering, PostgreSQL `SKIP LOCKED` claim support, stale lock recovery, and graceful start/stop loop. The model is registered in app auto-migration and test DB defaults. | `go test ./internal/worker -count=1`; `go test ./...`; `git diff --check`; full verification recorded in the P5.1 section. | Move P5.1 to Review; continue P5.2 dispatcher async routing and P5.4/P5.5 automation/CSAT job integration on top of the durable core. |
|
||||
| `test(parity): add enterprise browser smoke coverage` | Advances B12.3b with `scripts/parity_frontend_smoke.sh --enterprise-browser-smoke`. The browser driver now supports `--enterprise`, creates route-specific reused Chatwoot dashboard shells, and asserts browser-origin requests for SLA reports, CSAT reports, automation rules, macros, audit logs, custom roles, agent capacity list/edit, Captain settings/assistants, and Copilot thread requests. The smoke seed now enables the Chatwoot enterprise feature flags required by those route guards. | `bash -n scripts/parity_frontend_smoke.sh`; `node --check scripts/parity_frontend_browser_smoke.mjs`; `scripts/parity_frontend_smoke.sh --print`; `scripts/parity_frontend_smoke.sh --check`; `go test ./cmd/gochat -count=1`; full verification recorded in the B12.3b section. Live enterprise browser smoke requires running GoChat and Chatwoot Vite. | Move B12 verification harness to Review; continue Phase 5 job parity and convert any optional live-smoke failures into named slices. |
|
||||
| `test(parity): add enterprise api smoke coverage` | Advances B12.3a with `scripts/parity_frontend_smoke.sh --enterprise-smoke`. The command composes the core API smoke with enterprise API assertions for SLA reports/metrics/download, public and account CSAT reports/metrics/download, automation-rule create/list, macro create/execute, audit-log list, custom-role list, agent-capacity list/users, Captain preferences/assistants, and Copilot thread/message payloads. | `bash -n scripts/parity_frontend_smoke.sh`; `scripts/parity_frontend_smoke.sh --print`; `scripts/parity_frontend_smoke.sh --check`; focused verification recorded in the B12.3a section. Live enterprise smoke requires running GoChat with PostgreSQL/Redis/Meilisearch. | Continue B12.3b with reused-frontend browser navigation for the same enterprise screens. |
|
||||
| `test(parity): add browser smoke harness` | Advances B12.2b with `scripts/parity_frontend_smoke.sh --browser-smoke` and `scripts/parity_frontend_browser_smoke.mjs`. The browser smoke creates temporary Vite-served HTML shells for the reused Chatwoot `v3app` login and `dashboard` entrypoints, drives login through headless Chrome via the DevTools protocol without adding Node dependencies, confirms the Chatwoot auth cookie, and checks dashboard validation/conversation requests. | `bash -n scripts/parity_frontend_smoke.sh`; `node --check scripts/parity_frontend_browser_smoke.mjs`; `scripts/parity_frontend_smoke.sh --print`; `scripts/parity_frontend_smoke.sh --check`; full verification recorded in the B12.2b section. Live browser smoke requires running GoChat and Chatwoot Vite. | Continue B12.3 with enterprise screen browser/API smoke for SLA, CSAT reports, automation/macros, audit/custom roles, capacity, Captain, and Copilot. |
|
||||
@@ -1578,7 +1579,7 @@ Reference job families to keep in scope:
|
||||
|
||||
Known hotspots:
|
||||
|
||||
- `internal/worker/worker.go` is still mostly placeholder.
|
||||
- `internal/worker/worker.go` now has the P5.1 durable core, but no feature-specific producers/handlers are wired yet.
|
||||
- `internal/automation/action_service.go` has pending webhook/email transcript work.
|
||||
- `internal/automation/csat_survey_listener.go` now handles CSAT enablement, survey rules, generic send idempotency, and message-updated response building; durable queueing and external WhatsApp/Twilio template delivery remain follow-ups.
|
||||
- `internal/auth/webhook_registry.go` has pending signature verification for Facebook/WhatsApp.
|
||||
@@ -1586,8 +1587,8 @@ Known hotspots:
|
||||
|
||||
Checklist:
|
||||
|
||||
- [ ] Add `background_jobs` persistence with job type, payload, queue, status, attempt counters, scheduled/locked timestamps, idempotency key, last error, and completion timestamps.
|
||||
- [ ] Replace `internal/worker/worker.go` placeholder with enqueue, schedule, perform, retry/backoff, dead-letter, and graceful shutdown behavior.
|
||||
- [x] Add `background_jobs` persistence with job type, payload, queue, status, attempt counters, scheduled/locked timestamps, idempotency key, last error, and completion timestamps.
|
||||
- [x] Replace `internal/worker/worker.go` placeholder with enqueue, schedule, perform, retry/backoff, dead-letter, and graceful shutdown behavior.
|
||||
- [ ] Map Chatwoot jobs/listeners to Go worker responsibilities and record out-of-scope migration-only jobs explicitly.
|
||||
- [ ] Implement durable job dispatch for automation, CSAT, notifications, webhooks, SLA processing, Captain/Copilot, contact export, macros, and search indexing.
|
||||
- [ ] Add retry and failure logging for external calls.
|
||||
@@ -1602,7 +1603,7 @@ Tracking table:
|
||||
|
||||
| ID | Task | Reference source | Current Go hotspot | Done boundary | Status |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| P5.1 | Implement durable worker core and job model. | `reference/chatwoot/app/jobs/application_job.rb`, `mutex_application_job.rb` | `internal/worker/worker.go` | Job table, repository, enqueue API, worker loop, retry/backoff, scheduled jobs, mutex/idempotency keys, dead-letter state, and restart pickup tests exist. | Next |
|
||||
| P5.1 | Implement durable worker core and job model. | `reference/chatwoot/app/jobs/application_job.rb`, `mutex_application_job.rb` | `internal/worker/worker.go`, `internal/model/background_job.go`, `migrations/000026_add_background_jobs.*.sql` | Job table, worker persistence API, enqueue API, worker loop, retry/backoff, scheduled jobs, mutex/idempotency keys, dead-letter state, and restart pickup tests exist. | Review by `feat(worker): add durable background jobs` |
|
||||
| P5.2 | Route async dispatcher events through durable jobs. | `event_dispatcher_job.rb`, Chatwoot async dispatcher listeners | `internal/dispatch/dispatcher.go`, `internal/channel/dispatcher.go` | Heavy listeners can enqueue durable jobs without changing sync listener behavior; tests cover sync vs async routing and replay. | Todo |
|
||||
| P5.3 | Move Meilisearch indexing and reindex fan-out into retryable jobs. | Meilisearch plan plus Chatwoot callbacks/jobs that index searchable records | search services, contact/company/conversation indexing hooks | Create/update/delete indexing survives handler success, retries on Meilisearch failure, and optional live Meilisearch gate remains green. | Todo |
|
||||
| P5.4 | Queue automation webhook and transcript delivery. | `webhook_job.rb`, automation action execution services | `internal/automation/action_delivery.go`, `internal/automation/action_service.go` | Existing timeout/retry fakeable delivery is invoked by durable jobs; logs preserve attempt metadata and idempotency. | Todo |
|
||||
@@ -1616,6 +1617,22 @@ Tracking table:
|
||||
| P5.12 | Queue conversation maintenance jobs. | `conversations/resolution_job.rb`, `reopen_snoozed_conversations_job.rb`, `update_message_status_job.rb`, `bulk_actions_job.rb` | conversation service/handlers | Auto-resolution, snooze reopen, status updates, and bulk actions are scheduled/retryable with idempotent tests. | Todo |
|
||||
| P5.13 | Replace placeholder analytics/report builders that need background aggregation. | reporting jobs/services and report controllers | `internal/service/analytics_service.go`, reporting services | Frontend-visible reports no longer use placeholder values; any expensive aggregation is scheduled or cached with freshness rules. | Todo |
|
||||
|
||||
P5.1 current checkpoint:
|
||||
|
||||
- Added `background_jobs` as the shared durable job table with queue, job type, JSON payload, status, priority, attempt counters, scheduled/locked timestamps, idempotency key, last error, and completion/dead-letter timestamps.
|
||||
- Replaced the `WorkerPool` stub with enqueue/schedule APIs, handler registration, `ProcessOne`, start/stop worker loops, PostgreSQL `FOR UPDATE SKIP LOCKED` claiming, retry/backoff, dead-letter state, and stale-lock recovery for worker restart pickup.
|
||||
- Kept older `NewWorkerPool()` construction as a no-op-compatible path while adding `NewWorkerPoolWithOptions(db, ...)` for durable wiring.
|
||||
- Added focused tests proving enqueue/idempotency, due job completion, retry to dead-letter, queue/schedule filtering, and stale running job requeue.
|
||||
- Remaining Phase 5 work must now wire real producers/handlers: dispatcher async events, Meilisearch indexing, automation webhooks/transcripts, delayed actions/macros, CSAT templates, SLA scans, contact exports, provider webhooks, outbound delivery, Captain/Copilot, and conversation maintenance.
|
||||
|
||||
P5.1 verification:
|
||||
|
||||
```bash
|
||||
env GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go test ./internal/worker -count=1
|
||||
env TMPDIR=/home/rogee/Projects/gochat/.tmp/test-tmp GOCACHE=/tmp/gochat-gocache GOMODCACHE=/tmp/gochat-gomodcache go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
## Phase 6: Core Product Placeholder Burn-down
|
||||
|
||||
Status: doing.
|
||||
@@ -1807,3 +1824,4 @@ Verification milestone gates:
|
||||
- 2026-06-05: B12.2b browser smoke harness checkpoint prepared as `test(parity): add browser smoke harness`; `scripts/parity_frontend_smoke.sh --browser-smoke` now runs the live API smoke, verifies Chatwoot Vite readiness, and drives the reused Chatwoot login/dashboard entrypoints through headless Chrome DevTools without adding Playwright or frontend adapters. The smoke captures browser requests and console output in `.tmp/frontend-smoke/browser-smoke-report.json`, while default `--check` keeps live browser execution optional because it needs PostgreSQL, Redis, Meilisearch, GoChat, Vite, and Chrome. Verification: `bash -n`, `node --check`, `--print`, `--check`, full `go test ./...`, and `git diff --check`. Next slice is B12.3 enterprise screen smoke.
|
||||
- 2026-06-05: B12.3a enterprise API smoke checkpoint prepared as `test(parity): add enterprise api smoke coverage`; `scripts/parity_frontend_smoke.sh --enterprise-smoke` now composes the core API smoke with enterprise checks for SLA reports/download, CSAT public submit plus reports/download, automation/macros, audit/custom roles, capacity, Captain preferences/assistants, and Copilot thread/message payloads. Default verification keeps the live enterprise run optional because it needs PostgreSQL, Redis, Meilisearch, and GoChat. Verification: `bash -n`, `--print`, and `--check` passed locally. Next slice is B12.3b reused-frontend enterprise browser navigation.
|
||||
- 2026-06-05: B12.3b enterprise browser smoke checkpoint prepared as `test(parity): add enterprise browser smoke coverage`; `scripts/parity_frontend_smoke.sh --enterprise-browser-smoke` now runs enterprise API smoke, verifies Chatwoot Vite, and drives reused Chatwoot enterprise dashboard routes through the Chrome DevTools browser harness. Browser assertions cover SLA, CSAT, automation, macros, audit logs, custom roles, agent capacity, Captain settings/assistants, and Copilot thread request paths. Seed now enables the corresponding enterprise feature flags for route guards. Verification: `bash -n`, `node --check`, `--print`, `--check`, `go test ./cmd/gochat`, full `go test ./...`, and `git diff --check` passed locally. Next slice is Phase 5 job parity.
|
||||
- 2026-06-05: P5.1 durable worker checkpoint prepared as `feat(worker): add durable background jobs`; the `background_jobs` table/model and WorkerPool core now cover enqueueing, scheduled jobs, queue filtering, idempotency keys, PostgreSQL `SKIP LOCKED` claims, handler execution, retry/backoff, dead-letter state, stale-lock recovery, and graceful start/stop. Focused worker tests cover idempotent enqueue, completion, retries, schedule/queue gating, and restart pickup. Next slice is P5.2/P5.4 integration of dispatcher and automation jobs onto the durable core.
|
||||
|
||||
@@ -242,6 +242,7 @@ func autoMigrate(db *gorm.DB) error {
|
||||
&model.EmailTemplate{},
|
||||
&model.MessageReaction{},
|
||||
&model.Report{},
|
||||
&model.BackgroundJob{},
|
||||
// S6: WorkingHour — out-of-office / business hours per inbox
|
||||
&model.WorkingHour{},
|
||||
}
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
BackgroundJobStatusQueued = "queued"
|
||||
BackgroundJobStatusRunning = "running"
|
||||
BackgroundJobStatusRetrying = "retrying"
|
||||
BackgroundJobStatusCompleted = "completed"
|
||||
BackgroundJobStatusDead = "dead"
|
||||
)
|
||||
|
||||
const DefaultBackgroundJobQueue = "default"
|
||||
|
||||
// BackgroundJob stores durable background work in the database.
|
||||
// Reference: Chatwoot ActiveJob/Sidekiq jobs with retry, delayed scheduling,
|
||||
// mutex/idempotency, and observable failure state.
|
||||
type BackgroundJob struct {
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
Queue string `gorm:"size:100;not null;default:'default';index:idx_background_jobs_ready,priority:2" json:"queue"`
|
||||
JobType string `gorm:"size:150;not null;index" json:"job_type"`
|
||||
Payload json.RawMessage `gorm:"type:jsonb;not null;default:'{}'" json:"payload"`
|
||||
Status string `gorm:"size:50;not null;default:'queued';index:idx_background_jobs_ready,priority:1" json:"status"`
|
||||
Priority int `gorm:"not null;default:0" json:"priority"`
|
||||
Attempts int `gorm:"not null;default:0" json:"attempts"`
|
||||
MaxAttempts int `gorm:"not null;default:3" json:"max_attempts"`
|
||||
ScheduledAt time.Time `gorm:"not null;index:idx_background_jobs_ready,priority:3" json:"scheduled_at"`
|
||||
LockedAt *time.Time `gorm:"index" json:"locked_at,omitempty"`
|
||||
LockedBy string `gorm:"size:150" json:"locked_by,omitempty"`
|
||||
IdempotencyKey string `gorm:"size:255;index" json:"idempotency_key,omitempty"`
|
||||
LastError string `gorm:"type:text" json:"last_error,omitempty"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
FailedAt *time.Time `json:"failed_at,omitempty"`
|
||||
CreatedAt time.Time `gorm:"autoCreateTime" json:"created_at"`
|
||||
UpdatedAt time.Time `gorm:"autoUpdateTime" json:"updated_at"`
|
||||
}
|
||||
|
||||
func (BackgroundJob) TableName() string { return "background_jobs" }
|
||||
@@ -168,6 +168,7 @@ func defaultTestModels() []interface{} {
|
||||
&model.Mention{},
|
||||
&model.IntegrationHook{},
|
||||
&model.IntegrationApp{},
|
||||
&model.BackgroundJob{},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+407
-17
@@ -1,31 +1,421 @@
|
||||
package worker
|
||||
|
||||
// Worker processes background jobs such as:
|
||||
// - Auto-assignment of conversations to agents
|
||||
// - Sending outgoing messages through channels
|
||||
// - Processing webhook events asynchronously
|
||||
// - Notification delivery
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
// WorkerPool manages a set of background job processors.
|
||||
// This is a placeholder — will be implemented with proper job queue
|
||||
// (e.g., Redis-based or database-backed queue).
|
||||
"github.com/gochat/gochat/internal/model"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
)
|
||||
|
||||
var ErrWorkerDatabaseRequired = errors.New("worker database is required")
|
||||
|
||||
// JobHandler performs one durable background job.
|
||||
type JobHandler func(context.Context, *model.BackgroundJob) error
|
||||
|
||||
type BackoffFunc func(attempt int) time.Duration
|
||||
|
||||
// WorkerPool manages durable background job processors.
|
||||
// Reference: Chatwoot ActiveJob/Sidekiq workers use persistent jobs, scheduled
|
||||
// execution, retry, and dead-letter visibility for async side effects.
|
||||
type WorkerPool struct {
|
||||
// TODO: implement with job queue
|
||||
db *gorm.DB
|
||||
handlers map[string]JobHandler
|
||||
queues []string
|
||||
workerID string
|
||||
workerCount int
|
||||
pollInterval time.Duration
|
||||
staleLockTimeout time.Duration
|
||||
backoff BackoffFunc
|
||||
now func() time.Time
|
||||
|
||||
mu sync.RWMutex
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
// NewWorkerPool creates a new WorkerPool instance.
|
||||
func NewWorkerPool() *WorkerPool {
|
||||
return &WorkerPool{}
|
||||
type Option func(*WorkerPool)
|
||||
|
||||
// NewWorkerPool creates a WorkerPool. The variadic DB preserves older no-arg
|
||||
// construction while allowing Phase 5 durable workers to be wired incrementally.
|
||||
func NewWorkerPool(db ...*gorm.DB) *WorkerPool {
|
||||
wp := &WorkerPool{
|
||||
handlers: make(map[string]JobHandler),
|
||||
queues: []string{model.DefaultBackgroundJobQueue},
|
||||
workerID: fmt.Sprintf("worker-%d", time.Now().UnixNano()),
|
||||
workerCount: 1,
|
||||
pollInterval: 500 * time.Millisecond,
|
||||
staleLockTimeout: 15 * time.Minute,
|
||||
backoff: defaultBackoff,
|
||||
now: time.Now,
|
||||
}
|
||||
if len(db) > 0 {
|
||||
wp.db = db[0]
|
||||
}
|
||||
return wp
|
||||
}
|
||||
|
||||
func NewWorkerPoolWithOptions(db *gorm.DB, opts ...Option) *WorkerPool {
|
||||
wp := NewWorkerPool(db)
|
||||
for _, opt := range opts {
|
||||
opt(wp)
|
||||
}
|
||||
return wp
|
||||
}
|
||||
|
||||
func WithWorkerCount(count int) Option {
|
||||
return func(wp *WorkerPool) {
|
||||
if count > 0 {
|
||||
wp.workerCount = count
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithQueues(queues ...string) Option {
|
||||
return func(wp *WorkerPool) {
|
||||
if len(queues) > 0 {
|
||||
wp.queues = append([]string(nil), queues...)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithPollInterval(interval time.Duration) Option {
|
||||
return func(wp *WorkerPool) {
|
||||
if interval > 0 {
|
||||
wp.pollInterval = interval
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithWorkerID(id string) Option {
|
||||
return func(wp *WorkerPool) {
|
||||
if id != "" {
|
||||
wp.workerID = id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithStaleLockTimeout(timeout time.Duration) Option {
|
||||
return func(wp *WorkerPool) {
|
||||
if timeout > 0 {
|
||||
wp.staleLockTimeout = timeout
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithBackoff(backoff BackoffFunc) Option {
|
||||
return func(wp *WorkerPool) {
|
||||
if backoff != nil {
|
||||
wp.backoff = backoff
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithNow(now func() time.Time) Option {
|
||||
return func(wp *WorkerPool) {
|
||||
if now != nil {
|
||||
wp.now = now
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) Register(jobType string, handler JobHandler) {
|
||||
wp.mu.Lock()
|
||||
defer wp.mu.Unlock()
|
||||
wp.handlers[jobType] = handler
|
||||
}
|
||||
|
||||
type EnqueueOption func(*model.BackgroundJob)
|
||||
|
||||
func WithQueue(queue string) EnqueueOption {
|
||||
return func(job *model.BackgroundJob) {
|
||||
if queue != "" {
|
||||
job.Queue = queue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithScheduledAt(at time.Time) EnqueueOption {
|
||||
return func(job *model.BackgroundJob) {
|
||||
if !at.IsZero() {
|
||||
job.ScheduledAt = at
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithMaxAttempts(max int) EnqueueOption {
|
||||
return func(job *model.BackgroundJob) {
|
||||
if max > 0 {
|
||||
job.MaxAttempts = max
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func WithPriority(priority int) EnqueueOption {
|
||||
return func(job *model.BackgroundJob) {
|
||||
job.Priority = priority
|
||||
}
|
||||
}
|
||||
|
||||
func WithIdempotencyKey(key string) EnqueueOption {
|
||||
return func(job *model.BackgroundJob) {
|
||||
job.IdempotencyKey = key
|
||||
}
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) Enqueue(ctx context.Context, jobType string, payload any, opts ...EnqueueOption) (*model.BackgroundJob, error) {
|
||||
if wp.db == nil {
|
||||
return nil, ErrWorkerDatabaseRequired
|
||||
}
|
||||
if jobType == "" {
|
||||
return nil, errors.New("job type is required")
|
||||
}
|
||||
|
||||
payloadBytes, err := marshalPayload(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
job := &model.BackgroundJob{
|
||||
Queue: model.DefaultBackgroundJobQueue,
|
||||
JobType: jobType,
|
||||
Payload: payloadBytes,
|
||||
Status: model.BackgroundJobStatusQueued,
|
||||
MaxAttempts: 3,
|
||||
ScheduledAt: wp.now(),
|
||||
}
|
||||
for _, opt := range opts {
|
||||
opt(job)
|
||||
}
|
||||
if job.ScheduledAt.IsZero() {
|
||||
job.ScheduledAt = wp.now()
|
||||
}
|
||||
if job.MaxAttempts <= 0 {
|
||||
job.MaxAttempts = 3
|
||||
}
|
||||
|
||||
if job.IdempotencyKey != "" {
|
||||
var existing model.BackgroundJob
|
||||
err := wp.db.WithContext(ctx).Where("idempotency_key = ?", job.IdempotencyKey).First(&existing).Error
|
||||
if err == nil {
|
||||
return &existing, nil
|
||||
}
|
||||
if !errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
if err := wp.db.WithContext(ctx).Create(job).Error; err != nil {
|
||||
if job.IdempotencyKey != "" {
|
||||
var existing model.BackgroundJob
|
||||
if findErr := wp.db.WithContext(ctx).Where("idempotency_key = ?", job.IdempotencyKey).First(&existing).Error; findErr == nil {
|
||||
return &existing, nil
|
||||
}
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return job, nil
|
||||
}
|
||||
|
||||
// Start initializes and begins processing background jobs.
|
||||
func (wp *WorkerPool) Start() error {
|
||||
// TODO: implement
|
||||
if wp.db == nil {
|
||||
return nil
|
||||
}
|
||||
wp.mu.Lock()
|
||||
if wp.cancel != nil {
|
||||
wp.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
wp.ctx, wp.cancel = context.WithCancel(context.Background())
|
||||
ctx := wp.ctx
|
||||
workerCount := wp.workerCount
|
||||
wp.mu.Unlock()
|
||||
|
||||
if _, err := wp.RequeueStaleJobs(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
for i := 0; i < workerCount; i++ {
|
||||
wp.wg.Add(1)
|
||||
go wp.run(ctx)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop gracefully shuts down the worker pool.
|
||||
func (wp *WorkerPool) Stop() error {
|
||||
// TODO: implement
|
||||
wp.mu.Lock()
|
||||
cancel := wp.cancel
|
||||
wp.cancel = nil
|
||||
wp.ctx = nil
|
||||
wp.mu.Unlock()
|
||||
if cancel != nil {
|
||||
cancel()
|
||||
}
|
||||
wp.wg.Wait()
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) ProcessOne(ctx context.Context) (bool, error) {
|
||||
if wp.db == nil {
|
||||
return false, ErrWorkerDatabaseRequired
|
||||
}
|
||||
job, err := wp.claimNext(ctx)
|
||||
if err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return false, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
return true, wp.perform(ctx, job)
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) RequeueStaleJobs(ctx context.Context) (int64, error) {
|
||||
if wp.db == nil {
|
||||
return 0, ErrWorkerDatabaseRequired
|
||||
}
|
||||
cutoff := wp.now().Add(-wp.staleLockTimeout)
|
||||
updates := map[string]any{
|
||||
"status": model.BackgroundJobStatusRetrying,
|
||||
"locked_at": nil,
|
||||
"locked_by": "",
|
||||
}
|
||||
result := wp.db.WithContext(ctx).Model(&model.BackgroundJob{}).
|
||||
Where("status = ? AND locked_at IS NOT NULL AND locked_at < ?", model.BackgroundJobStatusRunning, cutoff).
|
||||
Updates(updates)
|
||||
return result.RowsAffected, result.Error
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) run(ctx context.Context) {
|
||||
defer wp.wg.Done()
|
||||
ticker := time.NewTicker(wp.pollInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
processed, _ := wp.ProcessOne(ctx)
|
||||
if processed {
|
||||
continue
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) claimNext(ctx context.Context) (*model.BackgroundJob, error) {
|
||||
var job model.BackgroundJob
|
||||
err := wp.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
query := tx.Where("status IN ? AND scheduled_at <= ?", []string{model.BackgroundJobStatusQueued, model.BackgroundJobStatusRetrying}, wp.now())
|
||||
if len(wp.queues) > 0 {
|
||||
query = query.Where("queue IN ?", wp.queues)
|
||||
}
|
||||
if tx.Dialector != nil && tx.Dialector.Name() == "postgres" {
|
||||
query = query.Clauses(clause.Locking{Strength: "UPDATE", Options: "SKIP LOCKED"})
|
||||
}
|
||||
if err := query.Order("priority DESC, scheduled_at ASC, id ASC").First(&job).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
lockedAt := wp.now()
|
||||
updates := map[string]any{
|
||||
"status": model.BackgroundJobStatusRunning,
|
||||
"locked_at": &lockedAt,
|
||||
"locked_by": wp.workerID,
|
||||
"attempts": gorm.Expr("attempts + 1"),
|
||||
}
|
||||
if err := tx.Model(&model.BackgroundJob{}).Where("id = ?", job.ID).Updates(updates).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.First(&job, job.ID).Error
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &job, nil
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) perform(ctx context.Context, job *model.BackgroundJob) error {
|
||||
handler := wp.handlerFor(job.JobType)
|
||||
if handler == nil {
|
||||
return wp.fail(ctx, job, fmt.Errorf("no handler registered for job type %q", job.JobType))
|
||||
}
|
||||
if err := handler(ctx, job); err != nil {
|
||||
return wp.fail(ctx, job, err)
|
||||
}
|
||||
finishedAt := wp.now()
|
||||
updates := map[string]any{
|
||||
"status": model.BackgroundJobStatusCompleted,
|
||||
"locked_at": nil,
|
||||
"locked_by": "",
|
||||
"last_error": "",
|
||||
"finished_at": &finishedAt,
|
||||
}
|
||||
return wp.db.WithContext(ctx).Model(&model.BackgroundJob{}).Where("id = ?", job.ID).Updates(updates).Error
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) fail(ctx context.Context, job *model.BackgroundJob, err error) error {
|
||||
now := wp.now()
|
||||
updates := map[string]any{
|
||||
"locked_at": nil,
|
||||
"locked_by": "",
|
||||
"last_error": err.Error(),
|
||||
}
|
||||
if job.Attempts >= job.MaxAttempts {
|
||||
updates["status"] = model.BackgroundJobStatusDead
|
||||
updates["failed_at"] = &now
|
||||
} else {
|
||||
updates["status"] = model.BackgroundJobStatusRetrying
|
||||
updates["scheduled_at"] = now.Add(wp.backoff(job.Attempts))
|
||||
}
|
||||
if updateErr := wp.db.WithContext(ctx).Model(&model.BackgroundJob{}).Where("id = ?", job.ID).Updates(updates).Error; updateErr != nil {
|
||||
return updateErr
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (wp *WorkerPool) handlerFor(jobType string) JobHandler {
|
||||
wp.mu.RLock()
|
||||
defer wp.mu.RUnlock()
|
||||
return wp.handlers[jobType]
|
||||
}
|
||||
|
||||
func marshalPayload(payload any) (json.RawMessage, error) {
|
||||
if payload == nil {
|
||||
return json.RawMessage(`{}`), nil
|
||||
}
|
||||
switch value := payload.(type) {
|
||||
case json.RawMessage:
|
||||
if !json.Valid(value) {
|
||||
return nil, errors.New("job payload must be valid JSON")
|
||||
}
|
||||
return value, nil
|
||||
case []byte:
|
||||
if !json.Valid(value) {
|
||||
return nil, errors.New("job payload must be valid JSON")
|
||||
}
|
||||
return json.RawMessage(value), nil
|
||||
default:
|
||||
payloadBytes, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return payloadBytes, nil
|
||||
}
|
||||
}
|
||||
|
||||
func defaultBackoff(attempt int) time.Duration {
|
||||
if attempt <= 0 {
|
||||
attempt = 1
|
||||
}
|
||||
seconds := math.Pow(2, float64(attempt-1))
|
||||
if seconds > 300 {
|
||||
seconds = 300
|
||||
}
|
||||
return time.Duration(seconds) * time.Second
|
||||
}
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/gochat/gochat/internal/model"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
func newWorkerTestDB(t *testing.T) *gorm.DB {
|
||||
t.Helper()
|
||||
db, err := gorm.Open(sqlite.Open("file:worker-test?mode=memory&cache=shared"), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
|
||||
if err != nil {
|
||||
t.Fatalf("open sqlite: %v", err)
|
||||
}
|
||||
sqlDB, err := db.DB()
|
||||
if err != nil {
|
||||
t.Fatalf("sqlite db handle: %v", err)
|
||||
}
|
||||
sqlDB.SetMaxOpenConns(1)
|
||||
if err := db.AutoMigrate(&model.BackgroundJob{}); err != nil {
|
||||
t.Fatalf("migrate background jobs: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
db.Exec("DELETE FROM background_jobs")
|
||||
sqlDB.Close()
|
||||
})
|
||||
return db
|
||||
}
|
||||
|
||||
func loadJob(t *testing.T, db *gorm.DB, id uint) model.BackgroundJob {
|
||||
t.Helper()
|
||||
var job model.BackgroundJob
|
||||
if err := db.First(&job, id).Error; err != nil {
|
||||
t.Fatalf("load job: %v", err)
|
||||
}
|
||||
return job
|
||||
}
|
||||
|
||||
func TestWorkerPoolEnqueueStoresPayloadAndIdempotency(t *testing.T) {
|
||||
db := newWorkerTestDB(t)
|
||||
now := time.Date(2026, 6, 5, 10, 0, 0, 0, time.UTC)
|
||||
wp := NewWorkerPoolWithOptions(db, WithNow(func() time.Time { return now }))
|
||||
|
||||
job, err := wp.Enqueue(context.Background(), "contact_export", map[string]any{"account_id": 1}, WithIdempotencyKey("contact-export:1"), WithQueue("exports"), WithMaxAttempts(5), WithPriority(10))
|
||||
if err != nil {
|
||||
t.Fatalf("enqueue: %v", err)
|
||||
}
|
||||
duplicate, err := wp.Enqueue(context.Background(), "contact_export", map[string]any{"account_id": 2}, WithIdempotencyKey("contact-export:1"))
|
||||
if err != nil {
|
||||
t.Fatalf("enqueue duplicate: %v", err)
|
||||
}
|
||||
if duplicate.ID != job.ID {
|
||||
t.Fatalf("expected duplicate enqueue to return existing job %d, got %d", job.ID, duplicate.ID)
|
||||
}
|
||||
|
||||
reloaded := loadJob(t, db, job.ID)
|
||||
if reloaded.Queue != "exports" || reloaded.JobType != "contact_export" || reloaded.Status != model.BackgroundJobStatusQueued || reloaded.MaxAttempts != 5 || reloaded.Priority != 10 {
|
||||
t.Fatalf("unexpected job fields: %+v", reloaded)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(reloaded.Payload, &payload); err != nil {
|
||||
t.Fatalf("unmarshal payload: %v", err)
|
||||
}
|
||||
if payload["account_id"].(float64) != 1 {
|
||||
t.Fatalf("unexpected payload: %s", string(reloaded.Payload))
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerPoolProcessOneCompletesDueJob(t *testing.T) {
|
||||
db := newWorkerTestDB(t)
|
||||
now := time.Date(2026, 6, 5, 10, 0, 0, 0, time.UTC)
|
||||
wp := NewWorkerPoolWithOptions(db, WithNow(func() time.Time { return now }), WithWorkerID("test-worker"))
|
||||
var handled atomic.Int32
|
||||
wp.Register("send_reply", func(ctx context.Context, job *model.BackgroundJob) error {
|
||||
handled.Add(1)
|
||||
if job.Attempts != 1 || job.LockedBy != "test-worker" || job.LockedAt == nil {
|
||||
t.Fatalf("job was not claimed before handler: %+v", job)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
job, err := wp.Enqueue(context.Background(), "send_reply", map[string]any{"message_id": 7})
|
||||
if err != nil {
|
||||
t.Fatalf("enqueue: %v", err)
|
||||
}
|
||||
|
||||
processed, err := wp.ProcessOne(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("process one: %v", err)
|
||||
}
|
||||
if !processed || handled.Load() != 1 {
|
||||
t.Fatalf("expected one handled job, processed=%v handled=%d", processed, handled.Load())
|
||||
}
|
||||
reloaded := loadJob(t, db, job.ID)
|
||||
if reloaded.Status != model.BackgroundJobStatusCompleted || reloaded.FinishedAt == nil || reloaded.LockedAt != nil || reloaded.LockedBy != "" {
|
||||
t.Fatalf("expected completed unlocked job: %+v", reloaded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerPoolRetriesThenDeadLettersFailures(t *testing.T) {
|
||||
db := newWorkerTestDB(t)
|
||||
now := time.Date(2026, 6, 5, 10, 0, 0, 0, time.UTC)
|
||||
wp := NewWorkerPoolWithOptions(db, WithNow(func() time.Time { return now }), WithBackoff(func(attempt int) time.Duration { return 0 }))
|
||||
boom := errors.New("provider timeout")
|
||||
wp.Register("webhook_delivery", func(ctx context.Context, job *model.BackgroundJob) error { return boom })
|
||||
job, err := wp.Enqueue(context.Background(), "webhook_delivery", nil, WithMaxAttempts(2))
|
||||
if err != nil {
|
||||
t.Fatalf("enqueue: %v", err)
|
||||
}
|
||||
|
||||
processed, err := wp.ProcessOne(context.Background())
|
||||
if !processed || !errors.Is(err, boom) {
|
||||
t.Fatalf("expected first failure, processed=%v err=%v", processed, err)
|
||||
}
|
||||
reloaded := loadJob(t, db, job.ID)
|
||||
if reloaded.Status != model.BackgroundJobStatusRetrying || reloaded.Attempts != 1 || reloaded.LastError != boom.Error() || reloaded.FailedAt != nil {
|
||||
t.Fatalf("expected retrying job after first failure: %+v", reloaded)
|
||||
}
|
||||
|
||||
processed, err = wp.ProcessOne(context.Background())
|
||||
if !processed || !errors.Is(err, boom) {
|
||||
t.Fatalf("expected second failure, processed=%v err=%v", processed, err)
|
||||
}
|
||||
reloaded = loadJob(t, db, job.ID)
|
||||
if reloaded.Status != model.BackgroundJobStatusDead || reloaded.Attempts != 2 || reloaded.FailedAt == nil || reloaded.LockedAt != nil {
|
||||
t.Fatalf("expected dead-lettered job: %+v", reloaded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerPoolRespectsScheduleAndQueues(t *testing.T) {
|
||||
db := newWorkerTestDB(t)
|
||||
now := time.Date(2026, 6, 5, 10, 0, 0, 0, time.UTC)
|
||||
wp := NewWorkerPoolWithOptions(db, WithNow(func() time.Time { return now }), WithQueues("critical"))
|
||||
wp.Register("sla_scan", func(ctx context.Context, job *model.BackgroundJob) error { return nil })
|
||||
if _, err := wp.Enqueue(context.Background(), "sla_scan", nil, WithQueue("default")); err != nil {
|
||||
t.Fatalf("enqueue default: %v", err)
|
||||
}
|
||||
if _, err := wp.Enqueue(context.Background(), "sla_scan", nil, WithQueue("critical"), WithScheduledAt(now.Add(time.Hour))); err != nil {
|
||||
t.Fatalf("enqueue future: %v", err)
|
||||
}
|
||||
processed, err := wp.ProcessOne(context.Background())
|
||||
if err != nil || processed {
|
||||
t.Fatalf("expected no eligible job, processed=%v err=%v", processed, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerPoolRequeuesStaleRunningJobs(t *testing.T) {
|
||||
db := newWorkerTestDB(t)
|
||||
now := time.Date(2026, 6, 5, 10, 0, 0, 0, time.UTC)
|
||||
wp := NewWorkerPoolWithOptions(db, WithNow(func() time.Time { return now }), WithStaleLockTimeout(time.Minute))
|
||||
lockedAt := now.Add(-2 * time.Minute)
|
||||
job := model.BackgroundJob{
|
||||
Queue: model.DefaultBackgroundJobQueue,
|
||||
JobType: "captain_document_sync",
|
||||
Payload: json.RawMessage(`{}`),
|
||||
Status: model.BackgroundJobStatusRunning,
|
||||
MaxAttempts: 3,
|
||||
ScheduledAt: now.Add(-time.Hour),
|
||||
LockedAt: &lockedAt,
|
||||
LockedBy: "dead-worker",
|
||||
}
|
||||
if err := db.Create(&job).Error; err != nil {
|
||||
t.Fatalf("create stale job: %v", err)
|
||||
}
|
||||
|
||||
count, err := wp.RequeueStaleJobs(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("requeue stale: %v", err)
|
||||
}
|
||||
if count != 1 {
|
||||
t.Fatalf("expected 1 stale job requeued, got %d", count)
|
||||
}
|
||||
reloaded := loadJob(t, db, job.ID)
|
||||
if reloaded.Status != model.BackgroundJobStatusRetrying || reloaded.LockedAt != nil || reloaded.LockedBy != "" {
|
||||
t.Fatalf("expected retrying unlocked stale job: %+v", reloaded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerPoolStartAndStopProcessJobs(t *testing.T) {
|
||||
db := newWorkerTestDB(t)
|
||||
now := time.Date(2026, 6, 5, 10, 0, 0, 0, time.UTC)
|
||||
wp := NewWorkerPoolWithOptions(db, WithNow(func() time.Time { return now }), WithPollInterval(5*time.Millisecond))
|
||||
var handled atomic.Int32
|
||||
wp.Register("event_dispatch", func(ctx context.Context, job *model.BackgroundJob) error {
|
||||
handled.Add(1)
|
||||
return nil
|
||||
})
|
||||
job, err := wp.Enqueue(context.Background(), "event_dispatch", map[string]any{"event": "conversation_created"})
|
||||
if err != nil {
|
||||
t.Fatalf("enqueue: %v", err)
|
||||
}
|
||||
if err := wp.Start(); err != nil {
|
||||
t.Fatalf("start worker: %v", err)
|
||||
}
|
||||
defer wp.Stop()
|
||||
|
||||
deadline := time.Now().Add(time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if handled.Load() == 1 {
|
||||
break
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if handled.Load() != 1 {
|
||||
t.Fatalf("worker loop did not process job")
|
||||
}
|
||||
if err := wp.Stop(); err != nil {
|
||||
t.Fatalf("stop worker: %v", err)
|
||||
}
|
||||
reloaded := loadJob(t, db, job.ID)
|
||||
if reloaded.Status != model.BackgroundJobStatusCompleted {
|
||||
t.Fatalf("expected completed job after worker loop: %+v", reloaded)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
DROP INDEX IF EXISTS idx_background_jobs_idempotency_key_unique;
|
||||
DROP INDEX IF EXISTS idx_background_jobs_idempotency_key;
|
||||
DROP INDEX IF EXISTS idx_background_jobs_locked_at;
|
||||
DROP INDEX IF EXISTS idx_background_jobs_job_type;
|
||||
DROP INDEX IF EXISTS idx_background_jobs_ready;
|
||||
DROP TABLE IF EXISTS background_jobs;
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
CREATE TABLE IF NOT EXISTS background_jobs (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
queue VARCHAR(100) NOT NULL DEFAULT 'default',
|
||||
job_type VARCHAR(150) NOT NULL,
|
||||
payload JSONB NOT NULL DEFAULT '{}',
|
||||
status VARCHAR(50) NOT NULL DEFAULT 'queued',
|
||||
priority INTEGER NOT NULL DEFAULT 0,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
max_attempts INTEGER NOT NULL DEFAULT 3,
|
||||
scheduled_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
locked_at TIMESTAMP WITH TIME ZONE,
|
||||
locked_by VARCHAR(150),
|
||||
idempotency_key VARCHAR(255),
|
||||
last_error TEXT,
|
||||
finished_at TIMESTAMP WITH TIME ZONE,
|
||||
failed_at TIMESTAMP WITH TIME ZONE,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_background_jobs_ready
|
||||
ON background_jobs(status, queue, scheduled_at, priority DESC, id);
|
||||
CREATE INDEX IF NOT EXISTS idx_background_jobs_job_type ON background_jobs(job_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_background_jobs_locked_at ON background_jobs(locked_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_background_jobs_idempotency_key ON background_jobs(idempotency_key);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_background_jobs_idempotency_key_unique
|
||||
ON background_jobs(idempotency_key)
|
||||
WHERE idempotency_key IS NOT NULL AND idempotency_key <> '';
|
||||
|
||||
Reference in New Issue
Block a user