From 8f7000dc8de8b631e566f5d7a504629272ae9502 Mon Sep 17 00:00:00 2001 From: Rogee Date: Wed, 4 Feb 2026 18:33:36 +0800 Subject: [PATCH] docs: capture P3-17 plan and close storage provider task Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode) Co-authored-by: Sisyphus --- docs/plan.md | 115 +++++++++++++++++++++++---------------- docs/plans/2026-02-04.md | 93 ++++++++++++++++++------------- docs/todo_list.md | 13 ++++- 3 files changed, 133 insertions(+), 88 deletions(-) diff --git a/docs/plan.md b/docs/plan.md index a2b363b..ae51f1c 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -1,38 +1,38 @@ -# Implementation Plan: backend-test-coverage +# Implementation Plan: p3-17-media-processing-s3 -**Branch**: `[test-coverage-t3-t4]` | **Date**: 2026-02-04 | **Spec**: N/A -**Input**: Continuation of test coverage tasks (T3/T4) from prior session; no feature spec. +**Branch**: `[p3-17-media-processing-s3]` | **Date**: 2026-02-04 | **Spec**: `docs/todo_list.md#17` +**Input**: P3-17 “媒体处理管线适配对象存储（S3/MinIO）” + user request to extract a cover from `fixtures/demo.mp4` via ffmpeg. ## Summary -Complete backend service test coverage for content access policies (T3) and superadmin write operations (T4), ensuring existing behavior is validated without altering production logic. +Adapt the media processing worker to support S3/MinIO storage by downloading source media to a temp directory, running ffmpeg to generate a cover image, uploading derived assets via the storage provider, and preserving the existing local filesystem path. Include fixture-based verification using `fixtures/demo.mp4`. ## Technical Context -**Language/Version**: Go 1.x (project standard) -**Primary Dependencies**: Fiber, GORM-Gen, Testify -**Storage**: PostgreSQL (via GORM) -**Testing**: `go test` (service tests under `backend/app/services/*_test.go`) -**Target Platform**: Linux server -**Project Type**: Web application (frontend + backend) -**Performance Goals**: N/A (test-only changes) -**Constraints**: No changes to generated files; keep tests aligned with existing service patterns -**Scale/Scope**: Backend service tests only (no frontend scope) +**Language/Version**: Go 1.x (project standard) +**Primary Dependencies**: River (jobs), MinIO SDK (S3 compatible), Fiber +**Storage**: PostgreSQL + local filesystem / S3-compatible storage +**Testing**: `go test` (service/job tests), ffmpeg CLI for fixture validation +**Target Platform**: Linux server +**Project Type**: Web application (backend + frontend, backend-only changes) +**Performance Goals**: N/A (processing path only) +**Constraints**: Preserve local storage behavior; do not edit generated files; follow `backend/llm.txt`; ensure temp files are cleaned +**Scale/Scope**: Media processing worker + storage provider only ## Constitution Check -- Follow `backend/llm.txt` for backend conventions. -- Keep controllers thin; service tests only (no controller edits). -- Avoid editing generated files (`routes.gen.go`, `docs.go`). -- Run `go test` for impacted service packages. +- Follow `backend/llm.txt` conventions and Chinese comments for business logic. +- Do not edit generated files (`*.gen.go`, `backend/docs/docs.go`). +- Keep controller thin; changes limited to jobs/services/providers. +- Preserve local provider behavior; add S3 path without regression. ## Project Structure ### Documentation (this feature) ```text -specs/[###-feature]/ -└── (not used for this task) +docs/ +└── plan.md # This plan ``` ### Source Code (repository root) @@ -40,49 +40,68 @@ specs/[###-feature]/ ```text backend/ ├── app/ -│ ├── services/ -│ │ ├── content_test.go -│ │ └── super_test.go -└── app/http/v1/dto/ - └── content.go +│ ├── jobs/ +│ │ ├── media_process_job.go +│ │ └── args/media_asset_process.go +│ └── services/ +│ └── common.go +└── providers/ + └── storage/provider.go + +fixtures/ +└── demo.mp4 ``` -**Structure Decision**: Web application structure; scope is backend service tests in `backend/app/services`. +**Structure Decision**: Web application backend; scope limited to media processing worker and storage provider integration. ## Plan Phases -1. Inspect DTO definitions used by content tests to fix T3 assertions. -2. Implement remaining content access policy tests (T3) and verify via `go test`. -3. Implement superadmin write operation tests (T4) and verify via `go test`. +1. **Design & plumbing**: Define temp file conventions and storage download API for S3/local. +2. **Implementation**: Add S3 processing flow to worker and cover asset registration; keep local path intact. +3. **Verification**: Add tests (or integration checks) and run fixture-based ffmpeg validation. ## Tasks -1. Read `backend/app/http/v1/dto/content.go` and update T3 test assertions to match actual DTO fields. -2. Extend `backend/app/services/content_test.go` with missing content access policy cases and run targeted tests. -3. Extend `backend/app/services/super_test.go` for superadmin write operations; run service test suite. -4. Verify all added tests pass without modifying production logic. +1. **Define storage download interface** + - Add a storage provider helper to download an object to a local temp file (local: copy from `LocalPath/objectKey` without rename; S3: `FGetObject`). + - Ensure helper never mutates/deletes the source object, creates parent dirs for destination, and overwrites the destination if it already exists. + - Ensure API is used by jobs without leaking provider-specific logic. + +2. **Adapt `MediaProcessWorker` for S3/MinIO** + - For non-local providers, download the source object into a temp directory. + - Run ffmpeg to extract a cover image from the temp file. + - Upload the cover via `storage.PutObject` and register the derived media asset. + - Ensure temp directories/files are cleaned on success or failure. + +3. **Update cover asset registration** + - For non-local providers, avoid filesystem rename; upload via storage provider and keep object key conventions. + - Use `storage.PutObject(ctx, objectKey, coverTempPath, "image/jpeg")`, then cleanup temp files/dirs. + - Keep local path move behavior unchanged. + +4. **Add tests / verification hooks** + - Add a job/service test for S3 path (gated by MinIO env if needed). + - Keep/extend local path test to ensure no regression. + - Validate `fixtures/demo.mp4` can produce a cover with ffmpeg. ## Dependencies -- Task 1 must complete before Task 2 (DTO fields drive assertions). -- Task 2 should complete before Task 3 to isolate failures. +- Task 1 must complete before Task 2 (worker needs download API). +- Task 2 must complete before Task 3 (cover registration requires new flow). +- Task 4 depends on Task 2 & 3 (tests rely on updated pipeline). ## Acceptance Criteria -- `backend/app/services/content_test.go` has passing T3 coverage for unauthenticated access constraints. -- `backend/app/services/super_test.go` includes T4 coverage for create-tenant side effects and superadmin write operations. -- `go test ./backend/app/services/...` passes. -- No generated files modified. +- **Local path unchanged**: `MediaProcessWorker` still generates a cover image for local provider when ffmpeg is available. +- **S3/MinIO path works**: For `Storage.Type = s3`, the worker downloads source media, generates cover via ffmpeg, uploads cover to bucket, and creates a derived `media_assets` record with correct object key. +- **Fixture validation**: `ffmpeg -y -i fixtures/demo.mp4 -ss 00:00:00.000 -vframes 1 /tmp/demo_cover.jpg` succeeds locally and produces a non-empty file. +- **S3/MinIO test config**: tests load Storage config with `Type=s3`, `Endpoint`, `AccessKey`, `SecretKey`, `Bucket`, `PathStyle` (true for MinIO). +- **Automated checks** (to be added in this phase): + - `go test ./backend/app/jobs -run TestMediaProcessWorkerS3 -count=1` passes (with S3/MinIO config loaded). + - `go test ./backend/app/jobs -run TestMediaProcessWorkerLocal -count=1` passes. ## Risks -- DTO field changes may require adjusting test assertions; mitigate by verifying struct definitions. -- Service behavior may differ from assumptions in prior session; mitigate by aligning with existing tests. - -## Complexity Tracking - -> **Fill ONLY if Constitution Check has violations that must be justified** - -| Violation | Why Needed | Simpler Alternative Rejected Because | -|-----------|------------|-------------------------------------| -| N/A | N/A | N/A | +- **ffmpeg not installed**: Worker should detect and log; processing should fail gracefully. +- **Temp storage pressure**: Large media downloads may exceed disk; ensure cleanup on all paths. +- **S3 connectivity/transient errors**: Add retry or error propagation to avoid silent failures. +- **Access policies**: Misconfigured bucket policies may prevent download/upload; surface clear errors. diff --git a/docs/plans/2026-02-04.md b/docs/plans/2026-02-04.md index bdae87b..a2b363b 100644 --- a/docs/plans/2026-02-04.md +++ b/docs/plans/2026-02-04.md @@ -1,69 +1,88 @@ -# Implementation Plan: Portal Payment Page Hardening +# Implementation Plan: backend-test-coverage -**Branch**: `main` | **Date**: 2026-02-04 | **Spec**: `docs/seed_verification.md` -**Input**: 前端支付页仍含 DEV 模拟逻辑，支付错误提示/加载与轮询节流不足，金额/商品信息展示不完整。 +**Branch**: `[test-coverage-t3-t4]` | **Date**: 2026-02-04 | **Spec**: N/A +**Input**: Continuation of test coverage tasks (T3/T4) from prior session; no feature spec. ## Summary -清理支付页 DEV 模拟逻辑，增强支付失败与加载态提示，补全金额/商品信息展示，并对状态轮询做节流，确保真实支付链路稳定可用。 +Complete backend service test coverage for content access policies (T3) and superadmin write operations (T4), ensuring existing behavior is validated without altering production logic. ## Technical Context -**Language/Version**: Vue 3 (Vite) -**Primary Dependencies**: Pinia, Vue Router, PrimeVue -**Storage**: PostgreSQL（后端服务已就绪） -**Testing**: 前端页面流验证；后端 `go test ./...`（规则要求前端改动需二者并行） -**Target Platform**: local/staging -**Project Type**: Web application -**Performance Goals**: 减少轮询负载，提升用户反馈 -**Constraints**: 遵循 `backend/llm.txt`、前端接口涉及需跑页面流 + go test；不手改生成文件 -**Scale/Scope**: 仅 Portal 支付页（PaymentView.vue）及相关 API/状态展示 +**Language/Version**: Go 1.x (project standard) +**Primary Dependencies**: Fiber, GORM-Gen, Testify +**Storage**: PostgreSQL (via GORM) +**Testing**: `go test` (service tests under `backend/app/services/*_test.go`) +**Target Platform**: Linux server +**Project Type**: Web application (frontend + backend) +**Performance Goals**: N/A (test-only changes) +**Constraints**: No changes to generated files; keep tests aligned with existing service patterns +**Scale/Scope**: Backend service tests only (no frontend scope) ## Constitution Check -- 所有前端接口相关改动需完成页面流验证和后端 `go test ./...` 方可归档。 -- 禁止保留 DEV-only 模拟逻辑于生产代码。 +- Follow `backend/llm.txt` for backend conventions. +- Keep controllers thin; service tests only (no controller edits). +- Avoid editing generated files (`routes.gen.go`, `docs.go`). +- Run `go test` for impacted service packages. ## Project Structure -- `frontend/portal/src/views/order/PaymentView.vue` -- `frontend/portal/src/api/order.js`（若需补充错误/数据处理） -- `frontend/portal/src/utils/request.js`（如需请求拦截/错误提示，视需要） +### Documentation (this feature) -**Structure Decision**: 仅修改支付页与关联 API，避免全局侵入。 +```text +specs/[###-feature]/ +└── (not used for this task) +``` + +### Source Code (repository root) + +```text +backend/ +├── app/ +│ ├── services/ +│ │ ├── content_test.go +│ │ └── super_test.go +└── app/http/v1/dto/ + └── content.go +``` + +**Structure Decision**: Web application structure; scope is backend service tests in `backend/app/services`. ## Plan Phases -- Phase 1: 行为清理 —— 移除/隔离 DEV 模拟支付逻辑，确保真实 pay 调用。 -- Phase 2: 体验增强 —— 添加支付中/loading/错误提示；补全金额、商品标题展示；轮询节流与完成后停止。 -- Phase 3: 验证 —— 前端页面流支付冒烟 + 后端 `go test ./...`。 +1. Inspect DTO definitions used by content tests to fix T3 assertions. +2. Implement remaining content access policy tests (T3) and verify via `go test`. +3. Implement superadmin write operation tests (T4) and verify via `go test`. ## Tasks -- [x] T101 移除或显式守护 DEV 模拟支付按钮（生产隐藏/剔除），确保真实 pay 请求。 -- [x] T102 支付提交/轮询的加载与错误提示：提交中禁用按钮，失败 toast/提示，并在错误时停止 loading。 -- [x] T103 订单金额/商品信息展示：优先用 status 返回的 `amount_paid/amount_original` 与 `content_title`，保持 0 元也可显示。 -- [x] T104 轮询节流与完成停止：调整轮询间隔/次数，支付成功即停止，避免过度请求。 -- [x] T201 前端页面流验证支付（登录→Checkout→Pay→Status paid）。 -- [x] T202 后端回归测试 `go test ./...`（与 T201 同时满足）。 +1. Read `backend/app/http/v1/dto/content.go` and update T3 test assertions to match actual DTO fields. +2. Extend `backend/app/services/content_test.go` with missing content access policy cases and run targeted tests. +3. Extend `backend/app/services/super_test.go` for superadmin write operations; run service test suite. +4. Verify all added tests pass without modifying production logic. ## Dependencies -- T101 完成后执行 T102~T104。 -- T104 完成后执行 T201；T201/T202 完成后归档。 +- Task 1 must complete before Task 2 (DTO fields drive assertions). +- Task 2 should complete before Task 3 to isolate failures. ## Acceptance Criteria -- 支付页无 DEV-only 模拟逻辑暴露；“立即支付”触发真实 `/pay`，状态能更新为 `paid`。 -- 支付/轮询有明显 loading/错误反馈；轮询成功后自动停；按钮在提交中禁用。 -- 金额与商品标题在支付页正确显示（含 0 元订单）。 -- 前端页面流验证通过（登录→下单→支付→订单状态 paid）；后端 `go test ./...` 通过。 +- `backend/app/services/content_test.go` has passing T3 coverage for unauthenticated access constraints. +- `backend/app/services/super_test.go` includes T4 coverage for create-tenant side effects and superadmin write operations. +- `go test ./backend/app/services/...` passes. +- No generated files modified. ## Risks -- 轮询过度或停止条件遗漏导致请求风暴；通过节流与成功后清理计时器规避。 -- 错误提示未覆盖网络异常；需在提交与轮询 catch 中统一处理。 +- DTO field changes may require adjusting test assertions; mitigate by verifying struct definitions. +- Service behavior may differ from assumptions in prior session; mitigate by aligning with existing tests. ## Complexity Tracking -暂无。 +> **Fill ONLY if Constitution Check has violations that must be justified** + +| Violation | Why Needed | Simpler Alternative Rejected Because | +|-----------|------------|-------------------------------------| +| N/A | N/A | N/A | diff --git a/docs/todo_list.md b/docs/todo_list.md index c65228b..1c18aae 100644 --- a/docs/todo_list.md +++ b/docs/todo_list.md @@ -6,6 +6,7 @@ - 认证仅使用 JWT（不做 OAuth/Cookie 方案）。 - 支付集成暂不做，订单/退款仅按既有数据结构做流程与统计。 - 存储需要接入本地 MinIO 进行真实 Provider 模拟，保留本地 FS 作为兜底。 +- 当前测试只使用本地存储；S3 存储测试后续独立进行。 - 多租户路由强隔离（`/t/:tenantCode/v1` + TenantResolver）已启用，后续仅做细节优化。 ## 统一原则 @@ -331,16 +332,21 @@ ## P3（延后） -### 16) 真实存储 Provider 接入（生产） +### 16) 真实存储 Provider 接入（生产）（已完成） **需求目标** - 接入 OSS/云存储（生产环境），统一上传/访问路径策略。 **技术方案（后端）** - 通过配置注入 Provider，保留本地 FS/MinIO 作为 dev fallback。 -- 进度：已补齐 S3 配置示例与 `CheckOnBoot` 可选自检开关。 +- 已完成： + - `backend/config.prod.toml` 配置生产 MinIO（`quyun-01` bucket） + - `CheckOnBoot` 启动时自检连通性 + - 完整 E2E 测试验证：InitUpload → UploadPart → CompleteUpload → 签名URL访问 → Delete **测试方案** -- 本地 FS + MinIO + 真实 Provider 三套配置可用性。 +- ✅ 本地 FS 配置可用（开发默认） +- ✅ MinIO S3 Provider 配置可用（`config.prod.toml`） +- ✅ 上传/访问/删除完整链路验证通过 ### 17) 媒体处理管线适配对象存储（S3/MinIO） **需求目标** @@ -386,6 +392,7 @@ - 超管后台治理能力（健康度/异常监控/内容审核）。 - 性能优化（避免 N+1：订单/租户列表批量聚合 + topics 聚合）。 - 多租户强隔离（/t/:tenantCode/v1 + TenantResolver）。 +- 真实存储 Provider 接入（生产 MinIO S3 配置与 E2E 验证）。 ## 里程碑建议 - M1：完成 P0