7 Commits

Author SHA1 Message Date
9a57949147 Fix upstream timeout for large downloads
Some checks failed
docker-release / build-and-push (push) Failing after 25m53s
2025-12-12 17:36:37 +08:00
3685b2129a fix: update content type handling for release.gpg 2025-11-18 16:13:05 +08:00
fc2c46a9df Refactor module binding to rely on Type 2025-11-18 16:11:13 +08:00
347eb3adc5 feat: remove module/rollout config key 2025-11-18 15:37:21 +08:00
dcd85a9f41 fix: apt cache
Some checks failed
docker-release / build-and-push (push) Failing after 9m58s
2025-11-18 14:21:12 +08:00
ba5544c28d feat: support apk 2025-11-18 12:16:28 +08:00
68b6bb78e6 add oc 2025-11-18 09:55:07 +08:00
61 changed files with 2053 additions and 575 deletions

View File

@@ -10,6 +10,8 @@ Auto-generated from all feature plans. Last updated: 2025-11-13
- 本地文件系统缓存目录 `StoragePath/<Hub>/<path>`,模块需直接复用原始路径布局 (004-modular-proxy-cache)
- 本地文件系统缓存目录 `StoragePath/<Hub>/<path>`(按模块定义的布局) (005-proxy-module-delegation)
- 本地文件系统缓存目录 `StoragePath/<Hub>/<path>`(由模块 Hook 定义布局) (006-module-hook-refactor)
- Go 1.25+ (静态链接,单二进制) + Fiber v3HTTP 服务、Viper配置、Logrus + Lumberjack、标准库 `net/http`/`io` (007-apt-apk-cache)
- 本地 `StoragePath/<Hub>/<path>` + `.meta`;索引需带 Last-Modified/ETag 元信息;包体按原路径落盘 (007-apt-apk-cache)
- Go 1.25+ (静态链接,单二进制交付) + Fiber v3HTTP 服务、Viper配置、Logrus + Lumberjack结构化日志 [EXTRACTED FROM ALL PLAN.MD FILES] 滚动)、标准库 `net/http`/`io` (001-config-bootstrap)
@@ -29,9 +31,9 @@ tests/
Go 1.25+ (静态链接,单二进制交付): Follow standard conventions
## Recent Changes
- 007-apt-apk-cache: Added Go 1.25+ (静态链接,单二进制) + Fiber v3HTTP 服务、Viper配置、Logrus + Lumberjack、标准库 `net/http`/`io`
- 006-module-hook-refactor: Added Go 1.25+ (静态链接,单二进制交付) + Fiber v3HTTP 服务、Viper配置、Logrus + Lumberjack结构化日志 & 滚动)、标准库 `net/http`/`io`
- 005-proxy-module-delegation: Added Go 1.25+ (静态链接,单二进制交付) + Fiber v3HTTP 服务、Viper配置、Logrus + Lumberjack结构化日志 & 滚动)、标准库 `net/http`/`io`
- 004-modular-proxy-cache: Added Go 1.25+ (静态链接,单二进制交付) + Fiber v3HTTP 服务、Viper配置、Logrus + Lumberjack结构化日志 & 滚动)、标准库 `net/http`/`io`
<!-- MANUAL ADDITIONS START -->

View File

@@ -38,7 +38,7 @@ Password = "s3cr3t"
1. 复制 `configs/config.example.toml` 为工作目录下的 `config.toml` 并调整 `[[Hub]]` 配置:
- 在全局段添加/修改 `ListenPort`,并从每个 Hub 中移除 `Port`
- 为 Hub 填写 `Type`并按需添加 `Module`(缺省为 `legacy`,自定义模块需在 `internal/hubmodule/<module-key>/` 注册)
- 为 Hub 填写 `Type`any-hub 会根据类型挑选对应模块 Hook如需扩展模块需在 `internal/hubmodule/<type>/` 注册并将新的类型纳入配置校验
- 根据 quickstart 示例设置 `Domain``Upstream``StoragePath` 等字段,并按需添加 `Username`/`Password`
2. 参考 [`specs/003-hub-auth-fields/quickstart.md`](specs/003-hub-auth-fields/quickstart.md) 完成配置校验、凭证验证与日志检查。
3. 常用命令:
@@ -48,20 +48,20 @@ Password = "s3cr3t"
## 模块化代理与示例
- `configs/config.example.toml` 展示了多个 Hub 的组合Docker Hub省略 `Module`,自动使用 `Type` 同名 Hook、Composer Hub显式指定 `Module = "composer"`)以及 legacy 兜底 Hub可直接复制修改
- 运行 `./scripts/demo-proxy.sh docker`(或 `npm`)即可加载示例配置并启动代理,日志中会附带 `module_key` 字段,便于确认命中的是 `legacy` 还是自定义模块。
- `configs/config.example.toml` 展示了多个 Hub 的组合Docker/NPM/Composer 等类型均自动绑定同名模块,仅需设置 `Type` 与上游信息即可启动
- 运行 `./scripts/demo-proxy.sh docker`(或 `npm`)即可加载示例配置并启动代理,日志中会附带 `module_key` 字段,便于确认命中的是 `docker``npm`模块。
- Hook 开发流程:
1. 复制 `internal/hubmodule/template/``internal/hubmodule/<module-key>/`,补全 `module.go``module_test.go`
2. 在模块 `init()` 中调用 `hubmodule.MustRegister` 注册 metadata并使用 `hooks.MustRegister` 注册 HookNormalizePath/ResolveUpstream/RewriteResponse 等)。
3. 为模块补充单元测试、`tests/integration/` 覆盖 miss→hit 流程,运行 `make modules-test`/`go test ./...`
4. 更新配置:`[[Hub]].Module` 留空,将根据 `Type` 自动选择 Hook也可显式设置 `Module = "<module-key>"` 并通过 `Rollout` 控制 legacy/dual/modular
4. 更新配置:为新的模块挑选一个唯一的 `Type`需要同步到配置校验列表Hub 只需填写该 `Type` 即可路由至新模块
5. 启动服务前,可通过 `curl -s /-/modules | jq '.hook_registry'` 确认 hook 注册情况;缺失时启动会直接失败,避免运行期回退到 legacy。
### 模块选择与 legacy
- `[[Hub]].Module` 为空时会自动回退到与 `Type` 同名的模块(若已注册),否则使用 `legacy` 兜底。
- diagnostics `/-/modules` 将展示 `hook_status`,当模块仍使用 legacy 时会标记 `legacy-only`,便于排查
- legacy 模块仅提供最小兜底能力,迁移完成后应显式将 `Module` 设置为对应仓库Hook。
- 示例操作手册、常见问题参见 [`specs/003-hub-auth-fields/quickstart.md`](specs/003-hub-auth-fields/quickstart.md) 以及本特性的 [`quickstart.md`](specs/004-modular-proxy-cache/quickstart.md)
### 模块选择
- Hub 的 `Type` 直接映射到同名模块;新增模块时需同步扩展 `internal/config/validation.go` 中的支持列表
- diagnostics `/-/modules` 仍会展示每个模块hook 注册状态与所有 Hub 绑定关系,便于排查配置错误
- legacy 模块仅作为历史兼容存在,不再通过配置字段触发
## CLI 标志

View File

@@ -19,7 +19,6 @@ Name = "docker"
Upstream = "https://registry-1.docker.io"
Proxy = ""
Type = "docker"
Module = "docker"
Username = ""
Password = ""
@@ -29,7 +28,6 @@ Name = "ghcr"
Upstream = "https://ghcr.io"
Proxy = ""
Type = "docker"
Module = "docker"
Username = ""
Password = ""
@@ -39,7 +37,6 @@ Name = "quay"
Upstream = "https://quay.io"
Proxy = ""
Type = "docker"
Module = "docker"
Username = ""
Password = ""
@@ -50,7 +47,6 @@ Name = "go"
Upstream = "https://proxy.golang.org"
Proxy = ""
Type = "go"
Module = "go"
Username = ""
Password = ""
@@ -61,7 +57,6 @@ Name = "npm"
Upstream = "https://registry.npmjs.org"
Proxy = ""
Type = "npm"
Module = "npm"
Username = ""
Password = ""
@@ -74,7 +69,6 @@ Proxy = ""
Username = ""
Password = ""
Type = "pypi"
Module = "pypi"
# Composer Repository
[[Hub]]
@@ -85,4 +79,17 @@ Proxy = ""
Username = ""
Password = ""
Type = "composer"
Module = "composer"
# Debian/Ubuntu APT 示例
[[Hub]]
Name = "apt-cache"
Domain = "apt.hub.local"
Upstream = "https://mirrors.edge.kernel.org/ubuntu"
Type = "debian"
# Alpine APK 示例
[[Hub]]
Name = "apk-cache"
Domain = "apk.hub.local"
Upstream = "https://dl-cdn.alpinelinux.org/alpine"
Type = "apk"

View File

@@ -17,9 +17,7 @@ Name = "docker-cache"
Domain = "docker.hub.local"
Upstream = "https://registry-1.docker.io"
Proxy = ""
Type = "docker" # 省略 Module 时自动选择与 Type 同名的 Hook此处为 docker
# Module = "docker" # 如需明确指定,可取消注释
Rollout = "modular"
Type = "docker"
Username = ""
Password = ""
CacheTTL = 43200
@@ -30,8 +28,6 @@ Name = "composer-cache"
Domain = "composer.hub.local"
Upstream = "https://repo.packagist.org"
Type = "composer"
Module = "composer" # 显式绑定 composer Hook启动时会验证 hook 是否已注册
Rollout = "dual" # 可选legacy-only/dual/modular
CacheTTL = 21600
[[Hub]]
@@ -39,5 +35,17 @@ Name = "legacy-fallback"
Domain = "legacy.hub.local"
Upstream = "https://registry.npmjs.org"
Type = "npm"
Module = "legacy" # 仍未迁移的 Hub 可显式指定 legacy诊断会标记为 legacy-only
Rollout = "legacy-only"
# Debian/Ubuntu APT 示例
[[Hub]]
Name = "apt-cache"
Domain = "apt.hub.local"
Upstream = "https://mirrors.edge.kernel.org/ubuntu"
Type = "debian"
# Alpine APK 示例
[[Hub]]
Name = "apk-cache"
Domain = "apk.hub.local"
Upstream = "https://dl-cdn.alpinelinux.org/alpine"
Type = "apk"

View File

@@ -0,0 +1,21 @@
# Logging & Cache Semantics (APT/APK)
## Common Fields
- `hub`/`domain`/`hub_type`:当前 Hub 标识与协议类型,例如 `debian`/`apk`
- `module_key`:命中的模块键(与 `Type` 同名)。
- `cache_hit``true` 表示直接复用缓存;`false` 表示从上游获取或已刷新。
- `upstream`/`upstream_status`:实际访问的上游地址与状态码。
- `action``proxy`,表明代理链路日志。
## APT (debian 模块)
- 索引路径Release/InRelease/Packages*`cache_hit=true` 仍会在后台进行 HEAD 再验证;命中 304 时保持缓存。
- 包体路径(`/pool/*``/dists/.../by-hash/...`):视为不可变,首次 GET 落盘,后续直接命中,无 HEAD。
- 日志可结合 `X-Any-Hub-Cache-Hit` 响应头进行对照。
## APK (apk 模块)
- APKINDEX 及签名:每次命中会触发 HEAD 再验证,缓存命中返回 304 时继续使用本地文件。
- 包体 (`packages/*.apk`):不可变资源,首轮 GET 落盘,后续直接命中,无 HEAD。
## Quick Checks
- 观察 `cache_hit``upstream_status``cache_hit=true``upstream_status=200/304` 表示缓存复用成功;`cache_hit=false` 表示回源或刷新。
-`module_key` 与配置的 `Type` 不符,检查该类型的 hook 是否已注册,或是否误用了旧版二进制。

View File

@@ -1,58 +1,33 @@
# Modular Hub Migration Playbook
# Module Binding Notes
This playbook describes how to cut a hub over from the shared legacy adapter to a dedicated module using the new rollout flags, diagnostics endpoint, and structured logs delivered in feature `004-modular-proxy-cache`.
Legacy rollout flags (`Module`/`Rollout`) have been removed. Hubs now bind to modules solely through their `Type` values, which map 1:1 to registered modules (docker, npm, go, pypi, composer, debian, apk, ...).
## Prerequisites
## Migrating to a New Module
- Target module must be registered via `hubmodule.MustRegister` and expose a proxy handler through `proxy.RegisterModuleHandler`.
- `config.toml` must already map the hub to its target module through `[[Hub]].Module`.
- Operators must have access to the running binary port (default `:5000`) to query `/-/modules`.
1. **Register the module**
Implement the new module under `internal/hubmodule/<type>/`, call `hubmodule.MustRegister` in `init()`, and register hooks via `hooks.MustRegister`.
## Rollout Workflow
2. **Expose a handler**
New modules continue to reuse the shared proxy handler registered via `proxy.RegisterModule`. No per-module handler wiring is required unless the module supplies a bespoke handler.
1. **Snapshot current state**
Run `curl -s http://localhost:5000/-/modules | jq '.hubs[] | select(.hub_name=="<hub>")'` to capture the current `module_key` and `rollout_flag`. Legacy hubs report `module_key=legacy` and `rollout_flag=legacy-only`.
2. **Prepare config for dual traffic**
Edit the hub block to target the new module while keeping rollback safety:
```toml
[[Hub]]
Name = "npm-prod"
Domain = "npm.example.com"
Upstream = "https://registry.npmjs.org"
Module = "npm"
Rollout = "dual"
```
Dual mode keeps routing on the new module but keeps observability tagged as a partial rollout.
3. **Deploy and monitor**
Restart the service and tail logs filtered by `module_key`:
```sh
jq 'select(.module_key=="npm" and .rollout_flag=="dual")' /var/log/any-hub.json
```
Every request now carries `module_key`/`rollout_flag`, allowing dashboards or `grep`-based analyses without extra parsing.
3. **Update the config schema**
Add the new type to `internal/config/validation.go`s `supportedHubTypes`, then redeploy. Every hub that should use the new module only needs `Type = "<type>"` plus the usual `Domain`/`Upstream` fields.
4. **Verify diagnostics**
Query `/-/modules/npm` to inspect the registered metadata and confirm cache strategy, or `/-/modules` to ensure the hub binding reflects `rollout_flag=dual`.
`curl http://127.0.0.1:<port>/-/modules` to ensure the new type appears under `modules[]` and that the desired hubs show `module_key="<type>"`.
5. **Promote to modular**
Once metrics are healthy, change `Rollout = "modular"` in config and redeploy. Continue monitoring logs to make sure both `module_key` and `rollout_flag` show the fully promoted state.
5. **Monitor logs**
Structured logs still carry `module_key`, making it easy to confirm that traffic is flowing through the expected module. Example:
6. **Rollback procedure**
To rollback, set `Rollout = "legacy-only"` (without touching `Module`). The runtime forces traffic through the legacy module while keeping the desired module declaration for later reattempts. Confirm via diagnostics (`module_key` reverts to `legacy`) before announcing rollback complete.
```json
{"action":"proxy","hub":"npm","module_key":"npm","cache_hit":false,"upstream_status":200}
```
## Observability Checklist
- **Logs**: Every proxy log line now contains `hub`, `module_key`, `rollout_flag`, upstream status, and `request_id`. Capture at least five minutes of traffic per flag change.
- **Diagnostics**: Store JSON snapshots from `/-/modules` before and after each rollout stage for incident timelines.
- **Config History**: Keep the `config.toml` diff (especially `Rollout` changes) attached to change records for auditability.
6. **Rollback**
Since modules are now type-driven, rollback is as simple as reverting the `Type` value (or config deployment) back to the previous modules type.
## Troubleshooting
- **Error: `module_not_found` during diagnostics** → module key not registered; ensure the module packages `init()` calls `hubmodule.MustRegister`.
- **Requests still tagged with `legacy-only` after promotion** → double-check the running process uses the updated config path (`ANY_HUB_CONFIG` vs `--config`) and restart the service.
- **Diagnostics 404** → confirm you are hitting the correct port and that the CLI user/network path allows HTTP access; the endpoint ignores Host headers, so `curl http://127.0.0.1:<port>/-/modules` should succeed locally.
- **`module_not_found` in diagnostics** → ensure the module registered via `hubmodule.MustRegister` before the hub references its type.
- **Hooks missing** → `/-/modules` exposes `hook_registry`; confirm the new type reports `registered`.
- **Unexpected module key in logs** → confirm the running binary includes your module (imported in `internal/config/modules.go`) and that the config `/--config` path matches the deployed file.

View File

@@ -5,7 +5,6 @@ import (
"path/filepath"
"reflect"
"strconv"
"strings"
"time"
"github.com/mitchellh/mapstructure"
@@ -89,24 +88,17 @@ func applyHubDefaults(h *HubConfig) {
if h.CacheTTL.DurationValue() < 0 {
h.CacheTTL = Duration(0)
}
if trimmed := strings.TrimSpace(h.Module); trimmed == "" {
typeKey := strings.ToLower(strings.TrimSpace(h.Type))
if meta, ok := hubmodule.Resolve(typeKey); ok {
h.Module = meta.Key
} else {
h.Module = hubmodule.DefaultModuleKey()
}
} else {
h.Module = strings.ToLower(trimmed)
}
if rollout := strings.TrimSpace(h.Rollout); rollout != "" {
h.Rollout = strings.ToLower(rollout)
}
if h.ValidationMode == "" {
h.ValidationMode = string(hubmodule.ValidationModeETag)
}
}
// NormalizeHubConfig 公开给无需依赖 loader 的调用方(例如测试)以应用 TTL/校验默认值。
func NormalizeHubConfig(h HubConfig) HubConfig {
applyHubDefaults(&h)
return h
}
func durationDecodeHook() mapstructure.DecodeHookFunc {
targetType := reflect.TypeOf(Duration(0))

View File

@@ -2,9 +2,11 @@ package config
import (
_ "github.com/any-hub/any-hub/internal/hubmodule/composer"
_ "github.com/any-hub/any-hub/internal/hubmodule/debian"
_ "github.com/any-hub/any-hub/internal/hubmodule/docker"
_ "github.com/any-hub/any-hub/internal/hubmodule/golang"
_ "github.com/any-hub/any-hub/internal/hubmodule/legacy"
_ "github.com/any-hub/any-hub/internal/hubmodule/apk"
_ "github.com/any-hub/any-hub/internal/hubmodule/npm"
_ "github.com/any-hub/any-hub/internal/hubmodule/pypi"
)

View File

@@ -4,7 +4,6 @@ import (
"time"
"github.com/any-hub/any-hub/internal/hubmodule"
"github.com/any-hub/any-hub/internal/hubmodule/legacy"
)
// HubRuntime 将 Hub 配置与模块元数据合并,方便运行时快速取用策略。
@@ -12,16 +11,14 @@ type HubRuntime struct {
Config HubConfig
Module hubmodule.ModuleMetadata
CacheStrategy hubmodule.CacheStrategyProfile
Rollout legacy.RolloutFlag
}
// BuildHubRuntime 根据 Hub 配置和模块元数据创建运行时描述,应用最终 TTL 覆盖。
func BuildHubRuntime(cfg HubConfig, meta hubmodule.ModuleMetadata, ttl time.Duration, flag legacy.RolloutFlag) HubRuntime {
func BuildHubRuntime(cfg HubConfig, meta hubmodule.ModuleMetadata, ttl time.Duration) HubRuntime {
strategy := hubmodule.ResolveStrategy(meta, cfg.StrategyOverrides(ttl))
return HubRuntime{
Config: cfg,
Module: meta,
CacheStrategy: strategy,
Rollout: flag,
}
}

View File

@@ -1,62 +0,0 @@
package config
import (
"fmt"
"strings"
"github.com/any-hub/any-hub/internal/hubmodule"
"github.com/any-hub/any-hub/internal/hubmodule/legacy"
)
// parseRolloutFlag 将配置中的 rollout 字段标准化,并结合模块类型输出最终状态。
func parseRolloutFlag(raw string, moduleKey string) (legacy.RolloutFlag, error) {
normalized := strings.ToLower(strings.TrimSpace(raw))
if normalized == "" {
return defaultRolloutFlag(moduleKey), nil
}
switch normalized {
case string(legacy.RolloutLegacyOnly):
return legacy.RolloutLegacyOnly, nil
case string(legacy.RolloutDual):
if moduleKey == hubmodule.DefaultModuleKey() {
return legacy.RolloutLegacyOnly, nil
}
return legacy.RolloutDual, nil
case string(legacy.RolloutModular):
if moduleKey == hubmodule.DefaultModuleKey() {
return legacy.RolloutLegacyOnly, nil
}
return legacy.RolloutModular, nil
default:
return "", fmt.Errorf("不支持的 rollout 值: %s", raw)
}
}
func defaultRolloutFlag(moduleKey string) legacy.RolloutFlag {
if strings.TrimSpace(moduleKey) == "" || moduleKey == hubmodule.DefaultModuleKey() {
return legacy.RolloutLegacyOnly
}
return legacy.RolloutModular
}
// EffectiveModuleKey 根据 rollout 状态计算真实运行的模块。
func EffectiveModuleKey(moduleKey string, flag legacy.RolloutFlag) string {
if flag == legacy.RolloutLegacyOnly {
return hubmodule.DefaultModuleKey()
}
normalized := strings.ToLower(strings.TrimSpace(moduleKey))
if normalized == "" {
return hubmodule.DefaultModuleKey()
}
return normalized
}
// RolloutFlagValue 返回当前 Hub 的 rollout flag假定 Validate 已经通过)。
func (h HubConfig) RolloutFlagValue() legacy.RolloutFlag {
flag := legacy.RolloutFlag(strings.ToLower(strings.TrimSpace(h.Rollout)))
if flag == "" {
return defaultRolloutFlag(h.Module)
}
return flag
}

View File

@@ -69,8 +69,6 @@ type HubConfig struct {
Upstream string `mapstructure:"Upstream"`
Proxy string `mapstructure:"Proxy"`
Type string `mapstructure:"Type"`
Module string `mapstructure:"Module"`
Rollout string `mapstructure:"Rollout"`
Username string `mapstructure:"Username"`
Password string `mapstructure:"Password"`
CacheTTL Duration `mapstructure:"CacheTTL"`

View File

@@ -16,9 +16,11 @@ var supportedHubTypes = map[string]struct{}{
"go": {},
"pypi": {},
"composer": {},
"debian": {},
"apk": {},
}
const supportedHubTypeList = "docker|npm|go|pypi|composer"
const supportedHubTypeList = "docker|npm|go|pypi|composer|debian|apk"
// Validate 针对语义级别做进一步校验,防止非法配置启动服务。
func (c *Config) Validate() error {
@@ -77,23 +79,9 @@ func (c *Config) Validate() error {
}
hub.Type = normalizedType
moduleKey := strings.ToLower(strings.TrimSpace(hub.Module))
if moduleKey == "" {
if _, ok := hubmodule.Resolve(normalizedType); ok && normalizedType != "" {
moduleKey = normalizedType
} else {
moduleKey = hubmodule.DefaultModuleKey()
}
if _, ok := hubmodule.Resolve(normalizedType); !ok {
return newFieldError(hubField(hub.Name, "Type"), fmt.Sprintf("未注册模块: %s", normalizedType))
}
if _, ok := hubmodule.Resolve(moduleKey); !ok {
return newFieldError(hubField(hub.Name, "Module"), fmt.Sprintf("未注册模块: %s", moduleKey))
}
hub.Module = moduleKey
flag, err := parseRolloutFlag(hub.Rollout, hub.Module)
if err != nil {
return newFieldError(hubField(hub.Name, "Rollout"), err.Error())
}
hub.Rollout = string(flag)
if hub.ValidationMode != "" {
mode := strings.ToLower(strings.TrimSpace(hub.ValidationMode))
switch mode {

View File

@@ -24,7 +24,7 @@ internal/hubmodule/
2. 填写模块特有逻辑与缓存策略,并确保包含中文注释解释设计。
3. 在模块目录添加 `module_test.go`,使用 `httptest.Server``t.TempDir()` 复现真实流量。
4. 运行 `make modules-test` 验证模块单元测试。
5. `[[Hub]].Module` 留空时会优先选择与 `Type` 同名的模块,实际迁移时仍建议显式填写,便于 diagnostics 标记 rollout
5. `[[Hub]].Type` 现已直接映射到同名模块;新增模块时记得将类型加入配置校验与示例配置
## 术语
- **Module Key**:模块唯一标识(如 `legacy``npm-tarball`)。

View File

@@ -0,0 +1,83 @@
// Package apk defines hook behaviors for Alpine APK proxying.
// APKINDEX/签名需要再验证packages/*.apk 视为不可变缓存。
package apk
import (
"path"
"strings"
"github.com/any-hub/any-hub/internal/proxy/hooks"
)
func init() {
hooks.MustRegister("apk", hooks.Hooks{
NormalizePath: normalizePath,
CachePolicy: cachePolicy,
ContentType: contentType,
})
}
func normalizePath(_ *hooks.RequestContext, p string, rawQuery []byte) (string, []byte) {
clean := path.Clean("/" + strings.TrimSpace(p))
return clean, rawQuery
}
func cachePolicy(_ *hooks.RequestContext, locatorPath string, current hooks.CachePolicy) hooks.CachePolicy {
clean := canonicalPath(locatorPath)
switch {
case isAPKIndexPath(clean), isAPKSignaturePath(clean):
// APKINDEX 及签名需要再验证,确保索引最新。
current.AllowCache = true
current.AllowStore = true
current.RequireRevalidate = true
case isAPKPackagePath(clean):
// 包体不可变,允许直接命中缓存,无需 HEAD。
current.AllowCache = true
current.AllowStore = true
current.RequireRevalidate = false
default:
current.AllowCache = false
current.AllowStore = false
current.RequireRevalidate = false
}
return current
}
func contentType(_ *hooks.RequestContext, locatorPath string) string {
clean := canonicalPath(locatorPath)
switch {
case strings.HasSuffix(clean, ".apk"):
return "application/vnd.android.package-archive"
case strings.HasSuffix(clean, ".tar.gz"):
return "application/gzip"
case strings.HasSuffix(clean, ".tar.gz.asc") || strings.HasSuffix(clean, ".tar.gz.sig"):
return "application/pgp-signature"
default:
return ""
}
}
func isAPKIndexPath(p string) bool {
clean := canonicalPath(p)
return strings.HasSuffix(clean, "/apkindex.tar.gz")
}
func isAPKSignaturePath(p string) bool {
clean := canonicalPath(p)
return strings.HasSuffix(clean, "/apkindex.tar.gz.asc") || strings.HasSuffix(clean, "/apkindex.tar.gz.sig")
}
func isAPKPackagePath(p string) bool {
clean := canonicalPath(p)
if isAPKIndexPath(clean) || isAPKSignaturePath(clean) {
return false
}
return strings.HasSuffix(clean, ".apk")
}
func canonicalPath(p string) string {
if p == "" {
return "/"
}
return strings.ToLower(path.Clean("/" + strings.TrimSpace(p)))
}

View File

@@ -0,0 +1,61 @@
package apk
import (
"testing"
"github.com/any-hub/any-hub/internal/proxy/hooks"
)
func TestCachePolicyIndexAndSignatureRevalidate(t *testing.T) {
paths := []string{
"/v3.19/main/x86_64/APKINDEX.tar.gz",
"/v3.19/main/x86_64/APKINDEX.tar.gz.asc",
"/v3.19/community/aarch64/apkindex.tar.gz.sig",
}
for _, p := range paths {
current := cachePolicy(nil, p, hooks.CachePolicy{})
if !current.AllowCache || !current.AllowStore || !current.RequireRevalidate {
t.Fatalf("expected index/signature to require revalidate for %s", p)
}
}
}
func TestCachePolicyPackageImmutable(t *testing.T) {
tests := []string{
"/v3.19/main/x86_64/packages/hello-1.0.apk",
"/v3.18/testing/aarch64/packages/../packages/hello-1.0-r1.APK",
"/v3.22/community/x86_64/tini-static-0.19.0-r3.apk", // 路径不含 /packages/ 也应视作包体
}
for _, p := range tests {
current := cachePolicy(nil, p, hooks.CachePolicy{})
if !current.AllowCache || !current.AllowStore || current.RequireRevalidate {
t.Fatalf("expected immutable cache for %s", p)
}
}
}
func TestCachePolicyNonAPKPath(t *testing.T) {
current := cachePolicy(nil, "/other/path", hooks.CachePolicy{})
if current.AllowCache || current.AllowStore || current.RequireRevalidate {
t.Fatalf("expected non-APK path to disable cache/store")
}
}
func TestNormalizePath(t *testing.T) {
p, _ := normalizePath(nil, "v3.19/main/x86_64/APKINDEX.tar.gz", nil)
if p != "/v3.19/main/x86_64/APKINDEX.tar.gz" {
t.Fatalf("unexpected normalized path: %s", p)
}
}
func TestContentType(t *testing.T) {
if ct := contentType(nil, "/v3.19/main/x86_64/APKINDEX.tar.gz"); ct != "application/gzip" {
t.Fatalf("expected gzip content type, got %s", ct)
}
if ct := contentType(nil, "/v3.19/main/x86_64/APKINDEX.tar.gz.asc"); ct != "application/pgp-signature" {
t.Fatalf("expected signature content type, got %s", ct)
}
if ct := contentType(nil, "/v3.19/main/x86_64/packages/hello.apk"); ct != "application/vnd.android.package-archive" {
t.Fatalf("expected apk content type, got %s", ct)
}
}

View File

@@ -0,0 +1,29 @@
// Package apk registers metadata for Alpine APK proxying.
package apk
import (
"time"
"github.com/any-hub/any-hub/internal/hubmodule"
)
const apkDefaultTTL = 6 * time.Hour
func init() {
// 模块元数据声明,具体 hooks 见 hooks.go已在 init 自动注册)。
hubmodule.MustRegister(hubmodule.ModuleMetadata{
Key: "apk",
Description: "Alpine APK proxy with cached indexes and packages",
MigrationState: hubmodule.MigrationStateBeta,
SupportedProtocols: []string{
"apk",
},
CacheStrategy: hubmodule.CacheStrategyProfile{
TTLHint: 0, // APKINDEX 每次再验证,包体直接命中
ValidationMode: hubmodule.ValidationModeLastModified, // APKINDEX 再验证
DiskLayout: "raw_path",
RequiresMetadataFile: false,
SupportsStreamingWrite: true, // 包体流式写
},
})
}

View File

@@ -0,0 +1,76 @@
// Package debian defines hook behaviors for APT (Debian/Ubuntu) proxying.
// 索引Release/InRelease/Packages*需要再验证包体pool/ 和 by-hash视为不可变直接缓存。
// 日志字段沿用通用 proxy命中/上游状态),无需额外改写。
package debian
import (
"path"
"strings"
"github.com/any-hub/any-hub/internal/proxy/hooks"
)
func init() {
hooks.MustRegister("debian", hooks.Hooks{
NormalizePath: normalizePath,
CachePolicy: cachePolicy,
ContentType: contentType,
})
}
func normalizePath(_ *hooks.RequestContext, p string, rawQuery []byte) (string, []byte) {
clean := path.Clean("/" + strings.TrimSpace(p))
return clean, rawQuery
}
func cachePolicy(_ *hooks.RequestContext, locatorPath string, current hooks.CachePolicy) hooks.CachePolicy {
clean := canonicalPath(locatorPath)
if strings.Contains(clean, "/by-hash/") || strings.Contains(clean, "/pool/") {
// pool/*.deb 与 by-hash 路径视为不可变,直接缓存后续不再 HEAD。
current.AllowCache = true
current.AllowStore = true
current.RequireRevalidate = false
return current
}
if strings.Contains(clean, "/dists/") {
// 索引类Release/Packages/Contents需要 If-None-Match/If-Modified-Since 再验证。
if strings.HasSuffix(clean, "/release") ||
strings.HasSuffix(clean, "/inrelease") ||
strings.HasSuffix(clean, "/release.gpg") {
current.AllowCache = true
current.AllowStore = true
current.RequireRevalidate = true
return current
}
}
current.AllowCache = false
current.AllowStore = false
current.RequireRevalidate = false
return current
}
func contentType(_ *hooks.RequestContext, locatorPath string) string {
clean := canonicalPath(locatorPath)
switch {
case strings.HasSuffix(clean, ".gz"):
return "application/gzip"
case strings.HasSuffix(clean, ".xz"):
return "application/x-xz"
case strings.HasSuffix(clean, "release.gpg"):
return "application/pgp-signature"
case strings.Contains(clean, "/dists/") &&
(strings.HasSuffix(clean, "/release") || strings.HasSuffix(clean, "/inrelease") || strings.HasSuffix(clean, "/release.gpg")):
return "text/plain"
default:
return ""
}
}
func canonicalPath(p string) string {
if p == "" {
return "/"
}
return strings.ToLower(path.Clean("/" + strings.TrimSpace(p)))
}

View File

@@ -0,0 +1,73 @@
package debian
import (
"testing"
"github.com/any-hub/any-hub/internal/proxy/hooks"
)
func TestCachePolicyIndexesRevalidate(t *testing.T) {
current := cachePolicy(nil, "/dists/bookworm/Release", hooks.CachePolicy{})
if !current.AllowCache || !current.AllowStore || !current.RequireRevalidate {
t.Fatalf("expected index to allow cache/store and revalidate")
}
current = cachePolicy(nil, "/dists/bookworm/main/binary-amd64/Packages.gz", hooks.CachePolicy{})
if !current.AllowCache || !current.AllowStore || !current.RequireRevalidate {
t.Fatalf("expected packages index to revalidate")
}
current = cachePolicy(nil, "/dists/bookworm/main/Contents-amd64.gz", hooks.CachePolicy{})
if !current.AllowCache || !current.AllowStore || !current.RequireRevalidate {
t.Fatalf("expected contents index to revalidate")
}
current = cachePolicy(nil, "/debian-security/dists/trixie/Contents-amd64.gz", hooks.CachePolicy{})
if !current.AllowCache || !current.AllowStore || !current.RequireRevalidate {
t.Fatalf("expected prefixed contents index to revalidate")
}
}
func TestCachePolicyImmutable(t *testing.T) {
tests := []struct {
name string
path string
}{
{name: "by-hash index snapshot", path: "/dists/bookworm/by-hash/sha256/abc"},
{name: "by-hash nested", path: "/dists/bookworm/main/binary-amd64/by-hash/SHA256/def"},
{name: "pool package", path: "/pool/main/h/hello.deb"},
{name: "pool canonicalized", path: " /PoOl/main/../main/h/hello_1.0_amd64.DeB "},
{name: "mirror prefix pool", path: "/debian/pool/main/h/hello.deb"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
current := cachePolicy(nil, tt.path, hooks.CachePolicy{})
if !current.AllowCache || !current.AllowStore || current.RequireRevalidate {
t.Fatalf("expected immutable cache for %s", tt.path)
}
})
}
}
func TestCachePolicyNonAptPath(t *testing.T) {
current := cachePolicy(nil, "/other/path", hooks.CachePolicy{})
if current.AllowCache || current.AllowStore || current.RequireRevalidate {
t.Fatalf("expected non-APT path to disable cache/store")
}
}
func TestNormalizePath(t *testing.T) {
path, _ := normalizePath(nil, "dists/bookworm/Release", nil)
if path != "/dists/bookworm/Release" {
t.Fatalf("unexpected path: %s", path)
}
}
func TestContentType(t *testing.T) {
if ct := contentType(nil, "/dists/bookworm/Release"); ct != "text/plain" {
t.Fatalf("expected text/plain for Release, got %s", ct)
}
if ct := contentType(nil, "/dists/bookworm/Release.gpg"); ct != "application/pgp-signature" {
t.Fatalf("expected signature content-type, got %s", ct)
}
if ct := contentType(nil, "/dists/bookworm/main/binary-amd64/Packages.gz"); ct != "application/gzip" {
t.Fatalf("expected gzip content-type, got %s", ct)
}
}

View File

@@ -0,0 +1,29 @@
// Package debian registers metadata for Debian/Ubuntu APT proxying.
package debian
import (
"time"
"github.com/any-hub/any-hub/internal/hubmodule"
)
const debianDefaultTTL = 6 * time.Hour
func init() {
// 仅声明模块元数据(缓存策略等);具体 hooks 在后续实现。
hubmodule.MustRegister(hubmodule.ModuleMetadata{
Key: "debian",
Description: "APT proxy with cached indexes and packages",
MigrationState: hubmodule.MigrationStateBeta,
SupportedProtocols: []string{
"debian",
},
CacheStrategy: hubmodule.CacheStrategyProfile{
TTLHint: 0, // 索引每次再验证,由 ETag/Last-Modified 控制
ValidationMode: hubmodule.ValidationModeLastModified, // 索引使用 Last-Modified/ETag 再验证
DiskLayout: "raw_path", // 复用通用原始路径布局
RequiresMetadataFile: false,
SupportsStreamingWrite: true, // 包体需要流式写入,避免大文件占用内存
},
})
}

View File

@@ -15,9 +15,6 @@ func init() {
}
func normalizePath(ctx *hooks.RequestContext, clean string, rawQuery []byte) (string, []byte) {
if !isDockerHubHost(ctx.UpstreamHost) {
return clean, rawQuery
}
repo, rest, ok := splitDockerRepoPath(clean)
if !ok || repo == "" || strings.Contains(repo, "/") || repo == "library" {
return clean, rawQuery

View File

@@ -19,7 +19,7 @@ func init() {
"docker",
},
CacheStrategy: hubmodule.CacheStrategyProfile{
TTLHint: dockerDefaultTTL,
TTLHint: 0, // manifests 需每次再验证,由 ETag 控制新鲜度
ValidationMode: hubmodule.ValidationModeETag,
DiskLayout: "raw_path",
RequiresMetadataFile: false,

View File

@@ -1,65 +0,0 @@
package legacy
import (
"sort"
"strings"
"sync"
)
// RolloutFlag 描述 legacy 模块迁移阶段。
type RolloutFlag string
const (
RolloutLegacyOnly RolloutFlag = "legacy-only"
RolloutDual RolloutFlag = "dual"
RolloutModular RolloutFlag = "modular"
)
// AdapterState 记录特定 Hub 在 legacy 适配器中的运行状态。
type AdapterState struct {
HubName string
ModuleKey string
Rollout RolloutFlag
}
var (
stateMu sync.RWMutex
state = make(map[string]AdapterState)
)
// RecordAdapterState 更新指定 Hub 的 rollout 状态,供诊断端和日志使用。
func RecordAdapterState(hubName, moduleKey string, flag RolloutFlag) {
if hubName == "" {
return
}
key := strings.ToLower(hubName)
stateMu.Lock()
state[key] = AdapterState{
HubName: hubName,
ModuleKey: moduleKey,
Rollout: flag,
}
stateMu.Unlock()
}
// SnapshotAdapterStates 返回所有 Hub 的 rollout 状态,按名称排序。
func SnapshotAdapterStates() []AdapterState {
stateMu.RLock()
defer stateMu.RUnlock()
if len(state) == 0 {
return nil
}
keys := make([]string, 0, len(state))
for k := range state {
keys = append(keys, k)
}
sort.Strings(keys)
result := make([]AdapterState, 0, len(keys))
for _, key := range keys {
result = append(result, state[key])
}
return result
}

View File

@@ -19,7 +19,7 @@ func init() {
"pypi",
},
CacheStrategy: hubmodule.CacheStrategyProfile{
TTLHint: pypiDefaultTTL,
TTLHint: 0, // simple index 每次再验证
ValidationMode: hubmodule.ValidationModeLastModified,
DiskLayout: "raw_path",
RequiresMetadataFile: false,

View File

@@ -11,7 +11,7 @@ type StrategyOptions struct {
// ResolveStrategy 将模块的默认策略与 hub 级覆盖合并。
func ResolveStrategy(meta ModuleMetadata, opts StrategyOptions) CacheStrategyProfile {
strategy := meta.CacheStrategy
if opts.TTLOverride > 0 {
if strategy.TTLHint > 0 && opts.TTLOverride > 0 {
strategy.TTLHint = opts.TTLOverride
}
if opts.ValidationOverride != "" {

View File

@@ -11,15 +11,13 @@ func BaseFields(action, configPath string) logrus.Fields {
}
// RequestFields 提供 hub/domain/命中状态字段,供代理请求日志复用。
func RequestFields(hub, domain, hubType, authMode, moduleKey, rolloutFlag string, cacheHit bool, legacyOnly bool) logrus.Fields {
func RequestFields(hub, domain, hubType, authMode, moduleKey string, cacheHit bool) logrus.Fields {
return logrus.Fields{
"hub": hub,
"domain": domain,
"hub_type": hubType,
"auth_mode": authMode,
"cache_hit": cacheHit,
"legacy_only": legacyOnly,
"module_key": moduleKey,
"rollout_flag": rolloutFlag,
"hub": hub,
"domain": domain,
"hub_type": hubType,
"auth_mode": authMode,
"cache_hit": cacheHit,
"module_key": moduleKey,
}
}

View File

@@ -8,7 +8,6 @@ import (
"github.com/gofiber/fiber/v3"
"github.com/sirupsen/logrus"
"github.com/any-hub/any-hub/internal/hubmodule"
"github.com/any-hub/any-hub/internal/logging"
"github.com/any-hub/any-hub/internal/server"
)
@@ -36,7 +35,7 @@ func RegisterModuleHandler(key string, handler server.ProxyHandler) {
MustRegisterModule(ModuleRegistration{Key: key, Handler: handler})
}
// Handle 实现 server.ProxyHandler根据 route.ModuleKey 选择 handler。
// Handle 实现 server.ProxyHandler根据 route.Module.Key 选择 handler。
func (f *Forwarder) Handle(c fiber.Ctx, route *server.HubRoute) error {
requestID := server.RequestID(c)
handler := f.lookup(route)
@@ -91,7 +90,7 @@ func (f *Forwarder) logModuleError(route *server.HubRoute, code string, err erro
func (f *Forwarder) lookup(route *server.HubRoute) server.ProxyHandler {
if route != nil {
if handler := lookupModuleHandler(route.ModuleKey); handler != nil {
if handler := lookupModuleHandler(route.Module.Key); handler != nil {
return handler
}
}
@@ -132,10 +131,8 @@ func (f *Forwarder) routeFields(route *server.HubRoute, requestID string) logrus
route.Config.Domain,
route.Config.Type,
route.Config.AuthMode(),
route.ModuleKey,
string(route.RolloutFlag),
route.Module.Key,
false,
route.ModuleKey == hubmodule.DefaultModuleKey(),
)
if requestID != "" {
fields["request_id"] = requestID

View File

@@ -10,7 +10,7 @@ import (
"github.com/valyala/fasthttp"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/hubmodule/legacy"
"github.com/any-hub/any-hub/internal/hubmodule"
"github.com/any-hub/any-hub/internal/server"
)
@@ -103,7 +103,8 @@ func testRouteWithModule(moduleKey string) *server.HubRoute {
Domain: "test.local",
Type: "custom",
},
ModuleKey: moduleKey,
RolloutFlag: legacy.RolloutModular,
Module: hubmodule.ModuleMetadata{
Key: moduleKey,
},
}
}

View File

@@ -65,8 +65,7 @@ func buildHookContext(route *server.HubRoute, c fiber.Ctx) *hooks.RequestContext
HubName: route.Config.Name,
Domain: route.Config.Domain,
HubType: route.Config.Type,
ModuleKey: route.ModuleKey,
RolloutFlag: string(route.RolloutFlag),
ModuleKey: route.Module.Key,
UpstreamHost: baseHost,
Method: c.Method(),
}
@@ -84,7 +83,7 @@ func hasHook(def hooks.Hooks) bool {
func (h *Handler) Handle(c fiber.Ctx, route *server.HubRoute) error {
started := time.Now()
requestID := server.RequestID(c)
hooksDef, ok := hooks.Fetch(route.ModuleKey)
hooksDef, ok := hooks.Fetch(route.Module.Key)
hookCtx := buildHookContext(route, c)
rawQuery := append([]byte(nil), c.Request().URI().QueryString()...)
cleanPath := normalizeRequestPath(route, string(c.Request().URI().Path()))
@@ -121,7 +120,7 @@ func (h *Handler) Handle(c fiber.Ctx, route *server.HubRoute) error {
// miss, continue
default:
h.logger.WithError(err).
WithFields(logrus.Fields{"hub": route.Config.Name, "module_key": route.ModuleKey}).
WithFields(logrus.Fields{"hub": route.Config.Name, "module_key": route.Module.Key}).
Warn("cache_get_failed")
}
}
@@ -135,7 +134,7 @@ func (h *Handler) Handle(c fiber.Ctx, route *server.HubRoute) error {
fresh, err := h.isCacheFresh(c, route, locator, cached.Entry, &hookState)
if err != nil {
h.logger.WithError(err).
WithFields(logrus.Fields{"hub": route.Config.Name, "module_key": route.ModuleKey}).
WithFields(logrus.Fields{"hub": route.Config.Name, "module_key": route.Module.Key}).
Warn("cache_revalidate_failed")
serve = false
} else if !fresh {
@@ -203,6 +202,12 @@ func (h *Handler) serveCache(
c.Status(status)
if method == http.MethodHead {
// 对 HEAD 请求仍向上游发起一次 HEAD以满足“显式请求 + 再验证”的期望。
if route != nil && route.UpstreamURL != nil {
if resp, err := h.revalidateRequest(c, route, resolveUpstreamURL(route, route.UpstreamURL, c, hook), result.Entry.Locator, ""); err == nil {
resp.Body.Close()
}
}
result.Reader.Close()
h.logResult(route, route.UpstreamURL.String(), requestID, status, true, started, nil)
return nil
@@ -358,6 +363,7 @@ func (h *Handler) cacheAndStream(
}
c.Status(resp.StatusCode)
// 使用 TeeReader 边向客户端回写边落盘,避免大文件在内存中完整缓冲。
reader := io.TeeReader(resp.Body, c.Response().BodyWriter())
opts := cache.PutOptions{ModTime: extractModTime(resp.Header)}
@@ -511,10 +517,8 @@ func (h *Handler) logResult(
route.Config.Domain,
route.Config.Type,
route.Config.AuthMode(),
route.ModuleKey,
string(route.RolloutFlag),
route.Module.Key,
cacheHit,
route.ModuleKey == hubmodule.DefaultModuleKey(),
)
fields["action"] = "proxy"
fields["upstream"] = upstream
@@ -733,7 +737,7 @@ func determineCachePolicyWithHook(route *server.HubRoute, locator cache.Locator,
}
func determineCachePolicy(route *server.HubRoute, locator cache.Locator, method string) cachePolicy {
if method != http.MethodGet {
if method != http.MethodGet && method != http.MethodHead {
return cachePolicy{}
}
return cachePolicy{allowCache: true, allowStore: true}
@@ -786,9 +790,12 @@ func (h *Handler) isCacheFresh(
case http.StatusNotModified:
return true, nil
case http.StatusOK:
if resp.Header.Get("Etag") == "" && resp.Header.Get("Docker-Content-Digest") == "" && resp.Header.Get("Last-Modified") == "" {
return true, nil
}
h.rememberETag(route, locator, resp)
remote := extractModTime(resp.Header)
if !remote.After(entry.ModTime.Add(time.Second)) {
if !remote.After(entry.ModTime) {
return true, nil
}
return false, nil
@@ -961,10 +968,8 @@ func (h *Handler) logAuthRetry(route *server.HubRoute, upstream string, requestI
route.Config.Domain,
route.Config.Type,
route.Config.AuthMode(),
route.ModuleKey,
string(route.RolloutFlag),
route.Module.Key,
false,
route.ModuleKey == hubmodule.DefaultModuleKey(),
)
fields["action"] = "proxy_retry"
fields["upstream"] = upstream
@@ -982,10 +987,8 @@ func (h *Handler) logAuthFailure(route *server.HubRoute, upstream string, reques
route.Config.Domain,
route.Config.Type,
route.Config.AuthMode(),
route.ModuleKey,
string(route.RolloutFlag),
route.Module.Key,
false,
route.ModuleKey == hubmodule.DefaultModuleKey(),
)
fields["action"] = "proxy"
fields["upstream"] = upstream

View File

@@ -13,7 +13,6 @@ type RequestContext struct {
Domain string
HubType string
ModuleKey string
RolloutFlag string
UpstreamHost string
Method string
}

View File

@@ -31,9 +31,12 @@ func NewUpstreamClient(cfg *config.Config) *http.Client {
timeout = cfg.Global.UpstreamTimeout.DurationValue()
}
transport := defaultTransport.Clone()
// Use UpstreamTimeout as ResponseHeaderTimeout to avoid killing long streaming downloads.
transport.ResponseHeaderTimeout = timeout
return &http.Client{
Timeout: timeout,
Transport: defaultTransport.Clone(),
Transport: transport,
}
}

View File

@@ -16,8 +16,12 @@ func TestNewUpstreamClientUsesConfigTimeout(t *testing.T) {
}
client := NewUpstreamClient(cfg)
if client.Timeout != 45*time.Second {
t.Fatalf("expected timeout 45s, got %s", client.Timeout)
transport, ok := client.Transport.(*http.Transport)
if !ok {
t.Fatalf("expected *http.Transport, got %T", client.Transport)
}
if transport.ResponseHeaderTimeout != 45*time.Second {
t.Fatalf("expected response header timeout 45s, got %s", transport.ResponseHeaderTimeout)
}
}

View File

@@ -11,7 +11,6 @@ import (
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/hubmodule"
"github.com/any-hub/any-hub/internal/hubmodule/legacy"
)
// HubRoute 将 Hub 配置与派生属性(如缓存 TTL、解析后的 Upstream/Proxy URL
@@ -26,13 +25,10 @@ type HubRoute struct {
// UpstreamURL/ProxyURL 在构造 Registry 时提前解析完成,便于后续请求快速复用。
UpstreamURL *url.URL
ProxyURL *url.URL
// ModuleKey/Module 记录当前 hub 选用的模块及其元数据,便于日志与观测。
ModuleKey string
Module hubmodule.ModuleMetadata
// Module 记录当前 hub 选用的模块元数据,便于日志与观测。
Module hubmodule.ModuleMetadata
// CacheStrategy 代表模块默认策略与 hub 覆盖后的最终结果。
CacheStrategy hubmodule.CacheStrategyProfile
// RolloutFlag 反映当前 hub 的 legacy → modular 迁移状态,供日志/诊断使用。
RolloutFlag legacy.RolloutFlag
}
// HubRegistry 提供 Host/Host:port 到 HubRoute 的查询能力,所有 Hub 共享同一个监听端口。
@@ -56,6 +52,7 @@ func NewHubRegistry(cfg *config.Config) (*HubRegistry, error) {
}
for _, hub := range cfg.Hubs {
hub = config.NormalizeHubConfig(hub)
normalizedHost := normalizeDomain(hub.Domain)
if normalizedHost == "" {
return nil, fmt.Errorf("invalid domain for hub %s", hub.Name)
@@ -105,9 +102,11 @@ func (r *HubRegistry) List() []HubRoute {
}
func buildHubRoute(cfg *config.Config, hub config.HubConfig) (*HubRoute, error) {
flag := hub.RolloutFlagValue()
effectiveKey := config.EffectiveModuleKey(hub.Module, flag)
meta, err := moduleMetadataForKey(effectiveKey)
moduleKey := strings.ToLower(strings.TrimSpace(hub.Type))
if moduleKey == "" {
return nil, fmt.Errorf("hub %s: 缺少 Type", hub.Name)
}
meta, err := moduleMetadataForKey(moduleKey)
if err != nil {
return nil, fmt.Errorf("hub %s: %w", hub.Name, err)
}
@@ -126,8 +125,7 @@ func buildHubRoute(cfg *config.Config, hub config.HubConfig) (*HubRoute, error)
}
effectiveTTL := cfg.EffectiveCacheTTL(hub)
runtime := config.BuildHubRuntime(hub, meta, effectiveTTL, flag)
legacy.RecordAdapterState(hub.Name, runtime.Module.Key, flag)
runtime := config.BuildHubRuntime(hub, meta, effectiveTTL)
return &HubRoute{
Config: hub,
@@ -135,10 +133,8 @@ func buildHubRoute(cfg *config.Config, hub config.HubConfig) (*HubRoute, error)
CacheTTL: effectiveTTL,
UpstreamURL: upstreamURL,
ProxyURL: proxyURL,
ModuleKey: runtime.Module.Key,
Module: runtime.Module,
CacheStrategy: runtime.CacheStrategy,
RolloutFlag: runtime.Rollout,
}, nil
}

View File

@@ -5,7 +5,6 @@ import (
"time"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/hubmodule/legacy"
)
func TestHubRegistryLookupByHost(t *testing.T) {
@@ -49,14 +48,14 @@ func TestHubRegistryLookupByHost(t *testing.T) {
if route.CacheTTL != cfg.EffectiveCacheTTL(route.Config) {
t.Errorf("cache ttl mismatch: got %s", route.CacheTTL)
}
if route.CacheStrategy.TTLHint != route.CacheTTL {
t.Errorf("cache strategy ttl mismatch: %s vs %s", route.CacheStrategy.TTLHint, route.CacheTTL)
if route.CacheStrategy.TTLHint != 0 {
t.Errorf("cache strategy ttl mismatch: %s vs %s", route.CacheStrategy.TTLHint, time.Duration(0))
}
if route.CacheStrategy.ValidationMode == "" {
t.Fatalf("cache strategy validation mode should not be empty")
}
if route.RolloutFlag != legacy.RolloutLegacyOnly {
t.Fatalf("default rollout flag should be legacy-only")
if route.Module.Key != "docker" {
t.Fatalf("expected docker module, got %s", route.Module.Key)
}
if route.UpstreamURL.String() != "https://registry-1.docker.io" {

View File

@@ -65,8 +65,6 @@ type hubBindingPayload struct {
ModuleKey string `json:"module_key"`
Domain string `json:"domain"`
Port int `json:"port"`
Rollout string `json:"rollout_flag"`
Legacy bool `json:"legacy_only"`
}
func encodeModules(mods []hubmodule.ModuleMetadata, status map[string]string) []modulePayload {
@@ -115,11 +113,9 @@ func encodeHubBindings(routes []server.HubRoute) []hubBindingPayload {
for _, route := range routes {
result = append(result, hubBindingPayload{
HubName: route.Config.Name,
ModuleKey: route.ModuleKey,
ModuleKey: route.Module.Key,
Domain: route.Config.Domain,
Port: route.ListenPort,
Rollout: string(route.RolloutFlag),
Legacy: route.ModuleKey == hubmodule.DefaultModuleKey(),
})
}
return result

View File

@@ -15,7 +15,7 @@
- `MaxMemoryCacheSize` (bytes, optional, default 268435456)
- `MaxRetries` (int >=0, default 3)
- `InitialBackoff` (duration, default 1s)
- `UpstreamTimeout` (duration, default 30s)
- `UpstreamTimeout` (duration, default 30s, used as upstream response header timeout; body can stream longer)
- **Validation Rules**: 路径必须存在或可创建;数值必须 >0LogLevel 必须匹配允许枚举。
- **Relationships**: 被 `Config` 聚合并为 `HubConfig` 提供默认值。

View File

@@ -94,6 +94,3 @@ components:
type: string
port:
type: integer
rollout_flag:
type: string
enum: [legacy-only, dual, modular]

View File

@@ -13,13 +13,12 @@ The modular architecture introduces explicit metadata describing which proxy+cac
- `Domain` *(string, required)* hostname clients access; must be unique per process.
- `Port` *(int, required)* listen port; validated to 165535.
- `Upstream` *(string, required)* base URL for upstream registry; must be HTTPS or explicitly whitelisted HTTP.
- `Module` *(string, optional, default `"legacy"`)* key resolved through module registry. Validation ensures module exists at load time.
- `Type` *(string, required)* module selector; validated against the registered module set.
- `CacheTTL`, `Proxy`, and other overrides *(optional)* reuse existing schema; modules may read these via dependency injection.
- **Relationships**:
- `HubConfigEntry.Module``ModuleMetadata.Key` (many-to-one).
- `HubConfigEntry.Type``ModuleMetadata.Key` (many-to-one).
- **Validation Rules**:
- Missing `Module` implicitly maps to `legacy` to preserve backward compatibility.
- Changing `Module` requires a migration plan; config loader logs module name for observability.
- Invalid `Type` values reject config load; operators must add the type to the supported list alongside module registration.
### 2. ModuleMetadata
- **Fields**:
@@ -57,20 +56,12 @@ The modular architecture introduces explicit metadata describing which proxy+cac
- TTL must be positive.
- Modules flagged as `SupportsStreamingWrite=false` must document fallback behavior before registration.
### 5. LegacyAdapterState
- **Purpose**: Tracks which hubs still run through the old shared implementation to support progressive migration.
- **Fields**:
- `HubName` *(string)* references `HubConfigEntry.Name`.
- ` rolloutFlag` *(enum: `legacy-only`, `dual`, `modular`)* indicates traffic split for that hub.
- `FallbackDeadline` *(timestamp, optional)* when legacy path will be removed.
- **Storage**: In-memory map derived from config + environment flags; optionally surfaced via diagnostics endpoint.
## State Transitions
1. **Module Adoption**
- Start: `HubConfigEntry.Module = "legacy"`.
- Transition: operator edits config to new module key, runs validation.
- Result: registry resolves new module, `LegacyAdapterState` updated to `dual` until rollout flag toggled fully.
- Start: `HubConfigEntry.Type = "legacy"` (or other baseline).
- Transition: operator edits config to new module type, runs validation.
- Result: registry resolves new module and routes all traffic to it immediately.
2. **Cache Strategy Update**
- Start: Module uses default TTL.

View File

@@ -21,7 +21,7 @@ Modularize the proxy and cache layers so every hub type (npm, Docker, PyPI, futu
**Constraints**: 禁止 Web UI 或账号体系;所有行为受单一 TOML 配置控制;每个 Hub 需独立 Domain/Port 绑定;仅匿名访问
**Scale/Scope**: 支撑 Docker/NPM/Go/PyPI 等多仓代理,面向弱网及离线缓存复用场景
**Module Registry Location**: `internal/hubmodule/registry.go` 暴露注册/解析 API模块子目录位于 `internal/hubmodule/<name>/`
**Config Binding for Modules**: `[[Hub]].Module` 字段控制模块名,默认 `legacy`,配置加载阶段校验必须命中已注册模块
**Config Binding for Modules**: `[[Hub]].Type` 字段控制模块名(同名映射),配置加载阶段校验类型对应的模块已注册
## Constitution Check
@@ -29,7 +29,7 @@ Modularize the proxy and cache layers so every hub type (npm, Docker, PyPI, futu
- Feature 仍然是“轻量多仓 CLI 代理”,未引入 Web UI、账号体系或与代理无关的能力。
- 仅使用 Go + 宪法指定依赖;任何新第三方库都已在本计划中说明理由与审核结论。
- 行为完全由 `config.toml` 控制,新增 `[[Hub]].Module` 配置项已规划默认值、校验与迁移策略
- 行为完全由 `config.toml` 控制,`[[Hub]].Type` 直接驱动模块绑定,校验列表随模块扩展更新
- 方案维持缓存优先 + 流式回源路径,并给出命中/回源/失败的日志与观测手段。
- 计划内列出了配置解析、缓存读写、Host Header 路由等强制测试与中文注释交付范围。
@@ -103,14 +103,14 @@ tests/ # `go test` 下的单元/集成测试,用临时目
### Post-Design Constitution Check
- New diagnostics endpoint remains internal and optional; no UI/login introduced. ✅ Principle I
- Code still single Go binary with existing dependency set. ✅ Principle II
- `Module` field documented with defaults, validation, and migration path; no extra config sources. ✅ Principle III
- `Type` 字段即模块绑定点,文档与校验同步更新;无额外配置源。 ✅ Principle III
- Cache strategy enforces“原始路径 == 磁盘路径”的布局与流式回源,相关观测需求写入 contracts。✅ Principle IV
- Logs/quickstart/test guidance ensure observability and Chinese documentation continue. ✅ Principle V
## Phase 2 Implementation Outlook (pre-tasks)
1. **Module Registry & Interfaces**: Create `internal/hubmodule` package, define shared interfaces, implement registry with tests, and expose diagnostics data source reused by HTTP endpoints.
2. **Config Loader & Validation**: Extend `internal/config/types.go` and `validation.go` to include `Module` with default `legacy`, plus wiring to registry resolution during startup.
2. **Config Loader & Validation**: Extend `internal/config/types.go` and `validation.go` to bind modules via `Type`, plus wiring to registry resolution during startup.
3. **Legacy Adapter & Migration Switches**: Provide adapter module that wraps current shared proxy/cache, plus feature flags or config toggles to control rollout states per hub.
4. **Module Implementations**: Carve existing npm/docker/pypi logic into dedicated modules within `internal/hubmodule/`, ensuring cache writer复用原始请求路径与必要的 telemetry 标签。
5. **Observability/Diagnostics**: Implement `//modules` endpoint (Fiber route) and log tags showing `module_key` on cache/proxy events.

View File

@@ -12,20 +12,20 @@
4. Add tests under the module directory and run `make modules-test` (delegates to `go test ./internal/hubmodule/...`).
## 3. Bind Module via Config
1. Edit `config.toml` and set `Module = "<module-key>"` inside the target `[[Hub]]` block (omit to use `legacy`).
2. While validating a new module, set `Rollout = "dual"` so you can flip back to legacy without editing other fields.
1. Add your module type to `internal/config/validation.go` and the sample config if it represents a new protocol.
2. Edit `config.toml` and set `Type = "<module-type>"` inside the target `[[Hub]]` block.
3. (Optional) Override cache behavior per hub using existing fields (`CacheTTL`, etc.).
4. Run `ANY_HUB_CONFIG=./config.toml go test ./...` (or `make modules-test`) to ensure loader validation passes and the module registry sees your key.
## 4. Run and Verify
1. Start the binary: `go run ./cmd/any-hub --config ./config.toml`.
2. Use `curl -H "Host: <hub-domain>" http://127.0.0.1:<port>/<path>` to produce traffic, then hit `curl http://127.0.0.1:<port>/-/modules` and confirm the hub binding points to your module with the expected `rollout_flag`.
2. Use `curl -H "Host: <hub-domain>" http://127.0.0.1:<port>/<path>` to produce traffic, then hit `curl http://127.0.0.1:<port>/-/modules` and confirm the hub binding points to your module key.
3. Inspect `./storage/<hub>/` to confirm the cached files mirror the upstream path (no suffix). When a path also has child entries (e.g., `/pkg` metadata plus `/pkg/-/...` tarballs), the metadata payload is stored in a `__content` file under that directory so both artifacts can coexist. PyPI Simple responses rewrite distribution links to `/files/<scheme>/<host>/<path>` so that wheels/tarballs are fetched through the proxy and cached alongside the HTML/JSON index. Verify TTL overrides are propagated.
4. Monitor `logs/any-hub.log` (or the sample `logs/module_migration_sample.log`) to verify each entry exposes `module_key` + `rollout_flag`. Example:
4. Monitor `logs/any-hub.log` (or the sample `logs/module_migration_sample.log`) to verify each entry exposes `module_key`. Example:
```json
{"action":"proxy","hub":"testhub","module_key":"testhub","rollout_flag":"dual","cache_hit":false,"upstream_status":200}
{"action":"proxy","hub":"testhub","module_key":"testhub","cache_hit":false,"upstream_status":200}
```
5. Exercise rollback by switching `Rollout = "legacy-only"` (or `Module = "legacy"` if needed) and re-running the traffic to ensure diagnostics/logs show the transition.
5. Exercise rollback by reverting the config change (or type rename) and re-running the traffic to ensure diagnostics/logs show the transition.
## 5. Ship
1. Commit module code + config docs.

View File

@@ -15,7 +15,7 @@
## Phase 2: Foundational (Blocking Prerequisites)
- [X] T003 Create shared module interfaces + registry in `internal/hubmodule/interfaces.go` and `internal/hubmodule/registry.go`
- [X] T004 Extend config schema with `[[Hub]].Module` defaults/validation plus sample configs in `internal/config/{types.go,validation.go,loader.go}` and `configs/*.toml`
- [X] T004 Extend config schema with module defaults/validation2025-03 起由 `Type` 直接驱动,`[[Hub]].Module` 已淘汰)
- [X] T005 [P] Wire server bootstrap to resolve modules once and inject into proxy/cache layers (`internal/server/bootstrap.go`, `internal/proxy/handler.go`)
**Checkpoint**: Registry + config plumbing complete; user story work may begin.
@@ -61,6 +61,8 @@
## Phase 5: User Story 3 - Operate Mixed Generations During Migration (Priority: P3)
> 2025-03: Rollout flags were removed; this section remains for historical tracking only.
**Goal**: Support dual-path deployments with diagnostics/logging to track legacy vs. modular hubs.
**Independent Test**: Run mixed legacy/modular hubs, flip rollout flags, and confirm logs + diagnostics show module ownership and allow rollback.
@@ -73,7 +75,7 @@
- [X] T019 [US3] Implement `LegacyAdapterState` tracker + rollout flag parsing (`internal/hubmodule/legacy/state.go`, `internal/config/runtime_flags.go`)
- [X] T020 [US3] Implement Fiber handler + routing for `//modules` diagnostics (`internal/server/routes/modules.go`, `internal/server/router.go`)
- [X] T021 [US3] Add structured log fields (`module_key`, `rollout_flag`) across logging middleware (`internal/server/middleware/logging.go`, `internal/proxy/logging.go`)
- [X] T021 [US3] Add structured log fields (`module_key`) across logging middleware (`internal/server/middleware/logging.go`, `internal/proxy/logging.go`)
- [X] T022 [US3] Document operational playbook for phased migration (`docs/operations/migration.md`)
---

View File

@@ -9,5 +9,5 @@ If future external endpoints are added, document them here with request/response
## Error Behaviors
- **module_handler_missing**: Forwarder无法找到给定 module_key 的 handler 时返回 `500 {"error":"module_handler_missing"}`,并记录 `hub/domain/module_key/rollout_flag` 等日志字段,便于排查配置缺失或注册遗漏。
- **module_handler_missing**: Forwarder无法找到给定 module_key 的 handler 时返回 `500 {"error":"module_handler_missing"}`,并记录 `hub/domain/module_key` 等日志字段,便于排查配置缺失或注册遗漏。
- **module_handler_panic**: Module handler panic 被捕获后返回 `500 {"error":"module_handler_panic"}`,同时输出结构化日志 `error=module_handler_panic`,防止进程崩溃并提供观测。

View File

@@ -14,7 +14,7 @@
- **Proxy Dispatcher**
- Attributes: handler map (module_key → handler), default handler fallback.
- Behavior: Lookup by route.ModuleKey and invoke handler; wraps errors/logging.
- Behavior: Lookup by the hub's module key (derived from Type / route.Module.Key) and invoke handler; wraps errors/logging.
- Constraints: If handler missing, return 5xx with observable logging.
- **Cache Policy**

View File

@@ -27,7 +27,7 @@
## Relationships
- Hub module 注册时同时在 HookRegistry 与 Forwarder handler map 建立关联。
- ProxyDispatcher 在请求进入后根据 route.ModuleKey 查询 Hook + handler。
- ProxyDispatcher 在请求进入后根据 route.Module.Key(来自 Hub Type查询 Hook + handler。
- Diagnostics 依赖 HookRegistry 与 HubRegistry 联合输出状态。
## Lifecycle

View File

@@ -40,5 +40,5 @@ curl -s http://localhost:8080/-/modules | jq '.modules[].hook_status'
```
- 确认新模块标记为 `registered`,未注册模块显示 `missing`legacy handler 仍可作为兜底。
- 如果需要查看全局状态,可检查 `hook_registry` 字段,它返回每个 module_key 的注册情况。
- `hubs[].legacy_only``true` 时表示该 Hub 仍绑定 legacy 模块;迁移完成后应显式设置 `[[Hub]].Module`
- `hubs[].module_key` 应与配置中的 `Type` 对齐legacy 模块仅作为兜底存在,推荐尽快替换为协议专用模块
- 启动阶段会验证每个模块是否注册 Hook缺失则直接退出避免运行期静默回退。

View File

@@ -0,0 +1,34 @@
# Specification Quality Checklist: APT/APK 包缓存模块
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2025-11-17
**Feature**: /home/rogee/Projects/any-hub/specs/007-apt-apk-cache/spec.md
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Items marked complete; ready to proceed to planning.

View File

@@ -0,0 +1,26 @@
# Proxy Path Contracts: APT/APK
## APT
| Downstream Request | Upstream Target | Caching | Notes |
|--------------------|-----------------|---------|-------|
| `/dists/<suite>/<component>/binary-<arch>/Packages[.gz/.xz]` | same path on Upstream | RequireRevalidate=true | Send If-None-Match/If-Modified-Since if cached |
| `/dists/<suite>/Release` | same path | RequireRevalidate=true | Preserve content; may have accompanying `.gpg` |
| `/dists/<suite>/InRelease` | same path | RequireRevalidate=true | Signed inline; no body changes |
| `/dists/<suite>/Release.gpg` | same path | RequireRevalidate=true | Signature only; no rewrite |
| `/pool/<vendor>/<name>.deb` | same path | AllowCache/AllowStore=true, RequireRevalidate=false | Treat as immutable |
| `/dists/<suite>/by-hash/<algo>/<hash>` | same path | AllowCache/AllowStore=true, RequireRevalidate=false | Path encodes hash, no extra validation |
## Alpine APK
| Downstream Request | Upstream Target | Caching | Notes |
|--------------------|-----------------|---------|-------|
| `/v3.<branch>/main/<arch>/APKINDEX.tar.gz` (and variants) | same path | RequireRevalidate=true | Preserve signature/headers |
| `/v3.<branch>/<repo>/<arch>/APKINDEX.tar.gz` | same path | RequireRevalidate=true | Same handling as above |
| `/v3.<branch>/<repo>/<arch>/packages/<file>.apk` | same path | AllowCache/AllowStore=true, RequireRevalidate=false | Immutable package |
| APKINDEX signature files | same path | RequireRevalidate=true | No rewrite |
## Behaviors
- No body/URL rewrite; proxy only normalizes cache policy per path shape.
- Preserve status codes, headers, and signature files exactly as upstream.
- Conditional requests (If-None-Match/If-Modified-Since) applied to all index/signature paths when cached.

View File

@@ -0,0 +1,36 @@
# Data Model: APT/APK 包缓存模块
## Entities
### HubConfig (APT/APK)
- Fields: `Name`, `Domain`, `Port`, `Upstream`, `Type` (`debian`/`apk`), optional `CacheTTL` override.
- Rules: `Type` 必须新增枚举;`Upstream` 必须为 HTTP/HTTPS每个 Hub 独立缓存前缀。
### IndexFile
- Represents: APT Release/InRelease/Packages*Alpine APKINDEX。
- Attributes: `Path`, `ETag` (if present), `Last-Modified`, `Hash` (if provided in index), `Size`, `ContentType`.
- Rules: RequireRevalidate=true缓存命中需携带条件请求内容不得修改。
### PackageFile
- Represents: APT `pool/*/*.deb` 包体Alpine `packages/<arch>/*.apk`
- Attributes: `Path`, `Size`, `Hash` (from upstream metadata if available), `StoredAt`.
- Rules: 视作不可变AllowCache/AllowStore=true不做本地内容改写。
### SignatureFile
- Represents: APT `Release.gpg`/`InRelease` 自带签名Alpine APKINDEX 签名文件。
- Attributes: `Path`, `Size`, `StoredAt`.
- Rules: 原样透传;与对应 IndexFile 绑定,同步缓存与再验证。
### CacheEntry
- Represents: 本地缓存记录(索引或包体)。
- Attributes: `LocatorPath`, `ModTime`, `ETag`, `Size`, `AllowStore`, `RequireRevalidate`.
- Rules: 读取时决定是否回源;写入需同步 `.meta`;路径格式 `StoragePath/<Hub>/<Path>`.
## Relationships
- HubConfig → IndexFile/PackageFile/SignatureFile 通过 Domain/Upstream 绑定。
- IndexFile ↔ SignatureFile同一索引文件的签名文件需同周期缓存与再验证。
- CacheEntry 聚合 IndexFile/PackageFile/SignatureFile 的落盘信息,用于策略判定。
## State/Transitions
- CacheEntry states: `miss``fetched``validated` (304) / `refreshed` (200) → `stale` (TTL 或上游 200 新内容) → `removed` (404 或清理)。
- Transitions triggered by upstream响应304 保持内容200 刷新404 删除相关缓存。

View File

@@ -0,0 +1,94 @@
# Implementation Plan: APT/APK 包缓存模块
**Branch**: `007-apt-apk-cache` | **Date**: 2025-11-17 | **Spec**: `/home/rogee/Projects/any-hub/specs/007-apt-apk-cache/spec.md`
**Input**: Feature specification from `/specs/007-apt-apk-cache/spec.md`
## Summary
为 any-hub 增加 APTDebian/Ubuntu与 Alpine APK 的缓存代理模块索引Release/InRelease/Packages*/APKINDEX必须再验证包体pool/*.deb、packages/*)视作不可变直接缓存,支持 Acquire-By-Hash/签名透传,不影响现有各模块。
## Technical Context
**Language/Version**: Go 1.25+ (静态链接,单二进制)
**Primary Dependencies**: Fiber v3HTTP 服务、Viper配置、Logrus + Lumberjack、标准库 `net/http`/`io`
**Storage**: 本地 `StoragePath/<Hub>/<path>` + `.meta`;索引需带 Last-Modified/ETag 元信息;包体按原路径落盘
**Protocols**: APT (`/dists/<suite>/<component>/binary-<arch>/Packages*`, `Release`, `InRelease`, `pool/*`), Acquire-By-HashAlpine (`APKINDEX.tar.gz`, `packages/<arch>/…`)
**Caching Rules**: 索引 RequireRevalidate=true包体 AllowStore/AllowCache=true、RequireRevalidate=false签名/校验文件原样透传
**Testing**: `go test ./...`,构造 httptest 上游返回 Release/Packages/APKINDEX 与包体,验证首次回源+再验证+命中;使用临时目录校验缓存落盘
**Target Platform**: Linux/Unix CLIsystemd/supervisor 托管,匿名客户端
**Constraints**: 单一 config.toml 控制;禁止新增第三方依赖;保持现有模块行为不变
**Risk/Unknowns**: Acquire-By-Hash 路径映射是否需额外校验逻辑(选择原样透传,路径即校验)
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
- Feature 仍然是“轻量多仓 CLI 代理”,未引入 Web UI、账号体系或与代理无关的能力。
- 仅使用 Go + 宪法指定依赖;任何新第三方库都已在本计划中说明理由与审核结论。
- 行为完全由 `config.toml` 控制,新增配置项已规划默认值、校验与迁移策略。
- 方案维持缓存优先 + 流式回源路径,并给出命中/回源/失败的日志与观测手段。
- 计划内列出了配置解析、缓存读写、Host Header 路由等强制测试与中文注释交付范围。
**Status**: PASS未引入新依赖/界面,新增配置仅增加模块枚举与示例)
## Project Structure
### Documentation (this feature)
```text
specs/007-apt-apk-cache/
├── plan.md # 本文件
├── research.md # Phase 0 输出
├── data-model.md # Phase 1 输出
├── quickstart.md # Phase 1 输出
├── contracts/ # Phase 1 输出
└── tasks.md # Phase 2 (/speckit.tasks 生成)
```
### Source Code (repository root)
```text
cmd/any-hub/main.go # CLI 入口、参数解析
internal/config/ # TOML 加载、默认值、校验
internal/server/ # Fiber 服务、路由、中间件
internal/cache/ # 磁盘/内存缓存与 .meta 管理
internal/proxy/ # 上游访问、缓存策略、流式复制
internal/hubmodule/* # 各模块 Hooks/元数据(新增 debian/apk 模块)
configs/ # 示例 config.toml如需
tests/ # go test 下的单元/集成测试
```
**Structure Decision**: 新增模块位于 `internal/hubmodule/debian` 或类似命名;复用现有 hook 注册与缓存策略接口;配置扩展在 `internal/config` 追加枚举与默认值。
## Complexity Tracking
> **Fill ONLY if Constitution Check has violations that must be justified**
| Violation | Why Needed | Simpler Alternative Rejected Because |
|-----------|------------|-------------------------------------|
| None | — | — |
## Phase 0: Outline & Research
### Unknowns / Research Tasks
- Acquire-By-Hash 处理策略:是否需要额外校验逻辑 → 选择透传路径并依赖上游校验。
- Alpine APKINDEX 签名校验需求 → 方案为透传签名文件,保持客户端校验。
### Research Output
- `/home/rogee/Projects/any-hub/specs/007-apt-apk-cache/research.md`
## Phase 1: Design & Contracts
### Artifacts
- `/home/rogee/Projects/any-hub/specs/007-apt-apk-cache/data-model.md`
- `/home/rogee/Projects/any-hub/specs/007-apt-apk-cache/contracts/proxy-paths.md`
- `/home/rogee/Projects/any-hub/specs/007-apt-apk-cache/quickstart.md`
- Agent context updated via `.specify/scripts/bash/update-agent-context.sh codex`
### Design Focus
- 定义 APT/APK 路径匹配与缓存策略(索引再验证、包体直接缓存)。
- Acquire-By-Hash/签名文件透传,避免破坏校验。
- 示例配置与测试入口httptest 上游 + 临时缓存目录)。
## Phase 2: Task Breakdown (deferred to /speckit.tasks)
- 基于用户故事和设计生成 tasks.md后续命令

View File

@@ -0,0 +1,42 @@
# Quickstart: APT/APK 缓存代理
## 1) 启用新 Hub
`config.toml` 增加示例:
```toml
[[Hub]]
Domain = "apt.hub.local"
Name = "apt"
Upstream = "https://mirrors.edge.kernel.org/ubuntu"
Type = "debian"
Module = "debian" # 待实现模块键
[[Hub]]
Domain = "apk.hub.local"
Name = "apk"
Upstream = "https://dl-cdn.alpinelinux.org/alpine"
Type = "apk"
Module = "apk" # 待实现模块键
```
## 2) 指向代理
- APT`/etc/apt/sources.list` 中的官方域名替换为 `http://apt.hub.local:5000`(或全局 ListenPort保持原 suite/component 路径。
- APK`/etc/apk/repositories` 中写入 `http://apk.hub.local:5000/v3.19/main` 等路径(与全局 ListenPort 一致)。
## 3) 验证
```bash
# APT
apt-get update
apt-get install -y curl
apt-get install -y curl # 第二次安装应直接命中缓存pool/* 与 by-hash/* 不再 HEAD 再验证)
# Alpine
apk update
apk add curl
apk add curl # 第二次安装应直接命中缓存APKINDEX 再验证packages/*.apk 直接命中)
```
观察 `logs/` 输出:首次请求应为回源,二次请求命中缓存(索引可能返回 304。如上游不可达且缓存已有包体应继续命中缓存无缓存则透传错误。

View File

@@ -0,0 +1,18 @@
# Research Notes: APT/APK 包缓存模块
## Decision Records
### 1) Acquire-By-Hash 处理
- **Decision**: 路径原样透传,缓存按完整路径存储;不做额外本地哈希校验,交由上游与客户端校验。
- **Rationale**: APT 自带哈希校验,路径即校验信息;本地重复计算增加 CPU/IO 成本且风险与上游标准重复。
- **Alternatives**: 额外本地哈希校验并拒绝不匹配(增加复杂度、可能与上游行为不一致);跳过缓存(失去加速价值)。
### 2) APT 索引再验证策略
- **Decision**: Release/InRelease/Packages* 请求统一带 If-None-Match/If-Modified-Since缓存 RequireRevalidate=true命中 304 继续用缓存200 刷新。
- **Rationale**: 与现有代理模式一致确保“latest” 索引及时更新,避免 stale。
- **Alternatives**: 固定 TTL 不再验证(风险:索引过期);强制每次全量 GET浪费带宽
### 3) APKINDEX 签名处理
- **Decision**: APKINDEX 及其签名文件原样透传并缓存,索引 RequireRevalidate=true包体直接缓存。
- **Rationale**: Alpine 客户端依靠签名文件校验,代理不应修改或剥离;再验证确保索引更新。
- **Alternatives**: 不缓存 APKINDEX失去加速效果仅缓存包体无法验证包版本更新

View File

@@ -0,0 +1,88 @@
# Feature Specification: APT/APK 包缓存模块
**Feature Branch**: `007-apt-apk-cache`
**Created**: 2025-11-17
**Status**: Draft
**Input**: User description: "Title: 设计并实现 APT/APK 包缓存模块以加速 Docker 构建 Goal: 为 any-hub 新增一个 debian/apk 模块,代理本地 Docker 构建中的 apt-get/apk add 请求,支持缓存索引与包文件并透明回源。 Constraints: - 适配 Debian/Ubuntu APT 路径Release/InRelease、Packages(.gz|.xz)、pool/*.deb及 Alpine APKINDEX/packages。 - 索引/Release 需 RequireRevalidate=true二进制包 AllowStore=true签名文件透传。 - 兼容 Acquire-By-HashAPT与 APKINDEX 签名校验,不破坏客户端校验。 - 复用现有缓存架构hooks.go + module.go + CacheStrategyProfile不得影响现有 docker/npm/pypi/go/composer 模块。 Inputs: - 代码库路径:/home/rogee/Projects/any-hub - 参考模块internal/hubmodule/docker/hooks.go (cachePolicy)、internal/hubmodule/golang/hooks.go、internal/proxy/handler.go (ETag/Last-Modified 再验证) Deliverables: - 新增 internal/hubmodule/debian或 alpine下的 hooks.go、module.go 实现 - 配套测试(模拟 apt-get update/install 或 apk update/add 的最小集) - 使用说明(简要配置示例,如何在 config.toml 添加 Hub AcceptanceCriteria: - “apt-get update” 或 “apk update” 指向代理时,首次 miss 回源,二次命中缓存;索引文件可被最新上游版本触发再验证;二进制包稳定命中缓存。 新分支开头数字是 007"
## User Scenarios & Testing *(mandatory)*
### User Story 1 - APT 更新通过代理 (Priority: P1)
本地构建镜像的开发者将 apt 源指向代理,运行 `apt-get update` 获取最新索引并希望后续重复构建无需再次访问官方仓。
**Why this priority**: APT 更新是后续安装的前提,需保证首次拉取成功且缓存生效以降低构建时间与外网依赖。
**Independent Test**: 将 apt 源指向代理后执行两次 `apt-get update`,验证首轮回源、次轮命中缓存且返回内容与官方一致。
**Acceptance Scenarios**:
1. **Given** 缓存为空,**When** 运行 `apt-get update` 指向代理,**Then** 返回官方索引内容且缓存写入成功Release/InRelease/Packages
2. **Given** 同一索引已缓存,**When** 再次运行 `apt-get update`**Then** 返回 200/304 且无额外上游 GET仅必要的条件请求输出内容与首轮一致。
---
### User Story 2 - APT 安装包命中缓存 (Priority: P2)
在构建步骤中安装常用 `.deb` 包,希望重复构建时直接从本地缓存提供 pool 下的包体。
**Why this priority**: 包文件体积大、下载耗时,命中缓存可显著缩短镜像构建时长。
**Independent Test**: 在完成 User Story 1 后,执行 `apt-get install <常见包>` 两次,确认首轮回源、次轮包体命中缓存,且校验/签名不受影响。
**Acceptance Scenarios**:
1. **Given** 缓存已有对应索引且无包体,**When** 首次安装某包,**Then** 包体成功下载并写入缓存,安装过程不因代理失败。
2. **Given** 同一包体已缓存,**When** 再次安装,**Then** 不触发上游包体下载,安装成功且校验通过。
---
### User Story 3 - Alpine APK 加速 (Priority: P3)
将 Alpine 利用代理执行 `apk update && apk add`,希望索引与包体同样缓存并可重复命中。
**Why this priority**: Alpine 镜像在容器场景常用,通过同一代理获得一致的构建加速。
**Independent Test**: 将 `/etc/apk/repositories` 指向代理,执行两轮 `apk update && apk add <常用包>`,验证首轮回源、次轮命中缓存且 APKINDEX/包体校验正常。
**Acceptance Scenarios**:
1. **Given** 代理配置完毕,**When** 首次执行 `apk update`**Then** 获取 APKINDEX 并写入缓存,随后 `apk add` 成功下载包体。
2. **Given** 同一索引与包体已缓存,**When** 再次执行 `apk update && apk add`**Then** 索引再验证后命中,包体直接命中缓存且安装成功。
---
### Edge Cases
- 上游索引更新Release/InRelease 或 APKINDEX 发生变更时,代理应通过条件请求检测到并刷新缓存。
- 校验失败Acquire-By-Hash 校验或 APKINDEX 签名不匹配时,应返回上游原始错误而非吞掉错误。
- 离线/超时:上游暂不可达时,若有缓存且策略允许,应优先返回缓存;无缓存则向客户端返回明确错误。
- 部分源未配置:多源场景下,某源缺失或路径拼写错误时,代理应返回与上游一致的 404/403。
- 大文件/磁盘不足:缓存写入包体时磁盘不足,应优雅失败并告警,不影响后续直接透传回源。
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: 代理必须支持 APT 获取与缓存 Release/InRelease、Packages(.gz/.xz) 及 pool 下的包体;索引请求需携带条件请求并可根据上游变更刷新缓存。
- **FR-002**: 代理必须支持 Acquire-By-Hash 路径,若哈希不匹配需返回上游一致的错误,不得用缓存污染返回。
- **FR-003**: 代理必须支持 Alpine APKINDEX 与 packages 路径的获取与缓存,索引需再验证,包体可直接命中缓存。
- **FR-004**: 首次命中失败的索引/包体要回源写入缓存;后续同一路径在缓存有效期内应直接命中,除非索引再验证判定有新版本。
- **FR-005**: 代理行为须保持透明:返回状态码、头信息、签名/校验文件与上游一致,日志需记录命中/回源/失败原因,便于构建诊断。
- **FR-006**: 新增 Hub 配置字段(如仓类型、上游地址)需有默认值与示例,且不得影响现有 docker/npm/pypi/go/composer 的配置与行为。
### Key Entities *(include if feature involves data)*
- **索引文件**APT 的 Release/InRelease/Packages*Alpine 的 APKINDEX含校验或签名信息需可再验证。
- **包体文件**`pool/*/*.deb`、Alpine `packages/*`;内容不可变,按路径/哈希缓存。
- **签名/校验文件**APT 的 Release.gpg、Acquire-By-Hash 路径;用于客户端校验,需原样透传。
- **缓存条目**:存储索引或包体的本地副本,包含写入时间与可选校验标识,用于命中与再验证判定。
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: 在同一构建机器上,第二次执行 `apt-get update``apk update` 对同一源时,上游请求数较首次减少 ≥90%(仅保留必要条件请求),且耗时降低至少 50%。
- **SC-002**: 已缓存的包体再次安装时,不触发上游下载,安装成功率达到 100%(以连续 20 次同包安装为样本)。
- **SC-003**: 当上游索引有新版本发布时,下一次索引请求能够在一次交互内获取最新数据(无旧数据残留),并保持客户端校验/签名通过率 100%。
- **SC-004**: 引入新模块后,现有 docker/npm/pypi/go/composer 代理的功能及配置使用无变化,回归用例全部通过。

View File

@@ -0,0 +1,119 @@
# Tasks: APT/APK 包缓存模块
**Input**: Design documents from `/specs/007-apt-apk-cache/`
**Prerequisites**: plan.md (required), spec.md (user stories), research.md, data-model.md, contracts/
## Format: `[ID] [P?] [Story] Description`
- **[P]**: Can run in parallel (different files, no dependencies)
- **[Story]**: User story label (US1, US2, US3)
- Include exact file paths in descriptions
## Phase 1: Setup (Shared Infrastructure)
**Purpose**: Prepare module folders and examples for new hubs.
- [X] T001 Create module directories `internal/hubmodule/debian/` and `internal/hubmodule/apk/` with placeholder go files (module.go/hooks.go scaffolds).
- [X] T002 Add sample hub entries for APT/APK in `configs/config.example.toml`.
---
## Phase 2: Foundational (Blocking Prerequisites)
**Purpose**: Core wiring for new hub types before story work.
- [X] T003 Update hub type validation to accept `debian``apk` in `internal/config/validation.go`.
- [X] T004 Register new modules in `internal/config/modules.go` and `internal/hubmodule/registry.go` (init side-effect includes debian/apk).
- [X] T005 [P] Define debian module metadata (cache strategy, TTL, validation mode) in `internal/hubmodule/debian/module.go`.
- [X] T006 [P] Define apk module metadata in `internal/hubmodule/apk/module.go`.
- [X] T007 Ensure path locator rewrite strategy (raw_path) reused for new modules in `internal/hubmodule/strategy.go` or module options if needed.
- [X] T008 Add constitution-mandated Chinese comments for new module metadata files.
**Checkpoint**: Foundation ready—new hub types recognized, modules load without runtime errors.
---
## Phase 3: User Story 1 - APT 更新通过代理 (Priority: P1) 🎯 MVP
**Goal**: 代理 APT 索引Release/InRelease/Packages*),首次回源,后续带条件请求再验证并命中缓存。
**Independent Test**: `apt-get update` 两次指向代理,首轮回源,次轮 304/命中缓存且内容与官方一致。
### Tests for User Story 1
- [X] T009 [P] [US1] Add unit tests for path classification与缓存策略索引 RequireRevalidate`internal/hubmodule/debian/hooks_test.go`.
- [X] T010 [US1] Add integration test `tests/integration/apt_update_proxy_test.go` covering first/second `apt-get update` (Release/InRelease/Packages) with httptest upstream and temp storage.
### Implementation for User Story 1
- [X] T011 [P] [US1] Implement APT hooks (NormalizePath/CachePolicy/ContentType/ResolveUpstream if needed) for index paths in `internal/hubmodule/debian/hooks.go`.
- [X] T012 [P] [US1] Support conditional requests (ETag/Last-Modified passthrough) for index responses in `internal/hubmodule/debian/hooks.go`.
- [X] T013 [US1] Wire debian module registration to use hooks in `internal/hubmodule/debian/module.go` and ensure hook registration in `hooks.go`.
- [X] T014 [US1] Ensure logging fields include cache hit/upstream for APT requests (reuse proxy logging) and document in comments `internal/hubmodule/debian/hooks.go`.
- [X] T015 [US1] Update quickstart instructions with APT usage validation steps in `specs/007-apt-apk-cache/quickstart.md`.
**Checkpoint**: APT 索引更新可独立验证并缓存。
---
## Phase 4: User Story 2 - APT 安装包命中缓存 (Priority: P2)
**Goal**: pool 下 `.deb` 包首次回源、后续直接命中缓存;保持 Acquire-By-Hash 路径透传且不污染哈希校验。
**Independent Test**: `apt-get install <包>` 两次,首轮下载并缓存,次轮无上游下载,安装成功且校验通过。
### Tests for User Story 2
- [X] T016 [P] [US2] Extend debian hook unit tests to cover `/pool/...``/by-hash/...` 缓存策略 in `internal/hubmodule/debian/hooks_test.go`.
- [X] T017 [US2] Integration test for package download caching and Acquire-By-Hash passthrough in `tests/integration/apt_package_proxy_test.go`.
### Implementation for User Story 2
- [X] T018 [P] [US2] Implement package/dist path handling (AllowCache/AllowStore, RequireRevalidate=false) in `internal/hubmodule/debian/hooks.go`.
- [X] T019 [P] [US2] Handle `/dists/<suite>/by-hash/<algo>/<hash>` as immutable cached resources in `internal/hubmodule/debian/hooks.go`.
- [X] T020 [US2] Validate cache writer/reader streaming for large deb files in `internal/proxy/handler.go` (ensure no full-buffer reads) with comments/tests if changes required.
- [X] T021 [US2] Update config docs/examples if additional APT-specific knobs are added in `configs/config.example.toml` or `README.md`.
**Checkpoint**: APT 包体可命中缓存且哈希/签名校验保持一致。
---
## Phase 5: User Story 3 - Alpine APK 加速 (Priority: P3)
**Goal**: 缓存 APKINDEX 并再验证包体packages/*.apk首次回源后直接命中缓存。
**Independent Test**: `apk update && apk add <包>` 两次,索引次轮 304/命中,包体次轮直接命中,安装成功。
### Tests for User Story 3
- [X] T022 [P] [US3] Add apk hook unit tests for index/package path policy in `internal/hubmodule/apk/hooks_test.go`.
- [X] T023 [US3] Integration test for apk update/install caching in `tests/integration/apk_proxy_test.go`.
### Implementation for User Story 3
- [X] T024 [P] [US3] Implement APK hooks (CachePolicy/ContentType/NormalizePath) for APKINDEX and packages in `internal/hubmodule/apk/hooks.go`.
- [X] T025 [P] [US3] Ensure APKINDEX/signature files RequireRevalidate and package files immutable cache in `internal/hubmodule/apk/hooks.go`.
- [X] T026 [US3] Register apk hooks in module init and update logging/observability comments in `internal/hubmodule/apk/module.go`.
- [X] T027 [US3] Add Alpine repository usage notes to `specs/007-apt-apk-cache/quickstart.md`.
**Checkpoint**: Alpine 索引与包体缓存可独立验证。
---
## Phase 6: Polish & Cross-Cutting Concerns
- [X] T028 [P] Add Chinese comments for key caching logic and path handling in new hook files (`internal/hubmodule/debian/hooks.go`, `internal/hubmodule/apk/hooks.go`).
- [X] T029 [P] Document log fields and cache semantics in `docs/` or `README.md` (structure log examples for APT/APK).
- [X] T030 Validate gofmt/go test ./... and update `specs/007-apt-apk-cache/quickstart.md` with final verification steps.
- [X] T031 [P] Confirm no regressions to existing modules via smoke test list in `tests/integration/` (reuse existing suites, adjust configs if needed).
---
## Dependencies & Execution Order
- Phase 1 → Phase 2 → User stories (Phase 3/4/5) → Phase 6.
- User Story 1 (P1) must complete before US2 (shares debian hooks); US3 can start after Phase 2 independently.
- Parallel opportunities:
- T005/T006 module metadata in parallel; hook/unit work can run in parallel within each story where marked [P].
- US3 tasks can run in parallel with US1 late-stage tasks once foundational ready (different modules/files).
- Suggested MVP: Complete Phases 1-3 (US1) to deliver APT 更新加速US2/US3 incrementally after validation.

View File

@@ -0,0 +1,344 @@
package integration
import (
"bytes"
"context"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"path"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/gofiber/fiber/v3"
"github.com/sirupsen/logrus"
"github.com/any-hub/any-hub/internal/cache"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/proxy"
"github.com/any-hub/any-hub/internal/server"
)
func TestAPKProxyCachesIndexAndPackages(t *testing.T) {
stub := newAPKStub(t)
defer stub.Close()
storageDir := t.TempDir()
cfg := &config.Config{
Global: config.GlobalConfig{
ListenPort: 5400,
CacheTTL: config.Duration(time.Hour),
StoragePath: storageDir,
},
Hubs: []config.HubConfig{
{
Name: "apk",
Domain: "apk.hub.local",
Type: "apk",
Upstream: stub.URL,
},
},
}
registry, err := server.NewHubRegistry(cfg)
if err != nil {
t.Fatalf("registry error: %v", err)
}
logger := logrus.New()
logger.SetOutput(io.Discard)
store, err := cache.NewStore(storageDir)
if err != nil {
t.Fatalf("store error: %v", err)
}
app, err := server.NewApp(server.AppOptions{
Logger: logger,
Registry: registry,
Proxy: proxy.NewHandler(server.NewUpstreamClient(cfg), logger, store),
ListenPort: 5400,
})
if err != nil {
t.Fatalf("app error: %v", err)
}
doRequest := func(p string) *http.Response {
req := httptest.NewRequest(http.MethodGet, "http://apk.hub.local"+p, nil)
req.Host = "apk.hub.local"
resp, err := app.Test(req)
if err != nil {
t.Fatalf("app.Test error: %v", err)
}
return resp
}
resp := doRequest(stub.indexPath)
if resp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for index, got %d", resp.StatusCode)
}
if hit := resp.Header.Get("X-Any-Hub-Cache-Hit"); hit != "false" {
t.Fatalf("expected cache miss on first index fetch, got %s", hit)
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if !bytes.Equal(body, stub.indexBody) {
t.Fatalf("index body mismatch on first fetch: %d bytes", len(body))
}
resp2 := doRequest(stub.indexPath)
if resp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached index, got %d", resp2.StatusCode)
}
if hit := resp2.Header.Get("X-Any-Hub-Cache-Hit"); hit != "true" {
t.Fatalf("expected cache hit for index, got %s", hit)
}
body2, _ := io.ReadAll(resp2.Body)
resp2.Body.Close()
if !bytes.Equal(body2, stub.indexBody) {
t.Fatalf("index body mismatch on cache hit: %d bytes", len(body2))
}
sigResp := doRequest(stub.signaturePath)
if sigResp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for signature, got %d", sigResp.StatusCode)
}
if hit := sigResp.Header.Get("X-Any-Hub-Cache-Hit"); hit != "false" {
t.Fatalf("expected cache miss on first signature fetch, got %s", hit)
}
_, _ = io.ReadAll(sigResp.Body)
sigResp.Body.Close()
sigResp2 := doRequest(stub.signaturePath)
if sigResp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached signature, got %d", sigResp2.StatusCode)
}
if hit := sigResp2.Header.Get("X-Any-Hub-Cache-Hit"); hit != "true" {
t.Fatalf("expected cache hit for signature, got %s", hit)
}
sigResp2.Body.Close()
pkgResp := doRequest(stub.packagePath)
if pkgResp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for package, got %d", pkgResp.StatusCode)
}
if hit := pkgResp.Header.Get("X-Any-Hub-Cache-Hit"); hit != "false" {
t.Fatalf("expected cache miss on first package fetch, got %s", hit)
}
pkgBody, _ := io.ReadAll(pkgResp.Body)
pkgResp.Body.Close()
if !bytes.Equal(pkgBody, stub.packageBody) {
t.Fatalf("package body mismatch on first fetch: %d bytes", len(pkgBody))
}
pkgResp2 := doRequest(stub.packagePath)
if pkgResp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached package, got %d", pkgResp2.StatusCode)
}
if hit := pkgResp2.Header.Get("X-Any-Hub-Cache-Hit"); hit != "true" {
t.Fatalf("expected cache hit for package, got %s", hit)
}
pkgBody2, _ := io.ReadAll(pkgResp2.Body)
pkgResp2.Body.Close()
if !bytes.Equal(pkgBody2, stub.packageBody) {
t.Fatalf("package body mismatch on cache hit: %d bytes", len(pkgBody2))
}
if stub.IndexGets() != 1 {
t.Fatalf("expected single index GET, got %d", stub.IndexGets())
}
if stub.IndexHeads() != 1 {
t.Fatalf("expected single index HEAD revalidate, got %d", stub.IndexHeads())
}
if stub.SignatureGets() != 1 {
t.Fatalf("expected single signature GET, got %d", stub.SignatureGets())
}
if stub.SignatureHeads() != 1 {
t.Fatalf("expected single signature HEAD revalidate, got %d", stub.SignatureHeads())
}
if stub.PackageGets() != 1 {
t.Fatalf("expected single package GET, got %d", stub.PackageGets())
}
if stub.PackageHeads() != 0 {
t.Fatalf("expected zero package HEAD revalidate, got %d", stub.PackageHeads())
}
verifyAPKStored(t, storageDir, "apk", stub.indexPath, int64(len(stub.indexBody)))
verifyAPKStored(t, storageDir, "apk", stub.signaturePath, int64(len(stub.signatureBody)))
verifyAPKStored(t, storageDir, "apk", stub.packagePath, int64(len(stub.packageBody)))
}
func verifyAPKStored(t *testing.T, basePath, hubName, locatorPath string, expectedSize int64) {
t.Helper()
clean := path.Clean("/" + locatorPath)
clean = strings.TrimPrefix(clean, "/")
fullPath := filepath.Join(basePath, hubName, clean)
info, err := os.Stat(fullPath)
if err != nil {
t.Fatalf("expected cached file at %s: %v", fullPath, err)
}
if info.Size() != expectedSize {
t.Fatalf("cached file %s size mismatch: got %d want %d", fullPath, info.Size(), expectedSize)
}
}
type apkStub struct {
server *http.Server
listener net.Listener
URL string
mu sync.Mutex
indexPath string
signaturePath string
packagePath string
indexBody []byte
signatureBody []byte
packageBody []byte
indexGets int
indexHeads int
signatureGets int
signatureHeads int
packageGets int
packageHeads int
}
func newAPKStub(t *testing.T) *apkStub {
t.Helper()
stub := &apkStub{
indexPath: "/v3.19/main/x86_64/APKINDEX.tar.gz",
signaturePath: "/v3.19/main/x86_64/APKINDEX.tar.gz.asc",
packagePath: "/v3.22/community/x86_64/tini-static-0.19.0-r3.apk",
indexBody: []byte("apk-index-body"),
signatureBody: []byte("apk-index-signature"),
packageBody: bytes.Repeat([]byte("apk-payload-"), 64*1024),
}
mux := http.NewServeMux()
mux.HandleFunc(stub.indexPath, stub.handleIndex)
mux.HandleFunc(stub.signaturePath, stub.handleSignature)
mux.HandleFunc(stub.packagePath, stub.handlePackage)
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("unable to start apk stub: %v", err)
}
srv := &http.Server{Handler: mux}
stub.server = srv
stub.listener = listener
stub.URL = "http://" + listener.Addr().String()
go func() {
_ = srv.Serve(listener)
}()
return stub
}
func (s *apkStub) handleIndex(w http.ResponseWriter, r *http.Request) {
s.handleWithETag(w, r, &s.indexGets, &s.indexHeads, s.indexBody, "application/gzip")
}
func (s *apkStub) handleSignature(w http.ResponseWriter, r *http.Request) {
s.handleWithETag(w, r, &s.signatureGets, &s.signatureHeads, s.signatureBody, "application/pgp-signature")
}
func (s *apkStub) handlePackage(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
defer s.mu.Unlock()
if r.Method == http.MethodHead {
s.packageHeads++
w.Header().Set("Content-Type", "application/vnd.android.package-archive")
w.WriteHeader(http.StatusOK)
return
}
s.packageGets++
w.Header().Set("Content-Type", "application/vnd.android.package-archive")
w.Header().Set("Content-Length", strconv.Itoa(len(s.packageBody)))
_, _ = w.Write(s.packageBody)
}
func (s *apkStub) handleWithETag(w http.ResponseWriter, r *http.Request, gets, heads *int, body []byte, contentType string) {
s.mu.Lock()
defer s.mu.Unlock()
etag := "\"apk-etag\""
if r.Method == http.MethodHead {
*heads++
w.Header().Set("ETag", etag)
w.Header().Set("Last-Modified", time.Now().UTC().Format(http.TimeFormat))
if matchETag(r, strings.Trim(etag, `"`)) {
w.WriteHeader(http.StatusNotModified)
return
}
if contentType != "" {
w.Header().Set("Content-Type", contentType)
}
return
}
*gets++
w.Header().Set("ETag", etag)
w.Header().Set("Last-Modified", time.Now().UTC().Format(http.TimeFormat))
if contentType != "" {
w.Header().Set("Content-Type", contentType)
}
_, _ = w.Write(body)
}
func (s *apkStub) IndexGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.indexGets
}
func (s *apkStub) IndexHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.indexHeads
}
func (s *apkStub) SignatureGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.signatureGets
}
func (s *apkStub) SignatureHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.signatureHeads
}
func (s *apkStub) PackageGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.packageGets
}
func (s *apkStub) PackageHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.packageHeads
}
func (s *apkStub) Close() {
if s == nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if s.server != nil {
_ = s.server.Shutdown(ctx)
}
if s.listener != nil {
_ = s.listener.Close()
}
}

View File

@@ -0,0 +1,276 @@
package integration
import (
"bytes"
"context"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"path"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/gofiber/fiber/v3"
"github.com/sirupsen/logrus"
"github.com/any-hub/any-hub/internal/cache"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/proxy"
"github.com/any-hub/any-hub/internal/server"
)
func TestAptPackagesCachedWithoutRevalidate(t *testing.T) {
stub := newAptPackageStub(t)
defer stub.Close()
storageDir := t.TempDir()
cfg := &config.Config{
Global: config.GlobalConfig{
ListenPort: 5000,
CacheTTL: config.Duration(time.Hour),
StoragePath: storageDir,
},
Hubs: []config.HubConfig{
{
Name: "apt",
Domain: "apt.hub.local",
Type: "debian",
Upstream: stub.URL,
},
},
}
registry, err := server.NewHubRegistry(cfg)
if err != nil {
t.Fatalf("registry error: %v", err)
}
logger := logrus.New()
logger.SetOutput(io.Discard)
store, err := cache.NewStore(storageDir)
if err != nil {
t.Fatalf("store error: %v", err)
}
app, err := server.NewApp(server.AppOptions{
Logger: logger,
Registry: registry,
Proxy: proxy.NewHandler(server.NewUpstreamClient(cfg), logger, store),
ListenPort: 5000,
})
if err != nil {
t.Fatalf("app error: %v", err)
}
doRequest := func(p string) *http.Response {
req := httptest.NewRequest(http.MethodGet, "http://apt.hub.local"+p, nil)
req.Host = "apt.hub.local"
resp, err := app.Test(req)
if err != nil {
t.Fatalf("app.Test error: %v", err)
}
return resp
}
resp := doRequest(stub.packagePath)
if resp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for package, got %d", resp.StatusCode)
}
if hit := resp.Header.Get("X-Any-Hub-Cache-Hit"); hit != "false" {
t.Fatalf("expected cache miss on first package fetch, got %s", hit)
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if !bytes.Equal(body, stub.packageBody) {
t.Fatalf("package body mismatch on first fetch: %d bytes", len(body))
}
resp2 := doRequest(stub.packagePath)
if resp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached package, got %d", resp2.StatusCode)
}
if hit := resp2.Header.Get("X-Any-Hub-Cache-Hit"); hit != "true" {
t.Fatalf("expected cache hit for package, got %s", hit)
}
body2, _ := io.ReadAll(resp2.Body)
resp2.Body.Close()
if !bytes.Equal(body2, stub.packageBody) {
t.Fatalf("package body mismatch on cache hit: %d bytes", len(body2))
}
hashResp := doRequest(stub.byHashPath)
if hashResp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for by-hash, got %d", hashResp.StatusCode)
}
if hit := hashResp.Header.Get("X-Any-Hub-Cache-Hit"); hit != "false" {
t.Fatalf("expected cache miss on first by-hash fetch, got %s", hit)
}
hashBody, _ := io.ReadAll(hashResp.Body)
hashResp.Body.Close()
if !bytes.Equal(hashBody, stub.byHashBody) {
t.Fatalf("by-hash body mismatch on first fetch: %d bytes", len(hashBody))
}
hashResp2 := doRequest(stub.byHashPath)
if hashResp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached by-hash, got %d", hashResp2.StatusCode)
}
if hit := hashResp2.Header.Get("X-Any-Hub-Cache-Hit"); hit != "true" {
t.Fatalf("expected cache hit for by-hash, got %s", hit)
}
hashBody2, _ := io.ReadAll(hashResp2.Body)
hashResp2.Body.Close()
if !bytes.Equal(hashBody2, stub.byHashBody) {
t.Fatalf("by-hash body mismatch on cache hit: %d bytes", len(hashBody2))
}
if stub.PackageGets() != 1 {
t.Fatalf("expected single package GET, got %d", stub.PackageGets())
}
if stub.PackageHeads() != 0 {
t.Fatalf("expected zero package HEAD revalidate, got %d", stub.PackageHeads())
}
if stub.ByHashGets() != 1 {
t.Fatalf("expected single by-hash GET, got %d", stub.ByHashGets())
}
if stub.ByHashHeads() != 0 {
t.Fatalf("expected zero by-hash HEAD revalidate, got %d", stub.ByHashHeads())
}
verifyStoredFile(t, storageDir, "apt", stub.packagePath, int64(len(stub.packageBody)))
verifyStoredFile(t, storageDir, "apt", stub.byHashPath, int64(len(stub.byHashBody)))
}
func verifyStoredFile(t *testing.T, basePath, hubName, locatorPath string, expectedSize int64) {
t.Helper()
clean := path.Clean("/" + locatorPath)
clean = strings.TrimPrefix(clean, "/")
fullPath := filepath.Join(basePath, hubName, clean)
info, err := os.Stat(fullPath)
if err != nil {
t.Fatalf("expected cached file at %s: %v", fullPath, err)
}
if info.Size() != expectedSize {
t.Fatalf("cached file %s size mismatch: got %d want %d", fullPath, info.Size(), expectedSize)
}
}
type aptPackageStub struct {
server *http.Server
listener net.Listener
URL string
mu sync.Mutex
packagePath string
byHashPath string
packageBody []byte
byHashBody []byte
packageGets int
packageHeads int
byHashGets int
byHashHeads int
}
func newAptPackageStub(t *testing.T) *aptPackageStub {
t.Helper()
stub := &aptPackageStub{
packagePath: "/pool/main/h/hello_1.0_amd64.deb",
byHashPath: "/dists/bookworm/by-hash/sha256/deadbeef",
packageBody: bytes.Repeat([]byte("deb-payload-"), 128*1024),
byHashBody: []byte("hash-index-body"),
}
mux := http.NewServeMux()
mux.HandleFunc(stub.packagePath, stub.handlePackage)
mux.HandleFunc(stub.byHashPath, stub.handleByHash)
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("unable to start apt package stub: %v", err)
}
srv := &http.Server{Handler: mux}
stub.server = srv
stub.listener = listener
stub.URL = "http://" + listener.Addr().String()
go func() {
_ = srv.Serve(listener)
}()
return stub
}
func (s *aptPackageStub) handlePackage(w http.ResponseWriter, r *http.Request) {
s.handleImmutable(w, r, &s.packageGets, &s.packageHeads, s.packageBody, "application/vnd.debian.binary-package")
}
func (s *aptPackageStub) handleByHash(w http.ResponseWriter, r *http.Request) {
s.handleImmutable(w, r, &s.byHashGets, &s.byHashHeads, s.byHashBody, "text/plain")
}
func (s *aptPackageStub) handleImmutable(w http.ResponseWriter, r *http.Request, gets, heads *int, body []byte, contentType string) {
s.mu.Lock()
defer s.mu.Unlock()
if r.Method == http.MethodHead {
*heads++
if contentType != "" {
w.Header().Set("Content-Type", contentType)
}
w.WriteHeader(http.StatusOK)
return
}
*gets++
if contentType != "" {
w.Header().Set("Content-Type", contentType)
}
w.Header().Set("Content-Length", strconv.Itoa(len(body)))
_, _ = w.Write(body)
}
func (s *aptPackageStub) PackageGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.packageGets
}
func (s *aptPackageStub) PackageHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.packageHeads
}
func (s *aptPackageStub) ByHashGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.byHashGets
}
func (s *aptPackageStub) ByHashHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.byHashHeads
}
func (s *aptPackageStub) Close() {
if s == nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if s.server != nil {
_ = s.server.Shutdown(ctx)
}
if s.listener != nil {
_ = s.listener.Close()
}
}

View File

@@ -0,0 +1,330 @@
package integration
import (
"context"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"github.com/gofiber/fiber/v3"
"github.com/sirupsen/logrus"
"github.com/any-hub/any-hub/internal/cache"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/proxy"
"github.com/any-hub/any-hub/internal/server"
)
func TestAptUpdateCachesIndexes(t *testing.T) {
stub := newAptStub(t)
defer stub.Close()
storageDir := t.TempDir()
cfg := &config.Config{
Global: config.GlobalConfig{
ListenPort: 5000,
CacheTTL: config.Duration(time.Hour),
StoragePath: storageDir,
},
Hubs: []config.HubConfig{
{
Name: "apt",
Domain: "apt.hub.local",
Type: "debian",
Upstream: stub.URL,
},
},
}
registry, err := server.NewHubRegistry(cfg)
if err != nil {
t.Fatalf("registry error: %v", err)
}
logger := logrus.New()
logger.SetOutput(io.Discard)
store, err := cache.NewStore(storageDir)
if err != nil {
t.Fatalf("store error: %v", err)
}
app, err := server.NewApp(server.AppOptions{
Logger: logger,
Registry: registry,
Proxy: proxy.NewHandler(server.NewUpstreamClient(cfg), logger, store),
ListenPort: 5000,
})
if err != nil {
t.Fatalf("app error: %v", err)
}
doRequest := func(path string) *http.Response {
req := httptest.NewRequest("GET", "http://apt.hub.local"+path, nil)
req.Host = "apt.hub.local"
resp, err := app.Test(req)
if err != nil {
t.Fatalf("app.Test error: %v", err)
}
return resp
}
releasePath := "/dists/bookworm/Release"
packagesPath := "/dists/bookworm/main/binary-amd64/Packages.gz"
contentsPath := "/dists/bookworm/main/Contents-amd64.gz"
resp := doRequest(releasePath)
if resp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for release, got %d", resp.StatusCode)
}
if resp.Header.Get("X-Any-Hub-Cache-Hit") != "false" {
t.Fatalf("expected cache miss for first release fetch")
}
resp.Body.Close()
resp2 := doRequest(releasePath)
if resp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached release, got %d", resp2.StatusCode)
}
if resp2.Header.Get("X-Any-Hub-Cache-Hit") != "true" {
t.Fatalf("expected cache hit for release")
}
resp2.Body.Close()
pkgResp := doRequest(packagesPath)
if pkgResp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for packages, got %d", pkgResp.StatusCode)
}
if pkgResp.Header.Get("X-Any-Hub-Cache-Hit") != "false" {
t.Fatalf("expected cache miss for packages")
}
pkgResp.Body.Close()
pkgResp2 := doRequest(packagesPath)
if pkgResp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached packages, got %d", pkgResp2.StatusCode)
}
if pkgResp2.Header.Get("X-Any-Hub-Cache-Hit") != "true" {
t.Fatalf("expected cache hit for packages")
}
pkgResp2.Body.Close()
contentsResp := doRequest(contentsPath)
if contentsResp.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for contents, got %d", contentsResp.StatusCode)
}
if contentsResp.Header.Get("X-Any-Hub-Cache-Hit") != "false" {
t.Fatalf("expected cache miss for contents")
}
contentsResp.Body.Close()
contentsResp2 := doRequest(contentsPath)
if contentsResp2.StatusCode != fiber.StatusOK {
t.Fatalf("expected 200 for cached contents, got %d", contentsResp2.StatusCode)
}
if contentsResp2.Header.Get("X-Any-Hub-Cache-Hit") != "true" {
t.Fatalf("expected cache hit for contents")
}
contentsResp2.Body.Close()
if stub.ReleaseGets() != 1 {
t.Fatalf("expected single release GET, got %d", stub.ReleaseGets())
}
if stub.ReleaseHeads() != 1 {
t.Fatalf("expected single release HEAD revalidate, got %d", stub.ReleaseHeads())
}
if stub.PackagesGets() != 1 {
t.Fatalf("expected single packages GET, got %d", stub.PackagesGets())
}
if stub.PackagesHeads() != 1 {
t.Fatalf("expected single packages HEAD revalidate, got %d", stub.PackagesHeads())
}
if stub.ContentsGets() != 1 {
t.Fatalf("expected single contents GET, got %d", stub.ContentsGets())
}
if stub.ContentsHeads() != 1 {
t.Fatalf("expected single contents HEAD revalidate, got %d", stub.ContentsHeads())
}
}
type aptStub struct {
server *http.Server
listener net.Listener
URL string
mu sync.Mutex
releaseBody string
packagesBody string
contentsBody string
releaseETag string
packagesETag string
contentsETag string
releaseGets int
releaseHeads int
packagesGets int
packagesHeads int
contentsGets int
contentsHeads int
releasePath string
packagesPath string
contentsPath string
}
func newAptStub(t *testing.T) *aptStub {
t.Helper()
stub := &aptStub{
releaseBody: "Release-body",
packagesBody: "Packages-body",
contentsBody: "Contents-body",
releaseETag: "r1",
packagesETag: "p1",
contentsETag: "c1",
releasePath: "/dists/bookworm/Release",
packagesPath: "/dists/bookworm/main/binary-amd64/Packages.gz",
contentsPath: "/dists/bookworm/main/Contents-amd64.gz",
}
mux := http.NewServeMux()
mux.HandleFunc(stub.releasePath, stub.handleRelease)
mux.HandleFunc(stub.packagesPath, stub.handlePackages)
mux.HandleFunc(stub.contentsPath, stub.handleContents)
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("unable to start apt stub: %v", err)
}
srv := &http.Server{Handler: mux}
stub.server = srv
stub.listener = listener
stub.URL = "http://" + listener.Addr().String()
go func() {
_ = srv.Serve(listener)
}()
return stub
}
func (s *aptStub) handleRelease(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
defer s.mu.Unlock()
if r.Method == http.MethodHead {
s.releaseHeads++
if matchETag(r, s.releaseETag) {
w.WriteHeader(http.StatusNotModified)
return
}
writeHeaders(w, s.releaseETag)
return
}
s.releaseGets++
writeHeaders(w, s.releaseETag)
_, _ = w.Write([]byte(s.releaseBody))
}
func (s *aptStub) handlePackages(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
defer s.mu.Unlock()
if r.Method == http.MethodHead {
s.packagesHeads++
if matchETag(r, s.packagesETag) {
w.WriteHeader(http.StatusNotModified)
return
}
writeHeaders(w, s.packagesETag)
w.Header().Set("Content-Type", "application/gzip")
return
}
s.packagesGets++
writeHeaders(w, s.packagesETag)
w.Header().Set("Content-Type", "application/gzip")
_, _ = w.Write([]byte(s.packagesBody))
}
func (s *aptStub) handleContents(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
defer s.mu.Unlock()
if r.Method == http.MethodHead {
s.contentsHeads++
if matchETag(r, s.contentsETag) {
w.WriteHeader(http.StatusNotModified)
return
}
writeHeaders(w, s.contentsETag)
w.Header().Set("Content-Type", "application/gzip")
return
}
s.contentsGets++
writeHeaders(w, s.contentsETag)
w.Header().Set("Content-Type", "application/gzip")
_, _ = w.Write([]byte(s.contentsBody))
}
func matchETag(r *http.Request, etag string) bool {
for _, candidate := range r.Header.Values("If-None-Match") {
c := strings.Trim(candidate, "\"")
if c == etag || candidate == etag {
return true
}
}
return false
}
func writeHeaders(w http.ResponseWriter, etag string) {
w.Header().Set("ETag", "\""+etag+"\"")
w.Header().Set("Last-Modified", time.Now().UTC().Format(http.TimeFormat))
}
func (s *aptStub) ReleaseGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.releaseGets
}
func (s *aptStub) ReleaseHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.releaseHeads
}
func (s *aptStub) PackagesGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.packagesGets
}
func (s *aptStub) PackagesHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.packagesHeads
}
func (s *aptStub) ContentsGets() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.contentsGets
}
func (s *aptStub) ContentsHeads() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.contentsHeads
}
func (s *aptStub) Close() {
if s == nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if s.server != nil {
_ = s.server.Shutdown(ctx)
}
if s.listener != nil {
_ = s.listener.Close()
}
}

View File

@@ -2,12 +2,14 @@ package integration
import (
"context"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
@@ -294,6 +296,7 @@ type cacheFlowStub struct {
lastRequest *http.Request
body []byte
etag string
etagVer int
lastMod string
}
@@ -302,6 +305,7 @@ func newCacheFlowStub(t *testing.T, paths ...string) *cacheFlowStub {
stub := &cacheFlowStub{
body: []byte("upstream payload"),
etag: `"etag-v1"`,
etagVer: 1,
lastMod: time.Now().UTC().Format(http.TimeFormat),
}
@@ -344,6 +348,8 @@ func (s *cacheFlowStub) Close() {
func (s *cacheFlowStub) handle(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
etag := s.etag
lastMod := s.lastMod
if r.Method == http.MethodHead {
s.headHits++
} else {
@@ -354,15 +360,21 @@ func (s *cacheFlowStub) handle(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead {
w.Header().Set("Content-Type", "text/plain")
w.Header().Set("Etag", s.etag)
w.Header().Set("Last-Modified", s.lastMod)
w.Header().Set("Etag", etag)
w.Header().Set("Last-Modified", lastMod)
for _, candidate := range r.Header.Values("If-None-Match") {
if strings.Trim(candidate, `"`) == strings.Trim(etag, `"`) {
w.WriteHeader(http.StatusNotModified)
return
}
}
w.WriteHeader(http.StatusOK)
return
}
w.Header().Set("Content-Type", "text/plain")
w.Header().Set("Etag", s.etag)
w.Header().Set("Last-Modified", s.lastMod)
w.Header().Set("Etag", etag)
w.Header().Set("Last-Modified", lastMod)
_, _ = w.Write(s.body)
}
@@ -370,5 +382,7 @@ func (s *cacheFlowStub) UpdateBody(body []byte) {
s.mu.Lock()
defer s.mu.Unlock()
s.body = body
s.lastMod = time.Now().UTC().Format(http.TimeFormat)
s.etagVer++
s.etag = fmt.Sprintf(`"etag-v%d"`, s.etagVer)
s.lastMod = time.Now().UTC().Add(2 * time.Second).Format(http.TimeFormat)
}

View File

@@ -35,7 +35,6 @@ func TestCacheStrategyOverrides(t *testing.T) {
Name: "npm-ttl",
Domain: "ttl.npm.local",
Type: "npm",
Module: "npm",
Upstream: stub.URL,
CacheTTL: config.Duration(ttl),
},
@@ -111,7 +110,6 @@ func TestCacheStrategyOverrides(t *testing.T) {
Name: "npm-novalidation",
Domain: "novalidation.npm.local",
Type: "npm",
Module: "npm",
Upstream: stub.URL,
CacheTTL: config.Duration(ttl),
ValidationMode: string(hubmodule.ValidationModeNever),

View File

@@ -1,118 +0,0 @@
package integration
import (
"io"
"net/http/httptest"
"testing"
"time"
"github.com/gofiber/fiber/v3"
"github.com/sirupsen/logrus"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/hubmodule"
"github.com/any-hub/any-hub/internal/hubmodule/legacy"
"github.com/any-hub/any-hub/internal/server"
)
func TestLegacyAdapterRolloutToggle(t *testing.T) {
const moduleKey = "rollout-toggle-test"
_ = hubmodule.Register(hubmodule.ModuleMetadata{Key: moduleKey})
logger := logrus.New()
logger.SetOutput(io.Discard)
baseHub := config.HubConfig{
Name: "dual-mode",
Domain: "dual.local",
Type: "docker",
Upstream: "https://registry.npmjs.org",
Module: moduleKey,
}
testCases := []struct {
name string
rolloutFlag string
expectKey string
expectFlag legacy.RolloutFlag
}{
{
name: "force legacy",
rolloutFlag: "legacy-only",
expectKey: hubmodule.DefaultModuleKey(),
expectFlag: legacy.RolloutLegacyOnly,
},
{
name: "dual mode",
rolloutFlag: "dual",
expectKey: moduleKey,
expectFlag: legacy.RolloutDual,
},
{
name: "full modular",
rolloutFlag: "modular",
expectKey: moduleKey,
expectFlag: legacy.RolloutModular,
},
{
name: "rollback to legacy",
rolloutFlag: "legacy-only",
expectKey: hubmodule.DefaultModuleKey(),
expectFlag: legacy.RolloutLegacyOnly,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
cfg := &config.Config{
Global: config.GlobalConfig{
ListenPort: 6100,
CacheTTL: config.Duration(time.Minute),
},
Hubs: []config.HubConfig{
func() config.HubConfig {
h := baseHub
h.Rollout = tc.rolloutFlag
return h
}(),
},
}
registry, err := server.NewHubRegistry(cfg)
if err != nil {
t.Fatalf("failed to build registry: %v", err)
}
recorder := &routeRecorder{}
app := mustNewApp(t, cfg.Global.ListenPort, logger, registry, recorder)
req := httptest.NewRequest("GET", "http://dual.local/v2/", nil)
req.Host = "dual.local"
resp, err := app.Test(req)
if err != nil {
t.Fatalf("request failed: %v", err)
}
if resp.StatusCode != fiber.StatusNoContent {
t.Fatalf("unexpected status: %d", resp.StatusCode)
}
if recorder.moduleKey != tc.expectKey {
t.Fatalf("expected module %s, got %s", tc.expectKey, recorder.moduleKey)
}
if recorder.rolloutFlag != tc.expectFlag {
t.Fatalf("expected rollout flag %s, got %s", tc.expectFlag, recorder.rolloutFlag)
}
})
}
}
type routeRecorder struct {
moduleKey string
rolloutFlag legacy.RolloutFlag
}
func (r *routeRecorder) Handle(c fiber.Ctx, route *server.HubRoute) error {
r.moduleKey = route.ModuleKey
r.rolloutFlag = route.RolloutFlag
return c.SendStatus(fiber.StatusNoContent)
}

View File

@@ -36,19 +36,17 @@ func TestModuleDiagnosticsEndpoints(t *testing.T) {
CacheTTL: config.Duration(30 * time.Minute),
},
Hubs: []config.HubConfig{
{
Name: "legacy-hub",
Domain: "legacy.local",
Type: "docker",
Upstream: "https://registry-1.docker.io",
},
{
Name: "modern-hub",
Domain: "modern.local",
Type: "npm",
Upstream: "https://registry.npmjs.org",
Module: moduleKey,
Rollout: "dual",
},
{
Name: "docker-hub",
Domain: "docker.local",
Type: "docker",
Upstream: "https://registry-1.docker.io",
},
},
}
@@ -74,12 +72,10 @@ func TestModuleDiagnosticsEndpoints(t *testing.T) {
var payload struct {
Modules []map[string]any `json:"modules"`
Hubs []struct {
HubName string `json:"hub_name"`
ModuleKey string `json:"module_key"`
Rollout string `json:"rollout_flag"`
Domain string `json:"domain"`
Port int `json:"port"`
LegacyOnly bool `json:"legacy_only"`
HubName string `json:"hub_name"`
ModuleKey string `json:"module_key"`
Domain string `json:"domain"`
Port int `json:"port"`
} `json:"hubs"`
}
body, _ := io.ReadAll(resp.Body)
@@ -108,22 +104,13 @@ func TestModuleDiagnosticsEndpoints(t *testing.T) {
}
for _, hub := range payload.Hubs {
switch hub.HubName {
case "legacy-hub":
if hub.ModuleKey != hubmodule.DefaultModuleKey() {
t.Fatalf("legacy hub should expose legacy module, got %s", hub.ModuleKey)
}
if !hub.LegacyOnly {
t.Fatalf("legacy hub should be marked legacy_only")
}
case "modern-hub":
if hub.ModuleKey != moduleKey {
t.Fatalf("modern hub should expose %s, got %s", moduleKey, hub.ModuleKey)
if hub.ModuleKey != "npm" {
t.Fatalf("modern hub should expose npm, got %s", hub.ModuleKey)
}
if hub.Rollout != "dual" {
t.Fatalf("modern hub rollout flag should be dual, got %s", hub.Rollout)
}
if hub.LegacyOnly {
t.Fatalf("modern hub should not be marked legacy_only")
case "docker-hub":
if hub.ModuleKey != "docker" {
t.Fatalf("docker hub should expose docker, got %s", hub.ModuleKey)
}
default:
t.Fatalf("unexpected hub %s", hub.HubName)

View File

@@ -1,107 +1,100 @@
package integration
import (
"io"
"net/http/httptest"
"testing"
"io"
"net/http/httptest"
"testing"
"github.com/gofiber/fiber/v3"
"github.com/sirupsen/logrus"
"github.com/gofiber/fiber/v3"
"github.com/sirupsen/logrus"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/hubmodule"
"github.com/any-hub/any-hub/internal/server"
"github.com/any-hub/any-hub/internal/config"
"github.com/any-hub/any-hub/internal/server"
)
func TestModuleRoutingIsolation(t *testing.T) {
_ = hubmodule.Register(hubmodule.ModuleMetadata{Key: "module-routing-test"})
cfg := &config.Config{
Global: config.GlobalConfig{
ListenPort: 6000,
CacheTTL: config.Duration(3600),
},
Hubs: []config.HubConfig{
{
Name: "docker-hub",
Domain: "legacy.hub.local",
Type: "docker",
Upstream: "https://registry-1.docker.io",
},
{
Name: "npm-hub",
Domain: "test.hub.local",
Type: "npm",
Upstream: "https://registry.example.com",
},
},
}
cfg := &config.Config{
Global: config.GlobalConfig{
ListenPort: 6000,
CacheTTL: config.Duration(3600),
},
Hubs: []config.HubConfig{
{
Name: "legacy",
Domain: "legacy.hub.local",
Type: "docker",
Module: "legacy",
Upstream: "https://registry-1.docker.io",
},
{
Name: "test",
Domain: "test.hub.local",
Type: "npm",
Module: "module-routing-test",
Upstream: "https://registry.example.com",
},
},
}
registry, err := server.NewHubRegistry(cfg)
if err != nil {
t.Fatalf("failed to create registry: %v", err)
}
registry, err := server.NewHubRegistry(cfg)
if err != nil {
t.Fatalf("failed to create registry: %v", err)
}
logger := logrus.New()
logger.SetOutput(io.Discard)
logger := logrus.New()
logger.SetOutput(io.Discard)
recorder := &moduleRecorder{}
app := mustNewApp(t, cfg.Global.ListenPort, logger, registry, recorder)
recorder := &moduleRecorder{}
app := mustNewApp(t, cfg.Global.ListenPort, logger, registry, recorder)
legacyReq := httptest.NewRequest("GET", "http://legacy.hub.local/v2/", nil)
legacyReq.Host = "legacy.hub.local"
legacyReq.Header.Set("Host", "legacy.hub.local")
resp, err := app.Test(legacyReq)
if err != nil {
t.Fatalf("legacy request failed: %v", err)
}
if resp.StatusCode != fiber.StatusNoContent {
t.Fatalf("legacy hub should return 204, got %d", resp.StatusCode)
}
if recorder.moduleKey != "docker" {
t.Fatalf("expected docker module, got %s", recorder.moduleKey)
}
legacyReq := httptest.NewRequest("GET", "http://legacy.hub.local/v2/", nil)
legacyReq.Host = "legacy.hub.local"
legacyReq.Header.Set("Host", "legacy.hub.local")
resp, err := app.Test(legacyReq)
if err != nil {
t.Fatalf("legacy request failed: %v", err)
}
if resp.StatusCode != fiber.StatusNoContent {
t.Fatalf("legacy hub should return 204, got %d", resp.StatusCode)
}
if recorder.moduleKey != "legacy" {
t.Fatalf("expected legacy module, got %s", recorder.moduleKey)
}
testReq := httptest.NewRequest("GET", "http://test.hub.local/v2/", nil)
testReq.Host = "test.hub.local"
testReq.Header.Set("Host", "test.hub.local")
resp2, err := app.Test(testReq)
if err != nil {
t.Fatalf("test request failed: %v", err)
}
if resp2.StatusCode != fiber.StatusNoContent {
t.Fatalf("test hub should return 204, got %d", resp2.StatusCode)
}
if recorder.moduleKey != "module-routing-test" {
t.Fatalf("expected module-routing-test module, got %s", recorder.moduleKey)
}
testReq := httptest.NewRequest("GET", "http://test.hub.local/v2/", nil)
testReq.Host = "test.hub.local"
testReq.Header.Set("Host", "test.hub.local")
resp2, err := app.Test(testReq)
if err != nil {
t.Fatalf("test request failed: %v", err)
}
if resp2.StatusCode != fiber.StatusNoContent {
t.Fatalf("test hub should return 204, got %d", resp2.StatusCode)
}
if recorder.moduleKey != "npm" {
t.Fatalf("expected npm module, got %s", recorder.moduleKey)
}
}
func mustNewApp(t *testing.T, port int, logger *logrus.Logger, registry *server.HubRegistry, handler server.ProxyHandler) *fiber.App {
t.Helper()
app, err := server.NewApp(server.AppOptions{
Logger: logger,
Registry: registry,
Proxy: handler,
ListenPort: port,
})
if err != nil {
t.Fatalf("failed to create app: %v", err)
}
return app
t.Helper()
app, err := server.NewApp(server.AppOptions{
Logger: logger,
Registry: registry,
Proxy: handler,
ListenPort: port,
})
if err != nil {
t.Fatalf("failed to create app: %v", err)
}
return app
}
type moduleRecorder struct {
routeName string
moduleKey string
rollout string
routeName string
moduleKey string
}
func (p *moduleRecorder) Handle(c fiber.Ctx, route *server.HubRoute) error {
p.routeName = route.Config.Name
p.moduleKey = route.ModuleKey
p.rollout = string(route.RolloutFlag)
return c.SendStatus(fiber.StatusNoContent)
p.routeName = route.Config.Name
p.moduleKey = route.Module.Key
return c.SendStatus(fiber.StatusNoContent)
}

View File

@@ -239,7 +239,7 @@ func (s *pypiStub) handleWheel(w http.ResponseWriter, r *http.Request) {
func (s *pypiStub) UpdateSimple(body []byte) {
s.mu.Lock()
s.simpleBody = append([]byte(nil), body...)
s.lastSimpleMod = time.Now().UTC().Format(http.TimeFormat)
s.lastSimpleMod = time.Now().UTC().Add(2 * time.Second).Format(http.TimeFormat)
s.mu.Unlock()
}