mirror of
https://github.com/obra/superpowers.git
synced 2026-08-01 23:21:34 +08:00
Compare commits
4 Commits
fix/docs-c
...
fix/t2-cod
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d8189d1587 | ||
|
|
db4538fcb8 | ||
|
|
9b8b14fe12 | ||
|
|
75756d2900 |
File diff suppressed because it is too large
Load Diff
@@ -1,252 +0,0 @@
|
|||||||
# Codex Efficiency Fixes — Design
|
|
||||||
|
|
||||||
Date: 2026-07-30
|
|
||||||
Status: approved by Jesse (in-session)
|
|
||||||
Branch: `codex-efficiency-fixes` off `dev`
|
|
||||||
|
|
||||||
## Sources
|
|
||||||
|
|
||||||
- Eval campaign closeout: `superpowers-autoresearch/reports/2026-07-codex-efficiency-campaign.md`
|
|
||||||
(treatment table §4; every treatment below has a scorer and a measured
|
|
||||||
`dev` baseline).
|
|
||||||
- Codex source recon: `superpowers-autoresearch/docs/2026-07-29-codex-multiagent-v2-capabilities.md`
|
|
||||||
(file:line citations against the Codex CLI source; grounds T2, T3, T5).
|
|
||||||
- Published experiment write-ups: `superpowers-evals/docs/experiments/`.
|
|
||||||
- Drew's spinout stack (PRs #2036, #2035) is **evidence, not adopted text**:
|
|
||||||
Jesse wants to dig into those fixes in more detail before adopting any
|
|
||||||
of them; they inform the problem statements only.
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Ship the five evidence-strong treatments from the codex-efficiency eval
|
|
||||||
campaign as superpowers skill/doc changes, each graded against its
|
|
||||||
pre-registered criterion by the campaign's scorers before its PR is cut.
|
|
||||||
Phase 2 (everything else in the closeout treatment table) follows, each
|
|
||||||
item gated on new baseline work first.
|
|
||||||
|
|
||||||
## Scope decisions (settled with Jesse)
|
|
||||||
|
|
||||||
- **Phase 1 = the evidence-strong five** (T1–T5 below). Phase 2 items
|
|
||||||
each need a failing baseline before any fix ships (discrimination
|
|
||||||
rule: inconclusive-by-zero is a stop).
|
|
||||||
- **One branch, PR per treatment.** Development and batteries happen on
|
|
||||||
`codex-efficiency-fixes`; when a treatment beats its criterion, it is
|
|
||||||
cut into its own PR against `dev` with its eval evidence. No merge
|
|
||||||
without Jesse's per-PR approval.
|
|
||||||
- **T4 ships cross-harness with a global regression battery** (Claude
|
|
||||||
Code, Codex, Gemini), variant C shape: ceremony scales, approval never
|
|
||||||
does.
|
|
||||||
|
|
||||||
## The five treatments
|
|
||||||
|
|
||||||
### T1. SDD worker-review prohibition
|
|
||||||
|
|
||||||
**Evidence:** 9/9 depth-2 spawns across 4 corpora were implementer-issued
|
|
||||||
reviewers; all 9 were same-task duplicates of the review the controller
|
|
||||||
dispatches anyway. The dispatch contract never says review is not the
|
|
||||||
worker's job; "self-review" in the implementer prompt gets reified into a
|
|
||||||
reviewer subagent on harnesses where children can spawn (Codex).
|
|
||||||
|
|
||||||
**Changes:**
|
|
||||||
- `skills/subagent-driven-development/implementer-prompt.md`: an explicit
|
|
||||||
"You do not dispatch subagents" clause — self-review means reading your
|
|
||||||
own diff; the controller owns all review dispatch; a reviewer you spawn
|
|
||||||
duplicates a review the process already provides.
|
|
||||||
- `skills/subagent-driven-development/SKILL.md`: one dispatch-contract
|
|
||||||
line in the task loop, plus a Red Flags row: "An independent review
|
|
||||||
would strengthen my report" → review is the controller's next step;
|
|
||||||
your reviewer is a duplicate seat.
|
|
||||||
- Harness-agnostic wording (no-op where children cannot spawn).
|
|
||||||
|
|
||||||
**Graded by:** `score_e6.py` (depth-2 spawns by spawner role, duplicate
|
|
||||||
review families); `score_e5.py` for the same-scope variant.
|
|
||||||
**Baseline:** 9/9 worker-issued, 0 counter-examples.
|
|
||||||
**Criterion:** 0 worker-issued depth-2 spawns AND review coverage
|
|
||||||
preserved (every task still gets exactly one controller-dispatched task
|
|
||||||
review).
|
|
||||||
|
|
||||||
### T2. Event-driven waiting
|
|
||||||
|
|
||||||
**Evidence:** 60–78% of `wait_agent` calls time out in every corpus
|
|
||||||
(dev 67.1%, spinout 60.2%). Source recon: V2 waits are event
|
|
||||||
subscriptions, not polls — one long wait has the same wake latency as a
|
|
||||||
10s poll at ~1/90th the calls; a completed child's FINAL_ANSWER is pushed
|
|
||||||
into the parent's mailbox and drained into the next model request with no
|
|
||||||
wait at all.
|
|
||||||
|
|
||||||
**Changes** (`skills/using-superpowers/references/codex-tools.md`):
|
|
||||||
- Never short-timeout poll.
|
|
||||||
- While local work remains, do not wait — child results arrive with your
|
|
||||||
next turn via the mailbox.
|
|
||||||
- When genuinely idle, issue ONE `wait_agent` with a long `timeout_ms`
|
|
||||||
(900000+; harness max 3600000).
|
|
||||||
- V2 caveat stated: completion mail carries `trigger_turn=false` and will
|
|
||||||
not wake an idle controller — that is the one job `wait_agent` has.
|
|
||||||
|
|
||||||
**Graded by:** `score_e7.py` (timeout rate, inter-poll cadence,
|
|
||||||
cache-rebill estimate — the rebill figure stays labeled as an estimate).
|
|
||||||
**Baseline:** dev 67.1% timeout rate.
|
|
||||||
**Criterion:** timeout rate < 25% with no loss of task completion.
|
|
||||||
|
|
||||||
### T3. codex-tools.md corrections
|
|
||||||
|
|
||||||
**Evidence:** five claims in the current guidance are contradicted by the
|
|
||||||
Codex source (all file:line-cited in the capabilities doc):
|
|
||||||
1. `close_agent` does not exist in multi-agent V2 (V1-only). V2 LRU-evicts
|
|
||||||
finished children automatically; not closing costs nothing;
|
|
||||||
`followup_task` transparently reloads an evicted child.
|
|
||||||
2. Fix rounds can always resume the implementer via `followup_task` —
|
|
||||||
dev's "if your harness cannot send another message to a spawned agent,
|
|
||||||
dispatch each fix round as a fresh implementer" branch is dead on V2.
|
|
||||||
3. Role files (`~/.codex/agents/**.toml`) DO attach to spawns via
|
|
||||||
`agent_type` on isolated forks (0.145+).
|
|
||||||
4. Full-history forks accept `model`/`reasoning_effort` overrides; only
|
|
||||||
`agent_type` is refused. (Isolated forks remain the SDD guidance for
|
|
||||||
context-hygiene reasons, stated accurately.)
|
|
||||||
5. Dispatch guidance must never name non-V2 model presets — the V2 spawn
|
|
||||||
allowlist is v2 presets only; others hard-error.
|
|
||||||
|
|
||||||
**Changes:** rewrite the multi-agent paragraph of
|
|
||||||
`skills/using-superpowers/references/codex-tools.md` to be
|
|
||||||
version-honest (V1 vs V2 behavior labeled where they differ).
|
|
||||||
|
|
||||||
**Graded by:** source citation (already verified); no scorer regressions
|
|
||||||
on the shared battery. `score_e8.py` is retained as a V1/V2 schema
|
|
||||||
detector, not a hygiene grader — no `close_agent` checklist ships.
|
|
||||||
|
|
||||||
### T4. Brainstorming three-path router (variant C: approval always)
|
|
||||||
|
|
||||||
**Evidence:** micro — the current HARD-GATE text pushes a bounded task to
|
|
||||||
FULL ceremony 5/5, while Z-null (no guidance) and a three-path router
|
|
||||||
both differentiate 5/5: the absolute wording suppresses discrimination
|
|
||||||
the model draws natively. FULL battery — ceremony volume scales
|
|
||||||
moderately (16.7 vs 24.0 tool calls, bounded vs arch), but the
|
|
||||||
two-document ritual (spec file → plan file) ran unconditionally in every
|
|
||||||
rep. The measured waste is the unconditional artifact ritual, not the
|
|
||||||
approval gate.
|
|
||||||
|
|
||||||
**Design (variant C):** three paths scale the ARTIFACT; every path keeps
|
|
||||||
human approval before implementation:
|
|
||||||
- **Spike** (feasibility question, explicitly throwaway): present the
|
|
||||||
question and the intended probe in 2–3 sentences, get a nod, go. No
|
|
||||||
docs. Findings return as a recommendation; anything built stays labeled
|
|
||||||
throwaway.
|
|
||||||
- **Bounded** (well-scoped change to an existing, understood flow):
|
|
||||||
present a short design in chat, get approval, implement. No spec file,
|
|
||||||
no writing-plans invocation.
|
|
||||||
- **Architectural** (restructures components, new subsystem, public
|
|
||||||
interface change): the full current flow — spec doc, review,
|
|
||||||
writing-plans.
|
|
||||||
|
|
||||||
**Guards (all ship with the router):**
|
|
||||||
- Classification is said out loud ("this looks bounded, so I'll present a
|
|
||||||
short design here rather than write a spec") so the human can override.
|
|
||||||
- When in doubt between two paths, take the heavier one.
|
|
||||||
- One-way ratchet: hidden complexity discovered mid-path upgrades the
|
|
||||||
path; never downgrade mid-task.
|
|
||||||
- New Red Flags rows targeting classification-as-escape-hatch ("I'll call
|
|
||||||
it bounded to skip the doc").
|
|
||||||
|
|
||||||
**Changes** (`skills/brainstorming/SKILL.md`): HARD-GATE keeps "no
|
|
||||||
implementation before approval" and drops "regardless of perceived
|
|
||||||
simplicity" as the ceremony driver; anti-pattern section reframed (the
|
|
||||||
sin is skipping approval, not skipping documents); checklist steps 6–9
|
|
||||||
become the architectural path; process-flow graph gains the router; Red
|
|
||||||
Flags rows added. This is carefully-tuned content — the edit follows
|
|
||||||
writing-skills methodology and ships only with the full eval evidence
|
|
||||||
below.
|
|
||||||
|
|
||||||
**Graded by (three layers):**
|
|
||||||
1. **Micro** (`ceremony-path-micro.py`, adapted): variant C literal text,
|
|
||||||
plus adversarially ambiguous briefs the campaign never tested (a task
|
|
||||||
that pattern-matches bounded but hides a public interface change).
|
|
||||||
Criteria: spike/bounded/arch differentiate (≥4/5 per cell); ambiguous
|
|
||||||
briefs escalate to FULL (≥4/5); arch never downgrades (5/5).
|
|
||||||
2. **Codex ceremony battery:** `cx-ceremony-{spike,bounded,arch}` on the
|
|
||||||
fix arm, 3 reps each, `score_e4.py` census. Criteria: bounded reps
|
|
||||||
show an approval turn but zero committed spec files and zero
|
|
||||||
writing-plans ritual; arch reps keep the full two-doc flow; spike reps
|
|
||||||
stay minimal.
|
|
||||||
3. **Global regression battery:** the same three ceremony scenarios on
|
|
||||||
Claude Code and Gemini (rig work: those scenarios are currently
|
|
||||||
codex-gated), 3 reps each; plus the triggering acceptance check
|
|
||||||
("Let's make a react todo list" auto-triggers brainstorming into the
|
|
||||||
full/architectural path) on all three harnesses.
|
|
||||||
|
|
||||||
### T5. Explicit model on child-issued spawns
|
|
||||||
|
|
||||||
**Evidence:** root spawns are 100% explicit-model at CLI 0.146 (dev
|
|
||||||
14/14); the live gap is depth-2 — 2/2 child-issued spawns omitted
|
|
||||||
`model`. Source recon: `model` without `reasoning_effort` resets effort
|
|
||||||
to the MODEL's default, not the parent's.
|
|
||||||
|
|
||||||
**Changes** (`skills/using-superpowers/references/codex-tools.md`):
|
|
||||||
- Every spawn you issue — including as a child — sets `model` AND
|
|
||||||
`reasoning_effort`; the effort-reset trap is named.
|
|
||||||
- Advise `[agents].default_subagent_model` and
|
|
||||||
`[agents].default_subagent_reasoning_effort` in `~/.codex/config.toml`
|
|
||||||
as the machine-level backstop for anything that slips through.
|
|
||||||
|
|
||||||
**Graded by:** `score_e1.py` (per-spawn explicit-model rate, by depth) on
|
|
||||||
the shared battery.
|
|
||||||
**Baseline:** depth-2: 0/2 explicit.
|
|
||||||
**Criterion:** every spawn at every depth carries explicit model +
|
|
||||||
effort. Pre-registered caveat: if T1 eliminates depth-2 spawns entirely,
|
|
||||||
T5 grades as root-spawn regression (hold 100%) plus doc correctness and
|
|
||||||
is recorded inconclusive-by-zero at depth-2 — the config backstop is then
|
|
||||||
the operative mechanism.
|
|
||||||
|
|
||||||
## Grading plan
|
|
||||||
|
|
||||||
- **Shared SDD battery** carries T1, T2, T5: `cx-sdd-small`, fix-branch
|
|
||||||
arm (`/tmp/sp-arm-fix`), 8 reps across both container lanes. Dev
|
|
||||||
baselines are already measured; no baseline re-runs.
|
|
||||||
- **T4 batteries** as listed above (micro + codex ceremony + global
|
|
||||||
regression).
|
|
||||||
- **Pre-registration:** every battery gets a hypothesis-log entry
|
|
||||||
(prediction, scorer, criterion) in
|
|
||||||
`superpowers-autoresearch/logs/2026-07-30-codex-efficiency-fixes.md`
|
|
||||||
BEFORE it runs. Standing rules carry over: append-only log, manual
|
|
||||||
inspection of scorer matches on fix-arm runs (non-circular
|
|
||||||
verification), no raw rollouts committed, correctness rides beside
|
|
||||||
cost in every verdict.
|
|
||||||
- **Attribution:** orthogonal scorers on one combined branch; unexpected
|
|
||||||
regressions bisect by treatment commit.
|
|
||||||
- **Budget:** shared battery ~$40, codex ceremony ~$40, global
|
|
||||||
regression ~$40–80, micros ~$5 → phase 1 ≈ $150–200 of the ~$850
|
|
||||||
remaining from the campaign's $1000.
|
|
||||||
|
|
||||||
## Process
|
|
||||||
|
|
||||||
- Work happens in the `codex-efficiency-fixes` worktree (branched off
|
|
||||||
`dev`); execution via subagent-driven-development from a written plan.
|
|
||||||
- Skill-text changes follow writing-skills methodology.
|
|
||||||
- Scenario/rig changes (un-gating ceremony scenarios for Claude
|
|
||||||
Code/Gemini, adversarial micro briefs) land in `superpowers-evals`
|
|
||||||
main, as authorized.
|
|
||||||
- PR-per-treatment against `dev`, each with its eval evidence and the
|
|
||||||
standard identification block; merges only on Jesse's per-PR approval.
|
|
||||||
|
|
||||||
## Phase 2 queue (baseline-first; not in this plan's tasks)
|
|
||||||
|
|
||||||
Each item requires a failing baseline before any fix ships:
|
|
||||||
1. **Dispatch routing / long-session drift** — needs a long-session
|
|
||||||
elicitation rig (fresh sessions don't reproduce the pathology at CLI
|
|
||||||
0.146). Drew's stack informs the treatment shape.
|
|
||||||
2. **Verification leases / evidence receipts** — needs the
|
|
||||||
substring-aware duplicate counter added to `score_e3.py` first
|
|
||||||
(current baseline 1/23 exact-string pairs is too weak).
|
|
||||||
3. **Remediation cap** — small-n baseline (2/3 reps) needs more reps.
|
|
||||||
4. **Cross-task-race probe redesign** — `score_e5.py`'s probe is
|
|
||||||
inconclusive-by-zero by design tradeoff; needs a stronger probe.
|
|
||||||
5. **E5 D4 shell-command parser** — fix-review-scope classifier cannot
|
|
||||||
parse compound commands; scorer work, not skill work.
|
|
||||||
|
|
||||||
## Out of scope
|
|
||||||
|
|
||||||
- Adopting Drew's spinout stack (#2036/#2035) or its text.
|
|
||||||
- RoboRev, Codex token telemetry (separate codebases).
|
|
||||||
- A `close_agent` hygiene checklist (V2 has no such tool — closed as
|
|
||||||
do-not-ship in the campaign).
|
|
||||||
- Claude Code/Gemini-specific efficiency treatments beyond the T4
|
|
||||||
regression battery.
|
|
||||||
@@ -197,6 +197,17 @@ Everything you paste into a dispatch prompt — and everything a subagent
|
|||||||
prints back — stays resident in your context for the rest of the session
|
prints back — stays resident in your context for the rest of the session
|
||||||
and is re-read on every later turn. Hand artifacts over as files.
|
and is re-read on every later turn. Hand artifacts over as files.
|
||||||
|
|
||||||
|
**Waiting on dispatched subagents:** never poll a wait interface with
|
||||||
|
short timeouts, and never sit in one silent, open-ended wait either.
|
||||||
|
While you have local work — ledger updates, packaging the next review,
|
||||||
|
reading reports — keep working; child results arrive on their own.
|
||||||
|
When you are genuinely idle, wait in bounded stretches (five to ten
|
||||||
|
minutes, where your platform allows), and between stretches post one
|
||||||
|
line of status and reconcile your live children: list them, and chase
|
||||||
|
any that finished without reporting. A bounded stretch keeps nearly
|
||||||
|
all of a long wait's efficiency while guaranteeing a stuck or lost
|
||||||
|
child is noticed within minutes, not at the end of the session.
|
||||||
|
|
||||||
### 1. Dispatch the implementer
|
### 1. Dispatch the implementer
|
||||||
|
|
||||||
Record BASE (`git rev-parse HEAD`) before dispatching — the review package
|
Record BASE (`git rev-parse HEAD`) before dispatching — the review package
|
||||||
|
|||||||
@@ -7,7 +7,57 @@ Add to your Codex config (`~/.codex/config.toml`):
|
|||||||
multi_agent = true
|
multi_agent = true
|
||||||
```
|
```
|
||||||
|
|
||||||
This enables `spawn_agent`, `wait_agent`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`. When using subagent-driven-development, close reviewer subagents when their review returns. Keep each implementer subagent open until its task's review passes — the fix loop resumes the implementer — then close it. If your harness cannot send another message to a spawned agent, dispatch each fix round as a fresh implementer carrying the brief, the report file, and the findings.
|
This enables the multi-agent tools that skills like
|
||||||
|
`dispatching-parallel-agents` and `subagent-driven-development` use.
|
||||||
|
Which tools you get depends on the multi-agent version your model
|
||||||
|
preset selects (current presets run V2; older ones run V1). Trust your
|
||||||
|
actual tool list over any table — including this one — when they
|
||||||
|
disagree.
|
||||||
|
|
||||||
|
- **Spawning:** give children a clean context with
|
||||||
|
`spawn_agent {fork_turns: "none"}`; the default `"all"` copies your
|
||||||
|
entire transcript into the child. On Codex 0.145+, role files under
|
||||||
|
`~/.codex/agents/` attach to isolated forks via `agent_type`.
|
||||||
|
Full-history forks accept `model` and `reasoning_effort` overrides
|
||||||
|
(only `agent_type` is refused there) — isolated forks are the SDD
|
||||||
|
default for context hygiene, not because overrides require them.
|
||||||
|
- **Fix rounds:** resume the implementer with `followup_task` — it
|
||||||
|
delivers your message, triggers a turn, and transparently reloads a
|
||||||
|
child the harness evicted. Never dispatch a fresh implementer on the
|
||||||
|
theory that a spawned agent cannot be messaged again; on V2 it
|
||||||
|
always can.
|
||||||
|
- **Lifecycle:** V2 has no `close_agent`. Finished children are
|
||||||
|
evicted automatically when slots are needed; leaving them unclosed
|
||||||
|
costs nothing. Only V1 sessions have `close_agent` — there, close
|
||||||
|
reviewers when their review returns, and close each implementer
|
||||||
|
after its task's review passes.
|
||||||
|
- **Model names:** never copy a model name from a skill, table, or old
|
||||||
|
session into `spawn_agent` without checking it against your current
|
||||||
|
spawn allowlist — V2 accepts only V2-capable presets and hard-errors
|
||||||
|
on the rest.
|
||||||
|
|
||||||
|
## Waiting on children
|
||||||
|
|
||||||
|
`wait_agent` is an event subscription, not a poll: a long wait wakes
|
||||||
|
the moment a child produces mailbox activity, with the same latency as
|
||||||
|
a short one. Short-timeout polling buys nothing and costs a tool call —
|
||||||
|
and a context rebill — per poll. In measured sessions, roughly
|
||||||
|
two-thirds of all wait calls were short polls that timed out.
|
||||||
|
|
||||||
|
- While you still have local work, do not wait at all. A completed
|
||||||
|
child's final answer is pushed into your mailbox and arrives with
|
||||||
|
your next turn.
|
||||||
|
- When you are genuinely idle with children outstanding, wait in
|
||||||
|
bounded stretches: `wait_agent` with `timeout_ms` 300000-600000
|
||||||
|
(5-10 minutes). After each stretch — wake or timeout — post one
|
||||||
|
status line, run `list_agents`, and chase any child that finished
|
||||||
|
without reporting. Never stack polls shorter than five minutes; the
|
||||||
|
event subscription wakes a bounded stretch just as fast as a short
|
||||||
|
one.
|
||||||
|
- Completion mail cannot wake an idle controller (it is delivered
|
||||||
|
without triggering a turn); covering that idle window is
|
||||||
|
`wait_agent`'s only job. A stretch that times out with no activity
|
||||||
|
is your cue to reconcile, not to shorten the next stretch.
|
||||||
|
|
||||||
## Environment Detection
|
## Environment Detection
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user