Compare commits

..

1 Commits

Author SHA1 Message Date
Jesse Vincent
3ff8d15f15 docs: codex-efficiency fix-cycle spec and plan (campaign record) 2026-07-31 10:52:05 -07:00
7 changed files with 1261 additions and 46 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,252 @@
# Codex Efficiency Fixes — Design
Date: 2026-07-30
Status: approved by Jesse (in-session)
Branch: `codex-efficiency-fixes` off `dev`
## Sources
- Eval campaign closeout: `superpowers-autoresearch/reports/2026-07-codex-efficiency-campaign.md`
(treatment table §4; every treatment below has a scorer and a measured
`dev` baseline).
- Codex source recon: `superpowers-autoresearch/docs/2026-07-29-codex-multiagent-v2-capabilities.md`
(file:line citations against the Codex CLI source; grounds T2, T3, T5).
- Published experiment write-ups: `superpowers-evals/docs/experiments/`.
- Drew's spinout stack (PRs #2036, #2035) is **evidence, not adopted text**:
Jesse wants to dig into those fixes in more detail before adopting any
of them; they inform the problem statements only.
## Goal
Ship the five evidence-strong treatments from the codex-efficiency eval
campaign as superpowers skill/doc changes, each graded against its
pre-registered criterion by the campaign's scorers before its PR is cut.
Phase 2 (everything else in the closeout treatment table) follows, each
item gated on new baseline work first.
## Scope decisions (settled with Jesse)
- **Phase 1 = the evidence-strong five** (T1T5 below). Phase 2 items
each need a failing baseline before any fix ships (discrimination
rule: inconclusive-by-zero is a stop).
- **One branch, PR per treatment.** Development and batteries happen on
`codex-efficiency-fixes`; when a treatment beats its criterion, it is
cut into its own PR against `dev` with its eval evidence. No merge
without Jesse's per-PR approval.
- **T4 ships cross-harness with a global regression battery** (Claude
Code, Codex, Gemini), variant C shape: ceremony scales, approval never
does.
## The five treatments
### T1. SDD worker-review prohibition
**Evidence:** 9/9 depth-2 spawns across 4 corpora were implementer-issued
reviewers; all 9 were same-task duplicates of the review the controller
dispatches anyway. The dispatch contract never says review is not the
worker's job; "self-review" in the implementer prompt gets reified into a
reviewer subagent on harnesses where children can spawn (Codex).
**Changes:**
- `skills/subagent-driven-development/implementer-prompt.md`: an explicit
"You do not dispatch subagents" clause — self-review means reading your
own diff; the controller owns all review dispatch; a reviewer you spawn
duplicates a review the process already provides.
- `skills/subagent-driven-development/SKILL.md`: one dispatch-contract
line in the task loop, plus a Red Flags row: "An independent review
would strengthen my report" → review is the controller's next step;
your reviewer is a duplicate seat.
- Harness-agnostic wording (no-op where children cannot spawn).
**Graded by:** `score_e6.py` (depth-2 spawns by spawner role, duplicate
review families); `score_e5.py` for the same-scope variant.
**Baseline:** 9/9 worker-issued, 0 counter-examples.
**Criterion:** 0 worker-issued depth-2 spawns AND review coverage
preserved (every task still gets exactly one controller-dispatched task
review).
### T2. Event-driven waiting
**Evidence:** 6078% of `wait_agent` calls time out in every corpus
(dev 67.1%, spinout 60.2%). Source recon: V2 waits are event
subscriptions, not polls — one long wait has the same wake latency as a
10s poll at ~1/90th the calls; a completed child's FINAL_ANSWER is pushed
into the parent's mailbox and drained into the next model request with no
wait at all.
**Changes** (`skills/using-superpowers/references/codex-tools.md`):
- Never short-timeout poll.
- While local work remains, do not wait — child results arrive with your
next turn via the mailbox.
- When genuinely idle, issue ONE `wait_agent` with a long `timeout_ms`
(900000+; harness max 3600000).
- V2 caveat stated: completion mail carries `trigger_turn=false` and will
not wake an idle controller — that is the one job `wait_agent` has.
**Graded by:** `score_e7.py` (timeout rate, inter-poll cadence,
cache-rebill estimate — the rebill figure stays labeled as an estimate).
**Baseline:** dev 67.1% timeout rate.
**Criterion:** timeout rate < 25% with no loss of task completion.
### T3. codex-tools.md corrections
**Evidence:** five claims in the current guidance are contradicted by the
Codex source (all file:line-cited in the capabilities doc):
1. `close_agent` does not exist in multi-agent V2 (V1-only). V2 LRU-evicts
finished children automatically; not closing costs nothing;
`followup_task` transparently reloads an evicted child.
2. Fix rounds can always resume the implementer via `followup_task`
dev's "if your harness cannot send another message to a spawned agent,
dispatch each fix round as a fresh implementer" branch is dead on V2.
3. Role files (`~/.codex/agents/**.toml`) DO attach to spawns via
`agent_type` on isolated forks (0.145+).
4. Full-history forks accept `model`/`reasoning_effort` overrides; only
`agent_type` is refused. (Isolated forks remain the SDD guidance for
context-hygiene reasons, stated accurately.)
5. Dispatch guidance must never name non-V2 model presets — the V2 spawn
allowlist is v2 presets only; others hard-error.
**Changes:** rewrite the multi-agent paragraph of
`skills/using-superpowers/references/codex-tools.md` to be
version-honest (V1 vs V2 behavior labeled where they differ).
**Graded by:** source citation (already verified); no scorer regressions
on the shared battery. `score_e8.py` is retained as a V1/V2 schema
detector, not a hygiene grader — no `close_agent` checklist ships.
### T4. Brainstorming three-path router (variant C: approval always)
**Evidence:** micro — the current HARD-GATE text pushes a bounded task to
FULL ceremony 5/5, while Z-null (no guidance) and a three-path router
both differentiate 5/5: the absolute wording suppresses discrimination
the model draws natively. FULL battery — ceremony volume scales
moderately (16.7 vs 24.0 tool calls, bounded vs arch), but the
two-document ritual (spec file → plan file) ran unconditionally in every
rep. The measured waste is the unconditional artifact ritual, not the
approval gate.
**Design (variant C):** three paths scale the ARTIFACT; every path keeps
human approval before implementation:
- **Spike** (feasibility question, explicitly throwaway): present the
question and the intended probe in 23 sentences, get a nod, go. No
docs. Findings return as a recommendation; anything built stays labeled
throwaway.
- **Bounded** (well-scoped change to an existing, understood flow):
present a short design in chat, get approval, implement. No spec file,
no writing-plans invocation.
- **Architectural** (restructures components, new subsystem, public
interface change): the full current flow — spec doc, review,
writing-plans.
**Guards (all ship with the router):**
- Classification is said out loud ("this looks bounded, so I'll present a
short design here rather than write a spec") so the human can override.
- When in doubt between two paths, take the heavier one.
- One-way ratchet: hidden complexity discovered mid-path upgrades the
path; never downgrade mid-task.
- New Red Flags rows targeting classification-as-escape-hatch ("I'll call
it bounded to skip the doc").
**Changes** (`skills/brainstorming/SKILL.md`): HARD-GATE keeps "no
implementation before approval" and drops "regardless of perceived
simplicity" as the ceremony driver; anti-pattern section reframed (the
sin is skipping approval, not skipping documents); checklist steps 69
become the architectural path; process-flow graph gains the router; Red
Flags rows added. This is carefully-tuned content — the edit follows
writing-skills methodology and ships only with the full eval evidence
below.
**Graded by (three layers):**
1. **Micro** (`ceremony-path-micro.py`, adapted): variant C literal text,
plus adversarially ambiguous briefs the campaign never tested (a task
that pattern-matches bounded but hides a public interface change).
Criteria: spike/bounded/arch differentiate (≥4/5 per cell); ambiguous
briefs escalate to FULL (≥4/5); arch never downgrades (5/5).
2. **Codex ceremony battery:** `cx-ceremony-{spike,bounded,arch}` on the
fix arm, 3 reps each, `score_e4.py` census. Criteria: bounded reps
show an approval turn but zero committed spec files and zero
writing-plans ritual; arch reps keep the full two-doc flow; spike reps
stay minimal.
3. **Global regression battery:** the same three ceremony scenarios on
Claude Code and Gemini (rig work: those scenarios are currently
codex-gated), 3 reps each; plus the triggering acceptance check
("Let's make a react todo list" auto-triggers brainstorming into the
full/architectural path) on all three harnesses.
### T5. Explicit model on child-issued spawns
**Evidence:** root spawns are 100% explicit-model at CLI 0.146 (dev
14/14); the live gap is depth-2 — 2/2 child-issued spawns omitted
`model`. Source recon: `model` without `reasoning_effort` resets effort
to the MODEL's default, not the parent's.
**Changes** (`skills/using-superpowers/references/codex-tools.md`):
- Every spawn you issue — including as a child — sets `model` AND
`reasoning_effort`; the effort-reset trap is named.
- Advise `[agents].default_subagent_model` and
`[agents].default_subagent_reasoning_effort` in `~/.codex/config.toml`
as the machine-level backstop for anything that slips through.
**Graded by:** `score_e1.py` (per-spawn explicit-model rate, by depth) on
the shared battery.
**Baseline:** depth-2: 0/2 explicit.
**Criterion:** every spawn at every depth carries explicit model +
effort. Pre-registered caveat: if T1 eliminates depth-2 spawns entirely,
T5 grades as root-spawn regression (hold 100%) plus doc correctness and
is recorded inconclusive-by-zero at depth-2 — the config backstop is then
the operative mechanism.
## Grading plan
- **Shared SDD battery** carries T1, T2, T5: `cx-sdd-small`, fix-branch
arm (`/tmp/sp-arm-fix`), 8 reps across both container lanes. Dev
baselines are already measured; no baseline re-runs.
- **T4 batteries** as listed above (micro + codex ceremony + global
regression).
- **Pre-registration:** every battery gets a hypothesis-log entry
(prediction, scorer, criterion) in
`superpowers-autoresearch/logs/2026-07-30-codex-efficiency-fixes.md`
BEFORE it runs. Standing rules carry over: append-only log, manual
inspection of scorer matches on fix-arm runs (non-circular
verification), no raw rollouts committed, correctness rides beside
cost in every verdict.
- **Attribution:** orthogonal scorers on one combined branch; unexpected
regressions bisect by treatment commit.
- **Budget:** shared battery ~$40, codex ceremony ~$40, global
regression ~$4080, micros ~$5 → phase 1 ≈ $150200 of the ~$850
remaining from the campaign's $1000.
## Process
- Work happens in the `codex-efficiency-fixes` worktree (branched off
`dev`); execution via subagent-driven-development from a written plan.
- Skill-text changes follow writing-skills methodology.
- Scenario/rig changes (un-gating ceremony scenarios for Claude
Code/Gemini, adversarial micro briefs) land in `superpowers-evals`
main, as authorized.
- PR-per-treatment against `dev`, each with its eval evidence and the
standard identification block; merges only on Jesse's per-PR approval.
## Phase 2 queue (baseline-first; not in this plan's tasks)
Each item requires a failing baseline before any fix ships:
1. **Dispatch routing / long-session drift** — needs a long-session
elicitation rig (fresh sessions don't reproduce the pathology at CLI
0.146). Drew's stack informs the treatment shape.
2. **Verification leases / evidence receipts** — needs the
substring-aware duplicate counter added to `score_e3.py` first
(current baseline 1/23 exact-string pairs is too weak).
3. **Remediation cap** — small-n baseline (2/3 reps) needs more reps.
4. **Cross-task-race probe redesign**`score_e5.py`'s probe is
inconclusive-by-zero by design tradeoff; needs a stronger probe.
5. **E5 D4 shell-command parser** — fix-review-scope classifier cannot
parse compound commands; scorer work, not skill work.
## Out of scope
- Adopting Drew's spinout stack (#2036/#2035) or its text.
- RoboRev, Codex token telemetry (separate codebases).
- A `close_agent` hygiene checklist (V2 has no such tool — closed as
do-not-ship in the campaign).
- Claude Code/Gemini-specific efficiency treatments beyond the T4
regression battery.

View File

@@ -34,15 +34,6 @@ Subagent (general-purpose):
Your review is read-only on this checkout. Do not mutate the working tree, the index, HEAD, or branch state in any way. Use tools like `git show`, `git diff`, and `git log` to inspect history. If you need a working copy of a different revision, check it out into a separate temporary directory (e.g. `git worktree add /tmp/review-[SHA] [SHA]`) — never move HEAD on this checkout. Your review is read-only on this checkout. Do not mutate the working tree, the index, HEAD, or branch state in any way. Use tools like `git show`, `git diff`, and `git log` to inspect history. If you need a working copy of a different revision, check it out into a separate temporary directory (e.g. `git worktree add /tmp/review-[SHA] [SHA]`) — never move HEAD on this checkout.
## You Do Not Dispatch Subagents
Do all of this review yourself. Never spawn a subagent to review part
of the diff, and never spawn another reviewer for a second opinion.
This process already provides every review seat the work gets; a
reviewer you spawn duplicates one of them at full cost, and its
verdict counts for nothing. If the diff feels too large for one
pass, review it in passes yourself and say so in your report.
## What to Check ## What to Check
**Plan alignment:** **Plan alignment:**

View File

@@ -223,12 +223,6 @@ and fix-round diffs need it.
later dispatches — a real session's dispatch hit 42k chars of which 99% later dispatches — a real session's dispatch hit 42k chars of which 99%
was pasted history. A fresh subagent needs its task, the interfaces it was pasted history. A fresh subagent needs its task, the interfaces it
touches, and the global constraints. Nothing else. touches, and the global constraints. Nothing else.
- The dispatch carries the no-subagents contract (it is in the
implementer template): the implementer never dispatches subagents —
not helpers, and never a reviewer. Review arrives from you, after the
report. In real sessions, every reviewer a worker spawned duplicated
the task review the controller dispatched anyway — a full extra
review seat per task.
- If an earlier task parked a finding in the area this task touches, carry - If an earlier task parked a finding in the area this task touches, carry
a pointer to that ledger entry in the dispatch. a pointer to that ledger entry in the dispatch.
- Record the implementer's agent identity from the dispatch result — - Record the implementer's agent identity from the dispatch result —
@@ -440,7 +434,6 @@ Use superpowers:finishing-a-development-branch.
| "The fix was small, skip the re-review" | Unreviewed fixes are how regressions land. Every round ends with a scoped re-review. | | "The fix was small, skip the re-review" | Unreviewed fixes are how regressions land. Every round ends with a scoped re-review. |
| "Reviews slow the loop down" | The loop without reviews is just unverified churn. Reviews are the loop's brakes and steering. | | "Reviews slow the loop down" | The loop without reviews is just unverified churn. Reviews are the loop's brakes and steering. |
| "Ledger bookkeeping is overhead" | The ledger is what survives compaction. Controllers without one have re-dispatched entire completed task sequences. | | "Ledger bookkeeping is overhead" | The ledger is what survives compaction. Controllers without one have re-dispatched entire completed task sequences. |
| "The implementer spawned its own reviewer — free extra assurance" | It's a duplicate seat reviewing the same diff; the task review is the gate. A worker-spawned reviewer is a defect to flag, not rigor. |
## Example Workflow ## Example Workflow

View File

@@ -47,18 +47,6 @@ Subagent (general-purpose):
While iterating, run the focused test for what you're changing; run the While iterating, run the focused test for what you're changing; run the
full suite once before committing, not after every edit. full suite once before committing, not after every edit.
## You Do Not Dispatch Subagents
Do all of this task's work yourself. Never spawn a subagent to
implement part of the task, and above all never spawn a reviewer to
check your work. Self-review (below) means reading your own diff.
Review is the controller's job: after you report, it dispatches a
fresh reviewer against your diff. A reviewer you spawn duplicates
that review at full cost, and its approval counts for nothing in
the process. If you catch yourself thinking "an independent review
would strengthen my report" — that review is already scheduled.
Report instead.
## Code Organization ## Code Organization
You reason best about code you can hold in context at once, and your edits are more You reason best about code you can hold in context at once, and your edits are more

View File

@@ -43,15 +43,6 @@ Subagent (general-purpose):
Your review is read-only on this checkout. Do not mutate the working Your review is read-only on this checkout. Do not mutate the working
tree, the index, HEAD, or branch state in any way. tree, the index, HEAD, or branch state in any way.
## You Do Not Dispatch Subagents
Do all of this review yourself. Never spawn a subagent to review part
of the diff, and never spawn another reviewer for a second opinion.
This process already provides every review seat the work gets; a
reviewer you spawn duplicates one of them at full cost, and its
verdict counts for nothing. If the diff feels too large for one
pass, review it in passes yourself and say so in your report.
## Scope ## Scope
Your scope is the findings list and the fix diff. Verdict every finding. Your scope is the findings list and the fix diff. Verdict every finding.

View File

@@ -52,15 +52,6 @@ Subagent (general-purpose):
Your review is read-only on this checkout. Do not mutate the working Your review is read-only on this checkout. Do not mutate the working
tree, the index, HEAD, or branch state in any way. tree, the index, HEAD, or branch state in any way.
## You Do Not Dispatch Subagents
Do all of this review yourself. Never spawn a subagent to review part
of the diff, and never spawn another reviewer for a second opinion.
This process already provides every review seat the work gets; a
reviewer you spawn duplicates one of them at full cost, and its
verdict counts for nothing. If the diff feels too large for one
pass, review it in passes yourself and say so in your report.
## Do Not Trust the Report ## Do Not Trust the Report
Treat the implementer's report as unverified claims about the code. It Treat the implementer's report as unverified claims about the code. It