Compare commits

..

3 Commits

Author SHA1 Message Date
Jesse Vincent
4dc71b10b3 fix(brainstorming): bounded means existing code in this repo, not a familiar app genre
Triggering battery: Claude Code classified a brand-new project bounded
3/3 by reading 'existing, understood flow' as genre familiarity — once
while explicitly noting the repo was empty. Gemini routed the same
prompt architectural 3/3.
2026-07-31 09:39:55 -07:00
Jesse Vincent
b68eaf96bb fix(brainstorming): bounded-path approval is a hard stop
Live ceremony battery: bounded reps produced zero doc ritual (the
measured win) but 2/3 implemented before any approval turn; the
bounded path now states the stop explicitly.
2026-07-31 09:39:55 -07:00
Jesse Vincent
6211388f4b feat(brainstorming): three-path router — ceremony scales, approval never does
Spike / bounded / architectural classification said out loud, one-way
upgrade ratchet, approval gate on every path. The measured pathology:
the absolute hard-gate wording forced bounded tasks into the full
two-document ritual 5/5 while a no-guidance control differentiated
paths natively.
2026-07-31 09:39:55 -07:00
3 changed files with 106 additions and 1268 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -1,252 +0,0 @@
# Codex Efficiency Fixes — Design
Date: 2026-07-30
Status: approved by Jesse (in-session)
Branch: `codex-efficiency-fixes` off `dev`
## Sources
- Eval campaign closeout: `superpowers-autoresearch/reports/2026-07-codex-efficiency-campaign.md`
(treatment table §4; every treatment below has a scorer and a measured
`dev` baseline).
- Codex source recon: `superpowers-autoresearch/docs/2026-07-29-codex-multiagent-v2-capabilities.md`
(file:line citations against the Codex CLI source; grounds T2, T3, T5).
- Published experiment write-ups: `superpowers-evals/docs/experiments/`.
- Drew's spinout stack (PRs #2036, #2035) is **evidence, not adopted text**:
Jesse wants to dig into those fixes in more detail before adopting any
of them; they inform the problem statements only.
## Goal
Ship the five evidence-strong treatments from the codex-efficiency eval
campaign as superpowers skill/doc changes, each graded against its
pre-registered criterion by the campaign's scorers before its PR is cut.
Phase 2 (everything else in the closeout treatment table) follows, each
item gated on new baseline work first.
## Scope decisions (settled with Jesse)
- **Phase 1 = the evidence-strong five** (T1T5 below). Phase 2 items
each need a failing baseline before any fix ships (discrimination
rule: inconclusive-by-zero is a stop).
- **One branch, PR per treatment.** Development and batteries happen on
`codex-efficiency-fixes`; when a treatment beats its criterion, it is
cut into its own PR against `dev` with its eval evidence. No merge
without Jesse's per-PR approval.
- **T4 ships cross-harness with a global regression battery** (Claude
Code, Codex, Gemini), variant C shape: ceremony scales, approval never
does.
## The five treatments
### T1. SDD worker-review prohibition
**Evidence:** 9/9 depth-2 spawns across 4 corpora were implementer-issued
reviewers; all 9 were same-task duplicates of the review the controller
dispatches anyway. The dispatch contract never says review is not the
worker's job; "self-review" in the implementer prompt gets reified into a
reviewer subagent on harnesses where children can spawn (Codex).
**Changes:**
- `skills/subagent-driven-development/implementer-prompt.md`: an explicit
"You do not dispatch subagents" clause — self-review means reading your
own diff; the controller owns all review dispatch; a reviewer you spawn
duplicates a review the process already provides.
- `skills/subagent-driven-development/SKILL.md`: one dispatch-contract
line in the task loop, plus a Red Flags row: "An independent review
would strengthen my report" → review is the controller's next step;
your reviewer is a duplicate seat.
- Harness-agnostic wording (no-op where children cannot spawn).
**Graded by:** `score_e6.py` (depth-2 spawns by spawner role, duplicate
review families); `score_e5.py` for the same-scope variant.
**Baseline:** 9/9 worker-issued, 0 counter-examples.
**Criterion:** 0 worker-issued depth-2 spawns AND review coverage
preserved (every task still gets exactly one controller-dispatched task
review).
### T2. Event-driven waiting
**Evidence:** 6078% of `wait_agent` calls time out in every corpus
(dev 67.1%, spinout 60.2%). Source recon: V2 waits are event
subscriptions, not polls — one long wait has the same wake latency as a
10s poll at ~1/90th the calls; a completed child's FINAL_ANSWER is pushed
into the parent's mailbox and drained into the next model request with no
wait at all.
**Changes** (`skills/using-superpowers/references/codex-tools.md`):
- Never short-timeout poll.
- While local work remains, do not wait — child results arrive with your
next turn via the mailbox.
- When genuinely idle, issue ONE `wait_agent` with a long `timeout_ms`
(900000+; harness max 3600000).
- V2 caveat stated: completion mail carries `trigger_turn=false` and will
not wake an idle controller — that is the one job `wait_agent` has.
**Graded by:** `score_e7.py` (timeout rate, inter-poll cadence,
cache-rebill estimate — the rebill figure stays labeled as an estimate).
**Baseline:** dev 67.1% timeout rate.
**Criterion:** timeout rate < 25% with no loss of task completion.
### T3. codex-tools.md corrections
**Evidence:** five claims in the current guidance are contradicted by the
Codex source (all file:line-cited in the capabilities doc):
1. `close_agent` does not exist in multi-agent V2 (V1-only). V2 LRU-evicts
finished children automatically; not closing costs nothing;
`followup_task` transparently reloads an evicted child.
2. Fix rounds can always resume the implementer via `followup_task`
dev's "if your harness cannot send another message to a spawned agent,
dispatch each fix round as a fresh implementer" branch is dead on V2.
3. Role files (`~/.codex/agents/**.toml`) DO attach to spawns via
`agent_type` on isolated forks (0.145+).
4. Full-history forks accept `model`/`reasoning_effort` overrides; only
`agent_type` is refused. (Isolated forks remain the SDD guidance for
context-hygiene reasons, stated accurately.)
5. Dispatch guidance must never name non-V2 model presets — the V2 spawn
allowlist is v2 presets only; others hard-error.
**Changes:** rewrite the multi-agent paragraph of
`skills/using-superpowers/references/codex-tools.md` to be
version-honest (V1 vs V2 behavior labeled where they differ).
**Graded by:** source citation (already verified); no scorer regressions
on the shared battery. `score_e8.py` is retained as a V1/V2 schema
detector, not a hygiene grader — no `close_agent` checklist ships.
### T4. Brainstorming three-path router (variant C: approval always)
**Evidence:** micro — the current HARD-GATE text pushes a bounded task to
FULL ceremony 5/5, while Z-null (no guidance) and a three-path router
both differentiate 5/5: the absolute wording suppresses discrimination
the model draws natively. FULL battery — ceremony volume scales
moderately (16.7 vs 24.0 tool calls, bounded vs arch), but the
two-document ritual (spec file → plan file) ran unconditionally in every
rep. The measured waste is the unconditional artifact ritual, not the
approval gate.
**Design (variant C):** three paths scale the ARTIFACT; every path keeps
human approval before implementation:
- **Spike** (feasibility question, explicitly throwaway): present the
question and the intended probe in 23 sentences, get a nod, go. No
docs. Findings return as a recommendation; anything built stays labeled
throwaway.
- **Bounded** (well-scoped change to an existing, understood flow):
present a short design in chat, get approval, implement. No spec file,
no writing-plans invocation.
- **Architectural** (restructures components, new subsystem, public
interface change): the full current flow — spec doc, review,
writing-plans.
**Guards (all ship with the router):**
- Classification is said out loud ("this looks bounded, so I'll present a
short design here rather than write a spec") so the human can override.
- When in doubt between two paths, take the heavier one.
- One-way ratchet: hidden complexity discovered mid-path upgrades the
path; never downgrade mid-task.
- New Red Flags rows targeting classification-as-escape-hatch ("I'll call
it bounded to skip the doc").
**Changes** (`skills/brainstorming/SKILL.md`): HARD-GATE keeps "no
implementation before approval" and drops "regardless of perceived
simplicity" as the ceremony driver; anti-pattern section reframed (the
sin is skipping approval, not skipping documents); checklist steps 69
become the architectural path; process-flow graph gains the router; Red
Flags rows added. This is carefully-tuned content — the edit follows
writing-skills methodology and ships only with the full eval evidence
below.
**Graded by (three layers):**
1. **Micro** (`ceremony-path-micro.py`, adapted): variant C literal text,
plus adversarially ambiguous briefs the campaign never tested (a task
that pattern-matches bounded but hides a public interface change).
Criteria: spike/bounded/arch differentiate (≥4/5 per cell); ambiguous
briefs escalate to FULL (≥4/5); arch never downgrades (5/5).
2. **Codex ceremony battery:** `cx-ceremony-{spike,bounded,arch}` on the
fix arm, 3 reps each, `score_e4.py` census. Criteria: bounded reps
show an approval turn but zero committed spec files and zero
writing-plans ritual; arch reps keep the full two-doc flow; spike reps
stay minimal.
3. **Global regression battery:** the same three ceremony scenarios on
Claude Code and Gemini (rig work: those scenarios are currently
codex-gated), 3 reps each; plus the triggering acceptance check
("Let's make a react todo list" auto-triggers brainstorming into the
full/architectural path) on all three harnesses.
### T5. Explicit model on child-issued spawns
**Evidence:** root spawns are 100% explicit-model at CLI 0.146 (dev
14/14); the live gap is depth-2 — 2/2 child-issued spawns omitted
`model`. Source recon: `model` without `reasoning_effort` resets effort
to the MODEL's default, not the parent's.
**Changes** (`skills/using-superpowers/references/codex-tools.md`):
- Every spawn you issue — including as a child — sets `model` AND
`reasoning_effort`; the effort-reset trap is named.
- Advise `[agents].default_subagent_model` and
`[agents].default_subagent_reasoning_effort` in `~/.codex/config.toml`
as the machine-level backstop for anything that slips through.
**Graded by:** `score_e1.py` (per-spawn explicit-model rate, by depth) on
the shared battery.
**Baseline:** depth-2: 0/2 explicit.
**Criterion:** every spawn at every depth carries explicit model +
effort. Pre-registered caveat: if T1 eliminates depth-2 spawns entirely,
T5 grades as root-spawn regression (hold 100%) plus doc correctness and
is recorded inconclusive-by-zero at depth-2 — the config backstop is then
the operative mechanism.
## Grading plan
- **Shared SDD battery** carries T1, T2, T5: `cx-sdd-small`, fix-branch
arm (`/tmp/sp-arm-fix`), 8 reps across both container lanes. Dev
baselines are already measured; no baseline re-runs.
- **T4 batteries** as listed above (micro + codex ceremony + global
regression).
- **Pre-registration:** every battery gets a hypothesis-log entry
(prediction, scorer, criterion) in
`superpowers-autoresearch/logs/2026-07-30-codex-efficiency-fixes.md`
BEFORE it runs. Standing rules carry over: append-only log, manual
inspection of scorer matches on fix-arm runs (non-circular
verification), no raw rollouts committed, correctness rides beside
cost in every verdict.
- **Attribution:** orthogonal scorers on one combined branch; unexpected
regressions bisect by treatment commit.
- **Budget:** shared battery ~$40, codex ceremony ~$40, global
regression ~$4080, micros ~$5 → phase 1 ≈ $150200 of the ~$850
remaining from the campaign's $1000.
## Process
- Work happens in the `codex-efficiency-fixes` worktree (branched off
`dev`); execution via subagent-driven-development from a written plan.
- Skill-text changes follow writing-skills methodology.
- Scenario/rig changes (un-gating ceremony scenarios for Claude
Code/Gemini, adversarial micro briefs) land in `superpowers-evals`
main, as authorized.
- PR-per-treatment against `dev`, each with its eval evidence and the
standard identification block; merges only on Jesse's per-PR approval.
## Phase 2 queue (baseline-first; not in this plan's tasks)
Each item requires a failing baseline before any fix ships:
1. **Dispatch routing / long-session drift** — needs a long-session
elicitation rig (fresh sessions don't reproduce the pathology at CLI
0.146). Drew's stack informs the treatment shape.
2. **Verification leases / evidence receipts** — needs the
substring-aware duplicate counter added to `score_e3.py` first
(current baseline 1/23 exact-string pairs is too weak).
3. **Remediation cap** — small-n baseline (2/3 reps) needs more reps.
4. **Cross-task-race probe redesign**`score_e5.py`'s probe is
inconclusive-by-zero by design tradeoff; needs a stronger probe.
5. **E5 D4 shell-command parser** — fix-review-scope classifier cannot
parse compound commands; scorer work, not skill work.
## Out of scope
- Adopting Drew's spinout stack (#2036/#2035) or its text.
- RoboRev, Codex token telemetry (separate codebases).
- A `close_agent` hygiene checklist (V2 has no such tool — closed as
do-not-ship in the campaign).
- Claude Code/Gemini-specific efficiency treatments beyond the T4
regression battery.

View File

@@ -7,20 +7,91 @@ description: "You MUST use this before any creative work - creating features, bu
Help turn ideas into fully formed designs and specs through natural collaborative dialogue.
Start by understanding the current project context, then ask questions one at a time to refine the idea. Once you understand what you're building, present the design and get user approval.
Start by classifying how much process the request needs, then work
through your path: understand the context, refine the idea, present a
design, and get your human partner's approval.
<HARD-GATE>
Do NOT invoke any implementation skill, write any code, scaffold any project, or take any implementation action until you have presented a design and the user has approved it. This applies to EVERY project regardless of perceived simplicity.
Do NOT invoke any implementation skill, write any code, scaffold any
project, or take any implementation action until you have told your
human partner what you intend and they have approved it. This applies
to EVERY task on EVERY path below — the ceremony scales with the task;
the approval gate never does.
</HARD-GATE>
## Anti-Pattern: "This Is Too Simple To Need A Design"
## Three Paths
Every project goes through this process. A todo list, a single-function utility, a config change — all of them. "Simple" projects are where unexamined assumptions cause the most wasted work. The design can be short (a few sentences for truly simple projects), but you MUST present it and get approval.
Before your first question, classify the request and say the
classification out loud — "this looks bounded, so I'll present a short
design here rather than write a spec" — so your human partner can
override it:
- **Spike** — a feasibility question ("can we...", "is it possible...",
"quick and dirty is fine") whose output is an answer, not code you
keep. Present the question and what you'll try in 2-3 sentences, get
a nod, then find out as cheaply as correctness allows. No design
doc, no spec file. Report findings as a recommendation; anything you
built stays labeled throwaway.
- **Bounded** — a well-scoped change to code that already exists in
this repo: a new flag, a small endpoint, a one-file fix.
Understanding the kind of app is not enough — bounded means the flow
you are changing is already here to read. If there is no existing
flow to change, the task is not bounded. Ask the clarifying
questions that matter, present a short design IN CHAT (a few
sentences to a few short paragraphs), and STOP. Implementation
starts only after your human partner says yes to that design — a
bounded task's approval is as hard a gate as an architectural
one. No spec file, no implementation plan document.
- **Architectural** — new projects, new subsystems, changes that
restructure how components fit together or alter interfaces others
depend on. Follow the full process: questions, approaches, sectioned
design, written spec, then the writing-plans skill.
When in doubt between two paths, take the heavier one. The ratchet is
one-way: hidden complexity discovered mid-task upgrades the path —
stop, say so, and step up. Nothing downgrades mid-task.
## Anti-Pattern: "Too Simple To Need Approval"
Every path ends with your human partner approving your intent before
implementation. A todo list, a single-function utility, a config
change — the design may be two sentences in chat, but you MUST present
it and get approval. "Simple" tasks are where unexamined assumptions
cause the most wasted work. What scales with simplicity is the
artifact, never the approval.
## Red Flags
| Thought | Reality |
|---------|---------|
| "This is too simple to need a design" | Simple means a short design, not no design. Two sentences in chat, then approval. |
| "I'll call it bounded and skip the spec" | Reaching for a label to skip work IS the doubt — take the heavier path. |
| "It's bounded and the design is obvious — I'll start while they read it" | The gate is the approval, not the design's length. Present, then stop until you hear yes. |
| "I understand this kind of app, so it's bounded" | Bounded measures the repo, not your familiarity. A new project has no existing flow — it is architectural. |
| "The spike works, so I'll keep the code" | A spike's output is an answer. Keeping the code is a new request — classify it. |
| "It grew, but I'm almost done — no need to re-classify" | Hidden complexity upgrades the path mid-task. Stop and say so. |
| "They approved the spike, so the follow-up change is approved too" | Each task gets its own classification and its own approval. |
## Checklist
You MUST create a task for each of these items and complete them in order:
Classify first, announce the path, then create a task for each item on
your path and complete them in order.
**Spike:**
1. **Explore project context** — enough to frame the probe
2. **Present question + probe plan** — 2-3 sentences
3. **Get approval** — a nod is enough
4. **Investigate** — as cheaply as correctness allows
5. **Report findings** — a recommendation; label anything built as throwaway
**Bounded:**
1. **Explore project context** — check files, docs, recent commits
2. **Ask clarifying questions** — one at a time, the ones that matter
3. **Present short design in chat** — approach, files touched, testing
4. **Get approval** — STOP and wait for an explicit yes; presenting the design and starting in the same breath is skipping the gate
5. **Implement** — proceed with the normal development workflow (TDD applies); no plan document
**Architectural:**
1. **Explore project context** — check files, docs, recent commits
2. **Offer the visual companion just-in-time** — NOT upfront. The first time a question would genuinely be clearer shown than described, offer it then (its own message); on approval its browser tab opens for you. If no visual question ever arises, never offer it. See the Visual Companion section below.
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
@@ -35,6 +106,13 @@ You MUST create a task for each of these items and complete them in order:
```dot
digraph brainstorming {
"Classify: spike / bounded / architectural" [shape=diamond];
"Present question + probe (2-3 sentences)" [shape=box];
"Ask clarifying questions (bounded)" [shape=box];
"Present short design in chat" [shape=box];
"Human approves?" [shape=diamond];
"Investigate; report recommendation" [shape=doublecircle];
"Implement via normal workflow (no plan doc)" [shape=doublecircle];
"Explore project context" [shape=box];
"Ask clarifying questions" [shape=box];
"Propose 2-3 approaches" [shape=box];
@@ -44,7 +122,17 @@ digraph brainstorming {
"Spec self-review\n(fix inline)" [shape=box];
"User reviews spec?" [shape=diamond];
"Invoke writing-plans skill" [shape=doublecircle];
"Hidden complexity? Upgrade path" [shape=box];
"Classify: spike / bounded / architectural" -> "Present question + probe (2-3 sentences)" [label="spike"];
"Classify: spike / bounded / architectural" -> "Ask clarifying questions (bounded)" [label="bounded"];
"Classify: spike / bounded / architectural" -> "Explore project context" [label="architectural"];
"Present question + probe (2-3 sentences)" -> "Human approves?";
"Ask clarifying questions (bounded)" -> "Present short design in chat";
"Present short design in chat" -> "Human approves?";
"Human approves?" -> "Investigate; report recommendation" [label="spike: yes"];
"Human approves?" -> "Implement via normal workflow (no plan doc)" [label="bounded: yes"];
"Hidden complexity? Upgrade path" -> "Classify: spike / bounded / architectural";
"Explore project context" -> "Ask clarifying questions";
"Ask clarifying questions" -> "Propose 2-3 approaches";
"Propose 2-3 approaches" -> "Present design sections";
@@ -58,10 +146,21 @@ digraph brainstorming {
}
```
**The terminal state is invoking writing-plans.** Do NOT invoke frontend-design, mcp-builder, or any other implementation skill. The ONLY skill you invoke after brainstorming is writing-plans.
**Terminal states are path-bound.** Architectural: the ONLY skill you
invoke after brainstorming is writing-plans — never frontend-design,
mcp-builder, or any other implementation skill. Bounded: after
approval, implementation proceeds directly through the normal
development workflow; no plan document. Spike: the terminal state is a
reported recommendation.
## The Process
The subsections below serve the bounded and architectural paths (a
spike stops at "present the probe, get a nod"). Sections from
**Exploring approaches** onward are architectural-path depth — for
bounded work, context plus a few questions plus a short in-chat design
is the whole process.
**Understanding the idea:**
- Check out the current project state first (files, docs, recent commits)
@@ -100,7 +199,7 @@ digraph brainstorming {
- Where existing code has problems that affect the work (e.g., a file that's grown too large, unclear boundaries, tangled responsibilities), include targeted improvements as part of the design - the way a good developer improves code they're working in.
- Don't propose unrelated refactoring. Stay focused on what serves the current goal.
## After the Design
## After the Design (architectural path)
**Documentation:**