mirror of
https://github.com/obra/superpowers.git
synced 2026-07-09 03:29:04 +08:00
docs(specs): record eval re-scope — blind adoption did not reproduce, claims narrowed
25/25 baseline reps refused the stale foreign ledger via git forensics; the spec's evaluation section now states the honest claims: structural fix + measured disambiguation-cost delta + same-plan-resume regression gate, shipping with explicit maintainer sign-off in place of a failing S1 baseline.
This commit is contained in:
@@ -141,32 +141,43 @@ plan's cleanup.
|
|||||||
`-integration.sh` audited for old-path expectations (none found in initial
|
`-integration.sh` audited for old-path expectations (none found in initial
|
||||||
grep; audit is a task gate anyway).
|
grep; audit is a task gate anyway).
|
||||||
|
|
||||||
### Evaluation (writing-skills RED → GREEN, the "basic eval")
|
### Evaluation (writing-skills RED → GREEN, re-scoped 2026-07-06)
|
||||||
|
|
||||||
Pressure scenarios run as fresh subagent sessions against a fixture repo in a
|
Pressure scenarios run as fresh sonnet subagent sessions against fixture repos
|
||||||
temp directory (never inside this worktree). Fixture: git repo with plan A
|
in temp directories (never inside this worktree), compaction-resume framing,
|
||||||
(17-task backend plan) and plan B (5-task follow-up), plus SDD workspace state
|
each rep hand-scored; the measured output is the controller's resume decision
|
||||||
as each scenario dictates. The agent under test is pointed at a specific
|
(no real implementer dispatches).
|
||||||
SKILL.md text (old = `git show` of the released text; new = this branch's) and
|
|
||||||
plan B, and asked to state concretely which task it starts with and what it
|
|
||||||
does about existing ledger state. No real implementer dispatches — the measured
|
|
||||||
output is the controller's resume decision.
|
|
||||||
|
|
||||||
- **S1 — stale ledger from a different plan (the reported bug):**
|
**RED outcome that forced the re-scope (maintainer decision, Jesse,
|
||||||
workspace contains plan A's completed ledger in the layout the skill text
|
2026-07-06):** the originally hypothesized failure — a controller blindly
|
||||||
under test prescribes. PASS = starts plan B at Task 1 (or explicitly
|
adopting a stale foreign ledger as its own progress — did **not** reproduce:
|
||||||
identifies the ledger as another plan's); FAIL = resumes past plan B tasks,
|
25/25 reps across three framings (fresh session, may-be-resumed, faithful
|
||||||
claims tasks complete, or adopts plan A's ledger.
|
post-compaction resume with the skill's "trust the ledger" line active)
|
||||||
- **S2 — same-plan resume (regression guard):** ledger for plan B marks Tasks
|
forensically cross-checked the ledger's cited commits against git history and
|
||||||
1–2 complete. PASS = resumes at Task 3 without re-dispatching 1–2. This
|
the plan files, refused the foreign ledger, and started plan B at Task 1 —
|
||||||
protects the ledger's original purpose; the fix must not break it.
|
spending 6–13 tool calls of cross-plan forensics per resume to do so. Two
|
||||||
|
fixture iterations were burned proving this honestly (v1: fabricated hashes
|
||||||
|
were dismissed on sight; v2: stub implementations were ruled false "review
|
||||||
|
clean" records — the S2 control failed both times). Full record in the
|
||||||
|
committed eval docs.
|
||||||
|
|
||||||
|
**Re-scoped claims and gates:**
|
||||||
|
|
||||||
|
- The change ships on the structural record (collisions, improvised side-band
|
||||||
|
names, overwritten briefs, git contamination — serf repo) plus the measured
|
||||||
|
disambiguation tax, with explicit maintainer sign-off standing in for the
|
||||||
|
writing-skills failing-baseline requirement on the SKILL.md text.
|
||||||
|
- **S1 GREEN (5/5 required):** stale plan-A workspace present in the new
|
||||||
|
scoped layout plus legacy flat litter; a resumed controller on plan B
|
||||||
|
resolves its own plan-scoped workspace directly and starts at Task 1;
|
||||||
|
per-rep `tool_uses` recorded against the RED baseline (7/13/9/10/6) as the
|
||||||
|
cost delta.
|
||||||
|
- **S2 RED control (≥4/5 required) and S2 GREEN (5/5 required)** on a
|
||||||
|
truthful v3 fixture (cited commits genuinely implement their tasks' specs,
|
||||||
|
rotating authors, spread timestamps): legitimate same-plan resume — tasks
|
||||||
|
1–2 recognized, Task 3 dispatched. This protects the ledger's original
|
||||||
|
purpose; the fix must not break it, and the control validates the fixture.
|
||||||
|
|
||||||
Reps: 5 per scenario per arm (RED = current released text, GREEN = new text),
|
|
||||||
every response read and scored by hand against the PASS/FAIL criteria above;
|
|
||||||
verbatim failure rationalizations captured. Expected: S1 RED fails ≥1/5 and
|
|
||||||
plausibly most runs (any failure validates the bug; if S1 RED passes 5/5, STOP
|
|
||||||
and reassess with Jesse before editing skill text — per writing-skills, no
|
|
||||||
skill change without a failing test). S1 GREEN and S2 both arms must pass 5/5.
|
|
||||||
Results land in `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md`
|
Results land in `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md`
|
||||||
and are summarized in the PR.
|
and are summarized in the PR.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user