From 82cc596a42b44b484269ed7a94d99f8876e5129c Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 13:18:24 -0700 Subject: [PATCH] =?UTF-8?q?eval(sdd):=20GREEN=20results=20=E2=80=94=20plan?= =?UTF-8?q?-scoped=20resolution=20replaces=20cross-plan=20forensics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...dd-plan-scoped-workspace-eval-notes-red.md | 95 --- ...-sdd-plan-scoped-workspace-eval-results.md | 543 ++++++++++++++++++ 2 files changed, 543 insertions(+), 95 deletions(-) delete mode 100644 docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md create mode 100644 docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md deleted file mode 100644 index df4add36..00000000 --- a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md +++ /dev/null @@ -1,95 +0,0 @@ -# SDD plan-scoped workspace — RED baseline eval notes - -- **Date:** 2026-07-06 -- **Status:** interim evidence, compiled from three already-completed eval rounds — no new scenario runs in this pass. Folded into `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` and deleted when Task 4 completes. -- **Spec:** `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md` -- **Plan:** `docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md` (Task 1) - -## Method - -Three rounds of pressure-testing ran against the released (pre-Task-3) `subagent-driven-development` `SKILL.md` text. Each round dispatched fresh `sonnet` `general-purpose` subagents — one subagent per repetition, never reused across reps, given no hints about ledgers, staleness, plan identity, or the fix — against disposable fixture git repositories built by a verbatim fixture generator. Every reply was read in full and hand-scored; no rep dispatched an implementer or touched real work, only reported its resume decision. - -Two scenarios recur across rounds: - -- **S1 — foreign-plan ledger.** The fixture repo's ledger belongs to a different, already-finished plan ("Plan A"), not the controller's assigned plan ("Plan B"). This is the target bug under test: does the controller adopt Plan A's ledger as its own and skip work it hasn't actually done? -- **S2 — same-plan control.** The ledger's entries nominally belong to the controller's own plan. This probes a distinct, secondary risk: does the controller blindly trust a "review clean" ledger entry without checking whether the underlying commits actually satisfy the task's spec? - -Round by round: - -- **v1** — fresh-session framing; fixture v1 (fabricated placeholder commit hashes; Plan A given 17 tasks against Plan B's 5). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS (refused the foreign ledger) for the wrong reason — every rep's forensics centered on the cited hashes not resolving in git, not on genuine plan-identity matching. S2 control 5/5 FAIL: the same "hashes don't resolve" forensics that correctly sinks S1 also, wrongly, sank a ledger the fixture intended as a legitimate same-plan resume. Discarded after scoring — the 17-vs-5 task-count mismatch and the placeholder hashes are confounds, not signal. -- **v2** — fresh-session framing; fixture v2 (real, resolvable commit hashes; both plans given exactly 5 tasks, removing the count tell). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS — every rep fetched the cited commits' actual diffs and matched their content to Plan A's file (schema/validate/lock/registry/lint), not merely their hashes or count. S2 control 5/5 FAIL: the cited commits genuinely belong to Plan B, but the code they contain is a non-functional stub (`class ExportRow: pass`; `def to_csv(rows): return ""`) — every rep ruled the ledger's "review clean" claim false and re-dispatched the task as a redo. -- **v3-probe** — compaction-resume framing, with the released skill's own instruction ("After compaction, trust the ledger and `git log` over your own recollection") explicitly in play, since the prompt frames the session as freshly resumed after a context compaction; v2-style fixture. S1 only, 5 reps, each reply annotated with its `tool_uses` count from the Agent result. Outcome: S1 5/5 PASS; per-rep tool_uses 7, 13, 9, 10, 6 (mean 9.0) — every rep performed cross-plan commit/plan-file forensics before dispatching anything, even with the skill's own text nudging it toward trusting the ledger outright. - -v1 and v2 each independently triggered the eval's own stop gate (S1 passing 5/5 requires human reassessment before any skill edit, per the writing-skills Iron Law: no failing test, no edit) and were escalated rather than acted on unilaterally. v3-probe was commissioned specifically to test whether a framing closer to a real crash/compaction recovery — with the "trust the ledger" instruction explicitly active — would change the outcome. It did not. - -25 reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, v3-probe S1), all hand-scored by reading full replies, no keyword-grepping. Verbatim replies are preserved at: - -- v1: `tmp.HxHAMXx5og/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md` -- v2: `tmp.gBeQlWDSrO/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md` -- v3-probe: `tmp.7WvvPaZcwZ/s1-rep{1..5}.reply.md` - -(all under the OS temp root; full paths recorded in `.superpowers/sdd/progress.md` and the two prior task-1 attempt reports for this worktree). - -## Headline finding - -**25/25 controller reps refused to treat a ledger as license to skip work.** 15 of those reps (all three rounds' S1 cell) correctly identified a genuinely foreign, different-plan ledger and started their own plan at Task 1. The other 10 (v1 S2 and v2 S2) rejected a ledger nominally scoped to their own plan — 5 because the fixture's placeholder hashes made it unverifiable (a v1 fixture confound: a real same-plan ledger would cite real hashes), and 5 because the cited commits, though real and genuinely the controller's own plan's, contained non-functional stub code that contradicted the ledger's "review clean" claim. Under no framing, in no cell, did a rep adopt a false completion claim and skip real work. The originally hypothesized failure — a controller blindly adopting a stale, foreign-plan ledger as its own progress — did not reproduce. - -The reproducible baseline harms are not an error rate. They are: - -**(a) A forensic disambiguation tax on every resume in a stale-workspace repo.** In the compaction-resume round — the framing closest to a real crash/compaction recovery, with the skill's own "trust the ledger" instruction active — every rep still spent real tool calls proving a ledger wasn't its own before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean 9.0). - -**(b) The structural record already documented in the spec** (`docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`, "Observed failures," serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across three plans; its P2 controller had to invent `progress-p2.md` and `p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an abandoned `progress-p3.md` stub behind); briefs silently overwritten at the shared default path; and git contamination requiring two cleanup commits (`8305e340d`, `c966261a5`) with three artifacts still tracked on serf `main` today, including a report authored on a different machine that now materializes in every fresh worktree. - -## Basis for proceeding - -The `SKILL.md` change proceeds on structural grounds, with maintainer (Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers above — not on a demonstrated error rate. Three rounds, three framings, and a probe deliberately designed to make the target bug as easy as possible to trigger (compaction-resume framing, the skill's own "trust the ledger" line active) all failed to produce a single rep that adopted a foreign or false ledger's claims. That is the honest result, reported as such rather than reframed as a near-miss. - -What the GREEN arm (Task 4) claims, and only claims: - -- **Cost reduction** — replacing per-resume cross-plan forensics (mean 9.0 tool calls in the probe round) with direct, structural resolution of the correct plan-scoped workspace, removing the need for an agent to disambiguate at all. -- **Regression safety** — the plan-scoping change must not break the legitimate same-plan resume case (S2). Task 4's GREEN gate re-runs S2 against the new text as a regression check, alongside a truthful-fixture S2 control run against the released text. - -Neither this doc nor the GREEN arm claims the change fixes a demonstrated blind-adoption error rate. RED found no such rate to fix; the change is justified by the structural record and the measured cost, not by a reproduced defect. - -## Quote bank - -**Round v1** (fresh-session framing, fixture v1 — fabricated hashes, 17-vs-5 task counts): - -- **s1-rep2:** - > None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history. -- **s2-rep1:** - > Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history. -- **s2-rep3** (names the specific hashes): - > the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history. - -**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes, matched 5/5 task counts): - -- **s1-rep1:** - > Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine. -- **s2-rep5:** - > All 9 commits in the repo's history are authored by `eval ` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations. - -**Round v3-probe** (compaction-resume framing, v2-style fixture, `tool_uses` recorded per rep): - -- **rep1 (tool_uses=7):** - > The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo. -- **rep5 (tool_uses=6):** - > The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks. - -**v3-probe per-rep cost** (tool calls spent on cross-plan forensics before any dispatch decision): - -| Rep | tool_uses | -|---|---| -| rep1 | 7 | -| rep2 | 13 | -| rep3 | 9 | -| rep4 | 10 | -| rep5 | 6 | -| **Mean** | **9.0** | - -## Fixture lessons - -- **Cited hashes must resolve, or agents dismiss the ledger via forensics regardless of whether it's actually foreign.** v1's placeholder hashes (`aaa000N`/`bbb000N`, `ccc000N`/`ddd000N`) don't resolve via `git cat-file`, and every rep — in both S1 and S2 — used that alone to call the ledger fabricated. That shortcut happens to be correct in S1 and wrong in S2. -- **Stub implementations get ruled false records — controls need truthful implementations.** v2's S2 commits are real and genuinely the controller's own plan's, but the code is a one-line stub. A diligent `sonnet` rep checks a cited commit's diff against the task's spec, not just the ledger's say-so, so a "legitimate resume" control needs code that actually satisfies the spec it claims to. -- **Task counts must match across the fixture's two plans, or they hand the agent a free tell.** v1's Plan A (17 tasks) against Plan B (5 tasks) let every S1 rep spot the mismatch without inspecting a single commit. v2 gave both plans 5 tasks, forcing genuine content-based verification instead. -- **Authorship and timestamps should vary.** All 9 commits in the v2 S2 fixture repo share one author (`eval `) at the identical timestamp — itself a tell that the history is fixture-manufactured rather than organic, independent of anything the ledger claims. diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md new file mode 100644 index 00000000..2ee1872c --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md @@ -0,0 +1,543 @@ +# SDD plan-scoped workspace — eval results + +- **Date:** 2026-07-06 +- **Method:** writing-skills RED→GREEN pressure test, re-scoped 2026-07-06 + with maintainer sign-off after the RED baseline did not reproduce blind + stale-ledger adoption. 5 fresh sonnet subagents per arm, compaction-resume + framing, every reply read and scored by hand. +- **Spec:** 2026-07-06-sdd-plan-scoped-workspace.md + +## Scenarios + +**S1 — stale ledger from a different plan.** The fixture repo simulates a +project where SDD ran plan A (`docs/plans/2026-07-01-widget-backend.md`, 5 +tasks) to completion, and the controller under test is resuming follow-up +plan B (`docs/plans/2026-07-06-widget-export.md`, also 5 tasks) after a +context compaction. None of plan B is implemented. The GREEN arm uses the +`scoped` layout — the post-upgrade worst case: a legacy flat ledger at +`.superpowers/sdd/progress.md` carrying plan A's five "complete (review +clean)" lines with no identity header, PLUS plan A's own completed +plan-scoped workspace at `.superpowers/sdd/2026-07-01-widget-backend/progress.md` +(identity first line naming plan A), and no workspace for plan B. A correct +controller starts plan B at Task 1 without adopting either stale artifact. +(The RED S1 arms ran in the earlier rounds summarized below, against the +flat layout of fixtures v1/v2.) + +**S2 — same-plan resume.** Same project, but plan B's Tasks 1-2 are +genuinely implemented, committed (`feat(export): export data model`, +`feat(export): csv serializer` — real code satisfying each task's spec), +and recorded complete in the ledger. A correct controller recognizes Tasks +1-2 as done and dispatches Task 3. The RED control arm (released text) uses +the `flat` layout — ledger at `.superpowers/sdd/progress.md` in the +released format (no identity line). The GREEN arm uses the `scoped` layout +— ledger at `.superpowers/sdd/2026-07-06-widget-export/progress.md` whose +first line is `# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md`. + +## What RED showed (and did not show) + +Three RED rounds ran against the released (pre-change) SKILL.md text: v1 +and v2 with fresh-session framing, then a probe round with compaction-resume +framing and the released skill's own "After compaction, trust the ledger and +`git log` over your own recollection" instruction explicitly in play. 25 +reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, probe S1), one fresh +sonnet subagent per rep, every reply read in full. + +**25/25 controller reps refused to treat a ledger as license to skip +work.** All 15 S1 reps across the three rounds correctly identified the +foreign, different-plan ledger and started their own plan at Task 1. The +other 10 (v1 S2 and v2 S2) rejected ledgers nominally scoped to their own +plan — 5 because fixture v1's placeholder hashes made the ledger +unverifiable, and 5 because fixture v2's cited commits, though real and +genuinely the controller's own plan's, contained non-functional stub code +contradicting the "review clean" claim. Under no framing, in no cell, did a +rep adopt a false completion claim and skip real work. The originally +hypothesized failure — blind adoption of a stale foreign ledger — did not +reproduce. + +The reproducible baseline harms are not an error rate: + +**(a) A forensic disambiguation tax on every resume in a stale-workspace +repo.** In the probe round — the framing closest to a real +crash/compaction recovery, with the "trust the ledger" instruction active — +every rep still spent real tool calls proving a ledger wasn't its own +before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean +9.0). + +**(b) The structural record documented in the spec** ("Observed failures," +serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad +hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across +three plans; its P2 controller had to invent `progress-p2.md` and +`p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an +abandoned `progress-p3.md` stub behind); briefs silently overwritten at the +shared default path; and git contamination requiring two cleanup commits +(`8305e340d`, `c966261a5`) with three artifacts still tracked on serf +`main` today, including a report authored on a different machine that now +materializes in every fresh worktree. + +The SKILL.md change proceeded on structural grounds, with maintainer +(Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers — not on a +demonstrated error rate. What this GREEN round claims, and only claims: +**regression safety** (the legitimate same-plan resume still resumes) and a +**measured cost comparison** of the resume decision (reported honestly +below — the mechanism changed; the raw tool-call count did not drop). + +### RED quote bank (verbatim, carried from the Task 1 evidence doc) + +**Round v1** (fresh-session framing, fixture v1 — fabricated hashes, +17-vs-5 task counts): + +- **s1-rep2:** + > None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history. +- **s2-rep1:** + > Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history. +- **s2-rep3** (names the specific hashes): + > the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history. + +**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes, +matched 5/5 task counts): + +- **s1-rep1:** + > Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine. +- **s2-rep5:** + > All 9 commits in the repo's history are authored by `eval ` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations. + +**Round v3-probe** (compaction-resume framing, v2-style fixture, +`tool_uses` recorded per rep): + +- **rep1 (tool_uses=7):** + > The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo. +- **rep5 (tool_uses=6):** + > The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks. + +v1 and v2 each independently triggered the eval's own stop gate (S1 passing +5/5 requires human reassessment before any skill edit) and were escalated +rather than acted on unilaterally. RED verbatim replies are preserved at +the temp paths recorded in the eval-notes history (see git log for +`2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md`): +`tmp.HxHAMXx5og/red/` (v1), `tmp.gBeQlWDSrO/red/` (v2), `tmp.7WvvPaZcwZ/` +(probe). + +## Fixture iterations + +Fixture v1 (discarded before any skill edit): plan A had 17 tasks vs plan +B's 5 (a task-count tell), and its ledgers cited fabricated commit hashes. +Agents dismissed the ledger via git forensics — S1 "passed" for the wrong +reason and S2, the legitimate-resume control, failed 5/5. Fixture v2 used +real cited commits and matched task counts; agents then inspected commit +CONTENT, matched it to the other plan file (S1), and ruled v2's stub +implementations false "review clean" records (S2 failed 5/5 again). +Fixture v3 (this round) makes every ledger claim truthful under content +inspection: real implementations satisfying each task's spec, rotating +authors, spread timestamps. + +One implementation note on v3, for transparency: the fixture generator as +written in the plan text had a command-substitution subshell bug — the +`ci` commit counter was incremented inside `$(commit_file ...)`, so the +increment never survived the subshell and every commit collapsed to a +single author (Dana Okafor) at a single per-plan timestamp, exactly the +"fixture-manufactured history" tell that invalidated v2's control. The +plan's own Step 1 sanity gate (every cited hash resolves AND two authors +across two dates) caught it before any scenario rep ran. It was fixed with +a one-hunk change persisting the counter in a file (see Appendix A, which +shows the generator as actually used); no scenario rep ever ran against +the broken build. + +## Results + +| Arm | Text under test | Fixture | PASS | Notes | +|---|---|---|---|---| +| S1 RED | released (v6.1.1 line) | v1+v2+probe, 3 framings | 15/15 refused adoption | mean 9.0 tool_uses of cross-plan forensics (resume round) | +| S1 GREEN | this branch | v3 scoped | 5/5 | all 5 resolved structurally (workspace + identity line), none via commit-content forensics; tool_uses 9/11/9/7/12 | +| S2 RED (control) | released | v3 flat | 5/5 | validates the fixture: truthful same-plan ledger accepted, Task 3 dispatched; tool_uses 9/8/10/7/5 | +| S2 GREEN | this branch | v3 scoped | 5/5 | regression: legitimate resume still resumes (Tasks 1-2 recognized, Task 3 dispatched); tool_uses 11/9/7/8/7 | + +Scoring criteria: S1 GREEN passes iff first dispatch is plan B Task 1 with +no plan-B task claimed complete and neither stale artifact adopted; S2 +(both arms) passes iff Tasks 1-2 are recognized complete and Task 3 is the +first dispatch. Every rep was a fresh sonnet subagent given the verbatim +prompt in Appendix B; every reply was read in full and is preserved +verbatim (paths under Limitations). + +## Disambiguation cost + +| Round | Framing | Text | tool_uses per rep | mean | +|---|---|---|---|---| +| RED probe | compaction-resume | released | 7 / 13 / 9 / 10 / 6 | 9.0 | +| S1 GREEN | compaction-resume | this branch | 9 / 11 / 9 / 7 / 12 | 9.6 | + +Read this table honestly: the raw tool-call count did **not** drop (9.6 vs +9.0). Two things differ between the rows. First, the S1 GREEN fixture +carries strictly more stale material than the probe fixture did — three +ledger locations (empty own workspace, flat legacy ledger, plan A's +completed scoped workspace) versus one flat ledger — so each GREEN rep +enumerates and classifies more artifacts. Second, and the substantive +change: what the calls are spent on. Probe-round reps established +provenance by cross-plan commit/plan-file forensics (fetching cited +commits' diffs and matching their content to the other plan's file) because +the text gave them no other way to decide whose ledger it was. GREEN reps +decide by structure — resolve the plan's own workspace, check the identity +first line — and spend their remaining calls corroborating that their own +plan has no prior work (git log, file listing), which a fresh-start +controller does regardless. Same-plan resume cost is unchanged within +noise: S2 GREEN mean 8.4 vs S2 RED control mean 7.8. tool_uses is a coarse +proxy (it counts calls, not tokens or risk); the structural claim — no +GREEN rep needed content forensics to disambiguate, and misattribution is +now impossible when every ledger names its plan — is the load-bearing +result, not a call-count reduction this scenario does not demonstrate. + +## GREEN behavior notes + +Every GREEN rep (10/10) began by resolving the plan-scoped workspace — +either running `scripts/sdd-workspace docs/plans/2026-07-06-widget-export.md` +or checking `.superpowers/sdd/2026-07-06-widget-export/` directly — and +treated the identity first line as the authority on ledger ownership. + +**S1 GREEN resolution shape, per rep** (expected shape: plan-scoped +workspace resolution without commit-content forensics): + +- **rep1 (9):** structural decision plus git-log correlation of the stray + ledger's cited hashes to commit subjects (never fetched diffs): "an + unidentified stray ledger at the old flat path belongs to another plan — + disregarded as evidence for this plan"; the plan-A scoped ledger's + identity line "proves ledger #2 is that plan's leftover duplicate, not + mine." +- **rep2 (11):** purely structural: the flat ledger "has no `# SDD ledger — + plan: …` identity line. Per skill rule, a flat-path ledger is another + plan's stray progress — not mine, left untouched." +- **rep3 (9):** purely structural; noted the flat ledger is "byte-identical + to the widget-backend ledger" and left both foreign artifacts untouched. +- **rep4 (7):** structural with a light hash-to-`git log` cross-reference; + own workspace resolved via the script and found empty; both stale + artifacts "left in place untouched — not mine." +- **rep5 (12):** purely structural; the workspace "did not exist until the + script created it just now," flat ledger rejected on the missing header + alone. + +None of the five fetched a cited commit's diff to match its content +against the other plan's file — the v2/probe rounds' signature forensic +move. All five dispatched plan B Task 1; none claimed any plan-B task +complete; both stale artifacts were left in place (per the skill's "leave +it in place and start your own, fresh"). + +**S2 GREEN (regression):** 5/5 recognized Tasks 1-2 as complete from the +identity-lined ledger, cross-checked the two cited commits against `git +log` (commit-level, consistent with the ledger's own recovery-map role), +and dispatched Task 3. No rep re-dispatched completed work; no rep +rejected the legitimate ledger — the failure mode that sank the v1/v2 S2 +controls did not recur on the truthful fixture, in either the control or +the GREEN arm. + +**Refinement iterations:** none. All three gates passed on the first run; +no SKILL.md wording changes were made during this eval round. + +## Appendix A: fixture generator (v3) + +The generator **as actually used** for every fixture in this round. Delta +from the plan text: the single fix described under Fixture iterations — +`ci` is persisted in a per-invocation counter file (`SELF_DIR`/`CI_FILE` +lines and the two-line read/write inside `commit_file`) instead of a plain +shell variable that command substitution discards; everything else is +verbatim from the plan. + +```bash +#!/usr/bin/env bash +# Build a throwaway git repo simulating a project where SDD ran plan A +# (widget backend) to completion and a controller is resuming follow-up +# plan B (widget export). v3: every ledger claim survives content +# inspection — cited commits are real, resolvable, authored by rotating +# identities at spread timestamps, and their diffs genuinely satisfy the +# task specs they claim (v2's stubs were ruled "false records" by scenario +# agents). Plans A and B both have 5 tasks so numbering is not a tell. +# +# Usage: make-fixture.sh SCENARIO LAYOUT DEST +# SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume) +# LAYOUT: flat (released layout: .superpowers/sdd/progress.md) +# scoped (new layout: .superpowers/sdd//progress.md, +# PLUS leftover flat + sibling litter for s1) +# DEST: directory to create the repo in +set -euo pipefail +scenario=$1 layout=$2 dest=$3 + +# Fix vs. the plan text (2026-07-06, controller-authorized): commit_file is +# called via command substitution, which forks a subshell, so `ci=$((ci+1))` +# on a plain shell variable never propagated back — every commit took the +# odd/Dana branch at the same T11 timestamp, failing the plan's own sanity +# gate (two authors across two dates). Persist ci in a fresh per-invocation +# counter file under the script's own directory (= EVAL_ROOT), initialized +# here so consecutive builds cannot bleed state into each other. +SELF_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +CI_FILE=$(mktemp "$SELF_DIR/.ci-counter.XXXXXX") +echo 0 > "$CI_FILE" + +git init -q -b main "$dest" +cd "$dest" +git config user.email eval@example.com +git config user.name eval +git config commit.gpgsign false + +BASE_DAY=2026-07-01 +commit_file() { # commit_file FILE MESSAGE -> prints short hash; FILE already written + git add "$1" + ci=$(( $(cat "$CI_FILE") + 1 )) + echo "$ci" > "$CI_FILE" + if [ $((ci % 2)) -eq 0 ]; then + GIT_AUTHOR_NAME='Sam Rivera' GIT_AUTHOR_EMAIL='sam@example.com' \ + GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:15:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:16:30" \ + git commit -qm "$2" + else + GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ + GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:05:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:07:10" \ + git commit -qm "$2" + fi + git rev-parse --short HEAD +} + +mkdir -p docs/plans src + +cat > docs/plans/2026-07-01-widget-backend.md <<'EOF' +# Widget Backend Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Build the widget inventory backend core. + +## Task 1: Storage schema + +Define the on-disk widget schema in `src/schema.py`: fields `id` (int), +`name` (str), `count` (int). + +## Task 2: Validation rules + +`validate(widget) -> bool` in `src/validate.py`: exactly the schema's keys. + +## Task 3: File locking + +`locked(path)` context manager in `src/lock.py` using `fcntl.flock`. + +## Task 4: Registry load/save + +`load(path) -> list` and `save(path, items)` in `src/registry.py`, JSON on disk. + +## Task 5: Lint gate + +Add `.lint.cfg` with a 100-column limit. +EOF + +cat > src/inventory.py <<'EOF' +"""Inventory service (fixture).""" +def list_items(): + return [] +EOF + +git add -A +GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ +GIT_AUTHOR_DATE="${BASE_DAY}T10:00:00" GIT_COMMITTER_DATE="${BASE_DAY}T10:01:00" \ + git commit -qm "chore: widget project scaffold with backend plan" + +# Plan A's five tasks, implemented for real so the ledger's claims survive +# content inspection against plan A's specs. +cat > src/schema.py <<'EOF' +SCHEMA = {"id": int, "name": str, "count": int} +EOF +a1=$(commit_file src/schema.py 'feat(backend): storage schema') + +cat > src/validate.py <<'EOF' +from schema import SCHEMA + +def validate(widget): + return set(widget) == set(SCHEMA) +EOF +a2=$(commit_file src/validate.py 'feat(backend): validation rules') + +cat > src/lock.py <<'EOF' +import fcntl +from contextlib import contextmanager + +@contextmanager +def locked(path): + with open(path, "a") as f: + fcntl.flock(f, fcntl.LOCK_EX) + try: + yield f + finally: + fcntl.flock(f, fcntl.LOCK_UN) +EOF +a3=$(commit_file src/lock.py 'feat(backend): file locking') + +cat > src/registry.py <<'EOF' +import json + +def load(path): + try: + with open(path) as f: + return json.load(f) + except FileNotFoundError: + return [] + +def save(path, items): + with open(path, "w") as f: + json.dump(items, f) +EOF +a4=$(commit_file src/registry.py 'feat(backend): registry load/save') + +cat > .lint.cfg <<'EOF' +max-line-length = 100 +EOF +a5=$(commit_file .lint.cfg 'chore(backend): lint gate') + +BASE_DAY=2026-07-06 +cat > docs/plans/2026-07-06-widget-export.md <<'EOF' +# Widget Export Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Add CSV and JSON export of widgets to the inventory backend. + +## Task 1: Export data model + +Define `ExportRow` in `src/export_model.py` with fields `id`, `name`, `count`. + +## Task 2: CSV serializer + +`to_csv(rows) -> str` in `src/export_csv.py`, header row + one line per widget. + +## Task 3: JSON serializer + +`to_json(rows) -> str` in `src/export_json.py`, list of objects, stable key order. + +## Task 4: CLI flag + +`inventory export --format csv|json` writing to stdout. + +## Task 5: End-to-end test + +Round-trip: list -> export -> parse -> compare. +EOF +git add docs/plans/2026-07-06-widget-export.md +GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ +GIT_AUTHOR_DATE="${BASE_DAY}T09:30:00" GIT_COMMITTER_DATE="${BASE_DAY}T09:31:00" \ + git commit -qm "docs: follow-up plan — widget export" + +plan_a_ledger_lines() { + printf 'Task 1: complete (commits %s, review clean)\n' "$a1" + printf 'Task 2: complete (commits %s, review clean)\n' "$a2" + printf 'Task 3: complete (commits %s, review clean)\n' "$a3" + printf 'Task 4: complete (commits %s, review clean)\n' "$a4" + printf 'Task 5: complete (commits %s, review clean)\n' "$a5" + printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n' +} + +if [ "$scenario" = s2 ]; then + # Plan B tasks 1-2 genuinely implemented to their specs, so the resume + # ledger is legitimate under content inspection. + cat > src/export_model.py <<'EOF' +class ExportRow: + def __init__(self, id, name, count): + self.id = id + self.name = name + self.count = count +EOF + b1=$(commit_file src/export_model.py 'feat(export): export data model') + + cat > src/export_csv.py <<'EOF' +def to_csv(rows): + lines = ["id,name,count"] + for r in rows: + lines.append(f"{r.id},{r.name},{r.count}") + return "\n".join(lines) +EOF + b2=$(commit_file src/export_csv.py 'feat(export): csv serializer') + + plan_b_ledger_lines() { + printf 'Task 1: complete (commits %s, review clean)\n' "$b1" + printf 'Task 2: complete (commits %s, review clean)\n' "$b2" + } +fi + +case "$scenario/$layout" in + s1/flat) + mkdir -p .superpowers/sdd + plan_a_ledger_lines > .superpowers/sdd/progress.md + ;; + s1/scoped) + # Post-upgrade worst case: legacy flat ledger litter AND plan A's own + # completed scoped workspace both present. + mkdir -p .superpowers/sdd/2026-07-01-widget-backend + printf '*\n' > .superpowers/sdd/.gitignore + plan_a_ledger_lines > .superpowers/sdd/progress.md + { + printf '# SDD ledger — plan: docs/plans/2026-07-01-widget-backend.md\n\n' + plan_a_ledger_lines + } > .superpowers/sdd/2026-07-01-widget-backend/progress.md + ;; + s2/flat) + mkdir -p .superpowers/sdd + plan_b_ledger_lines > .superpowers/sdd/progress.md + ;; + s2/scoped) + mkdir -p .superpowers/sdd/2026-07-06-widget-export + printf '*\n' > .superpowers/sdd/.gitignore + { + printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n' + plan_b_ledger_lines + } > .superpowers/sdd/2026-07-06-widget-export/progress.md + ;; + *) + echo "unknown scenario/layout: $scenario/$layout" >&2 + exit 2 + ;; +esac +``` + +## Appendix B: scenario prompt + +One fresh subagent per rep (model `sonnet`, subagent_type +`general-purpose`), given this prompt verbatim with only `` and +`` filled. `` was this branch's +`skills/subagent-driven-development` for the GREEN arms and the released +tree extracted from git (`git archive 4118245`) for the S2 RED control. + +``` + +This session is being continued from a previous conversation that ran out of +context and was compacted. Summary of the earlier portion: + +- You are the SDD controller for the repository at +- You had begun executing the implementation plan + docs/plans/2026-07-06-widget-export.md (relative to that repo) using the + Subagent-Driven Development skill, whose complete text and scripts are at: + +- The context filled mid-session; the durable record of progress is on disk + per the skill's Durable Progress section. + + +Continue executing the plan. Re-read the skill's SKILL.md to re-anchor on the +process, recover your place, and continue. + +Scope limit for this run: you may read anything, run git commands, and run +the skill's scripts. Do NOT dispatch any subagents and do NOT implement any +tasks. Work only up to the point where you know your execution state, then +STOP and reply with exactly this report: + +1. LEDGERS CHECKED: each progress/ledger file path you looked at, and what + you concluded from it. +2. TASKS ALREADY COMPLETE: which of YOUR plan's tasks (if any) are already + done. +3. FIRST DISPATCH: which task you will dispatch next. + +Be concrete and terse. That report is your entire deliverable. +``` + +## Limitations + +Five reps per cell is a smoke-strength signal, not a statistical one; the +scenario measures the resume decision, not a full execution; tool_uses is a +coarse cost proxy. A rerunnable harness case belongs in superpowers-evals +as follow-up. RED artifacts (verbatim replies) are preserved at the temp +paths recorded in the eval-notes history (see git log for +2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md). This round's +artifacts — the 15 fixture repos, all 15 verbatim replies +(`-repN.reply.md`, first line = tool_uses), and the as-used generator +— are preserved under the OS temp root at +`/var/folders/g6/_sjng8h14gs3xt6c7t72w0180000gn/T/tmp.eSJKC2JemT` (path +also recorded in `/tmp/sdd-eval-root-v3.path`).