Compare commits

..

6 Commits

Author SHA1 Message Date
Jesse Vincent
0e69a4d32c experiment: ground-up two-principle rewrite of writing-good-tests
Re-derived from scratch: every rule becomes a corollary of two principles
(every test names the break it catches; every test exercises the real
thing), one consolidated gate per principle, four example pairs kept, the
rest carried by prose. Scratch branch for comparison against the accreted
eight-rule version.
2026-07-05 18:47:55 -04:00
Jesse Vincent
8afa64b49d refactor(skills): compress writing-good-tests additions; doc changes earn no tests
Prose additions from the last two passes tightened to the terse guard
form: change-detector rule, string-presence trap, and Rule 7's release
valve each drop to a few sentences. Rule 7 now settles the jurisdiction
question outright: trivial code and human prose earn no test; skills and
prompts are pressure-tested per writing-skills when edits change
behavior, never text-asserted. Micro-tested: a subject with a README
rewrite plus a skill typo fix, under tests-with-every-PR pressure,
shipped zero tests — declining the string assertions and the ceremonial
subagent pressure-test alike.
2026-07-05 17:47:02 -04:00
Jesse Vincent
deb9d855cb fix(skills): close the change-detector hole in writing-good-tests
Fresh-eyes review found falsifiable-but-worthless tests passed every
rule: a constant assertion can fail, uses a literal, mocks nothing — and
protects nothing, firing on intentional decisions while sleeping through
bugs. Rule 1 gains the what-break-would-this-catch question (absorbed
from the source skill's quality gate, missed in the first pass) with a
gate stop for change detectors; Rule 6's trivial-code list regains
constants; Rule 7 gains the release valve that trivial-only changes earn
no ceremonial test; the coverage-theater and change-detector smells join
Warning Signs; the Rule 6 example stops modeling exact-copy brittleness.
Micro-tested: under a tests-with-every-PR norm, a subject rejected both
draft constant tests citing the new gate and replaced them with a test of
the retry behavior the constant controls.
2026-07-05 13:03:35 -07:00
Jesse Vincent
78fb4643da feat(skills): absorb falsifiability discipline into writing-good-tests
Generalized from agentsview's testing-without-tautologies skill: a new
Iron Law and lead rule (name the production change that would fail the
test, derive expectations independently of the code under test), a
test-your-code-not-the-framework rule with the characterization-test
exception and the trivial-code guidance, branch-specific doubles folded
into Mock at the Right Level, a closing Mutation Check, and six new
warning-sign smells. Rule 1 carries the string-presence trap by name:
grep-style tests on scripts, skills, and prompts counterfeit
falsifiability — the observable is the artifact's behavior, never its
text — with a hard stop in the gate function. Repo-specific content
(testify, backend parity, test-level ladder) stays in the source skill.
Micro-tested: 3/3 tautology verdicts with correct rule citations and the
mutation check named unprompted; a RED-pressure subject refused the
10-second grep test and wrote a behavioral one citing the trap.
2026-07-05 12:59:56 -07:00
Jesse Vincent
6f3eca4f2e fix(skills): broaden writing-good-tests trigger to any test writing
The pointer fired only on adding mocks or test utilities; the doc's own
load-when line already says writing or changing tests. The narrow trigger
would skip the rules exactly when an agent thinks no mocks are involved.
2026-07-05 12:51:57 -07:00
Jesse Vincent
0cfc0a16b4 refactor(skills): reframe testing-anti-patterns as writing-good-tests
The disclosure doc becomes a catalog of what to do: six positively named
rules (assert on real behavior, cleanup in test utilities, mock at the
right level, mirror real data, tests ship with implementation, prefer
real components), each leading with the GOOD example and keeping the
violation as contrast. Iron Laws, gate functions, human-partner lines,
and warning signs all survive; The Bottom Line recap and the
TDD-prevents-these section fold into one Overview sentence. SKILL.md's
pointer moves into the Good Tests section it belongs with. Micro-tested
2/2: a mock-existence assertion got rewritten to a real-behavior
assertion citing Rule 1, and a test-only teardown method plus a
to-be-safe mock were both rejected citing Rules 2 and 3.
2026-07-05 12:49:52 -07:00
12 changed files with 281 additions and 2353 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -1,543 +0,0 @@
# SDD plan-scoped workspace — eval results
- **Date:** 2026-07-06
- **Method:** writing-skills RED→GREEN pressure test, re-scoped 2026-07-06
with maintainer sign-off after the RED baseline did not reproduce blind
stale-ledger adoption. 5 fresh sonnet subagents per arm, compaction-resume
framing, every reply read and scored by hand.
- **Spec:** 2026-07-06-sdd-plan-scoped-workspace.md
## Scenarios
**S1 — stale ledger from a different plan.** The fixture repo simulates a
project where SDD ran plan A (`docs/plans/2026-07-01-widget-backend.md`, 5
tasks) to completion, and the controller under test is resuming follow-up
plan B (`docs/plans/2026-07-06-widget-export.md`, also 5 tasks) after a
context compaction. None of plan B is implemented. The GREEN arm uses the
`scoped` layout — the post-upgrade worst case: a legacy flat ledger at
`.superpowers/sdd/progress.md` carrying plan A's five "complete (review
clean)" lines with no identity header, PLUS plan A's own completed
plan-scoped workspace at `.superpowers/sdd/2026-07-01-widget-backend/progress.md`
(identity first line naming plan A), and no workspace for plan B. A correct
controller starts plan B at Task 1 without adopting either stale artifact.
(The RED S1 arms ran in the earlier rounds summarized below, against the
flat layout of fixtures v1/v2.)
**S2 — same-plan resume.** Same project, but plan B's Tasks 1-2 are
genuinely implemented, committed (`feat(export): export data model`,
`feat(export): csv serializer` — real code satisfying each task's spec),
and recorded complete in the ledger. A correct controller recognizes Tasks
1-2 as done and dispatches Task 3. The RED control arm (released text) uses
the `flat` layout — ledger at `.superpowers/sdd/progress.md` in the
released format (no identity line). The GREEN arm uses the `scoped` layout
— ledger at `.superpowers/sdd/2026-07-06-widget-export/progress.md` whose
first line is `# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md`.
## What RED showed (and did not show)
Three RED rounds ran against the released (pre-change) SKILL.md text: v1
and v2 with fresh-session framing, then a probe round with compaction-resume
framing and the released skill's own "After compaction, trust the ledger and
`git log` over your own recollection" instruction explicitly in play. 25
reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, probe S1), one fresh
sonnet subagent per rep, every reply read in full.
**25/25 controller reps refused to treat a ledger as license to skip
work.** All 15 S1 reps across the three rounds correctly identified the
foreign, different-plan ledger and started their own plan at Task 1. The
other 10 (v1 S2 and v2 S2) rejected ledgers nominally scoped to their own
plan — 5 because fixture v1's placeholder hashes made the ledger
unverifiable, and 5 because fixture v2's cited commits, though real and
genuinely the controller's own plan's, contained non-functional stub code
contradicting the "review clean" claim. Under no framing, in no cell, did a
rep adopt a false completion claim and skip real work. The originally
hypothesized failure — blind adoption of a stale foreign ledger — did not
reproduce.
The reproducible baseline harms are not an error rate:
**(a) A forensic disambiguation tax on every resume in a stale-workspace
repo.** In the probe round — the framing closest to a real
crash/compaction recovery, with the "trust the ledger" instruction active —
every rep still spent real tool calls proving a ledger wasn't its own
before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean
9.0).
**(b) The structural record documented in the spec** ("Observed failures,"
serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad
hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across
three plans; its P2 controller had to invent `progress-p2.md` and
`p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an
abandoned `progress-p3.md` stub behind); briefs silently overwritten at the
shared default path; and git contamination requiring two cleanup commits
(`8305e340d`, `c966261a5`) with three artifacts still tracked on serf
`main` today, including a report authored on a different machine that now
materializes in every fresh worktree.
The SKILL.md change proceeded on structural grounds, with maintainer
(Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers — not on a
demonstrated error rate. What this GREEN round claims, and only claims:
**regression safety** (the legitimate same-plan resume still resumes) and a
**measured cost comparison** of the resume decision (reported honestly
below — the mechanism changed; the raw tool-call count did not drop).
### RED quote bank (verbatim, carried from the Task 1 evidence doc)
**Round v1** (fresh-session framing, fixture v1 — fabricated hashes,
17-vs-5 task counts):
- **s1-rep2:**
> None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history.
- **s2-rep1:**
> Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history.
- **s2-rep3** (names the specific hashes):
> the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history.
**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes,
matched 5/5 task counts):
- **s1-rep1:**
> Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine.
- **s2-rep5:**
> All 9 commits in the repo's history are authored by `eval <eval@example.com>` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations.
**Round v3-probe** (compaction-resume framing, v2-style fixture,
`tool_uses` recorded per rep):
- **rep1 (tool_uses=7):**
> The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo.
- **rep5 (tool_uses=6):**
> The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks.
v1 and v2 each independently triggered the eval's own stop gate (S1 passing
5/5 requires human reassessment before any skill edit) and were escalated
rather than acted on unilaterally. RED verbatim replies are preserved at
the temp paths recorded in the eval-notes history (see git log for
`2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md`):
`tmp.HxHAMXx5og/red/` (v1), `tmp.gBeQlWDSrO/red/` (v2), `tmp.7WvvPaZcwZ/`
(probe).
## Fixture iterations
Fixture v1 (discarded before any skill edit): plan A had 17 tasks vs plan
B's 5 (a task-count tell), and its ledgers cited fabricated commit hashes.
Agents dismissed the ledger via git forensics — S1 "passed" for the wrong
reason and S2, the legitimate-resume control, failed 5/5. Fixture v2 used
real cited commits and matched task counts; agents then inspected commit
CONTENT, matched it to the other plan file (S1), and ruled v2's stub
implementations false "review clean" records (S2 failed 5/5 again).
Fixture v3 (this round) makes every ledger claim truthful under content
inspection: real implementations satisfying each task's spec, rotating
authors, spread timestamps.
One implementation note on v3, for transparency: the fixture generator as
written in the plan text had a command-substitution subshell bug — the
`ci` commit counter was incremented inside `$(commit_file ...)`, so the
increment never survived the subshell and every commit collapsed to a
single author (Dana Okafor) at a single per-plan timestamp, exactly the
"fixture-manufactured history" tell that invalidated v2's control. The
plan's own Step 1 sanity gate (every cited hash resolves AND two authors
across two dates) caught it before any scenario rep ran. It was fixed with
a one-hunk change persisting the counter in a file (see Appendix A, which
shows the generator as actually used); no scenario rep ever ran against
the broken build.
## Results
| Arm | Text under test | Fixture | PASS | Notes |
|---|---|---|---|---|
| S1 RED | released (v6.1.1 line) | v1+v2+probe, 3 framings | 15/15 refused adoption | mean 9.0 tool_uses of cross-plan forensics (resume round) |
| S1 GREEN | this branch | v3 scoped | 5/5 | all 5 resolved structurally (workspace + identity line), none via commit-content forensics; tool_uses 9/11/9/7/12 |
| S2 RED (control) | released | v3 flat | 5/5 | validates the fixture: truthful same-plan ledger accepted, Task 3 dispatched; tool_uses 9/8/10/7/5 |
| S2 GREEN | this branch | v3 scoped | 5/5 | regression: legitimate resume still resumes (Tasks 1-2 recognized, Task 3 dispatched); tool_uses 11/9/7/8/7 |
Scoring criteria: S1 GREEN passes iff first dispatch is plan B Task 1 with
no plan-B task claimed complete and neither stale artifact adopted; S2
(both arms) passes iff Tasks 1-2 are recognized complete and Task 3 is the
first dispatch. Every rep was a fresh sonnet subagent given the verbatim
prompt in Appendix B; every reply was read in full and is preserved
verbatim (paths under Limitations).
## Disambiguation cost
| Round | Framing | Text | tool_uses per rep | mean |
|---|---|---|---|---|
| RED probe | compaction-resume | released | 7 / 13 / 9 / 10 / 6 | 9.0 |
| S1 GREEN | compaction-resume | this branch | 9 / 11 / 9 / 7 / 12 | 9.6 |
Read this table honestly: the raw tool-call count did **not** drop (9.6 vs
9.0). Two things differ between the rows. First, the S1 GREEN fixture
carries strictly more stale material than the probe fixture did — three
ledger locations (empty own workspace, flat legacy ledger, plan A's
completed scoped workspace) versus one flat ledger — so each GREEN rep
enumerates and classifies more artifacts. Second, and the substantive
change: what the calls are spent on. Probe-round reps established
provenance by cross-plan commit/plan-file forensics (fetching cited
commits' diffs and matching their content to the other plan's file) because
the text gave them no other way to decide whose ledger it was. GREEN reps
decide by structure — resolve the plan's own workspace, check the identity
first line — and spend their remaining calls corroborating that their own
plan has no prior work (git log, file listing), which a fresh-start
controller does regardless. Same-plan resume cost is unchanged within
noise: S2 GREEN mean 8.4 vs S2 RED control mean 7.8. tool_uses is a coarse
proxy (it counts calls, not tokens or risk); the structural claim — no
GREEN rep needed content forensics to disambiguate, and misattribution is
now impossible when every ledger names its plan — is the load-bearing
result, not a call-count reduction this scenario does not demonstrate.
## GREEN behavior notes
Every GREEN rep (10/10) began by resolving the plan-scoped workspace —
either running `scripts/sdd-workspace docs/plans/2026-07-06-widget-export.md`
or checking `.superpowers/sdd/2026-07-06-widget-export/` directly — and
treated the identity first line as the authority on ledger ownership.
**S1 GREEN resolution shape, per rep** (expected shape: plan-scoped
workspace resolution without commit-content forensics):
- **rep1 (9):** structural decision plus git-log correlation of the stray
ledger's cited hashes to commit subjects (never fetched diffs): "an
unidentified stray ledger at the old flat path belongs to another plan —
disregarded as evidence for this plan"; the plan-A scoped ledger's
identity line "proves ledger #2 is that plan's leftover duplicate, not
mine."
- **rep2 (11):** purely structural: the flat ledger "has no `# SDD ledger —
plan: …` identity line. Per skill rule, a flat-path ledger is another
plan's stray progress — not mine, left untouched."
- **rep3 (9):** purely structural; noted the flat ledger is "byte-identical
to the widget-backend ledger" and left both foreign artifacts untouched.
- **rep4 (7):** structural with a light hash-to-`git log` cross-reference;
own workspace resolved via the script and found empty; both stale
artifacts "left in place untouched — not mine."
- **rep5 (12):** purely structural; the workspace "did not exist until the
script created it just now," flat ledger rejected on the missing header
alone.
None of the five fetched a cited commit's diff to match its content
against the other plan's file — the v2/probe rounds' signature forensic
move. All five dispatched plan B Task 1; none claimed any plan-B task
complete; both stale artifacts were left in place (per the skill's "leave
it in place and start your own, fresh").
**S2 GREEN (regression):** 5/5 recognized Tasks 1-2 as complete from the
identity-lined ledger, cross-checked the two cited commits against `git
log` (commit-level, consistent with the ledger's own recovery-map role),
and dispatched Task 3. No rep re-dispatched completed work; no rep
rejected the legitimate ledger — the failure mode that sank the v1/v2 S2
controls did not recur on the truthful fixture, in either the control or
the GREEN arm.
**Refinement iterations:** none. All three gates passed on the first run;
no SKILL.md wording changes were made during this eval round.
## Appendix A: fixture generator (v3)
The generator **as actually used** for every fixture in this round. Delta
from the plan text: the single fix described under Fixture iterations —
`ci` is persisted in a per-invocation counter file (`SELF_DIR`/`CI_FILE`
lines and the two-line read/write inside `commit_file`) instead of a plain
shell variable that command substitution discards; everything else is
verbatim from the plan.
```bash
#!/usr/bin/env bash
# Build a throwaway git repo simulating a project where SDD ran plan A
# (widget backend) to completion and a controller is resuming follow-up
# plan B (widget export). v3: every ledger claim survives content
# inspection — cited commits are real, resolvable, authored by rotating
# identities at spread timestamps, and their diffs genuinely satisfy the
# task specs they claim (v2's stubs were ruled "false records" by scenario
# agents). Plans A and B both have 5 tasks so numbering is not a tell.
#
# Usage: make-fixture.sh SCENARIO LAYOUT DEST
# SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume)
# LAYOUT: flat (released layout: .superpowers/sdd/progress.md)
# scoped (new layout: .superpowers/sdd/<plan-basename>/progress.md,
# PLUS leftover flat + sibling litter for s1)
# DEST: directory to create the repo in
set -euo pipefail
scenario=$1 layout=$2 dest=$3
# Fix vs. the plan text (2026-07-06, controller-authorized): commit_file is
# called via command substitution, which forks a subshell, so `ci=$((ci+1))`
# on a plain shell variable never propagated back — every commit took the
# odd/Dana branch at the same T11 timestamp, failing the plan's own sanity
# gate (two authors across two dates). Persist ci in a fresh per-invocation
# counter file under the script's own directory (= EVAL_ROOT), initialized
# here so consecutive builds cannot bleed state into each other.
SELF_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
CI_FILE=$(mktemp "$SELF_DIR/.ci-counter.XXXXXX")
echo 0 > "$CI_FILE"
git init -q -b main "$dest"
cd "$dest"
git config user.email eval@example.com
git config user.name eval
git config commit.gpgsign false
BASE_DAY=2026-07-01
commit_file() { # commit_file FILE MESSAGE -> prints short hash; FILE already written
git add "$1"
ci=$(( $(cat "$CI_FILE") + 1 ))
echo "$ci" > "$CI_FILE"
if [ $((ci % 2)) -eq 0 ]; then
GIT_AUTHOR_NAME='Sam Rivera' GIT_AUTHOR_EMAIL='sam@example.com' \
GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:15:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:16:30" \
git commit -qm "$2"
else
GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \
GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:05:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:07:10" \
git commit -qm "$2"
fi
git rev-parse --short HEAD
}
mkdir -p docs/plans src
cat > docs/plans/2026-07-01-widget-backend.md <<'EOF'
# Widget Backend Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development.
**Goal:** Build the widget inventory backend core.
## Task 1: Storage schema
Define the on-disk widget schema in `src/schema.py`: fields `id` (int),
`name` (str), `count` (int).
## Task 2: Validation rules
`validate(widget) -> bool` in `src/validate.py`: exactly the schema's keys.
## Task 3: File locking
`locked(path)` context manager in `src/lock.py` using `fcntl.flock`.
## Task 4: Registry load/save
`load(path) -> list` and `save(path, items)` in `src/registry.py`, JSON on disk.
## Task 5: Lint gate
Add `.lint.cfg` with a 100-column limit.
EOF
cat > src/inventory.py <<'EOF'
"""Inventory service (fixture)."""
def list_items():
return []
EOF
git add -A
GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \
GIT_AUTHOR_DATE="${BASE_DAY}T10:00:00" GIT_COMMITTER_DATE="${BASE_DAY}T10:01:00" \
git commit -qm "chore: widget project scaffold with backend plan"
# Plan A's five tasks, implemented for real so the ledger's claims survive
# content inspection against plan A's specs.
cat > src/schema.py <<'EOF'
SCHEMA = {"id": int, "name": str, "count": int}
EOF
a1=$(commit_file src/schema.py 'feat(backend): storage schema')
cat > src/validate.py <<'EOF'
from schema import SCHEMA
def validate(widget):
return set(widget) == set(SCHEMA)
EOF
a2=$(commit_file src/validate.py 'feat(backend): validation rules')
cat > src/lock.py <<'EOF'
import fcntl
from contextlib import contextmanager
@contextmanager
def locked(path):
with open(path, "a") as f:
fcntl.flock(f, fcntl.LOCK_EX)
try:
yield f
finally:
fcntl.flock(f, fcntl.LOCK_UN)
EOF
a3=$(commit_file src/lock.py 'feat(backend): file locking')
cat > src/registry.py <<'EOF'
import json
def load(path):
try:
with open(path) as f:
return json.load(f)
except FileNotFoundError:
return []
def save(path, items):
with open(path, "w") as f:
json.dump(items, f)
EOF
a4=$(commit_file src/registry.py 'feat(backend): registry load/save')
cat > .lint.cfg <<'EOF'
max-line-length = 100
EOF
a5=$(commit_file .lint.cfg 'chore(backend): lint gate')
BASE_DAY=2026-07-06
cat > docs/plans/2026-07-06-widget-export.md <<'EOF'
# Widget Export Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development.
**Goal:** Add CSV and JSON export of widgets to the inventory backend.
## Task 1: Export data model
Define `ExportRow` in `src/export_model.py` with fields `id`, `name`, `count`.
## Task 2: CSV serializer
`to_csv(rows) -> str` in `src/export_csv.py`, header row + one line per widget.
## Task 3: JSON serializer
`to_json(rows) -> str` in `src/export_json.py`, list of objects, stable key order.
## Task 4: CLI flag
`inventory export --format csv|json` writing to stdout.
## Task 5: End-to-end test
Round-trip: list -> export -> parse -> compare.
EOF
git add docs/plans/2026-07-06-widget-export.md
GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \
GIT_AUTHOR_DATE="${BASE_DAY}T09:30:00" GIT_COMMITTER_DATE="${BASE_DAY}T09:31:00" \
git commit -qm "docs: follow-up plan — widget export"
plan_a_ledger_lines() {
printf 'Task 1: complete (commits %s, review clean)\n' "$a1"
printf 'Task 2: complete (commits %s, review clean)\n' "$a2"
printf 'Task 3: complete (commits %s, review clean)\n' "$a3"
printf 'Task 4: complete (commits %s, review clean)\n' "$a4"
printf 'Task 5: complete (commits %s, review clean)\n' "$a5"
printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n'
}
if [ "$scenario" = s2 ]; then
# Plan B tasks 1-2 genuinely implemented to their specs, so the resume
# ledger is legitimate under content inspection.
cat > src/export_model.py <<'EOF'
class ExportRow:
def __init__(self, id, name, count):
self.id = id
self.name = name
self.count = count
EOF
b1=$(commit_file src/export_model.py 'feat(export): export data model')
cat > src/export_csv.py <<'EOF'
def to_csv(rows):
lines = ["id,name,count"]
for r in rows:
lines.append(f"{r.id},{r.name},{r.count}")
return "\n".join(lines)
EOF
b2=$(commit_file src/export_csv.py 'feat(export): csv serializer')
plan_b_ledger_lines() {
printf 'Task 1: complete (commits %s, review clean)\n' "$b1"
printf 'Task 2: complete (commits %s, review clean)\n' "$b2"
}
fi
case "$scenario/$layout" in
s1/flat)
mkdir -p .superpowers/sdd
plan_a_ledger_lines > .superpowers/sdd/progress.md
;;
s1/scoped)
# Post-upgrade worst case: legacy flat ledger litter AND plan A's own
# completed scoped workspace both present.
mkdir -p .superpowers/sdd/2026-07-01-widget-backend
printf '*\n' > .superpowers/sdd/.gitignore
plan_a_ledger_lines > .superpowers/sdd/progress.md
{
printf '# SDD ledger — plan: docs/plans/2026-07-01-widget-backend.md\n\n'
plan_a_ledger_lines
} > .superpowers/sdd/2026-07-01-widget-backend/progress.md
;;
s2/flat)
mkdir -p .superpowers/sdd
plan_b_ledger_lines > .superpowers/sdd/progress.md
;;
s2/scoped)
mkdir -p .superpowers/sdd/2026-07-06-widget-export
printf '*\n' > .superpowers/sdd/.gitignore
{
printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n'
plan_b_ledger_lines
} > .superpowers/sdd/2026-07-06-widget-export/progress.md
;;
*)
echo "unknown scenario/layout: $scenario/$layout" >&2
exit 2
;;
esac
```
## Appendix B: scenario prompt
One fresh subagent per rep (model `sonnet`, subagent_type
`general-purpose`), given this prompt verbatim with only `<SKILL_DIR>` and
`<FIXTURE_REPO>` filled. `<SKILL_DIR>` was this branch's
`skills/subagent-driven-development` for the GREEN arms and the released
tree extracted from git (`git archive 4118245`) for the S2 RED control.
```
<session-context>
This session is being continued from a previous conversation that ran out of
context and was compacted. Summary of the earlier portion:
- You are the SDD controller for the repository at <FIXTURE_REPO>
- You had begun executing the implementation plan
docs/plans/2026-07-06-widget-export.md (relative to that repo) using the
Subagent-Driven Development skill, whose complete text and scripts are at:
<SKILL_DIR>
- The context filled mid-session; the durable record of progress is on disk
per the skill's Durable Progress section.
</session-context>
Continue executing the plan. Re-read the skill's SKILL.md to re-anchor on the
process, recover your place, and continue.
Scope limit for this run: you may read anything, run git commands, and run
the skill's scripts. Do NOT dispatch any subagents and do NOT implement any
tasks. Work only up to the point where you know your execution state, then
STOP and reply with exactly this report:
1. LEDGERS CHECKED: each progress/ledger file path you looked at, and what
you concluded from it.
2. TASKS ALREADY COMPLETE: which of YOUR plan's tasks (if any) are already
done.
3. FIRST DISPATCH: which task you will dispatch next.
Be concrete and terse. That report is your entire deliverable.
```
## Limitations
Five reps per cell is a smoke-strength signal, not a statistical one; the
scenario measures the resume decision, not a full execution; tool_uses is a
coarse cost proxy. A rerunnable harness case belongs in superpowers-evals
as follow-up. RED artifacts (verbatim replies) are preserved at the temp
paths recorded in the eval-notes history (see git log for
2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md). This round's
artifacts — the 15 fixture repos, all 15 verbatim replies
(`<arm>-repN.reply.md`, first line = tool_uses), and the as-used generator
— are preserved under the OS temp root at
`/var/folders/g6/_sjng8h14gs3xt6c7t72w0180000gn/T/tmp.eSJKC2JemT` (path
also recorded in `/tmp/sdd-eval-root-v3.path`).

View File

@@ -1,196 +0,0 @@
# SDD plan-scoped workspace — design
- **Date:** 2026-07-06
- **Status:** approved direction (Jesse, 2026-07-06); this spec captures the investigation's recommended fix
- **Problem owner:** subagent-driven-development skill (`skills/subagent-driven-development/`)
## Problem
SDD's durable-progress workspace (`.superpowers/sdd/`, introduced v6.0.0/v6.0.3) has
no plan identity and no end-of-life. Every artifact is keyed by bare task number
(`progress.md`, `task-N-brief.md`, `task-N-report.md`), and SKILL.md instructs a
starting controller to treat whatever ledger it finds as its own progress:
> At skill start, check for a ledger:
> `cat "$(git rev-parse --show-toplevel)/.superpowers/sdd/progress.md"`. Tasks listed there
> as complete are DONE — do not re-dispatch them; resume at the first task
> not marked complete.
A fresh session executing a **follow-up plan** in the same worktree reads the
previous plan's ledger as its own. A straight-line reading of the skill tells it
to skip tasks. Nothing ever deletes the workspace, so the stale state persists
indefinitely and accumulates.
### Observed failures (serf repo, 2026-06-22 → 2026-07-05)
- **Cross-plan collisions, worked around ad hoc:** `cc-plugin-marketplaces`
worktree accumulated 68 files across three plans. The P2 controller had to
invent `progress-p2.md` and `p2-task-N-report.md` to dodge P1's ledger; P2's
briefs silently overwrote P1's at the default paths; an abandoned
`progress-p3.md` stub remains.
- **Git contamination, three times over:** SDD scratch was committed and needed
two cleanup commits (`8305e340d`, `c966261a5`); three artifacts are tracked on
serf main today, including a report authored on a different machine that now
materializes in every fresh worktree. A follow-up plan's task-1 report
overwrote an unrelated tracked one, leaving permanent `git status` noise.
- The self-ignoring `.gitignore` is written only when a script runs. Controllers
that hand-append the ledger (observed) never create it, and gitignore is
powerless once a file is tracked.
### Root cause
Identity lives nowhere in the data; correctness relies on cleanup that has no
trigger. Any fix that relies on end-of-plan cleanup alone fails exactly in the
crash/compaction cases the ledger exists to survive. Identity must be
structural.
## Design
### 1. Per-plan workspace directory (structural identity)
The workspace becomes `.superpowers/sdd/<plan-slug>/`, where `<plan-slug>` is
the plan file's basename without its `.md` extension (plan filenames are
already dated kebab-case, e.g. `2026-07-04-plugin-marketplaces-p1-backend-core`).
Artifacts from different plans can no longer collide; a stale sibling directory
is inert because no instruction ever points at it.
Script interface (all in `skills/subagent-driven-development/scripts/`):
- `sdd-workspace PLAN_FILE` — resolves and creates
`<repo-root>/.superpowers/sdd/<plan-slug>/`, maintains the self-ignoring
`.gitignore` at `.superpowers/sdd/.gitignore` (parent level, content `*`),
prints the plan directory's absolute path. Errors (exit 2) on missing
argument or nonexistent plan file. Slug must be non-empty after stripping.
- `task-brief PLAN_FILE N [OUTFILE]` — signature unchanged; default OUTFILE
moves to `<workspace>/task-N-brief.md` via `sdd-workspace PLAN_FILE`.
- `review-package PLAN_FILE BASE HEAD [OUTFILE]` — gains PLAN_FILE as first
argument; default OUTFILE moves to `<workspace>/review-<base7>..<head7>.diff`.
No compatibility path for the old flat layout: the scripts and SKILL.md ship
together in one plugin release, and nothing else invokes the scripts.
(Explicitly confirmed: no backward-compatibility handling.)
### 2. Ledger names its plan (belt for hand-rolled ledgers)
The ledger stays `<workspace>/progress.md`. When created, its first line MUST
be:
```
# SDD ledger — plan: docs/superpowers/plans/<plan-file>.md
```
SKILL.md's start-of-skill check becomes plan-scoped and carries a conditional
guard keyed to that observable line, phrased positively (recipe, not
prohibition): resolve your plan's workspace with `sdd-workspace PLAN_FILE`,
read `progress.md` there; a ledger whose plan line names a different plan file
is another plan's progress — leave it in place and use your own plan's
workspace. This covers controllers that hand-write ledgers without running the
scripts (observed in the serf ask_user session) and pre-upgrade litter at the
old flat path.
The exact wording of the guard is subordinate to eval results (see Evaluation);
counters are added only for failures actually observed in the RED baseline.
### 3. Workspace end-of-life (hygiene, not correctness)
When the final whole-branch review is clean and its fix wave (if any) is
merged — immediately before handing off to
`superpowers:finishing-a-development-branch` — the controller deletes its
plan's workspace directory (`rm -rf "$WORKSPACE"`). The record of the work is
the git history; the ledger's job (mid-plan compaction recovery) is over.
Sibling directories are never touched: crashed or parallel plans own their own
dirs, and deliberately parked cross-plan artifacts (observed pattern:
`WAVE1-HANDOFF.md`) live directly under `.superpowers/sdd/` untouched by any
plan's cleanup.
### 4. SKILL.md touch points
- **Durable Progress** section: workspace resolution via `sdd-workspace
PLAN_FILE`; ledger check scoped to the plan's own workspace; ledger-creation
format including the plan line; the mismatch guard; completion deletion; the
`git clean -fdx` hazard note updated to the new path.
- **Handling Implementer Status / Constructing Reviewer Prompts / File
Handoffs / Red Flags / Example Workflow**: update script invocations to the
new signatures (`review-package PLAN_FILE BASE HEAD`) and any path mentions.
`implementer-prompt.md` and `task-reviewer-prompt.md` contain no workspace
paths (verified) and need no changes.
- Red Flags additions only if the RED baseline shows a failure the structural
fix plus guard text does not close.
## Out of scope (deliberate)
- No changes to `finishing-a-development-branch` or any other skill.
- No git-level guards against committing `.superpowers/` beyond the existing
parent `.gitignore`.
- No retroactive cleanup of the serf repo (separate follow-up).
- No legacy-layout migration or fallback reads.
## Testing
### Deterministic shell tests (`tests/claude-code/test-sdd-workspace.sh`, extended)
- `sdd-workspace PLAN` prints `<root>/.superpowers/sdd/<slug>` and creates it;
errors without a plan arg; errors on missing plan file.
- Two different plan files resolve to two distinct directories; artifacts
written via `task-brief` land in their own plan's directory.
- `review-package PLAN BASE HEAD` writes under the plan's directory.
- Parent `.gitignore` self-ignores: workspace invisible to `git status` and
`git add -A` (existing assertions, re-anchored).
- Linked-worktree distinctness (existing assertion, re-anchored).
- Existing suites `test-subagent-driven-development.sh` /
`-integration.sh` audited for old-path expectations (none found in initial
grep; audit is a task gate anyway).
### Evaluation (writing-skills RED → GREEN, re-scoped 2026-07-06)
Pressure scenarios run as fresh sonnet subagent sessions against fixture repos
in temp directories (never inside this worktree), compaction-resume framing,
each rep hand-scored; the measured output is the controller's resume decision
(no real implementer dispatches).
**RED outcome that forced the re-scope (maintainer decision, Jesse,
2026-07-06):** the originally hypothesized failure — a controller blindly
adopting a stale foreign ledger as its own progress — did **not** reproduce:
25/25 reps across three framings (fresh session, may-be-resumed, faithful
post-compaction resume with the skill's "trust the ledger" line active)
forensically cross-checked the ledger's cited commits against git history and
the plan files, refused the foreign ledger, and started plan B at Task 1 —
spending 613 tool calls of cross-plan forensics per resume to do so. Two
fixture iterations were burned proving this honestly (v1: fabricated hashes
were dismissed on sight; v2: stub implementations were ruled false "review
clean" records — the S2 control failed both times). Full record in the
committed eval docs.
**Re-scoped claims and gates:**
- The change ships on the structural record (collisions, improvised side-band
names, overwritten briefs, git contamination — serf repo) plus the measured
disambiguation tax, with explicit maintainer sign-off standing in for the
writing-skills failing-baseline requirement on the SKILL.md text.
- **S1 GREEN (5/5 required):** stale plan-A workspace present in the new
scoped layout plus legacy flat litter; a resumed controller on plan B
resolves its own plan-scoped workspace directly and starts at Task 1;
per-rep `tool_uses` recorded against the RED baseline (7/13/9/10/6) as the
cost delta.
- **S2 RED control (≥4/5 required) and S2 GREEN (5/5 required)** on a
truthful v3 fixture (cited commits genuinely implement their tasks' specs,
rotating authors, spread timestamps): legitimate same-plan resume — tasks
12 recognized, Task 3 dispatched. This protects the ledger's original
purpose; the fix must not break it, and the control validates the fixture.
Results land in `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md`
and are summarized in the PR.
## Risks
- **Slug collisions between distinct plans with identical basenames** in
different directories: accepted; plan filenames are date-prefixed by
convention, and same-basename means same plan in practice (resume is then the
desired behavior).
- **Controllers skipping the scripts entirely** (hand-rolled everything): the
ledger plan-line guard is the mitigation; the eval's S1 measures whether the
text actually binds.
- **Re-running a completed plan from scratch after its workspace survived a
crash**: the ledger legitimately belongs to the same plan; resume-not-restart
is the designed behavior and `git log` cross-checking (existing skill text)
covers the divergence case.

View File

@@ -63,7 +63,6 @@ digraph process {
"Read plan, note context and global constraints, create todos" [shape=box];
"More tasks remain?" [shape=diamond];
"Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [shape=box];
"Final review clean: delete this plan's workspace" [shape=box];
"Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen];
"Read plan, note context and global constraints, create todos" -> "Dispatch implementer subagent (./implementer-prompt.md)";
@@ -79,8 +78,7 @@ digraph process {
"Mark task complete in todo list and progress ledger" -> "More tasks remain?";
"More tasks remain?" -> "Dispatch implementer subagent (./implementer-prompt.md)" [label="yes"];
"More tasks remain?" -> "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [label="no"];
"Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Final review clean: delete this plan's workspace";
"Final review clean: delete this plan's workspace" -> "Use superpowers:finishing-a-development-branch";
"Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Use superpowers:finishing-a-development-branch";
}
```
@@ -135,7 +133,7 @@ that implementer. Single-file mechanical fixes also take the cheapest tier.
Implementer subagents report one of four statuses. Handle each appropriately:
**DONE:** Generate the review package (`scripts/review-package PLAN_FILE BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path.
**DONE:** Generate the review package (`scripts/review-package BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path.
**DONE_WITH_CONCERNS:** The implementer completed the work but flagged doubts. Read the concerns before proceeding. If the concerns are about correctness or scope, address them before review. If they're observations (e.g., "this file is getting large"), note them and proceed to review.
@@ -181,10 +179,10 @@ final whole-branch review. When you fill a reviewer template:
test hygiene, review method) — the constraints block is for what THIS
project's spec demands.
- Hand the reviewer its diff as a file: run this skill's
`scripts/review-package PLAN_FILE BASE HEAD` and pass the reviewer the
file path it prints (or, without bash: `git log --oneline`,
`git diff --stat`, and `git diff -U10` for the range, redirected to one
uniquely named file). The output never enters your own context, and the reviewer sees
`scripts/review-package BASE HEAD` and pass the reviewer the file path
it prints (or, without bash: `git log --oneline`, `git diff --stat`,
and `git diff -U10` for the range, redirected to one uniquely named
file). The output never enters your own context, and the reviewer sees
the commit list, stat summary, and full diff with context in one Read
call. Use the BASE you recorded before dispatching the implementer —
never `HEAD~1`, which silently truncates multi-commit tasks.
@@ -203,8 +201,8 @@ final whole-branch review. When you fill a reviewer template:
Do not dismiss the finding because the plan mandates it, and do not
dispatch a fix that contradicts the plan without asking.
- The final whole-branch review gets a package too: run
`scripts/review-package PLAN_FILE MERGE_BASE HEAD` (MERGE_BASE = the
commit the branch started from, e.g. `git merge-base main HEAD`) and include the
`scripts/review-package MERGE_BASE HEAD` (MERGE_BASE = the commit the
branch started from, e.g. `git merge-base main HEAD`) and include the
printed path in the final review dispatch, so the final reviewer reads
one file instead of re-deriving the branch diff with git commands.
- Every fix dispatch carries the implementer contract: the fix subagent
@@ -252,31 +250,18 @@ controllers that lost their place have re-dispatched entire completed task
sequences — the single most expensive failure observed. Track progress in
a ledger file, not only in todos.
- Each plan owns a workspace: at skill start, run this skill's
`scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored
directory (`<repo-root>/.superpowers/sdd/<plan-basename>/`), home to
every artifact for THIS plan: ledger, briefs, reports, review packages.
Another plan's directory is never yours to read or write.
- Check for this plan's ledger at `<workspace>/progress.md`. If its first
line names your plan file, tasks listed there as complete are DONE — do
not re-dispatch them; resume at the first task not marked complete. A
ledger whose first line names a different plan file — or a stray ledger
at the old flat path `.superpowers/sdd/progress.md` — is another plan's
progress: leave it in place and start your own, fresh.
- Create the ledger with its identity as the first line:
`# SDD ledger — plan: <plan file path>`.
- At skill start, check for a ledger:
`cat "$(git rev-parse --show-toplevel)/.superpowers/sdd/progress.md"`. Tasks listed there
as complete are DONE — do not re-dispatch them; resume at the first task
not marked complete.
- When a task's review comes back clean, append one line to the ledger in
the same message as your other bookkeeping:
`Task N: complete (commits <base7>..<head7>, review clean)`.
- The ledger is your recovery map: the commits it names exist in git even
when your context no longer remembers creating them. After compaction,
trust the ledger and `git log` over your own recollection.
- `git clean -fdx` will destroy the workspace (it's git-ignored scratch); if
- `git clean -fdx` will destroy the ledger (it's git-ignored scratch); if
that happens, recover from `git log`.
- When the final whole-branch review is clean and its fixes are merged,
delete this plan's workspace (`rm -rf <workspace>`) — the git history
is the record now. Sibling directories belong to other plans; leave
them alone.
## Prompt Templates
@@ -290,7 +275,6 @@ a ledger file, not only in todos.
You: I'm using Subagent-Driven Development to execute this plan.
[Read plan file once: docs/superpowers/plans/feature-plan.md]
[Resolve workspace: scripts/sdd-workspace docs/superpowers/plans/feature-plan.md — no ledger inside, fresh start]
[Create todos for all tasks]
Task 1: Hook installation script
@@ -345,8 +329,6 @@ Task reviewer: Spec ✅. Task quality: Approved.
[Dispatch final code-reviewer]
Final reviewer: All requirements met, ready to merge
[Delete this plan's workspace — the record now lives in git]
Done!
```
@@ -400,8 +382,8 @@ Done!
dispatch prompt ("treat it as Minor at most") — the plan's example code is
a starting point, not evidence that its weaknesses were chosen
- Dispatch a task reviewer without a diff file — generate it first
(`scripts/review-package PLAN_FILE BASE HEAD`) and name the printed
path in the prompt
(`scripts/review-package BASE HEAD`) and name the printed path in the
prompt
- Move to next task while the review has open Critical/Important issues
- Re-dispatch a task the progress ledger already marks complete — check
the ledger (and `git log`) after any compaction or resume

View File

@@ -4,28 +4,26 @@
# call. Using the recorded per-task BASE (not HEAD~1) keeps multi-commit
# tasks intact.
#
# Usage: review-package PLAN_FILE BASE HEAD [OUTFILE]
# Default OUTFILE: <repo-root>/.superpowers/sdd/<plan-basename>/review-<base7>..<head7>.diff
# Usage: review-package BASE HEAD [OUTFILE]
# Default OUTFILE: <repo-root>/.superpowers/sdd/review-<base7>..<head7>.diff
# (named per range, so a re-review after fixes gets a distinct fresh file).
set -euo pipefail
if [ $# -lt 3 ] || [ $# -gt 4 ]; then
echo "usage: review-package PLAN_FILE BASE HEAD [OUTFILE]" >&2
if [ $# -lt 2 ] || [ $# -gt 3 ]; then
echo "usage: review-package BASE HEAD [OUTFILE]" >&2
exit 2
fi
plan=$1
base=$2
head=$3
[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; }
base=$1
head=$2
git rev-parse --verify --quiet "$base" >/dev/null || { echo "bad BASE: $base" >&2; exit 2; }
git rev-parse --verify --quiet "$head" >/dev/null || { echo "bad HEAD: $head" >&2; exit 2; }
if [ $# -eq 4 ]; then
out=$4
if [ $# -eq 3 ]; then
out=$3
else
dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan")
dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace")
out="$dir/review-$(git rev-parse --short "$base")..$(git rev-parse --short "$head").diff"
fi

View File

@@ -1,40 +1,22 @@
#!/usr/bin/env bash
# Resolve and ensure the working-tree directory SDD uses for one plan's
# short-lived artifacts: task briefs, implementer reports, review packages,
# and the progress ledger. Print the plan directory's absolute path.
#
# One directory per plan (.superpowers/sdd/<plan-basename>/) so a follow-up
# plan in the same working tree can never read or overwrite another plan's
# artifacts. A stale ledger misread as current progress makes controllers
# skip whole task sequences — plan-scoping removes that failure structurally.
# Resolve and ensure the working-tree directory SDD uses for its short-lived
# artifacts: task briefs, implementer reports, review packages, and the
# progress ledger. Print the directory's absolute path.
#
# The workspace lives in the working tree (not under .git/) because Claude Code
# treats .git/ as a protected path and denies agent writes there — which blocks
# an implementer subagent from writing its report file. A self-ignoring
# .gitignore at .superpowers/sdd/ keeps every plan's workspace out of
# `git status` and out of accidental commits without modifying any tracked file.
# .gitignore keeps the workspace out of `git status` and out of accidental
# commits without modifying any tracked file.
#
# Single source of truth for the workspace location, so task-brief and
# review-package cannot drift to different directories.
#
# Usage: sdd-workspace PLAN_FILE
# Usage: sdd-workspace
set -euo pipefail
if [ $# -ne 1 ]; then
echo "usage: sdd-workspace PLAN_FILE" >&2
exit 2
fi
plan=$1
[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; }
slug=$(basename "$plan" .md)
[ -n "$slug" ] && [ "$slug" != "." ] && [ "$slug" != ".." ] \
|| { echo "cannot derive a workspace name from: $plan" >&2; exit 2; }
root=$(git rev-parse --show-toplevel)
base="$root/.superpowers/sdd"
dir="$base/$slug"
dir="$root/.superpowers/sdd"
mkdir -p "$dir"
printf '*\n' > "$base/.gitignore"
printf '*\n' > "$dir/.gitignore"
cd "$dir" && pwd

View File

@@ -4,9 +4,8 @@
# through the controller's context.
#
# Usage: task-brief PLAN_FILE TASK_NUMBER [OUTFILE]
# Default OUTFILE: <repo-root>/.superpowers/sdd/<plan-basename>/task-<N>-brief.md
# (per plan and per worktree; concurrent runs of the SAME plan in the same
# working tree share it).
# Default OUTFILE: <repo-root>/.superpowers/sdd/task-<N>-brief.md
# (per worktree; concurrent runs in the same working tree share it).
set -euo pipefail
if [ $# -lt 2 ] || [ $# -gt 3 ]; then
@@ -21,7 +20,7 @@ n=$2
if [ $# -eq 3 ]; then
out=$3
else
dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan")
dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace")
out="$dir/task-${n}-brief.md"
fi

View File

@@ -178,8 +178,8 @@ Subagent (general-purpose):
- `[BASE_SHA]` — commit before this task
- `[HEAD_SHA]` — current commit
- `[DIFF_FILE]` — REQUIRED: the path the controller wrote the review
package to (`scripts/review-package PLAN_FILE BASE HEAD` prints the unique
path it wrote; the package never enters the controller's context)
package to (`scripts/review-package BASE HEAD` prints the unique path it
wrote; the package never enters the controller's context)
**Reviewer returns:** Spec Compliance verdict (✅/❌/⚠️), Strengths, Issues
(Critical/Important/Minor), Task quality verdict

View File

@@ -203,6 +203,12 @@ Next failing test for next feature.
| **Clear** | Name describes behavior | `test('test1')` |
| **Shows intent** | Demonstrates desired API | Obscures what code should do |
When writing or changing any test, read [writing-good-tests.md](writing-good-tests.md) for the rules that keep tests honest:
- Name the production change that would make the test fail — before writing it
- Assert on real behavior, never on mock behavior
- Keep test-only code in test utilities, out of production classes
- Understand a dependency's side effects before mocking it
## Why Order Matters
**"I'll write tests after to verify it works"**
@@ -354,13 +360,6 @@ Bug found? Write failing test reproducing it. Follow TDD cycle. Test proves fix
Never fix bugs without a test.
## Testing Anti-Patterns
When adding mocks or test utilities, read [testing-anti-patterns.md](testing-anti-patterns.md) to avoid common pitfalls:
- Testing mock behavior instead of real behavior
- Adding test-only methods to production classes
- Mocking without understanding dependencies
## Final Rule
```

View File

@@ -1,299 +0,0 @@
# Testing Anti-Patterns
**Load this reference when:** writing or changing tests, adding mocks, or tempted to add test-only methods to production code.
## Overview
Tests must verify real behavior, not mock behavior. Mocks are a means to isolate, not the thing being tested.
**Core principle:** Test what the code does, not what the mocks do.
**Following strict TDD prevents these anti-patterns.**
## The Iron Laws
```
1. NEVER test mock behavior
2. NEVER add test-only methods to production classes
3. NEVER mock without understanding dependencies
```
## Anti-Pattern 1: Testing Mock Behavior
**The violation:**
```typescript
// ❌ BAD: Testing that the mock exists
test('renders sidebar', () => {
render(<Page />);
expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument();
});
```
**Why this is wrong:**
- You're verifying the mock works, not that the component works
- Test passes when mock is present, fails when it's not
- Tells you nothing about real behavior
**your human partner's correction:** "Are we testing the behavior of a mock?"
**The fix:**
```typescript
// ✅ GOOD: Test real component or don't mock it
test('renders sidebar', () => {
render(<Page />); // Don't mock sidebar
expect(screen.getByRole('navigation')).toBeInTheDocument();
});
// OR if sidebar must be mocked for isolation:
// Don't assert on the mock - test Page's behavior with sidebar present
```
### Gate Function
```
BEFORE asserting on any mock element:
Ask: "Am I testing real component behavior or just mock existence?"
IF testing mock existence:
STOP - Delete the assertion or unmock the component
Test real behavior instead
```
## Anti-Pattern 2: Test-Only Methods in Production
**The violation:**
```typescript
// ❌ BAD: destroy() only used in tests
class Session {
async destroy() { // Looks like production API!
await this._workspaceManager?.destroyWorkspace(this.id);
// ... cleanup
}
}
// In tests
afterEach(() => session.destroy());
```
**Why this is wrong:**
- Production class polluted with test-only code
- Dangerous if accidentally called in production
- Violates YAGNI and separation of concerns
- Confuses object lifecycle with entity lifecycle
**The fix:**
```typescript
// ✅ GOOD: Test utilities handle test cleanup
// Session has no destroy() - it's stateless in production
// In test-utils/
export async function cleanupSession(session: Session) {
const workspace = session.getWorkspaceInfo();
if (workspace) {
await workspaceManager.destroyWorkspace(workspace.id);
}
}
// In tests
afterEach(() => cleanupSession(session));
```
### Gate Function
```
BEFORE adding any method to production class:
Ask: "Is this only used by tests?"
IF yes:
STOP - Don't add it
Put it in test utilities instead
Ask: "Does this class own this resource's lifecycle?"
IF no:
STOP - Wrong class for this method
```
## Anti-Pattern 3: Mocking Without Understanding
**The violation:**
```typescript
// ❌ BAD: Mock breaks test logic
test('detects duplicate server', () => {
// Mock prevents config write that test depends on!
vi.mock('ToolCatalog', () => ({
discoverAndCacheTools: vi.fn().mockResolvedValue(undefined)
}));
await addServer(config);
await addServer(config); // Should throw - but won't!
});
```
**Why this is wrong:**
- Mocked method had side effect test depended on (writing config)
- Over-mocking to "be safe" breaks actual behavior
- Test passes for wrong reason or fails mysteriously
**The fix:**
```typescript
// ✅ GOOD: Mock at correct level
test('detects duplicate server', () => {
// Mock the slow part, preserve behavior test needs
vi.mock('MCPServerManager'); // Just mock slow server startup
await addServer(config); // Config written
await addServer(config); // Duplicate detected ✓
});
```
### Gate Function
```
BEFORE mocking any method:
STOP - Don't mock yet
1. Ask: "What side effects does the real method have?"
2. Ask: "Does this test depend on any of those side effects?"
3. Ask: "Do I fully understand what this test needs?"
IF depends on side effects:
Mock at lower level (the actual slow/external operation)
OR use test doubles that preserve necessary behavior
NOT the high-level method the test depends on
IF unsure what test depends on:
Run test with real implementation FIRST
Observe what actually needs to happen
THEN add minimal mocking at the right level
Red flags:
- "I'll mock this to be safe"
- "This might be slow, better mock it"
- Mocking without understanding the dependency chain
```
## Anti-Pattern 4: Incomplete Mocks
**The violation:**
```typescript
// ❌ BAD: Partial mock - only fields you think you need
const mockResponse = {
status: 'success',
data: { userId: '123', name: 'Alice' }
// Missing: metadata that downstream code uses
};
// Later: breaks when code accesses response.metadata.requestId
```
**Why this is wrong:**
- **Partial mocks hide structural assumptions** - You only mocked fields you know about
- **Downstream code may depend on fields you didn't include** - Silent failures
- **Tests pass but integration fails** - Mock incomplete, real API complete
- **False confidence** - Test proves nothing about real behavior
**The Iron Rule:** Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses.
**The fix:**
```typescript
// ✅ GOOD: Mirror real API completeness
const mockResponse = {
status: 'success',
data: { userId: '123', name: 'Alice' },
metadata: { requestId: 'req-789', timestamp: 1234567890 }
// All fields real API returns
};
```
### Gate Function
```
BEFORE creating mock responses:
Check: "What fields does the real API response contain?"
Actions:
1. Examine actual API response from docs/examples
2. Include ALL fields system might consume downstream
3. Verify mock matches real response schema completely
Critical:
If you're creating a mock, you must understand the ENTIRE structure
Partial mocks fail silently when code depends on omitted fields
If uncertain: Include all documented fields
```
## Anti-Pattern 5: Integration Tests as Afterthought
**The violation:**
```
✅ Implementation complete
❌ No tests written
"Ready for testing"
```
**Why this is wrong:**
- Testing is part of implementation, not optional follow-up
- TDD would have caught this
- Can't claim complete without tests
**The fix:**
```
TDD cycle:
1. Write failing test
2. Implement to pass
3. Refactor
4. THEN claim complete
```
## When Mocks Become Too Complex
**Warning signs:**
- Mock setup longer than test logic
- Mocking everything to make test pass
- Mocks missing methods real components have
- Test breaks when mock changes
**your human partner's question:** "Do we need to be using a mock here?"
**Consider:** Integration tests with real components often simpler than complex mocks
## TDD Prevents These Anti-Patterns
**Why TDD helps:**
1. **Write test first** → Forces you to think about what you're actually testing
2. **Watch it fail** → Confirms test tests real behavior, not mocks
3. **Minimal implementation** → No test-only methods creep in
4. **Real dependencies** → You see what the test actually needs before mocking
**If you're testing mock behavior, you violated TDD** - you added mocks without watching test fail against real code first.
## Quick Reference
| Anti-Pattern | Fix |
|--------------|-----|
| Assert on mock elements | Test real component or unmock it |
| Test-only methods in production | Move to test utilities |
| Mock without understanding | Understand dependencies first, mock minimally |
| Incomplete mocks | Mirror real API completely |
| Tests as afterthought | TDD - tests first |
| Over-complex mocks | Consider integration tests |
## Red Flags
- Assertion checks for `*-mock` test IDs
- Methods only called in test files
- Mock setup is >50% of test
- Test fails when you remove mock
- Can't explain why mock is needed
- Mocking "just to be safe"
## The Bottom Line
**Mocks are tools to isolate, not things to test.**
If TDD reveals you're testing mock behavior, you've gone wrong.
Fix: Test real behavior or question why you're mocking at all.

View File

@@ -0,0 +1,198 @@
# Writing Good Tests
**Load this reference when:** writing or changing tests, adding mocks, or
adding cleanup/helper methods for tests.
## Overview
A test exists to catch a specific break. Two principles govern everything
here:
```
1. Every test names the break it catches
2. Every test exercises the real thing
```
Strict TDD produces both naturally: a test written first and watched
failing against real code has already proven it can fail, and only earns
a mock when the real dependency proves slow or external.
## Principle 1: Name the Break
Before writing the test body, answer: **what production change should
make this test fail — and is that change a bug or a decision?** A test
earns its place by catching a wrong branch, missing side effect, wrong
argument, boundary case, or broken contract.
**Derive expectations independently.** Use literals and hand-checked
fixtures; table-driven tests with literal `want` values are the preferred
shape. An expectation computed by the code under test — or its helpers —
passes no matter what that code does:
```typescript
// ❌ Mirror assertion: the same builder computes both sides — always true
const expected = buildSearchQuery({ tag: 'urgent' });
expect(buildSearchQuery({ tag: 'urgent' })).toBe(expected);
// ✅ Hand-derived literal
expect(buildSearchQuery({ tag: 'urgent' })).toBe('tag:"urgent"');
```
**No change detectors.** If only intentional decisions can fail a test —
a constant's value, exact message wording, private structure — it fires
on redesign and sleeps through bugs. Test the behavior that depends on
the decision: not `expect(MAX_RETRIES).toBe(5)` but "a failing call is
retried 5 times and the 6th attempt never happens."
**Behavior, not text.** Asserting that a script, skill, or config
contains an exact line proves only that the source is the source. Run
scripts against controlled inputs and assert outputs, side effects, or
exit codes. Documents that instruct agents are tested by the consuming
agent's behavior (superpowers:writing-skills); prose for humans earns no
test at all.
**Your code, not the framework.** Test the contract your code makes at
its boundaries — the route you register, the query you emit, the payload
you produce. Upstream mechanics are their maintainers' tests to write
(the classic: asserting your router invokes a registered handler — that
is the framework's test, not yours). When upstream behavior genuinely
surprised you, write one narrow characterization test naming the
assumption. The same boundary applies inside your code: constructors,
getters, constants, and trivial forwarding earn tests only when they
validate, normalize, default, derive, enforce, or cause side effects —
otherwise assert the first consumer-visible result that depends on them.
### Gate Function
```
BEFORE writing the test body:
Name the production change that would make this test fail.
Cannot name one → redesign around an observable behavior
"The source text changed" → run the artifact and assert its effects
Only intentional decisions → change detector; test the behavior
that depends on the decision
Confirm the expected value is derived without the code under test.
IF it reuses the code's logic or helpers:
Replace it with a literal or hand-checked fixture
```
## Principle 2: Exercise the Real Thing
**The mock earns no assertions.** A mock assertion passes when the mock
is present and fails when it is absent — it says nothing about the
component. Assert the real component's behavior; if the mock is what you
are checking, unmock it or delete the assertion.
```typescript
// ✅ Real behavior
expect(screen.getByRole('navigation')).toBeInTheDocument();
// ❌ Mock existence
expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument();
```
**your human partner's correction:** "Are we testing the behavior of a
mock?"
**Mock at the right level.** Learn every side effect of the real method
before replacing it; mock the slow or external operation and keep what
the test depends on real. When unsure, run the test against the real
implementation first and observe what actually needs to happen.
```typescript
// ❌ The mock swallows the config write that duplicate detection reads
vi.mock('ToolCatalog', () => ({
discoverAndCacheTools: vi.fn().mockResolvedValue(undefined)
}));
// ✅ Mock only the slow server startup; the config write stays real
vi.mock('MCPServerManager');
```
**Make doubles specific.** When arguments, call counts, or ordering are
part of the contract, assert them — a fake that accepts anything verifies
nothing. Give each branch (success, error, malformed) its own fixture or
spy, so the wrong branch cannot satisfy the expectation.
**Mirror real data completely.** Mock the complete structure as it exists
in reality — all documented fields — not just the ones your test reads.
Partial mocks fail silently when downstream code reads an omitted field:
the test passes while integration breaks.
**Production classes carry production methods only.** Cleanup that only
tests need lives in test utilities, never as a `destroy()` on the
production class. Ask: is this method called only from tests? Does this
class own this resource's lifecycle? Wrong answers → test utility.
**Prefer real components over complex mocks.** When mock setup outgrows
the test logic, mocks miss methods the real components have, or tests
break when the mock changes, switch to an integration test with real
components. **your human partner's question:** "Do we need to be using a
mock here?"
### Gate Function
```
BEFORE adding a mock or test helper:
List the real method's side effects; keep the ones the test
depends on real — mock the slow/external level below them.
Mock responses mirror the complete real structure.
A method only tests call lives in test utilities, not production.
About to assert on the mock itself?
Unmock it or delete the assertion.
```
## Tests Ship With the Implementation
The TDD cycle — failing test, minimal implementation, refactor — is what
"complete" means. Ship the tests the behavior needs and only those:
trivial code and human prose earn none, and a test written to satisfy
process costs maintenance forever.
## The Mutation Check
Before finishing, mentally mutate the production code; at least one test
should fail for each realistic mutation:
- Wrong constant or argument
- Wrong branch handler
- Missing state change or side effect
- Empty or default return
- Missing validation for zero, empty, nil, unauthorized, or malformed input
A mutation nothing catches marks the behavior as unprotected — or the
test as tautological.
## Quick Reference
| When you... | Do |
|-------------|-----|
| Write any test | Name the break it catches — a bug, not a decision |
| Build an expected value | Derive it by hand; never with the code under test |
| Test a script or document | Run it / pressure-test its consumer; never grep its text |
| Reach for a dependency test | Test your boundary contract, not their documented mechanics |
| Want to assert on a mocked element | Test the real component, or unmock it |
| Are about to mock a method | Learn its side effects; mock the slow/external level |
| Build a mock response | Mirror the real structure completely |
| Need cleanup only tests use | Put it in test utilities |
| Watch mock setup balloon | Switch to an integration test with real components |
| Finish a test file | Run the mutation check |
## Warning Signs
- Setup and assertion share the same object, guaranteeing equality
- The test can fail only through a panic, crash, or missing selector
- The test fails on every intentional change, never on accidental breakage
- Expected values are hidden behind loops, builders, or helpers
- The test greps source text, or asserts a removed symbol stays removed
- The test would still matter if only the framework remained
- The test exists for coverage, checking no side effect or outcome
- An assertion checks a `*-mock` test ID, or fails if you remove the mock
- A method is called only from test files
- Mock setup is more than half the test, or you can't explain why the mock is needed
- Mocking "just to be safe"

View File

@@ -1,7 +1,6 @@
#!/usr/bin/env bash
# Tests for the SDD workspace: scripts/sdd-workspace resolves a self-ignoring,
# PER-PLAN working-tree directory for SDD artifacts, and the SDD scripts write
# into their plan's directory.
# Tests for the SDD workspace: scripts/sdd-workspace resolves a self-ignoring
# working-tree directory for SDD artifacts, and the SDD scripts write into it.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -36,72 +35,26 @@ main() {
local repo
repo="$(cd "$TEST_ROOT/repo" && git rev-parse --show-toplevel)"
cat > "$repo/plan-a.md" <<'PLAN'
# Plan A
local dir
dir="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace")"
## Task 1: First thing
Do the first thing.
PLAN
cat > "$repo/plan-b.md" <<'PLAN'
# Plan B
## Task 1: Other thing
Do the other thing.
PLAN
# --- argument validation ---
local rc=0
(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" >/dev/null 2>&1) || rc=$?
if [[ "$rc" -eq 2 ]]; then
pass "sdd-workspace without a plan errors with exit 2"
if [[ "$dir" == "$repo/.superpowers/sdd" ]]; then
pass "prints <repo-root>/.superpowers/sdd"
else
fail "sdd-workspace without a plan errors with exit 2"
echo " exit: $rc"
fi
rc=0
(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" no-such-plan.md >/dev/null 2>&1) || rc=$?
if [[ "$rc" -eq 2 ]]; then
pass "sdd-workspace with a missing plan file errors with exit 2"
else
fail "sdd-workspace with a missing plan file errors with exit 2"
echo " exit: $rc"
fi
# --- per-plan resolution ---
local dir_a dir_b
dir_a="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" plan-a.md)"
dir_b="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" plan-b.md)"
if [[ "$dir_a" == "$repo/.superpowers/sdd/plan-a" ]]; then
pass "prints <repo-root>/.superpowers/sdd/<plan-basename>"
else
fail "prints <repo-root>/.superpowers/sdd/<plan-basename>"
echo " got: $dir_a"
fi
if [[ "$dir_a" != "$dir_b" && -d "$dir_a" && -d "$dir_b" ]]; then
pass "two plans resolve to two distinct directories"
else
fail "two plans resolve to two distinct directories"
echo " a: $dir_a"
echo " b: $dir_b"
fail "prints <repo-root>/.superpowers/sdd"
echo " got: $dir"
fi
if [[ -f "$repo/.superpowers/sdd/.gitignore" && "$(cat "$repo/.superpowers/sdd/.gitignore")" == "*" ]]; then
pass "self-ignoring .gitignore created at .superpowers/sdd/ with '*'"
pass "self-ignoring .gitignore created with '*'"
else
fail "self-ignoring .gitignore created at .superpowers/sdd/ with '*'"
fail "self-ignoring .gitignore created with '*'"
fi
printf 'x\n' > "$dir_a/artifact.md"
printf 'x\n' > "$repo/.superpowers/sdd/artifact.md"
local status
status="$(cd "$repo" && git status --porcelain)"
# plan-a.md/plan-b.md are intentionally untracked fixture files; only the
# workspace must be invisible.
if [[ "$status" != *".superpowers"* ]]; then
if [[ -z "$status" ]]; then
pass "workspace invisible to git status"
else
fail "workspace invisible to git status"
@@ -111,78 +64,67 @@ PLAN
( cd "$repo" && git add -A )
local staged
staged="$(cd "$repo" && git diff --cached --name-only)"
if [[ "$staged" != *".superpowers"* ]]; then
if [[ -z "$staged" ]]; then
pass "git add -A does not stage the workspace"
else
fail "git add -A does not stage the workspace"
echo " staged: $staged"
fi
# --- task-brief lands in its plan's directory ---
local brief_out brief_path
brief_out="$(cd "$repo" && "$SDD_SCRIPTS/task-brief" plan-a.md 1)"
brief_path="$(printf '%s\n' "$brief_out" | sed -n 's/^wrote \(.*\): [0-9][0-9]* lines$/\1/p')"
if [[ "$brief_path" == "$repo/.superpowers/sdd/plan-a/task-1-brief.md" ]]; then
pass "task-brief writes its brief under the plan's workspace"
else
fail "task-brief writes its brief under the plan's workspace"
echo " got: $brief_path"
fi
cat > "$repo/plan.md" <<'PLAN'
# Plan
## Task 1: First thing
Do the first thing.
PLAN
local brief_out brief_path
brief_out="$(cd "$repo" && "$SDD_SCRIPTS/task-brief" plan.md 1)"
brief_path="$(printf '%s\n' "$brief_out" | sed -n 's/^wrote \(.*\): [0-9][0-9]* lines$/\1/p')"
case "$brief_path" in
"$repo/.superpowers/sdd/"*) pass "task-brief writes its brief under the workspace" ;;
*)
fail "task-brief writes its brief under the workspace"
echo " got: $brief_path"
;;
esac
# --- review-package takes the plan first and lands in its directory ---
local git_id=(-c user.email=t@example.com -c user.name=t -c commit.gpgsign=false)
( cd "$repo" \
&& git add plan.md \
&& git "${git_id[@]}" commit -qm c1 \
&& printf 'y\n' > f && git add f \
&& git "${git_id[@]}" commit -qm c2 )
local rp_out rp_path
rp_out="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md HEAD~1 HEAD)"
rp_out="$(cd "$repo" && "$SDD_SCRIPTS/review-package" HEAD~1 HEAD)"
rp_path="$(printf '%s\n' "$rp_out" | sed -n 's/^wrote \(.*\): [0-9].*$/\1/p')"
case "$rp_path" in
"$repo/.superpowers/sdd/plan-a/review-"*.diff)
pass "review-package writes its diff under the plan's workspace" ;;
"$repo/.superpowers/sdd/"*) pass "review-package writes its diff under the workspace" ;;
*)
fail "review-package writes its diff under the plan's workspace"
fail "review-package writes its diff under the workspace"
echo " got: $rp_path"
;;
esac
rc=0
(cd "$repo" && "$SDD_SCRIPTS/review-package" HEAD~1 HEAD >/dev/null 2>&1) || rc=$?
if [[ "$rc" -eq 2 ]]; then
pass "review-package without a plan errors with exit 2"
else
fail "review-package without a plan errors with exit 2"
echo " exit: $rc"
fi
local rp_explicit
rp_explicit="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md HEAD~1 HEAD "$TEST_ROOT/explicit.diff")"
if [[ -s "$TEST_ROOT/explicit.diff" && "$rp_explicit" == *"$TEST_ROOT/explicit.diff"* ]]; then
pass "review-package honors an explicit OUTFILE"
else
fail "review-package honors an explicit OUTFILE"
echo " got: $rp_explicit"
fi
# --- Worktree isolation: a linked worktree resolves its own workspace ---
local wt="$TEST_ROOT/wt"
( cd "$repo" && git worktree add -q "$wt" -b wt-feature )
local wt_root wt_dir
wt_root="$(cd "$wt" && git rev-parse --show-toplevel)"
wt_dir="$(cd "$wt" && "$SDD_SCRIPTS/sdd-workspace" plan-a.md)"
if [[ "$wt_dir" == "$wt_root/.superpowers/sdd/plan-a" && "$wt_dir" != "$dir_a" ]]; then
wt_dir="$(cd "$wt" && "$SDD_SCRIPTS/sdd-workspace")"
if [[ "$wt_dir" == "$wt_root/.superpowers/sdd" && "$wt_dir" != "$dir" ]]; then
pass "linked worktree resolves its own distinct workspace"
else
fail "linked worktree resolves its own distinct workspace"
echo " main: $dir_a"
echo " main: $dir"
echo " wt: $wt_dir"
fi
printf 'y\n' > "$wt_dir/artifact.md"
printf 'y\n' > "$wt/.superpowers/sdd/artifact.md"
local wt_status
wt_status="$(cd "$wt" && git status --porcelain)"
if [[ "$wt_status" != *".superpowers"* ]]; then
if [[ -z "$wt_status" ]]; then
pass "worktree workspace invisible to git status"
else
fail "worktree workspace invisible to git status"