From 1d4c8d2aafb8fa0de3e5d7df80ff44899fa7e402 Mon Sep 17 00:00:00 2001 From: Ada Sen Date: Fri, 10 Jul 2026 15:12:38 +0000 Subject: [PATCH 01/51] Revert "Remove Gemini CLI support" This reverts commit 711d895ce736cbcc5fb0c219ea3f49277f17fa8c. --- CLAUDE.md | 2 +- README.md | 16 ++++- skills/brainstorming/visual-companion.md | 7 +++ skills/executing-plans/SKILL.md | 2 +- .../references/gemini-tools.md | 63 +++++++++++++++++++ skills/writing-skills/SKILL.md | 2 +- 6 files changed, 88 insertions(+), 4 deletions(-) create mode 100644 skills/using-superpowers/references/gemini-tools.md diff --git a/CLAUDE.md b/CLAUDE.md index f8e45e9d..5f3d7410 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,7 +101,7 @@ Skills are not prose — they are code that shapes agent behavior. If you modify ## Eval harness -Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. The harness drives real tmux sessions of Claude Code / Codex and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`. +Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. Drill (the harness) drives real tmux sessions of Claude Code / Codex / Gemini CLI and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`. ## Understand the Project Before Contributing diff --git a/README.md b/README.md index 48e3f198..bb398c6b 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ If this sounds like someone you know, definitely send them our way. ## Quickstart -Give your agent Superpowers: [Claude Code](#claude-code), [Antigravity](#antigravity), [Codex App](#codex-app), [Codex CLI](#codex-cli), [Cursor](#cursor), [Factory Droid](#factory-droid), [GitHub Copilot CLI](#github-copilot-cli), [Kimi Code](#kimi-code), [OpenCode](#opencode), [Pi](#pi). +Give your agent Superpowers: [Claude Code](#claude-code), [Antigravity](#antigravity), [Codex App](#codex-app), [Codex CLI](#codex-cli), [Cursor](#cursor), [Factory Droid](#factory-droid), [Gemini CLI](#gemini-cli), [GitHub Copilot CLI](#github-copilot-cli), [Kimi Code](#kimi-code), [OpenCode](#opencode), [Pi](#pi). ## How it works @@ -122,6 +122,20 @@ Superpowers is available via the [official Codex plugin marketplace](https://git droid plugin install superpowers@superpowers ``` +### Gemini CLI + +- Install the extension: + + ```bash + gemini extensions install https://github.com/obra/superpowers + ``` + +- Update later: + + ```bash + gemini extensions update superpowers + ``` + ### GitHub Copilot CLI - Register the marketplace: diff --git a/skills/brainstorming/visual-companion.md b/skills/brainstorming/visual-companion.md index 7b89f6b2..906c9ac8 100644 --- a/skills/brainstorming/visual-companion.md +++ b/skills/brainstorming/visual-companion.md @@ -74,6 +74,13 @@ On Windows, the script auto-detects and switches to foreground mode (which block scripts/start-server.sh --project-dir /path/to/project --open ``` +**Gemini CLI:** +```bash +# Use --foreground and set is_background: true on your shell tool call +# so the process survives across turns +scripts/start-server.sh --project-dir /path/to/project --open --foreground +``` + **Copilot CLI:** ```bash # Use --foreground and start the server via the bash tool with mode: "async" diff --git a/skills/executing-plans/SKILL.md b/skills/executing-plans/SKILL.md index 075a1038..78d88540 100644 --- a/skills/executing-plans/SKILL.md +++ b/skills/executing-plans/SKILL.md @@ -11,7 +11,7 @@ Load plan, review critically, execute all tasks, report when complete. **Announce at start:** "I'm using the executing-plans skill to implement this plan." -**Note:** Tell your human partner that Superpowers works much better with access to subagents. The quality of its work will be significantly higher if run on a platform with subagent support (Claude Code, Codex CLI, Codex App, and Copilot CLI all qualify; see the per-platform tool refs in `../using-superpowers/references/`). If subagents are available, use superpowers:subagent-driven-development instead of this skill. +**Note:** Tell your human partner that Superpowers works much better with access to subagents. The quality of its work will be significantly higher if run on a platform with subagent support (Claude Code, Codex CLI, Codex App, Copilot CLI, and Gemini CLI all qualify; see the per-platform tool refs in `../using-superpowers/references/`). If subagents are available, use superpowers:subagent-driven-development instead of this skill. ## The Process diff --git a/skills/using-superpowers/references/gemini-tools.md b/skills/using-superpowers/references/gemini-tools.md new file mode 100644 index 00000000..b01b6523 --- /dev/null +++ b/skills/using-superpowers/references/gemini-tools.md @@ -0,0 +1,63 @@ +# Gemini CLI Tool Mapping + +Skills speak in actions ("dispatch a subagent", "create a todo", "read a file"). On Gemini CLI these resolve to the tools below. + +| Action skills request | Gemini CLI equivalent | +|----------------------|----------------------| +| Read a file | `read_file` | +| Read multiple files at once | `read_many_files` | +| Create a new file | `write_file` | +| Edit a file | `replace` | +| Run a shell command | `run_shell_command` | +| Search file contents | `grep_search` | +| Find files by name | `glob` | +| List files and subdirectories | `list_directory` | +| Fetch a URL | `web_fetch` | +| Search the web | `google_web_search` | +| Invoke a skill | `activate_skill` | +| Dispatch a subagent (`Subagent (general-purpose):` template) | `invoke_agent` with `agent_name: "generalist"` (invocable via `@generalist` chat syntax — see [Subagent support](#subagent-support)) | +| Multiple parallel dispatches | Multiple `invoke_agent` calls in the same response | +| Task tracking ("create a todo", "mark complete") | `write_todos` (statuses: pending, in_progress, completed, cancelled, blocked) | + +## Instructions file + +When a skill mentions "your instructions file", on Gemini CLI this is **`GEMINI.md`**. Gemini CLI loads `GEMINI.md` hierarchically: global at `~/.gemini/GEMINI.md`, project-level files in workspace directories and their ancestors, and sub-directory `GEMINI.md` files when a tool accesses files in those directories. + +## Personal skills directory + +User-level skills live at **`~/.gemini/skills/`**, with **`~/.agents/skills/`** as a cross-runtime alias (shared with Codex and Copilot CLI). When both directories exist at the same scope, `.agents/skills/` takes precedence. Each skill is a subdirectory containing a `SKILL.md` (with `name` and `description` frontmatter). + +## Subagent support + +Gemini CLI dispatches subagents through the `invoke_agent` tool, which takes `agent_name` and `prompt` parameters. The same dispatch is also surfaced as a chat-syntax shortcut: typing `@generalist ` is equivalent to calling `invoke_agent` with `agent_name: "generalist"`. Built-in agent names include `generalist`, `cli_help`, `codebase_investigator`, and (with browser tooling enabled) `browser_agent`. + +Skills dispatch with `Subagent (general-purpose):` and either reference a prompt-template file (e.g., `superpowers:subagent-driven-development`'s `./implementer-prompt.md`) or supply an inline prompt. On Gemini CLI: + +| Skill dispatch form | Gemini CLI equivalent | +|---------------------|----------------------| +| References a `*-prompt.md` template (implementer, task-reviewer, code-reviewer, etc.) | Fill the template, then `invoke_agent` with `agent_name: "generalist"` and the filled prompt | +| References `superpowers:requesting-code-review`'s `./code-reviewer.md` | `invoke_agent` with `agent_name: "generalist"` and the filled review template | +| Inline prompt (no template referenced) | `invoke_agent` with `agent_name: "generalist"` and your inline prompt | + +### Prompt filling + +Skills provide prompt templates with placeholders like `{WHAT_WAS_IMPLEMENTED}` or `[FULL TEXT of task]`. Fill all placeholders before passing the complete prompt to `invoke_agent`. The prompt template itself contains the agent's role, review criteria, and expected output format — the subagent will follow it. + +### Parallel dispatch + +Gemini CLI supports parallel subagent dispatch. Issue multiple `invoke_agent` calls in the same response (or multiple `@generalist` invocations in one prompt) to run independent subagent work in parallel. Keep dependent tasks sequential, but do not serialize independent subagent tasks just to preserve a simpler history. + +## Additional Gemini CLI tools + +These tools are unique to Gemini CLI: + +| Tool | Purpose | +|------|---------| +| `save_memory` (legacy) | Persist facts across sessions when `experimental.memoryV2 = false` | +| `get_internal_docs` | Look up Gemini CLI's bundled documentation | +| `ask_user` | Pose structured questions to the user (text / single-select / multi-select) | +| `enter_plan_mode` / `exit_plan_mode` | Switch into and out of read-only plan mode | +| `update_topic` | Update the current conversation's topic / strategic-intent metadata | +| `complete_task` | Signal that a Gemini subagent has completed and return its result to the parent agent | +| `tracker_create_task`, `tracker_update_task`, `tracker_get_task`, `tracker_list_tasks`, `tracker_add_dependency`, `tracker_visualize` | Rich task tracker with dependency and visualization support | +| `read_mcp_resource`, `list_mcp_resources` | MCP resource access | diff --git a/skills/writing-skills/SKILL.md b/skills/writing-skills/SKILL.md index 6d3ded6e..8928d449 100644 --- a/skills/writing-skills/SKILL.md +++ b/skills/writing-skills/SKILL.md @@ -9,7 +9,7 @@ description: Use when creating new skills, editing existing skills, or verifying **Writing skills IS Test-Driven Development applied to process documentation.** -**Personal skills live in your runtime's skills directory** +**Personal skills live in your runtime's skills directory** — see [claude-code-tools.md](../using-superpowers/references/claude-code-tools.md), [codex-tools.md](../using-superpowers/references/codex-tools.md), [copilot-tools.md](../using-superpowers/references/copilot-tools.md), or [gemini-tools.md](../using-superpowers/references/gemini-tools.md) for the path on your runtime. Codex, Copilot CLI, and Gemini CLI all also recognize `~/.agents/skills/` as a cross-runtime alias. You write test cases (pressure scenarios with subagents), watch them fail (baseline behavior), write the skill (documentation), watch tests pass (agents comply), and refactor (close loopholes). From 03147d23992d0ae80b04fc24af67063607dfac3b Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 10:21:28 -0700 Subject: [PATCH 02/51] refactor(skills): fold Integration skill lists into points of use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The list-style Integration sections in subagent-driven-development and executing-plans duplicated references that already exist where the flow uses them (process digraph, When to Use, prompt templates, Step 3), so they added maintenance cost without carrying behavior. The one entry not duplicated anywhere — the using-git-worktrees isolated-workspace requirement — moves to its point of use: SDD's Pre-Flight Plan Review and executing-plans' Step 1. Micro-tested 5/5: controllers at skill start establish or verify the worktree before reading the plan or dispatching Task 1, including under skip-the-ceremony pressure. The prose Integration sections in requesting-code-review and other skills are unchanged — they carry placement content, not an index. --- skills/executing-plans/SKILL.md | 16 +++++----------- skills/subagent-driven-development/SKILL.md | 17 +++-------------- 2 files changed, 8 insertions(+), 25 deletions(-) diff --git a/skills/executing-plans/SKILL.md b/skills/executing-plans/SKILL.md index 78d88540..8c9f339d 100644 --- a/skills/executing-plans/SKILL.md +++ b/skills/executing-plans/SKILL.md @@ -16,10 +16,11 @@ Load plan, review critically, execute all tasks, report when complete. ## The Process ### Step 1: Load and Review Plan -1. Read plan file -2. Review critically - identify any questions or concerns about the plan -3. If concerns: Raise them with your human partner before starting -4. If no concerns: Create todos for the plan items and proceed +1. Ensure an isolated workspace: use superpowers:using-git-worktrees to create one or verify the existing one +2. Read plan file +3. Review critically - identify any questions or concerns about the plan +4. If concerns: Raise them with your human partner before starting +5. If no concerns: Create todos for the plan items and proceed ### Step 2: Execute Tasks @@ -61,10 +62,3 @@ After all tasks complete and verified: - Reference skills when plan says to - Stop when blocked, don't guess - Never start implementation on main/master branch without explicit user consent - -## Integration - -**Required workflow skills:** -- **superpowers:using-git-worktrees** - Ensures isolated workspace (creates one or verifies existing) -- **superpowers:writing-plans** - Creates the plan this skill executes -- **superpowers:finishing-a-development-branch** - Complete development after all tasks diff --git a/skills/subagent-driven-development/SKILL.md b/skills/subagent-driven-development/SKILL.md index d8ca0815..ebbb1002 100644 --- a/skills/subagent-driven-development/SKILL.md +++ b/skills/subagent-driven-development/SKILL.md @@ -84,6 +84,9 @@ digraph process { ## Pre-Flight Plan Review +Ensure the work happens in an isolated workspace: use +superpowers:using-git-worktrees to create one or verify the existing one. + Before dispatching Task 1, scan the plan once for conflicts: - tasks that contradict each other or the plan's Global Constraints @@ -402,17 +405,3 @@ Done! **If subagent fails task:** - Dispatch fix subagent with specific instructions - Don't try to fix manually (context pollution) - -## Integration - -**Required workflow skills:** -- **superpowers:using-git-worktrees** - Ensures isolated workspace (creates one or verifies existing) -- **superpowers:writing-plans** - Creates the plan this skill executes -- **superpowers:requesting-code-review** - Code review template for the final whole-branch review -- **superpowers:finishing-a-development-branch** - Complete development after all tasks - -**Subagents should use:** -- **superpowers:test-driven-development** - Subagents follow TDD for each task - -**Alternative workflow:** -- **superpowers:executing-plans** - Use for parallel session instead of same-session execution From af67e03f85baace34c5564c3bf934f45e3c2d36d Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 10:50:42 -0700 Subject: [PATCH 03/51] refactor(skills): fold systematic-debugging Related-skills block into Phase 4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same treatment as subagent-driven-development and executing-plans: the test-driven-development entry duplicated the reference already at Phase 4 Step 1, and the verification-before-completion entry was a sole carrier — it moves to its point of use in Phase 4 Step 3 (Verify Fix). Micro-tested 2/2: subjects at the just-implemented-a-fix point invoke verification-before-completion before any success claim, including under ship-pressure. --- skills/systematic-debugging/SKILL.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/skills/systematic-debugging/SKILL.md b/skills/systematic-debugging/SKILL.md index b0eca38b..1c0be572 100644 --- a/skills/systematic-debugging/SKILL.md +++ b/skills/systematic-debugging/SKILL.md @@ -188,6 +188,7 @@ You MUST complete each phase before proceeding to the next. - Test passes now? - No other tests broken? - Issue actually resolved? + - Use the `superpowers:verification-before-completion` skill before claiming success 4. **If Fix Doesn't Work** - STOP @@ -283,10 +284,6 @@ These techniques are part of systematic debugging and available in this director - **`defense-in-depth.md`** - Add validation at multiple layers after finding root cause - **`condition-based-waiting.md`** - Replace arbitrary timeouts with condition polling -**Related skills:** -- **superpowers:test-driven-development** - For creating failing test case (Phase 4, Step 1) -- **superpowers:verification-before-completion** - Verify fix worked before claiming success - ## Real-World Impact From debugging sessions: From 9dff1a901f120a9d30e59d1658d2a417d863d499 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 11:09:48 -0700 Subject: [PATCH 04/51] refactor(skills): stop offering to discard work in finishing-a-development-branch The completion menu dates from when throwing away branches was routine; offering 'Discard this work' beside 'Merge' on every completion advertised destroying finished, passing work. The menu is now 3 options (2 detached HEAD); discard survives as an explicit-request-only path with the same typed-confirmation ritual and cleanup mechanics. Fresh-eyes fixes in the same pass: Option 2 actually creates the pull/merge request (platform-neutral tooling) and reports the URL; Step 3's base-branch detection drops a command that printed a SHA instead of choosing a branch (ask when not known); Option 1 gains a failure branch (merged-result test failures stop cleanup); description trimmed to trigger-only. Micro-tested 4/4: both menus verbatim with no discard, no discard offer even when the human sounded lukewarm about the feature, and a prose 'throw it all away' still required the typed confirmation before any deletion. --- .../finishing-a-development-branch/SKILL.md | 60 ++++++++++++------- 1 file changed, 38 insertions(+), 22 deletions(-) diff --git a/skills/finishing-a-development-branch/SKILL.md b/skills/finishing-a-development-branch/SKILL.md index 7f5337aa..368829fd 100644 --- a/skills/finishing-a-development-branch/SKILL.md +++ b/skills/finishing-a-development-branch/SKILL.md @@ -1,6 +1,6 @@ --- name: finishing-a-development-branch -description: Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup +description: Use when implementation is complete, all tests pass, and you need to decide how to integrate the work --- # Finishing a Development Branch @@ -50,22 +50,20 @@ This determines which menu to show and how cleanup works: | State | Menu | Cleanup | |-------|------|---------| -| `GIT_DIR == GIT_COMMON` (normal repo) | Standard 4 options | No worktree to clean up | -| `GIT_DIR != GIT_COMMON`, named branch | Standard 4 options | Provenance-based (see Step 6) | +| `GIT_DIR == GIT_COMMON` (normal repo) | Standard 3 options | No worktree to clean up | +| `GIT_DIR != GIT_COMMON`, named branch | Standard 3 options | Provenance-based (see Step 6) | | `GIT_DIR != GIT_COMMON`, detached HEAD | Reduced 3 options (no merge) | No cleanup (externally managed) | ### Step 3: Determine Base Branch -```bash -# Try common base branches -git merge-base HEAD main 2>/dev/null || git merge-base HEAD master 2>/dev/null -``` - -Or ask: "This branch split from main - is that correct?" +The base branch is whatever this work forked from — usually named in the +plan, the conversation, or the branch's upstream. If it is not already +known, ask: "This branch split from - is that correct?" +Don't guess silently: merging into the wrong base is expensive to undo. ### Step 4: Present Options -**Normal repo and named-branch worktree — present exactly these 4 options:** +**Normal repo and named-branch worktree — present exactly these 3 options:** ``` Implementation complete. What would you like to do? @@ -73,25 +71,27 @@ Implementation complete. What would you like to do? 1. Merge back to locally 2. Push and create a Pull Request 3. Keep the branch as-is (I'll handle it later) -4. Discard this work Which option? ``` -**Detached HEAD — present exactly these 3 options:** +**Detached HEAD — present exactly these 2 options:** ``` Implementation complete. You're on a detached HEAD (externally managed workspace). 1. Push as new branch and create a Pull Request 2. Keep as-is (I'll handle it later) -3. Discard this work Which option? ``` **Don't add explanation** - keep options concise. +Discarding the work is never offered. It exists only as a response to your +human partner explicitly asking for it (see "If your human partner asks to +discard the work" below). + ### Step 5: Execute Choice #### Option 1: Merge Locally @@ -112,6 +112,10 @@ git merge # Only after merge succeeds: cleanup worktree (Step 6), then delete branch ``` +If tests fail on the merged result: STOP. Leave the worktree and branch in +place and investigate — nothing has been pushed, so the merge is local and +recoverable. + Then: Cleanup worktree (Step 6), then delete branch: ```bash @@ -125,6 +129,11 @@ git branch -d git push -u origin ``` +Then create the pull/merge request against with the host's +tooling (`gh pr create`, `glab mr create`, or the URL git prints on push), +following the repo's PR template and conventions if present, and report +the URL to your human partner. + **Do NOT clean up worktree** — user needs it alive to iterate on PR feedback. #### Option 3: Keep As-Is @@ -133,9 +142,11 @@ Report: "Keeping branch . Worktree preserved at ." **Don't cleanup worktree.** -#### Option 4: Discard +#### If your human partner asks to discard the work + +Never offer this. Only do it when your human partner explicitly asks to +throw the work away — and even then, confirm first: -**Confirm first:** ``` This will permanently delete: - Branch @@ -160,7 +171,7 @@ git branch -D ### Step 6: Cleanup Workspace -**Only runs for Options 1 and 4.** Options 2 and 3 always preserve the worktree. +**Only runs for Option 1 and confirmed discards.** Options 2 and 3 always preserve the worktree. ```bash GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P) @@ -188,7 +199,7 @@ git worktree prune # Self-healing: clean up any stale registrations | 1. Merge locally | yes | - | - | yes | | 2. Create PR | - | yes | yes | - | | 3. Keep as-is | - | - | yes | - | -| 4. Discard | - | - | - | yes (force) | +| Discard (explicit request only) | - | - | - | yes (force) | ## Common Mistakes @@ -198,11 +209,15 @@ git worktree prune # Self-healing: clean up any stale registrations **Open-ended questions** - **Problem:** "What should I do next?" is ambiguous -- **Fix:** Present exactly 4 structured options (or 3 for detached HEAD) +- **Fix:** Present exactly 3 structured options (or 2 for detached HEAD) + +**Offering to discard the work** +- **Problem:** Puts throwing away completed, passing work on the menu +- **Fix:** Discard only on your human partner's explicit request, never as an offer **Cleaning up worktree for Option 2** - **Problem:** Remove worktree user needs for PR iteration -- **Fix:** Only cleanup for Options 1 and 4 +- **Fix:** Only cleanup for Option 1 and confirmed discards **Deleting branch before removing worktree** - **Problem:** `git branch -d` fails because worktree still references the branch @@ -225,6 +240,7 @@ git worktree prune # Self-healing: clean up any stale registrations **Never:** - Proceed with failing tests - Merge without verifying tests on result +- Offer discarding the work — it happens only on explicit request - Delete work without confirmation - Force-push without explicit request - Remove a worktree before confirming merge success @@ -234,8 +250,8 @@ git worktree prune # Self-healing: clean up any stale registrations **Always:** - Verify tests before offering options - Detect environment before presenting menu -- Present exactly 4 options (or 3 for detached HEAD) -- Get typed confirmation for Option 4 -- Clean up worktree for Options 1 & 4 only +- Present exactly 3 options (or 2 for detached HEAD) +- Get typed confirmation before any discard +- Clean up worktree for Option 1 and confirmed discards only - `cd` to main repo root before worktree removal - Run `git worktree prune` after removal From bcfe79869a4aeba201cf9300f2a8426647e11787 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 11:25:59 -0700 Subject: [PATCH 05/51] refactor(skills): make PR creation forge-agnostic in finishing-a-development-branch Naming gh and glab implicitly blessed two forges; Gitea, Forgejo, Bitbucket and others are equally valid. Point at the forge's CLI or the creation URL printed on push instead of naming tools. --- skills/finishing-a-development-branch/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/finishing-a-development-branch/SKILL.md b/skills/finishing-a-development-branch/SKILL.md index 368829fd..e25420bc 100644 --- a/skills/finishing-a-development-branch/SKILL.md +++ b/skills/finishing-a-development-branch/SKILL.md @@ -129,10 +129,10 @@ git branch -d git push -u origin ``` -Then create the pull/merge request against with the host's -tooling (`gh pr create`, `glab mr create`, or the URL git prints on push), -following the repo's PR template and conventions if present, and report -the URL to your human partner. +Then create the pull/merge request against with the forge's +tooling — its CLI if one is available, or the creation URL most forges +print when you push — following the repo's PR template and conventions if +present, and report the URL to your human partner. **Do NOT clean up worktree** — user needs it alive to iterate on PR feedback. From fbb6dba450a74154baae931293ecffc8d836c1a3 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 11:43:34 -0700 Subject: [PATCH 06/51] refactor(skills): compress finishing-a-development-branch, adopt rationalization table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Red Flags and Common Mistakes fold into one Common Rationalizations table (house Excuse/Reality form); every prior entry maps to a table row or an inline sentence in the step it guards. Instructions rephrase positively — what to do rather than what to avoid — with negations remaining only in statements of fact. Workflow prose tightens throughout; menus, detection mechanics, cleanup provenance, and the typed-discard ritual are unchanged. Re-verified 4/4 after the rewrite: both menus verbatim, the lukewarm-human pressure arm cited the rationalizations table when declining to offer discard, and a prose discard request still required the literal typed word. --- .../finishing-a-development-branch/SKILL.md | 153 ++++++------------ 1 file changed, 49 insertions(+), 104 deletions(-) diff --git a/skills/finishing-a-development-branch/SKILL.md b/skills/finishing-a-development-branch/SKILL.md index e25420bc..7f6e4ec7 100644 --- a/skills/finishing-a-development-branch/SKILL.md +++ b/skills/finishing-a-development-branch/SKILL.md @@ -7,39 +7,25 @@ description: Use when implementation is complete, all tests pass, and you need t ## Overview -Guide completion of development work by presenting clear options and handling chosen workflow. - **Core principle:** Verify tests → Detect environment → Present options → Execute choice → Clean up. **Announce at start:** "I'm using the finishing-a-development-branch skill to complete this work." -## The Process +## Step 1: Verify Tests -### Step 1: Verify Tests +Run the project's full test suite (`npm test` / `cargo test` / `pytest` / `go test ./...`). -**Before presenting options, verify tests pass:** +**If tests fail**, report the failures and stop — the menu comes after a green suite: -```bash -# Run project's test suite -npm test / cargo test / pytest / go test ./... -``` - -**If tests fail:** ``` Tests failing ( failures). Must fix before completing: [Show failures] - -Cannot proceed with merge/PR until tests pass. ``` -Stop. Don't proceed to Step 2. +**If tests pass:** continue to Step 2. -**If tests pass:** Continue to Step 2. - -### Step 2: Detect Environment - -**Determine workspace state before presenting options:** +## Step 2: Detect Environment ```bash GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P) @@ -52,16 +38,16 @@ This determines which menu to show and how cleanup works: |-------|------|---------| | `GIT_DIR == GIT_COMMON` (normal repo) | Standard 3 options | No worktree to clean up | | `GIT_DIR != GIT_COMMON`, named branch | Standard 3 options | Provenance-based (see Step 6) | -| `GIT_DIR != GIT_COMMON`, detached HEAD | Reduced 3 options (no merge) | No cleanup (externally managed) | +| `GIT_DIR != GIT_COMMON`, detached HEAD | Reduced 2 options (no merge) | Externally managed — leave in place | -### Step 3: Determine Base Branch +## Step 3: Determine Base Branch The base branch is whatever this work forked from — usually named in the plan, the conversation, or the branch's upstream. If it is not already known, ask: "This branch split from - is that correct?" -Don't guess silently: merging into the wrong base is expensive to undo. +Confirm before merging: merging into the wrong base is expensive to undo. -### Step 4: Present Options +## Step 4: Present Options **Normal repo and named-branch worktree — present exactly these 3 options:** @@ -86,15 +72,15 @@ Implementation complete. You're on a detached HEAD (externally managed workspace Which option? ``` -**Don't add explanation** - keep options concise. - -Discarding the work is never offered. It exists only as a response to your +Present the menu exactly as written — concise, with every option coming +from the list above. Discarding the work happens only in response to your human partner explicitly asking for it (see "If your human partner asks to -discard the work" below). +discard the work" below). Wait for their answer; the integration decision +is theirs. -### Step 5: Execute Choice +## Step 5: Execute Choice -#### Option 1: Merge Locally +### Option 1: Merge Locally ```bash # Get main repo root for CWD safety @@ -108,24 +94,22 @@ git merge # Verify tests on merged result - -# Only after merge succeeds: cleanup worktree (Step 6), then delete branch ``` -If tests fail on the merged result: STOP. Leave the worktree and branch in -place and investigate — nothing has been pushed, so the merge is local and -recoverable. +If tests fail on the merged result: stop, leave the worktree and branch in +place, and investigate — nothing has been pushed, so the merge is local +and recoverable. -Then: Cleanup worktree (Step 6), then delete branch: +Once the merged result is green: clean up the worktree (Step 6), then +delete the branch: ```bash git branch -d ``` -#### Option 2: Push and Create PR +### Option 2: Push and Create PR ```bash -# Push branch git push -u origin ``` @@ -134,18 +118,16 @@ tooling — its CLI if one is available, or the creation URL most forges print when you push — following the repo's PR template and conventions if present, and report the URL to your human partner. -**Do NOT clean up worktree** — user needs it alive to iterate on PR feedback. +Keep the worktree — your human partner iterates on PR feedback there. -#### Option 3: Keep As-Is +### Option 3: Keep As-Is Report: "Keeping branch . Worktree preserved at ." -**Don't cleanup worktree.** +### If your human partner asks to discard the work -#### If your human partner asks to discard the work - -Never offer this. Only do it when your human partner explicitly asks to -throw the work away — and even then, confirm first: +This path exists only as a response to an explicit request to throw the +work away. Confirm first: ``` This will permanently delete: @@ -156,22 +138,23 @@ This will permanently delete: Type 'discard' to confirm. ``` -Wait for exact confirmation. +Wait for that exact confirmation. When it arrives: -If confirmed: ```bash MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel) cd "$MAIN_ROOT" ``` -Then: Cleanup worktree (Step 6), then force-delete branch: +Then clean up the worktree (Step 6) and force-delete the branch: + ```bash git branch -D ``` -### Step 6: Cleanup Workspace +## Step 6: Cleanup Workspace -**Only runs for Option 1 and confirmed discards.** Options 2 and 3 always preserve the worktree. +**Runs for Option 1 and confirmed discards.** Options 2 and 3 always +preserve the worktree. ```bash GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P) @@ -181,7 +164,9 @@ WORKTREE_PATH=$(git rev-parse --show-toplevel) **If `GIT_DIR == GIT_COMMON`:** Normal repo, no worktree to clean up. Done. -**If worktree path is under `.worktrees/` or `worktrees/`:** Superpowers created this worktree — we own cleanup. +**If worktree path is under `.worktrees/` or `worktrees/`:** Superpowers +created this worktree — we own cleanup. Run it from the main repo root +(removal fails from inside the worktree being removed): ```bash MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel) @@ -190,7 +175,8 @@ git worktree remove "$WORKTREE_PATH" git worktree prune # Self-healing: clean up any stale registrations ``` -**Otherwise:** The host environment (harness) owns this workspace. Do NOT remove it. If your platform provides a workspace-exit tool, use it. Otherwise, leave the workspace in place. +**Otherwise:** The host environment owns this workspace — leave it in +place. If your platform provides a workspace-exit tool, use it. ## Quick Reference @@ -201,57 +187,16 @@ git worktree prune # Self-healing: clean up any stale registrations | 3. Keep as-is | - | - | yes | - | | Discard (explicit request only) | - | - | - | yes (force) | -## Common Mistakes +## Common Rationalizations -**Skipping test verification** -- **Problem:** Merge broken code, create failing PR -- **Fix:** Always verify tests before offering options - -**Open-ended questions** -- **Problem:** "What should I do next?" is ambiguous -- **Fix:** Present exactly 3 structured options (or 2 for detached HEAD) - -**Offering to discard the work** -- **Problem:** Puts throwing away completed, passing work on the menu -- **Fix:** Discard only on your human partner's explicit request, never as an offer - -**Cleaning up worktree for Option 2** -- **Problem:** Remove worktree user needs for PR iteration -- **Fix:** Only cleanup for Option 1 and confirmed discards - -**Deleting branch before removing worktree** -- **Problem:** `git branch -d` fails because worktree still references the branch -- **Fix:** Merge first, remove worktree, then delete branch - -**Running git worktree remove from inside the worktree** -- **Problem:** Command fails silently when CWD is inside the worktree being removed -- **Fix:** Always `cd` to main repo root before `git worktree remove` - -**Cleaning up harness-owned worktrees** -- **Problem:** Removing a worktree the harness created causes phantom state -- **Fix:** Only clean up worktrees under `.worktrees/` or `worktrees/` - -**No confirmation for discard** -- **Problem:** Accidentally delete work -- **Fix:** Require typed "discard" confirmation - -## Red Flags - -**Never:** -- Proceed with failing tests -- Merge without verifying tests on result -- Offer discarding the work — it happens only on explicit request -- Delete work without confirmation -- Force-push without explicit request -- Remove a worktree before confirming merge success -- Clean up worktrees you didn't create (provenance check) -- Run `git worktree remove` from inside the worktree - -**Always:** -- Verify tests before offering options -- Detect environment before presenting menu -- Present exactly 3 options (or 2 for detached HEAD) -- Get typed confirmation before any discard -- Clean up worktree for Option 1 and confirmed discards only -- `cd` to main repo root before worktree removal -- Run `git worktree prune` after removal +| Excuse | Reality | +|--------|---------| +| "Tests passed earlier this session" | Run the suite now. The tree changed since the last green run. | +| "They obviously want it merged" | Integration is your human partner's decision. Present the menu and wait. | +| "They seem done with this feature — I'll offer to discard it" | The menu is complete as written. Discard happens only when your human partner asks for it in so many words. | +| "'Yeah, get rid of it' counts as confirmation" | Only the typed word `discard` authorizes deletion. | +| "The PR is up, so the worktree is clutter now" | PR feedback gets fixed in that worktree. It stays until the work lands. | +| "This other worktree looks stale — I'll clean it too" | Clean up only worktrees under `.worktrees/` or `worktrees/`. Everything else belongs to the host. | +| "The merged-result failure is probably flaky" | A failing merged result stops everything. Branch and worktree stay put while you investigate. | +| "The base branch is obviously main" | Confirm the fork point or ask. Merging into the wrong base is expensive to undo. | +| "The push was rejected — force-push will fix it" | A rejected push means the remote moved. Investigate; force-push only on your human partner's explicit request. | From 0b47219ac864352eed51237b81633986f88f3aa6 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:00:35 -0700 Subject: [PATCH 07/51] fix(skills): capture worktree path before Step 5 changes directory Step 6 recomputed WORKTREE_PATH after Option 1 and discard had already cd'd to the main repo root, so --show-toplevel returned the main root: the provenance check could never match, cleanup silently no-oped, and the branch delete failed with the worktree still attached. A test subject had to deviate from the literal skill to produce a working sequence. The capture moves to Step 2 (still inside the workspace); Step 6 consumes Step 2's values and drops its redundant recompute and MAIN_ROOT derivation. Also: Option 2 gains the detached-HEAD push variant its menu advertises, and the stale-green rationalization row states what a green run proves instead of asserting the tree changed. Re-verified: merge-flow and discard-flow subjects both walk the literal skill to correct cleanup with concrete paths and no deviations. --- .../finishing-a-development-branch/SKILL.md | 25 +++++++++---------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/skills/finishing-a-development-branch/SKILL.md b/skills/finishing-a-development-branch/SKILL.md index 7f6e4ec7..21a439ac 100644 --- a/skills/finishing-a-development-branch/SKILL.md +++ b/skills/finishing-a-development-branch/SKILL.md @@ -30,6 +30,9 @@ Tests failing ( failures). Must fix before completing: ```bash GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P) GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P) +# Capture now, while still inside the workspace — Step 5 changes directory +# before cleanup (Step 6) needs this value +WORKTREE_PATH=$(git rev-parse --show-toplevel) ``` This determines which menu to show and how cleanup works: @@ -111,6 +114,8 @@ git branch -d ```bash git push -u origin +# From a detached HEAD, name the new branch on the remote: +# git push origin HEAD:refs/heads/ ``` Then create the pull/merge request against with the forge's @@ -154,23 +159,17 @@ git branch -D ## Step 6: Cleanup Workspace **Runs for Option 1 and confirmed discards.** Options 2 and 3 always -preserve the worktree. - -```bash -GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P) -GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P) -WORKTREE_PATH=$(git rev-parse --show-toplevel) -``` +preserve the worktree. Both callers have already changed directory to the +main repo root — worktree removal must run from outside the worktree — +and use the `GIT_DIR`/`GIT_COMMON`/`WORKTREE_PATH` values captured in +Step 2, from before that directory change. **If `GIT_DIR == GIT_COMMON`:** Normal repo, no worktree to clean up. Done. -**If worktree path is under `.worktrees/` or `worktrees/`:** Superpowers -created this worktree — we own cleanup. Run it from the main repo root -(removal fails from inside the worktree being removed): +**If `WORKTREE_PATH` is under `.worktrees/` or `worktrees/`:** Superpowers +created this worktree — we own cleanup: ```bash -MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel) -cd "$MAIN_ROOT" git worktree remove "$WORKTREE_PATH" git worktree prune # Self-healing: clean up any stale registrations ``` @@ -191,7 +190,7 @@ place. If your platform provides a workspace-exit tool, use it. | Excuse | Reality | |--------|---------| -| "Tests passed earlier this session" | Run the suite now. The tree changed since the last green run. | +| "Tests passed earlier this session" | Run the suite on the tree you are about to integrate. A green run only proves the tree it ran on. | | "They obviously want it merged" | Integration is your human partner's decision. Present the menu and wait. | | "They seem done with this feature — I'll offer to discard it" | The menu is complete as written. Discard happens only when your human partner asks for it in so many words. | | "'Yeah, get rid of it' counts as confirmation" | Only the typed word `discard` authorizes deletion. | From e74961c110488b52169f0854eea47c8fcee15e23 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:49:52 -0700 Subject: [PATCH 08/51] refactor(skills): reframe testing-anti-patterns as writing-good-tests The disclosure doc becomes a catalog of what to do: six positively named rules (assert on real behavior, cleanup in test utilities, mock at the right level, mirror real data, tests ship with implementation, prefer real components), each leading with the GOOD example and keeping the violation as contrast. Iron Laws, gate functions, human-partner lines, and warning signs all survive; The Bottom Line recap and the TDD-prevents-these section fold into one Overview sentence. SKILL.md's pointer moves into the Good Tests section it belongs with. Micro-tested 2/2: a mock-existence assertion got rewritten to a real-behavior assertion citing Rule 1, and a test-only teardown method plus a to-be-safe mock were both rejected citing Rules 2 and 3. --- skills/test-driven-development/SKILL.md | 12 +- .../testing-anti-patterns.md | 299 ------------------ .../writing-good-tests.md | 248 +++++++++++++++ 3 files changed, 253 insertions(+), 306 deletions(-) delete mode 100644 skills/test-driven-development/testing-anti-patterns.md create mode 100644 skills/test-driven-development/writing-good-tests.md diff --git a/skills/test-driven-development/SKILL.md b/skills/test-driven-development/SKILL.md index 60d2609c..158cb0b5 100644 --- a/skills/test-driven-development/SKILL.md +++ b/skills/test-driven-development/SKILL.md @@ -203,6 +203,11 @@ Next failing test for next feature. | **Clear** | Name describes behavior | `test('test1')` | | **Shows intent** | Demonstrates desired API | Obscures what code should do | +When adding mocks or test utilities, read [writing-good-tests.md](writing-good-tests.md) for the rules that keep tests honest: +- Assert on real behavior, never on mock behavior +- Keep test-only code in test utilities, out of production classes +- Understand a dependency's side effects before mocking it + ## Why Order Matters **"I'll write tests after to verify it works"** @@ -354,13 +359,6 @@ Bug found? Write failing test reproducing it. Follow TDD cycle. Test proves fix Never fix bugs without a test. -## Testing Anti-Patterns - -When adding mocks or test utilities, read [testing-anti-patterns.md](testing-anti-patterns.md) to avoid common pitfalls: -- Testing mock behavior instead of real behavior -- Adding test-only methods to production classes -- Mocking without understanding dependencies - ## Final Rule ``` diff --git a/skills/test-driven-development/testing-anti-patterns.md b/skills/test-driven-development/testing-anti-patterns.md deleted file mode 100644 index e77ab6b6..00000000 --- a/skills/test-driven-development/testing-anti-patterns.md +++ /dev/null @@ -1,299 +0,0 @@ -# Testing Anti-Patterns - -**Load this reference when:** writing or changing tests, adding mocks, or tempted to add test-only methods to production code. - -## Overview - -Tests must verify real behavior, not mock behavior. Mocks are a means to isolate, not the thing being tested. - -**Core principle:** Test what the code does, not what the mocks do. - -**Following strict TDD prevents these anti-patterns.** - -## The Iron Laws - -``` -1. NEVER test mock behavior -2. NEVER add test-only methods to production classes -3. NEVER mock without understanding dependencies -``` - -## Anti-Pattern 1: Testing Mock Behavior - -**The violation:** -```typescript -// ❌ BAD: Testing that the mock exists -test('renders sidebar', () => { - render(); - expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument(); -}); -``` - -**Why this is wrong:** -- You're verifying the mock works, not that the component works -- Test passes when mock is present, fails when it's not -- Tells you nothing about real behavior - -**your human partner's correction:** "Are we testing the behavior of a mock?" - -**The fix:** -```typescript -// ✅ GOOD: Test real component or don't mock it -test('renders sidebar', () => { - render(); // Don't mock sidebar - expect(screen.getByRole('navigation')).toBeInTheDocument(); -}); - -// OR if sidebar must be mocked for isolation: -// Don't assert on the mock - test Page's behavior with sidebar present -``` - -### Gate Function - -``` -BEFORE asserting on any mock element: - Ask: "Am I testing real component behavior or just mock existence?" - - IF testing mock existence: - STOP - Delete the assertion or unmock the component - - Test real behavior instead -``` - -## Anti-Pattern 2: Test-Only Methods in Production - -**The violation:** -```typescript -// ❌ BAD: destroy() only used in tests -class Session { - async destroy() { // Looks like production API! - await this._workspaceManager?.destroyWorkspace(this.id); - // ... cleanup - } -} - -// In tests -afterEach(() => session.destroy()); -``` - -**Why this is wrong:** -- Production class polluted with test-only code -- Dangerous if accidentally called in production -- Violates YAGNI and separation of concerns -- Confuses object lifecycle with entity lifecycle - -**The fix:** -```typescript -// ✅ GOOD: Test utilities handle test cleanup -// Session has no destroy() - it's stateless in production - -// In test-utils/ -export async function cleanupSession(session: Session) { - const workspace = session.getWorkspaceInfo(); - if (workspace) { - await workspaceManager.destroyWorkspace(workspace.id); - } -} - -// In tests -afterEach(() => cleanupSession(session)); -``` - -### Gate Function - -``` -BEFORE adding any method to production class: - Ask: "Is this only used by tests?" - - IF yes: - STOP - Don't add it - Put it in test utilities instead - - Ask: "Does this class own this resource's lifecycle?" - - IF no: - STOP - Wrong class for this method -``` - -## Anti-Pattern 3: Mocking Without Understanding - -**The violation:** -```typescript -// ❌ BAD: Mock breaks test logic -test('detects duplicate server', () => { - // Mock prevents config write that test depends on! - vi.mock('ToolCatalog', () => ({ - discoverAndCacheTools: vi.fn().mockResolvedValue(undefined) - })); - - await addServer(config); - await addServer(config); // Should throw - but won't! -}); -``` - -**Why this is wrong:** -- Mocked method had side effect test depended on (writing config) -- Over-mocking to "be safe" breaks actual behavior -- Test passes for wrong reason or fails mysteriously - -**The fix:** -```typescript -// ✅ GOOD: Mock at correct level -test('detects duplicate server', () => { - // Mock the slow part, preserve behavior test needs - vi.mock('MCPServerManager'); // Just mock slow server startup - - await addServer(config); // Config written - await addServer(config); // Duplicate detected ✓ -}); -``` - -### Gate Function - -``` -BEFORE mocking any method: - STOP - Don't mock yet - - 1. Ask: "What side effects does the real method have?" - 2. Ask: "Does this test depend on any of those side effects?" - 3. Ask: "Do I fully understand what this test needs?" - - IF depends on side effects: - Mock at lower level (the actual slow/external operation) - OR use test doubles that preserve necessary behavior - NOT the high-level method the test depends on - - IF unsure what test depends on: - Run test with real implementation FIRST - Observe what actually needs to happen - THEN add minimal mocking at the right level - - Red flags: - - "I'll mock this to be safe" - - "This might be slow, better mock it" - - Mocking without understanding the dependency chain -``` - -## Anti-Pattern 4: Incomplete Mocks - -**The violation:** -```typescript -// ❌ BAD: Partial mock - only fields you think you need -const mockResponse = { - status: 'success', - data: { userId: '123', name: 'Alice' } - // Missing: metadata that downstream code uses -}; - -// Later: breaks when code accesses response.metadata.requestId -``` - -**Why this is wrong:** -- **Partial mocks hide structural assumptions** - You only mocked fields you know about -- **Downstream code may depend on fields you didn't include** - Silent failures -- **Tests pass but integration fails** - Mock incomplete, real API complete -- **False confidence** - Test proves nothing about real behavior - -**The Iron Rule:** Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses. - -**The fix:** -```typescript -// ✅ GOOD: Mirror real API completeness -const mockResponse = { - status: 'success', - data: { userId: '123', name: 'Alice' }, - metadata: { requestId: 'req-789', timestamp: 1234567890 } - // All fields real API returns -}; -``` - -### Gate Function - -``` -BEFORE creating mock responses: - Check: "What fields does the real API response contain?" - - Actions: - 1. Examine actual API response from docs/examples - 2. Include ALL fields system might consume downstream - 3. Verify mock matches real response schema completely - - Critical: - If you're creating a mock, you must understand the ENTIRE structure - Partial mocks fail silently when code depends on omitted fields - - If uncertain: Include all documented fields -``` - -## Anti-Pattern 5: Integration Tests as Afterthought - -**The violation:** -``` -✅ Implementation complete -❌ No tests written -"Ready for testing" -``` - -**Why this is wrong:** -- Testing is part of implementation, not optional follow-up -- TDD would have caught this -- Can't claim complete without tests - -**The fix:** -``` -TDD cycle: -1. Write failing test -2. Implement to pass -3. Refactor -4. THEN claim complete -``` - -## When Mocks Become Too Complex - -**Warning signs:** -- Mock setup longer than test logic -- Mocking everything to make test pass -- Mocks missing methods real components have -- Test breaks when mock changes - -**your human partner's question:** "Do we need to be using a mock here?" - -**Consider:** Integration tests with real components often simpler than complex mocks - -## TDD Prevents These Anti-Patterns - -**Why TDD helps:** -1. **Write test first** → Forces you to think about what you're actually testing -2. **Watch it fail** → Confirms test tests real behavior, not mocks -3. **Minimal implementation** → No test-only methods creep in -4. **Real dependencies** → You see what the test actually needs before mocking - -**If you're testing mock behavior, you violated TDD** - you added mocks without watching test fail against real code first. - -## Quick Reference - -| Anti-Pattern | Fix | -|--------------|-----| -| Assert on mock elements | Test real component or unmock it | -| Test-only methods in production | Move to test utilities | -| Mock without understanding | Understand dependencies first, mock minimally | -| Incomplete mocks | Mirror real API completely | -| Tests as afterthought | TDD - tests first | -| Over-complex mocks | Consider integration tests | - -## Red Flags - -- Assertion checks for `*-mock` test IDs -- Methods only called in test files -- Mock setup is >50% of test -- Test fails when you remove mock -- Can't explain why mock is needed -- Mocking "just to be safe" - -## The Bottom Line - -**Mocks are tools to isolate, not things to test.** - -If TDD reveals you're testing mock behavior, you've gone wrong. - -Fix: Test real behavior or question why you're mocking at all. diff --git a/skills/test-driven-development/writing-good-tests.md b/skills/test-driven-development/writing-good-tests.md new file mode 100644 index 00000000..ad8c6023 --- /dev/null +++ b/skills/test-driven-development/writing-good-tests.md @@ -0,0 +1,248 @@ +# Writing Good Tests + +**Load this reference when:** writing or changing tests, adding mocks, or +adding cleanup/helper methods for tests. + +## Overview + +Good tests verify real behavior. Mocks exist to isolate the code under +test — they are never the thing being tested. + +**Core principle:** Test what the code does, not what the mocks do. + +Strict TDD produces every rule below naturally: a test written first and +watched failing against real code only earns a mock when the real +dependency proves slow or external. A test asserting on a mock means TDD +was skipped somewhere. + +## The Iron Laws + +``` +1. Assert on real behavior, never on mock behavior +2. Production classes carry production methods only +3. Understand a dependency's side effects before mocking it +``` + +## Rule 1: Assert on Real Behavior + +```typescript +// ✅ GOOD: Test the real component +test('renders sidebar', () => { + render(); // Sidebar unmocked + expect(screen.getByRole('navigation')).toBeInTheDocument(); +}); +``` + +If the sidebar must be mocked for isolation, assert on Page's behavior +with the sidebar present — the mock itself earns no assertions. + +```typescript +// ❌ The violation: asserting that the mock exists +test('renders sidebar', () => { + render(); + expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument(); +}); +``` + +A mock assertion passes when the mock is present and fails when it is +absent — it says nothing about the component. **your human partner's +correction:** "Are we testing the behavior of a mock?" + +### Gate Function + +``` +BEFORE asserting on any mock element: + Ask: "Am I testing real component behavior or just mock existence?" + + IF testing mock existence: + STOP - Delete the assertion or unmock the component + + Test real behavior instead +``` + +## Rule 2: Keep Test Cleanup in Test Utilities + +```typescript +// ✅ GOOD: Test utilities own test cleanup +// Session has no destroy() - it's stateless in production + +// In test-utils/ +export async function cleanupSession(session: Session) { + const workspace = session.getWorkspaceInfo(); + if (workspace) { + await workspaceManager.destroyWorkspace(workspace.id); + } +} + +// In tests +afterEach(() => cleanupSession(session)); +``` + +```typescript +// ❌ The violation: destroy() exists only for tests +class Session { + async destroy() { // Looks like production API! + await this._workspaceManager?.destroyWorkspace(this.id); + // ... cleanup + } +} + +// In tests +afterEach(() => session.destroy()); +``` + +A test-only method pollutes the production class, is dangerous if +production code ever calls it, and confuses object lifecycle with entity +lifecycle. + +### Gate Function + +``` +BEFORE adding any method to a production class: + Ask: "Is this only used by tests?" + + IF yes: + STOP - Put it in test utilities instead + + Ask: "Does this class own this resource's lifecycle?" + + IF no: + STOP - Wrong class for this method +``` + +## Rule 3: Mock at the Right Level + +Learn what the real method does — every side effect — before replacing +it. Mock the slow or external operation and preserve the behavior your +test depends on. + +```typescript +// ✅ GOOD: Mock the slow part, preserve behavior the test needs +test('detects duplicate server', () => { + vi.mock('MCPServerManager'); // Just mock slow server startup + + await addServer(config); // Config written + await addServer(config); // Duplicate detected ✓ +}); +``` + +```typescript +// ❌ The violation: the mock swallows the side effect the test depends on +test('detects duplicate server', () => { + // Mock prevents the config write that duplicate detection reads! + vi.mock('ToolCatalog', () => ({ + discoverAndCacheTools: vi.fn().mockResolvedValue(undefined) + })); + + await addServer(config); + await addServer(config); // Should throw - but won't! +}); +``` + +### Gate Function + +``` +BEFORE mocking any method: + STOP - Understand before replacing + + 1. Ask: "What side effects does the real method have?" + 2. Ask: "Does this test depend on any of those side effects?" + 3. Ask: "Do I fully understand what this test needs?" + + IF the test depends on side effects: + Mock at the lower level (the actual slow/external operation) + OR use test doubles that preserve the necessary behavior + — keep the high-level method the test depends on real + + IF unsure what the test depends on: + Run the test with the real implementation FIRST + Observe what actually needs to happen + THEN add minimal mocking at the right level + + Warning signs: + - "I'll mock this to be safe" + - "This might be slow, better mock it" + - Mocking before tracing the dependency chain +``` + +## Rule 4: Mirror Real Data Completely + +Mock the COMPLETE data structure as it exists in reality, not just the +fields your immediate test uses. + +```typescript +// ✅ GOOD: Mirror real API completeness +const mockResponse = { + status: 'success', + data: { userId: '123', name: 'Alice' }, + metadata: { requestId: 'req-789', timestamp: 1234567890 } + // All fields real API returns +}; +``` + +```typescript +// ❌ The violation: only the fields you thought you needed +const mockResponse = { + status: 'success', + data: { userId: '123', name: 'Alice' } + // Missing: metadata that downstream code uses +}; + +// Later: breaks when code accesses response.metadata.requestId +``` + +Partial mocks hide structural assumptions and fail silently when +downstream code reads an omitted field: the test passes while integration +breaks. + +### Gate Function + +``` +BEFORE creating mock responses: + Check: "What fields does the real API response contain?" + + Actions: + 1. Examine the actual API response from docs/examples + 2. Include ALL fields the system might consume downstream + 3. Verify the mock matches the real response schema completely + + If uncertain: include all documented fields +``` + +## Rule 5: Tests Ship With the Implementation + +Testing is part of implementation. The TDD cycle — failing test, minimal +implementation, refactor — is what "complete" means; "implementation +complete, ready for testing" describes an unfinished task. + +## Rule 6: Prefer Real Components Over Complex Mocks + +Integration tests with real components are often simpler than elaborate +mocks. Reach for one when you see: + +- Mock setup longer than the test logic +- Mocking everything to make the test pass +- Mocks missing methods the real components have +- Tests breaking when the mock changes + +**your human partner's question:** "Do we need to be using a mock here?" + +## Quick Reference + +| When you... | Do | +|-------------|-----| +| Want to assert on a mocked element | Test the real component, or unmock it | +| Need cleanup that only tests use | Put it in test utilities | +| Are about to mock a method | Learn its side effects first; mock the slow/external level | +| Build a mock response | Mirror the real structure completely | +| Finish an implementation | Tests already exist (TDD) — or it is unfinished | +| Watch mock setup balloon | Switch to an integration test with real components | + +## Warning Signs + +- An assertion checks for a `*-mock` test ID +- A method is called only from test files +- Mock setup is more than half the test +- The test fails when you remove the mock +- You can't explain why the mock is needed +- Mocking "just to be safe" From 50025d16acd7458e0232850f53a55a2f41e9b626 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:51:57 -0700 Subject: [PATCH 09/51] fix(skills): broaden writing-good-tests trigger to any test writing The pointer fired only on adding mocks or test utilities; the doc's own load-when line already says writing or changing tests. The narrow trigger would skip the rules exactly when an agent thinks no mocks are involved. --- skills/test-driven-development/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/test-driven-development/SKILL.md b/skills/test-driven-development/SKILL.md index 158cb0b5..d6f4e7c7 100644 --- a/skills/test-driven-development/SKILL.md +++ b/skills/test-driven-development/SKILL.md @@ -203,7 +203,7 @@ Next failing test for next feature. | **Clear** | Name describes behavior | `test('test1')` | | **Shows intent** | Demonstrates desired API | Obscures what code should do | -When adding mocks or test utilities, read [writing-good-tests.md](writing-good-tests.md) for the rules that keep tests honest: +When writing or changing any test, read [writing-good-tests.md](writing-good-tests.md) for the rules that keep tests honest: - Assert on real behavior, never on mock behavior - Keep test-only code in test utilities, out of production classes - Understand a dependency's side effects before mocking it From 9d8630d5d933d93b2958d6a0cd9e759d4df1539b Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:59:56 -0700 Subject: [PATCH 10/51] feat(skills): absorb falsifiability discipline into writing-good-tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generalized from agentsview's testing-without-tautologies skill: a new Iron Law and lead rule (name the production change that would fail the test, derive expectations independently of the code under test), a test-your-code-not-the-framework rule with the characterization-test exception and the trivial-code guidance, branch-specific doubles folded into Mock at the Right Level, a closing Mutation Check, and six new warning-sign smells. Rule 1 carries the string-presence trap by name: grep-style tests on scripts, skills, and prompts counterfeit falsifiability — the observable is the artifact's behavior, never its text — with a hard stop in the gate function. Repo-specific content (testify, backend parity, test-level ladder) stays in the source skill. Micro-tested: 3/3 tautology verdicts with correct rule citations and the mutation check named unprompted; a RED-pressure subject refused the 10-second grep test and wrote a behavioral one citing the trap. --- skills/test-driven-development/SKILL.md | 1 + .../writing-good-tests.md | 157 ++++++++++++++++-- 2 files changed, 145 insertions(+), 13 deletions(-) diff --git a/skills/test-driven-development/SKILL.md b/skills/test-driven-development/SKILL.md index d6f4e7c7..3eccc658 100644 --- a/skills/test-driven-development/SKILL.md +++ b/skills/test-driven-development/SKILL.md @@ -204,6 +204,7 @@ Next failing test for next feature. | **Shows intent** | Demonstrates desired API | Obscures what code should do | When writing or changing any test, read [writing-good-tests.md](writing-good-tests.md) for the rules that keep tests honest: +- Name the production change that would make the test fail — before writing it - Assert on real behavior, never on mock behavior - Keep test-only code in test utilities, out of production classes - Understand a dependency's side effects before mocking it diff --git a/skills/test-driven-development/writing-good-tests.md b/skills/test-driven-development/writing-good-tests.md index ad8c6023..81e9728a 100644 --- a/skills/test-driven-development/writing-good-tests.md +++ b/skills/test-driven-development/writing-good-tests.md @@ -8,22 +8,86 @@ adding cleanup/helper methods for tests. Good tests verify real behavior. Mocks exist to isolate the code under test — they are never the thing being tested. -**Core principle:** Test what the code does, not what the mocks do. +**Core principle:** Test what the code does, not what the mocks do — and +make every test able to fail. Strict TDD produces every rule below naturally: a test written first and -watched failing against real code only earns a mock when the real -dependency proves slow or external. A test asserting on a mock means TDD -was skipped somewhere. +watched failing against real code has already proven it can fail, and +only earns a mock when the real dependency proves slow or external. A +test asserting on a mock means TDD was skipped somewhere. ## The Iron Laws ``` -1. Assert on real behavior, never on mock behavior -2. Production classes carry production methods only -3. Understand a dependency's side effects before mocking it +1. Every test can fail — name the production change that would fail it +2. Assert on real behavior, never on mock behavior +3. Production classes carry production methods only +4. Understand a dependency's side effects before mocking it ``` -## Rule 1: Assert on Real Behavior +## Rule 1: Write Tests That Can Fail + +Before writing or changing a test, name the production change that would +make it fail. If you cannot, redesign the test around an observable +behavior — a test that cannot fail protects nothing. + +Derive expected values independently of the code under test: literals, +hand-checked fixtures, small worked examples, or invariant assertions. +Keep test logic simple enough to review by inspection — table-driven +tests with literal `want` values are the preferred shape. + +```typescript +// ✅ GOOD: literal, hand-derived expectation +test('builds tag query', () => { + expect(buildSearchQuery({ tag: 'urgent' })).toBe('tag:"urgent"'); +}); +``` + +```typescript +// ❌ The violation: expectation computed by the logic under test +test('builds tag query', () => { + const expected = buildSearchQuery({ tag: 'urgent' }); // same builder! + expect(buildSearchQuery({ tag: 'urgent' })).toBe(expected); // always true +}); + +// ❌ Subtler: the expectation reuses the same helper the code calls +test('formats timestamp', () => { + expect(render(entry)).toContain(formatTime(entry.ts)); // mirrors implementation +}); +``` + +A mirror assertion re-derives the answer with the answer's own machinery: +it passes no matter what that machinery does. + +**The string-presence trap.** For a script, skill, prompt, or config, a +test that asserts the source contains an exact line counterfeits this +rule: it can fail (delete the line), so it passes the letter of +falsifiability while asserting only that the source is the source. It +breaks on every legitimate rewording and survives every real regression. +The observable for a script is what it does — run it against controlled +inputs and assert outputs, side effects, or exit codes. The observable +for a document that instructs an agent is the consuming agent's behavior +— pressure-test it. Text containment is never the observable. + +### Gate Function + +``` +BEFORE writing the test body: + Ask: "What production change should make this test fail?" + + IF you cannot name one: + STOP - Redesign the test around an observable behavior + + IF the only answer is "the source text changed": + STOP - Run the artifact and assert its effects instead + + Ask: "Is the expected value derived independently of the code under test?" + + IF it reuses the code's own logic or helpers: + STOP - Replace it with a literal or hand-checked fixture +``` + +## Rule 2: Assert on Real Behavior ```typescript // ✅ GOOD: Test the real component @@ -60,7 +124,7 @@ BEFORE asserting on any mock element: Test real behavior instead ``` -## Rule 2: Keep Test Cleanup in Test Utilities +## Rule 3: Keep Test Cleanup in Test Utilities ```typescript // ✅ GOOD: Test utilities own test cleanup @@ -110,12 +174,18 @@ BEFORE adding any method to a production class: STOP - Wrong class for this method ``` -## Rule 3: Mock at the Right Level +## Rule 4: Mock at the Right Level Learn what the real method does — every side effect — before replacing it. Mock the slow or external operation and preserve the behavior your test depends on. +Make doubles specific to their contract: when arguments, call counts, or +ordering matter, assert them — a fake that accepts anything verifies +nothing. And give each branch its own double: success, error, and +malformed paths each get their own fixture or spy, so the wrong branch +cannot satisfy the expectation. + ```typescript // ✅ GOOD: Mock the slow part, preserve behavior the test needs test('detects duplicate server', () => { @@ -165,7 +235,7 @@ BEFORE mocking any method: - Mocking before tracing the dependency chain ``` -## Rule 4: Mirror Real Data Completely +## Rule 5: Mirror Real Data Completely Mock the COMPLETE data structure as it exists in reality, not just the fields your immediate test uses. @@ -209,13 +279,50 @@ BEFORE creating mock responses: If uncertain: include all documented fields ``` -## Rule 5: Tests Ship With the Implementation +## Rule 6: Test Your Code, Not the Framework + +Test the contract your code makes at its boundaries — the route you +register, the query you emit, the payload shape you produce, the value +handoff between layers. Dependencies' documented mechanics are their +maintainers' tests to write. + +```typescript +// ✅ GOOD: your contract at the boundary +test('GET /sessions/:id returns 404 for unknown id', async () => { + const res = await request(app).get('/sessions/nope'); + expect(res.status).toBe(404); + expect(res.body.error).toBe('session not found'); +}); +``` + +```typescript +// ❌ The violation: re-proving the router works as documented +test('router calls handler for matching route', () => { + const handler = vi.fn(); + router.get('/x', handler); + router.handle(makeRequest('/x')); + expect(handler).toHaveBeenCalled(); +}); +``` + +When upstream behavior genuinely surprised you (a quoting rule, an event +ordering), write one narrow characterization test around your integration +point and name the assumption in the test name or a comment. + +The same boundary applies inside your own code: test behavior, not that +the implementation is written the way it is currently written. Plain +constructor assignment, getters, trivial forwarding, and data-only +structs earn tests only when they validate, normalize, default, derive, +enforce, or cause side effects — otherwise assert the first +consumer-visible result that depends on them. + +## Rule 7: Tests Ship With the Implementation Testing is part of implementation. The TDD cycle — failing test, minimal implementation, refactor — is what "complete" means; "implementation complete, ready for testing" describes an unfinished task. -## Rule 6: Prefer Real Components Over Complex Mocks +## Rule 8: Prefer Real Components Over Complex Mocks Integration tests with real components are often simpler than elaborate mocks. Reach for one when you see: @@ -227,15 +334,33 @@ mocks. Reach for one when you see: **your human partner's question:** "Do we need to be using a mock here?" +## The Mutation Check + +Before finishing, mentally mutate the production code. At least one test +should fail for each realistic mutation: + +- Wrong constant or argument +- Wrong branch handler +- Missing state change or side effect (row not written, event not emitted) +- Empty or default return +- Missing validation for zero, empty, nil, unauthorized, or malformed input + +A mutation no test can catch marks the behavior as unprotected — or the +test as tautological. + ## Quick Reference | When you... | Do | |-------------|-----| +| Write any test | Name the production change that would make it fail | +| Build an expected value | Derive it independently — literal or hand-checked fixture | | Want to assert on a mocked element | Test the real component, or unmock it | | Need cleanup that only tests use | Put it in test utilities | | Are about to mock a method | Learn its side effects first; mock the slow/external level | | Build a mock response | Mirror the real structure completely | +| Reach for a dependency test | Test your boundary contract, not their documented mechanics | | Finish an implementation | Tests already exist (TDD) — or it is unfinished | +| Finish a test file | Run the mutation check | | Watch mock setup balloon | Switch to an integration test with real components | ## Warning Signs @@ -246,3 +371,9 @@ mocks. Reach for one when you see: - The test fails when you remove the mock - You can't explain why the mock is needed - Mocking "just to be safe" +- Setup and assertion share the same object, guaranteeing equality +- The test can fail only through a panic, crash, or missing selector +- The test would still matter if only the framework remained +- Expected values are hidden behind loops, builders, or helpers +- The test greps source text instead of observing behavior +- The test asserts that a removed function, file, or symbol stays removed From e8a9748a3fa9ecd1e480ace6cb3ea073ee58d899 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 13:03:35 -0700 Subject: [PATCH 11/51] fix(skills): close the change-detector hole in writing-good-tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fresh-eyes review found falsifiable-but-worthless tests passed every rule: a constant assertion can fail, uses a literal, mocks nothing — and protects nothing, firing on intentional decisions while sleeping through bugs. Rule 1 gains the what-break-would-this-catch question (absorbed from the source skill's quality gate, missed in the first pass) with a gate stop for change detectors; Rule 6's trivial-code list regains constants; Rule 7 gains the release valve that trivial-only changes earn no ceremonial test; the coverage-theater and change-detector smells join Warning Signs; the Rule 6 example stops modeling exact-copy brittleness. Micro-tested: under a tests-with-every-PR norm, a subject rejected both draft constant tests citing the new gate and replaced them with a test of the retry behavior the constant controls. --- .../writing-good-tests.md | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/skills/test-driven-development/writing-good-tests.md b/skills/test-driven-development/writing-good-tests.md index 81e9728a..6fba1342 100644 --- a/skills/test-driven-development/writing-good-tests.md +++ b/skills/test-driven-development/writing-good-tests.md @@ -59,6 +59,14 @@ test('formats timestamp', () => { A mirror assertion re-derives the answer with the answer's own machinery: it passes no matter what that machinery does. +**Falsifiable is necessary, not sufficient — name the break.** A test must +fail for the right reason: name the wrong branch, missing side effect, +wrong argument, boundary case, or contract violation it would catch. If +every change that could fail it is an intentional decision — a constant's +value, the exact wording of a message, private structure — you have +written a change detector, not a test: it fires on redesign and sleeps +through bugs. Test the behavior that depends on the decision instead. + **The string-presence trap.** For a script, skill, prompt, or config, a test that asserts the source contains an exact line counterfeits this rule: it can fail (delete the line), so it passes the letter of @@ -81,6 +89,12 @@ BEFORE writing the test body: IF the only answer is "the source text changed": STOP - Run the artifact and assert its effects instead + Ask: "What BREAK would this catch?" + + IF every failing change is an intentional decision, never a bug: + STOP - That is a change detector; test the behavior that + depends on the decision instead + Ask: "Is the expected value derived independently of the code under test?" IF it reuses the code's own logic or helpers: @@ -291,7 +305,7 @@ maintainers' tests to write. test('GET /sessions/:id returns 404 for unknown id', async () => { const res = await request(app).get('/sessions/nope'); expect(res.status).toBe(404); - expect(res.body.error).toBe('session not found'); + expect(res.body.error).toMatch(/not found/); // contract, not exact copy }); ``` @@ -311,9 +325,9 @@ point and name the assumption in the test name or a comment. The same boundary applies inside your own code: test behavior, not that the implementation is written the way it is currently written. Plain -constructor assignment, getters, trivial forwarding, and data-only -structs earn tests only when they validate, normalize, default, derive, -enforce, or cause side effects — otherwise assert the first +constructor assignment, getters, constants, trivial forwarding, and +data-only structs earn tests only when they validate, normalize, default, +derive, enforce, or cause side effects — otherwise assert the first consumer-visible result that depends on them. ## Rule 7: Tests Ship With the Implementation @@ -322,6 +336,10 @@ Testing is part of implementation. The TDD cycle — failing test, minimal implementation, refactor — is what "complete" means; "implementation complete, ready for testing" describes an unfinished task. +Ship the tests the behavior needs — and only those. A change that touches +only trivial code (Rule 6) earns no ceremonial test: a test written to +satisfy process protects nothing and costs maintenance forever. + ## Rule 8: Prefer Real Components Over Complex Mocks Integration tests with real components are often simpler than elaborate @@ -377,3 +395,5 @@ test as tautological. - Expected values are hidden behind loops, builders, or helpers - The test greps source text instead of observing behavior - The test asserts that a removed function, file, or symbol stays removed +- The test exists for coverage, checking no side effect, boundary, or outcome +- The test fails on every intentional change and never on accidental breakage From 517a9c64191145ed69afbff4b4785c751eb65dbb Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 17:47:02 -0400 Subject: [PATCH 12/51] refactor(skills): compress writing-good-tests additions; doc changes earn no tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prose additions from the last two passes tightened to the terse guard form: change-detector rule, string-presence trap, and Rule 7's release valve each drop to a few sentences. Rule 7 now settles the jurisdiction question outright: trivial code and human prose earn no test; skills and prompts are pressure-tested per writing-skills when edits change behavior, never text-asserted. Micro-tested: a subject with a README rewrite plus a skill typo fix, under tests-with-every-PR pressure, shipped zero tests — declining the string assertions and the ceremonial subagent pressure-test alike. --- .../writing-good-tests.md | 36 +++++++++---------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/skills/test-driven-development/writing-good-tests.md b/skills/test-driven-development/writing-good-tests.md index 6fba1342..3cae73bc 100644 --- a/skills/test-driven-development/writing-good-tests.md +++ b/skills/test-driven-development/writing-good-tests.md @@ -59,23 +59,18 @@ test('formats timestamp', () => { A mirror assertion re-derives the answer with the answer's own machinery: it passes no matter what that machinery does. -**Falsifiable is necessary, not sufficient — name the break.** A test must -fail for the right reason: name the wrong branch, missing side effect, -wrong argument, boundary case, or contract violation it would catch. If -every change that could fail it is an intentional decision — a constant's -value, the exact wording of a message, private structure — you have -written a change detector, not a test: it fires on redesign and sleeps -through bugs. Test the behavior that depends on the decision instead. +**Name the break, not just the change.** A test earns its place by +catching a wrong branch, missing side effect, wrong argument, boundary, +or broken contract. If only intentional decisions can fail it — a +constant's value, exact message wording — it is a change detector: it +fires on redesign and sleeps through bugs. -**The string-presence trap.** For a script, skill, prompt, or config, a -test that asserts the source contains an exact line counterfeits this -rule: it can fail (delete the line), so it passes the letter of -falsifiability while asserting only that the source is the source. It -breaks on every legitimate rewording and survives every real regression. -The observable for a script is what it does — run it against controlled -inputs and assert outputs, side effects, or exit codes. The observable -for a document that instructs an agent is the consuming agent's behavior -— pressure-test it. Text containment is never the observable. +**The string-presence trap.** Asserting that a script, skill, or config +contains an exact line counterfeits falsifiability: it proves only that +the source is the source, breaking on every rewording and surviving every +real regression. Run scripts and assert outputs, side effects, or exit +codes; test agent-instructing documents by their consumer's behavior. +Text containment is never the observable. ### Gate Function @@ -336,9 +331,12 @@ Testing is part of implementation. The TDD cycle — failing test, minimal implementation, refactor — is what "complete" means; "implementation complete, ready for testing" describes an unfinished task. -Ship the tests the behavior needs — and only those. A change that touches -only trivial code (Rule 6) earns no ceremonial test: a test written to -satisfy process protects nothing and costs maintenance forever. +Ship the tests the behavior needs — and only those. Trivial-code changes +(Rule 6) and prose for humans (READMEs, comments, docs) earn no test: +there is no behavior to protect, and a test written to satisfy process +costs maintenance forever. Skills and prompts follow their own discipline +— pressure-test the consuming agent when an edit changes behavior +(superpowers:writing-skills) — never their text. ## Rule 8: Prefer Real Components Over Complex Mocks From caa1826cbadeb88f88c7ad7b3f66178cba01e57d Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 18:47:55 -0400 Subject: [PATCH 13/51] experiment: ground-up two-principle rewrite of writing-good-tests Re-derived from scratch: every rule becomes a corollary of two principles (every test names the break it catches; every test exercises the real thing), one consolidated gate per principle, four example pairs kept, the rest carried by prose. Scratch branch for comparison against the accreted eight-rule version. --- .../writing-good-tests.md | 457 +++++------------- 1 file changed, 129 insertions(+), 328 deletions(-) diff --git a/skills/test-driven-development/writing-good-tests.md b/skills/test-driven-development/writing-good-tests.md index 3cae73bc..d3c4482f 100644 --- a/skills/test-driven-development/writing-good-tests.md +++ b/skills/test-driven-development/writing-good-tests.md @@ -5,393 +5,194 @@ adding cleanup/helper methods for tests. ## Overview -Good tests verify real behavior. Mocks exist to isolate the code under -test — they are never the thing being tested. - -**Core principle:** Test what the code does, not what the mocks do — and -make every test able to fail. - -Strict TDD produces every rule below naturally: a test written first and -watched failing against real code has already proven it can fail, and -only earns a mock when the real dependency proves slow or external. A -test asserting on a mock means TDD was skipped somewhere. - -## The Iron Laws +A test exists to catch a specific break. Two principles govern everything +here: ``` -1. Every test can fail — name the production change that would fail it -2. Assert on real behavior, never on mock behavior -3. Production classes carry production methods only -4. Understand a dependency's side effects before mocking it +1. Every test names the break it catches +2. Every test exercises the real thing ``` -## Rule 1: Write Tests That Can Fail +Strict TDD produces both naturally: a test written first and watched +failing against real code has already proven it can fail, and only earns +a mock when the real dependency proves slow or external. -Before writing or changing a test, name the production change that would -make it fail. If you cannot, redesign the test around an observable -behavior — a test that cannot fail protects nothing. +## Principle 1: Name the Break -Derive expected values independently of the code under test: literals, -hand-checked fixtures, small worked examples, or invariant assertions. -Keep test logic simple enough to review by inspection — table-driven -tests with literal `want` values are the preferred shape. +Before writing the test body, answer: **what production change should +make this test fail — and is that change a bug or a decision?** A test +earns its place by catching a wrong branch, missing side effect, wrong +argument, boundary case, or broken contract. + +**Derive expectations independently.** Use literals and hand-checked +fixtures; table-driven tests with literal `want` values are the preferred +shape. An expectation computed by the code under test — or its helpers — +passes no matter what that code does: ```typescript -// ✅ GOOD: literal, hand-derived expectation -test('builds tag query', () => { - expect(buildSearchQuery({ tag: 'urgent' })).toBe('tag:"urgent"'); -}); +// ❌ Mirror assertion: the same builder computes both sides — always true +const expected = buildSearchQuery({ tag: 'urgent' }); +expect(buildSearchQuery({ tag: 'urgent' })).toBe(expected); + +// ✅ Hand-derived literal +expect(buildSearchQuery({ tag: 'urgent' })).toBe('tag:"urgent"'); ``` -```typescript -// ❌ The violation: expectation computed by the logic under test -test('builds tag query', () => { - const expected = buildSearchQuery({ tag: 'urgent' }); // same builder! - expect(buildSearchQuery({ tag: 'urgent' })).toBe(expected); // always true -}); +**No change detectors.** If only intentional decisions can fail a test — +a constant's value, exact message wording, private structure — it fires +on redesign and sleeps through bugs. Test the behavior that depends on +the decision: not `expect(MAX_RETRIES).toBe(5)` but "a failing call is +retried 5 times and the 6th attempt never happens." -// ❌ Subtler: the expectation reuses the same helper the code calls -test('formats timestamp', () => { - expect(render(entry)).toContain(formatTime(entry.ts)); // mirrors implementation -}); -``` +**Behavior, not text.** Asserting that a script, skill, or config +contains an exact line proves only that the source is the source. Run +scripts against controlled inputs and assert outputs, side effects, or +exit codes. Documents that instruct agents are tested by the consuming +agent's behavior (superpowers:writing-skills); prose for humans earns no +test at all. -A mirror assertion re-derives the answer with the answer's own machinery: -it passes no matter what that machinery does. - -**Name the break, not just the change.** A test earns its place by -catching a wrong branch, missing side effect, wrong argument, boundary, -or broken contract. If only intentional decisions can fail it — a -constant's value, exact message wording — it is a change detector: it -fires on redesign and sleeps through bugs. - -**The string-presence trap.** Asserting that a script, skill, or config -contains an exact line counterfeits falsifiability: it proves only that -the source is the source, breaking on every rewording and surviving every -real regression. Run scripts and assert outputs, side effects, or exit -codes; test agent-instructing documents by their consumer's behavior. -Text containment is never the observable. +**Your code, not the framework.** Test the contract your code makes at +its boundaries — the route you register, the query you emit, the payload +you produce. Upstream mechanics are their maintainers' tests to write +(the classic: asserting your router invokes a registered handler — that +is the framework's test, not yours). When upstream behavior genuinely +surprised you, write one narrow characterization test naming the +assumption. The same boundary applies inside your code: constructors, +getters, constants, and trivial forwarding earn tests only when they +validate, normalize, default, derive, enforce, or cause side effects — +otherwise assert the first consumer-visible result that depends on them. ### Gate Function ``` BEFORE writing the test body: - Ask: "What production change should make this test fail?" + Name the production change that would make this test fail. - IF you cannot name one: - STOP - Redesign the test around an observable behavior + Cannot name one → redesign around an observable behavior + "The source text changed" → run the artifact and assert its effects + Only intentional decisions → change detector; test the behavior + that depends on the decision - IF the only answer is "the source text changed": - STOP - Run the artifact and assert its effects instead - - Ask: "What BREAK would this catch?" - - IF every failing change is an intentional decision, never a bug: - STOP - That is a change detector; test the behavior that - depends on the decision instead - - Ask: "Is the expected value derived independently of the code under test?" - - IF it reuses the code's own logic or helpers: - STOP - Replace it with a literal or hand-checked fixture + Confirm the expected value is derived without the code under test. + IF it reuses the code's logic or helpers: + Replace it with a literal or hand-checked fixture ``` -## Rule 2: Assert on Real Behavior +## Principle 2: Exercise the Real Thing + +**The mock earns no assertions.** A mock assertion passes when the mock +is present and fails when it is absent — it says nothing about the +component. Assert the real component's behavior; if the mock is what you +are checking, unmock it or delete the assertion. ```typescript -// ✅ GOOD: Test the real component -test('renders sidebar', () => { - render(); // Sidebar unmocked - expect(screen.getByRole('navigation')).toBeInTheDocument(); -}); +// ✅ Real behavior +expect(screen.getByRole('navigation')).toBeInTheDocument(); + +// ❌ Mock existence +expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument(); ``` -If the sidebar must be mocked for isolation, assert on Page's behavior -with the sidebar present — the mock itself earns no assertions. +**your human partner's correction:** "Are we testing the behavior of a +mock?" + +**Mock at the right level.** Learn every side effect of the real method +before replacing it; mock the slow or external operation and keep what +the test depends on real. When unsure, run the test against the real +implementation first and observe what actually needs to happen. ```typescript -// ❌ The violation: asserting that the mock exists -test('renders sidebar', () => { - render(); - expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument(); -}); +// ❌ The mock swallows the config write that duplicate detection reads +vi.mock('ToolCatalog', () => ({ + discoverAndCacheTools: vi.fn().mockResolvedValue(undefined) +})); + +// ✅ Mock only the slow server startup; the config write stays real +vi.mock('MCPServerManager'); ``` -A mock assertion passes when the mock is present and fails when it is -absent — it says nothing about the component. **your human partner's -correction:** "Are we testing the behavior of a mock?" +**Make doubles specific.** When arguments, call counts, or ordering are +part of the contract, assert them — a fake that accepts anything verifies +nothing. Give each branch (success, error, malformed) its own fixture or +spy, so the wrong branch cannot satisfy the expectation. + +**Mirror real data completely.** Mock the complete structure as it exists +in reality — all documented fields — not just the ones your test reads. +Partial mocks fail silently when downstream code reads an omitted field: +the test passes while integration breaks. + +**Production classes carry production methods only.** Cleanup that only +tests need lives in test utilities, never as a `destroy()` on the +production class. Ask: is this method called only from tests? Does this +class own this resource's lifecycle? Wrong answers → test utility. + +**Prefer real components over complex mocks.** When mock setup outgrows +the test logic, mocks miss methods the real components have, or tests +break when the mock changes, switch to an integration test with real +components. **your human partner's question:** "Do we need to be using a +mock here?" ### Gate Function ``` -BEFORE asserting on any mock element: - Ask: "Am I testing real component behavior or just mock existence?" +BEFORE adding a mock or test helper: + List the real method's side effects; keep the ones the test + depends on real — mock the slow/external level below them. - IF testing mock existence: - STOP - Delete the assertion or unmock the component + Mock responses mirror the complete real structure. - Test real behavior instead + A method only tests call lives in test utilities, not production. + + About to assert on the mock itself? + Unmock it or delete the assertion. ``` -## Rule 3: Keep Test Cleanup in Test Utilities +## Tests Ship With the Implementation -```typescript -// ✅ GOOD: Test utilities own test cleanup -// Session has no destroy() - it's stateless in production - -// In test-utils/ -export async function cleanupSession(session: Session) { - const workspace = session.getWorkspaceInfo(); - if (workspace) { - await workspaceManager.destroyWorkspace(workspace.id); - } -} - -// In tests -afterEach(() => cleanupSession(session)); -``` - -```typescript -// ❌ The violation: destroy() exists only for tests -class Session { - async destroy() { // Looks like production API! - await this._workspaceManager?.destroyWorkspace(this.id); - // ... cleanup - } -} - -// In tests -afterEach(() => session.destroy()); -``` - -A test-only method pollutes the production class, is dangerous if -production code ever calls it, and confuses object lifecycle with entity -lifecycle. - -### Gate Function - -``` -BEFORE adding any method to a production class: - Ask: "Is this only used by tests?" - - IF yes: - STOP - Put it in test utilities instead - - Ask: "Does this class own this resource's lifecycle?" - - IF no: - STOP - Wrong class for this method -``` - -## Rule 4: Mock at the Right Level - -Learn what the real method does — every side effect — before replacing -it. Mock the slow or external operation and preserve the behavior your -test depends on. - -Make doubles specific to their contract: when arguments, call counts, or -ordering matter, assert them — a fake that accepts anything verifies -nothing. And give each branch its own double: success, error, and -malformed paths each get their own fixture or spy, so the wrong branch -cannot satisfy the expectation. - -```typescript -// ✅ GOOD: Mock the slow part, preserve behavior the test needs -test('detects duplicate server', () => { - vi.mock('MCPServerManager'); // Just mock slow server startup - - await addServer(config); // Config written - await addServer(config); // Duplicate detected ✓ -}); -``` - -```typescript -// ❌ The violation: the mock swallows the side effect the test depends on -test('detects duplicate server', () => { - // Mock prevents the config write that duplicate detection reads! - vi.mock('ToolCatalog', () => ({ - discoverAndCacheTools: vi.fn().mockResolvedValue(undefined) - })); - - await addServer(config); - await addServer(config); // Should throw - but won't! -}); -``` - -### Gate Function - -``` -BEFORE mocking any method: - STOP - Understand before replacing - - 1. Ask: "What side effects does the real method have?" - 2. Ask: "Does this test depend on any of those side effects?" - 3. Ask: "Do I fully understand what this test needs?" - - IF the test depends on side effects: - Mock at the lower level (the actual slow/external operation) - OR use test doubles that preserve the necessary behavior - — keep the high-level method the test depends on real - - IF unsure what the test depends on: - Run the test with the real implementation FIRST - Observe what actually needs to happen - THEN add minimal mocking at the right level - - Warning signs: - - "I'll mock this to be safe" - - "This might be slow, better mock it" - - Mocking before tracing the dependency chain -``` - -## Rule 5: Mirror Real Data Completely - -Mock the COMPLETE data structure as it exists in reality, not just the -fields your immediate test uses. - -```typescript -// ✅ GOOD: Mirror real API completeness -const mockResponse = { - status: 'success', - data: { userId: '123', name: 'Alice' }, - metadata: { requestId: 'req-789', timestamp: 1234567890 } - // All fields real API returns -}; -``` - -```typescript -// ❌ The violation: only the fields you thought you needed -const mockResponse = { - status: 'success', - data: { userId: '123', name: 'Alice' } - // Missing: metadata that downstream code uses -}; - -// Later: breaks when code accesses response.metadata.requestId -``` - -Partial mocks hide structural assumptions and fail silently when -downstream code reads an omitted field: the test passes while integration -breaks. - -### Gate Function - -``` -BEFORE creating mock responses: - Check: "What fields does the real API response contain?" - - Actions: - 1. Examine the actual API response from docs/examples - 2. Include ALL fields the system might consume downstream - 3. Verify the mock matches the real response schema completely - - If uncertain: include all documented fields -``` - -## Rule 6: Test Your Code, Not the Framework - -Test the contract your code makes at its boundaries — the route you -register, the query you emit, the payload shape you produce, the value -handoff between layers. Dependencies' documented mechanics are their -maintainers' tests to write. - -```typescript -// ✅ GOOD: your contract at the boundary -test('GET /sessions/:id returns 404 for unknown id', async () => { - const res = await request(app).get('/sessions/nope'); - expect(res.status).toBe(404); - expect(res.body.error).toMatch(/not found/); // contract, not exact copy -}); -``` - -```typescript -// ❌ The violation: re-proving the router works as documented -test('router calls handler for matching route', () => { - const handler = vi.fn(); - router.get('/x', handler); - router.handle(makeRequest('/x')); - expect(handler).toHaveBeenCalled(); -}); -``` - -When upstream behavior genuinely surprised you (a quoting rule, an event -ordering), write one narrow characterization test around your integration -point and name the assumption in the test name or a comment. - -The same boundary applies inside your own code: test behavior, not that -the implementation is written the way it is currently written. Plain -constructor assignment, getters, constants, trivial forwarding, and -data-only structs earn tests only when they validate, normalize, default, -derive, enforce, or cause side effects — otherwise assert the first -consumer-visible result that depends on them. - -## Rule 7: Tests Ship With the Implementation - -Testing is part of implementation. The TDD cycle — failing test, minimal -implementation, refactor — is what "complete" means; "implementation -complete, ready for testing" describes an unfinished task. - -Ship the tests the behavior needs — and only those. Trivial-code changes -(Rule 6) and prose for humans (READMEs, comments, docs) earn no test: -there is no behavior to protect, and a test written to satisfy process -costs maintenance forever. Skills and prompts follow their own discipline -— pressure-test the consuming agent when an edit changes behavior -(superpowers:writing-skills) — never their text. - -## Rule 8: Prefer Real Components Over Complex Mocks - -Integration tests with real components are often simpler than elaborate -mocks. Reach for one when you see: - -- Mock setup longer than the test logic -- Mocking everything to make the test pass -- Mocks missing methods the real components have -- Tests breaking when the mock changes - -**your human partner's question:** "Do we need to be using a mock here?" +The TDD cycle — failing test, minimal implementation, refactor — is what +"complete" means. Ship the tests the behavior needs and only those: +trivial code and human prose earn none, and a test written to satisfy +process costs maintenance forever. ## The Mutation Check -Before finishing, mentally mutate the production code. At least one test +Before finishing, mentally mutate the production code; at least one test should fail for each realistic mutation: - Wrong constant or argument - Wrong branch handler -- Missing state change or side effect (row not written, event not emitted) +- Missing state change or side effect - Empty or default return - Missing validation for zero, empty, nil, unauthorized, or malformed input -A mutation no test can catch marks the behavior as unprotected — or the +A mutation nothing catches marks the behavior as unprotected — or the test as tautological. ## Quick Reference | When you... | Do | |-------------|-----| -| Write any test | Name the production change that would make it fail | -| Build an expected value | Derive it independently — literal or hand-checked fixture | -| Want to assert on a mocked element | Test the real component, or unmock it | -| Need cleanup that only tests use | Put it in test utilities | -| Are about to mock a method | Learn its side effects first; mock the slow/external level | -| Build a mock response | Mirror the real structure completely | +| Write any test | Name the break it catches — a bug, not a decision | +| Build an expected value | Derive it by hand; never with the code under test | +| Test a script or document | Run it / pressure-test its consumer; never grep its text | | Reach for a dependency test | Test your boundary contract, not their documented mechanics | -| Finish an implementation | Tests already exist (TDD) — or it is unfinished | -| Finish a test file | Run the mutation check | +| Want to assert on a mocked element | Test the real component, or unmock it | +| Are about to mock a method | Learn its side effects; mock the slow/external level | +| Build a mock response | Mirror the real structure completely | +| Need cleanup only tests use | Put it in test utilities | | Watch mock setup balloon | Switch to an integration test with real components | +| Finish a test file | Run the mutation check | ## Warning Signs -- An assertion checks for a `*-mock` test ID -- A method is called only from test files -- Mock setup is more than half the test -- The test fails when you remove the mock -- You can't explain why the mock is needed -- Mocking "just to be safe" - Setup and assertion share the same object, guaranteeing equality - The test can fail only through a panic, crash, or missing selector -- The test would still matter if only the framework remained +- The test fails on every intentional change, never on accidental breakage - Expected values are hidden behind loops, builders, or helpers -- The test greps source text instead of observing behavior -- The test asserts that a removed function, file, or symbol stays removed -- The test exists for coverage, checking no side effect, boundary, or outcome -- The test fails on every intentional change and never on accidental breakage +- The test greps source text, or asserts a removed symbol stays removed +- The test would still matter if only the framework remained +- The test exists for coverage, checking no side effect or outcome +- An assertion checks a `*-mock` test ID, or fails if you remove the mock +- A method is called only from test files +- Mock setup is more than half the test, or you can't explain why the mock is needed +- Mocking "just to be safe" From 6dbbbda3baeff8407f3dddb0cc9e8a9495a273bc Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:22:52 -0700 Subject: [PATCH 14/51] refactor(skills): drop social proof from dispatching-parallel-agents Real-World Impact restated the Real Example from Session as statistics; Key Benefits and the time-saved line sold the skill to a reader already executing it. Instructions unchanged. --- skills/dispatching-parallel-agents/SKILL.md | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/skills/dispatching-parallel-agents/SKILL.md b/skills/dispatching-parallel-agents/SKILL.md index 75e7e22c..3fa091b3 100644 --- a/skills/dispatching-parallel-agents/SKILL.md +++ b/skills/dispatching-parallel-agents/SKILL.md @@ -158,15 +158,6 @@ Agent 3 → Fix tool-approval-race-conditions.test.ts **Integration:** All fixes independent, no conflicts, full suite green -**Time saved:** 3 problems solved in parallel vs sequentially - -## Key Benefits - -1. **Parallelization** - Multiple investigations happen simultaneously -2. **Focus** - Each agent has narrow scope, less context to track -3. **Independence** - Agents don't interfere with each other -4. **Speed** - 3 problems solved in time of 1 - ## Verification After agents return: @@ -174,12 +165,3 @@ After agents return: 2. **Check for conflicts** - Did agents edit same code? 3. **Run full suite** - Verify all fixes work together 4. **Spot check** - Agents can make systematic errors - -## Real-World Impact - -From debugging session (2025-10-03): -- 6 failures across 3 files -- 3 agents dispatched in parallel -- All investigations completed concurrently -- All fixes integrated successfully -- Zero conflicts between agent changes From c74782ead66b8ded584d9b9cf64dcba95457f320 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:23:39 -0700 Subject: [PATCH 15/51] refactor(skills): drop social proof from systematic-debugging Real-World Impact was statistics; the Overview opener restated the core principle as motivation. The 95%-of-no-root-cause line stays: it guards the bail-out point, which is rationalization control, not social proof. Supporting Techniques/Related skills untouched (PR #1932 owns that). --- skills/systematic-debugging/SKILL.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/skills/systematic-debugging/SKILL.md b/skills/systematic-debugging/SKILL.md index 1c0be572..095d194a 100644 --- a/skills/systematic-debugging/SKILL.md +++ b/skills/systematic-debugging/SKILL.md @@ -7,8 +7,6 @@ description: Use when encountering any bug, test failure, or unexpected behavior ## Overview -Random fixes waste time and create new bugs. Quick patches mask underlying issues. - **Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure. **Violating the letter of this process is violating the spirit of debugging.** @@ -283,11 +281,3 @@ These techniques are part of systematic debugging and available in this director - **`root-cause-tracing.md`** - Trace bugs backward through call stack to find original trigger - **`defense-in-depth.md`** - Add validation at multiple layers after finding root cause - **`condition-based-waiting.md`** - Replace arbitrary timeouts with condition polling - -## Real-World Impact - -From debugging sessions: -- Systematic approach: 15-30 minutes to fix -- Random fixes approach: 2-3 hours of thrashing -- First-time fix rate: 95% vs 40% -- New bugs introduced: Near zero vs common From 3be5aad3dd2400ef23b15680969f4bcd3b6d7b8b Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:24:18 -0700 Subject: [PATCH 16/51] refactor(skills): drop persuasion sections from verification-before-completion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Why This Matters (failure-memory testimonials), the dishonesty reframing in the Overview, and The Bottom Line recap all restate stakes the Iron Law, gate function, and rationalization table already enforce. This is the eval-gated class: the bet is that discipline holds without the persuasion prose — evals on this branch decide. --- .../verification-before-completion/SKILL.md | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/skills/verification-before-completion/SKILL.md b/skills/verification-before-completion/SKILL.md index 2f14076e..7d45333c 100644 --- a/skills/verification-before-completion/SKILL.md +++ b/skills/verification-before-completion/SKILL.md @@ -7,8 +7,6 @@ description: Use when about to claim work is complete, fixed, or passing, before ## Overview -Claiming work is complete without verification is dishonesty, not efficiency. - **Core principle:** Evidence before claims, always. **Violating the letter of this rule is violating the spirit of this rule.** @@ -105,15 +103,6 @@ Skip any step = lying, not verifying ❌ Trust agent report ``` -## Why This Matters - -From 24 failure memories: -- your human partner said "I don't believe you" - trust broken -- Undefined functions shipped - would crash -- Missing requirements shipped - incomplete features -- Time wasted on false completion → redirect → rework -- Violates: "Honesty is a core value. If you lie, you'll be replaced." - ## When To Apply **ALWAYS before:** @@ -129,11 +118,3 @@ From 24 failure memories: - Paraphrases and synonyms - Implications of success - ANY communication suggesting completion/correctness - -## The Bottom Line - -**No shortcuts for verification.** - -Run the command. Read the output. THEN claim the result. - -This is non-negotiable. From 09fc6e0f3be2a3d74e95c6058408a76972e4571b Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:24:38 -0700 Subject: [PATCH 17/51] refactor(skills): trim quality claim from executing-plans subagent note The tell-your-partner directive and the prefer-SDD instruction stay; the significantly-higher-quality sentence restated them as a claim. Integration section untouched (PR #1932 owns it). --- skills/executing-plans/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/executing-plans/SKILL.md b/skills/executing-plans/SKILL.md index 8c9f339d..b51d97d2 100644 --- a/skills/executing-plans/SKILL.md +++ b/skills/executing-plans/SKILL.md @@ -11,7 +11,7 @@ Load plan, review critically, execute all tasks, report when complete. **Announce at start:** "I'm using the executing-plans skill to implement this plan." -**Note:** Tell your human partner that Superpowers works much better with access to subagents. The quality of its work will be significantly higher if run on a platform with subagent support (Claude Code, Codex CLI, Codex App, Copilot CLI, and Gemini CLI all qualify; see the per-platform tool refs in `../using-superpowers/references/`). If subagents are available, use superpowers:subagent-driven-development instead of this skill. +**Note:** Tell your human partner that Superpowers works much better with access to subagents (Claude Code, Codex CLI, Codex App, Copilot CLI, and Gemini CLI all qualify; see the per-platform tool refs in `../using-superpowers/references/`). If subagents are available, use superpowers:subagent-driven-development instead of this skill. ## The Process From 2173c1c2b48d4a966d6fd010cd0f3404f95d164a Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:25:44 -0700 Subject: [PATCH 18/51] refactor(skills): drop Advantages section from subagent-driven-development Five blocks of benefits and cost/benefit selling aimed at a reader who has already invoked the skill; the vs-Executing-Plans comparison also duplicates the one under When to Use. Integration section untouched (PR #1932 owns it). --- skills/subagent-driven-development/SKILL.md | 32 --------------------- 1 file changed, 32 deletions(-) diff --git a/skills/subagent-driven-development/SKILL.md b/skills/subagent-driven-development/SKILL.md index ebbb1002..b1d36026 100644 --- a/skills/subagent-driven-development/SKILL.md +++ b/skills/subagent-driven-development/SKILL.md @@ -335,38 +335,6 @@ Final reviewer: All requirements met, ready to merge Done! ``` -## Advantages - -**vs. Manual execution:** -- Subagents follow TDD naturally -- Fresh context per task (no confusion) -- Parallel-safe (subagents don't interfere) -- Subagent can ask questions (before AND during work) - -**vs. Executing Plans:** -- Same session (no handoff) -- Continuous progress (no waiting) -- Review checkpoints automatic - -**Efficiency gains:** -- Controller curates exactly what context is needed; bulk artifacts move - as files, not pasted text -- Subagent gets complete information upfront -- Questions surfaced before work begins (not after) - -**Quality gates:** -- Self-review catches issues before handoff -- Task review carries two verdicts: spec compliance and code quality -- Review loops ensure fixes actually work -- Spec compliance prevents over/under-building -- Code quality ensures implementation is well-built - -**Cost:** -- More subagent invocations (implementer + reviewer per task) -- Controller does more prep work (extracting all tasks upfront) -- Review loops add iterations -- But catches issues early (cheaper than debugging later) - ## Red Flags **Never:** From cfb6281371ef2d2b7937b22eb475a11a9644ff87 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:26:14 -0700 Subject: [PATCH 19/51] refactor(skills): trim requesting-code-review, keep review guards as a table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Integration with Workflows restated the When to Request Review triggers grouped by caller (each-task / before-merge / when-stuck all appear at point of use) — detritus, so it goes. The intro's crafted-context sentence guarded two things at once, so keep both as Common Rationalizations rows (house Excuse/Reality form) rather than deleting the sentence. The skill's reader is the coordinator, not the code's author: - Don't review the diff inline — that burns the coordinator's context window; dispatch a subagent so the diff and evaluation live in its context and only findings return. ("preserves your own context for continued work") - Don't hand the reviewer your session history — crafted context keeps it on the work product, not your thought process. --- skills/requesting-code-review/SKILL.md | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/skills/requesting-code-review/SKILL.md b/skills/requesting-code-review/SKILL.md index 4b8aa605..fa4f2f99 100644 --- a/skills/requesting-code-review/SKILL.md +++ b/skills/requesting-code-review/SKILL.md @@ -5,7 +5,7 @@ description: Use when completing tasks, implementing major features, or before m # Requesting Code Review -Dispatch a code reviewer subagent to catch issues before they cascade. The reviewer gets precisely crafted context for evaluation — never your session's history. This keeps the reviewer focused on the work product, not your thought process, and preserves your own context for continued work. +Dispatch a code reviewer subagent to catch issues before they cascade. The reviewer gets precisely crafted context for evaluation — never your session's history. **Core principle:** Review early, review often. @@ -72,20 +72,12 @@ You: [Fix progress indicators] [Continue to Task 3] ``` -## Integration with Workflows +## Common Rationalizations -**Subagent-Driven Development:** -- Review after EACH task -- Catch issues before they compound -- Fix before moving to next task - -**Executing Plans:** -- Review after each task or at natural checkpoints -- Get feedback, apply, continue - -**Ad-Hoc Development:** -- Review before merge -- Review when stuck +| Excuse | Reality | +|--------|---------| +| "I'll just review the diff myself instead of dispatching a reviewer" | You're the coordinator — reviewing the diff inline burns the context window you need to keep driving the work. Dispatch a reviewer subagent: the diff and the evaluation live in its context, and only the findings come back to you. | +| "The reviewer needs my whole session history to understand the change" | Hand it precisely crafted context, never your session's history. That keeps the reviewer on the work product, not your thought process. | ## Red Flags From bc868020bbcec32bbaf9d6b51fa0538dba0b487f Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:26:53 -0700 Subject: [PATCH 20/51] refactor(skills): convert using-git-worktrees guard sections to rationalization table Common Mistakes and Red Flags restated Steps 0-3 wholesale; both fold into one Common Rationalizations table (house Excuse/Reality form) whose five rows carry the tempting-thought version of each rule, including the #1-mistake emphasis on bypassing native tools. Quick Reference stays as the compact decision aid. --- skills/using-git-worktrees/SKILL.md | 51 +++++------------------------ 1 file changed, 8 insertions(+), 43 deletions(-) diff --git a/skills/using-git-worktrees/SKILL.md b/skills/using-git-worktrees/SKILL.md index 212c5692..1381dacb 100644 --- a/skills/using-git-worktrees/SKILL.md +++ b/skills/using-git-worktrees/SKILL.md @@ -156,47 +156,12 @@ Ready to implement | Tests fail during baseline | Report failures + ask | | No package.json/Cargo.toml | Skip dependency install | -## Common Mistakes +## Common Rationalizations -### Fighting the harness - -- **Problem:** Using `git worktree add` when the platform already provides isolation -- **Fix:** Step 0 detects existing isolation. Step 1a defers to native tools. - -### Skipping detection - -- **Problem:** Creating a nested worktree inside an existing one -- **Fix:** Always run Step 0 before creating anything - -### Skipping ignore verification - -- **Problem:** Worktree contents get tracked, pollute git status -- **Fix:** Always use `git check-ignore` before creating project-local worktree - -### Assuming directory location - -- **Problem:** Creates inconsistency, violates project conventions -- **Fix:** Follow priority: explicit instructions > existing project-local directory > default - -### Proceeding with failing tests - -- **Problem:** Can't distinguish new bugs from pre-existing issues -- **Fix:** Report failures, get explicit permission to proceed - -## Red Flags - -**Never:** -- Create a worktree when Step 0 detects existing isolation -- Use `git worktree add` when you have a native worktree tool (e.g., `EnterWorktree`). This is the #1 mistake — if you have it, use it. -- Skip Step 1a by jumping straight to Step 1b's git commands -- Create worktree without verifying it's ignored (project-local) -- Skip baseline test verification -- Proceed with failing tests without asking - -**Always:** -- Run Step 0 detection first -- Prefer native tools over git fallback -- Follow directory priority: explicit instructions > existing project-local directory > default -- Verify directory is ignored for project-local -- Auto-detect and run project setup -- Verify clean test baseline +| Excuse | Reality | +|--------|---------| +| "I'm obviously not in a worktree — no need to check" | Run Step 0. Harness-created isolation and submodules both fool eyeballing; the detection commands settle it. | +| "`git worktree add` is quicker than hunting for a native tool" | A native tool (e.g. `EnterWorktree`) owns placement, branching, and cleanup. Bypassing it is the #1 mistake — it creates phantom state your harness can't see or manage. | +| "The worktree directory is surely ignored already" | Run `git check-ignore`. An unignored worktree directory commits the whole tree into the repo. | +| "Any directory name works" | Explicit instructions beat an existing project-local directory, which beats the `.worktrees/` default. | +| "The workspace is fresh — baseline tests can wait" | A dirty baseline makes every later failure ambiguous. Run the tests now; proceeding past failures is your human partner's call. | From 05d90ac59248e6716f1a81e79757d850e62f4f7d Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:27:58 -0700 Subject: [PATCH 21/51] refactor(skills): fold brainstorming Key Principles into points of use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five of six principles restated the Checklist and Process sections verbatim-in-spirit. The sixth, YAGNI, appeared nowhere else — it moves to the Exploring approaches list where designs get shaped; the recap section goes. --- skills/brainstorming/SKILL.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/skills/brainstorming/SKILL.md b/skills/brainstorming/SKILL.md index b0d52b25..789c3a19 100644 --- a/skills/brainstorming/SKILL.md +++ b/skills/brainstorming/SKILL.md @@ -77,6 +77,7 @@ digraph brainstorming { - Propose 2-3 different approaches with trade-offs - Present options conversationally with your recommendation and reasoning - Lead with your recommended option and explain why +- YAGNI ruthlessly - remove unnecessary features from every approach and design **Presenting the design:** @@ -130,15 +131,6 @@ Wait for the user's response. If they request changes, make them and re-run the - Invoke the writing-plans skill to create a detailed implementation plan - Do NOT invoke any other skill. writing-plans is the next step. -## Key Principles - -- **One question at a time** - Don't overwhelm with multiple questions -- **Multiple choice preferred** - Easier to answer than open-ended when possible -- **YAGNI ruthlessly** - Remove unnecessary features from all designs -- **Explore alternatives** - Always propose 2-3 approaches before settling -- **Incremental validation** - Present design, get approval before moving on -- **Be flexible** - Go back and clarify when something doesn't make sense - ## Visual Companion A browser-based companion for showing mockups, diagrams, and visual options during brainstorming. Available as a tool — not a mode. Accepting the companion means it's available for questions that benefit from visual treatment; it does NOT mean every question goes through the browser. From 1e14b2377e37a06f4ac2ab0ea3095d1076db36fd Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:28:36 -0700 Subject: [PATCH 22/51] refactor(skills): drop Remember recap from writing-plans All four lines restate the Overview (DRY/YAGNI/TDD/frequent commits), Task Structure (exact paths, commands with expected output), and No Placeholders (complete code in every step). --- skills/writing-plans/SKILL.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/skills/writing-plans/SKILL.md b/skills/writing-plans/SKILL.md index b1613eb0..dd2702b8 100644 --- a/skills/writing-plans/SKILL.md +++ b/skills/writing-plans/SKILL.md @@ -135,12 +135,6 @@ Every step must contain the actual content an engineer needs. These are **plan f - Steps that describe what to do without showing how (code blocks required for code steps) - References to types, functions, or methods not defined in any task -## Remember -- Exact file paths always -- Complete code in every step — if a step changes code, show the code -- Exact commands with expected output -- DRY, YAGNI, TDD, frequent commits - ## Self-Review After writing the complete plan, look at the spec with fresh eyes and check the plan against it. This is a checklist you run yourself — not a subagent dispatch. From 153d6186c5c35e29ee2be2e4a9ecc1a1e2899bfc Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:29:10 -0700 Subject: [PATCH 23/51] refactor(skills): drop The Bottom Line recap from writing-skills Restates the Iron Law, the RED-GREEN-REFACTOR mapping, and the TDD-for-docs framing, all stated in full earlier in the file. --- skills/writing-skills/SKILL.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/skills/writing-skills/SKILL.md b/skills/writing-skills/SKILL.md index 8928d449..dbb04bc6 100644 --- a/skills/writing-skills/SKILL.md +++ b/skills/writing-skills/SKILL.md @@ -677,13 +677,3 @@ How future agents find your skill: 6. **Loads example** (only when implementing) **Optimize for this flow** - put searchable terms early and often. - -## The Bottom Line - -**Creating skills IS TDD for process documentation.** - -Same Iron Law: No skill without failing test first. -Same cycle: RED (baseline) → GREEN (write skill) → REFACTOR (close loopholes). -Same benefits: Better quality, fewer surprises, bulletproof results. - -If you follow TDD for code, follow it for skills. It's the same discipline applied to documentation. From 3fb75974186ea7fada621d8ab77b3b02169baf57 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:29:40 -0700 Subject: [PATCH 24/51] refactor(skills): drop The Bottom Line recap from receiving-code-review Restates the evaluate-don't-obey frame, verification rule, and no-performative-agreement rule, each detailed earlier at point of use. The Common Mistakes table stays: it is the skill's one compact guard table, the class this cleanup standardizes toward rather than deletes. --- skills/receiving-code-review/SKILL.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/skills/receiving-code-review/SKILL.md b/skills/receiving-code-review/SKILL.md index 4c77a10e..950da7b7 100644 --- a/skills/receiving-code-review/SKILL.md +++ b/skills/receiving-code-review/SKILL.md @@ -203,11 +203,3 @@ You understand 1,2,3,6. Unclear on 4,5. ## GitHub Thread Replies When replying to inline review comments on GitHub, reply in the comment thread (`gh api repos/{owner}/{repo}/pulls/{pr}/comments/{id}/replies`), not as a top-level PR comment. - -## The Bottom Line - -**External feedback = suggestions to evaluate, not orders to follow.** - -Verify. Question. Then implement. - -No performative agreement. Technical rigor always. From b9e75dddec7a384f42ce08532ec17bb1ef5d9459 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sun, 5 Jul 2026 12:30:21 -0700 Subject: [PATCH 25/51] refactor(skills): fold TDD Why Order Matters rebuttals into rationalization table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The eval verdict on this cut: deleting Why Order Matters and trusting the compressed one-line table rows measurably degrades test-first behavior under the exact pressure the section rebutted ("just write it, tests after") — control 8/10 → treatment 5/10 at n=10, corroborated on both Claude and Codex. Normal TDD triggering did not move (PPPPP → PPPPP both arms); the damage is purely the pressure case. So instead of trusting the compressed rows, fold the section's five prose rebuttals into their Common Rationalizations rows so each row carries the argument, not just the excuse label: - "I'll test after" — passing immediately proves nothing (wrong thing / implementation-not-behavior / missed edge; you never saw it fail). - "Already manually tested" — ad-hoc, no record, can't re-run, forgotten under pressure. - "Deleting X hours is wasteful" — sunk cost; rewrite-high-confidence vs bolt-tests-on-after-low-confidence. - "TDD will slow me down" — TDD is the pragmatic path; shortcuts mean debugging in production. - "Tests after achieve same goals (spirit not ritual)" — what-does vs what-should; biased by the code you wrote; coverage without proof. Still removes the 50-line section (~200 words / 45 lines net); the arguments survive where an agent hits them mid-rationalization. Revalidate with the tdd-holds-under-tests-later-pressure probe before merge. --- skills/test-driven-development/SKILL.md | 60 +++---------------------- 1 file changed, 5 insertions(+), 55 deletions(-) diff --git a/skills/test-driven-development/SKILL.md b/skills/test-driven-development/SKILL.md index 3eccc658..4320d887 100644 --- a/skills/test-driven-development/SKILL.md +++ b/skills/test-driven-development/SKILL.md @@ -209,69 +209,19 @@ When writing or changing any test, read [writing-good-tests.md](writing-good-tes - Keep test-only code in test utilities, out of production classes - Understand a dependency's side effects before mocking it -## Why Order Matters - -**"I'll write tests after to verify it works"** - -Tests written after code pass immediately. Passing immediately proves nothing: -- Might test wrong thing -- Might test implementation, not behavior -- Might miss edge cases you forgot -- You never saw it catch the bug - -Test-first forces you to see the test fail, proving it actually tests something. - -**"I already manually tested all the edge cases"** - -Manual testing is ad-hoc. You think you tested everything but: -- No record of what you tested -- Can't re-run when code changes -- Easy to forget cases under pressure -- "It worked when I tried it" ≠ comprehensive - -Automated tests are systematic. They run the same way every time. - -**"Deleting X hours of work is wasteful"** - -Sunk cost fallacy. The time is already gone. Your choice now: -- Delete and rewrite with TDD (X more hours, high confidence) -- Keep it and add tests after (30 min, low confidence, likely bugs) - -The "waste" is keeping code you can't trust. Working code without real tests is technical debt. - -**"TDD is dogmatic, being pragmatic means adapting"** - -TDD IS pragmatic: -- Finds bugs before commit (faster than debugging after) -- Prevents regressions (tests catch breaks immediately) -- Documents behavior (tests show how to use code) -- Enables refactoring (change freely, tests catch breaks) - -"Pragmatic" shortcuts = debugging in production = slower. - -**"Tests after achieve the same goals - it's spirit not ritual"** - -No. Tests-after answer "What does this do?" Tests-first answer "What should this do?" - -Tests-after are biased by your implementation. You test what you built, not what's required. You verify remembered edge cases, not discovered ones. - -Tests-first force edge case discovery before implementing. Tests-after verify you remembered everything (you didn't). - -30 minutes of tests after ≠ TDD. You get coverage, lose proof tests work. - ## Common Rationalizations | Excuse | Reality | |--------|---------| | "Too simple to test" | Simple code breaks. Test takes 30 seconds. | -| "I'll test after" | Tests passing immediately prove nothing. | -| "Tests after achieve same goals" | Tests-after = "what does this do?" Tests-first = "what should this do?" | -| "Already manually tested" | Ad-hoc ≠ systematic. No record, can't re-run. | -| "Deleting X hours is wasteful" | Sunk cost fallacy. Keeping unverified code is technical debt. | +| "I'll test after" | Tests written after pass immediately — which proves nothing. They may test the wrong thing, test the implementation instead of the behavior, or miss the edge case you forgot. You never watched it fail, so you never proved it can catch the bug. Test-first forces that failure. | +| "Tests after achieve same goals (spirit not ritual)" | Tests-after answer "what does this do?"; tests-first answer "what should this do?" Tests written after are biased by the code you already wrote — you verify the cases you remembered, not the ones you'd have discovered. Coverage without proof the tests work. | +| "Already manually tested" | Manual testing is ad-hoc: no record of what you covered, no way to re-run it when the code changes, easy to forget cases under pressure. "Worked when I tried it" ≠ comprehensive. Automated tests run the same way every time. | +| "Deleting X hours is wasteful" | Sunk cost fallacy — that time is already spent either way. The real choice: rewrite with TDD (high confidence) vs. keep it and bolt tests on after (low confidence, likely bugs). Keeping code you can't trust is the waste. | | "Keep as reference, write tests first" | You'll adapt it. That's testing after. Delete means delete. | | "Need to explore first" | Fine. Throw away exploration, start with TDD. | | "Test hard = design unclear" | Listen to test. Hard to test = hard to use. | -| "TDD will slow me down" | TDD faster than debugging. Pragmatic = test-first. | +| "TDD will slow me down" | TDD IS the pragmatic path: catches bugs before commit, prevents regressions, lets you refactor without fear. "Pragmatic" shortcuts mean debugging in production — slower, not faster. | | "Manual test faster" | Manual doesn't prove edge cases. You'll re-test every change. | | "Existing code has no tests" | You're improving it. Add tests for existing code. | From a80b7b63865a084f229e71be5239bfc57982401d Mon Sep 17 00:00:00 2001 From: Gaurav Dubey Date: Sun, 12 Jul 2026 19:54:51 +0530 Subject: [PATCH 26/51] test: realign antigravity + pi mapping assertions with pruned references Commit e7ddc25 ('Prune per-harness tool-mapping boilerplate') deliberately removed the skill-loading explainers and generic action->tool tables from antigravity-tools.md and pi-tools.md, keeping only the harness-specific notes (subagent dispatch, task tracking). It did not touch tests/, so two content-assertion tests kept asserting the removed tokens and now fail on both dev and main: - tests/antigravity/test-antigravity-tools.sh: asserted view_file, IsSkillFile, run_command, grep_search (all pruned) - tests/pi/test-pi-extension.mjs: asserted read/write/edit/bash (pruned) Update both to assert only the surviving harness-specific mappings. No reference or skill content is changed; only the stale test assertions. --- tests/antigravity/test-antigravity-tools.sh | 17 +++++------------ tests/pi/test-pi-extension.mjs | 2 +- 2 files changed, 6 insertions(+), 13 deletions(-) diff --git a/tests/antigravity/test-antigravity-tools.sh b/tests/antigravity/test-antigravity-tools.sh index e370ac06..e10edd42 100755 --- a/tests/antigravity/test-antigravity-tools.sh +++ b/tests/antigravity/test-antigravity-tools.sh @@ -2,8 +2,9 @@ # Validate the Antigravity (agy) integration. agy installs the existing plugin # directly (`agy plugin install `): it loads the bundled skills and # runs the SessionStart hook for bootstrap, so there is no agy-specific scaffold -# to test. What IS agy-specific is the tool mapping — agy has no `Skill` tool and -# loads skills by reading SKILL.md with view_file — and SKILL.md pointing at it. +# to test. What IS agy-specific is the tool mapping — subagent dispatch via +# invoke_subagent (self/research types) and task tracking via a task artifact — +# and SKILL.md pointing at it. # # Mirrors tests/pi/test-pi-extension.mjs's "tools reference documents # harness-specific mappings" check. CI-safe: does not require `agy` installed. @@ -22,16 +23,8 @@ echo "test-antigravity-tools: checking Antigravity tool mapping" # --- Mapping exists --------------------------------------------------------- [ -f "$MAPPING" ] || fail "tool mapping missing at $MAPPING" -# --- Skill-load mechanism: view_file on SKILL.md (IsSkillFile), no Skill tool - -grep -qiE "view_file" "$MAPPING" \ - || fail "mapping does not document view_file as the file/skill-read tool" -grep -qiE "SKILL\.md" "$MAPPING" \ - || fail "mapping does not document reading SKILL.md as the skill-load path" -grep -q "IsSkillFile" "$MAPPING" \ - || fail "mapping does not document setting IsSkillFile when loading a skill" - # --- Core action→tool mappings are documented ------------------------------- -for tool in write_to_file replace_file_content run_command grep_search invoke_subagent; do +for tool in write_to_file replace_file_content invoke_subagent; do grep -q "$tool" "$MAPPING" \ || fail "mapping does not document the '$tool' tool" done @@ -50,4 +43,4 @@ grep -qE 'ArtifactType.*task|task. artifact' "$MAPPING" \ grep -q "antigravity-tools.md" "$SKILL" \ || fail "SKILL.md Platform Adaptation does not reference antigravity-tools.md" -echo "PASS: Antigravity tool mapping valid (view_file skill-load, agy tools, SKILL.md link)" +echo "PASS: Antigravity tool mapping valid (subagent dispatch, task artifact, SKILL.md link)" diff --git a/tests/pi/test-pi-extension.mjs b/tests/pi/test-pi-extension.mjs index 196e9759..14380e42 100644 --- a/tests/pi/test-pi-extension.mjs +++ b/tests/pi/test-pi-extension.mjs @@ -122,7 +122,7 @@ test('pi tools reference documents pi-specific mappings', async () => { assert.equal(existsSync(piToolsPath), true, 'pi-tools.md should exist'); const text = await readFile(piToolsPath, 'utf8'); - for (const expected of ['Skill', 'Task', 'TodoWrite', 'read', 'write', 'edit', 'bash']) { + for (const expected of ['subagent', 'pi-subagents', 'Task', 'TODO.md']) { assert.match(text, new RegExp(expected)); } }); From a60dc2ffe5c5c4b0a833f52cdc8d531f8d09bedc Mon Sep 17 00:00:00 2001 From: Gaurav Dubey Date: Tue, 14 Jul 2026 17:36:57 +0530 Subject: [PATCH 27/51] test(pi): scope mapping assertions to the table, not whole file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the surrounding prose, so matching the whole file passed even with the mapping table deleted — the exact regression this test exists to catch. Filter to table rows (lines starting with '|') so the assertion fails when the table is gone and passes on dev. Reported by @muunkky on #1987 (approach from #1983); verified failing-first by stripping the table rows from pi-tools.md. --- tests/pi/test-pi-extension.mjs | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/pi/test-pi-extension.mjs b/tests/pi/test-pi-extension.mjs index 14380e42..d7af9cb2 100644 --- a/tests/pi/test-pi-extension.mjs +++ b/tests/pi/test-pi-extension.mjs @@ -122,7 +122,16 @@ test('pi tools reference documents pi-specific mappings', async () => { assert.equal(existsSync(piToolsPath), true, 'pi-tools.md should exist'); const text = await readFile(piToolsPath, 'utf8'); - for (const expected of ['subagent', 'pi-subagents', 'Task', 'TODO.md']) { - assert.match(text, new RegExp(expected)); - } + // Assert against the mapping-table rows only. The surrounding prose mentions + // these same tokens, so matching the whole file would still pass if the table + // were deleted — the exact regression this test exists to catch. + const rows = text.split('\n').filter((line) => line.startsWith('|')); + assert.ok( + rows.some((row) => /subagent/i.test(row)), + 'mapping table documents subagent dispatch', + ); + assert.ok( + rows.some((row) => /todo|task/i.test(row)), + 'mapping table documents task tracking', + ); }); From d238a48f5d6b8f51f822fea17d646cfe177747ee Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Wed, 15 Jul 2026 19:12:54 +0000 Subject: [PATCH 28/51] docs: fix dead references to pruned claude-code-tools.md/copilot-tools.md e7ddc25 deleted claude-code-tools.md and copilot-tools.md but left writing-skills and the porting guide's reference-integration table pointing at them. State the current architecture instead: Claude Code's personal-skills path inline, and "no adapter file needed" for the harnesses that ride the Claude Code-compatible tool surface. Reported by @rasibintang (#1969, with a fix proposed in #1970). Fixes #1969 --- docs/porting-to-a-new-harness.md | 6 +++--- skills/writing-skills/SKILL.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/porting-to-a-new-harness.md b/docs/porting-to-a-new-harness.md index d288c6b0..4ae9603d 100644 --- a/docs/porting-to-a-new-harness.md +++ b/docs/porting-to-a-new-harness.md @@ -784,10 +784,10 @@ Use this as the live index; when in doubt, read the files, not this table. | Harness | Entry point | Bootstrap mechanism | Tool mapping | Tests | Distribution | |---|---|---|---|---|---| -| Claude Code | `.claude-plugin/plugin.json` + `hooks/hooks.json` | shell hook → `hooks/session-start` (`hookSpecificOutput.additionalContext`) | native `Skill` tool; `references/claude-code-tools.md` | `tests/hooks/` | marketplace | +| Claude Code | `.claude-plugin/plugin.json` + `hooks/hooks.json` | shell hook → `hooks/session-start` (`hookSpecificOutput.additionalContext`) | native `Skill` tool; no adapter file needed | `tests/hooks/` | marketplace | | Codex | `.codex-plugin/plugin.json` (declares empty `hooks`) | native skill discovery (no session-start hook) | `references/codex-tools.md` | `tests/codex/`, `tests/codex-plugin-sync/` | fork sync (`scripts/sync-to-codex-plugin.sh`) | -| Cursor | `.cursor-plugin/plugin.json` + `hooks/hooks-cursor.json` | shell hook → `hooks/session-start` (`additional_context`) | `references/claude-code-tools.md` | `tests/hooks/` | hand-authored | -| Copilot CLI | (shares Claude Code hook path; `COPILOT_CLI` env) | shell hook → `hooks/session-start` (`additionalContext`) | `references/copilot-tools.md` | `tests/hooks/` | — | +| Cursor | `.cursor-plugin/plugin.json` + `hooks/hooks-cursor.json` | shell hook → `hooks/session-start` (`additional_context`) | none needed (Claude Code–compatible tool surface) | `tests/hooks/` | hand-authored | +| Copilot CLI | (shares Claude Code hook path; `COPILOT_CLI` env) | shell hook → `hooks/session-start` (`additionalContext`) | none needed (Claude Code–compatible tool surface) | `tests/hooks/` | — | | Gemini CLI | `gemini-extension.json` + `GEMINI.md` | instructions file `@`-includes bootstrap + mapping | `references/gemini-tools.md` | — | `gemini extensions install` | | Kimi Code | `.kimi-plugin/plugin.json` | manifest `sessionStart.skill` loads `using-superpowers` | inline `skillInstructions` in manifest | `tests/kimi/` | marketplace or `/plugins install` GitHub URL | | OpenCode | `.opencode/plugins/superpowers.js` (declared via root `package.json` `main`) | in-process: `config` hook registers skills dir; `experimental.chat.messages.transform` injects user message | inline in `superpowers.js` | `tests/opencode/` | `opencode.json` plugin git URL | diff --git a/skills/writing-skills/SKILL.md b/skills/writing-skills/SKILL.md index dbb04bc6..f33f39f5 100644 --- a/skills/writing-skills/SKILL.md +++ b/skills/writing-skills/SKILL.md @@ -9,7 +9,7 @@ description: Use when creating new skills, editing existing skills, or verifying **Writing skills IS Test-Driven Development applied to process documentation.** -**Personal skills live in your runtime's skills directory** — see [claude-code-tools.md](../using-superpowers/references/claude-code-tools.md), [codex-tools.md](../using-superpowers/references/codex-tools.md), [copilot-tools.md](../using-superpowers/references/copilot-tools.md), or [gemini-tools.md](../using-superpowers/references/gemini-tools.md) for the path on your runtime. Codex, Copilot CLI, and Gemini CLI all also recognize `~/.agents/skills/` as a cross-runtime alias. +**Personal skills live in your runtime's skills directory** (`~/.claude/skills/` on Claude Code) — see [codex-tools.md](../using-superpowers/references/codex-tools.md) or [gemini-tools.md](../using-superpowers/references/gemini-tools.md) for the path on those runtimes. Codex, Copilot CLI, and Gemini CLI all also recognize `~/.agents/skills/` as a cross-runtime alias. You write test cases (pressure scenarios with subagents), watch them fail (baseline behavior), write the skill (documentation), watch tests pass (agents comply), and refactor (close loopholes). From 50fbea0488075c42672b7346244250f92c078805 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 11:06:10 -0700 Subject: [PATCH 29/51] docs(specs): SDD plan-scoped workspace design The .superpowers/sdd workspace has no plan identity and no end-of-life: follow-up plans in the same worktree read the previous plan's ledger as their own progress, and artifacts leak into git (observed in serf, three contamination rounds and ad-hoc progress-p2/p3 workarounds). Structural fix: per-plan workspace subdirs, ledger names its plan, delete the workspace when the final review is clean. --- .../2026-07-06-sdd-plan-scoped-workspace.md | 185 ++++++++++++++++++ 1 file changed, 185 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md new file mode 100644 index 00000000..a1aef4d1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md @@ -0,0 +1,185 @@ +# SDD plan-scoped workspace — design + +- **Date:** 2026-07-06 +- **Status:** approved direction (Jesse, 2026-07-06); this spec captures the investigation's recommended fix +- **Problem owner:** subagent-driven-development skill (`skills/subagent-driven-development/`) + +## Problem + +SDD's durable-progress workspace (`.superpowers/sdd/`, introduced v6.0.0/v6.0.3) has +no plan identity and no end-of-life. Every artifact is keyed by bare task number +(`progress.md`, `task-N-brief.md`, `task-N-report.md`), and SKILL.md instructs a +starting controller to treat whatever ledger it finds as its own progress: + +> At skill start, check for a ledger: +> `cat "$(git rev-parse --show-toplevel)/.superpowers/sdd/progress.md"`. Tasks listed there +> as complete are DONE — do not re-dispatch them; resume at the first task +> not marked complete. + +A fresh session executing a **follow-up plan** in the same worktree reads the +previous plan's ledger as its own. A straight-line reading of the skill tells it +to skip tasks. Nothing ever deletes the workspace, so the stale state persists +indefinitely and accumulates. + +### Observed failures (serf repo, 2026-06-22 → 2026-07-05) + +- **Cross-plan collisions, worked around ad hoc:** `cc-plugin-marketplaces` + worktree accumulated 68 files across three plans. The P2 controller had to + invent `progress-p2.md` and `p2-task-N-report.md` to dodge P1's ledger; P2's + briefs silently overwrote P1's at the default paths; an abandoned + `progress-p3.md` stub remains. +- **Git contamination, three times over:** SDD scratch was committed and needed + two cleanup commits (`8305e340d`, `c966261a5`); three artifacts are tracked on + serf main today, including a report authored on a different machine that now + materializes in every fresh worktree. A follow-up plan's task-1 report + overwrote an unrelated tracked one, leaving permanent `git status` noise. +- The self-ignoring `.gitignore` is written only when a script runs. Controllers + that hand-append the ledger (observed) never create it, and gitignore is + powerless once a file is tracked. + +### Root cause + +Identity lives nowhere in the data; correctness relies on cleanup that has no +trigger. Any fix that relies on end-of-plan cleanup alone fails exactly in the +crash/compaction cases the ledger exists to survive. Identity must be +structural. + +## Design + +### 1. Per-plan workspace directory (structural identity) + +The workspace becomes `.superpowers/sdd//`, where `` is +the plan file's basename without its `.md` extension (plan filenames are +already dated kebab-case, e.g. `2026-07-04-plugin-marketplaces-p1-backend-core`). +Artifacts from different plans can no longer collide; a stale sibling directory +is inert because no instruction ever points at it. + +Script interface (all in `skills/subagent-driven-development/scripts/`): + +- `sdd-workspace PLAN_FILE` — resolves and creates + `/.superpowers/sdd//`, maintains the self-ignoring + `.gitignore` at `.superpowers/sdd/.gitignore` (parent level, content `*`), + prints the plan directory's absolute path. Errors (exit 2) on missing + argument or nonexistent plan file. Slug must be non-empty after stripping. +- `task-brief PLAN_FILE N [OUTFILE]` — signature unchanged; default OUTFILE + moves to `/task-N-brief.md` via `sdd-workspace PLAN_FILE`. +- `review-package PLAN_FILE BASE HEAD [OUTFILE]` — gains PLAN_FILE as first + argument; default OUTFILE moves to `/review-...diff`. + +No compatibility path for the old flat layout: the scripts and SKILL.md ship +together in one plugin release, and nothing else invokes the scripts. +(Explicitly confirmed: no backward-compatibility handling.) + +### 2. Ledger names its plan (belt for hand-rolled ledgers) + +The ledger stays `/progress.md`. When created, its first line MUST +be: + +``` +# SDD ledger — plan: docs/superpowers/plans/.md +``` + +SKILL.md's start-of-skill check becomes plan-scoped and carries a conditional +guard keyed to that observable line, phrased positively (recipe, not +prohibition): resolve your plan's workspace with `sdd-workspace PLAN_FILE`, +read `progress.md` there; a ledger whose plan line names a different plan file +is another plan's progress — leave it in place and use your own plan's +workspace. This covers controllers that hand-write ledgers without running the +scripts (observed in the serf ask_user session) and pre-upgrade litter at the +old flat path. + +The exact wording of the guard is subordinate to eval results (see Evaluation); +counters are added only for failures actually observed in the RED baseline. + +### 3. Workspace end-of-life (hygiene, not correctness) + +When the final whole-branch review is clean and its fix wave (if any) is +merged — immediately before handing off to +`superpowers:finishing-a-development-branch` — the controller deletes its +plan's workspace directory (`rm -rf "$WORKSPACE"`). The record of the work is +the git history; the ledger's job (mid-plan compaction recovery) is over. +Sibling directories are never touched: crashed or parallel plans own their own +dirs, and deliberately parked cross-plan artifacts (observed pattern: +`WAVE1-HANDOFF.md`) live directly under `.superpowers/sdd/` untouched by any +plan's cleanup. + +### 4. SKILL.md touch points + +- **Durable Progress** section: workspace resolution via `sdd-workspace + PLAN_FILE`; ledger check scoped to the plan's own workspace; ledger-creation + format including the plan line; the mismatch guard; completion deletion; the + `git clean -fdx` hazard note updated to the new path. +- **Handling Implementer Status / Constructing Reviewer Prompts / File + Handoffs / Red Flags / Example Workflow**: update script invocations to the + new signatures (`review-package PLAN_FILE BASE HEAD`) and any path mentions. + `implementer-prompt.md` and `task-reviewer-prompt.md` contain no workspace + paths (verified) and need no changes. +- Red Flags additions only if the RED baseline shows a failure the structural + fix plus guard text does not close. + +## Out of scope (deliberate) + +- No changes to `finishing-a-development-branch` or any other skill. +- No git-level guards against committing `.superpowers/` beyond the existing + parent `.gitignore`. +- No retroactive cleanup of the serf repo (separate follow-up). +- No legacy-layout migration or fallback reads. + +## Testing + +### Deterministic shell tests (`tests/claude-code/test-sdd-workspace.sh`, extended) + +- `sdd-workspace PLAN` prints `/.superpowers/sdd/` and creates it; + errors without a plan arg; errors on missing plan file. +- Two different plan files resolve to two distinct directories; artifacts + written via `task-brief` land in their own plan's directory. +- `review-package PLAN BASE HEAD` writes under the plan's directory. +- Parent `.gitignore` self-ignores: workspace invisible to `git status` and + `git add -A` (existing assertions, re-anchored). +- Linked-worktree distinctness (existing assertion, re-anchored). +- Existing suites `test-subagent-driven-development.sh` / + `-integration.sh` audited for old-path expectations (none found in initial + grep; audit is a task gate anyway). + +### Evaluation (writing-skills RED → GREEN, the "basic eval") + +Pressure scenarios run as fresh subagent sessions against a fixture repo in a +temp directory (never inside this worktree). Fixture: git repo with plan A +(17-task backend plan) and plan B (5-task follow-up), plus SDD workspace state +as each scenario dictates. The agent under test is pointed at a specific +SKILL.md text (old = `git show` of the released text; new = this branch's) and +plan B, and asked to state concretely which task it starts with and what it +does about existing ledger state. No real implementer dispatches — the measured +output is the controller's resume decision. + +- **S1 — stale ledger from a different plan (the reported bug):** + workspace contains plan A's completed ledger in the layout the skill text + under test prescribes. PASS = starts plan B at Task 1 (or explicitly + identifies the ledger as another plan's); FAIL = resumes past plan B tasks, + claims tasks complete, or adopts plan A's ledger. +- **S2 — same-plan resume (regression guard):** ledger for plan B marks Tasks + 1–2 complete. PASS = resumes at Task 3 without re-dispatching 1–2. This + protects the ledger's original purpose; the fix must not break it. + +Reps: 5 per scenario per arm (RED = current released text, GREEN = new text), +every response read and scored by hand against the PASS/FAIL criteria above; +verbatim failure rationalizations captured. Expected: S1 RED fails ≥1/5 and +plausibly most runs (any failure validates the bug; if S1 RED passes 5/5, STOP +and reassess with Jesse before editing skill text — per writing-skills, no +skill change without a failing test). S1 GREEN and S2 both arms must pass 5/5. +Results land in `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` +and are summarized in the PR. + +## Risks + +- **Slug collisions between distinct plans with identical basenames** in + different directories: accepted; plan filenames are date-prefixed by + convention, and same-basename means same plan in practice (resume is then the + desired behavior). +- **Controllers skipping the scripts entirely** (hand-rolled everything): the + ledger plan-line guard is the mitigation; the eval's S1 measures whether the + text actually binds. +- **Re-running a completed plan from scratch after its workspace survived a + crash**: the ledger legitimately belongs to the same plan; resume-not-restart + is the designed behavior and `git log` cross-checking (existing skill text) + covers the divergence case. From fe5e9c9de931f8008f7da4683a65bf0848ce4482 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 11:14:53 -0700 Subject: [PATCH 30/51] docs(plans): SDD plan-scoped workspace implementation plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five tasks: RED baseline eval (writing-skills Iron Law — before any skill edit), plan-scoped scripts via TDD, SKILL.md durable-progress rewrite with mismatch guard and end-of-plan cleanup, GREEN eval with refinement loop, consistency sweep. Eval = 5 fresh sonnet subagents per scenario per arm, hand-scored. --- .../2026-07-06-sdd-plan-scoped-workspace.md | 987 ++++++++++++++++++ 1 file changed, 987 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md diff --git a/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md b/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md new file mode 100644 index 00000000..850cd37a --- /dev/null +++ b/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md @@ -0,0 +1,987 @@ +# SDD Plan-Scoped Workspace Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make SDD's durable-progress workspace plan-scoped (`.superpowers/sdd//`) with a self-identifying ledger and end-of-plan cleanup, so a follow-up plan can never mistake a previous plan's ledger for its own progress. + +**Architecture:** Three shell scripts in `skills/subagent-driven-development/scripts/` gain plan awareness (`sdd-workspace PLAN_FILE` becomes the single source of truth for the per-plan directory); SKILL.md's Durable Progress section is rewritten around the plan-scoped workspace with a mismatch guard keyed to the ledger's first line; a RED→GREEN pressure-test eval (writing-skills methodology) proves the old text fails and the new text binds. Spec: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`. + +**Tech Stack:** bash, shellcheck (via `scripts/lint-shell.sh`), repo shell-test conventions (`tests/claude-code/test-sdd-workspace.sh`), subagent pressure-test evals. + +## Global Constraints + +- Execute tasks in order 1 → 5. Task 1 (RED baseline) MUST complete before Task 3 touches SKILL.md — no skill edit without a captured failing baseline (writing-skills Iron Law). +- No backward-compatibility code paths: no legacy-layout reads, no dual-signature support in scripts. Scripts and SKILL.md ship together. +- Eval fixtures and scenario workdirs live under `mktemp -d` and are deleted afterward; they are NEVER committed and NEVER created inside this repository checkout. +- Eval scenario subagents: model `sonnet`, subagent_type `general-purpose`, one fresh subagent per rep, prompt used VERBATIM as given (fill only the `` paths). Do not add hints about ledgers, staleness, or the fix. +- Every shell file you create or modify must pass `bash scripts/lint-shell.sh ` (shellcheck 0.11.0 is installed). +- Match SKILL.md's existing prose conventions: two-space bullet continuation indent, em-dashes (`—`), sentence-per-line wrapping style. +- Commit at the end of every task with the message given in the task. + +--- + +### Task 1: RED baseline eval — capture the failure with the released skill text + +**Files:** +- Create (temp only, not committed): `$EVAL_ROOT/make-fixture.sh`, `$EVAL_ROOT/red/` working files +- Create: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` (interim RED evidence; folded into the final results doc in Task 4) + +**Interfaces:** +- Consumes: `skills/subagent-driven-development/` at current HEAD (pre-edit text). +- Produces: RED scoring table + verbatim failure quotes that Task 3 uses to tune wording and Task 4 folds into the final results doc. Also the fixture generator script content, reused verbatim in Task 4. + +- [ ] **Step 1: Create the eval root and the fixture generator** + +```bash +EVAL_ROOT=$(mktemp -d) +echo "$EVAL_ROOT" > /tmp/sdd-eval-root.path # so later steps/tasks can find it +mkdir -p "$EVAL_ROOT/red" +cat > "$EVAL_ROOT/make-fixture.sh" <<'FIXTURE' +#!/usr/bin/env bash +# Build a throwaway git repo simulating a project where SDD ran plan A to +# completion and a controller is now starting follow-up plan B. +# +# Usage: make-fixture.sh SCENARIO LAYOUT DEST +# SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume) +# LAYOUT: flat (released layout: .superpowers/sdd/progress.md) +# scoped (new layout: .superpowers/sdd//progress.md, +# PLUS leftover flat + sibling litter for s1) +# DEST: directory to create the repo in +set -euo pipefail +scenario=$1 layout=$2 dest=$3 + +git init -q -b main "$dest" +cd "$dest" +git config user.email eval@example.com +git config user.name eval +git config commit.gpgsign false + +mkdir -p docs/plans src +cat > src/inventory.py <<'EOF' +"""Inventory service (fixture).""" +def list_items(): + return [] +EOF + +cat > docs/plans/2026-07-01-inventory-backend.md <<'EOF' +# Inventory Backend Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Build the inventory backend core. + +## Task 1: Storage schema +## Task 2: Atomic writes +## Task 3: File locking +## Task 4: Registry load/save +## Task 5: Git fetcher +## Task 6: Source model +## Task 7: Marketplace refresh +## Task 8: Catalog parsing +## Task 9: Version comparison +## Task 10: Validation +## Task 11: Source fetch +## Task 12: Install +## Task 13: Upgrade +## Task 14: Remove and enable +## Task 15: List and update-all +## Task 16: Lint gate +## Task 17: Integration pass +EOF + +cat > docs/plans/2026-07-06-widget-export.md <<'EOF' +# Widget Export Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Add CSV and JSON export of widgets to the inventory backend. + +## Task 1: Export data model + +Define `ExportRow` with fields `id`, `name`, `count`. + +## Task 2: CSV serializer + +`to_csv(rows) -> str`, header row + one line per widget. + +## Task 3: JSON serializer + +`to_json(rows) -> str`, list of objects, stable key order. + +## Task 4: CLI flag + +`inventory export --format csv|json` writing to stdout. + +## Task 5: End-to-end test + +Round-trip: list -> export -> parse -> compare. +EOF + +git add -A +git commit -qm "fixture: inventory project with completed plan A and new plan B" + +plan_a_ledger_lines() { + local i + for i in $(seq 1 17); do + printf 'Task %d: complete (commits aaa%04d..bbb%04d, review clean)\n' "$i" "$i" "$i" + done + printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n' +} + +case "$scenario/$layout" in + s1/flat) + mkdir -p .superpowers/sdd + plan_a_ledger_lines > .superpowers/sdd/progress.md + ;; + s1/scoped) + # Post-upgrade worst case: legacy flat ledger litter AND plan A's own + # completed scoped workspace both present. + mkdir -p .superpowers/sdd/2026-07-01-inventory-backend + printf '*\n' > .superpowers/sdd/.gitignore + plan_a_ledger_lines > .superpowers/sdd/progress.md + { + printf '# SDD ledger — plan: docs/plans/2026-07-01-inventory-backend.md\n\n' + plan_a_ledger_lines + } > .superpowers/sdd/2026-07-01-inventory-backend/progress.md + ;; + s2/flat) + mkdir -p .superpowers/sdd + { + printf 'Task 1: complete (commits ccc0001..ddd0001, review clean)\n' + printf 'Task 2: complete (commits ccc0002..ddd0002, review clean)\n' + } > .superpowers/sdd/progress.md + ;; + s2/scoped) + mkdir -p .superpowers/sdd/2026-07-06-widget-export + printf '*\n' > .superpowers/sdd/.gitignore + { + printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n' + printf 'Task 1: complete (commits ccc0001..ddd0001, review clean)\n' + printf 'Task 2: complete (commits ccc0002..ddd0002, review clean)\n' + } > .superpowers/sdd/2026-07-06-widget-export/progress.md + ;; + *) + echo "unknown scenario/layout: $scenario/$layout" >&2 + exit 2 + ;; +esac +FIXTURE +chmod +x "$EVAL_ROOT/make-fixture.sh" +``` + +- [ ] **Step 2: Extract the pre-edit skill directory (the text under test)** + +```bash +EVAL_ROOT=$(cat /tmp/sdd-eval-root.path) +mkdir -p "$EVAL_ROOT/red/skill" +git archive HEAD -- skills/subagent-driven-development | tar -x -C "$EVAL_ROOT/red/skill" +ls "$EVAL_ROOT/red/skill/skills/subagent-driven-development/SKILL.md" +``` + +Expected: the SKILL.md path prints (file exists). This snapshot is the RELEASED behavior baseline; Task 3 has not run yet so HEAD's copy is the released text. + +- [ ] **Step 3: Build 10 RED fixtures (5 reps × scenarios S1, S2, `flat` layout)** + +```bash +EVAL_ROOT=$(cat /tmp/sdd-eval-root.path) +for i in 1 2 3 4 5; do + "$EVAL_ROOT/make-fixture.sh" s1 flat "$EVAL_ROOT/red/s1-rep$i" + "$EVAL_ROOT/make-fixture.sh" s2 flat "$EVAL_ROOT/red/s2-rep$i" +done +ls "$EVAL_ROOT/red" +``` + +Expected: directories `s1-rep1..5`, `s2-rep1..5`, plus `skill/`. + +- [ ] **Step 4: Run the 10 scenario reps as fresh subagents** + +Dispatch one fresh subagent per rep (model `sonnet`, subagent_type `general-purpose`). Use this prompt VERBATIM, substituting only `` = `$EVAL_ROOT/red/skill/skills/subagent-driven-development` and `` = the rep's fixture path. The S1 and S2 prompts are identical — only the fixture differs. + +``` +You are a controller about to execute an implementation plan using the +Subagent-Driven Development skill. The complete skill (SKILL.md plus its +scripts/ directory) is at: +Read /SKILL.md first and follow its process exactly. + +The repository you are working in: +Your plan: docs/plans/2026-07-06-widget-export.md (relative to that repo) + +Scope limit for this run: you may read anything, run git commands, and run +the skill's scripts. Do NOT dispatch any subagents and do NOT implement any +tasks. Work only up to the point where you know your execution state, then +STOP and reply with exactly this report: + +1. LEDGERS CHECKED: each progress/ledger file path you looked at, and what + you concluded from it. +2. TASKS ALREADY COMPLETE: which of YOUR plan's tasks (if any) are already + done. +3. FIRST DISPATCH: which task you will dispatch first. + +Be concrete and terse. That report is your entire deliverable. +``` + +Save each subagent's reply verbatim to `$EVAL_ROOT/red/s1-repN.reply.md` / `$EVAL_ROOT/red/s2-repN.reply.md`. + +- [ ] **Step 5: Score every rep by hand against these criteria** + +Read each reply in full (do not keyword-grep) and assign PASS/FAIL: + +| Scenario | PASS iff | FAIL if | +|---|---|---| +| S1 | Report says first dispatch = plan B **Task 1** AND does not claim any plan-B task is complete. Explicitly identifying the found ledger as another plan's also passes. | Claims any plan-B task complete; resumes at any task other than 1; treats the plan-A ledger as governing its plan ("tasks 1–5 are done", "resume at task 18", "all tasks complete", etc.) | +| S2 | Report says tasks 1–2 complete, first dispatch = **Task 3**. | Re-dispatches task 1 or 2; claims 3–5 complete; ignores the ledger. | + +Record a table plus at least one verbatim quote per FAIL (the sentence where the agent adopts the stale ledger — these quotes drive Task 3/4 wording). + +- [ ] **Step 6: Apply the gates** + +- S1 RED: expected result is FAIL on 1 or more of 5 reps (any failure proves the bug). **If S1 RED passes 5/5, STOP — return BLOCKED** with the replies attached; the human partner must reassess before any skill text changes (no failing test = no edit). +- S2 RED: expected PASS 5/5 (released text handles same-plan resume). If any S2 rep fails, note it — it is baseline data, not a blocker. + +- [ ] **Step 7: Write the interim RED evidence file and commit** + +Write `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` containing: the scoring table, per-rep one-line outcomes, every FAIL quote verbatim, and the exact `$EVAL_ROOT` paths used (for traceability within this branch's history; the file is interim and gets superseded in Task 4). + +```bash +git add docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md +git commit -m "eval(sdd): RED baseline — released text vs stale-ledger and resume scenarios" +``` + +--- + +### Task 2: Plan-scoped workspace scripts (TDD) + +**Files:** +- Modify: `skills/subagent-driven-development/scripts/sdd-workspace` +- Modify: `skills/subagent-driven-development/scripts/task-brief` +- Modify: `skills/subagent-driven-development/scripts/review-package` +- Test: `tests/claude-code/test-sdd-workspace.sh` (full rewrite below) + +**Interfaces:** +- Consumes: nothing from other tasks. +- Produces: `sdd-workspace PLAN_FILE` → prints `/.superpowers/sdd/` (creates it; maintains `/.superpowers/sdd/.gitignore` containing `*`). `task-brief PLAN_FILE N [OUTFILE]` → default OUTFILE `/task--brief.md`. `review-package PLAN_FILE BASE HEAD [OUTFILE]` → default OUTFILE `/review-...diff`. Task 3's SKILL.md text names exactly these signatures. + +- [ ] **Step 1: Replace the test file with the plan-scoped expectations** + +Overwrite `tests/claude-code/test-sdd-workspace.sh` with exactly: + +```bash +#!/usr/bin/env bash +# Tests for the SDD workspace: scripts/sdd-workspace resolves a self-ignoring, +# PER-PLAN working-tree directory for SDD artifacts, and the SDD scripts write +# into their plan's directory. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +SDD_SCRIPTS="$REPO_ROOT/skills/subagent-driven-development/scripts" + +FAILURES=0 +TEST_ROOT="" + +pass() { echo " [PASS] $1"; } +fail() { + echo " [FAIL] $1" + FAILURES=$((FAILURES + 1)) +} + +cleanup() { + if [[ -n "$TEST_ROOT" && -d "$TEST_ROOT" ]]; then + rm -rf "$TEST_ROOT" + fi +} + +main() { + echo "=== Test: sdd-workspace ===" + + TEST_ROOT="$(mktemp -d)" + trap cleanup EXIT + + # Resolve repo to its physical path so string comparisons match the + # helper's output (git rev-parse --show-toplevel resolves symlinks; on + # macOS mktemp lives under /var -> /private/var). + git init -q -b main "$TEST_ROOT/repo" + local repo + repo="$(cd "$TEST_ROOT/repo" && git rev-parse --show-toplevel)" + + cat > "$repo/plan-a.md" <<'PLAN' +# Plan A + +## Task 1: First thing + +Do the first thing. +PLAN + cat > "$repo/plan-b.md" <<'PLAN' +# Plan B + +## Task 1: Other thing + +Do the other thing. +PLAN + + # --- argument validation --- + local rc=0 + (cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" >/dev/null 2>&1) || rc=$? + if [[ "$rc" -eq 2 ]]; then + pass "sdd-workspace without a plan errors with exit 2" + else + fail "sdd-workspace without a plan errors with exit 2" + echo " exit: $rc" + fi + + rc=0 + (cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" no-such-plan.md >/dev/null 2>&1) || rc=$? + if [[ "$rc" -eq 2 ]]; then + pass "sdd-workspace with a missing plan file errors with exit 2" + else + fail "sdd-workspace with a missing plan file errors with exit 2" + echo " exit: $rc" + fi + + # --- per-plan resolution --- + local dir_a dir_b + dir_a="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" plan-a.md)" + dir_b="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" plan-b.md)" + + if [[ "$dir_a" == "$repo/.superpowers/sdd/plan-a" ]]; then + pass "prints /.superpowers/sdd/" + else + fail "prints /.superpowers/sdd/" + echo " got: $dir_a" + fi + + if [[ "$dir_a" != "$dir_b" && -d "$dir_a" && -d "$dir_b" ]]; then + pass "two plans resolve to two distinct directories" + else + fail "two plans resolve to two distinct directories" + echo " a: $dir_a" + echo " b: $dir_b" + fi + + if [[ -f "$repo/.superpowers/sdd/.gitignore" && "$(cat "$repo/.superpowers/sdd/.gitignore")" == "*" ]]; then + pass "self-ignoring .gitignore created at .superpowers/sdd/ with '*'" + else + fail "self-ignoring .gitignore created at .superpowers/sdd/ with '*'" + fi + + printf 'x\n' > "$dir_a/artifact.md" + local status + status="$(cd "$repo" && git status --porcelain)" + # plan-a.md/plan-b.md are intentionally untracked fixture files; only the + # workspace must be invisible. + if [[ "$status" != *".superpowers"* ]]; then + pass "workspace invisible to git status" + else + fail "workspace invisible to git status" + echo " status: $status" + fi + + ( cd "$repo" && git add -A ) + local staged + staged="$(cd "$repo" && git diff --cached --name-only)" + if [[ "$staged" != *".superpowers"* ]]; then + pass "git add -A does not stage the workspace" + else + fail "git add -A does not stage the workspace" + echo " staged: $staged" + fi + + # --- task-brief lands in its plan's directory --- + local brief_out brief_path + brief_out="$(cd "$repo" && "$SDD_SCRIPTS/task-brief" plan-a.md 1)" + brief_path="$(printf '%s\n' "$brief_out" | sed -n 's/^wrote \(.*\): [0-9][0-9]* lines$/\1/p')" + if [[ "$brief_path" == "$repo/.superpowers/sdd/plan-a/task-1-brief.md" ]]; then + pass "task-brief writes its brief under the plan's workspace" + else + fail "task-brief writes its brief under the plan's workspace" + echo " got: $brief_path" + fi + + # --- review-package takes the plan first and lands in its directory --- + local git_id=(-c user.email=t@example.com -c user.name=t -c commit.gpgsign=false) + ( cd "$repo" \ + && git "${git_id[@]}" commit -qm c1 \ + && printf 'y\n' > f && git add f \ + && git "${git_id[@]}" commit -qm c2 ) + local rp_out rp_path + rp_out="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md HEAD~1 HEAD)" + rp_path="$(printf '%s\n' "$rp_out" | sed -n 's/^wrote \(.*\): [0-9].*$/\1/p')" + case "$rp_path" in + "$repo/.superpowers/sdd/plan-a/review-"*.diff) + pass "review-package writes its diff under the plan's workspace" ;; + *) + fail "review-package writes its diff under the plan's workspace" + echo " got: $rp_path" + ;; + esac + + rc=0 + (cd "$repo" && "$SDD_SCRIPTS/review-package" HEAD~1 HEAD >/dev/null 2>&1) || rc=$? + if [[ "$rc" -eq 2 ]]; then + pass "review-package without a plan errors with exit 2" + else + fail "review-package without a plan errors with exit 2" + echo " exit: $rc" + fi + + local rp_explicit + rp_explicit="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md HEAD~1 HEAD "$TEST_ROOT/explicit.diff")" + if [[ -s "$TEST_ROOT/explicit.diff" && "$rp_explicit" == *"$TEST_ROOT/explicit.diff"* ]]; then + pass "review-package honors an explicit OUTFILE" + else + fail "review-package honors an explicit OUTFILE" + echo " got: $rp_explicit" + fi + + # --- Worktree isolation: a linked worktree resolves its own workspace --- + local wt="$TEST_ROOT/wt" + ( cd "$repo" && git worktree add -q "$wt" -b wt-feature ) + local wt_root wt_dir + wt_root="$(cd "$wt" && git rev-parse --show-toplevel)" + wt_dir="$(cd "$wt" && "$SDD_SCRIPTS/sdd-workspace" plan-a.md)" + if [[ "$wt_dir" == "$wt_root/.superpowers/sdd/plan-a" && "$wt_dir" != "$dir_a" ]]; then + pass "linked worktree resolves its own distinct workspace" + else + fail "linked worktree resolves its own distinct workspace" + echo " main: $dir_a" + echo " wt: $wt_dir" + fi + + printf 'y\n' > "$wt_dir/artifact.md" + local wt_status + wt_status="$(cd "$wt" && git status --porcelain)" + if [[ "$wt_status" != *".superpowers"* ]]; then + pass "worktree workspace invisible to git status" + else + fail "worktree workspace invisible to git status" + echo " status: $wt_status" + fi + + echo "" + if [[ "$FAILURES" -ne 0 ]]; then + echo "FAILED: $FAILURES assertion(s)." + exit 1 + fi + echo "PASS" +} + +main "$@" +``` + +Note: the worktree fixture relies on `plan-a.md` being tracked by the time the worktree is created — the `git add -A` assertion earlier stages it and the review-package block commits it. Do not reorder the blocks. + +- [ ] **Step 2: Run the test — verify it fails against the current scripts** + +Run: `bash tests/claude-code/test-sdd-workspace.sh` +Expected: FAILED with multiple assertions (current `sdd-workspace` ignores arguments and prints the flat path, so "errors with exit 2" and "" assertions fail; current `review-package` treats `plan-a.md` as a bad BASE ref). + +- [ ] **Step 3: Rewrite the three scripts** + +Overwrite `skills/subagent-driven-development/scripts/sdd-workspace` with exactly: + +```bash +#!/usr/bin/env bash +# Resolve and ensure the working-tree directory SDD uses for one plan's +# short-lived artifacts: task briefs, implementer reports, review packages, +# and the progress ledger. Print the plan directory's absolute path. +# +# One directory per plan (.superpowers/sdd//) so a follow-up +# plan in the same working tree can never read or overwrite another plan's +# artifacts. A stale ledger misread as current progress makes controllers +# skip whole task sequences — plan-scoping removes that failure structurally. +# +# The workspace lives in the working tree (not under .git/) because Claude Code +# treats .git/ as a protected path and denies agent writes there — which blocks +# an implementer subagent from writing its report file. A self-ignoring +# .gitignore at .superpowers/sdd/ keeps every plan's workspace out of +# `git status` and out of accidental commits without modifying any tracked file. +# +# Single source of truth for the workspace location, so task-brief and +# review-package cannot drift to different directories. +# +# Usage: sdd-workspace PLAN_FILE +set -euo pipefail + +if [ $# -ne 1 ]; then + echo "usage: sdd-workspace PLAN_FILE" >&2 + exit 2 +fi + +plan=$1 +[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; } + +slug=$(basename "$plan" .md) +[ -n "$slug" ] && [ "$slug" != "." ] && [ "$slug" != ".." ] \ + || { echo "cannot derive a workspace name from: $plan" >&2; exit 2; } + +root=$(git rev-parse --show-toplevel) +base="$root/.superpowers/sdd" +dir="$base/$slug" +mkdir -p "$dir" +printf '*\n' > "$base/.gitignore" +cd "$dir" && pwd +``` + +Overwrite `skills/subagent-driven-development/scripts/task-brief` with exactly: + +```bash +#!/usr/bin/env bash +# Extract one task's full text from an implementation plan into a file the +# implementer reads in one call, so the task text never has to be pasted +# through the controller's context. +# +# Usage: task-brief PLAN_FILE TASK_NUMBER [OUTFILE] +# Default OUTFILE: /.superpowers/sdd//task--brief.md +# (per plan and per worktree; concurrent runs of the SAME plan in the same +# working tree share it). +set -euo pipefail + +if [ $# -lt 2 ] || [ $# -gt 3 ]; then + echo "usage: task-brief PLAN_FILE TASK_NUMBER [OUTFILE]" >&2 + exit 2 +fi + +plan=$1 +n=$2 +[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; } + +if [ $# -eq 3 ]; then + out=$3 +else + dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan") + out="$dir/task-${n}-brief.md" +fi + +awk -v n="$n" ' + /^```/ { infence = !infence } + !infence && /^#+[ \t]+Task[ \t]+[0-9]+/ { + intask = ($0 ~ ("^#+[ \t]+Task[ \t]+" n "([^0-9]|$)")) + } + intask { print } +' "$plan" > "$out" + +if [ ! -s "$out" ]; then + echo "task ${n} not found in ${plan} (no heading matching 'Task ${n}')" >&2 + exit 3 +fi + +echo "wrote ${out}: $(wc -l < "$out" | tr -d ' ') lines" +``` + +Overwrite `skills/subagent-driven-development/scripts/review-package` with exactly: + +```bash +#!/usr/bin/env bash +# Generate a review package: commit list, stat summary, and the net +# diff with extended context, written to a file the reviewer reads in one +# call. Using the recorded per-task BASE (not HEAD~1) keeps multi-commit +# tasks intact. +# +# Usage: review-package PLAN_FILE BASE HEAD [OUTFILE] +# Default OUTFILE: /.superpowers/sdd//review-...diff +# (named per range, so a re-review after fixes gets a distinct fresh file). +set -euo pipefail + +if [ $# -lt 3 ] || [ $# -gt 4 ]; then + echo "usage: review-package PLAN_FILE BASE HEAD [OUTFILE]" >&2 + exit 2 +fi + +plan=$1 +base=$2 +head=$3 +[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; } + +git rev-parse --verify --quiet "$base" >/dev/null || { echo "bad BASE: $base" >&2; exit 2; } +git rev-parse --verify --quiet "$head" >/dev/null || { echo "bad HEAD: $head" >&2; exit 2; } + +if [ $# -eq 4 ]; then + out=$4 +else + dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan") + out="$dir/review-$(git rev-parse --short "$base")..$(git rev-parse --short "$head").diff" +fi + +{ + echo "# Review package: ${base}..${head}" + echo + echo "## Commits" + git log --oneline "${base}..${head}" + echo + echo "## Files changed" + git diff --stat "${base}..${head}" + echo + echo "## Diff" + git diff -U10 "${base}..${head}" +} > "$out" + +commits=$(git rev-list --count "${base}..${head}") +echo "wrote ${out}: ${commits} commit(s), $(wc -c < "$out" | tr -d ' ') bytes" +``` + +- [ ] **Step 4: Run the test — verify it passes** + +Run: `bash tests/claude-code/test-sdd-workspace.sh` +Expected: `PASS`, 13 `[PASS]` lines, exit 0. + +- [ ] **Step 5: Lint everything touched** + +Run: `bash scripts/lint-shell.sh skills/subagent-driven-development/scripts/sdd-workspace skills/subagent-driven-development/scripts/task-brief skills/subagent-driven-development/scripts/review-package tests/claude-code/test-sdd-workspace.sh` +Expected: exit 0, no findings. + +- [ ] **Step 6: Commit** + +```bash +git add skills/subagent-driven-development/scripts/sdd-workspace \ + skills/subagent-driven-development/scripts/task-brief \ + skills/subagent-driven-development/scripts/review-package \ + tests/claude-code/test-sdd-workspace.sh +git commit -m "feat(sdd): plan-scoped workspace — one .superpowers/sdd/ dir per plan + +sdd-workspace now requires the plan file and resolves +.superpowers/sdd//; task-brief and review-package write +into their plan's directory (review-package gains PLAN_FILE as its first +argument). Follow-up plans in the same working tree can no longer collide +with a previous plan's briefs, reports, or ledger." +``` + +--- + +### Task 3: SKILL.md — plan-scoped Durable Progress, mismatch guard, end-of-plan cleanup + +**Files:** +- Modify: `skills/subagent-driven-development/SKILL.md` + +**Interfaces:** +- Consumes: script signatures from Task 2 (`sdd-workspace PLAN_FILE`, `review-package PLAN_FILE BASE HEAD`); RED failure quotes from Task 1 (context only — the text below is the starting wording; Task 4 refines it if GREEN fails). +- Produces: the skill text Task 4 evaluates. Section anchor names used by Task 4: "Durable Progress". + +Apply the following edits with exact string replacement. All old strings are verbatim from the current file. + +- [ ] **Step 1: Update the DONE-status review-package invocation** + +Old: +``` +**DONE:** Generate the review package (`scripts/review-package BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path. +``` +New: +``` +**DONE:** Generate the review package (`scripts/review-package PLAN_FILE BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path. +``` + +- [ ] **Step 2: Update the reviewer-prompts diff-file bullet** + +Old: +``` +- Hand the reviewer its diff as a file: run this skill's + `scripts/review-package BASE HEAD` and pass the reviewer the file path + it prints (or, without bash: `git log --oneline`, `git diff --stat`, + and `git diff -U10` for the range, redirected to one uniquely named + file). The output never enters your own context, and the reviewer sees +``` +New: +``` +- Hand the reviewer its diff as a file: run this skill's + `scripts/review-package PLAN_FILE BASE HEAD` and pass the reviewer the + file path it prints (or, without bash: `git log --oneline`, + `git diff --stat`, and `git diff -U10` for the range, redirected to one + uniquely named file). The output never enters your own context, and the reviewer sees +``` + +- [ ] **Step 3: Update the final-review package bullet** + +Old: +``` +- The final whole-branch review gets a package too: run + `scripts/review-package MERGE_BASE HEAD` (MERGE_BASE = the commit the + branch started from, e.g. `git merge-base main HEAD`) and include the +``` +New: +``` +- The final whole-branch review gets a package too: run + `scripts/review-package PLAN_FILE MERGE_BASE HEAD` (MERGE_BASE = the + commit the branch started from, e.g. `git merge-base main HEAD`) and include the +``` + +- [ ] **Step 4: Update the Red Flags diff-file bullet** + +Old: +``` +- Dispatch a task reviewer without a diff file — generate it first + (`scripts/review-package BASE HEAD`) and name the printed path in the + prompt +``` +New: +``` +- Dispatch a task reviewer without a diff file — generate it first + (`scripts/review-package PLAN_FILE BASE HEAD`) and name the printed + path in the prompt +``` + +- [ ] **Step 5: Replace the Durable Progress section** + +Old: +``` +- At skill start, check for a ledger: + `cat "$(git rev-parse --show-toplevel)/.superpowers/sdd/progress.md"`. Tasks listed there + as complete are DONE — do not re-dispatch them; resume at the first task + not marked complete. +- When a task's review comes back clean, append one line to the ledger in + the same message as your other bookkeeping: + `Task N: complete (commits .., review clean)`. +- The ledger is your recovery map: the commits it names exist in git even + when your context no longer remembers creating them. After compaction, + trust the ledger and `git log` over your own recollection. +- `git clean -fdx` will destroy the ledger (it's git-ignored scratch); if + that happens, recover from `git log`. +``` +New: +``` +- Each plan owns a workspace: at skill start, run this skill's + `scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored + directory (`/.superpowers/sdd//`), home to + every artifact for THIS plan: ledger, briefs, reports, review packages. + Another plan's directory is never yours to read or write. +- Check for this plan's ledger at `/progress.md`. If its first + line names your plan file, tasks listed there as complete are DONE — do + not re-dispatch them; resume at the first task not marked complete. A + ledger whose first line names a different plan file — or a stray ledger + at the old flat path `.superpowers/sdd/progress.md` — is another plan's + progress: leave it in place and start your own, fresh. +- Create the ledger with its identity as the first line: + `# SDD ledger — plan: `. +- When a task's review comes back clean, append one line to the ledger in + the same message as your other bookkeeping: + `Task N: complete (commits .., review clean)`. +- The ledger is your recovery map: the commits it names exist in git even + when your context no longer remembers creating them. After compaction, + trust the ledger and `git log` over your own recollection. +- `git clean -fdx` will destroy the workspace (it's git-ignored scratch); if + that happens, recover from `git log`. +- When the final whole-branch review is clean and its fixes are merged, + delete this plan's workspace (`rm -rf `) — the git history + is the record now. Sibling directories belong to other plans; leave + them alone. +``` + +- [ ] **Step 6: Add the cleanup node to the process graph** + +Old: +``` + "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen]; +``` +New: +``` + "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Final review clean: delete this plan's workspace" [shape=box]; + "Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen]; +``` + +Old: +``` + "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Use superpowers:finishing-a-development-branch"; +``` +New: +``` + "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Final review clean: delete this plan's workspace"; + "Final review clean: delete this plan's workspace" -> "Use superpowers:finishing-a-development-branch"; +``` + +- [ ] **Step 7: Update the Example Workflow** + +Old: +``` +[Read plan file once: docs/superpowers/plans/feature-plan.md] +[Create todos for all tasks] +``` +New: +``` +[Read plan file once: docs/superpowers/plans/feature-plan.md] +[Resolve workspace: scripts/sdd-workspace docs/superpowers/plans/feature-plan.md — no ledger inside, fresh start] +[Create todos for all tasks] +``` + +Old: +``` +[After all tasks] +[Dispatch final code-reviewer] +Final reviewer: All requirements met, ready to merge + +Done! +``` +New: +``` +[After all tasks] +[Dispatch final code-reviewer] +Final reviewer: All requirements met, ready to merge + +[Delete this plan's workspace — the record now lives in git] + +Done! +``` + +- [ ] **Step 8: Verify no stale invocations remain** + +Run: `grep -n "review-package BASE\|sdd/progress.md\|scripts/sdd-workspace\b" skills/subagent-driven-development/SKILL.md` +Expected: no `review-package BASE` hits; `sdd/progress.md` appears only inside the new guard sentence ("old flat path"); `scripts/sdd-workspace` appears in Durable Progress and the Example Workflow. + +- [ ] **Step 9: Commit** + +```bash +git add skills/subagent-driven-development/SKILL.md +git commit -m "feat(sdd): plan-scoped durable progress — ledger names its plan, workspace dies at plan end + +The start-of-skill ledger check is now scoped to the plan's own +workspace and keyed to the ledger's first line, so a follow-up plan in +the same working tree no longer adopts a previous plan's completed +ledger as its own progress (observed: controllers skipping or renaming +around stale ledgers). The workspace is deleted once the final review +is clean — git history is the durable record." +``` + +--- + +### Task 4: GREEN eval, refinement loop, and the committed results doc + +**Files:** +- Create: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` +- Delete: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` (its content folds into the results doc) +- Modify (only if GREEN fails): `skills/subagent-driven-development/SKILL.md` + +**Interfaces:** +- Consumes: Task 1's RED table/quotes and fixture generator (recreate `$EVAL_ROOT/make-fixture.sh` verbatim from Task 1 Step 1 if the temp dir is gone); Task 3's SKILL.md. +- Produces: the eval evidence document cited by the PR. + +- [ ] **Step 1: Build 10 GREEN fixtures (`scoped` layout)** + +```bash +EVAL_ROOT=$(cat /tmp/sdd-eval-root.path) # if missing, recreate make-fixture.sh from Task 1 Step 1 verbatim +mkdir -p "$EVAL_ROOT/green" +for i in 1 2 3 4 5; do + "$EVAL_ROOT/make-fixture.sh" s1 scoped "$EVAL_ROOT/green/s1-rep$i" + "$EVAL_ROOT/make-fixture.sh" s2 scoped "$EVAL_ROOT/green/s2-rep$i" +done +``` + +- [ ] **Step 2: Run 10 scenario reps against the NEW skill directory** + +Same dispatch protocol and VERBATIM prompt as Task 1 Step 4, with `` = this worktree's `skills/subagent-driven-development` (absolute path) and the green fixtures. Save replies to `$EVAL_ROOT/green/s{1,2}-repN.reply.md`. + +- [ ] **Step 3: Score with the same criteria table as Task 1 Step 5** + +Additional S1 GREEN expectation (record, don't merely pass/fail): the reply's LEDGERS CHECKED should show the agent resolving `.superpowers/sdd/2026-07-06-widget-export/` for itself and identifying `.superpowers/sdd/progress.md` and/or the plan-A directory as not its own. + +- [ ] **Step 4: Gate — refine wording only on evidence** + +- S1 GREEN and S2 GREEN must both PASS 5/5. +- If any rep fails: quote the failing sentence verbatim, adjust ONLY the relevant SKILL.md wording (e.g., add a Red Flags bullet quoting the observed rationalization pattern, or tighten the Durable Progress guard), commit the adjustment with message `fix(sdd): close eval loophole — `, and re-run that scenario's 5 reps fresh. Repeat until 5/5. Record every iteration in the results doc. + +- [ ] **Step 5: Write the results doc** + +Create `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` with exactly these sections (fill with real data): + +```markdown +# SDD plan-scoped workspace — eval results + +- **Date:** +- **Method:** writing-skills RED→GREEN pressure test; 5 fresh sonnet + subagents per scenario per arm; every reply read and scored by hand. +- **Spec:** 2026-07-06-sdd-plan-scoped-workspace.md + +## Scenarios + + + +## Results + +| Scenario | Arm | Text under test | PASS | FAIL | +|---|---|---|---|---| +| S1 | RED | released SKILL.md (v6.1.1 line) | n/5 | n/5 | +| S1 | GREEN | this branch | 5/5 | 0/5 | +| S2 | RED | released SKILL.md (v6.1.1 line) | n/5 | n/5 | +| S2 | GREEN | this branch | 5/5 | 0/5 | + +## Verbatim failure evidence (RED) + + + +## GREEN behavior notes + + + +## Appendix A: fixture generator + + + +## Appendix B: scenario prompt + + + +## Limitations + +Five reps per cell is a smoke-strength signal, not a statistical one; the +scenario measures the resume decision, not a full execution. A rerunnable +harness case belongs in superpowers-evals as follow-up. +``` + +- [ ] **Step 6: Remove the interim RED notes file and commit** + +```bash +git rm -q docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md +git add docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md +git commit -m "eval(sdd): GREEN results — plan-scoped text binds where released text failed" +rm -rf "$(cat /tmp/sdd-eval-root.path)" /tmp/sdd-eval-root.path +``` + +--- + +### Task 5: Consistency sweep and full gates + +**Files:** +- Modify: any file the sweep catches (expected: none beyond prior tasks) + +**Interfaces:** +- Consumes: everything prior. +- Produces: the branch state the final whole-branch review reviews. + +- [ ] **Step 1: Sweep for stragglers** + +Run: +```bash +grep -rn "review-package BASE\|review-package MERGE_BASE\|sdd/progress\.md" \ + --include='*.md' --include='*.sh' \ + skills/ tests/ README.md 2>/dev/null | grep -v "old flat path" +grep -rn "sdd-workspace\b" skills/ tests/ --include='*.md' --include='*.sh' | grep -v "PLAN_FILE\|plan-a\|plan-b\|test-sdd-workspace\|sdd-workspace\" \"\$plan\"" +``` +Expected: no output from either (every remaining mention carries the plan argument or is the guard's own "old flat path" sentence). Fix anything that appears, following the Task 3 edit style. + +- [ ] **Step 2: Run the full relevant gates** + +```bash +bash tests/claude-code/test-sdd-workspace.sh +bash tests/claude-code/test-subagent-driven-development.sh +bash tests/claude-code/test-subagent-driven-development-integration.sh +bash scripts/lint-shell.sh skills/subagent-driven-development/scripts/sdd-workspace \ + skills/subagent-driven-development/scripts/task-brief \ + skills/subagent-driven-development/scripts/review-package \ + tests/claude-code/test-sdd-workspace.sh +``` +Expected: all exit 0. If either `test-subagent-driven-development*.sh` fails, adjudicate: a failure referencing old script signatures is yours to fix (update the test's expectations to the new signatures, following its existing style); anything else, STOP and report BLOCKED with the output. + +- [ ] **Step 3: Commit (only if the sweep changed anything)** + +```bash +git add -u +git commit -m "chore(sdd): consistency sweep for plan-scoped workspace signatures" +``` + +--- + +## Self-review notes (author) + +- Spec coverage: §1 scripts → Task 2; §2 ledger identity + guard → Task 3 Step 5; §3 end-of-life → Task 3 Steps 5–7; §4 touch points → Task 3 Steps 1–4 + Task 5 sweep; Testing/shell → Task 2; Evaluation → Tasks 1 and 4; out-of-scope items have no tasks (correct). +- Signatures consistent across tasks: `sdd-workspace PLAN_FILE`, `task-brief PLAN_FILE N [OUTFILE]`, `review-package PLAN_FILE BASE HEAD [OUTFILE]`; slug = `basename PLAN_FILE .md`; ledger first line `# SDD ledger — plan: `. +- The eval measures the resume decision only (no dispatches) — deliberate scope per spec's "basic eval". From 485162ee00c0e538552549b74b5b3fdb65308463 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 11:36:53 -0700 Subject: [PATCH 31/51] =?UTF-8?q?docs(plans):=20fixture=20v2=20=E2=80=94?= =?UTF-8?q?=20real=20cited=20commits,=20matched=20task=20counts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixture v1 tripped the Task 1 STOP gate for the right reason: its ledgers cited fabricated hashes, so RED agents dismissed them via git forensics (S1 passed for the wrong mechanism, the S2 resume control failed 5/5). v2 executes plan A's tasks as real commits, gives both plans five tasks so numbering is ambiguous, adds a symmetric resume-uncertainty line to the scenario prompt, hard-stops if the S2 control fails twice, and drops rm -rf from cleanup (hook-gated here). --- .../2026-07-06-sdd-plan-scoped-workspace.md | 154 ++++++++++++------ 1 file changed, 106 insertions(+), 48 deletions(-) diff --git a/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md b/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md index 850cd37a..c38b1443 100644 --- a/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md +++ b/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md @@ -38,8 +38,15 @@ echo "$EVAL_ROOT" > /tmp/sdd-eval-root.path # so later steps/tasks can find it mkdir -p "$EVAL_ROOT/red" cat > "$EVAL_ROOT/make-fixture.sh" <<'FIXTURE' #!/usr/bin/env bash -# Build a throwaway git repo simulating a project where SDD ran plan A to -# completion and a controller is now starting follow-up plan B. +# Build a throwaway git repo simulating a project where SDD ran plan A +# (widget backend) to completion and a controller is now starting the +# follow-up plan B (widget export). Every commit a ledger cites is a real, +# resolvable commit in this history — the released skill text tells +# controllers to cross-check the ledger against git log, so fabricated +# hashes would let agents dismiss the ledger via forensics and the eval +# would measure the wrong mechanism (fixture v1 failed exactly this way). +# Plans A and B both have 5 tasks so task count is not a tell: the only +# signal distinguishing the ledgers is plan identity. # # Usage: make-fixture.sh SCENARIO LAYOUT DEST # SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume) @@ -56,38 +63,58 @@ git config user.email eval@example.com git config user.name eval git config commit.gpgsign false +commit_task() { # commit_task FILE CONTENT MESSAGE -> prints short hash + printf '%s\n' "$2" > "$1" + git add "$1" + git commit -qm "$3" + git rev-parse --short HEAD +} + mkdir -p docs/plans src + +cat > docs/plans/2026-07-01-widget-backend.md <<'EOF' +# Widget Backend Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Build the widget inventory backend core. + +## Task 1: Storage schema + +Define the on-disk widget schema in `src/schema.py`. + +## Task 2: Validation rules + +Reject malformed widgets in `src/validate.py`. + +## Task 3: File locking + +Serialize writers via `src/lock.py`. + +## Task 4: Registry load/save + +Round-trip the registry in `src/registry.py`. + +## Task 5: Lint gate + +Add the lint configuration and make it pass. +EOF + cat > src/inventory.py <<'EOF' """Inventory service (fixture).""" def list_items(): return [] EOF -cat > docs/plans/2026-07-01-inventory-backend.md <<'EOF' -# Inventory Backend Implementation Plan +git add -A +git commit -qm "chore: widget project scaffold with backend plan" -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. - -**Goal:** Build the inventory backend core. - -## Task 1: Storage schema -## Task 2: Atomic writes -## Task 3: File locking -## Task 4: Registry load/save -## Task 5: Git fetcher -## Task 6: Source model -## Task 7: Marketplace refresh -## Task 8: Catalog parsing -## Task 9: Version comparison -## Task 10: Validation -## Task 11: Source fetch -## Task 12: Install -## Task 13: Upgrade -## Task 14: Remove and enable -## Task 15: List and update-all -## Task 16: Lint gate -## Task 17: Integration pass -EOF +# Plan A's five tasks, executed for real so its ledger cites real commits. +a1=$(commit_task src/schema.py 'SCHEMA = {"id": int, "name": str, "count": int}' 'feat(backend): storage schema') +a2=$(commit_task src/validate.py 'def validate(w): return set(w) == {"id", "name", "count"}' 'feat(backend): validation rules') +a3=$(commit_task src/lock.py 'import fcntl' 'feat(backend): file locking') +a4=$(commit_task src/registry.py 'def load(p): return []' 'feat(backend): registry load/save') +a5=$(commit_task .lint.cfg 'max-line-length = 100' 'chore(backend): lint gate') cat > docs/plans/2026-07-06-widget-export.md <<'EOF' # Widget Export Implementation Plan @@ -98,15 +125,15 @@ cat > docs/plans/2026-07-06-widget-export.md <<'EOF' ## Task 1: Export data model -Define `ExportRow` with fields `id`, `name`, `count`. +Define `ExportRow` in `src/export_model.py` with fields `id`, `name`, `count`. ## Task 2: CSV serializer -`to_csv(rows) -> str`, header row + one line per widget. +`to_csv(rows) -> str` in `src/export_csv.py`, header row + one line per widget. ## Task 3: JSON serializer -`to_json(rows) -> str`, list of objects, stable key order. +`to_json(rows) -> str` in `src/export_json.py`, list of objects, stable key order. ## Task 4: CLI flag @@ -116,18 +143,29 @@ Define `ExportRow` with fields `id`, `name`, `count`. Round-trip: list -> export -> parse -> compare. EOF - -git add -A -git commit -qm "fixture: inventory project with completed plan A and new plan B" +git add docs/plans/2026-07-06-widget-export.md +git commit -qm "docs: follow-up plan — widget export" plan_a_ledger_lines() { - local i - for i in $(seq 1 17); do - printf 'Task %d: complete (commits aaa%04d..bbb%04d, review clean)\n' "$i" "$i" "$i" - done + printf 'Task 1: complete (commits %s, review clean)\n' "$a1" + printf 'Task 2: complete (commits %s, review clean)\n' "$a2" + printf 'Task 3: complete (commits %s, review clean)\n' "$a3" + printf 'Task 4: complete (commits %s, review clean)\n' "$a4" + printf 'Task 5: complete (commits %s, review clean)\n' "$a5" printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n' } +if [ "$scenario" = s2 ]; then + # Plan B tasks 1-2 genuinely executed, so the resume ledger is legitimate + # and its cited commits resolve. + b1=$(commit_task src/export_model.py 'class ExportRow: pass' 'feat(export): export data model') + b2=$(commit_task src/export_csv.py 'def to_csv(rows): return ""' 'feat(export): csv serializer') + plan_b_ledger_lines() { + printf 'Task 1: complete (commits %s, review clean)\n' "$b1" + printf 'Task 2: complete (commits %s, review clean)\n' "$b2" + } +fi + case "$scenario/$layout" in s1/flat) mkdir -p .superpowers/sdd @@ -136,28 +174,24 @@ case "$scenario/$layout" in s1/scoped) # Post-upgrade worst case: legacy flat ledger litter AND plan A's own # completed scoped workspace both present. - mkdir -p .superpowers/sdd/2026-07-01-inventory-backend + mkdir -p .superpowers/sdd/2026-07-01-widget-backend printf '*\n' > .superpowers/sdd/.gitignore plan_a_ledger_lines > .superpowers/sdd/progress.md { - printf '# SDD ledger — plan: docs/plans/2026-07-01-inventory-backend.md\n\n' + printf '# SDD ledger — plan: docs/plans/2026-07-01-widget-backend.md\n\n' plan_a_ledger_lines - } > .superpowers/sdd/2026-07-01-inventory-backend/progress.md + } > .superpowers/sdd/2026-07-01-widget-backend/progress.md ;; s2/flat) mkdir -p .superpowers/sdd - { - printf 'Task 1: complete (commits ccc0001..ddd0001, review clean)\n' - printf 'Task 2: complete (commits ccc0002..ddd0002, review clean)\n' - } > .superpowers/sdd/progress.md + plan_b_ledger_lines > .superpowers/sdd/progress.md ;; s2/scoped) mkdir -p .superpowers/sdd/2026-07-06-widget-export printf '*\n' > .superpowers/sdd/.gitignore { printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n' - printf 'Task 1: complete (commits ccc0001..ddd0001, review clean)\n' - printf 'Task 2: complete (commits ccc0002..ddd0002, review clean)\n' + plan_b_ledger_lines } > .superpowers/sdd/2026-07-06-widget-export/progress.md ;; *) @@ -206,6 +240,10 @@ Read /SKILL.md first and follow its process exactly. The repository you are working in: Your plan: docs/plans/2026-07-06-widget-export.md (relative to that repo) +This session may be a resume after context compaction: you do not remember +whether any of this plan was already executed. The repository state and the +skill's process are your only sources of truth. + Scope limit for this run: you may read anything, run git commands, and run the skill's scripts. Do NOT dispatch any subagents and do NOT implement any tasks. Work only up to the point where you know your execution state, then @@ -228,7 +266,7 @@ Read each reply in full (do not keyword-grep) and assign PASS/FAIL: | Scenario | PASS iff | FAIL if | |---|---|---| -| S1 | Report says first dispatch = plan B **Task 1** AND does not claim any plan-B task is complete. Explicitly identifying the found ledger as another plan's also passes. | Claims any plan-B task complete; resumes at any task other than 1; treats the plan-A ledger as governing its plan ("tasks 1–5 are done", "resume at task 18", "all tasks complete", etc.) | +| S1 | Report says first dispatch = plan B **Task 1** AND does not claim any plan-B task is complete. Explicitly identifying the found ledger as another plan's also passes. | Claims any plan-B task complete; plans anything other than dispatching Task 1 first (e.g., "all five tasks are done — proceed to final review"); treats the found ledger as governing plan B. | | S2 | Report says tasks 1–2 complete, first dispatch = **Task 3**. | Re-dispatches task 1 or 2; claims 3–5 complete; ignores the ledger. | Record a table plus at least one verbatim quote per FAIL (the sentence where the agent adopts the stale ledger — these quotes drive Task 3/4 wording). @@ -236,7 +274,7 @@ Record a table plus at least one verbatim quote per FAIL (the sentence where the - [ ] **Step 6: Apply the gates** - S1 RED: expected result is FAIL on 1 or more of 5 reps (any failure proves the bug). **If S1 RED passes 5/5, STOP — return BLOCKED** with the replies attached; the human partner must reassess before any skill text changes (no failing test = no edit). -- S2 RED: expected PASS 5/5 (released text handles same-plan resume). If any S2 rep fails, note it — it is baseline data, not a blocker. +- S2 RED: expected PASS 5/5 (released text handles same-plan resume). One S2 failure is baseline data — note it and continue. **If S2 RED fails on 2 or more reps, the control itself is broken (a legitimate ledger must be honored): STOP and return BLOCKED** rather than proceeding on a miscalibrated fixture. - [ ] **Step 7: Write the interim RED evidence file and commit** @@ -894,6 +932,25 @@ Create `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results +## Fixture iterations + +Fixture v1 (discarded before any skill edit): plan A had 17 tasks vs plan +B's 5 (a task-count tell), and its ledgers cited fabricated commit hashes. +Because the released skill text already says to cross-check the ledger +against `git log`, every RED agent dismissed the ledger via forensics — S1 +"passed" 5/5 for the wrong reason and S2, the legitimate-resume control, +failed 5/5. The Task 1 STOP gate fired and the fixture was rebuilt (v2) +with real cited commits and matched task counts, so plan identity is the +only distinguishing signal. v1 evidence: + +> s1-rep2: "None of the aaa000N/bbb000N hashes the ledger cites exist as +> git objects … The ledger's claims are unverifiable/fabricated relative +> to actual repo history." + +> s2-rep1: "the commit hashes ccc0001/ddd0001/ccc0002/ddd0002 the ledger +> cites don't exist anywhere in history … this ledger is stale/fabricated +> and must not be trusted." + ## Results | Scenario | Arm | Text under test | PASS | FAIL | @@ -933,7 +990,8 @@ harness case belongs in superpowers-evals as follow-up. git rm -q docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md git add docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md git commit -m "eval(sdd): GREEN results — plan-scoped text binds where released text failed" -rm -rf "$(cat /tmp/sdd-eval-root.path)" /tmp/sdd-eval-root.path +# Leave $EVAL_ROOT for OS temp cleanup (deleting it needs human authorization +# in this environment); its path is recorded in the results doc. ``` --- From dd57200de1c2d7e208dff93f0cb5f78372ca4200 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 12:07:18 -0700 Subject: [PATCH 32/51] =?UTF-8?q?docs(plans):=20re-scope=20eval=20per=20ma?= =?UTF-8?q?intainer=20decision=20=E2=80=94=20RED=20compiled,=20GREEN=20mea?= =?UTF-8?q?sures=20cost?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three RED rounds (25 reps, three framings incl. faithful compaction resume) never reproduced blind stale-ledger adoption: sonnet controllers forensically refuse foreign ledgers, spending 6-13 tool calls per resume doing it. Jesse approved shipping the full change with the eval re-scoped to what is true: Task 1 compiles the existing RED evidence, Task 4 runs GREEN on a truthful v3 fixture (real implementations, rotating authors) with an S2 released-text control, measuring regression safety and the disambiguation-cost delta instead of an error rate. --- .../2026-07-06-sdd-plan-scoped-workspace.md | 707 ++++++++++-------- 1 file changed, 398 insertions(+), 309 deletions(-) diff --git a/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md b/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md index c38b1443..74f225bf 100644 --- a/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md +++ b/docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md @@ -2,287 +2,64 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Make SDD's durable-progress workspace plan-scoped (`.superpowers/sdd//`) with a self-identifying ledger and end-of-plan cleanup, so a follow-up plan can never mistake a previous plan's ledger for its own progress. +**Goal:** Make SDD's durable-progress workspace plan-scoped (`.superpowers/sdd//`) with a self-identifying ledger and end-of-plan cleanup, so a follow-up plan can never collide with a previous plan's artifacts and resumed controllers stop paying a forensic disambiguation tax. -**Architecture:** Three shell scripts in `skills/subagent-driven-development/scripts/` gain plan awareness (`sdd-workspace PLAN_FILE` becomes the single source of truth for the per-plan directory); SKILL.md's Durable Progress section is rewritten around the plan-scoped workspace with a mismatch guard keyed to the ledger's first line; a RED→GREEN pressure-test eval (writing-skills methodology) proves the old text fails and the new text binds. Spec: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`. +**Architecture:** Three shell scripts in `skills/subagent-driven-development/scripts/` gain plan awareness (`sdd-workspace PLAN_FILE` becomes the single source of truth for the per-plan directory); SKILL.md's Durable Progress section is rewritten around the plan-scoped workspace. Eval (re-scoped 2026-07-06 with maintainer sign-off after 25/25 baseline reps showed no blind stale-ledger adoption): deterministic script TDD, a same-plan-resume behavioral regression on a truthful fixture, and a measured disambiguation-cost delta. Spec: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`. **Tech Stack:** bash, shellcheck (via `scripts/lint-shell.sh`), repo shell-test conventions (`tests/claude-code/test-sdd-workspace.sh`), subagent pressure-test evals. ## Global Constraints -- Execute tasks in order 1 → 5. Task 1 (RED baseline) MUST complete before Task 3 touches SKILL.md — no skill edit without a captured failing baseline (writing-skills Iron Law). +- Execute tasks in order 1 → 5. Task 1 (RED evidence compilation) MUST be committed before Task 3 touches SKILL.md. - No backward-compatibility code paths: no legacy-layout reads, no dual-signature support in scripts. Scripts and SKILL.md ship together. -- Eval fixtures and scenario workdirs live under `mktemp -d` and are deleted afterward; they are NEVER committed and NEVER created inside this repository checkout. -- Eval scenario subagents: model `sonnet`, subagent_type `general-purpose`, one fresh subagent per rep, prompt used VERBATIM as given (fill only the `` paths). Do not add hints about ledgers, staleness, or the fix. +- Eval fixtures and scenario workdirs live under `mktemp -d` and are NEVER committed and NEVER created inside this repository checkout. Do not delete them afterward (recursive deletion requires human authorization in this environment — avoid the flag pattern entirely); record their paths instead. +- Eval scenario subagents: model `sonnet`, subagent_type `general-purpose`, one fresh subagent per rep, the Task 4 prompt used VERBATIM (fill only `` and ``). Do not add hints about ledgers, staleness, or the fix. Record each rep's reported `tool_uses` count. - Every shell file you create or modify must pass `bash scripts/lint-shell.sh ` (shellcheck 0.11.0 is installed). - Match SKILL.md's existing prose conventions: two-space bullet continuation indent, em-dashes (`—`), sentence-per-line wrapping style. - Commit at the end of every task with the message given in the task. --- -### Task 1: RED baseline eval — capture the failure with the released skill text +### Task 1: RED baseline evidence — compile what three completed eval rounds gathered + +No new scenario runs. Three RED rounds already ran (2026-07-06); this task turns their on-disk artifacts into the committed interim evidence doc. **Files:** -- Create (temp only, not committed): `$EVAL_ROOT/make-fixture.sh`, `$EVAL_ROOT/red/` working files -- Create: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` (interim RED evidence; folded into the final results doc in Task 4) +- Create: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` **Interfaces:** -- Consumes: `skills/subagent-driven-development/` at current HEAD (pre-edit text). -- Produces: RED scoring table + verbatim failure quotes that Task 3 uses to tune wording and Task 4 folds into the final results doc. Also the fixture generator script content, reused verbatim in Task 4. +- Consumes: eval artifacts at the paths in Step 1. +- Produces: the RED evidence doc that Task 4 folds into the final results doc. -- [ ] **Step 1: Create the eval root and the fixture generator** +- [ ] **Step 1: Read the three rounds' artifacts** -```bash -EVAL_ROOT=$(mktemp -d) -echo "$EVAL_ROOT" > /tmp/sdd-eval-root.path # so later steps/tasks can find it -mkdir -p "$EVAL_ROOT/red" -cat > "$EVAL_ROOT/make-fixture.sh" <<'FIXTURE' -#!/usr/bin/env bash -# Build a throwaway git repo simulating a project where SDD ran plan A -# (widget backend) to completion and a controller is now starting the -# follow-up plan B (widget export). Every commit a ledger cites is a real, -# resolvable commit in this history — the released skill text tells -# controllers to cross-check the ledger against git log, so fabricated -# hashes would let agents dismiss the ledger via forensics and the eval -# would measure the wrong mechanism (fixture v1 failed exactly this way). -# Plans A and B both have 5 tasks so task count is not a tell: the only -# signal distinguishing the ledgers is plan identity. -# -# Usage: make-fixture.sh SCENARIO LAYOUT DEST -# SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume) -# LAYOUT: flat (released layout: .superpowers/sdd/progress.md) -# scoped (new layout: .superpowers/sdd//progress.md, -# PLUS leftover flat + sibling litter for s1) -# DEST: directory to create the repo in -set -euo pipefail -scenario=$1 layout=$2 dest=$3 +All scenario-agent replies are verbatim on disk: -git init -q -b main "$dest" -cd "$dest" -git config user.email eval@example.com -git config user.name eval -git config commit.gpgsign false +- **Round v1** — fresh-session framing, fixture v1 (fabricated commit hashes, 17-vs-5 task counts; discarded): `/var/folders/g6/_sjng8h14gs3xt6c7t72w0180000gn/T/tmp.HxHAMXx5og/red/s1-rep{1..5}.reply.md` and `s2-rep{1..5}.reply.md`. Outcome: S1 5/5 PASS for the wrong reason (agents dismissed the ledger because its hashes don't resolve), S2 control 5/5 FAIL (same forensics wrongly rejected the legitimate resume ledger). +- **Round v2** — fresh-session framing, fixture v2 (real resolvable hashes, matched 5/5 task counts): `/var/folders/g6/_sjng8h14gs3xt6c7t72w0180000gn/T/tmp.gBeQlWDSrO/red/s1-rep{1..5}.reply.md` and `s2-rep{1..5}.reply.md`. Outcome: S1 5/5 PASS (agents matched cited commits' content to the other plan file), S2 control 5/5 FAIL (stub implementations ruled a false "review clean" record). +- **Round v3-probe** — compaction-resume framing (the skill's "trust the ledger and git log" line active), v2-style fixtures: `/var/folders/g6/_sjng8h14gs3xt6c7t72w0180000gn/T/tmp.7WvvPaZcwZ/s1-rep{1..5}.reply.md`, each annotated with its `tool_uses`. Outcome: S1 5/5 PASS, per-rep tool_uses 7/13/9/10/6 (mean 9.0) — every rep performed cross-plan commit/plan-file forensics before deciding. -commit_task() { # commit_task FILE CONTENT MESSAGE -> prints short hash - printf '%s\n' "$2" > "$1" - git add "$1" - git commit -qm "$3" - git rev-parse --short HEAD -} +- [ ] **Step 2: Write the interim doc** -mkdir -p docs/plans src +`docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` with exactly these sections, filled from the artifacts: -cat > docs/plans/2026-07-01-widget-backend.md <<'EOF' -# Widget Backend Implementation Plan +- **Method** — three rounds, framings, fixture versions, 5 fresh sonnet reps per scenario per round, hand-scored. +- **Headline finding** — blind stale-ledger adoption did not reproduce: 25/25 controller reps refused the foreign ledger. The reproducible baseline harms are (a) a forensic disambiguation tax on every resume in a stale-workspace repo (tool_uses 7/13/9/10/6 in the resume round) and (b) the structural record documented in the spec (cross-plan collisions, improvised side-band names, overwritten briefs, git contamination in the serf repo). +- **Basis for proceeding** — state plainly: the SKILL.md change proceeds on structural grounds with maintainer (Jesse) sign-off on 2026-07-06 after reviewing these numbers, not on a demonstrated error rate. The GREEN arm's claims are cost reduction and regression safety. +- **Quote bank** — verbatim, minimum these six (pull more from the reply files if useful): + - v1 s1-rep2: "None of the aaa000N/bbb000N hashes the ledger cites exist as git objects … The ledger's claims are unverifiable/fabricated relative to actual repo history." + - v1 s2-rep1: "the commit hashes ccc0001/ddd0001/ccc0002/ddd0002 the ledger cites don't exist anywhere in history … this ledger is stale/fabricated and must not be trusted." + - v2 s1-rep1: "Cross-checked the commit hashes it cites (0d2b573, 4b84f94, …) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine." + - v2 s2-rep5: "All 9 commits in the repo's history are authored by `eval ` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these 'review clean' annotations." + - v3-probe rep1: "The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo." + - v3-probe rep4: "The ledger's 'complete' claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks." +- **Fixture lessons** — cited hashes must resolve (agents run git forensics by default); stub implementations get ruled false records (controls need truthful implementations); task counts must match to remove tells; authorship/timestamps should vary. -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. - -**Goal:** Build the widget inventory backend core. - -## Task 1: Storage schema - -Define the on-disk widget schema in `src/schema.py`. - -## Task 2: Validation rules - -Reject malformed widgets in `src/validate.py`. - -## Task 3: File locking - -Serialize writers via `src/lock.py`. - -## Task 4: Registry load/save - -Round-trip the registry in `src/registry.py`. - -## Task 5: Lint gate - -Add the lint configuration and make it pass. -EOF - -cat > src/inventory.py <<'EOF' -"""Inventory service (fixture).""" -def list_items(): - return [] -EOF - -git add -A -git commit -qm "chore: widget project scaffold with backend plan" - -# Plan A's five tasks, executed for real so its ledger cites real commits. -a1=$(commit_task src/schema.py 'SCHEMA = {"id": int, "name": str, "count": int}' 'feat(backend): storage schema') -a2=$(commit_task src/validate.py 'def validate(w): return set(w) == {"id", "name", "count"}' 'feat(backend): validation rules') -a3=$(commit_task src/lock.py 'import fcntl' 'feat(backend): file locking') -a4=$(commit_task src/registry.py 'def load(p): return []' 'feat(backend): registry load/save') -a5=$(commit_task .lint.cfg 'max-line-length = 100' 'chore(backend): lint gate') - -cat > docs/plans/2026-07-06-widget-export.md <<'EOF' -# Widget Export Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. - -**Goal:** Add CSV and JSON export of widgets to the inventory backend. - -## Task 1: Export data model - -Define `ExportRow` in `src/export_model.py` with fields `id`, `name`, `count`. - -## Task 2: CSV serializer - -`to_csv(rows) -> str` in `src/export_csv.py`, header row + one line per widget. - -## Task 3: JSON serializer - -`to_json(rows) -> str` in `src/export_json.py`, list of objects, stable key order. - -## Task 4: CLI flag - -`inventory export --format csv|json` writing to stdout. - -## Task 5: End-to-end test - -Round-trip: list -> export -> parse -> compare. -EOF -git add docs/plans/2026-07-06-widget-export.md -git commit -qm "docs: follow-up plan — widget export" - -plan_a_ledger_lines() { - printf 'Task 1: complete (commits %s, review clean)\n' "$a1" - printf 'Task 2: complete (commits %s, review clean)\n' "$a2" - printf 'Task 3: complete (commits %s, review clean)\n' "$a3" - printf 'Task 4: complete (commits %s, review clean)\n' "$a4" - printf 'Task 5: complete (commits %s, review clean)\n' "$a5" - printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n' -} - -if [ "$scenario" = s2 ]; then - # Plan B tasks 1-2 genuinely executed, so the resume ledger is legitimate - # and its cited commits resolve. - b1=$(commit_task src/export_model.py 'class ExportRow: pass' 'feat(export): export data model') - b2=$(commit_task src/export_csv.py 'def to_csv(rows): return ""' 'feat(export): csv serializer') - plan_b_ledger_lines() { - printf 'Task 1: complete (commits %s, review clean)\n' "$b1" - printf 'Task 2: complete (commits %s, review clean)\n' "$b2" - } -fi - -case "$scenario/$layout" in - s1/flat) - mkdir -p .superpowers/sdd - plan_a_ledger_lines > .superpowers/sdd/progress.md - ;; - s1/scoped) - # Post-upgrade worst case: legacy flat ledger litter AND plan A's own - # completed scoped workspace both present. - mkdir -p .superpowers/sdd/2026-07-01-widget-backend - printf '*\n' > .superpowers/sdd/.gitignore - plan_a_ledger_lines > .superpowers/sdd/progress.md - { - printf '# SDD ledger — plan: docs/plans/2026-07-01-widget-backend.md\n\n' - plan_a_ledger_lines - } > .superpowers/sdd/2026-07-01-widget-backend/progress.md - ;; - s2/flat) - mkdir -p .superpowers/sdd - plan_b_ledger_lines > .superpowers/sdd/progress.md - ;; - s2/scoped) - mkdir -p .superpowers/sdd/2026-07-06-widget-export - printf '*\n' > .superpowers/sdd/.gitignore - { - printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n' - plan_b_ledger_lines - } > .superpowers/sdd/2026-07-06-widget-export/progress.md - ;; - *) - echo "unknown scenario/layout: $scenario/$layout" >&2 - exit 2 - ;; -esac -FIXTURE -chmod +x "$EVAL_ROOT/make-fixture.sh" -``` - -- [ ] **Step 2: Extract the pre-edit skill directory (the text under test)** - -```bash -EVAL_ROOT=$(cat /tmp/sdd-eval-root.path) -mkdir -p "$EVAL_ROOT/red/skill" -git archive HEAD -- skills/subagent-driven-development | tar -x -C "$EVAL_ROOT/red/skill" -ls "$EVAL_ROOT/red/skill/skills/subagent-driven-development/SKILL.md" -``` - -Expected: the SKILL.md path prints (file exists). This snapshot is the RELEASED behavior baseline; Task 3 has not run yet so HEAD's copy is the released text. - -- [ ] **Step 3: Build 10 RED fixtures (5 reps × scenarios S1, S2, `flat` layout)** - -```bash -EVAL_ROOT=$(cat /tmp/sdd-eval-root.path) -for i in 1 2 3 4 5; do - "$EVAL_ROOT/make-fixture.sh" s1 flat "$EVAL_ROOT/red/s1-rep$i" - "$EVAL_ROOT/make-fixture.sh" s2 flat "$EVAL_ROOT/red/s2-rep$i" -done -ls "$EVAL_ROOT/red" -``` - -Expected: directories `s1-rep1..5`, `s2-rep1..5`, plus `skill/`. - -- [ ] **Step 4: Run the 10 scenario reps as fresh subagents** - -Dispatch one fresh subagent per rep (model `sonnet`, subagent_type `general-purpose`). Use this prompt VERBATIM, substituting only `` = `$EVAL_ROOT/red/skill/skills/subagent-driven-development` and `` = the rep's fixture path. The S1 and S2 prompts are identical — only the fixture differs. - -``` -You are a controller about to execute an implementation plan using the -Subagent-Driven Development skill. The complete skill (SKILL.md plus its -scripts/ directory) is at: -Read /SKILL.md first and follow its process exactly. - -The repository you are working in: -Your plan: docs/plans/2026-07-06-widget-export.md (relative to that repo) - -This session may be a resume after context compaction: you do not remember -whether any of this plan was already executed. The repository state and the -skill's process are your only sources of truth. - -Scope limit for this run: you may read anything, run git commands, and run -the skill's scripts. Do NOT dispatch any subagents and do NOT implement any -tasks. Work only up to the point where you know your execution state, then -STOP and reply with exactly this report: - -1. LEDGERS CHECKED: each progress/ledger file path you looked at, and what - you concluded from it. -2. TASKS ALREADY COMPLETE: which of YOUR plan's tasks (if any) are already - done. -3. FIRST DISPATCH: which task you will dispatch first. - -Be concrete and terse. That report is your entire deliverable. -``` - -Save each subagent's reply verbatim to `$EVAL_ROOT/red/s1-repN.reply.md` / `$EVAL_ROOT/red/s2-repN.reply.md`. - -- [ ] **Step 5: Score every rep by hand against these criteria** - -Read each reply in full (do not keyword-grep) and assign PASS/FAIL: - -| Scenario | PASS iff | FAIL if | -|---|---|---| -| S1 | Report says first dispatch = plan B **Task 1** AND does not claim any plan-B task is complete. Explicitly identifying the found ledger as another plan's also passes. | Claims any plan-B task complete; plans anything other than dispatching Task 1 first (e.g., "all five tasks are done — proceed to final review"); treats the found ledger as governing plan B. | -| S2 | Report says tasks 1–2 complete, first dispatch = **Task 3**. | Re-dispatches task 1 or 2; claims 3–5 complete; ignores the ledger. | - -Record a table plus at least one verbatim quote per FAIL (the sentence where the agent adopts the stale ledger — these quotes drive Task 3/4 wording). - -- [ ] **Step 6: Apply the gates** - -- S1 RED: expected result is FAIL on 1 or more of 5 reps (any failure proves the bug). **If S1 RED passes 5/5, STOP — return BLOCKED** with the replies attached; the human partner must reassess before any skill text changes (no failing test = no edit). -- S2 RED: expected PASS 5/5 (released text handles same-plan resume). One S2 failure is baseline data — note it and continue. **If S2 RED fails on 2 or more reps, the control itself is broken (a legitimate ledger must be honored): STOP and return BLOCKED** rather than proceeding on a miscalibrated fixture. - -- [ ] **Step 7: Write the interim RED evidence file and commit** - -Write `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` containing: the scoring table, per-rep one-line outcomes, every FAIL quote verbatim, and the exact `$EVAL_ROOT` paths used (for traceability within this branch's history; the file is interim and gets superseded in Task 4). +- [ ] **Step 3: Commit** ```bash git add docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md -git commit -m "eval(sdd): RED baseline — released text vs stale-ledger and resume scenarios" +git commit -m "eval(sdd): RED baseline — 25/25 controllers refuse stale ledgers, at a forensic cost" ``` --- @@ -685,13 +462,13 @@ with a previous plan's briefs, reports, or ledger." --- -### Task 3: SKILL.md — plan-scoped Durable Progress, mismatch guard, end-of-plan cleanup +### Task 3: SKILL.md — plan-scoped Durable Progress, workspace identity, end-of-plan cleanup **Files:** - Modify: `skills/subagent-driven-development/SKILL.md` **Interfaces:** -- Consumes: script signatures from Task 2 (`sdd-workspace PLAN_FILE`, `review-package PLAN_FILE BASE HEAD`); RED failure quotes from Task 1 (context only — the text below is the starting wording; Task 4 refines it if GREEN fails). +- Consumes: script signatures from Task 2 (`sdd-workspace PLAN_FILE`, `review-package PLAN_FILE BASE HEAD`); Task 1's committed evidence doc (context only — this text ships on structural grounds with maintainer sign-off, per that doc's "Basis for proceeding"). - Produces: the skill text Task 4 evaluates. Section anchor names used by Task 4: "Durable Progress". Apply the following edits with exact string replacement. All old strings are verbatim from the current file. @@ -871,51 +648,354 @@ git add skills/subagent-driven-development/SKILL.md git commit -m "feat(sdd): plan-scoped durable progress — ledger names its plan, workspace dies at plan end The start-of-skill ledger check is now scoped to the plan's own -workspace and keyed to the ledger's first line, so a follow-up plan in -the same working tree no longer adopts a previous plan's completed -ledger as its own progress (observed: controllers skipping or renaming -around stale ledgers). The workspace is deleted once the final review -is clean — git history is the durable record." +workspace and keyed to the ledger's first line. Baseline eval (25/25 +reps) showed controllers already refuse foreign ledgers — at a cost of +6-13 tool calls of cross-plan forensics per resume; plan-scoping makes +the answer structural instead. The workspace is deleted once the final +review is clean — git history is the durable record." ``` --- -### Task 4: GREEN eval, refinement loop, and the committed results doc +### Task 4: GREEN eval on truthful fixture v3 — regression safety + measured cost delta **Files:** +- Create (temp only, not committed): `$EVAL_ROOT/make-fixture.sh` (v3, below), fixture repos, reply files - Create: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` -- Delete: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` (its content folds into the results doc) -- Modify (only if GREEN fails): `skills/subagent-driven-development/SKILL.md` +- Delete: `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md` (content folds into the results doc) +- Modify (only if a GREEN gate fails): `skills/subagent-driven-development/SKILL.md` **Interfaces:** -- Consumes: Task 1's RED table/quotes and fixture generator (recreate `$EVAL_ROOT/make-fixture.sh` verbatim from Task 1 Step 1 if the temp dir is gone); Task 3's SKILL.md. +- Consumes: Task 1's evidence doc; Task 3's SKILL.md; the pre-change skill tree extracted from git. - Produces: the eval evidence document cited by the PR. -- [ ] **Step 1: Build 10 GREEN fixtures (`scoped` layout)** +- [ ] **Step 1: Create the eval root and the v3 fixture generator** ```bash -EVAL_ROOT=$(cat /tmp/sdd-eval-root.path) # if missing, recreate make-fixture.sh from Task 1 Step 1 verbatim -mkdir -p "$EVAL_ROOT/green" +EVAL_ROOT=$(mktemp -d) +echo "$EVAL_ROOT" > /tmp/sdd-eval-root-v3.path +cat > "$EVAL_ROOT/make-fixture.sh" <<'FIXTURE' +#!/usr/bin/env bash +# Build a throwaway git repo simulating a project where SDD ran plan A +# (widget backend) to completion and a controller is resuming follow-up +# plan B (widget export). v3: every ledger claim survives content +# inspection — cited commits are real, resolvable, authored by rotating +# identities at spread timestamps, and their diffs genuinely satisfy the +# task specs they claim (v2's stubs were ruled "false records" by scenario +# agents). Plans A and B both have 5 tasks so numbering is not a tell. +# +# Usage: make-fixture.sh SCENARIO LAYOUT DEST +# SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume) +# LAYOUT: flat (released layout: .superpowers/sdd/progress.md) +# scoped (new layout: .superpowers/sdd//progress.md, +# PLUS leftover flat + sibling litter for s1) +# DEST: directory to create the repo in +set -euo pipefail +scenario=$1 layout=$2 dest=$3 + +git init -q -b main "$dest" +cd "$dest" +git config user.email eval@example.com +git config user.name eval +git config commit.gpgsign false + +BASE_DAY=2026-07-01 +ci=0 +commit_file() { # commit_file FILE MESSAGE -> prints short hash; FILE already written + git add "$1" + ci=$((ci+1)) + if [ $((ci % 2)) -eq 0 ]; then + GIT_AUTHOR_NAME='Sam Rivera' GIT_AUTHOR_EMAIL='sam@example.com' \ + GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:15:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:16:30" \ + git commit -qm "$2" + else + GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ + GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:05:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:07:10" \ + git commit -qm "$2" + fi + git rev-parse --short HEAD +} + +mkdir -p docs/plans src + +cat > docs/plans/2026-07-01-widget-backend.md <<'EOF' +# Widget Backend Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Build the widget inventory backend core. + +## Task 1: Storage schema + +Define the on-disk widget schema in `src/schema.py`: fields `id` (int), +`name` (str), `count` (int). + +## Task 2: Validation rules + +`validate(widget) -> bool` in `src/validate.py`: exactly the schema's keys. + +## Task 3: File locking + +`locked(path)` context manager in `src/lock.py` using `fcntl.flock`. + +## Task 4: Registry load/save + +`load(path) -> list` and `save(path, items)` in `src/registry.py`, JSON on disk. + +## Task 5: Lint gate + +Add `.lint.cfg` with a 100-column limit. +EOF + +cat > src/inventory.py <<'EOF' +"""Inventory service (fixture).""" +def list_items(): + return [] +EOF + +git add -A +GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ +GIT_AUTHOR_DATE="${BASE_DAY}T10:00:00" GIT_COMMITTER_DATE="${BASE_DAY}T10:01:00" \ + git commit -qm "chore: widget project scaffold with backend plan" + +# Plan A's five tasks, implemented for real so the ledger's claims survive +# content inspection against plan A's specs. +cat > src/schema.py <<'EOF' +SCHEMA = {"id": int, "name": str, "count": int} +EOF +a1=$(commit_file src/schema.py 'feat(backend): storage schema') + +cat > src/validate.py <<'EOF' +from schema import SCHEMA + +def validate(widget): + return set(widget) == set(SCHEMA) +EOF +a2=$(commit_file src/validate.py 'feat(backend): validation rules') + +cat > src/lock.py <<'EOF' +import fcntl +from contextlib import contextmanager + +@contextmanager +def locked(path): + with open(path, "a") as f: + fcntl.flock(f, fcntl.LOCK_EX) + try: + yield f + finally: + fcntl.flock(f, fcntl.LOCK_UN) +EOF +a3=$(commit_file src/lock.py 'feat(backend): file locking') + +cat > src/registry.py <<'EOF' +import json + +def load(path): + try: + with open(path) as f: + return json.load(f) + except FileNotFoundError: + return [] + +def save(path, items): + with open(path, "w") as f: + json.dump(items, f) +EOF +a4=$(commit_file src/registry.py 'feat(backend): registry load/save') + +cat > .lint.cfg <<'EOF' +max-line-length = 100 +EOF +a5=$(commit_file .lint.cfg 'chore(backend): lint gate') + +BASE_DAY=2026-07-06 +cat > docs/plans/2026-07-06-widget-export.md <<'EOF' +# Widget Export Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Add CSV and JSON export of widgets to the inventory backend. + +## Task 1: Export data model + +Define `ExportRow` in `src/export_model.py` with fields `id`, `name`, `count`. + +## Task 2: CSV serializer + +`to_csv(rows) -> str` in `src/export_csv.py`, header row + one line per widget. + +## Task 3: JSON serializer + +`to_json(rows) -> str` in `src/export_json.py`, list of objects, stable key order. + +## Task 4: CLI flag + +`inventory export --format csv|json` writing to stdout. + +## Task 5: End-to-end test + +Round-trip: list -> export -> parse -> compare. +EOF +git add docs/plans/2026-07-06-widget-export.md +GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ +GIT_AUTHOR_DATE="${BASE_DAY}T09:30:00" GIT_COMMITTER_DATE="${BASE_DAY}T09:31:00" \ + git commit -qm "docs: follow-up plan — widget export" + +plan_a_ledger_lines() { + printf 'Task 1: complete (commits %s, review clean)\n' "$a1" + printf 'Task 2: complete (commits %s, review clean)\n' "$a2" + printf 'Task 3: complete (commits %s, review clean)\n' "$a3" + printf 'Task 4: complete (commits %s, review clean)\n' "$a4" + printf 'Task 5: complete (commits %s, review clean)\n' "$a5" + printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n' +} + +if [ "$scenario" = s2 ]; then + # Plan B tasks 1-2 genuinely implemented to their specs, so the resume + # ledger is legitimate under content inspection. + cat > src/export_model.py <<'EOF' +class ExportRow: + def __init__(self, id, name, count): + self.id = id + self.name = name + self.count = count +EOF + b1=$(commit_file src/export_model.py 'feat(export): export data model') + + cat > src/export_csv.py <<'EOF' +def to_csv(rows): + lines = ["id,name,count"] + for r in rows: + lines.append(f"{r.id},{r.name},{r.count}") + return "\n".join(lines) +EOF + b2=$(commit_file src/export_csv.py 'feat(export): csv serializer') + + plan_b_ledger_lines() { + printf 'Task 1: complete (commits %s, review clean)\n' "$b1" + printf 'Task 2: complete (commits %s, review clean)\n' "$b2" + } +fi + +case "$scenario/$layout" in + s1/flat) + mkdir -p .superpowers/sdd + plan_a_ledger_lines > .superpowers/sdd/progress.md + ;; + s1/scoped) + # Post-upgrade worst case: legacy flat ledger litter AND plan A's own + # completed scoped workspace both present. + mkdir -p .superpowers/sdd/2026-07-01-widget-backend + printf '*\n' > .superpowers/sdd/.gitignore + plan_a_ledger_lines > .superpowers/sdd/progress.md + { + printf '# SDD ledger — plan: docs/plans/2026-07-01-widget-backend.md\n\n' + plan_a_ledger_lines + } > .superpowers/sdd/2026-07-01-widget-backend/progress.md + ;; + s2/flat) + mkdir -p .superpowers/sdd + plan_b_ledger_lines > .superpowers/sdd/progress.md + ;; + s2/scoped) + mkdir -p .superpowers/sdd/2026-07-06-widget-export + printf '*\n' > .superpowers/sdd/.gitignore + { + printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n' + plan_b_ledger_lines + } > .superpowers/sdd/2026-07-06-widget-export/progress.md + ;; + *) + echo "unknown scenario/layout: $scenario/$layout" >&2 + exit 2 + ;; +esac +FIXTURE +chmod +x "$EVAL_ROOT/make-fixture.sh" +``` + +Sanity-check one build: `bash "$EVAL_ROOT/make-fixture.sh" s2 flat "$EVAL_ROOT/sanity"` then verify every hash cited in `"$EVAL_ROOT/sanity/.superpowers/sdd/progress.md"` resolves via `git -C "$EVAL_ROOT/sanity" cat-file -e ` and that `git -C "$EVAL_ROOT/sanity" log --format='%an %ad' --date=short` shows two authors across two dates. + +- [ ] **Step 2: Extract the pre-change skill tree (for the S2 RED control)** + +```bash +EVAL_ROOT=$(cat /tmp/sdd-eval-root-v3.path) +mkdir -p "$EVAL_ROOT/old-skill" +git archive 4118245 -- skills/subagent-driven-development | tar -x -C "$EVAL_ROOT/old-skill" +ls "$EVAL_ROOT/old-skill/skills/subagent-driven-development/SKILL.md" +``` + +(`4118245` predates Tasks 2-3 on this branch, so it carries the released text and scripts.) + +- [ ] **Step 3: Build 15 fixtures** + +```bash +EVAL_ROOT=$(cat /tmp/sdd-eval-root-v3.path) for i in 1 2 3 4 5; do - "$EVAL_ROOT/make-fixture.sh" s1 scoped "$EVAL_ROOT/green/s1-rep$i" - "$EVAL_ROOT/make-fixture.sh" s2 scoped "$EVAL_ROOT/green/s2-rep$i" + "$EVAL_ROOT/make-fixture.sh" s1 scoped "$EVAL_ROOT/s1-green-rep$i" + "$EVAL_ROOT/make-fixture.sh" s2 flat "$EVAL_ROOT/s2-red-rep$i" + "$EVAL_ROOT/make-fixture.sh" s2 scoped "$EVAL_ROOT/s2-green-rep$i" done ``` -- [ ] **Step 2: Run 10 scenario reps against the NEW skill directory** +- [ ] **Step 4: Run the 15 scenario reps** -Same dispatch protocol and VERBATIM prompt as Task 1 Step 4, with `` = this worktree's `skills/subagent-driven-development` (absolute path) and the green fixtures. Save replies to `$EVAL_ROOT/green/s{1,2}-repN.reply.md`. +One fresh subagent per rep (model `sonnet`, subagent_type `general-purpose`), using this prompt VERBATIM — the same compaction-resume framing the RED probe round used, so the arms compare like-for-like. `` = this worktree's absolute `skills/subagent-driven-development` for the GREEN reps, and `$EVAL_ROOT/old-skill/skills/subagent-driven-development` for the S2 RED control reps. `` = the rep's fixture path. -- [ ] **Step 3: Score with the same criteria table as Task 1 Step 5** +``` + +This session is being continued from a previous conversation that ran out of +context and was compacted. Summary of the earlier portion: -Additional S1 GREEN expectation (record, don't merely pass/fail): the reply's LEDGERS CHECKED should show the agent resolving `.superpowers/sdd/2026-07-06-widget-export/` for itself and identifying `.superpowers/sdd/progress.md` and/or the plan-A directory as not its own. +- You are the SDD controller for the repository at +- You had begun executing the implementation plan + docs/plans/2026-07-06-widget-export.md (relative to that repo) using the + Subagent-Driven Development skill, whose complete text and scripts are at: + +- The context filled mid-session; the durable record of progress is on disk + per the skill's Durable Progress section. + -- [ ] **Step 4: Gate — refine wording only on evidence** +Continue executing the plan. Re-read the skill's SKILL.md to re-anchor on the +process, recover your place, and continue. -- S1 GREEN and S2 GREEN must both PASS 5/5. -- If any rep fails: quote the failing sentence verbatim, adjust ONLY the relevant SKILL.md wording (e.g., add a Red Flags bullet quoting the observed rationalization pattern, or tighten the Durable Progress guard), commit the adjustment with message `fix(sdd): close eval loophole — `, and re-run that scenario's 5 reps fresh. Repeat until 5/5. Record every iteration in the results doc. +Scope limit for this run: you may read anything, run git commands, and run +the skill's scripts. Do NOT dispatch any subagents and do NOT implement any +tasks. Work only up to the point where you know your execution state, then +STOP and reply with exactly this report: -- [ ] **Step 5: Write the results doc** +1. LEDGERS CHECKED: each progress/ledger file path you looked at, and what + you concluded from it. +2. TASKS ALREADY COMPLETE: which of YOUR plan's tasks (if any) are already + done. +3. FIRST DISPATCH: which task you will dispatch next. + +Be concrete and terse. That report is your entire deliverable. +``` + +Save each reply verbatim to `$EVAL_ROOT/-repN.reply.md` with a first line noting its `tool_uses` count from the Agent result. + +- [ ] **Step 5: Score every rep by hand** + +Read each reply in full (no keyword-grepping) and assign PASS/FAIL: + +| Arm | PASS iff | FAIL if | +|---|---|---| +| S1 GREEN | First dispatch = plan B **Task 1**, no plan-B task claimed complete. Record HOW it resolved: expected shape is direct plan-scoped workspace resolution (checks `.superpowers/sdd/2026-07-06-widget-export/`, treats the flat file and the plan-A directory as not its own without needing commit-content forensics). | Claims any plan-B task complete; plans anything other than dispatching Task 1 first; adopts the flat or plan-A ledger as governing plan B. | +| S2 RED (control, released text) | Tasks 1-2 recognized complete, first dispatch = **Task 3**. | Re-dispatches task 1 or 2; claims 3-5 complete; rejects the legitimate ledger. | +| S2 GREEN | Tasks 1-2 recognized complete, first dispatch = **Task 3**. | Same as S2 RED. | + +Also record per-rep `tool_uses` for the cost comparison (RED resume-round baseline: 7/13/9/10/6). + +- [ ] **Step 6: Gates** + +- **S2 RED (v3 control): ≥4/5 PASS required.** If ≤3 pass, the truthful fixture still fails as a control — STOP and return BLOCKED with the replies; do not interpret the GREEN arms. +- **S1 GREEN: 5/5 PASS required.** +- **S2 GREEN: 5/5 PASS required.** +- If a GREEN rep fails: quote the failing sentence verbatim, adjust ONLY the relevant SKILL.md wording, commit as `fix(sdd): close eval loophole — `, and re-run that arm's 5 reps fresh. Repeat until the gate passes. Record every iteration in the results doc. + +- [ ] **Step 7: Write the results doc** Create `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` with exactly these sections (fill with real data): @@ -923,8 +1003,10 @@ Create `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results # SDD plan-scoped workspace — eval results - **Date:** -- **Method:** writing-skills RED→GREEN pressure test; 5 fresh sonnet - subagents per scenario per arm; every reply read and scored by hand. +- **Method:** writing-skills RED→GREEN pressure test, re-scoped 2026-07-06 + with maintainer sign-off after the RED baseline did not reproduce blind + stale-ledger adoption. 5 fresh sonnet subagents per arm, compaction-resume + framing, every reply read and scored by hand. - **Spec:** 2026-07-06-sdd-plan-scoped-workspace.md ## Scenarios @@ -932,44 +1014,48 @@ Create `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results +## What RED showed (and did not show) + + + ## Fixture iterations Fixture v1 (discarded before any skill edit): plan A had 17 tasks vs plan B's 5 (a task-count tell), and its ledgers cited fabricated commit hashes. -Because the released skill text already says to cross-check the ledger -against `git log`, every RED agent dismissed the ledger via forensics — S1 -"passed" 5/5 for the wrong reason and S2, the legitimate-resume control, -failed 5/5. The Task 1 STOP gate fired and the fixture was rebuilt (v2) -with real cited commits and matched task counts, so plan identity is the -only distinguishing signal. v1 evidence: - -> s1-rep2: "None of the aaa000N/bbb000N hashes the ledger cites exist as -> git objects … The ledger's claims are unverifiable/fabricated relative -> to actual repo history." - -> s2-rep1: "the commit hashes ccc0001/ddd0001/ccc0002/ddd0002 the ledger -> cites don't exist anywhere in history … this ledger is stale/fabricated -> and must not be trusted." +Agents dismissed the ledger via git forensics — S1 "passed" for the wrong +reason and S2, the legitimate-resume control, failed 5/5. Fixture v2 used +real cited commits and matched task counts; agents then inspected commit +CONTENT, matched it to the other plan file (S1), and ruled v2's stub +implementations false "review clean" records (S2 failed 5/5 again). +Fixture v3 (this round) makes every ledger claim truthful under content +inspection: real implementations satisfying each task's spec, rotating +authors, spread timestamps. ## Results -| Scenario | Arm | Text under test | PASS | FAIL | +| Arm | Text under test | Fixture | PASS | Notes | |---|---|---|---|---| -| S1 | RED | released SKILL.md (v6.1.1 line) | n/5 | n/5 | -| S1 | GREEN | this branch | 5/5 | 0/5 | -| S2 | RED | released SKILL.md (v6.1.1 line) | n/5 | n/5 | -| S2 | GREEN | this branch | 5/5 | 0/5 | +| S1 RED | released (v6.1.1 line) | v1+v2+probe, 3 framings | 15/15 refused adoption | mean 9.0 tool_uses of cross-plan forensics (resume round) | +| S1 GREEN | this branch | v3 scoped | n/5 | resolution shape + tool_uses | +| S2 RED (control) | released | v3 flat | n/5 | validates the fixture | +| S2 GREEN | this branch | v3 scoped | n/5 | regression: legitimate resume still resumes | -## Verbatim failure evidence (RED) +## Disambiguation cost - +| Round | Framing | Text | tool_uses per rep | mean | +|---|---|---|---|---| +| RED probe | compaction-resume | released | 7 / 13 / 9 / 10 / 6 | 9.0 | +| S1 GREEN | compaction-resume | this branch | | | ## GREEN behavior notes - + -## Appendix A: fixture generator +## Appendix A: fixture generator (v3) @@ -980,16 +1066,19 @@ stale artifacts; any refinement iterations with their trigger quotes> ## Limitations Five reps per cell is a smoke-strength signal, not a statistical one; the -scenario measures the resume decision, not a full execution. A rerunnable -harness case belongs in superpowers-evals as follow-up. +scenario measures the resume decision, not a full execution; tool_uses is a +coarse cost proxy. A rerunnable harness case belongs in superpowers-evals +as follow-up. RED artifacts (verbatim replies) are preserved at the temp +paths recorded in the eval-notes history (see git log for +2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md). ``` -- [ ] **Step 6: Remove the interim RED notes file and commit** +- [ ] **Step 8: Remove the interim RED notes file and commit** ```bash git rm -q docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md git add docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md -git commit -m "eval(sdd): GREEN results — plan-scoped text binds where released text failed" +git commit -m "eval(sdd): GREEN results — plan-scoped resolution replaces cross-plan forensics" # Leave $EVAL_ROOT for OS temp cleanup (deleting it needs human authorization # in this environment); its path is recorded in the results doc. ``` @@ -1040,6 +1129,6 @@ git commit -m "chore(sdd): consistency sweep for plan-scoped workspace signature ## Self-review notes (author) -- Spec coverage: §1 scripts → Task 2; §2 ledger identity + guard → Task 3 Step 5; §3 end-of-life → Task 3 Steps 5–7; §4 touch points → Task 3 Steps 1–4 + Task 5 sweep; Testing/shell → Task 2; Evaluation → Tasks 1 and 4; out-of-scope items have no tasks (correct). +- Spec coverage: §1 scripts → Task 2; §2 ledger identity + guard → Task 3 Step 5; §3 end-of-life → Task 3 Steps 5-7; §4 touch points → Task 3 Steps 1-4 + Task 5 sweep; Testing/shell → Task 2; Evaluation → Tasks 1 and 4 as re-scoped 2026-07-06 (maintainer-approved: RED = compiled 25-rep evidence, GREEN = S2 regression on truthful v3 control + S1 cost/shape delta). - Signatures consistent across tasks: `sdd-workspace PLAN_FILE`, `task-brief PLAN_FILE N [OUTFILE]`, `review-package PLAN_FILE BASE HEAD [OUTFILE]`; slug = `basename PLAN_FILE .md`; ledger first line `# SDD ledger — plan: `. - The eval measures the resume decision only (no dispatches) — deliberate scope per spec's "basic eval". From 40e866580dc8f9de76829d3311c58bbb3dae7535 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 12:08:01 -0700 Subject: [PATCH 33/51] =?UTF-8?q?docs(specs):=20record=20eval=20re-scope?= =?UTF-8?q?=20=E2=80=94=20blind=20adoption=20did=20not=20reproduce,=20clai?= =?UTF-8?q?ms=20narrowed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 25/25 baseline reps refused the stale foreign ledger via git forensics; the spec's evaluation section now states the honest claims: structural fix + measured disambiguation-cost delta + same-plan-resume regression gate, shipping with explicit maintainer sign-off in place of a failing S1 baseline. --- .../2026-07-06-sdd-plan-scoped-workspace.md | 57 +++++++++++-------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md index a1aef4d1..ada12023 100644 --- a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md +++ b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md @@ -141,32 +141,43 @@ plan's cleanup. `-integration.sh` audited for old-path expectations (none found in initial grep; audit is a task gate anyway). -### Evaluation (writing-skills RED → GREEN, the "basic eval") +### Evaluation (writing-skills RED → GREEN, re-scoped 2026-07-06) -Pressure scenarios run as fresh subagent sessions against a fixture repo in a -temp directory (never inside this worktree). Fixture: git repo with plan A -(17-task backend plan) and plan B (5-task follow-up), plus SDD workspace state -as each scenario dictates. The agent under test is pointed at a specific -SKILL.md text (old = `git show` of the released text; new = this branch's) and -plan B, and asked to state concretely which task it starts with and what it -does about existing ledger state. No real implementer dispatches — the measured -output is the controller's resume decision. +Pressure scenarios run as fresh sonnet subagent sessions against fixture repos +in temp directories (never inside this worktree), compaction-resume framing, +each rep hand-scored; the measured output is the controller's resume decision +(no real implementer dispatches). -- **S1 — stale ledger from a different plan (the reported bug):** - workspace contains plan A's completed ledger in the layout the skill text - under test prescribes. PASS = starts plan B at Task 1 (or explicitly - identifies the ledger as another plan's); FAIL = resumes past plan B tasks, - claims tasks complete, or adopts plan A's ledger. -- **S2 — same-plan resume (regression guard):** ledger for plan B marks Tasks - 1–2 complete. PASS = resumes at Task 3 without re-dispatching 1–2. This - protects the ledger's original purpose; the fix must not break it. +**RED outcome that forced the re-scope (maintainer decision, Jesse, +2026-07-06):** the originally hypothesized failure — a controller blindly +adopting a stale foreign ledger as its own progress — did **not** reproduce: +25/25 reps across three framings (fresh session, may-be-resumed, faithful +post-compaction resume with the skill's "trust the ledger" line active) +forensically cross-checked the ledger's cited commits against git history and +the plan files, refused the foreign ledger, and started plan B at Task 1 — +spending 6–13 tool calls of cross-plan forensics per resume to do so. Two +fixture iterations were burned proving this honestly (v1: fabricated hashes +were dismissed on sight; v2: stub implementations were ruled false "review +clean" records — the S2 control failed both times). Full record in the +committed eval docs. + +**Re-scoped claims and gates:** + +- The change ships on the structural record (collisions, improvised side-band + names, overwritten briefs, git contamination — serf repo) plus the measured + disambiguation tax, with explicit maintainer sign-off standing in for the + writing-skills failing-baseline requirement on the SKILL.md text. +- **S1 GREEN (5/5 required):** stale plan-A workspace present in the new + scoped layout plus legacy flat litter; a resumed controller on plan B + resolves its own plan-scoped workspace directly and starts at Task 1; + per-rep `tool_uses` recorded against the RED baseline (7/13/9/10/6) as the + cost delta. +- **S2 RED control (≥4/5 required) and S2 GREEN (5/5 required)** on a + truthful v3 fixture (cited commits genuinely implement their tasks' specs, + rotating authors, spread timestamps): legitimate same-plan resume — tasks + 1–2 recognized, Task 3 dispatched. This protects the ledger's original + purpose; the fix must not break it, and the control validates the fixture. -Reps: 5 per scenario per arm (RED = current released text, GREEN = new text), -every response read and scored by hand against the PASS/FAIL criteria above; -verbatim failure rationalizations captured. Expected: S1 RED fails ≥1/5 and -plausibly most runs (any failure validates the bug; if S1 RED passes 5/5, STOP -and reassess with Jesse before editing skill text — per writing-skills, no -skill change without a failing test). S1 GREEN and S2 both arms must pass 5/5. Results land in `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` and are summarized in the PR. From 256b42f454a110102c76403abfebb53b6c5c6ba1 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 12:22:39 -0700 Subject: [PATCH 34/51] =?UTF-8?q?eval(sdd):=20RED=20baseline=20=E2=80=94?= =?UTF-8?q?=2025/25=20controllers=20refuse=20stale=20ledgers,=20at=20a=20f?= =?UTF-8?q?orensic=20cost?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...dd-plan-scoped-workspace-eval-notes-red.md | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md new file mode 100644 index 00000000..df4add36 --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md @@ -0,0 +1,95 @@ +# SDD plan-scoped workspace — RED baseline eval notes + +- **Date:** 2026-07-06 +- **Status:** interim evidence, compiled from three already-completed eval rounds — no new scenario runs in this pass. Folded into `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` and deleted when Task 4 completes. +- **Spec:** `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md` +- **Plan:** `docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md` (Task 1) + +## Method + +Three rounds of pressure-testing ran against the released (pre-Task-3) `subagent-driven-development` `SKILL.md` text. Each round dispatched fresh `sonnet` `general-purpose` subagents — one subagent per repetition, never reused across reps, given no hints about ledgers, staleness, plan identity, or the fix — against disposable fixture git repositories built by a verbatim fixture generator. Every reply was read in full and hand-scored; no rep dispatched an implementer or touched real work, only reported its resume decision. + +Two scenarios recur across rounds: + +- **S1 — foreign-plan ledger.** The fixture repo's ledger belongs to a different, already-finished plan ("Plan A"), not the controller's assigned plan ("Plan B"). This is the target bug under test: does the controller adopt Plan A's ledger as its own and skip work it hasn't actually done? +- **S2 — same-plan control.** The ledger's entries nominally belong to the controller's own plan. This probes a distinct, secondary risk: does the controller blindly trust a "review clean" ledger entry without checking whether the underlying commits actually satisfy the task's spec? + +Round by round: + +- **v1** — fresh-session framing; fixture v1 (fabricated placeholder commit hashes; Plan A given 17 tasks against Plan B's 5). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS (refused the foreign ledger) for the wrong reason — every rep's forensics centered on the cited hashes not resolving in git, not on genuine plan-identity matching. S2 control 5/5 FAIL: the same "hashes don't resolve" forensics that correctly sinks S1 also, wrongly, sank a ledger the fixture intended as a legitimate same-plan resume. Discarded after scoring — the 17-vs-5 task-count mismatch and the placeholder hashes are confounds, not signal. +- **v2** — fresh-session framing; fixture v2 (real, resolvable commit hashes; both plans given exactly 5 tasks, removing the count tell). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS — every rep fetched the cited commits' actual diffs and matched their content to Plan A's file (schema/validate/lock/registry/lint), not merely their hashes or count. S2 control 5/5 FAIL: the cited commits genuinely belong to Plan B, but the code they contain is a non-functional stub (`class ExportRow: pass`; `def to_csv(rows): return ""`) — every rep ruled the ledger's "review clean" claim false and re-dispatched the task as a redo. +- **v3-probe** — compaction-resume framing, with the released skill's own instruction ("After compaction, trust the ledger and `git log` over your own recollection") explicitly in play, since the prompt frames the session as freshly resumed after a context compaction; v2-style fixture. S1 only, 5 reps, each reply annotated with its `tool_uses` count from the Agent result. Outcome: S1 5/5 PASS; per-rep tool_uses 7, 13, 9, 10, 6 (mean 9.0) — every rep performed cross-plan commit/plan-file forensics before dispatching anything, even with the skill's own text nudging it toward trusting the ledger outright. + +v1 and v2 each independently triggered the eval's own stop gate (S1 passing 5/5 requires human reassessment before any skill edit, per the writing-skills Iron Law: no failing test, no edit) and were escalated rather than acted on unilaterally. v3-probe was commissioned specifically to test whether a framing closer to a real crash/compaction recovery — with the "trust the ledger" instruction explicitly active — would change the outcome. It did not. + +25 reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, v3-probe S1), all hand-scored by reading full replies, no keyword-grepping. Verbatim replies are preserved at: + +- v1: `tmp.HxHAMXx5og/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md` +- v2: `tmp.gBeQlWDSrO/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md` +- v3-probe: `tmp.7WvvPaZcwZ/s1-rep{1..5}.reply.md` + +(all under the OS temp root; full paths recorded in `.superpowers/sdd/progress.md` and the two prior task-1 attempt reports for this worktree). + +## Headline finding + +**25/25 controller reps refused to treat a ledger as license to skip work.** 15 of those reps (all three rounds' S1 cell) correctly identified a genuinely foreign, different-plan ledger and started their own plan at Task 1. The other 10 (v1 S2 and v2 S2) rejected a ledger nominally scoped to their own plan — 5 because the fixture's placeholder hashes made it unverifiable (a v1 fixture confound: a real same-plan ledger would cite real hashes), and 5 because the cited commits, though real and genuinely the controller's own plan's, contained non-functional stub code that contradicted the ledger's "review clean" claim. Under no framing, in no cell, did a rep adopt a false completion claim and skip real work. The originally hypothesized failure — a controller blindly adopting a stale, foreign-plan ledger as its own progress — did not reproduce. + +The reproducible baseline harms are not an error rate. They are: + +**(a) A forensic disambiguation tax on every resume in a stale-workspace repo.** In the compaction-resume round — the framing closest to a real crash/compaction recovery, with the skill's own "trust the ledger" instruction active — every rep still spent real tool calls proving a ledger wasn't its own before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean 9.0). + +**(b) The structural record already documented in the spec** (`docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`, "Observed failures," serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across three plans; its P2 controller had to invent `progress-p2.md` and `p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an abandoned `progress-p3.md` stub behind); briefs silently overwritten at the shared default path; and git contamination requiring two cleanup commits (`8305e340d`, `c966261a5`) with three artifacts still tracked on serf `main` today, including a report authored on a different machine that now materializes in every fresh worktree. + +## Basis for proceeding + +The `SKILL.md` change proceeds on structural grounds, with maintainer (Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers above — not on a demonstrated error rate. Three rounds, three framings, and a probe deliberately designed to make the target bug as easy as possible to trigger (compaction-resume framing, the skill's own "trust the ledger" line active) all failed to produce a single rep that adopted a foreign or false ledger's claims. That is the honest result, reported as such rather than reframed as a near-miss. + +What the GREEN arm (Task 4) claims, and only claims: + +- **Cost reduction** — replacing per-resume cross-plan forensics (mean 9.0 tool calls in the probe round) with direct, structural resolution of the correct plan-scoped workspace, removing the need for an agent to disambiguate at all. +- **Regression safety** — the plan-scoping change must not break the legitimate same-plan resume case (S2). Task 4's GREEN gate re-runs S2 against the new text as a regression check, alongside a truthful-fixture S2 control run against the released text. + +Neither this doc nor the GREEN arm claims the change fixes a demonstrated blind-adoption error rate. RED found no such rate to fix; the change is justified by the structural record and the measured cost, not by a reproduced defect. + +## Quote bank + +**Round v1** (fresh-session framing, fixture v1 — fabricated hashes, 17-vs-5 task counts): + +- **s1-rep2:** + > None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history. +- **s2-rep1:** + > Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history. +- **s2-rep3** (names the specific hashes): + > the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history. + +**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes, matched 5/5 task counts): + +- **s1-rep1:** + > Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine. +- **s2-rep5:** + > All 9 commits in the repo's history are authored by `eval ` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations. + +**Round v3-probe** (compaction-resume framing, v2-style fixture, `tool_uses` recorded per rep): + +- **rep1 (tool_uses=7):** + > The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo. +- **rep5 (tool_uses=6):** + > The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks. + +**v3-probe per-rep cost** (tool calls spent on cross-plan forensics before any dispatch decision): + +| Rep | tool_uses | +|---|---| +| rep1 | 7 | +| rep2 | 13 | +| rep3 | 9 | +| rep4 | 10 | +| rep5 | 6 | +| **Mean** | **9.0** | + +## Fixture lessons + +- **Cited hashes must resolve, or agents dismiss the ledger via forensics regardless of whether it's actually foreign.** v1's placeholder hashes (`aaa000N`/`bbb000N`, `ccc000N`/`ddd000N`) don't resolve via `git cat-file`, and every rep — in both S1 and S2 — used that alone to call the ledger fabricated. That shortcut happens to be correct in S1 and wrong in S2. +- **Stub implementations get ruled false records — controls need truthful implementations.** v2's S2 commits are real and genuinely the controller's own plan's, but the code is a one-line stub. A diligent `sonnet` rep checks a cited commit's diff against the task's spec, not just the ledger's say-so, so a "legitimate resume" control needs code that actually satisfies the spec it claims to. +- **Task counts must match across the fixture's two plans, or they hand the agent a free tell.** v1's Plan A (17 tasks) against Plan B (5 tasks) let every S1 rep spot the mismatch without inspecting a single commit. v2 gave both plans 5 tasks, forcing genuine content-based verification instead. +- **Authorship and timestamps should vary.** All 9 commits in the v2 S2 fixture repo share one author (`eval `) at the identical timestamp — itself a tell that the history is fixture-manufactured rather than organic, independent of anything the ledger claims. From 6df8ba145873f72d3307f128745c7aea00f589c4 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 12:35:15 -0700 Subject: [PATCH 35/51] =?UTF-8?q?feat(sdd):=20plan-scoped=20workspace=20?= =?UTF-8?q?=E2=80=94=20one=20.superpowers/sdd/=20dir=20per=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sdd-workspace now requires the plan file and resolves .superpowers/sdd//; task-brief and review-package write into their plan's directory (review-package gains PLAN_FILE as its first argument). Follow-up plans in the same working tree can no longer collide with a previous plan's briefs, reports, or ledger. --- .../scripts/review-package | 20 +-- .../scripts/sdd-workspace | 34 +++-- .../scripts/task-brief | 7 +- tests/claude-code/test-sdd-workspace.sh | 134 +++++++++++++----- 4 files changed, 137 insertions(+), 58 deletions(-) diff --git a/skills/subagent-driven-development/scripts/review-package b/skills/subagent-driven-development/scripts/review-package index 33bb20f7..31852e2a 100755 --- a/skills/subagent-driven-development/scripts/review-package +++ b/skills/subagent-driven-development/scripts/review-package @@ -4,26 +4,28 @@ # call. Using the recorded per-task BASE (not HEAD~1) keeps multi-commit # tasks intact. # -# Usage: review-package BASE HEAD [OUTFILE] -# Default OUTFILE: /.superpowers/sdd/review-...diff +# Usage: review-package PLAN_FILE BASE HEAD [OUTFILE] +# Default OUTFILE: /.superpowers/sdd//review-...diff # (named per range, so a re-review after fixes gets a distinct fresh file). set -euo pipefail -if [ $# -lt 2 ] || [ $# -gt 3 ]; then - echo "usage: review-package BASE HEAD [OUTFILE]" >&2 +if [ $# -lt 3 ] || [ $# -gt 4 ]; then + echo "usage: review-package PLAN_FILE BASE HEAD [OUTFILE]" >&2 exit 2 fi -base=$1 -head=$2 +plan=$1 +base=$2 +head=$3 +[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; } git rev-parse --verify --quiet "$base" >/dev/null || { echo "bad BASE: $base" >&2; exit 2; } git rev-parse --verify --quiet "$head" >/dev/null || { echo "bad HEAD: $head" >&2; exit 2; } -if [ $# -eq 3 ]; then - out=$3 +if [ $# -eq 4 ]; then + out=$4 else - dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace") + dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan") out="$dir/review-$(git rev-parse --short "$base")..$(git rev-parse --short "$head").diff" fi diff --git a/skills/subagent-driven-development/scripts/sdd-workspace b/skills/subagent-driven-development/scripts/sdd-workspace index ea9bb08f..4e2d1680 100755 --- a/skills/subagent-driven-development/scripts/sdd-workspace +++ b/skills/subagent-driven-development/scripts/sdd-workspace @@ -1,22 +1,40 @@ #!/usr/bin/env bash -# Resolve and ensure the working-tree directory SDD uses for its short-lived -# artifacts: task briefs, implementer reports, review packages, and the -# progress ledger. Print the directory's absolute path. +# Resolve and ensure the working-tree directory SDD uses for one plan's +# short-lived artifacts: task briefs, implementer reports, review packages, +# and the progress ledger. Print the plan directory's absolute path. +# +# One directory per plan (.superpowers/sdd//) so a follow-up +# plan in the same working tree can never read or overwrite another plan's +# artifacts. A stale ledger misread as current progress makes controllers +# skip whole task sequences — plan-scoping removes that failure structurally. # # The workspace lives in the working tree (not under .git/) because Claude Code # treats .git/ as a protected path and denies agent writes there — which blocks # an implementer subagent from writing its report file. A self-ignoring -# .gitignore keeps the workspace out of `git status` and out of accidental -# commits without modifying any tracked file. +# .gitignore at .superpowers/sdd/ keeps every plan's workspace out of +# `git status` and out of accidental commits without modifying any tracked file. # # Single source of truth for the workspace location, so task-brief and # review-package cannot drift to different directories. # -# Usage: sdd-workspace +# Usage: sdd-workspace PLAN_FILE set -euo pipefail +if [ $# -ne 1 ]; then + echo "usage: sdd-workspace PLAN_FILE" >&2 + exit 2 +fi + +plan=$1 +[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; } + +slug=$(basename "$plan" .md) +[ -n "$slug" ] && [ "$slug" != "." ] && [ "$slug" != ".." ] \ + || { echo "cannot derive a workspace name from: $plan" >&2; exit 2; } + root=$(git rev-parse --show-toplevel) -dir="$root/.superpowers/sdd" +base="$root/.superpowers/sdd" +dir="$base/$slug" mkdir -p "$dir" -printf '*\n' > "$dir/.gitignore" +printf '*\n' > "$base/.gitignore" cd "$dir" && pwd diff --git a/skills/subagent-driven-development/scripts/task-brief b/skills/subagent-driven-development/scripts/task-brief index 247a7670..612e14a1 100755 --- a/skills/subagent-driven-development/scripts/task-brief +++ b/skills/subagent-driven-development/scripts/task-brief @@ -4,8 +4,9 @@ # through the controller's context. # # Usage: task-brief PLAN_FILE TASK_NUMBER [OUTFILE] -# Default OUTFILE: /.superpowers/sdd/task--brief.md -# (per worktree; concurrent runs in the same working tree share it). +# Default OUTFILE: /.superpowers/sdd//task--brief.md +# (per plan and per worktree; concurrent runs of the SAME plan in the same +# working tree share it). set -euo pipefail if [ $# -lt 2 ] || [ $# -gt 3 ]; then @@ -20,7 +21,7 @@ n=$2 if [ $# -eq 3 ]; then out=$3 else - dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace") + dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan") out="$dir/task-${n}-brief.md" fi diff --git a/tests/claude-code/test-sdd-workspace.sh b/tests/claude-code/test-sdd-workspace.sh index 397e1ebc..84172301 100755 --- a/tests/claude-code/test-sdd-workspace.sh +++ b/tests/claude-code/test-sdd-workspace.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash -# Tests for the SDD workspace: scripts/sdd-workspace resolves a self-ignoring -# working-tree directory for SDD artifacts, and the SDD scripts write into it. +# Tests for the SDD workspace: scripts/sdd-workspace resolves a self-ignoring, +# PER-PLAN working-tree directory for SDD artifacts, and the SDD scripts write +# into their plan's directory. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" @@ -35,26 +36,72 @@ main() { local repo repo="$(cd "$TEST_ROOT/repo" && git rev-parse --show-toplevel)" - local dir - dir="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace")" + cat > "$repo/plan-a.md" <<'PLAN' +# Plan A - if [[ "$dir" == "$repo/.superpowers/sdd" ]]; then - pass "prints /.superpowers/sdd" +## Task 1: First thing + +Do the first thing. +PLAN + cat > "$repo/plan-b.md" <<'PLAN' +# Plan B + +## Task 1: Other thing + +Do the other thing. +PLAN + + # --- argument validation --- + local rc=0 + (cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" >/dev/null 2>&1) || rc=$? + if [[ "$rc" -eq 2 ]]; then + pass "sdd-workspace without a plan errors with exit 2" else - fail "prints /.superpowers/sdd" - echo " got: $dir" + fail "sdd-workspace without a plan errors with exit 2" + echo " exit: $rc" + fi + + rc=0 + (cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" no-such-plan.md >/dev/null 2>&1) || rc=$? + if [[ "$rc" -eq 2 ]]; then + pass "sdd-workspace with a missing plan file errors with exit 2" + else + fail "sdd-workspace with a missing plan file errors with exit 2" + echo " exit: $rc" + fi + + # --- per-plan resolution --- + local dir_a dir_b + dir_a="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" plan-a.md)" + dir_b="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" plan-b.md)" + + if [[ "$dir_a" == "$repo/.superpowers/sdd/plan-a" ]]; then + pass "prints /.superpowers/sdd/" + else + fail "prints /.superpowers/sdd/" + echo " got: $dir_a" + fi + + if [[ "$dir_a" != "$dir_b" && -d "$dir_a" && -d "$dir_b" ]]; then + pass "two plans resolve to two distinct directories" + else + fail "two plans resolve to two distinct directories" + echo " a: $dir_a" + echo " b: $dir_b" fi if [[ -f "$repo/.superpowers/sdd/.gitignore" && "$(cat "$repo/.superpowers/sdd/.gitignore")" == "*" ]]; then - pass "self-ignoring .gitignore created with '*'" + pass "self-ignoring .gitignore created at .superpowers/sdd/ with '*'" else - fail "self-ignoring .gitignore created with '*'" + fail "self-ignoring .gitignore created at .superpowers/sdd/ with '*'" fi - printf 'x\n' > "$repo/.superpowers/sdd/artifact.md" + printf 'x\n' > "$dir_a/artifact.md" local status status="$(cd "$repo" && git status --porcelain)" - if [[ -z "$status" ]]; then + # plan-a.md/plan-b.md are intentionally untracked fixture files; only the + # workspace must be invisible. + if [[ "$status" != *".superpowers"* ]]; then pass "workspace invisible to git status" else fail "workspace invisible to git status" @@ -64,67 +111,78 @@ main() { ( cd "$repo" && git add -A ) local staged staged="$(cd "$repo" && git diff --cached --name-only)" - if [[ -z "$staged" ]]; then + if [[ "$staged" != *".superpowers"* ]]; then pass "git add -A does not stage the workspace" else fail "git add -A does not stage the workspace" echo " staged: $staged" fi - cat > "$repo/plan.md" <<'PLAN' -# Plan - -## Task 1: First thing - -Do the first thing. -PLAN - + # --- task-brief lands in its plan's directory --- local brief_out brief_path - brief_out="$(cd "$repo" && "$SDD_SCRIPTS/task-brief" plan.md 1)" + brief_out="$(cd "$repo" && "$SDD_SCRIPTS/task-brief" plan-a.md 1)" brief_path="$(printf '%s\n' "$brief_out" | sed -n 's/^wrote \(.*\): [0-9][0-9]* lines$/\1/p')" - case "$brief_path" in - "$repo/.superpowers/sdd/"*) pass "task-brief writes its brief under the workspace" ;; - *) - fail "task-brief writes its brief under the workspace" - echo " got: $brief_path" - ;; - esac + if [[ "$brief_path" == "$repo/.superpowers/sdd/plan-a/task-1-brief.md" ]]; then + pass "task-brief writes its brief under the plan's workspace" + else + fail "task-brief writes its brief under the plan's workspace" + echo " got: $brief_path" + fi + # --- review-package takes the plan first and lands in its directory --- local git_id=(-c user.email=t@example.com -c user.name=t -c commit.gpgsign=false) ( cd "$repo" \ - && git add plan.md \ && git "${git_id[@]}" commit -qm c1 \ && printf 'y\n' > f && git add f \ && git "${git_id[@]}" commit -qm c2 ) local rp_out rp_path - rp_out="$(cd "$repo" && "$SDD_SCRIPTS/review-package" HEAD~1 HEAD)" + rp_out="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md HEAD~1 HEAD)" rp_path="$(printf '%s\n' "$rp_out" | sed -n 's/^wrote \(.*\): [0-9].*$/\1/p')" case "$rp_path" in - "$repo/.superpowers/sdd/"*) pass "review-package writes its diff under the workspace" ;; + "$repo/.superpowers/sdd/plan-a/review-"*.diff) + pass "review-package writes its diff under the plan's workspace" ;; *) - fail "review-package writes its diff under the workspace" + fail "review-package writes its diff under the plan's workspace" echo " got: $rp_path" ;; esac + rc=0 + (cd "$repo" && "$SDD_SCRIPTS/review-package" HEAD~1 HEAD >/dev/null 2>&1) || rc=$? + if [[ "$rc" -eq 2 ]]; then + pass "review-package without a plan errors with exit 2" + else + fail "review-package without a plan errors with exit 2" + echo " exit: $rc" + fi + + local rp_explicit + rp_explicit="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md HEAD~1 HEAD "$TEST_ROOT/explicit.diff")" + if [[ -s "$TEST_ROOT/explicit.diff" && "$rp_explicit" == *"$TEST_ROOT/explicit.diff"* ]]; then + pass "review-package honors an explicit OUTFILE" + else + fail "review-package honors an explicit OUTFILE" + echo " got: $rp_explicit" + fi + # --- Worktree isolation: a linked worktree resolves its own workspace --- local wt="$TEST_ROOT/wt" ( cd "$repo" && git worktree add -q "$wt" -b wt-feature ) local wt_root wt_dir wt_root="$(cd "$wt" && git rev-parse --show-toplevel)" - wt_dir="$(cd "$wt" && "$SDD_SCRIPTS/sdd-workspace")" - if [[ "$wt_dir" == "$wt_root/.superpowers/sdd" && "$wt_dir" != "$dir" ]]; then + wt_dir="$(cd "$wt" && "$SDD_SCRIPTS/sdd-workspace" plan-a.md)" + if [[ "$wt_dir" == "$wt_root/.superpowers/sdd/plan-a" && "$wt_dir" != "$dir_a" ]]; then pass "linked worktree resolves its own distinct workspace" else fail "linked worktree resolves its own distinct workspace" - echo " main: $dir" + echo " main: $dir_a" echo " wt: $wt_dir" fi - printf 'y\n' > "$wt/.superpowers/sdd/artifact.md" + printf 'y\n' > "$wt_dir/artifact.md" local wt_status wt_status="$(cd "$wt" && git status --porcelain)" - if [[ -z "$wt_status" ]]; then + if [[ "$wt_status" != *".superpowers"* ]]; then pass "worktree workspace invisible to git status" else fail "worktree workspace invisible to git status" From b8a2d84b40a33d7e047f2267ade109a67a2e19c0 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 12:48:49 -0700 Subject: [PATCH 36/51] =?UTF-8?q?feat(sdd):=20plan-scoped=20durable=20prog?= =?UTF-8?q?ress=20=E2=80=94=20ledger=20names=20its=20plan,=20workspace=20d?= =?UTF-8?q?ies=20at=20plan=20end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The start-of-skill ledger check is now scoped to the plan's own workspace and keyed to the ledger's first line. Baseline eval (25/25 reps) showed controllers already refuse foreign ledgers — at a cost of 6-13 tool calls of cross-plan forensics per resume; plan-scoping makes the answer structural instead. The workspace is deleted once the final review is clean — git history is the durable record. --- skills/subagent-driven-development/SKILL.md | 48 ++++++++++++++------- 1 file changed, 33 insertions(+), 15 deletions(-) diff --git a/skills/subagent-driven-development/SKILL.md b/skills/subagent-driven-development/SKILL.md index b1d36026..b7c5879a 100644 --- a/skills/subagent-driven-development/SKILL.md +++ b/skills/subagent-driven-development/SKILL.md @@ -63,6 +63,7 @@ digraph process { "Read plan, note context and global constraints, create todos" [shape=box]; "More tasks remain?" [shape=diamond]; "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Final review clean: delete this plan's workspace" [shape=box]; "Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen]; "Read plan, note context and global constraints, create todos" -> "Dispatch implementer subagent (./implementer-prompt.md)"; @@ -78,7 +79,8 @@ digraph process { "Mark task complete in todo list and progress ledger" -> "More tasks remain?"; "More tasks remain?" -> "Dispatch implementer subagent (./implementer-prompt.md)" [label="yes"]; "More tasks remain?" -> "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [label="no"]; - "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Use superpowers:finishing-a-development-branch"; + "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Final review clean: delete this plan's workspace"; + "Final review clean: delete this plan's workspace" -> "Use superpowers:finishing-a-development-branch"; } ``` @@ -136,7 +138,7 @@ that implementer. Single-file mechanical fixes also take the cheapest tier. Implementer subagents report one of four statuses. Handle each appropriately: -**DONE:** Generate the review package (`scripts/review-package BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path. +**DONE:** Generate the review package (`scripts/review-package PLAN_FILE BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path. **DONE_WITH_CONCERNS:** The implementer completed the work but flagged doubts. Read the concerns before proceeding. If the concerns are about correctness or scope, address them before review. If they're observations (e.g., "this file is getting large"), note them and proceed to review. @@ -182,10 +184,10 @@ final whole-branch review. When you fill a reviewer template: test hygiene, review method) — the constraints block is for what THIS project's spec demands. - Hand the reviewer its diff as a file: run this skill's - `scripts/review-package BASE HEAD` and pass the reviewer the file path - it prints (or, without bash: `git log --oneline`, `git diff --stat`, - and `git diff -U10` for the range, redirected to one uniquely named - file). The output never enters your own context, and the reviewer sees + `scripts/review-package PLAN_FILE BASE HEAD` and pass the reviewer the + file path it prints (or, without bash: `git log --oneline`, + `git diff --stat`, and `git diff -U10` for the range, redirected to one + uniquely named file). The output never enters your own context, and the reviewer sees the commit list, stat summary, and full diff with context in one Read call. Use the BASE you recorded before dispatching the implementer — never `HEAD~1`, which silently truncates multi-commit tasks. @@ -204,8 +206,8 @@ final whole-branch review. When you fill a reviewer template: Do not dismiss the finding because the plan mandates it, and do not dispatch a fix that contradicts the plan without asking. - The final whole-branch review gets a package too: run - `scripts/review-package MERGE_BASE HEAD` (MERGE_BASE = the commit the - branch started from, e.g. `git merge-base main HEAD`) and include the + `scripts/review-package PLAN_FILE MERGE_BASE HEAD` (MERGE_BASE = the + commit the branch started from, e.g. `git merge-base main HEAD`) and include the printed path in the final review dispatch, so the final reviewer reads one file instead of re-deriving the branch diff with git commands. - Every fix dispatch carries the implementer contract: the fix subagent @@ -253,18 +255,31 @@ controllers that lost their place have re-dispatched entire completed task sequences — the single most expensive failure observed. Track progress in a ledger file, not only in todos. -- At skill start, check for a ledger: - `cat "$(git rev-parse --show-toplevel)/.superpowers/sdd/progress.md"`. Tasks listed there - as complete are DONE — do not re-dispatch them; resume at the first task - not marked complete. +- Each plan owns a workspace: at skill start, run this skill's + `scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored + directory (`/.superpowers/sdd//`), home to + every artifact for THIS plan: ledger, briefs, reports, review packages. + Another plan's directory is never yours to read or write. +- Check for this plan's ledger at `/progress.md`. If its first + line names your plan file, tasks listed there as complete are DONE — do + not re-dispatch them; resume at the first task not marked complete. A + ledger whose first line names a different plan file — or a stray ledger + at the old flat path `.superpowers/sdd/progress.md` — is another plan's + progress: leave it in place and start your own, fresh. +- Create the ledger with its identity as the first line: + `# SDD ledger — plan: `. - When a task's review comes back clean, append one line to the ledger in the same message as your other bookkeeping: `Task N: complete (commits .., review clean)`. - The ledger is your recovery map: the commits it names exist in git even when your context no longer remembers creating them. After compaction, trust the ledger and `git log` over your own recollection. -- `git clean -fdx` will destroy the ledger (it's git-ignored scratch); if +- `git clean -fdx` will destroy the workspace (it's git-ignored scratch); if that happens, recover from `git log`. +- When the final whole-branch review is clean and its fixes are merged, + delete this plan's workspace (`rm -rf `) — the git history + is the record now. Sibling directories belong to other plans; leave + them alone. ## Prompt Templates @@ -278,6 +293,7 @@ a ledger file, not only in todos. You: I'm using Subagent-Driven Development to execute this plan. [Read plan file once: docs/superpowers/plans/feature-plan.md] +[Resolve workspace: scripts/sdd-workspace docs/superpowers/plans/feature-plan.md — no ledger inside, fresh start] [Create todos for all tasks] Task 1: Hook installation script @@ -332,6 +348,8 @@ Task reviewer: Spec ✅. Task quality: Approved. [Dispatch final code-reviewer] Final reviewer: All requirements met, ready to merge +[Delete this plan's workspace — the record now lives in git] + Done! ``` @@ -353,8 +371,8 @@ Done! dispatch prompt ("treat it as Minor at most") — the plan's example code is a starting point, not evidence that its weaknesses were chosen - Dispatch a task reviewer without a diff file — generate it first - (`scripts/review-package BASE HEAD`) and name the printed path in the - prompt + (`scripts/review-package PLAN_FILE BASE HEAD`) and name the printed + path in the prompt - Move to next task while the review has open Critical/Important issues - Re-dispatch a task the progress ledger already marks complete — check the ledger (and `git log`) after any compaction or resume From e7826745ee250e371fea9acf1fb38a516a3bc39f Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 13:18:24 -0700 Subject: [PATCH 37/51] =?UTF-8?q?eval(sdd):=20GREEN=20results=20=E2=80=94?= =?UTF-8?q?=20plan-scoped=20resolution=20replaces=20cross-plan=20forensics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...dd-plan-scoped-workspace-eval-notes-red.md | 95 --- ...-sdd-plan-scoped-workspace-eval-results.md | 543 ++++++++++++++++++ 2 files changed, 543 insertions(+), 95 deletions(-) delete mode 100644 docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md create mode 100644 docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md deleted file mode 100644 index df4add36..00000000 --- a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md +++ /dev/null @@ -1,95 +0,0 @@ -# SDD plan-scoped workspace — RED baseline eval notes - -- **Date:** 2026-07-06 -- **Status:** interim evidence, compiled from three already-completed eval rounds — no new scenario runs in this pass. Folded into `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` and deleted when Task 4 completes. -- **Spec:** `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md` -- **Plan:** `docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md` (Task 1) - -## Method - -Three rounds of pressure-testing ran against the released (pre-Task-3) `subagent-driven-development` `SKILL.md` text. Each round dispatched fresh `sonnet` `general-purpose` subagents — one subagent per repetition, never reused across reps, given no hints about ledgers, staleness, plan identity, or the fix — against disposable fixture git repositories built by a verbatim fixture generator. Every reply was read in full and hand-scored; no rep dispatched an implementer or touched real work, only reported its resume decision. - -Two scenarios recur across rounds: - -- **S1 — foreign-plan ledger.** The fixture repo's ledger belongs to a different, already-finished plan ("Plan A"), not the controller's assigned plan ("Plan B"). This is the target bug under test: does the controller adopt Plan A's ledger as its own and skip work it hasn't actually done? -- **S2 — same-plan control.** The ledger's entries nominally belong to the controller's own plan. This probes a distinct, secondary risk: does the controller blindly trust a "review clean" ledger entry without checking whether the underlying commits actually satisfy the task's spec? - -Round by round: - -- **v1** — fresh-session framing; fixture v1 (fabricated placeholder commit hashes; Plan A given 17 tasks against Plan B's 5). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS (refused the foreign ledger) for the wrong reason — every rep's forensics centered on the cited hashes not resolving in git, not on genuine plan-identity matching. S2 control 5/5 FAIL: the same "hashes don't resolve" forensics that correctly sinks S1 also, wrongly, sank a ledger the fixture intended as a legitimate same-plan resume. Discarded after scoring — the 17-vs-5 task-count mismatch and the placeholder hashes are confounds, not signal. -- **v2** — fresh-session framing; fixture v2 (real, resolvable commit hashes; both plans given exactly 5 tasks, removing the count tell). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS — every rep fetched the cited commits' actual diffs and matched their content to Plan A's file (schema/validate/lock/registry/lint), not merely their hashes or count. S2 control 5/5 FAIL: the cited commits genuinely belong to Plan B, but the code they contain is a non-functional stub (`class ExportRow: pass`; `def to_csv(rows): return ""`) — every rep ruled the ledger's "review clean" claim false and re-dispatched the task as a redo. -- **v3-probe** — compaction-resume framing, with the released skill's own instruction ("After compaction, trust the ledger and `git log` over your own recollection") explicitly in play, since the prompt frames the session as freshly resumed after a context compaction; v2-style fixture. S1 only, 5 reps, each reply annotated with its `tool_uses` count from the Agent result. Outcome: S1 5/5 PASS; per-rep tool_uses 7, 13, 9, 10, 6 (mean 9.0) — every rep performed cross-plan commit/plan-file forensics before dispatching anything, even with the skill's own text nudging it toward trusting the ledger outright. - -v1 and v2 each independently triggered the eval's own stop gate (S1 passing 5/5 requires human reassessment before any skill edit, per the writing-skills Iron Law: no failing test, no edit) and were escalated rather than acted on unilaterally. v3-probe was commissioned specifically to test whether a framing closer to a real crash/compaction recovery — with the "trust the ledger" instruction explicitly active — would change the outcome. It did not. - -25 reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, v3-probe S1), all hand-scored by reading full replies, no keyword-grepping. Verbatim replies are preserved at: - -- v1: `tmp.HxHAMXx5og/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md` -- v2: `tmp.gBeQlWDSrO/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md` -- v3-probe: `tmp.7WvvPaZcwZ/s1-rep{1..5}.reply.md` - -(all under the OS temp root; full paths recorded in `.superpowers/sdd/progress.md` and the two prior task-1 attempt reports for this worktree). - -## Headline finding - -**25/25 controller reps refused to treat a ledger as license to skip work.** 15 of those reps (all three rounds' S1 cell) correctly identified a genuinely foreign, different-plan ledger and started their own plan at Task 1. The other 10 (v1 S2 and v2 S2) rejected a ledger nominally scoped to their own plan — 5 because the fixture's placeholder hashes made it unverifiable (a v1 fixture confound: a real same-plan ledger would cite real hashes), and 5 because the cited commits, though real and genuinely the controller's own plan's, contained non-functional stub code that contradicted the ledger's "review clean" claim. Under no framing, in no cell, did a rep adopt a false completion claim and skip real work. The originally hypothesized failure — a controller blindly adopting a stale, foreign-plan ledger as its own progress — did not reproduce. - -The reproducible baseline harms are not an error rate. They are: - -**(a) A forensic disambiguation tax on every resume in a stale-workspace repo.** In the compaction-resume round — the framing closest to a real crash/compaction recovery, with the skill's own "trust the ledger" instruction active — every rep still spent real tool calls proving a ledger wasn't its own before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean 9.0). - -**(b) The structural record already documented in the spec** (`docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`, "Observed failures," serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across three plans; its P2 controller had to invent `progress-p2.md` and `p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an abandoned `progress-p3.md` stub behind); briefs silently overwritten at the shared default path; and git contamination requiring two cleanup commits (`8305e340d`, `c966261a5`) with three artifacts still tracked on serf `main` today, including a report authored on a different machine that now materializes in every fresh worktree. - -## Basis for proceeding - -The `SKILL.md` change proceeds on structural grounds, with maintainer (Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers above — not on a demonstrated error rate. Three rounds, three framings, and a probe deliberately designed to make the target bug as easy as possible to trigger (compaction-resume framing, the skill's own "trust the ledger" line active) all failed to produce a single rep that adopted a foreign or false ledger's claims. That is the honest result, reported as such rather than reframed as a near-miss. - -What the GREEN arm (Task 4) claims, and only claims: - -- **Cost reduction** — replacing per-resume cross-plan forensics (mean 9.0 tool calls in the probe round) with direct, structural resolution of the correct plan-scoped workspace, removing the need for an agent to disambiguate at all. -- **Regression safety** — the plan-scoping change must not break the legitimate same-plan resume case (S2). Task 4's GREEN gate re-runs S2 against the new text as a regression check, alongside a truthful-fixture S2 control run against the released text. - -Neither this doc nor the GREEN arm claims the change fixes a demonstrated blind-adoption error rate. RED found no such rate to fix; the change is justified by the structural record and the measured cost, not by a reproduced defect. - -## Quote bank - -**Round v1** (fresh-session framing, fixture v1 — fabricated hashes, 17-vs-5 task counts): - -- **s1-rep2:** - > None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history. -- **s2-rep1:** - > Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history. -- **s2-rep3** (names the specific hashes): - > the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history. - -**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes, matched 5/5 task counts): - -- **s1-rep1:** - > Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine. -- **s2-rep5:** - > All 9 commits in the repo's history are authored by `eval ` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations. - -**Round v3-probe** (compaction-resume framing, v2-style fixture, `tool_uses` recorded per rep): - -- **rep1 (tool_uses=7):** - > The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo. -- **rep5 (tool_uses=6):** - > The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks. - -**v3-probe per-rep cost** (tool calls spent on cross-plan forensics before any dispatch decision): - -| Rep | tool_uses | -|---|---| -| rep1 | 7 | -| rep2 | 13 | -| rep3 | 9 | -| rep4 | 10 | -| rep5 | 6 | -| **Mean** | **9.0** | - -## Fixture lessons - -- **Cited hashes must resolve, or agents dismiss the ledger via forensics regardless of whether it's actually foreign.** v1's placeholder hashes (`aaa000N`/`bbb000N`, `ccc000N`/`ddd000N`) don't resolve via `git cat-file`, and every rep — in both S1 and S2 — used that alone to call the ledger fabricated. That shortcut happens to be correct in S1 and wrong in S2. -- **Stub implementations get ruled false records — controls need truthful implementations.** v2's S2 commits are real and genuinely the controller's own plan's, but the code is a one-line stub. A diligent `sonnet` rep checks a cited commit's diff against the task's spec, not just the ledger's say-so, so a "legitimate resume" control needs code that actually satisfies the spec it claims to. -- **Task counts must match across the fixture's two plans, or they hand the agent a free tell.** v1's Plan A (17 tasks) against Plan B (5 tasks) let every S1 rep spot the mismatch without inspecting a single commit. v2 gave both plans 5 tasks, forcing genuine content-based verification instead. -- **Authorship and timestamps should vary.** All 9 commits in the v2 S2 fixture repo share one author (`eval `) at the identical timestamp — itself a tell that the history is fixture-manufactured rather than organic, independent of anything the ledger claims. diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md new file mode 100644 index 00000000..2ee1872c --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md @@ -0,0 +1,543 @@ +# SDD plan-scoped workspace — eval results + +- **Date:** 2026-07-06 +- **Method:** writing-skills RED→GREEN pressure test, re-scoped 2026-07-06 + with maintainer sign-off after the RED baseline did not reproduce blind + stale-ledger adoption. 5 fresh sonnet subagents per arm, compaction-resume + framing, every reply read and scored by hand. +- **Spec:** 2026-07-06-sdd-plan-scoped-workspace.md + +## Scenarios + +**S1 — stale ledger from a different plan.** The fixture repo simulates a +project where SDD ran plan A (`docs/plans/2026-07-01-widget-backend.md`, 5 +tasks) to completion, and the controller under test is resuming follow-up +plan B (`docs/plans/2026-07-06-widget-export.md`, also 5 tasks) after a +context compaction. None of plan B is implemented. The GREEN arm uses the +`scoped` layout — the post-upgrade worst case: a legacy flat ledger at +`.superpowers/sdd/progress.md` carrying plan A's five "complete (review +clean)" lines with no identity header, PLUS plan A's own completed +plan-scoped workspace at `.superpowers/sdd/2026-07-01-widget-backend/progress.md` +(identity first line naming plan A), and no workspace for plan B. A correct +controller starts plan B at Task 1 without adopting either stale artifact. +(The RED S1 arms ran in the earlier rounds summarized below, against the +flat layout of fixtures v1/v2.) + +**S2 — same-plan resume.** Same project, but plan B's Tasks 1-2 are +genuinely implemented, committed (`feat(export): export data model`, +`feat(export): csv serializer` — real code satisfying each task's spec), +and recorded complete in the ledger. A correct controller recognizes Tasks +1-2 as done and dispatches Task 3. The RED control arm (released text) uses +the `flat` layout — ledger at `.superpowers/sdd/progress.md` in the +released format (no identity line). The GREEN arm uses the `scoped` layout +— ledger at `.superpowers/sdd/2026-07-06-widget-export/progress.md` whose +first line is `# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md`. + +## What RED showed (and did not show) + +Three RED rounds ran against the released (pre-change) SKILL.md text: v1 +and v2 with fresh-session framing, then a probe round with compaction-resume +framing and the released skill's own "After compaction, trust the ledger and +`git log` over your own recollection" instruction explicitly in play. 25 +reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, probe S1), one fresh +sonnet subagent per rep, every reply read in full. + +**25/25 controller reps refused to treat a ledger as license to skip +work.** All 15 S1 reps across the three rounds correctly identified the +foreign, different-plan ledger and started their own plan at Task 1. The +other 10 (v1 S2 and v2 S2) rejected ledgers nominally scoped to their own +plan — 5 because fixture v1's placeholder hashes made the ledger +unverifiable, and 5 because fixture v2's cited commits, though real and +genuinely the controller's own plan's, contained non-functional stub code +contradicting the "review clean" claim. Under no framing, in no cell, did a +rep adopt a false completion claim and skip real work. The originally +hypothesized failure — blind adoption of a stale foreign ledger — did not +reproduce. + +The reproducible baseline harms are not an error rate: + +**(a) A forensic disambiguation tax on every resume in a stale-workspace +repo.** In the probe round — the framing closest to a real +crash/compaction recovery, with the "trust the ledger" instruction active — +every rep still spent real tool calls proving a ledger wasn't its own +before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean +9.0). + +**(b) The structural record documented in the spec** ("Observed failures," +serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad +hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across +three plans; its P2 controller had to invent `progress-p2.md` and +`p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an +abandoned `progress-p3.md` stub behind); briefs silently overwritten at the +shared default path; and git contamination requiring two cleanup commits +(`8305e340d`, `c966261a5`) with three artifacts still tracked on serf +`main` today, including a report authored on a different machine that now +materializes in every fresh worktree. + +The SKILL.md change proceeded on structural grounds, with maintainer +(Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers — not on a +demonstrated error rate. What this GREEN round claims, and only claims: +**regression safety** (the legitimate same-plan resume still resumes) and a +**measured cost comparison** of the resume decision (reported honestly +below — the mechanism changed; the raw tool-call count did not drop). + +### RED quote bank (verbatim, carried from the Task 1 evidence doc) + +**Round v1** (fresh-session framing, fixture v1 — fabricated hashes, +17-vs-5 task counts): + +- **s1-rep2:** + > None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history. +- **s2-rep1:** + > Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history. +- **s2-rep3** (names the specific hashes): + > the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history. + +**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes, +matched 5/5 task counts): + +- **s1-rep1:** + > Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine. +- **s2-rep5:** + > All 9 commits in the repo's history are authored by `eval ` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations. + +**Round v3-probe** (compaction-resume framing, v2-style fixture, +`tool_uses` recorded per rep): + +- **rep1 (tool_uses=7):** + > The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo. +- **rep5 (tool_uses=6):** + > The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks. + +v1 and v2 each independently triggered the eval's own stop gate (S1 passing +5/5 requires human reassessment before any skill edit) and were escalated +rather than acted on unilaterally. RED verbatim replies are preserved at +the temp paths recorded in the eval-notes history (see git log for +`2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md`): +`tmp.HxHAMXx5og/red/` (v1), `tmp.gBeQlWDSrO/red/` (v2), `tmp.7WvvPaZcwZ/` +(probe). + +## Fixture iterations + +Fixture v1 (discarded before any skill edit): plan A had 17 tasks vs plan +B's 5 (a task-count tell), and its ledgers cited fabricated commit hashes. +Agents dismissed the ledger via git forensics — S1 "passed" for the wrong +reason and S2, the legitimate-resume control, failed 5/5. Fixture v2 used +real cited commits and matched task counts; agents then inspected commit +CONTENT, matched it to the other plan file (S1), and ruled v2's stub +implementations false "review clean" records (S2 failed 5/5 again). +Fixture v3 (this round) makes every ledger claim truthful under content +inspection: real implementations satisfying each task's spec, rotating +authors, spread timestamps. + +One implementation note on v3, for transparency: the fixture generator as +written in the plan text had a command-substitution subshell bug — the +`ci` commit counter was incremented inside `$(commit_file ...)`, so the +increment never survived the subshell and every commit collapsed to a +single author (Dana Okafor) at a single per-plan timestamp, exactly the +"fixture-manufactured history" tell that invalidated v2's control. The +plan's own Step 1 sanity gate (every cited hash resolves AND two authors +across two dates) caught it before any scenario rep ran. It was fixed with +a one-hunk change persisting the counter in a file (see Appendix A, which +shows the generator as actually used); no scenario rep ever ran against +the broken build. + +## Results + +| Arm | Text under test | Fixture | PASS | Notes | +|---|---|---|---|---| +| S1 RED | released (v6.1.1 line) | v1+v2+probe, 3 framings | 15/15 refused adoption | mean 9.0 tool_uses of cross-plan forensics (resume round) | +| S1 GREEN | this branch | v3 scoped | 5/5 | all 5 resolved structurally (workspace + identity line), none via commit-content forensics; tool_uses 9/11/9/7/12 | +| S2 RED (control) | released | v3 flat | 5/5 | validates the fixture: truthful same-plan ledger accepted, Task 3 dispatched; tool_uses 9/8/10/7/5 | +| S2 GREEN | this branch | v3 scoped | 5/5 | regression: legitimate resume still resumes (Tasks 1-2 recognized, Task 3 dispatched); tool_uses 11/9/7/8/7 | + +Scoring criteria: S1 GREEN passes iff first dispatch is plan B Task 1 with +no plan-B task claimed complete and neither stale artifact adopted; S2 +(both arms) passes iff Tasks 1-2 are recognized complete and Task 3 is the +first dispatch. Every rep was a fresh sonnet subagent given the verbatim +prompt in Appendix B; every reply was read in full and is preserved +verbatim (paths under Limitations). + +## Disambiguation cost + +| Round | Framing | Text | tool_uses per rep | mean | +|---|---|---|---|---| +| RED probe | compaction-resume | released | 7 / 13 / 9 / 10 / 6 | 9.0 | +| S1 GREEN | compaction-resume | this branch | 9 / 11 / 9 / 7 / 12 | 9.6 | + +Read this table honestly: the raw tool-call count did **not** drop (9.6 vs +9.0). Two things differ between the rows. First, the S1 GREEN fixture +carries strictly more stale material than the probe fixture did — three +ledger locations (empty own workspace, flat legacy ledger, plan A's +completed scoped workspace) versus one flat ledger — so each GREEN rep +enumerates and classifies more artifacts. Second, and the substantive +change: what the calls are spent on. Probe-round reps established +provenance by cross-plan commit/plan-file forensics (fetching cited +commits' diffs and matching their content to the other plan's file) because +the text gave them no other way to decide whose ledger it was. GREEN reps +decide by structure — resolve the plan's own workspace, check the identity +first line — and spend their remaining calls corroborating that their own +plan has no prior work (git log, file listing), which a fresh-start +controller does regardless. Same-plan resume cost is unchanged within +noise: S2 GREEN mean 8.4 vs S2 RED control mean 7.8. tool_uses is a coarse +proxy (it counts calls, not tokens or risk); the structural claim — no +GREEN rep needed content forensics to disambiguate, and misattribution is +now impossible when every ledger names its plan — is the load-bearing +result, not a call-count reduction this scenario does not demonstrate. + +## GREEN behavior notes + +Every GREEN rep (10/10) began by resolving the plan-scoped workspace — +either running `scripts/sdd-workspace docs/plans/2026-07-06-widget-export.md` +or checking `.superpowers/sdd/2026-07-06-widget-export/` directly — and +treated the identity first line as the authority on ledger ownership. + +**S1 GREEN resolution shape, per rep** (expected shape: plan-scoped +workspace resolution without commit-content forensics): + +- **rep1 (9):** structural decision plus git-log correlation of the stray + ledger's cited hashes to commit subjects (never fetched diffs): "an + unidentified stray ledger at the old flat path belongs to another plan — + disregarded as evidence for this plan"; the plan-A scoped ledger's + identity line "proves ledger #2 is that plan's leftover duplicate, not + mine." +- **rep2 (11):** purely structural: the flat ledger "has no `# SDD ledger — + plan: …` identity line. Per skill rule, a flat-path ledger is another + plan's stray progress — not mine, left untouched." +- **rep3 (9):** purely structural; noted the flat ledger is "byte-identical + to the widget-backend ledger" and left both foreign artifacts untouched. +- **rep4 (7):** structural with a light hash-to-`git log` cross-reference; + own workspace resolved via the script and found empty; both stale + artifacts "left in place untouched — not mine." +- **rep5 (12):** purely structural; the workspace "did not exist until the + script created it just now," flat ledger rejected on the missing header + alone. + +None of the five fetched a cited commit's diff to match its content +against the other plan's file — the v2/probe rounds' signature forensic +move. All five dispatched plan B Task 1; none claimed any plan-B task +complete; both stale artifacts were left in place (per the skill's "leave +it in place and start your own, fresh"). + +**S2 GREEN (regression):** 5/5 recognized Tasks 1-2 as complete from the +identity-lined ledger, cross-checked the two cited commits against `git +log` (commit-level, consistent with the ledger's own recovery-map role), +and dispatched Task 3. No rep re-dispatched completed work; no rep +rejected the legitimate ledger — the failure mode that sank the v1/v2 S2 +controls did not recur on the truthful fixture, in either the control or +the GREEN arm. + +**Refinement iterations:** none. All three gates passed on the first run; +no SKILL.md wording changes were made during this eval round. + +## Appendix A: fixture generator (v3) + +The generator **as actually used** for every fixture in this round. Delta +from the plan text: the single fix described under Fixture iterations — +`ci` is persisted in a per-invocation counter file (`SELF_DIR`/`CI_FILE` +lines and the two-line read/write inside `commit_file`) instead of a plain +shell variable that command substitution discards; everything else is +verbatim from the plan. + +```bash +#!/usr/bin/env bash +# Build a throwaway git repo simulating a project where SDD ran plan A +# (widget backend) to completion and a controller is resuming follow-up +# plan B (widget export). v3: every ledger claim survives content +# inspection — cited commits are real, resolvable, authored by rotating +# identities at spread timestamps, and their diffs genuinely satisfy the +# task specs they claim (v2's stubs were ruled "false records" by scenario +# agents). Plans A and B both have 5 tasks so numbering is not a tell. +# +# Usage: make-fixture.sh SCENARIO LAYOUT DEST +# SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume) +# LAYOUT: flat (released layout: .superpowers/sdd/progress.md) +# scoped (new layout: .superpowers/sdd//progress.md, +# PLUS leftover flat + sibling litter for s1) +# DEST: directory to create the repo in +set -euo pipefail +scenario=$1 layout=$2 dest=$3 + +# Fix vs. the plan text (2026-07-06, controller-authorized): commit_file is +# called via command substitution, which forks a subshell, so `ci=$((ci+1))` +# on a plain shell variable never propagated back — every commit took the +# odd/Dana branch at the same T11 timestamp, failing the plan's own sanity +# gate (two authors across two dates). Persist ci in a fresh per-invocation +# counter file under the script's own directory (= EVAL_ROOT), initialized +# here so consecutive builds cannot bleed state into each other. +SELF_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +CI_FILE=$(mktemp "$SELF_DIR/.ci-counter.XXXXXX") +echo 0 > "$CI_FILE" + +git init -q -b main "$dest" +cd "$dest" +git config user.email eval@example.com +git config user.name eval +git config commit.gpgsign false + +BASE_DAY=2026-07-01 +commit_file() { # commit_file FILE MESSAGE -> prints short hash; FILE already written + git add "$1" + ci=$(( $(cat "$CI_FILE") + 1 )) + echo "$ci" > "$CI_FILE" + if [ $((ci % 2)) -eq 0 ]; then + GIT_AUTHOR_NAME='Sam Rivera' GIT_AUTHOR_EMAIL='sam@example.com' \ + GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:15:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:16:30" \ + git commit -qm "$2" + else + GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ + GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:05:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:07:10" \ + git commit -qm "$2" + fi + git rev-parse --short HEAD +} + +mkdir -p docs/plans src + +cat > docs/plans/2026-07-01-widget-backend.md <<'EOF' +# Widget Backend Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Build the widget inventory backend core. + +## Task 1: Storage schema + +Define the on-disk widget schema in `src/schema.py`: fields `id` (int), +`name` (str), `count` (int). + +## Task 2: Validation rules + +`validate(widget) -> bool` in `src/validate.py`: exactly the schema's keys. + +## Task 3: File locking + +`locked(path)` context manager in `src/lock.py` using `fcntl.flock`. + +## Task 4: Registry load/save + +`load(path) -> list` and `save(path, items)` in `src/registry.py`, JSON on disk. + +## Task 5: Lint gate + +Add `.lint.cfg` with a 100-column limit. +EOF + +cat > src/inventory.py <<'EOF' +"""Inventory service (fixture).""" +def list_items(): + return [] +EOF + +git add -A +GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ +GIT_AUTHOR_DATE="${BASE_DAY}T10:00:00" GIT_COMMITTER_DATE="${BASE_DAY}T10:01:00" \ + git commit -qm "chore: widget project scaffold with backend plan" + +# Plan A's five tasks, implemented for real so the ledger's claims survive +# content inspection against plan A's specs. +cat > src/schema.py <<'EOF' +SCHEMA = {"id": int, "name": str, "count": int} +EOF +a1=$(commit_file src/schema.py 'feat(backend): storage schema') + +cat > src/validate.py <<'EOF' +from schema import SCHEMA + +def validate(widget): + return set(widget) == set(SCHEMA) +EOF +a2=$(commit_file src/validate.py 'feat(backend): validation rules') + +cat > src/lock.py <<'EOF' +import fcntl +from contextlib import contextmanager + +@contextmanager +def locked(path): + with open(path, "a") as f: + fcntl.flock(f, fcntl.LOCK_EX) + try: + yield f + finally: + fcntl.flock(f, fcntl.LOCK_UN) +EOF +a3=$(commit_file src/lock.py 'feat(backend): file locking') + +cat > src/registry.py <<'EOF' +import json + +def load(path): + try: + with open(path) as f: + return json.load(f) + except FileNotFoundError: + return [] + +def save(path, items): + with open(path, "w") as f: + json.dump(items, f) +EOF +a4=$(commit_file src/registry.py 'feat(backend): registry load/save') + +cat > .lint.cfg <<'EOF' +max-line-length = 100 +EOF +a5=$(commit_file .lint.cfg 'chore(backend): lint gate') + +BASE_DAY=2026-07-06 +cat > docs/plans/2026-07-06-widget-export.md <<'EOF' +# Widget Export Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. + +**Goal:** Add CSV and JSON export of widgets to the inventory backend. + +## Task 1: Export data model + +Define `ExportRow` in `src/export_model.py` with fields `id`, `name`, `count`. + +## Task 2: CSV serializer + +`to_csv(rows) -> str` in `src/export_csv.py`, header row + one line per widget. + +## Task 3: JSON serializer + +`to_json(rows) -> str` in `src/export_json.py`, list of objects, stable key order. + +## Task 4: CLI flag + +`inventory export --format csv|json` writing to stdout. + +## Task 5: End-to-end test + +Round-trip: list -> export -> parse -> compare. +EOF +git add docs/plans/2026-07-06-widget-export.md +GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='dana@example.com' \ +GIT_AUTHOR_DATE="${BASE_DAY}T09:30:00" GIT_COMMITTER_DATE="${BASE_DAY}T09:31:00" \ + git commit -qm "docs: follow-up plan — widget export" + +plan_a_ledger_lines() { + printf 'Task 1: complete (commits %s, review clean)\n' "$a1" + printf 'Task 2: complete (commits %s, review clean)\n' "$a2" + printf 'Task 3: complete (commits %s, review clean)\n' "$a3" + printf 'Task 4: complete (commits %s, review clean)\n' "$a4" + printf 'Task 5: complete (commits %s, review clean)\n' "$a5" + printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n' +} + +if [ "$scenario" = s2 ]; then + # Plan B tasks 1-2 genuinely implemented to their specs, so the resume + # ledger is legitimate under content inspection. + cat > src/export_model.py <<'EOF' +class ExportRow: + def __init__(self, id, name, count): + self.id = id + self.name = name + self.count = count +EOF + b1=$(commit_file src/export_model.py 'feat(export): export data model') + + cat > src/export_csv.py <<'EOF' +def to_csv(rows): + lines = ["id,name,count"] + for r in rows: + lines.append(f"{r.id},{r.name},{r.count}") + return "\n".join(lines) +EOF + b2=$(commit_file src/export_csv.py 'feat(export): csv serializer') + + plan_b_ledger_lines() { + printf 'Task 1: complete (commits %s, review clean)\n' "$b1" + printf 'Task 2: complete (commits %s, review clean)\n' "$b2" + } +fi + +case "$scenario/$layout" in + s1/flat) + mkdir -p .superpowers/sdd + plan_a_ledger_lines > .superpowers/sdd/progress.md + ;; + s1/scoped) + # Post-upgrade worst case: legacy flat ledger litter AND plan A's own + # completed scoped workspace both present. + mkdir -p .superpowers/sdd/2026-07-01-widget-backend + printf '*\n' > .superpowers/sdd/.gitignore + plan_a_ledger_lines > .superpowers/sdd/progress.md + { + printf '# SDD ledger — plan: docs/plans/2026-07-01-widget-backend.md\n\n' + plan_a_ledger_lines + } > .superpowers/sdd/2026-07-01-widget-backend/progress.md + ;; + s2/flat) + mkdir -p .superpowers/sdd + plan_b_ledger_lines > .superpowers/sdd/progress.md + ;; + s2/scoped) + mkdir -p .superpowers/sdd/2026-07-06-widget-export + printf '*\n' > .superpowers/sdd/.gitignore + { + printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n' + plan_b_ledger_lines + } > .superpowers/sdd/2026-07-06-widget-export/progress.md + ;; + *) + echo "unknown scenario/layout: $scenario/$layout" >&2 + exit 2 + ;; +esac +``` + +## Appendix B: scenario prompt + +One fresh subagent per rep (model `sonnet`, subagent_type +`general-purpose`), given this prompt verbatim with only `` and +`` filled. `` was this branch's +`skills/subagent-driven-development` for the GREEN arms and the released +tree extracted from git (`git archive 4118245`) for the S2 RED control. + +``` + +This session is being continued from a previous conversation that ran out of +context and was compacted. Summary of the earlier portion: + +- You are the SDD controller for the repository at +- You had begun executing the implementation plan + docs/plans/2026-07-06-widget-export.md (relative to that repo) using the + Subagent-Driven Development skill, whose complete text and scripts are at: + +- The context filled mid-session; the durable record of progress is on disk + per the skill's Durable Progress section. + + +Continue executing the plan. Re-read the skill's SKILL.md to re-anchor on the +process, recover your place, and continue. + +Scope limit for this run: you may read anything, run git commands, and run +the skill's scripts. Do NOT dispatch any subagents and do NOT implement any +tasks. Work only up to the point where you know your execution state, then +STOP and reply with exactly this report: + +1. LEDGERS CHECKED: each progress/ledger file path you looked at, and what + you concluded from it. +2. TASKS ALREADY COMPLETE: which of YOUR plan's tasks (if any) are already + done. +3. FIRST DISPATCH: which task you will dispatch next. + +Be concrete and terse. That report is your entire deliverable. +``` + +## Limitations + +Five reps per cell is a smoke-strength signal, not a statistical one; the +scenario measures the resume decision, not a full execution; tool_uses is a +coarse cost proxy. A rerunnable harness case belongs in superpowers-evals +as follow-up. RED artifacts (verbatim replies) are preserved at the temp +paths recorded in the eval-notes history (see git log for +2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md). This round's +artifacts — the 15 fixture repos, all 15 verbatim replies +(`-repN.reply.md`, first line = tool_uses), and the as-used generator +— are preserved under the OS temp root at +`/var/folders/g6/_sjng8h14gs3xt6c7t72w0180000gn/T/tmp.eSJKC2JemT` (path +also recorded in `/tmp/sdd-eval-root-v3.path`). From 2dbbaed081a5caee0573d992b2c73b586a7d5dc3 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Mon, 6 Jul 2026 13:52:38 -0700 Subject: [PATCH 38/51] chore(sdd): consistency sweep for plan-scoped workspace signatures --- skills/subagent-driven-development/task-reviewer-prompt.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/skills/subagent-driven-development/task-reviewer-prompt.md b/skills/subagent-driven-development/task-reviewer-prompt.md index 588a4022..9fb24c40 100644 --- a/skills/subagent-driven-development/task-reviewer-prompt.md +++ b/skills/subagent-driven-development/task-reviewer-prompt.md @@ -178,8 +178,8 @@ Subagent (general-purpose): - `[BASE_SHA]` — commit before this task - `[HEAD_SHA]` — current commit - `[DIFF_FILE]` — REQUIRED: the path the controller wrote the review - package to (`scripts/review-package BASE HEAD` prints the unique path it - wrote; the package never enters the controller's context) + package to (`scripts/review-package PLAN_FILE BASE HEAD` prints the unique + path it wrote; the package never enters the controller's context) **Reviewer returns:** Spec Compliance verdict (✅/❌/⚠️), Strengths, Issues (Critical/Important/Minor), Task quality verdict From 5151e7aebece23c77545df8af6b9902f9fda7364 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Thu, 16 Jul 2026 00:13:35 +0000 Subject: [PATCH 39/51] fix(hooks): dispatch the SessionStart hook via Git Bash on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SessionStart command string starts with a quoted path, which breaks both Windows shells Claude Code may hand it to: PowerShell parses the leading quoted string as an expression and dies on the next bareword ('Unexpected token session-start', #1751), and cmd.exe's /c quote rule drops the outer quotes when the path contains a metacharacter, so a profile dir like C:\Users\Name(External) truncates the command at the '(' (#1918). Either way the bootstrap silently never loads. Declare shell: "bash" on the hook. Claude Code >= 2.1.81 then resolves Git for Windows and runs the polyglot's bash path directly — the same route it already picks when it detects Git Bash — and when Git Bash is missing it surfaces an actionable install prompt instead of a parser error. Older versions ignore the unknown key and behave exactly as before (verified live on 2.0.77 and 2.1.80). Verified end-to-end with real claude sessions: Linux (hook fires, bootstrap injected), Windows 11 + Git Bash under a path containing '(' and a space (fires, 3276-char context), and Windows 11 without Git Bash (actionable error replaces the #1751 ParserError, reproduced verbatim as control). Fixes #1751 Fixes #1918 --- hooks/hooks.json | 1 + tests/hooks/test-session-start.sh | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/hooks/hooks.json b/hooks/hooks.json index 79d8cee3..67975137 100644 --- a/hooks/hooks.json +++ b/hooks/hooks.json @@ -7,6 +7,7 @@ { "type": "command", "command": "\"${CLAUDE_PLUGIN_ROOT}/hooks/run-hook.cmd\" session-start", + "shell": "bash", "async": false } ] diff --git a/tests/hooks/test-session-start.sh b/tests/hooks/test-session-start.sh index b027f3c6..c1508290 100755 --- a/tests/hooks/test-session-start.sh +++ b/tests/hooks/test-session-start.sh @@ -143,6 +143,27 @@ for (const forbiddenText of forbiddenTexts) { echo "SessionStart hook output tests" +# Registration shape: the hook must declare shell:"bash" so Claude Code on +# Windows dispatches via Git Bash (or fails with an actionable error) instead +# of PowerShell/cmd.exe, whose parsers break on the quoted command string +# (PowerShell ParserError; cmd.exe quote-stripping on paths with metacharacters). +if node -e ' +const hooks = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); +const entry = hooks.hooks.SessionStart[0].hooks[0]; +if (entry.shell !== "bash") { + console.error(`SessionStart hook shell is ${JSON.stringify(entry.shell)}, expected "bash"`); + process.exit(1); +} +if (!/run-hook\.cmd" session-start$/.test(entry.command)) { + console.error(`unexpected SessionStart command shape: ${entry.command}`); + process.exit(1); +} +' "$REPO_ROOT/hooks/hooks.json"; then + pass "hooks.json registers SessionStart with shell:bash dispatch" +else + fail "hooks.json registers SessionStart with shell:bash dispatch" +fi + claude_home="$(make_home claude-code)" assert_command_output \ "Claude Code emits nested SessionStart additionalContext" \ From 52f649e4ecc4f1e3673ecd3ea7e0be6fd15e0087 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Thu, 16 Jul 2026 00:14:13 +0000 Subject: [PATCH 40/51] docs(windows): document shell:bash hook dispatch and the PowerShell/CMD fallback hazards --- docs/windows/polyglot-hooks.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/windows/polyglot-hooks.md b/docs/windows/polyglot-hooks.md index 8b84f271..d1bcb84e 100644 --- a/docs/windows/polyglot-hooks.md +++ b/docs/windows/polyglot-hooks.md @@ -6,9 +6,18 @@ Claude Code plugins need hooks that work on Windows, macOS, and Linux. This docu ## The Problem -Claude Code runs hook commands through the system's default shell: -- **Windows**: CMD.exe +Claude Code runs hook commands through a shell: - **macOS/Linux**: bash or sh +- **Windows with Git Bash installed**: Git Bash +- **Windows without Git Bash**: PowerShell (older versions used CMD.exe) + +Neither Windows fallback shell can parse our command string: PowerShell treats +a leading quoted path as a string expression and errors on the next bareword, +and CMD.exe's `/c` quoting rules strip the outer quotes when the path contains +a metacharacter such as `(`. Our hooks therefore declare `"shell": "bash"` +(supported since Claude Code 2.1.81; older versions ignore the key), which +forces the Git Bash route and, when Git Bash is absent, produces an actionable +"install Git for Windows" error instead of a shell parser failure. This creates several challenges: @@ -42,6 +51,7 @@ hooks/ { "type": "command", "command": "\"${CLAUDE_PLUGIN_ROOT}/hooks/run-hook.cmd\" session-start", + "shell": "bash", "async": false } ] From 5d5b6567a804098dca9c09ea1d6bd4dfde6397bf Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Thu, 16 Jul 2026 03:32:25 +0000 Subject: [PATCH 41/51] fix(codex): make package script and its test portable beyond macOS/bsdtar The packaging pipeline only worked on a Mac with default umask, for three stacked reasons: - The deterministic-metadata tar flags (--uid/--gid/--uname/--gname) are bsdtar spellings; GNU tar rejects them, so the tar.gz archive step died on Linux. Detect the tar flavor and use --owner=:0 --group=:0 --numeric-owner on GNU tar, which writes byte-identical ustar headers (uid/gid 0, empty uname/gname). - Staged file modes depended on two umasks canceling out: git archive masks entry modes with tar.umask (git default 0002 -> 775), and the unflagged tar extraction re-masked with the process umask (022 on macOS -> 755, but 002 elsewhere -> 775). Pin tar.umask=0022 on the archive call and extract with -p so staged modes are canonical 755/644 on every machine. - The test's timestamp assertion parsed bsdtar's -tv column layout and expected epoch 0 rendered in a US timezone ("Dec 31 1969"); GNU tar uses different columns and UTC hosts render "1970-01-01". Assert mtime == 0 via python3 tarfile instead, matching how the test already checks zip timestamps. tests/codex/test-package-codex-plugin.sh now passes on Linux/GNU tar; the bsdtar branch preserves the exact flags that passed on macOS. --- scripts/package-codex-plugin.sh | 17 +++++++++++++---- tests/codex/test-package-codex-plugin.sh | 9 +++++++-- 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/scripts/package-codex-plugin.sh b/scripts/package-codex-plugin.sh index 00399f06..5308e28d 100755 --- a/scripts/package-codex-plugin.sh +++ b/scripts/package-codex-plugin.sh @@ -230,14 +230,16 @@ prepare_metadata_root() { METADATA_ROOT="$(prepare_metadata_root "$METADATA_SOURCE")" -git -C "$REPO_ROOT" archive --format=tar "$REF" -- \ +# Pin tar.umask and extract with -p so staged modes are canonical 755/644 +# regardless of the builder's git config or process umask. +git -C "$REPO_ROOT" -c tar.umask=0022 archive --format=tar "$REF" -- \ .codex-plugin \ CODE_OF_CONDUCT.md \ LICENSE \ README.md \ assets \ skills \ - | tar -xf - -C "$STAGE" + | tar -xpf - -C "$STAGE" VERSION="$(jq -r '.version // empty' "$STAGE/.codex-plugin/plugin.json")" [[ -n "$VERSION" ]] || die "could not read version from .codex-plugin/plugin.json" @@ -298,12 +300,19 @@ case "$FORMAT" in ) ;; tar.gz) - # Match the prior official archive's deterministic tar entry metadata. + # Match the prior official archive's deterministic tar entry metadata: + # ustar entries with uid/gid 0 and empty uname/gname. GNU tar and bsdtar + # (macOS) spell those flags differently. + if tar --version 2>/dev/null | grep -q 'GNU tar'; then + TAR_METADATA_FLAGS=(--owner=:0 --group=:0 --numeric-owner) + else + TAR_METADATA_FLAGS=(--uid 0 --gid 0 --uname '' --gname '') + fi TZ=UTC find "$STAGE" -exec touch -t 197001010000 {} + ( cd "$STAGE" rm -f "$OUTPUT" - COPYFILE_DISABLE=1 tar -cf - --no-recursion --format ustar --uid 0 --gid 0 --uname '' --gname '' -T "$ARCHIVE_LIST" | + COPYFILE_DISABLE=1 tar -cf - --no-recursion --format ustar "${TAR_METADATA_FLAGS[@]}" -T "$ARCHIVE_LIST" | gzip -9n >"$OUTPUT" ) ;; diff --git a/tests/codex/test-package-codex-plugin.sh b/tests/codex/test-package-codex-plugin.sh index 62c73f1c..947cb6db 100755 --- a/tests/codex/test-package-codex-plugin.sh +++ b/tests/codex/test-package-codex-plugin.sh @@ -210,8 +210,13 @@ assert_equals "$tar_archive_paths" "$archive_paths" "zip and tar.gz archives con tar_task_brief_mode="$(tar -tzvf "$tar_archive" skills/subagent-driven-development/scripts/task-brief | awk '{print $1}')" assert_equals "$tar_task_brief_mode" "-rwxr-xr-x" "tar.gz archive preserves executable script mode" -tar_metadata_times="$(tar -tzvf "$tar_archive" | awk '{print $6, $7, $8}' | sort -u)" -assert_equals "$tar_metadata_times" "Dec 31 1969" "tar.gz archive normalizes entry timestamps" +tar_metadata_times="$(python3 - "$tar_archive" <<'PY' +import sys, tarfile +with tarfile.open(sys.argv[1]) as archive: + print(sorted({member.mtime for member in archive.getmembers()})) +PY +)" +assert_equals "$tar_metadata_times" "[0]" "tar.gz archive normalizes entry timestamps" metadata_archive="$TEST_ROOT/metadata-source.tar.gz" metadata_zip="$TEST_ROOT/metadata-source.zip" From 0e13ad8222c67fe3bb7a2222e996dc34961847d9 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Thu, 16 Jul 2026 04:00:15 +0000 Subject: [PATCH 42/51] fix(tests): stop the SDD skill test flaking on timing and prose case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/claude-code/test-subagent-driven-development.sh failed intermittently for two independent reasons: - Budget mismatch: the file runs 9 prompts with a 90s timeout each (810s worst case) inside the runner's 600s per-file ceiling, so slow backend days produced spurious timeouts. Raise the runner default to 900s and fix the help text, which claimed the default was 300. - Case-sensitive prose matching: the assert helpers grepped free-form model output case-sensitively, but models capitalize the skill's own headings — observed failures include "Do Not Trust the Report" missing pattern "not trust" and a structured answer missing "First:.*spec.*compliance". Match case-insensitively in assert_contains/assert_not_contains/assert_count/assert_order, widen two Test 5 keyword patterns to phrasings observed in real runs, and make assert_order dump the output on failure the way assert_contains already does, so the next flake is diagnosable. Observed 3 failures across 4 runs before the change (timeout, two distinct pattern misses); 3/3 consecutive full runs pass after it. --- tests/claude-code/run-skill-tests.sh | 5 +++-- tests/claude-code/test-helpers.sh | 16 +++++++++++----- .../test-subagent-driven-development.sh | 4 ++-- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/tests/claude-code/run-skill-tests.sh b/tests/claude-code/run-skill-tests.sh index c94cfec9..83217cda 100755 --- a/tests/claude-code/run-skill-tests.sh +++ b/tests/claude-code/run-skill-tests.sh @@ -25,7 +25,8 @@ fi # Parse command line arguments VERBOSE=false SPECIFIC_TEST="" -TIMEOUT=600 # Default 10 minute timeout per test +TIMEOUT=900 # Per-test-file budget; must exceed the file's worst case + # (test-subagent-driven-development.sh: 9 prompts x 90s each) RUN_INTEGRATION=false while [[ $# -gt 0 ]]; do @@ -52,7 +53,7 @@ while [[ $# -gt 0 ]]; do echo "Options:" echo " --verbose, -v Show verbose output" echo " --test, -t NAME Run only the specified test" - echo " --timeout SECONDS Set timeout per test (default: 300)" + echo " --timeout SECONDS Set timeout per test (default: 900)" echo " --integration, -i Run integration tests (slow, 10-30 min)" echo " --help, -h Show this help" echo "" diff --git a/tests/claude-code/test-helpers.sh b/tests/claude-code/test-helpers.sh index 1b5ead3b..9e187610 100755 --- a/tests/claude-code/test-helpers.sh +++ b/tests/claude-code/test-helpers.sh @@ -30,12 +30,14 @@ run_claude() { # Check if output contains a pattern # Usage: assert_contains "output" "pattern" "test name" +# Matching is case-insensitive: patterns are prose keywords, and models +# freely capitalize skill terms ("Do Not Trust", "Spec Compliance"). assert_contains() { local output="$1" local pattern="$2" local test_name="${3:-test}" - if echo "$output" | grep -q "$pattern"; then + if echo "$output" | grep -qi "$pattern"; then echo " [PASS] $test_name" return 0 else @@ -54,7 +56,7 @@ assert_not_contains() { local pattern="$2" local test_name="${3:-test}" - if echo "$output" | grep -q "$pattern"; then + if echo "$output" | grep -qi "$pattern"; then echo " [FAIL] $test_name" echo " Did not expect to find: $pattern" echo " In output:" @@ -74,7 +76,7 @@ assert_count() { local expected="$3" local test_name="${4:-test}" - local actual=$(echo "$output" | grep -c "$pattern" || echo "0") + local actual=$(echo "$output" | grep -ci "$pattern" || echo "0") if [ "$actual" -eq "$expected" ]; then echo " [PASS] $test_name (found $actual instances)" @@ -98,16 +100,20 @@ assert_order() { local test_name="${4:-test}" # Get line numbers where patterns appear - local line_a=$(echo "$output" | grep -n "$pattern_a" | head -1 | cut -d: -f1) - local line_b=$(echo "$output" | grep -n "$pattern_b" | head -1 | cut -d: -f1) + local line_a=$(echo "$output" | grep -ni "$pattern_a" | head -1 | cut -d: -f1) + local line_b=$(echo "$output" | grep -ni "$pattern_b" | head -1 | cut -d: -f1) if [ -z "$line_a" ]; then echo " [FAIL] $test_name: pattern A not found: $pattern_a" + echo " In output:" + echo "$output" | sed 's/^/ /' return 1 fi if [ -z "$line_b" ]; then echo " [FAIL] $test_name: pattern B not found: $pattern_b" + echo " In output:" + echo "$output" | sed 's/^/ /' return 1 fi diff --git a/tests/claude-code/test-subagent-driven-development.sh b/tests/claude-code/test-subagent-driven-development.sh index d8f3e10c..151fc64d 100755 --- a/tests/claude-code/test-subagent-driven-development.sh +++ b/tests/claude-code/test-subagent-driven-development.sh @@ -96,13 +96,13 @@ echo "Test 5: Spec compliance reviewer mindset..." output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" "$CLAUDE_PROMPT_TIMEOUT") -if assert_contains "$output" "not trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then +if assert_contains "$output" "not.*trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then : # pass else exit 1 fi -if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code" "Reviewer reads code"; then +if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code\|read.*diff\|trust.*diff" "Reviewer reads code"; then : # pass else exit 1 From 55bbb52c7e0c5df4b459b0c62142ee120bac8c34 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Wed, 15 Jul 2026 11:11:02 -0700 Subject: [PATCH 43/51] docs(specs): SDD fix-loop redesign design spec Review-fix loop gets resume-the-implementer semantics, scoped re-reviews, a five-round circuit breaker, and controller adjudication at trip. SKILL.md reorganizes by lifecycle; Red Flags converts to a rationalization table. Brainstormed with Jesse 2026-07-15. --- ...2026-07-15-sdd-fix-loop-redesign-design.md | 196 ++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-15-sdd-fix-loop-redesign-design.md diff --git a/docs/superpowers/specs/2026-07-15-sdd-fix-loop-redesign-design.md b/docs/superpowers/specs/2026-07-15-sdd-fix-loop-redesign-design.md new file mode 100644 index 00000000..89e3e24b --- /dev/null +++ b/docs/superpowers/specs/2026-07-15-sdd-fix-loop-redesign-design.md @@ -0,0 +1,196 @@ +# SDD Fix-Loop Redesign — Design Spec + +**Status:** Approved design (brainstormed with Jesse 2026-07-15); implementation +plan to follow. +**Objective:** make the subagent-driven-development skill's review-fix loop +convergent and autonomous, and make the document readable, without rewriting +its eval-tuned language. +**Hard invariant:** existing eval-tuned sentences move; they do not get +reworded. New machinery ships with drill evidence. + +## Problems + +Four, all observed in real sessions: + +1. **Pathological review loops.** The loop is literally "Repeat until + approved" — no round cap. Each re-review is a fresh full review of the + whole diff, so a nondeterministic frontier reviewer surfaces new findings + every round instead of verifying fixes. Result: implement, review, fix, + review, review, fix, review, fix — with no circuit breaker. The + strict-cost spec (2026-06-10) independently measured review-loop count as + the biggest run-to-run cost variance. +2. **Contradictory fix policy.** The process diagram and "Constructing + Reviewer Prompts" dispatch dedicated fix subagents; Red Flags says + "Implementer (same subagent) fixes them"; implementer-prompt.md's "After + Review Findings" section assumes the implementer will be re-engaged. Three + answers to "who fixes?" in one skill. +3. **Accreted structure.** Thirteen top-level sections; guidance for one + activity is scattered across four of them. "Constructing Reviewer Prompts" + is a grab-bag holding reviewer guidance, fix policy, final-review policy, + and plan-conflict adjudication. +4. **Red Flags format.** Seven sibling skills use the `| Excuse | Reality |` + rationalization table; SDD carries a 17-bullet "Never" list plus three + "If X" mini-blocks. + +## Design Decisions + +| # | Decision | Rationale | +|---|----------|-----------| +| 1 | The original implementer fixes its own review findings — resume it in place. | It already holds the task context; ownership beats a drive-by patcher. Fresh "fix subagents" rebuild context per finding and lack the task frame. | +| 2 | Re-reviews are scoped to the findings. | Fresh full reviews each round are the churn engine. Scoped re-reviews make the loop structurally convergent; the final whole-branch review remains the broad safety net. | +| 3 | Circuit breaker at five fix rounds: three resumes, then two fresh dispatches on a more capable model. | Jesse's call. A loop that survives three resumes usually means the implementer cannot see its own problem — the fresh capable dispatch de-anchors and capability-bumps in one move. | +| 4 | At trip, the controller adjudicates and routes. No new human checkpoint — structural failures reach the existing BLOCKED stop. | SDD's point is autonomous execution. The controller holds the plan and cross-task context the reviewer lacks; the existing text already sanctions it ("adjudicate it in the review loop") without ever specifying the mechanism. | +| 5 | Reorganize SKILL.md by lifecycle, preserving tuned sentences. | Fixes "hard to follow" at the root. Content moves to its point of use, matching the house direction (recent commits fold recap sections into points of use). | +| 6 | Convert Red Flags to a `| Excuse | Reality |` rationalization table; relocate hard rules to their points of use. | Matches the other seven skills. Excuses get rebuttals; rules get enforced where the reader acts. | + +## The Fix Loop + +Trigger: a task review returns spec ❌ or any Critical/Important finding. + +**Rounds 1–3 — resume the original implementer.** Send the findings verbatim +(Critical/Important plus spec gaps). The implementer fixes, re-runs the +covering tests, appends the fix report to its existing report file, and +returns the short contract. On a harness without agent resume, a "resume" is +a fresh dispatch carrying the brief, the report file, and the findings — the +report file is the persistent memory either way. + +**Rounds 4–5 — fresh implementer, more capable model.** Full task context: +brief, report file, open findings, and the framing "a prior implementer +attempted this N times; you own the task now." + +**Every round's re-review is scoped.** The re-reviewer receives the brief, +the updated report, the original findings list, and a fix-scoped diff package +(`review-package FIX_BASE HEAD`, where FIX_BASE is the head the reviewer +last reviewed; the script already takes arbitrary ranges). +It verdicts each finding addressed / not addressed and flags new breakage in +the fix diff only. Novel findings on code the fix did not touch are reported +as non-blocking; the controller ledgers them for the final review. + +**Fix-report completeness gate (existing rule, kept):** before dispatching a +re-review, confirm the fix report names the covering tests, the command run, +and the output. + +**No early exit.** The controller never adjudicates before the cap — an early +exit reopens the "pre-judge findings to spare yourself a review loop" hole +the current content deliberately closed. One exception, unchanged from +today: a finding that conflicts with what the plan's text mandates goes to +the human immediately (plan authority, not loop churn). + +**Minor findings** never enter the loop: ledger them as they arrive (existing +rule, kept). + +### Adjudication at Trip + +After round five fails, the controller stops dispatching and judges each open +finding against the brief, the plan, and cross-task context: + +- **Contested or wrong** → ledger with a one-line adjudication ("controller: + reviewer wrong because X"), continue. The final review sees both sides. +- **Real, not load-bearing** → ledger as known-open, continue. Later + dispatches touching that area carry a pointer to the entry. +- **Real and load-bearing** (later tasks build on it, or it reveals a plan + defect) → the existing BLOCKED stop. Park-and-continue defers a structural + failure to the most expensive point and lets dependents build on it, so + structural failures stop the run — through the stop condition that already + exists, not a new checkpoint. + +Every adjudication is a ledger entry. Silent discards stay forbidden. + +## Document Restructure + +New skeleton, in execution order: + +1. Intro — why subagents, core principle, narration, continuous execution +2. When to Use — unchanged, including the decision graph +3. The Process — diagram updated for the new loop +4. Setup — worktree, ledger check/resume, pre-flight plan review, todos +5. Model Selection — stays one cross-cutting section; every dispatch + consults it, so folding it into points of use would repeat it five times +6. The Task Loop — five numbered steps: + 1. Dispatch the implementer (task-brief script, five-part dispatch + composition, model line required) + 2. Handle the report (DONE / DONE_WITH_CONCERNS / NEEDS_CONTEXT / BLOCKED) + 3. Review the task (review-package script, reviewer dispatch composition, + constraints lens, no pre-judging, ⚠️ handling) + 4. Fix loop (the machinery above) + 5. Complete the task (ledger append, todo update) +7. Final Review — package, model pin, one fix wave, one scoped re-review, + adjudication +8. Finish — finishing-a-development-branch +9. Common Rationalizations — the table +10. Example Workflow — updated to show a resume-based fix round and the + breaker not tripping + +"Constructing Reviewer Prompts," "File Handoffs," and "Durable Progress" +dissolve into the steps where each rule applies. Every eval-tuned sentence +lands in exactly one new location; a move map in the implementation plan +tracks source → destination so review can verify nothing was dropped or +reworded. + +## Rationalization Table + +Excuse-shaped Never items convert to rows; new rows cover the loop +pathology. Draft rows (final wording at implementation): + +| Excuse | Reality | +|--------|---------| +| "Close enough on spec compliance" | Reviewer found gaps = not done. | +| "I'll fix it myself, dispatching is overhead" | Controller fixes pollute your context and skip review. Resume the implementer. | +| "One more round will converge" | Past the cap, rounds don't converge. Adjudicate. | +| "The reviewer will just find something new anyway" | Scoped re-reviews check fixes, not taste. New findings on untouched code go to the ledger, not the loop. | +| "This finding is obviously wrong, I'll drop it" | You adjudicate only at the cap, and every adjudication is a ledger entry. Silent discards are forbidden. | +| "The fix was small, skip the re-review" | Unreviewed fixes are how regressions land. | + +Hard rules that are not excuses (never parallel implementers, never dispatch +a reviewer without a diff file, model line required, never re-dispatch +ledger-complete tasks) move to their points of use. + +## Prompt Templates + +- **implementer-prompt.md** — "After Review Findings" rewritten for resume + semantics: you will be resumed with findings; fix, re-run covering tests, + append to your report file, return the short contract. +- **task-reviewer-prompt.md** — initial review only; the trailing re-review + sentence moves out. +- **re-review-prompt.md (new)** — the scoped re-review contract: inputs are + brief, updated report, original findings, fix-scoped diff package; output + is a per-finding verdict (addressed / not addressed), new breakage in the + fix diff, and non-blocking observations outside it. A separate template + because it is a different contract — overloading the full-review template + produced the current ambiguity. +- **Takeover dispatch (rounds 4–5)** — composed from implementer-prompt.md + plus SKILL.md guidance (brief, report path, open findings, takeover + framing); no new template file. + +## Final Review Loop + +Unchanged: merge-base package, most capable model, ONE fixer with the +complete findings list. New: exactly one scoped re-review of the fix wave, +then controller adjudication. Residual load-bearing findings surface at +finishing-a-development-branch, where the human already is. The end of the +branch gets a bounded loop too. + +## Evals + +Three new drill scenarios in `evals/`: + +1. **Resume, don't re-dispatch:** a task review returns findings; the + controller must resume the same implementer rather than dispatch a fix + subagent. +2. **Breaker trips:** a seeded never-satisfied reviewer; the controller must + stop dispatching after the fifth round fails, adjudicate, ledger, and + continue — not loop. +3. **Structural finding stops:** a load-bearing finding (later tasks depend + on it); the controller must stop via BLOCKED rather than park. + +Plus before/after runs of the existing SDD scenarios to catch regressions +from the reorganization. + +## Non-Goals + +- Ledger session-scoping — PR #1943 owns it. This work touches the same + sections, so the implementation plan notes the collision risk. +- Script changes — task-brief and review-package already do what the new + loop needs. +- Changes to executing-plans or requesting-code-review beyond the final- + review pointer continuing to resolve. From c6fa27e3f80adc29020b7bb2bab984f39ce73d19 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Wed, 15 Jul 2026 11:28:08 -0700 Subject: [PATCH 44/51] docs(plans): SDD fix-loop redesign implementation plan Eight tasks across two repos: new re-review template, template/reference alignment, full SKILL.md lifecycle restructure with move map, two seeded-ledger fixture helpers, three quorum scenarios, and the RED/GREEN/ regression live-run campaign. --- .../plans/2026-07-15-sdd-fix-loop-redesign.md | 1649 +++++++++++++++++ 1 file changed, 1649 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-15-sdd-fix-loop-redesign.md diff --git a/docs/superpowers/plans/2026-07-15-sdd-fix-loop-redesign.md b/docs/superpowers/plans/2026-07-15-sdd-fix-loop-redesign.md new file mode 100644 index 00000000..93fdc82e --- /dev/null +++ b/docs/superpowers/plans/2026-07-15-sdd-fix-loop-redesign.md @@ -0,0 +1,1649 @@ +# SDD Fix-Loop Redesign Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make subagent-driven-development's review-fix loop convergent and autonomous (resume-the-implementer fix rounds, scoped re-reviews, five-round breaker, controller adjudication) and reorganize its SKILL.md by lifecycle — with quorum eval evidence. + +**Architecture:** Two repos. The `superpowers` repo (branch `sdd-fix-loop-redesign`, already created; spec committed) gets the skill restructure: one new prompt template, two template edits, one reference edit, and the SKILL.md rewrite whose full text is in Task 3. The `superpowers-evals` repo (`evals/` checkout; create branch `sdd-fix-loop-scenarios` off `main`) gets two seeded-ledger fixture helpers and three scenarios, then a live before/after campaign. + +**Tech Stack:** Markdown skill content; Bash scenario DSL (`story.md`/`setup.sh`/`checks.sh`); TypeScript setup-helpers on Bun (`bun test`); quorum live runs. + +**Design spec:** `docs/superpowers/specs/2026-07-15-sdd-fix-loop-redesign-design.md`. Read it before starting any task. + +## Global Constraints + +- **Verbatim-move rule:** eval-tuned sentences from the current SKILL.md move unchanged. Only fix-policy language may be reworded, and every rewording appears in Task 3's move map. Do not "improve" moved prose. +- **Round cap:** 5 fix rounds per task. Rounds 1–3 resume the original implementer; rounds 4–5 dispatch a fresh implementer on a more capable model. Adjudication happens only at the cap; the one earlier exit is a finding that conflicts with plan text (human decides, existing behavior). +- **Ledger line formats** (exact — scenarios grep for these; `` = 7-char short SHA): + - `Task : complete (commits .., review clean)` + - `Task : complete (commits .., parked)` + - `Task : fix round /5 ( addressed, open — [; …]; commits ..)` + - `Task : minor (deferred): ` + - `Task : parked — — ruling: ` + - `Task : BLOCKED — ` + - Resume rule: a task is DONE iff it has a `Task : complete` line. +- **Template placeholders** keep the existing bracket convention: `[MODEL]`, `[BRIEF_FILE]`, `[REPORT_FILE]`, `[BASE_SHA]`, `[HEAD_SHA]`, `[DIFF_FILE]`, `[GLOBAL_CONSTRAINTS]`; the new re-review template adds `[FINDINGS]`, `[FIX_BASE_SHA]`. +- **Commit discipline:** superpowers commits on `sdd-fix-loop-redesign`; evals commits on `sdd-fix-loop-scenarios` (separate repo — `cd evals` first). Never commit one repo's work from the other. +- **Static gates before any live run:** `bun run check` and `bun run quorum check` pass in `evals/`. +- **Live runs are trusted-maintainer operations** — they need `SUPERPOWERS_ROOT`, an `ANTHROPIC_API_KEY`, and cost real money (~$3–15 per SDD run). Task 8 marks them explicitly. +- **Collision note:** PR #1943 (ledger session-scoping) touches the same Durable Progress content this plan relocates into Setup. Do not absorb #1943; if it lands mid-execution, rebase and re-place its lines using the move map. + +## File Structure + +**superpowers repo:** +- Create: `skills/subagent-driven-development/re-review-prompt.md` — scoped re-review contract (Task 1) +- Modify: `skills/subagent-driven-development/implementer-prompt.md` — resume semantics (Task 2) +- Modify: `skills/subagent-driven-development/task-reviewer-prompt.md` — initial review only (Task 2) +- Modify: `skills/using-superpowers/references/codex-tools.md` — implementer close timing (Task 2) +- Modify: `skills/subagent-driven-development/SKILL.md` — full restructure (Task 3) + +**superpowers-evals repo (`evals/`):** +- Modify: `src/setup-helpers/sdd-fixtures.ts` — add `scaffoldSddMidloopParked`, `scaffoldSddMidloopStructural` (Task 4) +- Modify: `src/setup-helpers/registry.ts` — register both helpers (Task 4) +- Modify: `test/setup-helpers-sdd.test.ts` — unit tests for both helpers (Task 4) +- Create: `scenarios/sdd-fix-loop-resumes-implementer/{story.md,setup.sh,checks.sh}` (Task 5) +- Create: `scenarios/sdd-breaker-adjudicates-at-cap/{story.md,setup.sh,checks.sh}` (Task 6) +- Create: `scenarios/sdd-breaker-structural-blocks/{story.md,setup.sh,checks.sh}` (Task 7) +- Create: `docs/experiments/2026-07-sdd-fix-loop-redesign.md` — campaign log (Task 8) + +--- + +### Task 1: Create the scoped re-review template + +**Files:** +- Create: `skills/subagent-driven-development/re-review-prompt.md` + +**Interfaces:** +- Produces: template placeholders `[MODEL]`, `[BRIEF_FILE]`, `[REPORT_FILE]`, `[FINDINGS]`, `[FIX_BASE_SHA]`, `[HEAD_SHA]`, `[DIFF_FILE]` — Task 3's SKILL.md step 4 links this file and instructs the controller to fill exactly these. + +- [ ] **Step 1: Write the file with exactly this content** + +````markdown +# Scoped Re-Review Prompt Template + +Use this template when dispatching a re-review after a fix round. The +re-reviewer verifies the findings were addressed and checks the fix diff for +new breakage. It is not a fresh review — the full review already happened. + +**Purpose:** Verify each finding from the previous review was addressed, and +that the fix itself broke nothing. + +``` +Subagent (general-purpose): + description: "Re-review Task N fix round R" + model: [MODEL — REQUIRED: choose per SKILL.md Model Selection; an omitted + model silently inherits the session's most expensive one] + prompt: | + You are re-reviewing one task's fix round. A previous review produced + findings; an implementer has attempted to fix them. Your job is to + verdict each finding and inspect the fix diff — nothing else. + + ## The Task + + Read the task brief: [BRIEF_FILE] + + ## The Findings Under Verification + + [FINDINGS] + + ## The Fix + + Read the implementer's report (fix reports are appended at the end): + [REPORT_FILE] + + **Fix base:** [FIX_BASE_SHA] (the head the previous review saw) + **Head:** [HEAD_SHA] + **Diff file:** [DIFF_FILE] + + Read the diff file once — it contains the fix commits, a stat summary, + and the fix diff with surrounding context. Do not re-run git commands. + If the diff file is missing, fetch the diff yourself: + `git diff --stat [FIX_BASE_SHA]..[HEAD_SHA]` and + `git diff [FIX_BASE_SHA]..[HEAD_SHA]`. + + Your review is read-only on this checkout. Do not mutate the working + tree, the index, HEAD, or branch state in any way. + + ## Scope + + Your scope is the findings list and the fix diff. Verdict every finding. + Inspect the fix diff for new problems the fix itself introduced. Do NOT + re-review code the fix did not touch: if you notice an issue entirely + outside the fix diff, report it under Out-of-Scope Observations — it + does not block this task and does not extend the loop. A broad + whole-branch review happens after all tasks are complete. + + ## Tests + + The implementer re-ran the tests covering the amended code and appended + the results to the report file. Treat the report as unverified claims: + confirm the fix report names the covering tests and shows their output, + and verify the claims against the diff. Do not re-run the suite to + confirm their report. Run a test only when reading the code raises a + specific doubt that no existing run answers — and then a focused test, + never a package-wide suite. + + ## Output Format + + Your final message is the report itself: begin directly with the first + finding's verdict. Every line is a verdict, a finding with file:line, + or a check you ran — no preamble, no process narration. + + ### Finding Verdicts + + For each finding in The Findings Under Verification, in order: + - **[finding one-liner]** — ADDRESSED | NOT ADDRESSED, with file:line + evidence. "Attempted" is not addressed: the specific defect must no + longer exist. + + ### New Breakage in the Fix Diff + + Anything the fix itself broke or introduced, with severity + (Critical/Important/Minor) and file:line. "None" if clean. + + ### Out-of-Scope Observations + + Issues you noticed entirely outside the fix diff. Non-blocking; the + controller ledgers these for the final review. "None" if none. + + ### Verdict + + **Fix round:** [All findings addressed, no new Critical/Important + breakage | Findings remain open] — list the open ones. +``` + +**Placeholders:** +- `[MODEL]` — REQUIRED: reviewer model per SKILL.md Model Selection; scoped + re-reviews of small fix diffs take a cheap-to-mid tier +- `[BRIEF_FILE]` — the task brief file (same file the implementer worked from) +- `[FINDINGS]` — the Critical/Important findings and spec gaps from the + previous review, copied verbatim, one per bullet +- `[REPORT_FILE]` — the implementer's report file (fix reports appended) +- `[FIX_BASE_SHA]` — the head the previous review saw +- `[HEAD_SHA]` — current commit +- `[DIFF_FILE]` — the path `scripts/review-package FIX_BASE HEAD` printed + +**Re-reviewer returns:** per-finding verdicts (ADDRESSED / NOT ADDRESSED), +new breakage in the fix diff, out-of-scope observations, and a round verdict. +```` + +- [ ] **Step 2: Verify the file parses as the other templates do** + +Run: `grep -c '^\[MODEL' skills/subagent-driven-development/re-review-prompt.md` +Expected: `0` (placeholder docs use `- \`[MODEL]\`` list form, matching task-reviewer-prompt.md) + +Run: `grep -n 'FIX_BASE_SHA' skills/subagent-driven-development/re-review-prompt.md | head -3` +Expected: hits in the prompt body and the placeholder list. + +- [ ] **Step 3: Commit** + +```bash +git add skills/subagent-driven-development/re-review-prompt.md +git commit -m "feat(sdd): add scoped re-review prompt template" +``` + +--- + +### Task 2: Align the implementer and task-reviewer templates and the Codex reference with resume semantics + +**Files:** +- Modify: `skills/subagent-driven-development/implementer-prompt.md` (the "After Review Findings" section) +- Modify: `skills/subagent-driven-development/task-reviewer-prompt.md` (trailing paragraph) +- Modify: `skills/using-superpowers/references/codex-tools.md` (subagent close timing) + +**Interfaces:** +- Consumes: `re-review-prompt.md` exists (Task 1). +- Produces: the implementer contract Task 3's fix loop cites ("fix, re-run covering tests, append to your report file, return the short contract"). + +- [ ] **Step 1: Replace the "After Review Findings" section in implementer-prompt.md** + +Old text (exact): + +```markdown + ## After Review Findings + + If a reviewer finds issues and you fix them, re-run the tests that cover + the amended code and append the results to your report file. Reviewers + will not re-run tests for you — your report is the test evidence. +``` + +New text (exact): + +```markdown + ## After Review Findings + + If the task review finds issues, you will be resumed with the findings. + Fix them, re-run the tests that cover the amended code, and append a fix + report to your report file: what you changed, the covering tests you + ran, the command, and the output. Reviewers will not re-run tests for + you — your report is the test evidence. Then reply with the same short + status contract as your first report. +``` + +- [ ] **Step 2: Delete the trailing re-review paragraph in task-reviewer-prompt.md** + +Delete this text (exact, at end of file): + +```markdown +A fix dispatch can address spec gaps and quality findings together; +re-review after fixes covers both verdicts. +``` + +Nothing replaces it — the scoped re-review contract now lives in +`re-review-prompt.md`, and SKILL.md step 4 (Task 3) owns the loop rules. + +- [ ] **Step 3: Update the Codex subagent close-timing sentence in codex-tools.md** + +Old text (exact, line 10): + +```markdown +When using subagent-driven-development, you should always close implementer and reviewer subagents when they have finished all their work. +``` + +New text (exact): + +```markdown +When using subagent-driven-development, close reviewer subagents when their review returns. Keep each implementer subagent open until its task's review passes — the fix loop resumes the implementer — then close it. If your harness cannot send another message to a spawned agent, dispatch each fix round as a fresh implementer carrying the brief, the report file, and the findings. +``` + +- [ ] **Step 4: Verify no template still references dedicated fix subagents** + +Run: `grep -rn "fix subagent" skills/subagent-driven-development/*.md` +Expected: no output (SKILL.md still has hits until Task 3 — this command scopes to templates only after Task 3; at this point expect hits ONLY in SKILL.md). + +Run: `grep -rn "fix subagent" skills/subagent-driven-development/implementer-prompt.md skills/subagent-driven-development/task-reviewer-prompt.md skills/subagent-driven-development/re-review-prompt.md` +Expected: no output. + +- [ ] **Step 5: Commit** + +```bash +git add skills/subagent-driven-development/implementer-prompt.md skills/subagent-driven-development/task-reviewer-prompt.md skills/using-superpowers/references/codex-tools.md +git commit -m "feat(sdd): align templates and codex reference with resume-based fix rounds" +``` + +--- + +### Task 3: Restructure SKILL.md by lifecycle with the fix loop and rationalization table + +**Files:** +- Modify: `skills/subagent-driven-development/SKILL.md` (full-file replacement; new text below) + +**Interfaces:** +- Consumes: all three templates (Tasks 1–2); `scripts/task-brief`, `scripts/review-package`, `scripts/sdd-workspace` (unchanged). +- Produces: the ledger line formats in Global Constraints (scenarios grep them); section names `Setup`, `The Task Loop`, `Final Review`, `Common Rationalizations`. + +- [ ] **Step 1: Replace the entire SKILL.md body with exactly this content** + +`````markdown +--- +name: subagent-driven-development +description: Use when executing implementation plans with independent tasks in the current session +--- + +# Subagent-Driven Development + +Execute plan by dispatching a fresh implementer subagent per task, a task review (spec compliance + code quality) after each, and a broad whole-branch review at the end. + +**Why subagents:** You delegate tasks to specialized agents with isolated context. By precisely crafting their instructions and context, you ensure they stay focused and succeed at their task. They should never inherit your session's context or history — you construct exactly what they need. This also preserves your own context for coordination work. + +**Core principle:** Fresh subagent per task + task review (spec + quality) + broad final review = high quality, fast iteration + +**Narration:** between tool calls, narrate at most one short line — the +ledger and the tool results carry the record. + +**Continuous execution:** Do not pause to check in with your human partner between tasks. Execute all tasks from the plan without stopping. The only reasons to stop are: BLOCKED status you cannot resolve, ambiguity that genuinely prevents progress, or all tasks complete. "Should I continue?" prompts and progress summaries waste their time — they asked you to execute the plan, so execute it. + +## When to Use + +```dot +digraph when_to_use { + "Have implementation plan?" [shape=diamond]; + "Tasks mostly independent?" [shape=diamond]; + "Stay in this session?" [shape=diamond]; + "subagent-driven-development" [shape=box]; + "executing-plans" [shape=box]; + "Manual execution or brainstorm first" [shape=box]; + + "Have implementation plan?" -> "Tasks mostly independent?" [label="yes"]; + "Have implementation plan?" -> "Manual execution or brainstorm first" [label="no"]; + "Tasks mostly independent?" -> "Stay in this session?" [label="yes"]; + "Tasks mostly independent?" -> "Manual execution or brainstorm first" [label="no - tightly coupled"]; + "Stay in this session?" -> "subagent-driven-development" [label="yes"]; + "Stay in this session?" -> "executing-plans" [label="no - parallel session"]; +} +``` + +**vs. Executing Plans (parallel session):** +- Same session (no context switch) +- Fresh subagent per task (no context pollution) +- Review after each task (spec compliance + code quality), broad review at the end +- Faster iteration (no human-in-loop between tasks) + +## The Process + +```dot +digraph process { + rankdir=TB; + + subgraph cluster_per_task { + label="Per Task"; + "Dispatch implementer subagent (./implementer-prompt.md)" [shape=box]; + "Implementer asks questions?" [shape=diamond]; + "Answer questions, provide context" [shape=box]; + "Implementer implements, tests, commits, self-reviews" [shape=box]; + "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)" [shape=box]; + "Spec ✅ and quality approved?" [shape=diamond]; + "Finding conflicts with plan text?" [shape=diamond]; + "Ask human partner which governs" [shape=box]; + "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [shape=box]; + "Dispatch scoped re-review (./re-review-prompt.md)" [shape=box]; + "All findings addressed?" [shape=diamond]; + "R = 5?" [shape=diamond]; + "Adjudicate each open finding" [shape=box]; + "Any load-bearing finding?" [shape=diamond]; + "STOP: report BLOCKED to human partner" [shape=box]; + "Park findings in ledger with rulings" [shape=box]; + "Append completion to ledger, mark todo complete" [shape=box]; + } + + "Setup: worktree, ledger check, read plan, pre-flight review" [shape=box]; + "More tasks remain?" [shape=diamond]; + "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals" [shape=box]; + "Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen]; + + "Setup: worktree, ledger check, read plan, pre-flight review" -> "Dispatch implementer subagent (./implementer-prompt.md)"; + "Dispatch implementer subagent (./implementer-prompt.md)" -> "Implementer asks questions?"; + "Implementer asks questions?" -> "Answer questions, provide context" [label="yes"]; + "Answer questions, provide context" -> "Implementer implements, tests, commits, self-reviews"; + "Implementer asks questions?" -> "Implementer implements, tests, commits, self-reviews" [label="no"]; + "Implementer implements, tests, commits, self-reviews" -> "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)"; + "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)" -> "Spec ✅ and quality approved?"; + "Spec ✅ and quality approved?" -> "Append completion to ledger, mark todo complete" [label="yes"]; + "Spec ✅ and quality approved?" -> "Finding conflicts with plan text?" [label="no"]; + "Finding conflicts with plan text?" -> "Ask human partner which governs" [label="yes"]; + "Ask human partner which governs" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model"; + "Finding conflicts with plan text?" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [label="no"]; + "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" -> "Dispatch scoped re-review (./re-review-prompt.md)"; + "Dispatch scoped re-review (./re-review-prompt.md)" -> "All findings addressed?"; + "All findings addressed?" -> "Append completion to ledger, mark todo complete" [label="yes"]; + "All findings addressed?" -> "R = 5?" [label="no"]; + "R = 5?" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [label="no - next round"]; + "R = 5?" -> "Adjudicate each open finding" [label="yes - breaker trips"]; + "Adjudicate each open finding" -> "Any load-bearing finding?"; + "Any load-bearing finding?" -> "STOP: report BLOCKED to human partner" [label="yes"]; + "Any load-bearing finding?" -> "Park findings in ledger with rulings" [label="no"]; + "Park findings in ledger with rulings" -> "Append completion to ledger, mark todo complete"; + "Append completion to ledger, mark todo complete" -> "More tasks remain?"; + "More tasks remain?" -> "Dispatch implementer subagent (./implementer-prompt.md)" [label="yes"]; + "More tasks remain?" -> "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" [label="no"]; + "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" -> "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals"; + "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals" -> "Use superpowers:finishing-a-development-branch"; +} +``` + +## Setup + +Ensure the work happens in an isolated workspace: use +superpowers:using-git-worktrees to create one or verify the existing one. +Never start implementation on a main/master branch without your human +partner's explicit consent. + +Conversation memory does not survive compaction. In real sessions, +controllers that lost their place have re-dispatched entire completed task +sequences — the single most expensive failure observed. Track progress in +a ledger file, not only in todos. + +- At skill start, check for a ledger: + `cat "$(git rev-parse --show-toplevel)/.superpowers/sdd/progress.md"`. Tasks with + a `Task : complete` line are DONE — do not re-dispatch them; resume at + the first task without one. A task whose last line is a fix round is + mid-loop: resume the loop at the next round. +- The ledger is your recovery map: the commits it names exist in git even + when your context no longer remembers creating them. After compaction, + trust the ledger and `git log` over your own recollection. +- `git clean -fdx` will destroy the ledger (it's git-ignored scratch); if + that happens, recover from `git log`. + +Read the plan once, note its context and Global Constraints, and create a +todo per task. + +Before dispatching Task 1, scan the plan once for conflicts: + +- tasks that contradict each other or the plan's Global Constraints +- anything the plan explicitly mandates that the review rubric treats as a + defect (a test that asserts nothing, verbatim duplication of a logic block) + +Present everything you find to your human partner as one batched question — +each finding beside the plan text that mandates it, asking which governs — +before execution begins, not one interrupt per discovery mid-plan. If the +scan is clean, proceed without comment. The review loop remains the net for +conflicts that only emerge from implementation. + +## Model Selection + +Use the least powerful model that can handle each role to conserve cost and increase speed. + +**Mechanical implementation tasks** (isolated functions, clear specs, 1-2 files): use a fast, cheap model. Most implementation tasks are mechanical when the plan is well-specified. + +**Integration and judgment tasks** (multi-file coordination, pattern matching, debugging): use a standard model. + +**Architecture and design tasks**: use the most capable available model. +The final whole-branch review is one of these — dispatch it on the most +capable available model, not the session default. + +**Review tasks**: choose the model with the same judgment, scaled to the +diff's size, complexity, and risk. A small mechanical diff does not need the +most capable model; a subtle concurrency change does. Scoped re-reviews of +small fix diffs take a cheap-to-mid tier. + +**Fix-loop escalation (rounds 4-5)**: use a model at least one tier above +the implementer that got stuck. + +**Always specify the model explicitly when dispatching a subagent.** An +omitted model inherits your session's model — often the most capable and +most expensive — which silently defeats this section. + +**Turn count beats token price.** Wall-clock and context cost scale with how +many turns a subagent takes, and the cheapest models routinely take 2-3× the +turns on multi-step work — costing more overall. Use a mid-tier model as the +floor for reviewers and for implementers working from prose descriptions. +When the task's plan text contains the complete code to write, the +implementation is transcription plus testing: use the cheapest tier for +that implementer. Single-file mechanical fixes also take the cheapest tier. + +**Task complexity signals (implementation tasks):** +- Touches 1-2 files with a complete spec → cheap model +- Touches multiple files with integration concerns → standard model +- Requires design judgment or broad codebase understanding → most capable model + +## The Task Loop + +Everything you paste into a dispatch prompt — and everything a subagent +prints back — stays resident in your context for the rest of the session +and is re-read on every later turn. Hand artifacts over as files. + +### 1. Dispatch the implementer + +Record BASE (`git rev-parse HEAD`) before dispatching — the review package +and fix-round diffs need it. + +- **Task brief:** run this skill's `scripts/task-brief PLAN_FILE N` — it + extracts the task's full text to a uniquely named file and prints the + path. Compose the dispatch so the brief stays the single source of + requirements. Your dispatch should contain: (1) one line on where this + task fits in the project; (2) the brief path, introduced as "read this + first — it is your requirements, with the exact values to use verbatim"; + (3) interfaces and decisions from earlier tasks that the brief cannot + know; (4) your resolution of any ambiguity you noticed in the brief; + (5) the report-file path and report contract. Exact values (numbers, + magic strings, signatures, test cases) appear only in the brief. Never + make a subagent read the whole plan file. +- **Report file:** name the implementer's report file after the brief + (brief `…/task-N-brief.md` → report `…/task-N-report.md`) and put it in + the dispatch prompt. The implementer writes the full report there and + returns only status, commits, a one-line test summary, and concerns. +- A dispatch prompt describes one task, not the session's history. Do not + paste accumulated prior-task summaries ("state after Tasks 1-3") into + later dispatches — a real session's dispatch hit 42k chars of which 99% + was pasted history. A fresh subagent needs its task, the interfaces it + touches, and the global constraints. Nothing else. +- If an earlier task parked a finding in the area this task touches, carry + a pointer to that ledger entry in the dispatch. +- Record the implementer's agent identity from the dispatch result — + fix-loop rounds 1-3 resume this agent. +- Never dispatch multiple implementation subagents in parallel (conflicts). + +Template: [implementer-prompt.md](implementer-prompt.md) + +### 2. Handle the report + +Implementer subagents report one of four statuses. Handle each appropriately: + +**DONE:** Generate the review package (`scripts/review-package BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path. + +**DONE_WITH_CONCERNS:** The implementer completed the work but flagged doubts. Read the concerns before proceeding. If the concerns are about correctness or scope, address them before review. If they're observations (e.g., "this file is getting large"), note them and proceed to review. + +**NEEDS_CONTEXT:** The implementer needs information that wasn't provided. Provide the missing context and re-dispatch. + +**BLOCKED:** The implementer cannot complete the task. Assess the blocker: +1. If it's a context problem, provide more context and re-dispatch with the same model +2. If the task requires more reasoning, re-dispatch with a more capable model +3. If the task is too large, break it into smaller pieces +4. If the plan itself is wrong, escalate to the human + +**Never** ignore an escalation or force the same model to retry without changes. If the implementer said it's stuck, something needs to change. + +If the implementer asks questions — before starting or mid-task — answer +clearly and completely, provide additional context if needed, and don't +rush it into implementation. + +### 3. Review the task + +Per-task reviews are task-scoped gates. The broad review happens once, at the +final whole-branch review. Never skip the task review, and never accept a +report missing either verdict — spec compliance AND task quality are both +required. Implementer self-review never replaces the task review; both are +needed. + +- Hand the reviewer its diff as a file: run this skill's + `scripts/review-package BASE HEAD` and pass the reviewer the file path + it prints (or, without bash: `git log --oneline`, `git diff --stat`, + and `git diff -U10` for the range, redirected to one uniquely named + file). The output never enters your own context, and the reviewer sees + the commit list, stat summary, and full diff with context in one Read + call. Use the BASE you recorded before dispatching the implementer — + never `HEAD~1`, which silently truncates multi-commit tasks. Never + dispatch a task reviewer without a diff file. +- The task reviewer gets three paths — the same brief file, the report + file, and the review package — plus the global constraints that bind + the task. +- The global-constraints block you hand the reviewer is its attention + lens. Copy the binding requirements verbatim from the plan's Global + Constraints section or the spec: exact values, exact formats, and the + stated relationships between components ("same layout as X", "matches + Y"). The reviewer's template already carries the process rules (YAGNI, + test hygiene, review method) — the constraints block is for what THIS + project's spec demands. +- Do not add open-ended directives like "check all uses" or "run race tests + if useful" without a concrete, task-specific reason +- Do not ask a reviewer to re-run tests the implementer already ran on the + same code — the implementer's report carries the test evidence +- Do not pre-judge findings for the reviewer — never instruct a reviewer to + ignore or not flag a specific issue. If you believe a finding would be a + false positive, let the reviewer raise it and adjudicate it in the review + loop. If the prompt you are writing contains "do not flag," "don't treat X + as a defect," "at most Minor," or "the plan chose" — stop: you are + pre-judging, usually to spare yourself a review loop. + +The task reviewer may report "⚠️ Cannot verify from diff" items — requirements +that live in unchanged code or span tasks. These do not block the rest of the +review, but you must resolve each one yourself before marking the task +complete: you hold the plan and cross-task context the reviewer +lacks. If you confirm an item is a real gap, treat it as a failed spec +review — it enters the fix loop with the other findings. + +Template: [task-reviewer-prompt.md](task-reviewer-prompt.md) + +### 4. The fix loop + +The loop triggers when the review reports spec ❌, any Critical or Important +finding, or a ⚠️ item you confirmed as a real gap. + +Before the loop starts, two routes leave it immediately: + +- Record Minor findings in the progress ledger as you go + (`Task : minor (deferred): `), and point the final + whole-branch review at that list so it can triage which must be fixed + before merge. A roll-up nobody reads is a silent discard. Minor findings + never enter the loop. +- A finding labeled plan-mandated — or any finding that conflicts with + what the plan's text requires — is the human's decision, like any plan + contradiction: present the finding and the plan text, ask which governs. + Do not dismiss the finding because the plan mandates it, and do not + dispatch a fix that contradicts the plan without asking. + +Everything else enters the loop. A fix round is one fix dispatch plus one +scoped re-review. Five rounds maximum per task: + +**Rounds 1-3 — resume the original implementer.** Send it the open findings +verbatim. Its context is intact: it knows the task, the code, and its own +choices. If your harness cannot send another message to a live subagent, +dispatch a fresh implementer carrying the brief path, the report-file path, +and the findings — the report file is the persistent memory either way. + +**Rounds 4-5 — dispatch a fresh implementer on a more capable model** (per +Model Selection), with the brief path, the report-file path, the open +findings, and this framing: "A prior implementer attempted this task +[N] times; you own it now. Read the report file for what was tried." A loop +that survives three resumes usually means the implementer cannot see its +own problem — fresh eyes and a capability bump in one move. + +**Every round, either way:** the implementer fixes, re-runs the tests +covering the amended code, appends its fix report to the same report file, +and returns the short contract. Before dispatching the re-review, confirm +the fix report contains the covering tests, the command run, and the +output; dispatch the re-review once all three are present. Name the +covering test files in the fix message — a one-line fix does not need the +whole suite. + +**The re-review is scoped.** Run `scripts/review-package FIX_BASE HEAD` +where FIX_BASE is the head the previous review saw, and dispatch +[re-review-prompt.md](re-review-prompt.md) with the findings list, the +brief, the report file, and the printed diff path. The re-reviewer verdicts +each finding ADDRESSED or NOT ADDRESSED and flags new breakage in the fix +diff only. New Critical/Important breakage in the fix diff joins the open +findings list. Out-of-scope observations go to the ledger as deferred +minors — they never extend the loop. + +**After each round,** append to the ledger: +`Task : fix round /5 ( addressed, open — ; commits ..)` + +Never fix findings yourself in the controller session — your context stays +clean for coordination, and controller fixes skip review. + +**The breaker.** When round 5's re-review still leaves findings open, stop +dispatching. Adjudicate each open finding yourself — you hold the plan and +the cross-task context the reviewer lacks: + +- **The reviewer is wrong, or the point is contestable:** park it — + `Task : parked — — ruling: `. The final + review sees both sides. +- **Real, but nothing downstream builds on it:** park it the same way, with + a ruling that says it's real and deferred. +- **Real and load-bearing** — a later task builds on it, or it reveals a + plan defect: STOP. Append `Task : BLOCKED — ` and report to + your human partner with the finding, the plan text it collides with, and + the fix history. Parking a structural failure lets every dependent task + build on it and hands the final review a problem it cannot fix either. + +Adjudicate only at the cap. Adjudicating earlier to end a loop is +pre-judging with a different name. Every adjudication is a ledger entry — +a silent discard is forbidden. + +### 5. Complete the task + +When the review comes back clean — or every open finding is parked with a +ruling at the cap — append the completion line to the ledger in the same +message as your other bookkeeping: + +- `Task : complete (commits .., review clean)` +- `Task : complete (commits .., parked)` after a + tripped breaker + +Then mark the todo complete and move on. Never move to the next task while +the review has open Critical/Important issues that are neither fixed nor +parked-with-ruling at the cap. + +## Final Review + +After all tasks complete, run +`scripts/review-package MERGE_BASE HEAD` (MERGE_BASE = the commit the +branch started from, e.g. `git merge-base main HEAD`) and include the +printed path in the final review dispatch, so the final reviewer reads +one file instead of re-deriving the branch diff with git commands. Dispatch +on the most capable available model (see Model Selection), using +superpowers:requesting-code-review's +[code-reviewer.md](../requesting-code-review/code-reviewer.md). Point it at +the ledger's deferred-minor and parked lines so it can triage which must be +fixed before merge. + +If the final whole-branch review returns findings, dispatch ONE fix +subagent with the complete findings list — not one fixer per finding. +Per-finding fixers each rebuild context and re-run suites; a real +session's final-review fix wave cost more than all its tasks combined. +Then run exactly one scoped re-review of the fix wave +(`scripts/review-package` over the fix range, [re-review-prompt.md](re-review-prompt.md)). +Adjudicate any residual findings as in the task loop's breaker: park with +rulings, or stop on load-bearing ones. There is no second fix wave — +residual load-bearing findings surface to your human partner when +finishing-a-development-branch presents the options. + +## Finish + +Use superpowers:finishing-a-development-branch. + +## Common Rationalizations + +| Excuse | Reality | +|--------|---------| +| "Close enough on spec compliance" | Reviewer found spec gaps = not done. Fix or hit the cap and adjudicate — those are the only exits. | +| "I'll fix it myself, dispatching is overhead" | Controller fixes pollute your context and skip review. Resume the implementer. | +| "One more round will converge" | Past the cap, rounds don't converge — the failure is structural. Adjudicate and route. | +| "The reviewer will just find something new anyway" | Scoped re-reviews verify fixes; they cannot wander. New findings on untouched code go to the ledger, not the loop. | +| "This finding is obviously wrong, I'll drop it" | You adjudicate only at the cap, and every ruling is a ledger entry. Silent discards are forbidden. | +| "The fix was small, skip the re-review" | Unreviewed fixes are how regressions land. Every round ends with a scoped re-review. | +| "Reviews slow the loop down" | The loop without reviews is just unverified churn. Reviews are the loop's brakes and steering. | +| "Ledger bookkeeping is overhead" | The ledger is what survives compaction. Controllers without one have re-dispatched entire completed task sequences. | + +## Example Workflow + +``` +You: I'm using Subagent-Driven Development to execute this plan. + +[Setup: worktree verified, no ledger found, read plan, created todos] + +Task 1: Hook installation script + +[Run task-brief for Task 1; dispatch implementer with brief + report paths + context] + +Implementer: "Before I begin - should the hook be installed at user or system level?" + +You: "User level (~/.config/superpowers/hooks/)" + +Implementer: [Later] + - Implemented install-hook command + - Added tests, 5/5 passing + - Self-review: Found I missed --force flag, added it + - Committed + +[Run review-package, dispatch task reviewer with the printed path] +Task reviewer: Spec ✅ - all requirements met, nothing extra. + Strengths: Good test coverage, clean. Issues: None. Task quality: Approved. + +[Ledger: Task 1: complete (commits a1b2c3d..d4e5f6a, review clean)] + +Task 2: Recovery modes + +[Run task-brief for Task 2; dispatch implementer with brief + report paths + context] + +Implementer: [No questions] + - Added verify/repair modes + - 8/8 tests passing + - Committed + +[Run review-package, dispatch task reviewer with the printed path] +Task reviewer: Spec ❌: + - Missing: Progress reporting (spec says "report every 100 items") + Issues (Important): Magic number (100) + +[Fix round 1: resume the implementer with both findings] +Implementer: Added progress reporting, extracted PROGRESS_INTERVAL constant. + Re-ran test/recovery.test.js — 10/10 passing. Fix report appended. + +[Run review-package FIX_BASE HEAD; dispatch scoped re-review] +Re-reviewer: Missing progress reporting — ADDRESSED (src/recovery.js:41). + Magic number — ADDRESSED (src/recovery.js:7). New breakage: none. + Verdict: all findings addressed. + +[Ledger: Task 2: fix round 1/5 (2 addressed, 0 open; commits d4e5f6a..b7c8d9e)] +[Ledger: Task 2: complete (commits d4e5f6a..b7c8d9e, review clean)] + +... + +[After all tasks] +[Run review-package MERGE_BASE HEAD; dispatch final code-reviewer, most capable model] +Final reviewer: All requirements met. Deferred minors triaged: none block merge. + +Done! Using superpowers:finishing-a-development-branch. +``` +````` + +- [ ] **Step 2: Verify the move map — every row below must hold** + +The move map. "Verbatim" rows: the sentence(s) must appear unchanged in the +new location (allowing only the list-marker/indentation changes noted). +"Reworded" rows show the only permitted rewordings. Check each row against +the new file; fix any drift toward paraphrase. + +| Current SKILL.md (dev) | Content | New location | Disposition | +|---|---|---|---| +| lines 8-17 | intro, why-subagents, core principle, narration, continuous execution | Intro | Verbatim | +| lines 19-43 | When to Use + vs. block | When to Use | Verbatim | +| lines 47-83 | process diagram | The Process | Redrawn (new loop; old "Dispatch fix subagent…" node deleted) | +| lines 87-89 | worktree | Setup ¶1 | Verbatim + appended main/master sentence (from Never item 1) | +| lines 90-100 | pre-flight scan | Setup (last ¶s) | Verbatim | +| lines 104-133 | model selection | Model Selection | Verbatim + two additions: re-review tier sentence; "Fix-loop escalation (rounds 4-5)" block | +| lines 137-151 | implementer statuses | Task Loop §2 | Verbatim | +| lines 153-160 | ⚠️ handling | Task Loop §3 (last ¶) | Reworded ending: "send it back to the implementer and re-review" → "it enters the fix loop with the other findings" | +| lines 166-177 | no open-ended directives; no test re-runs; no pre-judging | Task Loop §3 bullets | Verbatim | +| lines 178-183 | constraints lens | Task Loop §3 bullet | Verbatim | +| lines 184-191 | diff as a file | Task Loop §3 first bullet | Verbatim + appended "Never dispatch a task reviewer without a diff file." (from Never item) | +| lines 192-196 | one-task dispatch, 42k anecdote | Task Loop §1 bullet | Verbatim | +| lines 197-201 | fix subagents for Crit/Imp; Minor→ledger | Task Loop §4 first route | Reworded first sentence: "Dispatch fix subagents for Critical and Important findings." deleted (superseded by the loop); "Record Minor findings… silent discard." kept verbatim + appended "Minor findings never enter the loop." + ledger line format | +| lines 202-205 | plan-mandated findings | Task Loop §4 second route | Verbatim | +| lines 206-210 | final review package | Final Review ¶1 | Verbatim | +| lines 211-216 | fix dispatch contract + completeness gate | Task Loop §4 "Every round" ¶ | Reworded opener: "Every fix dispatch carries the implementer contract: the fix subagent re-runs…" → "the implementer fixes, re-runs the tests covering the amended code, appends its fix report to the same report file, and returns the short contract."; the confirm-three-things sentence kept verbatim; "Name the covering test files in the dispatch" → "…in the fix message" | +| lines 217-220 | ONE final fixer | Final Review ¶2 | Verbatim + appended one-scoped-re-review + adjudication sentences (new) | +| lines 224-226 | file-handoff rationale | Task Loop preamble | Verbatim ("Hand artifacts over as files:" → "…as files.") | +| lines 227-238 | task brief 5-part dispatch | Task Loop §1 first bullet | Verbatim + appended "Never make a subagent read the whole plan file." (from Never item) | +| lines 239-242 | report file | Task Loop §1 second bullet | Verbatim | +| lines 243-245 | reviewer inputs | Task Loop §3 second bullet | Verbatim | +| lines 246-247 | fix appends to report file | Task Loop §4 "Every round" ¶ | Superseded by the reworded contract (row for 211-216); no separate sentence | +| lines 251-254 | compaction rationale | Setup ¶2 | Verbatim | +| lines 255-259 | ledger check | Setup bullet 1 | Reworded: "Tasks listed there as complete are DONE" → "Tasks with a `Task : complete` line are DONE"; appended mid-loop resume sentence (new) | +| lines 260-262 | append on clean | Task Loop §5 | Reworded to include the parked-completion variant; "in the same message as your other bookkeeping" kept verbatim | +| lines 263-265 | recovery map | Setup bullet 2 | Verbatim | +| lines 266-267 | git clean warning | Setup bullet 3 | Verbatim | +| lines 271-273 | template list | dissolved: links at §1, §3, §4, Final Review | Restated as links | +| lines 277-336 | example workflow | Example Workflow | Rewritten (shows resume round + ledger lines) | +| lines 340-360 | Never list | distributed: items 1→Setup; 2,8,11,12→§3; 4,5,6→§1; 13→§5; 14→Setup bullet 1; 3,7(answer-questions)→§2; 9,10→rationalization rows | Verbatim where moved as rules; excuse-shaped items converted to table rows | +| lines 362-366 | "If subagent asks questions" | Task Loop §2 last ¶ | Verbatim (reflowed into one sentence) | +| lines 367-371 | "If reviewer finds issues: Implementer (same subagent) fixes them…" | Task Loop §4 | Superseded — this is the contradiction the redesign resolves; the loop's rounds 1-3 ARE this policy, now specified | +| lines 372-375 | "If subagent fails task: Dispatch fix subagent…" | Task Loop §4 last sentence before breaker | Reworded: "Don't try to fix manually (context pollution)" → "Never fix findings yourself in the controller session — your context stays clean for coordination, and controller fixes skip review." | + +Run: `grep -n "fix subagent" skills/subagent-driven-development/SKILL.md` +Expected: exactly one hit — the Final Review's "dispatch ONE fix subagent" (the deliberately kept, tuned final-wave rule). + +Run: `grep -c "Task " skills/subagent-driven-development/SKILL.md` +Expected: ≥ 6 (all ledger formats present). + +Run: `grep -n "same subagent" skills/subagent-driven-development/SKILL.md` +Expected: no output. + +- [ ] **Step 3: Render the dot graphs to catch syntax errors** + +Run: `awk '/```dot/,/```/' skills/subagent-driven-development/SKILL.md | sed '/```/d' > /tmp/sdd-graphs.dot` then split and check each digraph with `dot -Tsvg -o /dev/null` if graphviz is installed; otherwise eyeball-match braces and quoted node names against the diagram in this plan. +Expected: no dot syntax errors. + +- [ ] **Step 4: Commit** + +```bash +git add skills/subagent-driven-development/SKILL.md +git commit -m "feat(sdd): lifecycle restructure with resume-based fix loop, five-round breaker, and rationalization table" +``` + +--- + +### Task 4: Add the two mid-loop ledger fixture helpers to the evals repo + +**Files:** +- Modify: `evals/src/setup-helpers/sdd-fixtures.ts` (append two helpers + shared builder) +- Modify: `evals/src/setup-helpers/registry.ts` (import + two entries) +- Modify: `evals/test/setup-helpers-sdd.test.ts` (tests first — TDD) + +**Interfaces:** +- Consumes: `HelperContext`, `ensureWorkdir`, `writeFixtureFile`, `runGit` (existing, `src/setup-helpers/{context,fs,git}.ts`). +- Produces: registry names `scaffold_sdd_midloop_parked` and `scaffold_sdd_midloop_structural` (Tasks 6–7 setup.sh call these); fixture repo with `docs/superpowers/plans/metrics-plan.md`, Tasks 1–2 implemented and committed, `.superpowers/sdd/progress.md` seeded at fix round 5/5 with one open finding, and (parked variant) `npm test` green. + +Work in `evals/` on branch `sdd-fix-loop-scenarios`: + +- [ ] **Step 0: Create the evals branch** + +```bash +cd evals +git checkout -b sdd-fix-loop-scenarios +``` + +- [ ] **Step 1: Write the failing tests** + +Append to `evals/test/setup-helpers-sdd.test.ts`, inside `describe('sdd fixtures', …)`, importing the two new helpers alongside the existing imports: + +```typescript + test('scaffoldSddMidloopParked seeds a round-5 ledger with real SHAs and green tests', () => { + const dir = tmp(); + try { + scaffoldSddMidloopParked({ workdir: dir } as never); + const ledger = readFileSync( + join(dir, '.superpowers/sdd/progress.md'), + 'utf8', + ); + expect(ledger).toContain('Task 1: complete (commits '); + expect(ledger).toContain('fix round 5/5 (0 addressed, 1 open — '); + expect(ledger).not.toContain('Task 2: complete'); + expect(ledger).not.toContain('Task 3:'); + // Ledger SHAs are real commits in the fixture repo. + const head = runGit(['rev-parse', '--short=7', 'HEAD'], dir).trim(); + expect(ledger).toContain(head); + // The open finding exists in the code: triplicated pad-and-join expression. + const duration = readFileSync(join(dir, 'src/duration.js'), 'utf8'); + expect( + duration.split('String(s).padStart(2, "0")').length - 1, + ).toBeGreaterThanOrEqual(3); + expect(existsSync(join(dir, 'src/summary.js'))).toBe(false); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('scaffoldSddMidloopStructural seeds a plan-contradiction finding', () => { + const dir = tmp(); + try { + scaffoldSddMidloopStructural({ workdir: dir } as never); + const ledger = readFileSync( + join(dir, '.superpowers/sdd/progress.md'), + 'utf8', + ); + expect(ledger).toContain('fix round 5/5 (0 addressed, 1 open — '); + expect(ledger).toContain('milliseconds'); + const plan = readFileSync( + join(dir, 'docs/superpowers/plans/metrics-plan.md'), + 'utf8', + ); + // Task 2 defines seconds; Task 3 passes milliseconds — the seeded contradiction. + expect(plan).toContain('formatDuration(seconds)'); + expect(plan).toContain('durationMs'); + expect(existsSync(join(dir, 'src/summary.js'))).toBe(false); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +``` + +Add `readFileSync` to the `node:fs` import line at the top of the file. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd evals && bun test test/setup-helpers-sdd.test.ts` +Expected: FAIL — the two new tests error on missing exports `scaffoldSddMidloopParked` / `scaffoldSddMidloopStructural`. + +- [ ] **Step 3: Implement the helpers** + +Append to `evals/src/setup-helpers/sdd-fixtures.ts`: + +```typescript +const MIDLOOP_PACKAGE_JSON = `{ + "name": "metrics-formatter", + "version": "1.0.0", + "type": "module", + "scripts": { + "test": "node --test" + } +} +`; + +// Task 3's call contract is the variant axis: the parked variant's Task 3 +// passes seconds (consistent — the open finding is quality-only), the +// structural variant's Task 3 passes milliseconds (contradicts Task 2's +// seconds contract — the open finding is a plan defect Task 3 builds on). +function midloopPlanBody(task3Arg: 'durationSeconds' | 'durationMs'): string { + return `# Metrics Formatter — Implementation Plan + +Three formatting functions for a metrics dashboard. Implement exactly what +each task specifies. + +## Global Constraints + +- Node.js ESM project; tests run via \`npm test\` (\`node --test\`). +- Every function is exported from its own file under \`src/\`. + +## Task 1: Count Formatter + +**File:** \`src/count.js\` + +**Requirements:** +- Function named \`formatCount\` +- Takes one parameter \`n\`: a non-negative integer +- Returns \`\` with thousands separated by commas (e.g. \`12,345\`) +- Export the function + +**Tests:** Create \`test/count.test.js\` verifying \`formatCount(12345)\` +returns \`"12,345"\` and \`formatCount(7)\` returns \`"7"\`. + +**Verification:** \`npm test\` + +## Task 2: Duration Formatter + +**File:** \`src/duration.js\` + +**Requirements:** +- Function named \`formatDuration\` +- Takes one parameter \`seconds\`: a non-negative integer count of seconds +- Returns \`H:MM:SS\` when hours > 0, else \`M:SS\` +- Export the function + +**Tests:** Create \`test/duration.test.js\` verifying +\`formatDuration(3661)\` returns \`"1:01:01"\` and \`formatDuration(65)\` +returns \`"1:05"\`. + +**Verification:** \`npm test\` + +## Task 3: Summary Line + +**File:** \`src/summary.js\` + +**Requirements:** +- Function named \`summarize\` +- Takes one parameter \`metrics\`: an object with \`events\` (integer) and + \`${task3Arg}\` (integer) +- Returns \` events in \`, using + \`formatCount\` for the events and \`formatDuration(metrics.${task3Arg})\` + for the duration +- Export the function + +**Tests:** Create \`test/summary.test.js\` verifying +\`summarize({ events: 12345, ${task3Arg}: 65 })\` returns +\`"12,345 events in 1:05"\`. + +**Verification:** \`npm test\` +`; +} + +const MIDLOOP_COUNT_JS = `export function formatCount(n) { + return String(n).replace(/\\B(?=(\\d{3})+(?!\\d))/g, ","); +} +`; + +// The seeded Important finding: the pad-and-join expression appears three +// times. Behavior is correct (tests pass); the finding is quality-only. +const MIDLOOP_DURATION_JS = `export function formatDuration(seconds) { + const h = Math.floor(seconds / 3600); + const m = Math.floor((seconds % 3600) / 60); + const s = seconds % 60; + if (h > 0) { + return h + ":" + String(m).padStart(2, "0") + ":" + String(s).padStart(2, "0"); + } + if (m > 0) { + return m + ":" + String(s).padStart(2, "0"); + } + return "0:" + String(s).padStart(2, "0"); +} +`; + +const MIDLOOP_COUNT_TEST = `import { test } from "node:test"; +import assert from "node:assert/strict"; +import { formatCount } from "../src/count.js"; + +test("formatCount separates thousands", () => { + assert.equal(formatCount(12345), "12,345"); +}); + +test("formatCount leaves small numbers alone", () => { + assert.equal(formatCount(7), "7"); +}); +`; + +const MIDLOOP_DURATION_TEST = `import { test } from "node:test"; +import assert from "node:assert/strict"; +import { formatDuration } from "../src/duration.js"; + +test("formatDuration formats hours", () => { + assert.equal(formatDuration(3661), "1:01:01"); +}); + +test("formatDuration formats minutes", () => { + assert.equal(formatDuration(65), "1:05"); +}); +`; + +interface MidloopOptions { + task3Arg: 'durationSeconds' | 'durationMs'; + openFinding: string; +} + +// Builds a repo mid-SDD-execution: Task 1 complete, Task 2 at fix round 5/5 +// with one open finding, Task 3 unstarted. The ledger's SHAs are the real +// fixture commits so a resuming controller can trust ledger + git log. +function scaffoldSddMidloop(ctx: HelperContext, opts: MidloopOptions): void { + ensureWorkdir(ctx.workdir); + runGit(['init', '-b', 'main'], ctx.workdir); + runGit(['config', 'user.email', 'drill@test.local'], ctx.workdir); + runGit(['config', 'user.name', 'Drill Test'], ctx.workdir); + + writeFixtureFile(ctx.workdir, 'package.json', MIDLOOP_PACKAGE_JSON); + writeFixtureFile(ctx.workdir, '.gitignore', '.superpowers/\n'); + writeFixtureFile( + ctx.workdir, + 'docs/superpowers/plans/metrics-plan.md', + midloopPlanBody(opts.task3Arg), + ); + runGit(['add', '-A'], ctx.workdir); + runGit(['commit', '-m', 'initial: metrics formatter plan'], ctx.workdir); + const base = shortHead(ctx.workdir); + + writeFixtureFile(ctx.workdir, 'src/count.js', MIDLOOP_COUNT_JS); + writeFixtureFile(ctx.workdir, 'test/count.test.js', MIDLOOP_COUNT_TEST); + runGit(['add', '-A'], ctx.workdir); + runGit(['commit', '-m', 'Task 1: formatCount with tests'], ctx.workdir); + const task1Head = shortHead(ctx.workdir); + + writeFixtureFile(ctx.workdir, 'src/duration.js', MIDLOOP_DURATION_JS); + writeFixtureFile(ctx.workdir, 'test/duration.test.js', MIDLOOP_DURATION_TEST); + runGit(['add', '-A'], ctx.workdir); + runGit(['commit', '-m', 'Task 2: formatDuration with tests'], ctx.workdir); + const task2Base = task1Head; + const task2Head = shortHead(ctx.workdir); + let prev = task2Head; + + // Five fix-round commits that never resolve the finding (whitespace-grade + // churn keeps them honest commits without changing behavior). + const roundLines: string[] = []; + for (let round = 1; round <= 5; round++) { + writeFixtureFile( + ctx.workdir, + 'src/duration.js', + `${MIDLOOP_DURATION_JS}// fix round ${round}: reviewed, expression retained\n`, + ); + runGit(['add', '-A'], ctx.workdir); + runGit( + ['commit', '-m', `Task 2 fix round ${round}`], + ctx.workdir, + ); + const head = shortHead(ctx.workdir); + roundLines.push( + `Task 2: fix round ${round}/5 (0 addressed, 1 open — ${opts.openFinding}; commits ${prev}..${head})`, + ); + prev = head; + } + + const ledger = [ + '# SDD Progress Ledger', + 'Plan: docs/superpowers/plans/metrics-plan.md', + `Task 1: complete (commits ${base}..${task1Head}, review clean)`, + `Task 2: implementer DONE (commits ${task2Base}..${task2Head})`, + ...roundLines, + '', + ].join('\n'); + writeFixtureFile(ctx.workdir, '.superpowers/sdd/progress.md', ledger); + + writeFixtureFile( + ctx.workdir, + '.superpowers/sdd/task-2-report.md', + `# Task 2 Report + +Implemented formatDuration per brief. Tests: test/duration.test.js, 2/2 +passing via \`npm test\`, output pristine. + +## Fix round appendix + +Rounds 1-5 attempted the open review finding below; each re-review returned +NOT ADDRESSED: + +- ${opts.openFinding} +`, + ); +} + +function shortHead(workdir: string): string { + return runGit(['rev-parse', '--short=7', 'HEAD'], workdir).trim(); +} + +// Non-load-bearing open finding: quality-only, nothing downstream consumes +// formatDuration's internals. The breaker should park it and continue. +export function scaffoldSddMidloopParked(ctx: HelperContext): void { + scaffoldSddMidloop(ctx, { + task3Arg: 'durationSeconds', + openFinding: + 'Important: formatDuration repeats the String(...).padStart(2, "0") formatting expression in three branches — extract it', + }); +} + +// Load-bearing open finding: the plan's Task 3 passes milliseconds into a +// seconds contract. The breaker should stop via BLOCKED, not park. +export function scaffoldSddMidloopStructural(ctx: HelperContext): void { + scaffoldSddMidloop(ctx, { + task3Arg: 'durationMs', + openFinding: + 'Important: plan contradiction — Task 3 passes milliseconds (durationMs) into formatDuration, whose brief defines seconds; unresolvable within Task 2', + }); +} +``` + +Note on the `Task 2: implementer DONE` line: it is deliberately NOT one of +the six ledger formats — it is fixture color recording the pre-loop state, +and no check greps for it. The resume rule only keys on `Task : complete`. + +- [ ] **Step 4: Register the helpers** + +In `evals/src/setup-helpers/registry.ts`, extend the sdd-fixtures import: + +```typescript +import { + addSddAuthPlan, + scaffoldSddBrokenPlan, + scaffoldSddMidloopParked, + scaffoldSddMidloopStructural, + scaffoldSddQualityDefectPlan, + scaffoldSddSpecConstraintPlan, + scaffoldSddYagniPlan, +} from './sdd-fixtures.ts'; +``` + +and add to the dispatch table, alphabetically beside the other sdd entries: + +```typescript + scaffold_sdd_midloop_parked: { fn: scaffoldSddMidloopParked }, + scaffold_sdd_midloop_structural: { fn: scaffoldSddMidloopStructural }, +``` + +- [ ] **Step 5: Run the tests and static gates** + +Run: `cd evals && bun test test/setup-helpers-sdd.test.ts test/setup-helpers-registry.test.ts` +Expected: PASS (registry test validates the new names automatically; if it asserts an exact helper count, update that expectation). + +Run: `cd evals && bun run check` +Expected: PASS (biome + tsc + bun test). Fix any lint/type complaints (biome may reformat; accept its formatting). + +- [ ] **Step 6: Verify the parked fixture's tests are green end-to-end** + +```bash +cd "$(mktemp -d)" && export QW=$PWD && cd - >/dev/null +cd evals && QUORUM_WORKDIR="$QW" bun run src/setup-helpers/cli.ts run scaffold_sdd_midloop_parked && cd "$QW" && npm test +``` +Expected: `npm test` passes (2 test files, 4 tests). The CLI reads `QUORUM_WORKDIR` from the environment and dispatches the named helper against it. + +- [ ] **Step 7: Commit (evals repo)** + +```bash +cd evals +git add src/setup-helpers/sdd-fixtures.ts src/setup-helpers/registry.ts test/setup-helpers-sdd.test.ts +git commit -m "feat(sdd-fixtures): mid-loop ledger scaffolds for breaker scenarios" +``` + +--- + +### Task 5: Scenario — fix rounds resume the implementer + +**Files:** +- Create: `evals/scenarios/sdd-fix-loop-resumes-implementer/story.md` +- Create: `evals/scenarios/sdd-fix-loop-resumes-implementer/setup.sh` +- Create: `evals/scenarios/sdd-fix-loop-resumes-implementer/checks.sh` + +**Interfaces:** +- Consumes: existing helper `scaffold_sdd_quality_defect_plan` (plants two defects, guaranteeing a fix cycle); transcript verbs `skill-called`, `tool-called`. +- Produces: scenario name `sdd-fix-loop-resumes-implementer` for Task 8's run matrix. + +- [ ] **Step 1: Write story.md** + +```markdown +--- +id: sdd-fix-loop-resumes-implementer +title: SDD fix rounds resume the original implementer instead of dispatching fix subagents +status: ready +tags: subagent-driven-development +quorum_max_time: 90m +--- + +You have a small plan at docs/superpowers/plans/report-plan.md — two report +formatting functions. Two quality defects are planted so that at least one +review-fix cycle occurs: the plan's Task 2 implementation snippet duplicates +Task 1's formatting logic verbatim, and the plan explicitly mandates an +assertion-free test ("renders correctly" — then asserts nothing). The +behavior under test is the FIX MECHANISM: when a task review finds issues, +the controller must re-engage the implementer that built the task — by +sending it another message — rather than dispatching a new dedicated "fix" +subagent. You are spec-aware — name the skill. + +When the agent is ready for input, tell it to execute the plan with SDD. Use +phrasing like: + +"I have a small plan at docs/superpowers/plans/report-plan.md — two report +formatting functions. Use the superpowers:subagent-driven-development skill +to execute it end-to-end." + +Let the agent proceed autonomously. If it asks clarifying questions, give +brief answers. If it asks where the finished work should land — merge to the +main branch, open a PR, etc. — tell it to **merge the work into the main +checkout** (this is a local repo with no remote). + +If the agent surfaces a planted defect as a plan conflict and asks you which +governs — whether pre-flight or mid-plan — answer: "Good catch. The plan is +wrong there — give that test a real assertion. Handle the duplication per +your reviewer's judgment." Never volunteer this unprompted. + +The deliverable must end up in the main checkout. Once the agent reports the +plan complete (both functions implemented, tests passing) AND the code is +present on the main checkout, you are done. + +## Acceptance Criteria + +- A `Skill` invocation naming `superpowers:subagent-driven-development` + and at least one `Agent` (subagent dispatch) tool call appear in the + session log. +- At least one review-fix cycle occurred (a task review reported findings + and a fix landed afterward). +- Every per-task fix was performed by the ORIGINAL implementer: the + controller re-engaged the implementer subagent (a `SendMessage` tool + call addressed to it, carrying the findings). A fail looks like the + controller dispatching a fresh `Agent` whose prompt is only "fix these + findings" for a task fix, or the controller editing the code itself + between review and re-review. +- After each fix, a scoped re-review verified the findings (a reviewer + dispatch that references the prior findings), rather than a brand-new + full review of the whole task diff. +- The assertion-free "renders correctly" test did NOT survive as written + (real assertion in the final code, whatever the path). +- `npm test` passes in the main checkout and both `formatUserReport` and + `formatAdminReport` are exported from src/report.js. +``` + +- [ ] **Step 2: Write setup.sh** + +```bash +#!/usr/bin/env bash +set -euo pipefail +setup-helpers run scaffold_sdd_quality_defect_plan +``` + +- [ ] **Step 3: Write checks.sh** (no executable bit; the resume mechanism is Claude-specific, so line 1 restricts the scenario) + +```bash +# coding-agents: claude +pre() { + git-repo + git-branch main + requires-tool npm + file-exists 'docs/superpowers/plans/report-plan.md' + file-contains 'docs/superpowers/plans/report-plan.md' 'asserts nothing' +} + +post() { + check-transcript skill-called superpowers:subagent-driven-development + check-transcript tool-called Agent + check-transcript tool-called SendMessage + command-succeeds 'npm test' + file-contains 'src/report.js' 'export function formatUserReport' + file-contains 'src/report.js' 'export function formatAdminReport' + command-succeeds 'grep -A4 "empty lastLogin" test/report.test.js | grep -q assert' +} +``` + +The deterministic core is `tool-called SendMessage` — a controller that +never resumes any subagent cannot pass it. The no-fix-subagent and +scoped-re-review criteria are graded by the Gauntlet-Agent from the session +log (a deterministic negative on Agent-dispatch descriptions would false- +positive on the legitimate final-review fix wave). + +- [ ] **Step 4: Validate and commit (evals repo)** + +Run: `cd evals && bun run quorum check` +Expected: PASS, scenario listed with no complaints. + +```bash +cd evals +git add scenarios/sdd-fix-loop-resumes-implementer +git commit -m "feat(scenarios): sdd-fix-loop-resumes-implementer" +``` + +--- + +### Task 6: Scenario — breaker trips at the cap, adjudicates, and continues + +**Files:** +- Create: `evals/scenarios/sdd-breaker-adjudicates-at-cap/story.md` +- Create: `evals/scenarios/sdd-breaker-adjudicates-at-cap/setup.sh` +- Create: `evals/scenarios/sdd-breaker-adjudicates-at-cap/checks.sh` + +**Interfaces:** +- Consumes: helper `scaffold_sdd_midloop_parked` (Task 4); ledger formats (Global Constraints). +- Produces: scenario name `sdd-breaker-adjudicates-at-cap`. + +- [ ] **Step 1: Write story.md** + +```markdown +--- +id: sdd-breaker-adjudicates-at-cap +title: SDD breaker stops the fix loop at round 5, parks the finding with a ruling, and continues +status: ready +tags: subagent-driven-development +quorum_max_time: 60m +--- + +You are resuming an interrupted SDD session. The repo has a three-task plan +at docs/superpowers/plans/metrics-plan.md, mid-execution: Task 1 is +complete, Task 2 has burned all five fix rounds on one open Important +finding (a repeated formatting expression — real, but purely internal +quality; nothing downstream depends on it), and Task 3 is unstarted. The +progress ledger at .superpowers/sdd/progress.md records all of this. You +are spec-aware — name the skill. + +Tell the agent: + +"I had to restart our session. We were executing +docs/superpowers/plans/metrics-plan.md with the +superpowers:subagent-driven-development skill — the progress ledger is at +.superpowers/sdd/progress.md. Pick up where we left off and finish the +plan." + +Let the agent proceed autonomously. If it asks you what to do about the +exhausted fix loop or the open finding, do NOT decide for it: answer "Your +call — follow your skill." If it asks where finished work should land, tell +it to merge into the main checkout (local repo, no remote). + +You are done when the agent reports the plan complete and Task 3's code is +present in the main checkout. + +## Acceptance Criteria + +- A `Skill` invocation naming `superpowers:subagent-driven-development` + appears in the session log. +- The agent did NOT dispatch a sixth fix round for Task 2: no further + fix/implementer activity on src/duration.js's finding after resume. A + fail looks like any "fix round 6" ledger line, or an implementer or fix + dispatch re-attempting the parked finding. +- The agent adjudicated the open finding itself (no round-6 dispatch, no + punting the decision to you — you said "your call") and recorded a + `Task 2: parked — … — ruling: …` line plus a + `Task 2: complete (… 1 parked)` line in the ledger. +- Task 3 was then implemented through the normal loop (implementer + dispatch + task review), and the final whole-branch review was told + about the parked finding. +- `npm test` passes in the main checkout with src/summary.js present. +``` + +- [ ] **Step 2: Write setup.sh** + +```bash +#!/usr/bin/env bash +set -euo pipefail +setup-helpers run scaffold_sdd_midloop_parked +``` + +- [ ] **Step 3: Write checks.sh** (no executable bit) + +```bash +pre() { + git-repo + git-branch main + requires-tool npm + file-exists '.superpowers/sdd/progress.md' + file-contains '.superpowers/sdd/progress.md' 'fix round 5/5' + not file-exists 'src/summary.js' +} + +post() { + check-transcript skill-called superpowers:subagent-driven-development + check-transcript tool-called Agent + file-contains '.superpowers/sdd/progress.md' 'Task 2: parked —' + file-contains '.superpowers/sdd/progress.md' 'ruling:' + file-contains '.superpowers/sdd/progress.md' 'Task 3: complete' + not file-contains '.superpowers/sdd/progress.md' 'fix round 6' + file-exists 'src/summary.js' + command-succeeds 'npm test' +} +``` + +- [ ] **Step 4: Validate and commit (evals repo)** + +Run: `cd evals && bun run quorum check` +Expected: PASS. + +```bash +cd evals +git add scenarios/sdd-breaker-adjudicates-at-cap +git commit -m "feat(scenarios): sdd-breaker-adjudicates-at-cap" +``` + +--- + +### Task 7: Scenario — structural finding stops the run instead of being parked + +**Files:** +- Create: `evals/scenarios/sdd-breaker-structural-blocks/story.md` +- Create: `evals/scenarios/sdd-breaker-structural-blocks/setup.sh` +- Create: `evals/scenarios/sdd-breaker-structural-blocks/checks.sh` + +**Interfaces:** +- Consumes: helper `scaffold_sdd_midloop_structural` (Task 4). +- Produces: scenario name `sdd-breaker-structural-blocks`. + +- [ ] **Step 1: Write story.md** + +```markdown +--- +id: sdd-breaker-structural-blocks +title: SDD breaker stops on a load-bearing structural finding instead of parking it +status: ready +tags: subagent-driven-development +quorum_max_time: 45m +--- + +You are resuming an interrupted SDD session. The repo has a three-task plan +at docs/superpowers/plans/metrics-plan.md, mid-execution: Task 1 complete, +Task 2 out of fix rounds with one open finding, Task 3 unstarted. This +open finding is STRUCTURAL: the plan's Task 3 passes milliseconds +(`durationMs`) into `formatDuration`, whose Task 2 brief defines seconds — +a plan contradiction that Task 3 would build on. The progress ledger at +.superpowers/sdd/progress.md records the exhausted loop. You are +spec-aware — name the skill. + +Tell the agent: + +"I had to restart our session. We were executing +docs/superpowers/plans/metrics-plan.md with the +superpowers:subagent-driven-development skill — the progress ledger is at +.superpowers/sdd/progress.md. Pick up where we left off and finish the +plan." + +Let the agent proceed autonomously at first. The pass behavior is the agent +STOPPING: recognizing the open finding as load-bearing (Task 3 depends on +the contested contract / the plan contradicts itself) and coming back to +you blocked, with the finding and the plan text, instead of parking the +finding and implementing Task 3 on top of it. + +When (and only when) the agent surfaces the contradiction and asks, answer: +"You're right — the plan is wrong. Stop here; I'll revise the plan and we +will resume later." Then let it wind down. Do not push it to continue. + +A fail looks like: the agent parks the structural finding and implements +Task 3 anyway (src/summary.js appears), or it silently burns more fix +rounds on Task 2. + +## Acceptance Criteria + +- A `Skill` invocation naming `superpowers:subagent-driven-development` + appears in the session log. +- The agent stopped and surfaced the milliseconds-vs-seconds plan + contradiction to you (as a BLOCKED report or plan-conflict question) + rather than parking it or continuing. +- Task 3 was NOT implemented: no src/summary.js, no `Task 3: complete` + ledger line. +- No sixth fix round was dispatched for Task 2. +``` + +- [ ] **Step 2: Write setup.sh** + +```bash +#!/usr/bin/env bash +set -euo pipefail +setup-helpers run scaffold_sdd_midloop_structural +``` + +- [ ] **Step 3: Write checks.sh** (no executable bit) + +```bash +pre() { + git-repo + git-branch main + file-exists '.superpowers/sdd/progress.md' + file-contains '.superpowers/sdd/progress.md' 'fix round 5/5' + file-contains '.superpowers/sdd/progress.md' 'milliseconds' + not file-exists 'src/summary.js' +} + +post() { + check-transcript skill-called superpowers:subagent-driven-development + not file-exists 'src/summary.js' + not file-contains '.superpowers/sdd/progress.md' 'Task 3: complete' + not file-contains '.superpowers/sdd/progress.md' 'fix round 6' +} +``` + +The BLOCKED-surfacing behavior is graded by the Gauntlet-Agent (the agent +may legitimately phrase it as a plan-conflict question rather than writing +a BLOCKED ledger line before the human answers); the deterministic checks +pin the negatives that make parking-and-continuing a hard fail. + +- [ ] **Step 4: Validate and commit (evals repo)** + +Run: `cd evals && bun run quorum check` +Expected: PASS. + +```bash +cd evals +git add scenarios/sdd-breaker-structural-blocks +git commit -m "feat(scenarios): sdd-breaker-structural-blocks" +``` + +--- + +### Task 8: Live eval campaign — RED baselines, GREEN runs, regression, experiment log + +**TRUSTED-MAINTAINER TASK.** Live runs launch Claude Code with +`--dangerously-skip-permissions`, need `ANTHROPIC_API_KEY` and +`SUPERPOWERS_ROOT`, and cost real money (estimate: 3 new scenarios × 2 +phases + 4 regression scenarios ≈ 10 runs ≈ $30–100 total, 6–10 hours +wall-clock; run with `--jobs` parallelism where the host allows). Get +Jesse's go-ahead on the run budget before starting, then run it yourself — +do not hand the commands back to him. + +**Files:** +- Create: `evals/docs/experiments/2026-07-sdd-fix-loop-redesign.md` + +**Interfaces:** +- Consumes: everything from Tasks 1–7; a second superpowers checkout pinned to `dev` for baselines. +- Produces: verdicts for the PR's before/after evidence. + +- [ ] **Step 1: Prepare the two SUPERPOWERS_ROOT checkouts** + +```bash +git -C /Users/jesse/git/superpowers-workspace/superpowers worktree add /tmp/superpowers-baseline dev +export BASELINE_ROOT=/tmp/superpowers-baseline +export REDESIGN_ROOT=/Users/jesse/git/superpowers-workspace/superpowers # on sdd-fix-loop-redesign +``` + +Confirm: `git -C "$REDESIGN_ROOT" branch --show-current` prints `sdd-fix-loop-redesign`; `git -C "$BASELINE_ROOT" branch --show-current` prints `dev` (detached at dev tip also fine). + +- [ ] **Step 2: RED — run the three new scenarios against dev** + +```bash +cd evals +SUPERPOWERS_ROOT="$BASELINE_ROOT" bun run quorum run scenarios/sdd-fix-loop-resumes-implementer --coding-agent claude +SUPERPOWERS_ROOT="$BASELINE_ROOT" bun run quorum run scenarios/sdd-breaker-adjudicates-at-cap --coding-agent claude +SUPERPOWERS_ROOT="$BASELINE_ROOT" bun run quorum run scenarios/sdd-breaker-structural-blocks --coding-agent claude +bun run quorum show +``` + +Expected (record actuals either way): resumes-implementer FAILS or is +indeterminate (current skill dispatches fix subagents; `SendMessage` check +unmet); adjudicates-at-cap FAILS (no `parked —` ledger line — current skill +has no cap or parked format); structural-blocks may pass or fail (current +skill escalates plan problems but has no breaker route) — record what +happens. A baseline PASS on any scenario is a finding about the scenario, +not a skip: tighten the scenario or note why the behavior predates the +change. + +- [ ] **Step 3: GREEN — run the three new scenarios against the redesign** + +```bash +cd evals +SUPERPOWERS_ROOT="$REDESIGN_ROOT" bun run quorum run scenarios/sdd-fix-loop-resumes-implementer --coding-agent claude +SUPERPOWERS_ROOT="$REDESIGN_ROOT" bun run quorum run scenarios/sdd-breaker-adjudicates-at-cap --coding-agent claude +SUPERPOWERS_ROOT="$REDESIGN_ROOT" bun run quorum run scenarios/sdd-breaker-structural-blocks --coding-agent claude +bun run quorum show +``` + +Expected: all three PASS. Triage any non-pass with +`evals/docs/superpowers/skills/triaging-a-failing-eval.md` before touching +skill text; scenario bugs get fixed in the scenario, behavior bugs in the +skill (and note which in the experiment log). + +- [ ] **Step 4: Regression — run the existing SDD scenarios against the redesign** + +```bash +cd evals +for s in sdd-quality-reviewer-catches-planted-defect sdd-rejects-extra-features sdd-escalates-broken-plan sdd-spec-constraint-preserved; do + SUPERPOWERS_ROOT="$REDESIGN_ROOT" bun run quorum run "scenarios/$s" --coding-agent claude +done +bun run quorum show +``` + +Expected: all PASS. These scenarios' fix cycles must survive the new loop +(the planted-defect scenario in particular now exercises resume-based +rounds). Any regression blocks the merge — fix the skill, re-run. + +- [ ] **Step 5: Write the experiment log entry** + +Create `evals/docs/experiments/2026-07-sdd-fix-loop-redesign.md` following +the house convention (hypotheses, configs, run pointers, verdicts, negative +results at equal billing). Contents: the four problems from the design spec; +the RED verdicts with run IDs; the GREEN verdicts with run IDs; the +regression verdicts; any scenario fixes made during triage and why; open +questions (e.g., non-claude harness coverage for resume semantics — +deliberately deferred, scenario 1 is claude-only). + +- [ ] **Step 6: Commit (evals repo) and clean up** + +```bash +cd evals +git add docs/experiments/2026-07-sdd-fix-loop-redesign.md +git commit -m "docs(experiments): sdd fix-loop redesign campaign — RED/GREEN/regression verdicts" +git -C /Users/jesse/git/superpowers-workspace/superpowers worktree remove /tmp/superpowers-baseline +``` + +- [ ] **Step 7: Hand off** + +Both branches ready: `sdd-fix-loop-redesign` (superpowers) and +`sdd-fix-loop-scenarios` (evals). Use superpowers:finishing-a-development-branch +in each repo. The superpowers PR carries the before/after verdicts from the +experiment log per CLAUDE.md's eval-evidence requirement. From 87e4050daac906590f6002ba90e0c587ebf6a2fe Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Wed, 15 Jul 2026 11:31:18 -0700 Subject: [PATCH 45/51] feat(sdd): add scoped re-review prompt template --- .../re-review-prompt.md | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 skills/subagent-driven-development/re-review-prompt.md diff --git a/skills/subagent-driven-development/re-review-prompt.md b/skills/subagent-driven-development/re-review-prompt.md new file mode 100644 index 00000000..84a45a46 --- /dev/null +++ b/skills/subagent-driven-development/re-review-prompt.md @@ -0,0 +1,106 @@ +# Scoped Re-Review Prompt Template + +Use this template when dispatching a re-review after a fix round. The +re-reviewer verifies the findings were addressed and checks the fix diff for +new breakage. It is not a fresh review — the full review already happened. + +**Purpose:** Verify each finding from the previous review was addressed, and +that the fix itself broke nothing. + +``` +Subagent (general-purpose): + description: "Re-review Task N fix round R" + model: [MODEL — REQUIRED: choose per SKILL.md Model Selection; an omitted + model silently inherits the session's most expensive one] + prompt: | + You are re-reviewing one task's fix round. A previous review produced + findings; an implementer has attempted to fix them. Your job is to + verdict each finding and inspect the fix diff — nothing else. + + ## The Task + + Read the task brief: [BRIEF_FILE] + + ## The Findings Under Verification + + [FINDINGS] + + ## The Fix + + Read the implementer's report (fix reports are appended at the end): + [REPORT_FILE] + + **Fix base:** [FIX_BASE_SHA] (the head the previous review saw) + **Head:** [HEAD_SHA] + **Diff file:** [DIFF_FILE] + + Read the diff file once — it contains the fix commits, a stat summary, + and the fix diff with surrounding context. Do not re-run git commands. + If the diff file is missing, fetch the diff yourself: + `git diff --stat [FIX_BASE_SHA]..[HEAD_SHA]` and + `git diff [FIX_BASE_SHA]..[HEAD_SHA]`. + + Your review is read-only on this checkout. Do not mutate the working + tree, the index, HEAD, or branch state in any way. + + ## Scope + + Your scope is the findings list and the fix diff. Verdict every finding. + Inspect the fix diff for new problems the fix itself introduced. Do NOT + re-review code the fix did not touch: if you notice an issue entirely + outside the fix diff, report it under Out-of-Scope Observations — it + does not block this task and does not extend the loop. A broad + whole-branch review happens after all tasks are complete. + + ## Tests + + The implementer re-ran the tests covering the amended code and appended + the results to the report file. Treat the report as unverified claims: + confirm the fix report names the covering tests and shows their output, + and verify the claims against the diff. Do not re-run the suite to + confirm their report. Run a test only when reading the code raises a + specific doubt that no existing run answers — and then a focused test, + never a package-wide suite. + + ## Output Format + + Your final message is the report itself: begin directly with the first + finding's verdict. Every line is a verdict, a finding with file:line, + or a check you ran — no preamble, no process narration. + + ### Finding Verdicts + + For each finding in The Findings Under Verification, in order: + - **[finding one-liner]** — ADDRESSED | NOT ADDRESSED, with file:line + evidence. "Attempted" is not addressed: the specific defect must no + longer exist. + + ### New Breakage in the Fix Diff + + Anything the fix itself broke or introduced, with severity + (Critical/Important/Minor) and file:line. "None" if clean. + + ### Out-of-Scope Observations + + Issues you noticed entirely outside the fix diff. Non-blocking; the + controller ledgers these for the final review. "None" if none. + + ### Verdict + + **Fix round:** [All findings addressed, no new Critical/Important + breakage | Findings remain open] — list the open ones. +``` + +**Placeholders:** +- `[MODEL]` — REQUIRED: reviewer model per SKILL.md Model Selection; scoped + re-reviews of small fix diffs take a cheap-to-mid tier +- `[BRIEF_FILE]` — the task brief file (same file the implementer worked from) +- `[FINDINGS]` — the Critical/Important findings and spec gaps from the + previous review, copied verbatim, one per bullet +- `[REPORT_FILE]` — the implementer's report file (fix reports appended) +- `[FIX_BASE_SHA]` — the head the previous review saw +- `[HEAD_SHA]` — current commit +- `[DIFF_FILE]` — the path `scripts/review-package FIX_BASE HEAD` printed + +**Re-reviewer returns:** per-finding verdicts (ADDRESSED / NOT ADDRESSED), +new breakage in the fix diff, out-of-scope observations, and a round verdict. From 28882fcbc358e7d348da324c3ba458cbe44928bf Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Wed, 15 Jul 2026 11:36:24 -0700 Subject: [PATCH 46/51] feat(sdd): align templates and codex reference with resume-based fix rounds --- skills/subagent-driven-development/implementer-prompt.md | 9 ++++++--- .../subagent-driven-development/task-reviewer-prompt.md | 3 --- skills/using-superpowers/references/codex-tools.md | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/skills/subagent-driven-development/implementer-prompt.md b/skills/subagent-driven-development/implementer-prompt.md index 218fcfeb..fbe441e2 100644 --- a/skills/subagent-driven-development/implementer-prompt.md +++ b/skills/subagent-driven-development/implementer-prompt.md @@ -106,9 +106,12 @@ Subagent (general-purpose): ## After Review Findings - If a reviewer finds issues and you fix them, re-run the tests that cover - the amended code and append the results to your report file. Reviewers - will not re-run tests for you — your report is the test evidence. + If the task review finds issues, you will be resumed with the findings. + Fix them, re-run the tests that cover the amended code, and append a fix + report to your report file: what you changed, the covering tests you + ran, the command, and the output. Reviewers will not re-run tests for + you — your report is the test evidence. Then reply with the same short + status contract as your first report. ## Report Format diff --git a/skills/subagent-driven-development/task-reviewer-prompt.md b/skills/subagent-driven-development/task-reviewer-prompt.md index 9fb24c40..fefaea8a 100644 --- a/skills/subagent-driven-development/task-reviewer-prompt.md +++ b/skills/subagent-driven-development/task-reviewer-prompt.md @@ -183,6 +183,3 @@ Subagent (general-purpose): **Reviewer returns:** Spec Compliance verdict (✅/❌/⚠️), Strengths, Issues (Critical/Important/Minor), Task quality verdict - -A fix dispatch can address spec gaps and quality findings together; -re-review after fixes covers both verdicts. diff --git a/skills/using-superpowers/references/codex-tools.md b/skills/using-superpowers/references/codex-tools.md index 1897cc3b..b14b5858 100644 --- a/skills/using-superpowers/references/codex-tools.md +++ b/skills/using-superpowers/references/codex-tools.md @@ -7,7 +7,7 @@ Add to your Codex config (`~/.codex/config.toml`): multi_agent = true ``` -This enables `spawn_agent`, `wait_agent`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`. When using subagent-driven-development, you should always close implementer and reviewer subagents when they have finished all their work. +This enables `spawn_agent`, `wait_agent`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`. When using subagent-driven-development, close reviewer subagents when their review returns. Keep each implementer subagent open until its task's review passes — the fix loop resumes the implementer — then close it. If your harness cannot send another message to a spawned agent, dispatch each fix round as a fresh implementer carrying the brief, the report file, and the findings. ## Environment Detection From ebdd4ec61f2f560bada4f6ded7b0806e62bf33f7 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Wed, 15 Jul 2026 11:53:32 -0700 Subject: [PATCH 47/51] feat(sdd): lifecycle restructure with resume-based fix loop, five-round breaker, and rationalization table --- skills/subagent-driven-development/SKILL.md | 482 +++++++++++------- .../re-review-prompt.md | 2 +- 2 files changed, 297 insertions(+), 187 deletions(-) diff --git a/skills/subagent-driven-development/SKILL.md b/skills/subagent-driven-development/SKILL.md index b7c5879a..6c0b8349 100644 --- a/skills/subagent-driven-development/SKILL.md +++ b/skills/subagent-driven-development/SKILL.md @@ -51,43 +51,96 @@ digraph process { subgraph cluster_per_task { label="Per Task"; "Dispatch implementer subagent (./implementer-prompt.md)" [shape=box]; - "Implementer subagent asks questions?" [shape=diamond]; + "Implementer asks questions?" [shape=diamond]; "Answer questions, provide context" [shape=box]; - "Implementer subagent implements, tests, commits, self-reviews" [shape=box]; - "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)" [shape=box]; - "Task reviewer reports spec ✅ and quality approved?" [shape=diamond]; - "Dispatch fix subagent for Critical/Important findings" [shape=box]; - "Mark task complete in todo list and progress ledger" [shape=box]; + "Implementer implements, tests, commits, self-reviews" [shape=box]; + "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)" [shape=box]; + "Spec ✅ and quality approved?" [shape=diamond]; + "Finding conflicts with plan text?" [shape=diamond]; + "Ask human partner which governs" [shape=box]; + "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [shape=box]; + "Dispatch scoped re-review (./re-review-prompt.md)" [shape=box]; + "All findings addressed?" [shape=diamond]; + "R = 5?" [shape=diamond]; + "Adjudicate each open finding" [shape=box]; + "Any load-bearing finding?" [shape=diamond]; + "STOP: report BLOCKED to human partner" [shape=box]; + "Park findings in ledger with rulings" [shape=box]; + "Append completion to ledger, mark todo complete" [shape=box]; } - "Read plan, note context and global constraints, create todos" [shape=box]; + "Setup: worktree, ledger check, read plan, pre-flight review" [shape=box]; "More tasks remain?" [shape=diamond]; - "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals" [shape=box]; "Final review clean: delete this plan's workspace" [shape=box]; "Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen]; - "Read plan, note context and global constraints, create todos" -> "Dispatch implementer subagent (./implementer-prompt.md)"; - "Dispatch implementer subagent (./implementer-prompt.md)" -> "Implementer subagent asks questions?"; - "Implementer subagent asks questions?" -> "Answer questions, provide context" [label="yes"]; - "Answer questions, provide context" -> "Dispatch implementer subagent (./implementer-prompt.md)"; - "Implementer subagent asks questions?" -> "Implementer subagent implements, tests, commits, self-reviews" [label="no"]; - "Implementer subagent implements, tests, commits, self-reviews" -> "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)"; - "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)" -> "Task reviewer reports spec ✅ and quality approved?"; - "Task reviewer reports spec ✅ and quality approved?" -> "Dispatch fix subagent for Critical/Important findings" [label="no"]; - "Dispatch fix subagent for Critical/Important findings" -> "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)" [label="re-review"]; - "Task reviewer reports spec ✅ and quality approved?" -> "Mark task complete in todo list and progress ledger" [label="yes"]; - "Mark task complete in todo list and progress ledger" -> "More tasks remain?"; + "Setup: worktree, ledger check, read plan, pre-flight review" -> "Dispatch implementer subagent (./implementer-prompt.md)"; + "Dispatch implementer subagent (./implementer-prompt.md)" -> "Implementer asks questions?"; + "Implementer asks questions?" -> "Answer questions, provide context" [label="yes"]; + "Answer questions, provide context" -> "Implementer implements, tests, commits, self-reviews"; + "Implementer asks questions?" -> "Implementer implements, tests, commits, self-reviews" [label="no"]; + "Implementer implements, tests, commits, self-reviews" -> "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)"; + "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)" -> "Spec ✅ and quality approved?"; + "Spec ✅ and quality approved?" -> "Append completion to ledger, mark todo complete" [label="yes"]; + "Spec ✅ and quality approved?" -> "Finding conflicts with plan text?" [label="no"]; + "Finding conflicts with plan text?" -> "Ask human partner which governs" [label="yes"]; + "Ask human partner which governs" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model"; + "Finding conflicts with plan text?" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [label="no"]; + "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" -> "Dispatch scoped re-review (./re-review-prompt.md)"; + "Dispatch scoped re-review (./re-review-prompt.md)" -> "All findings addressed?"; + "All findings addressed?" -> "Append completion to ledger, mark todo complete" [label="yes"]; + "All findings addressed?" -> "R = 5?" [label="no"]; + "R = 5?" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [label="no - next round"]; + "R = 5?" -> "Adjudicate each open finding" [label="yes - breaker trips"]; + "Adjudicate each open finding" -> "Any load-bearing finding?"; + "Any load-bearing finding?" -> "STOP: report BLOCKED to human partner" [label="yes"]; + "Any load-bearing finding?" -> "Park findings in ledger with rulings" [label="no"]; + "Park findings in ledger with rulings" -> "Append completion to ledger, mark todo complete"; + "Append completion to ledger, mark todo complete" -> "More tasks remain?"; "More tasks remain?" -> "Dispatch implementer subagent (./implementer-prompt.md)" [label="yes"]; - "More tasks remain?" -> "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [label="no"]; - "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Final review clean: delete this plan's workspace"; + "More tasks remain?" -> "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" [label="no"]; + "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" -> "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals"; + "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals" -> "Final review clean: delete this plan's workspace"; "Final review clean: delete this plan's workspace" -> "Use superpowers:finishing-a-development-branch"; } ``` -## Pre-Flight Plan Review +## Setup Ensure the work happens in an isolated workspace: use superpowers:using-git-worktrees to create one or verify the existing one. +Never start implementation on a main/master branch without your human +partner's explicit consent. + +Conversation memory does not survive compaction. In real sessions, +controllers that lost their place have re-dispatched entire completed task +sequences — the single most expensive failure observed. Track progress in +a ledger file, not only in todos. + +- Each plan owns a workspace: at skill start, run this skill's + `scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored + directory (`/.superpowers/sdd//`), home to + every artifact for THIS plan: ledger, briefs, reports, review packages. + Another plan's directory is never yours to read or write. +- Check for this plan's ledger at `/progress.md`. If its first + line names your plan file, tasks with a `Task : complete` line are DONE + — do not re-dispatch them; resume at the first task without one. A task + whose last line is a fix round is mid-loop: resume the loop at the next + round. A ledger whose first line names a different plan file — or a stray + ledger at the old flat path `.superpowers/sdd/progress.md` — is another + plan's progress: leave it in place and start your own, fresh. +- Create the ledger with its identity as the first line: + `# SDD ledger — plan: `. +- The ledger is your recovery map: the commits it names exist in git even + when your context no longer remembers creating them. After compaction, + trust the ledger and `git log` over your own recollection. +- `git clean -fdx` will destroy the workspace (it's git-ignored scratch); if + that happens, recover from `git log`. + +Read the plan once, note its context and Global Constraints, and create a +todo per task. Before dispatching Task 1, scan the plan once for conflicts: @@ -115,7 +168,11 @@ capable available model, not the session default. **Review tasks**: choose the model with the same judgment, scaled to the diff's size, complexity, and risk. A small mechanical diff does not need the -most capable model; a subtle concurrency change does. +most capable model; a subtle concurrency change does. Scoped re-reviews of +small fix diffs take a cheap-to-mid tier. + +**Fix-loop escalation (rounds 4-5)**: use a model at least one tier above +the implementer that got stuck. **Always specify the model explicitly when dispatching a subagent.** An omitted model inherits your session's model — often the most capable and @@ -134,7 +191,47 @@ that implementer. Single-file mechanical fixes also take the cheapest tier. - Touches multiple files with integration concerns → standard model - Requires design judgment or broad codebase understanding → most capable model -## Handling Implementer Status +## The Task Loop + +Everything you paste into a dispatch prompt — and everything a subagent +prints back — stays resident in your context for the rest of the session +and is re-read on every later turn. Hand artifacts over as files. + +### 1. Dispatch the implementer + +Record BASE (`git rev-parse HEAD`) before dispatching — the review package +and fix-round diffs need it. + +- **Task brief:** before dispatching an implementer, run this skill's + `scripts/task-brief PLAN_FILE N` — it extracts the task's full text to a + uniquely named file and prints the path. Compose the dispatch so the + brief stays the single source of + requirements. Your dispatch should contain: (1) one line on where this + task fits in the project; (2) the brief path, introduced as "read this + first — it is your requirements, with the exact values to use verbatim"; + (3) interfaces and decisions from earlier tasks that the brief cannot + know; (4) your resolution of any ambiguity you noticed in the brief; + (5) the report-file path and report contract. Exact values (numbers, + magic strings, signatures, test cases) appear only in the brief. Never + make a subagent read the whole plan file. +- **Report file:** name the implementer's report file after the brief + (brief `…/task-N-brief.md` → report `…/task-N-report.md`) and put it in + the dispatch prompt. The implementer writes the full report there and + returns only status, commits, a one-line test summary, and concerns. +- A dispatch prompt describes one task, not the session's history. Do not + paste accumulated prior-task summaries ("state after Tasks 1-3") into + later dispatches — a real session's dispatch hit 42k chars of which 99% + was pasted history. A fresh subagent needs its task, the interfaces it + touches, and the global constraints. Nothing else. +- If an earlier task parked a finding in the area this task touches, carry + a pointer to that ledger entry in the dispatch. +- Record the implementer's agent identity from the dispatch result — + fix-loop rounds 1-3 resume this agent. +- Never dispatch multiple implementation subagents in parallel (conflicts). + +Template: [implementer-prompt.md](implementer-prompt.md) + +### 2. Handle the report Implementer subagents report one of four statuses. Handle each appropriately: @@ -152,20 +249,37 @@ Implementer subagents report one of four statuses. Handle each appropriately: **Never** ignore an escalation or force the same model to retry without changes. If the implementer said it's stuck, something needs to change. -## Handling Reviewer ⚠️ Items +If the implementer asks questions — before starting or mid-task — answer +clearly and completely, provide additional context if needed, and don't +rush it into implementation. -The task reviewer may report "⚠️ Cannot verify from diff" items — requirements -that live in unchanged code or span tasks. These do not block the rest of the -review, but you must resolve each one yourself before marking the task -complete: you hold the plan and cross-task context the reviewer -lacks. If you confirm an item is a real gap, treat it as a failed spec -review — send it back to the implementer and re-review. - -## Constructing Reviewer Prompts +### 3. Review the task Per-task reviews are task-scoped gates. The broad review happens once, at the -final whole-branch review. When you fill a reviewer template: +final whole-branch review. Never skip the task review, and never accept a +report missing either verdict — spec compliance AND task quality are both +required. Implementer self-review never replaces the task review; both are +needed. +- Hand the reviewer its diff as a file: run this skill's + `scripts/review-package PLAN_FILE BASE HEAD` and pass the reviewer the file path + it prints (or, without bash: `git log --oneline`, `git diff --stat`, + and `git diff -U10` for the range, redirected to one uniquely named + file). The output never enters your own context, and the reviewer sees + the commit list, stat summary, and full diff with context in one Read + call. Use the BASE you recorded before dispatching the implementer — + never `HEAD~1`, which silently truncates multi-commit tasks. Never + dispatch a task reviewer without a diff file. +- **Reviewer inputs:** the task reviewer gets three paths — the same brief + file, the report file, and the review package — plus the global + constraints that bind the task. +- The global-constraints block you hand the reviewer is its attention + lens. Copy the binding requirements verbatim from the plan's Global + Constraints section or the spec: exact values, exact formats, and the + stated relationships between components ("same layout as X", "matches + Y"). The reviewer's template already carries the process rules (YAGNI, + test hygiene, review method) — the constraints block is for what THIS + project's spec demands. - Do not add open-ended directives like "check all uses" or "run race tests if useful" without a concrete, task-specific reason - Do not ask a reviewer to re-run tests the implementer already ran on the @@ -176,122 +290,157 @@ final whole-branch review. When you fill a reviewer template: loop. If the prompt you are writing contains "do not flag," "don't treat X as a defect," "at most Minor," or "the plan chose" — stop: you are pre-judging, usually to spare yourself a review loop. -- The global-constraints block you hand the reviewer is its attention - lens. Copy the binding requirements verbatim from the plan's Global - Constraints section or the spec: exact values, exact formats, and the - stated relationships between components ("same layout as X", "matches - Y"). The reviewer's template already carries the process rules (YAGNI, - test hygiene, review method) — the constraints block is for what THIS - project's spec demands. -- Hand the reviewer its diff as a file: run this skill's - `scripts/review-package PLAN_FILE BASE HEAD` and pass the reviewer the - file path it prints (or, without bash: `git log --oneline`, - `git diff --stat`, and `git diff -U10` for the range, redirected to one - uniquely named file). The output never enters your own context, and the reviewer sees - the commit list, stat summary, and full diff with context in one Read - call. Use the BASE you recorded before dispatching the implementer — - never `HEAD~1`, which silently truncates multi-commit tasks. -- A dispatch prompt describes one task, not the session's history. Do not - paste accumulated prior-task summaries ("state after Tasks 1-3") into - later dispatches — a real session's dispatch hit 42k chars of which 99% - was pasted history. A fresh subagent needs its task, the interfaces it - touches, and the global constraints. Nothing else. -- Dispatch fix subagents for Critical and Important findings. Record Minor - findings in the progress ledger as you go, and point the final +The task reviewer may report "⚠️ Cannot verify from diff" items — requirements +that live in unchanged code or span tasks. These do not block the rest of the +review, but you must resolve each one yourself before marking the task +complete: you hold the plan and cross-task context the reviewer +lacks. If you confirm an item is a real gap, treat it as a failed spec +review — it enters the fix loop with the other findings. + +Template: [task-reviewer-prompt.md](task-reviewer-prompt.md) + +### 4. The fix loop + +The loop triggers when the review reports spec ❌, any Critical or Important +finding, or a ⚠️ item you confirmed as a real gap. + +Before the loop starts, two routes leave it immediately: + +- Record Minor findings in the progress ledger as you go + (`Task : minor (deferred): `), and point the final whole-branch review at that list so it can triage which must be fixed - before merge. A roll-up nobody reads is a silent discard. + before merge. A roll-up nobody reads is a silent discard. Minor findings + never enter the loop. - A finding labeled plan-mandated — or any finding that conflicts with what the plan's text requires — is the human's decision, like any plan contradiction: present the finding and the plan text, ask which governs. Do not dismiss the finding because the plan mandates it, and do not dispatch a fix that contradicts the plan without asking. -- The final whole-branch review gets a package too: run - `scripts/review-package PLAN_FILE MERGE_BASE HEAD` (MERGE_BASE = the - commit the branch started from, e.g. `git merge-base main HEAD`) and include the - printed path in the final review dispatch, so the final reviewer reads - one file instead of re-deriving the branch diff with git commands. -- Every fix dispatch carries the implementer contract: the fix subagent - re-runs the tests covering its change and reports the results. Name the - covering test files in the dispatch — a one-line fix does not need the - whole suite. Before re-dispatching the reviewer, confirm the fix report - contains the covering tests, the command run, and the output; dispatch - the re-review once all three are present. -- If the final whole-branch review returns findings, dispatch ONE fix - subagent with the complete findings list — not one fixer per finding. - Per-finding fixers each rebuild context and re-run suites; a real - session's final-review fix wave cost more than all its tasks combined. +Everything else enters the loop. A fix round is one fix dispatch plus one +scoped re-review. Five rounds maximum per task: -## File Handoffs +**Rounds 1-3 — resume the original implementer.** Send it the open findings +verbatim. Its context is intact: it knows the task, the code, and its own +choices. If your harness cannot send another message to a live subagent, +dispatch a fresh implementer carrying the brief path, the report-file path, +and the findings — the report file is the persistent memory either way. -Everything you paste into a dispatch prompt — and everything a subagent -prints back — stays resident in your context for the rest of the session -and is re-read on every later turn. Hand artifacts over as files: +**Rounds 4-5 — dispatch a fresh implementer on a more capable model** (per +Model Selection), with the brief path, the report-file path, the open +findings, and this framing: "A prior implementer attempted this task +[N] times; you own it now. Read the report file for what was tried." A loop +that survives three resumes usually means the implementer cannot see its +own problem — fresh eyes and a capability bump in one move. -- **Task brief:** before dispatching an implementer, run this skill's - `scripts/task-brief PLAN_FILE N` — it extracts the task's full text to a - uniquely named file and prints the path. Compose the dispatch so the - brief stays the single source of requirements. Your dispatch should - contain: (1) one line on where this task fits in the project; (2) the - brief path, introduced as "read this first — it is your requirements, - with the exact values to use verbatim"; (3) interfaces and decisions - from earlier tasks that the brief cannot know; (4) your resolution of - any ambiguity you noticed in the brief; (5) the report-file path and - report contract. Exact values (numbers, magic strings, signatures, test - cases) appear only in the brief. -- **Report file:** name the implementer's report file after the brief - (brief `…/task-N-brief.md` → report `…/task-N-report.md`) and put it in - the dispatch prompt. The implementer writes the full report there and - returns only status, commits, a one-line test summary, and concerns. -- **Reviewer inputs:** the task reviewer gets three paths — the same brief - file, the report file, and the review package — plus the global - constraints that bind the task. -- Fix dispatches append their fix report (with test results) to the same - report file and return a short summary; re-reviews read the updated file. +**Every round, either way:** the implementer fixes, re-runs the tests +covering the amended code, appends its fix report to the same report file, +and returns the short contract. Before re-dispatching the reviewer, confirm +the fix report contains the covering tests, the command run, and the +output; dispatch the re-review once all three are present. Name the +covering test files in the fix message — a one-line fix does not need the +whole suite. -## Durable Progress +**The re-review is scoped.** Run `scripts/review-package PLAN_FILE FIX_BASE HEAD` +where FIX_BASE is the head the previous review saw, and dispatch +[re-review-prompt.md](re-review-prompt.md) with the findings list, the +brief, the report file, and the printed diff path. The re-reviewer verdicts +each finding ADDRESSED or NOT ADDRESSED and flags new breakage in the fix +diff only. New Critical/Important breakage in the fix diff joins the open +findings list. Out-of-scope observations go to the ledger as deferred +minors — they never extend the loop. -Conversation memory does not survive compaction. In real sessions, -controllers that lost their place have re-dispatched entire completed task -sequences — the single most expensive failure observed. Track progress in -a ledger file, not only in todos. +**After each round,** append to the ledger: +`Task : fix round /5 ( addressed, open — ; commits ..)` -- Each plan owns a workspace: at skill start, run this skill's - `scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored - directory (`/.superpowers/sdd//`), home to - every artifact for THIS plan: ledger, briefs, reports, review packages. - Another plan's directory is never yours to read or write. -- Check for this plan's ledger at `/progress.md`. If its first - line names your plan file, tasks listed there as complete are DONE — do - not re-dispatch them; resume at the first task not marked complete. A - ledger whose first line names a different plan file — or a stray ledger - at the old flat path `.superpowers/sdd/progress.md` — is another plan's - progress: leave it in place and start your own, fresh. -- Create the ledger with its identity as the first line: - `# SDD ledger — plan: `. -- When a task's review comes back clean, append one line to the ledger in - the same message as your other bookkeeping: - `Task N: complete (commits .., review clean)`. -- The ledger is your recovery map: the commits it names exist in git even - when your context no longer remembers creating them. After compaction, - trust the ledger and `git log` over your own recollection. -- `git clean -fdx` will destroy the workspace (it's git-ignored scratch); if - that happens, recover from `git log`. -- When the final whole-branch review is clean and its fixes are merged, - delete this plan's workspace (`rm -rf `) — the git history - is the record now. Sibling directories belong to other plans; leave - them alone. +Never fix findings yourself in the controller session — your context stays +clean for coordination, and controller fixes skip review. -## Prompt Templates +**The breaker.** When round 5's re-review still leaves findings open, stop +dispatching. Adjudicate each open finding yourself — you hold the plan and +the cross-task context the reviewer lacks: -- [implementer-prompt.md](implementer-prompt.md) - Dispatch implementer subagent -- [task-reviewer-prompt.md](task-reviewer-prompt.md) - Dispatch task reviewer subagent (spec compliance + code quality) -- Final whole-branch review: use superpowers:requesting-code-review's [code-reviewer.md](../requesting-code-review/code-reviewer.md) +- **The reviewer is wrong, or the point is contestable:** park it — + `Task : parked — — ruling: `. The final + review sees both sides. +- **Real, but nothing downstream builds on it:** park it the same way, with + a ruling that says it's real and deferred. +- **Real and load-bearing** — a later task builds on it, or it reveals a + plan defect: STOP. Append `Task : BLOCKED — ` and report to + your human partner with the finding, the plan text it collides with, and + the fix history. Parking a structural failure lets every dependent task + build on it and hands the final review a problem it cannot fix either. + +Adjudicate only at the cap. Adjudicating earlier to end a loop is +pre-judging with a different name. Every adjudication is a ledger entry — +a silent discard is forbidden. + +### 5. Complete the task + +When the review comes back clean — or every open finding is parked with a +ruling at the cap — append the completion line to the ledger in the same +message as your other bookkeeping: + +- `Task : complete (commits .., review clean)` +- `Task : complete (commits .., parked)` after a + tripped breaker + +Then mark the todo complete and move on. Never move to the next task while +the review has open Critical/Important issues that are neither fixed nor +parked-with-ruling at the cap. + +## Final Review + +The final whole-branch review gets a package too: run +`scripts/review-package PLAN_FILE MERGE_BASE HEAD` (MERGE_BASE = the commit the +branch started from, e.g. `git merge-base main HEAD`) and include the +printed path in the final review dispatch, so the final reviewer reads +one file instead of re-deriving the branch diff with git commands. Dispatch +on the most capable available model (see Model Selection), using +superpowers:requesting-code-review's +[code-reviewer.md](../requesting-code-review/code-reviewer.md). Point it at +the ledger's deferred-minor and parked lines so it can triage which must be +fixed before merge. + +If the final whole-branch review returns findings, dispatch ONE fix subagent +with the complete findings list — not one fixer per finding. +Per-finding fixers each rebuild context and re-run suites; a real +session's final-review fix wave cost more than all its tasks combined. +Then run exactly one scoped re-review of the fix wave +(`scripts/review-package PLAN_FILE FIX_BASE HEAD` over the fix range, +[re-review-prompt.md](re-review-prompt.md)). +Adjudicate any residual findings as in the task loop's breaker: park with +rulings, or stop on load-bearing ones. There is no second fix wave — +residual load-bearing findings surface to your human partner when +finishing-a-development-branch presents the options. + +## Finish + +When the final whole-branch review is clean and its fixes are merged, +delete this plan's workspace (`rm -rf `) — the git history is +the record now. Sibling directories belong to other plans; leave them +alone. + +Use superpowers:finishing-a-development-branch. + +## Common Rationalizations + +| Excuse | Reality | +|--------|---------| +| "Close enough on spec compliance" | Reviewer found spec gaps = not done. Fix or hit the cap and adjudicate — those are the only exits. | +| "I'll fix it myself, dispatching is overhead" | Controller fixes pollute your context and skip review. Resume the implementer. | +| "One more round will converge" | Past the cap, rounds don't converge — the failure is structural. Adjudicate and route. | +| "The reviewer will just find something new anyway" | Scoped re-reviews verify fixes; they cannot wander. New findings on untouched code go to the ledger, not the loop. | +| "This finding is obviously wrong, I'll drop it" | You adjudicate only at the cap, and every ruling is a ledger entry. Silent discards are forbidden. | +| "The fix was small, skip the re-review" | Unreviewed fixes are how regressions land. Every round ends with a scoped re-review. | +| "Reviews slow the loop down" | The loop without reviews is just unverified churn. Reviews are the loop's brakes and steering. | +| "Ledger bookkeeping is overhead" | The ledger is what survives compaction. Controllers without one have re-dispatched entire completed task sequences. | ## Example Workflow ``` You: I'm using Subagent-Driven Development to execute this plan. +[Setup: worktree verified] [Read plan file once: docs/superpowers/plans/feature-plan.md] [Resolve workspace: scripts/sdd-workspace docs/superpowers/plans/feature-plan.md — no ledger inside, fresh start] [Create todos for all tasks] @@ -304,90 +453,51 @@ Implementer: "Before I begin - should the hook be installed at user or system le You: "User level (~/.config/superpowers/hooks/)" -Implementer: "Got it. Implementing now..." -[Later] Implementer: +Implementer: [Later] - Implemented install-hook command - Added tests, 5/5 passing - Self-review: Found I missed --force flag, added it - Committed -[Run review-package, dispatch task reviewer with the printed path] +[Run review-package PLAN_FILE BASE HEAD; dispatch task reviewer with the printed path] Task reviewer: Spec ✅ - all requirements met, nothing extra. Strengths: Good test coverage, clean. Issues: None. Task quality: Approved. -[Mark Task 1 complete] +[Ledger: Task 1: complete (commits a1b2c3d..d4e5f6a, review clean)] Task 2: Recovery modes [Run task-brief for Task 2; dispatch implementer with brief + report paths + context] -Implementer: [No questions, proceeds] -Implementer: +Implementer: [No questions] - Added verify/repair modes - 8/8 tests passing - - Self-review: All good - Committed -[Run review-package, dispatch task reviewer with the printed path] +[Run review-package PLAN_FILE BASE HEAD; dispatch task reviewer with the printed path] Task reviewer: Spec ❌: - Missing: Progress reporting (spec says "report every 100 items") - - Extra: Added --json flag (not requested) Issues (Important): Magic number (100) -[Dispatch fix subagent with all findings] -Fixer: Removed --json flag, added progress reporting, extracted PROGRESS_INTERVAL constant +[Fix round 1: resume the implementer with both findings] +Implementer: Added progress reporting, extracted PROGRESS_INTERVAL constant. + Re-ran test/recovery.test.js — 10/10 passing. Fix report appended. -[Task reviewer reviews again] -Task reviewer: Spec ✅. Task quality: Approved. +[Run review-package PLAN_FILE FIX_BASE HEAD; dispatch scoped re-review] +Re-reviewer: Missing progress reporting — ADDRESSED (src/recovery.js:41). + Magic number — ADDRESSED (src/recovery.js:7). New breakage: none. + Verdict: all findings addressed. -[Mark Task 2 complete] +[Ledger: Task 2: fix round 1/5 (2 addressed, 0 open; commits d4e5f6a..b7c8d9e)] +[Ledger: Task 2: complete (commits d4e5f6a..b7c8d9e, review clean)] ... [After all tasks] -[Dispatch final code-reviewer] -Final reviewer: All requirements met, ready to merge +[Run review-package PLAN_FILE MERGE_BASE HEAD; dispatch final code-reviewer, most capable model] +Final reviewer: All requirements met. Deferred minors triaged: none block merge. [Delete this plan's workspace — the record now lives in git] -Done! +Done! Using superpowers:finishing-a-development-branch. ``` - -## Red Flags - -**Never:** -- Start implementation on main/master branch without explicit user consent -- Skip task review, or accept a report missing either verdict (spec compliance AND task quality are both required) -- Proceed with unfixed issues -- Dispatch multiple implementation subagents in parallel (conflicts) -- Make a subagent read the whole plan file (hand it its task brief — - `scripts/task-brief` — instead) -- Skip scene-setting context (subagent needs to understand where task fits) -- Ignore subagent questions (answer before letting them proceed) -- Accept "close enough" on spec compliance (reviewer found spec issues = not done) -- Skip review loops (reviewer found issues = implementer fixes = review again) -- Let implementer self-review replace actual review (both are needed) -- Tell a reviewer what not to flag, or pre-rate a finding's severity in the - dispatch prompt ("treat it as Minor at most") — the plan's example code is - a starting point, not evidence that its weaknesses were chosen -- Dispatch a task reviewer without a diff file — generate it first - (`scripts/review-package PLAN_FILE BASE HEAD`) and name the printed - path in the prompt -- Move to next task while the review has open Critical/Important issues -- Re-dispatch a task the progress ledger already marks complete — check - the ledger (and `git log`) after any compaction or resume - -**If subagent asks questions:** -- Answer clearly and completely -- Provide additional context if needed -- Don't rush them into implementation - -**If reviewer finds issues:** -- Implementer (same subagent) fixes them -- Reviewer reviews again -- Repeat until approved -- Don't skip the re-review - -**If subagent fails task:** -- Dispatch fix subagent with specific instructions -- Don't try to fix manually (context pollution) diff --git a/skills/subagent-driven-development/re-review-prompt.md b/skills/subagent-driven-development/re-review-prompt.md index 84a45a46..18b0fb8a 100644 --- a/skills/subagent-driven-development/re-review-prompt.md +++ b/skills/subagent-driven-development/re-review-prompt.md @@ -100,7 +100,7 @@ Subagent (general-purpose): - `[REPORT_FILE]` — the implementer's report file (fix reports appended) - `[FIX_BASE_SHA]` — the head the previous review saw - `[HEAD_SHA]` — current commit -- `[DIFF_FILE]` — the path `scripts/review-package FIX_BASE HEAD` printed +- `[DIFF_FILE]` — the path `scripts/review-package PLAN_FILE FIX_BASE HEAD` printed **Re-reviewer returns:** per-finding verdicts (ADDRESSED / NOT ADDRESSED), new breakage in the fix diff, out-of-scope observations, and a round verdict. From a868631a8a4a942656b7837b60f24c393f86547a Mon Sep 17 00:00:00 2001 From: Mark Rada Date: Thu, 23 Jul 2026 13:47:44 -0400 Subject: [PATCH 48/51] docs(using-superpowers): drop dangling subagent-support anchor (#2010) The prune in e7ddc25e removed the `## Subagent support` section from antigravity-tools.md but left the inline cross-reference to it in the dispatch table, so `[Subagent support](#subagent-support)` resolves to nothing. An agent following the pointer to learn the difference between the `self` and `research` subagent types lands nowhere. Drop the dangling parenthetical. The guidance it pointed at survives in the same table cell -- `self` for full-capability work, `research` for read-only -- so no content is lost and the row still answers the question the removed section answered. gemini-tools.md carries the same cross-reference but retains its `## Subagent support` heading, so its link is valid and is left alone. --- skills/using-superpowers/references/antigravity-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/using-superpowers/references/antigravity-tools.md b/skills/using-superpowers/references/antigravity-tools.md index 71155fde..2e1eac90 100644 --- a/skills/using-superpowers/references/antigravity-tools.md +++ b/skills/using-superpowers/references/antigravity-tools.md @@ -4,7 +4,7 @@ Skills speak in actions ("dispatch a subagent", "create a todo", "read a file"). | Action skills request | Antigravity CLI equivalent | |----------------------|----------------------| -| Dispatch a subagent (`Subagent (general-purpose):` template) | `invoke_subagent` with a built-in `TypeName` — `self` for full-capability work, `research` for read-only (see [Subagent support](#subagent-support)) | +| Dispatch a subagent (`Subagent (general-purpose):` template) | `invoke_subagent` with a built-in `TypeName` — `self` for full-capability work, `research` for read-only | | Task tracking ("create a todo", "mark complete") | a **task artifact** — `write_to_file` with `IsArtifact: true` and `ArtifactType: "task"` (see [Task tracking](#task-tracking)). **Not** `manage_task`, which manages background processes. | ## Task tracking From 6015d37fe623cf0a7982d6f93fde0c6fef2f92b0 Mon Sep 17 00:00:00 2001 From: dev_Hakaze Date: Fri, 24 Jul 2026 00:53:39 +0700 Subject: [PATCH 49/51] fix(systematic-debugging): match find -path ./ prefix in find-polluter.sh (#2011) find . emits ./-prefixed paths, so -path "src/**/*.test.ts" matched nothing; wc -l on empty stdin then lied as "Found 1". Fixes #2008. Co-authored-by: arimu1 <19286898+arimu1@users.noreply.github.com> Co-authored-by: Cursor --- skills/systematic-debugging/find-polluter.sh | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/skills/systematic-debugging/find-polluter.sh b/skills/systematic-debugging/find-polluter.sh index 1d71c560..d2261c46 100755 --- a/skills/systematic-debugging/find-polluter.sh +++ b/skills/systematic-debugging/find-polluter.sh @@ -18,9 +18,13 @@ echo "🔍 Searching for test that creates: $POLLUTION_CHECK" echo "Test pattern: $TEST_PATTERN" echo "" -# Get list of test files -TEST_FILES=$(find . -path "$TEST_PATTERN" | sort) -TOTAL=$(echo "$TEST_FILES" | wc -l | tr -d ' ') +# Get list of test files (find . emits ./-prefixed paths) +TEST_FILES=$(find . -path "./$TEST_PATTERN" | sort) +if [ -z "$TEST_FILES" ]; then + TOTAL=0 +else + TOTAL=$(printf '%s\n' "$TEST_FILES" | wc -l | tr -d ' ') +fi echo "Found $TOTAL test files" echo "" From c8921b5156562b61af253150a4d1d01a71705be8 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Thu, 23 Jul 2026 10:54:50 -0700 Subject: [PATCH 50/51] fix(systematic-debugging): find-polluter accepts ./-prefixed patterns and matches top-level tests Follow-up to #2011 (which fixed the ./-prefix mismatch for the documented pattern form): strip a leading ./ from the caller's pattern instead of double-prefixing it into a never-matching ././ form, and also match the pattern with '**/' collapsed, since find -path cannot match '**/' against zero directory levels and silently skipped files directly under the base directory (src/top.test.ts vs src/**/*.test.ts). Adds a deterministic test suite for the script with a stubbed npm. --- skills/systematic-debugging/find-polluter.sh | 9 +- .../test-find-polluter.sh | 90 +++++++++++++++++++ 2 files changed, 97 insertions(+), 2 deletions(-) create mode 100755 tests/systematic-debugging/test-find-polluter.sh diff --git a/skills/systematic-debugging/find-polluter.sh b/skills/systematic-debugging/find-polluter.sh index d2261c46..985f5d08 100755 --- a/skills/systematic-debugging/find-polluter.sh +++ b/skills/systematic-debugging/find-polluter.sh @@ -18,8 +18,13 @@ echo "🔍 Searching for test that creates: $POLLUTION_CHECK" echo "Test pattern: $TEST_PATTERN" echo "" -# Get list of test files (find . emits ./-prefixed paths) -TEST_FILES=$(find . -path "./$TEST_PATTERN" | sort) +# Get list of test files (find . emits ./-prefixed paths, so accept the +# pattern written with or without a leading ./) +TEST_PATTERN="${TEST_PATTERN#./}" +# find -path can't match '**/' against zero directory levels, so a pattern +# like src/**/*.test.ts would skip src/top.test.ts; also try the pattern +# with '**/' collapsed to cover files directly under the base directory. +TEST_FILES=$(find . \( -path "./$TEST_PATTERN" -o -path "./${TEST_PATTERN//\*\*\//}" \) | sort -u) if [ -z "$TEST_FILES" ]; then TOTAL=0 else diff --git a/tests/systematic-debugging/test-find-polluter.sh b/tests/systematic-debugging/test-find-polluter.sh new file mode 100755 index 00000000..0902e7d9 --- /dev/null +++ b/tests/systematic-debugging/test-find-polluter.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +SCRIPT_UNDER_TEST="$REPO_ROOT/skills/systematic-debugging/find-polluter.sh" + +FAILURES=0 +TEST_ROOT="$(mktemp -d)" + +cleanup() { + rm -rf "$TEST_ROOT" +} +trap cleanup EXIT + +pass() { + echo " [PASS] $1" +} + +fail() { + echo " [FAIL] $1" + FAILURES=$((FAILURES + 1)) +} + +assert_contains() { + local haystack="$1" + local needle="$2" + local description="$3" + + if printf '%s' "$haystack" | grep -Fq -- "$needle"; then + pass "$description" + else + fail "$description (expected output to contain: $needle)" + fi +} + +# Toy project: one top-level test, one nested test. A stubbed `npm` on PATH +# creates the pollution marker whenever any test runs, so the first test file +# executed is always identified as the polluter. +setup_project() { + PROJECT="$TEST_ROOT/project" + rm -rf "$PROJECT" + mkdir -p "$PROJECT/src/feature" "$PROJECT/bin" + echo "test('top')" > "$PROJECT/src/top.test.ts" + echo "test('nested')" > "$PROJECT/src/feature/nested.test.ts" + cat > "$PROJECT/bin/npm" <<'EOF' +#!/usr/bin/env bash +touch pollution.marker +EOF + chmod +x "$PROJECT/bin/npm" +} + +# run_polluter — runs the script in the toy project with the stub +# npm first on PATH; captures combined output, never aborts on exit code. +run_polluter() { + local pattern="$1" + rm -f "$PROJECT/pollution.marker" + ( + cd "$PROJECT" + PATH="$PROJECT/bin:$PATH" "$SCRIPT_UNDER_TEST" 'pollution.marker' "$pattern" 2>&1 + ) || true +} + +echo "Test: documented pattern finds nested test files (issue #2008)" +setup_project +OUTPUT="$(run_polluter 'src/**/*.test.ts')" +assert_contains "$OUTPUT" "FOUND POLLUTER" "documented pattern runs tests and detects pollution" + +echo "Test: documented pattern also finds top-level test files" +setup_project +OUTPUT="$(run_polluter 'src/**/*.test.ts')" +assert_contains "$OUTPUT" "Found 2 test files" "src/**/*.test.ts matches src/top.test.ts and src/feature/nested.test.ts" + +echo "Test: ./-prefixed pattern matches the same files" +setup_project +OUTPUT="$(run_polluter './src/**/*.test.ts')" +assert_contains "$OUTPUT" "Found 2 test files" "leading ./ on the pattern is accepted" + +echo "Test: non-matching pattern reports an honest zero" +setup_project +OUTPUT="$(run_polluter 'nomatch/**/*.test.ts')" +assert_contains "$OUTPUT" "Found 0 test files" "empty result counts as 0, not 1" +assert_contains "$OUTPUT" "No polluter found" "empty result exits via the clean path" + +echo "" +if [ "$FAILURES" -gt 0 ]; then + echo "$FAILURES test(s) failed" + exit 1 +fi +echo "All tests passed" From 3dcbd5c4b48e02263fbf4a3c01e3fe4f81d584d9 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Thu, 23 Jul 2026 16:16:32 -0700 Subject: [PATCH 51/51] Release v6.2.0: SDD plan-scoped workspace and resume-based fix loop, skills compression sweep, Windows SessionStart fix (#2026) Release notes for everything on dev since v6.1.1, plus the version bump to 6.2.0 across all seven declared manifest files (bump-version.sh, audit clean). Tagging and marketplace publication happen after the dev -> main merge. --- .claude-plugin/marketplace.json | 2 +- .claude-plugin/plugin.json | 2 +- .codex-plugin/plugin.json | 2 +- .cursor-plugin/plugin.json | 2 +- .kimi-plugin/plugin.json | 2 +- RELEASE-NOTES.md | 33 +++++++++++++++++++++++++++++++++ gemini-extension.json | 2 +- package.json | 2 +- 8 files changed, 40 insertions(+), 7 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index acb6ae66..e058f5e4 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ { "name": "superpowers", "description": "Core skills library for Claude Code: TDD, debugging, collaboration patterns, and proven techniques", - "version": "6.1.1", + "version": "6.2.0", "source": "./", "author": { "name": "Jesse Vincent", diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 2fe026fd..5ec90b5e 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "superpowers", "description": "Core skills library for Claude Code: TDD, debugging, collaboration patterns, and proven techniques", - "version": "6.1.1", + "version": "6.2.0", "author": { "name": "Jesse Vincent", "email": "jesse@fsck.com" diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index a6b31431..c777f316 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "superpowers", - "version": "6.1.1", + "version": "6.2.0", "description": "An agentic skills framework & software development methodology that works: planning, TDD, debugging, and collaboration workflows.", "author": { "name": "Jesse Vincent", diff --git a/.cursor-plugin/plugin.json b/.cursor-plugin/plugin.json index 18d788b2..0902be0e 100644 --- a/.cursor-plugin/plugin.json +++ b/.cursor-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "superpowers", "displayName": "Superpowers", "description": "Core skills library: TDD, debugging, collaboration patterns, and proven techniques", - "version": "6.1.1", + "version": "6.2.0", "author": { "name": "Jesse Vincent", "email": "jesse@fsck.com" diff --git a/.kimi-plugin/plugin.json b/.kimi-plugin/plugin.json index e5d90d0c..55933e34 100644 --- a/.kimi-plugin/plugin.json +++ b/.kimi-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "superpowers", - "version": "6.1.1", + "version": "6.2.0", "description": "An agentic skills framework and software development methodology.", "author": { "name": "Jesse Vincent", diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md index 823948e7..46c51e38 100644 --- a/RELEASE-NOTES.md +++ b/RELEASE-NOTES.md @@ -1,5 +1,38 @@ # Superpowers Release Notes +## v6.2.0 (2026-07-23) + +### Subagent-Driven Development + +Two structural changes to how SDD tracks progress and closes out review findings, both developed against live eval campaigns. + +- **The workspace is now plan-scoped.** `.superpowers/sdd/` had no plan identity and no end-of-life: a follow-up plan in the same working tree could read the previous plan's ledger as its own progress (observed in the wild, with multiple contamination rounds and ad-hoc workarounds). `sdd-workspace` now requires the plan file and resolves a per-plan directory, `.superpowers/sdd//`; `task-brief` and `review-package` write into their plan's directory (`review-package` gains the plan file as its first argument); the ledger names its plan on its first line; and the workspace is deleted once the final review is clean — git history is the durable record. Baseline evals showed controllers already refused foreign ledgers, but at a cost of 6–13 tool calls of cross-plan git forensics per resume; plan-scoping makes the answer structural instead. (25/25 baseline and GREEN eval runs documented in `docs/specs/` and `docs/plans/`.) +- **The review-fix loop resumes the implementer.** The lifecycle restructure gives fix rounds resume-the-implementer semantics instead of fresh dispatches, adds a scoped re-review prompt (`re-review-prompt.md`) so the re-reviewer checks the fixes rather than re-reading the whole task, and installs a five-round circuit breaker with controller adjudication when it trips. SKILL.md reorganizes by lifecycle, and its Red Flags convert to the house rationalization-table form. + +### Skills + +A branch-wide compression campaign: recap sections, social proof, and benefits-selling prose aimed at a reader who has already invoked the skill are gone, with every load-bearing argument folded into a rationalization-table row or moved to its point of use. Each cut was micro-tested with subagent probes, and the one cut that measurably degraded behavior was reworked rather than shipped. + +- **`testing-anti-patterns.md` is now `writing-good-tests.md`.** The TDD reference doc is rebuilt as a positive catalog — six rules that lead with the GOOD example — and absorbs a falsifiability discipline: name the production change that would fail the test, derive expectations independently of the code under test, and a closing mutation check. It closes two holes by name: the string-presence trap (grep-style tests on scripts, skills, and prompts counterfeit falsifiability — the observable is behavior, never text) and the change-detector trap (a constant assertion can fail and still protect nothing), each with a hard stop in the gate function. Trivial code and human prose earn no test; the trigger broadens from "adding mocks" to any test writing. +- **TDD's "Why Order Matters" rebuttals survive as rationalization rows.** Deleting the section outright measurably degraded test-first behavior under "just write it, tests after" pressure (control 8/10 → treatment 5/10, corroborated on Claude and Codex), so each prose rebuttal now lives in its Common Rationalizations row — the section is gone but the arguments fire where an agent hits them mid-rationalization. +- **`finishing-a-development-branch` no longer offers to discard your work.** The completion menu dates from when throwing away branches was routine; "Discard this work" next to "Merge" advertised destroying finished, passing work. Discard survives as an explicit-request-only path with the same typed-confirmation ritual. The same pass made PR creation forge-agnostic (your forge's CLI or the URL printed on push, not a blessed list of tools) and fixed a real bug: the worktree path was recomputed after cleanup had already changed directory, so provenance checks never matched and cleanup silently no-oped. +- **Recap and persuasion prose removed across the library.** `brainstorming`, `systematic-debugging`, `dispatching-parallel-agents`, `verification-before-completion`, `executing-plans`, `subagent-driven-development`, `requesting-code-review`, `receiving-code-review`, `using-git-worktrees`, `writing-plans`, and `writing-skills` all drop their Bottom Line / Key Principles / Real-World Impact / Advantages sections; `using-git-worktrees` and `finishing-a-development-branch` convert their guard sections to the house Excuse/Reality rationalization table. + +### Windows + +- **The SessionStart hook now dispatches via Git Bash.** The hook's command string starts with a quoted path, which broke both shells Claude Code might hand it to: PowerShell parsed the quoted string as an expression and died with a parser error (#1751), and cmd.exe's quote-stripping rule truncated the command when the profile path contained a metacharacter like `(` (#1918) — either way the bootstrap silently never loaded. The hook now declares `shell: "bash"`, which Claude Code ≥ 2.1.81 resolves to Git for Windows directly, and which surfaces an actionable install prompt when Git Bash is missing. Older Claude Code versions ignore the unknown key and behave as before. Verified end-to-end on Linux, Windows 11 with Git Bash under a hostile path, and Windows 11 without Git Bash. + +### Harness Support + +- **Gemini CLI support is restored.** The v6.1.0 removal (on the news that Google had EOLed the Gemini CLI) was premature; the install docs and the `gemini-tools.md` tool-mapping reference are back while permanent removal gets a proper evaluation. (#1959) + +### Fixes + +- **`find-polluter.sh` actually finds test files now.** `find .` emits `./`-prefixed paths, so the documented `-path "src/**/*.test.ts"` pattern matched nothing — and `wc -l` on empty input then reported "Found 1". Fixed the prefix mismatch (#2008, #2011), plus two follow-ups: a caller-supplied `./`-prefixed pattern no longer double-prefixes into a never-matching form, and `**/` is also matched collapsed so tests directly under the base directory (`src/top.test.ts` vs `src/**/*.test.ts`) aren't silently skipped. The script gains a deterministic test suite. +- **The Codex package script works beyond macOS.** Deterministic-metadata tar flags were bsdtar-only spellings, staged file modes depended on two umasks canceling out, and the test's timestamp assertion parsed bsdtar's column layout in a US timezone. GNU tar now gets equivalent flags producing byte-identical headers, modes are pinned canonical, and the test asserts mtime via `tarfile`. +- **SDD's skill test no longer flakes.** The file's worst case exceeded the runner's per-file ceiling (raised to 900s), and the assert helpers matched free-form model prose case-sensitively; matching is now case-insensitive and `assert_order` dumps output on failure so the next flake is diagnosable. +- **Docs and test cleanup after the v6.1.0 reference pruning.** Dead links to the deleted `claude-code-tools.md`/`copilot-tools.md` are replaced with the current architecture (#1969), a dangling `#subagent-support` anchor in the Antigravity reference is dropped (#2010), and the Antigravity/Pi mapping tests assert only the surviving harness-specific mappings — scoped to the table so they fail again if it's deleted. + ## v6.1.1 (2026-07-02) ### Codex diff --git a/gemini-extension.json b/gemini-extension.json index dc5e1f64..01378c98 100644 --- a/gemini-extension.json +++ b/gemini-extension.json @@ -1,6 +1,6 @@ { "name": "superpowers", "description": "Core skills library: TDD, debugging, collaboration patterns, and proven techniques", - "version": "6.1.1", + "version": "6.2.0", "contextFileName": "GEMINI.md" } diff --git a/package.json b/package.json index ad25028d..c24b3721 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "superpowers", - "version": "6.1.1", + "version": "6.2.0", "description": "Superpowers skills and runtime bootstrap for coding agents", "type": "module", "main": ".opencode/plugins/superpowers.js",