Compare commits

..

3 Commits

Author SHA1 Message Date
Jesse Vincent
9887c17b31 fix(writing-skills): hang backfire mechanism on the separated prohibition-vs-recipe comparison (NEW-4); control comparison stated as trend 2026-06-11 11:30:31 -07:00
Jesse Vincent
391c33edf8 fix(writing-skills): scope empirical claims, honest noise reporting, conditionalize micro-test checklist line
Adversarial review findings 1/3/9: the head-to-head result is now scoped
to its context (dispatch-prompt guidance) with an explicit micro-test-your-
own-case instruction; the nuance-clause result is reported as
consistent->noisy rather than 'measurably dilutes'; the checklist line is
scoped to behavior-shaping guidance and the micro method no longer assumes
raw API access.
2026-06-11 11:10:33 -07:00
Jesse Vincent
95795c2e7b feat(writing-skills): form-selection table + micro-test wording method
RED battery (35 opus authoring samples against the current skill) showed
authors default to prohibition+rationalization-table for composition-
shaping problems (T1: 5/5), where that form measurably backfires
(prohibition 4.4 vs 3.6 no-guidance control vs 3.0 recipe restatement
errors), and design only full-subagent verification with no wording
micro-tests, no mandatory no-guidance control, no manual inspection of
automated matches, no variance signal (T7: 5/5).

Adds: Match the Form to the Failure (failure-type -> form table, nuance/
exemption rules), scope note on Bulletproofing, Micro-Test Wording
subsection, two checklist lines. Deliberately narrow: T3/T4/T5/T6 RED
samples showed Iron Law / elicit-first behavior already strong.
2026-06-11 10:20:24 -07:00
7 changed files with 40 additions and 17 deletions

View File

@@ -26,7 +26,7 @@ You MUST create a task for each of these items and complete them in order:
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria 3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
4. **Propose 2-3 approaches** — with trade-offs and your recommendation 4. **Propose 2-3 approaches** — with trade-offs and your recommendation
5. **Present design** — in sections scaled to their complexity, get user approval after each section 5. **Present design** — in sections scaled to their complexity, get user approval after each section
6. **Write design doc** — save to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md` and commit (exactly this path — not `docs/specs/`) 6. **Write design doc** — save to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md` and commit
7. **Spec self-review** — quick inline check for placeholders, contradictions, ambiguity, scope (see below) 7. **Spec self-review** — quick inline check for placeholders, contradictions, ambiguity, scope (see below)
8. **User reviews written spec** — ask user to review the spec file before proceeding 8. **User reviews written spec** — ask user to review the spec file before proceeding
9. **Transition to implementation** — invoke writing-plans skill to create implementation plan 9. **Transition to implementation** — invoke writing-plans skill to create implementation plan
@@ -109,7 +109,7 @@ digraph brainstorming {
**Documentation:** **Documentation:**
- Write the validated design (spec) to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md` - Write the validated design (spec) to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md`
- (An explicit user instruction overrides this default; an existing differently-named docs directory does not) - (User preferences for spec location override this default)
- Use elements-of-style:writing-clearly-and-concisely skill if available - Use elements-of-style:writing-clearly-and-concisely skill if available
- Commit the design document to git - Commit the design document to git

View File

@@ -16,7 +16,7 @@ Load plan, review critically, execute all tasks, report when complete.
## The Process ## The Process
### Step 1: Load and Review Plan ### Step 1: Load and Review Plan
1. Read plan file, and the spec it cites in its `**Spec:**` header (plans reference requirements rather than restating them) 1. Read plan file
2. Review critically - identify any questions or concerns about the plan 2. Review critically - identify any questions or concerns about the plan
3. If concerns: Raise them with your human partner before starting 3. If concerns: Raise them with your human partner before starting
4. If no concerns: Create todos for the plan items and proceed 4. If no concerns: Create todos for the plan items and proceed

View File

@@ -86,10 +86,6 @@ digraph process {
} }
``` ```
## Spec Context
If the plan's header cites a spec (`**Spec:** <path>`), read it once during plan extraction. Plans reference requirements rather than restating them — when a task cites a spec section, paste that section's text into the implementer and spec-reviewer prompts along with the task text. Implementer subagents never read the spec file themselves; the spec reviewer may additionally read it at the cited path (its prompt says so).
## Model Selection ## Model Selection
Use the least powerful model that can handle each role to conserve cost and increase speed. Use the least powerful model that can handle each role to conserve cost and increase speed.

View File

@@ -12,8 +12,6 @@ Subagent (general-purpose):
[FULL TEXT of task from plan - paste it here, don't make subagent read file] [FULL TEXT of task from plan - paste it here, don't make subagent read file]
[If the task cites spec sections, paste the cited sections' text here too]
## Context ## Context
[Scene-setting: where this fits, dependencies, architectural context] [Scene-setting: where this fits, dependencies, architectural context]

View File

@@ -12,7 +12,7 @@ Subagent (general-purpose):
## What Was Requested ## What Was Requested
[FULL TEXT of task requirements, including the text of any spec sections the task cites] [FULL TEXT of task requirements]
## What Implementer Claims They Built ## What Implementer Claims They Built
@@ -28,7 +28,7 @@ Subagent (general-purpose):
git diff [BASE_SHA]..[HEAD_SHA] git diff [BASE_SHA]..[HEAD_SHA]
``` ```
Only read files in this diff. Do not crawl the broader codebase. (One exception: if the requirements cite a spec document, you may read that spec at its cited path.) Only read files in this diff. Do not crawl the broader codebase.
## Read-Only Review ## Read-Only Review

View File

@@ -7,18 +7,16 @@ description: Use when you have a spec or requirements for a multi-step task, bef
## Overview ## Overview
Write comprehensive implementation plans assuming the engineer has zero context for our codebase and questionable taste. Document everything they need to execute: which files to touch for each task, code, testing, docs they might need to check, how to test it. Give them the whole plan as bite-sized tasks. DRY. YAGNI. TDD. Frequent commits. Write comprehensive implementation plans assuming the engineer has zero context for our codebase and questionable taste. Document everything they need to know: which files to touch for each task, code, testing, docs they might need to check, how to test it. Give them the whole plan as bite-sized tasks. DRY. YAGNI. TDD. Frequent commits.
Assume they are a skilled developer, but know almost nothing about our toolset or problem domain. Assume they don't know good test design very well. Assume they are a skilled developer, but know almost nothing about our toolset or problem domain. Assume they don't know good test design very well.
**Plans reference the spec; they never restate, paraphrase, or summarize it.** The spec owns the WHAT and WHY — requirements, acceptance criteria, design decisions; the plan owns the HOW — tasks, files, code, commands. Cite it by path in the header and by section where a task needs context. Reference discipline never means skipping the spec: if brainstorming produced one, it exists and the plan cites it. No Placeholders still requires repeating code and commands WITHIN the plan; copying FROM the spec is different: a step that needs a requirement's prose is under-specified — turn it into a concrete action. Snapshotting spec text into the plan hides drift, not prevents it. "Zero context" means each step is mechanically executable, not that the plan repeats the spec.
**Announce at start:** "I'm using the writing-plans skill to create the implementation plan." **Announce at start:** "I'm using the writing-plans skill to create the implementation plan."
**Context:** If working in an isolated worktree, it should have been created via the `superpowers:using-git-worktrees` skill at execution time. **Context:** If working in an isolated worktree, it should have been created via the `superpowers:using-git-worktrees` skill at execution time.
**Save plans to:** `docs/superpowers/plans/YYYY-MM-DD-<feature-name>.md` **Save plans to:** `docs/superpowers/plans/YYYY-MM-DD-<feature-name>.md`
- (An explicit user instruction overrides this default; an existing differently-named docs directory does not) - (User preferences for plan location override this default)
## Scope Check ## Scope Check
@@ -55,8 +53,6 @@ This structure informs the task decomposition. Each task should produce self-con
**Goal:** [One sentence describing what this builds] **Goal:** [One sentence describing what this builds]
**Spec:** [Path to the spec doc, e.g. `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md` — requirements and design decisions live there; do not restate them here. Only if no spec doc exists (requirements arrived conversationally; brainstorming never ran): write "none — requirements:" and state them once here, not per task]
**Architecture:** [2-3 sentences about approach] **Architecture:** [2-3 sentences about approach]
**Tech Stack:** [Key technologies/libraries] **Tech Stack:** [Key technologies/libraries]

View File

@@ -456,10 +456,29 @@ Different skill types need different test approaches:
**All of these mean: Test before deploying. No exceptions.** **All of these mean: Test before deploying. No exceptions.**
## Match the Form to the Failure
Before writing guidance, classify the baseline failure. The form that bulletproofs one failure type measurably backfires on another.
| Baseline failure | Right form | Wrong form |
|---|---|---|
| Skips/violates a rule under pressure (knows better, does it anyway) | Prohibition + rationalization table + red flags (see Bulletproofing below) | Soft guidance ("prefer...", "consider...") |
| Complies, but output has the wrong shape (bloated prompt, buried verdict, restated spec) | Positive recipe or contract: state what the output IS — its parts, in order | Prohibition list ("don't restate", "never narrate") |
| Omits a required element from something they already produce | Structural: REQUIRED field or slot in the template they fill in | Prose reminders near the template |
| Behavior should depend on a condition | Conditional keyed to an observable predicate ("if the brief exists, reference it") | Unconditional rule + exemption clauses |
**Why prohibitions backfire on shaping problems:** under a competing incentive ("make the prompt self-contained"), agents negotiate with "don't X". In head-to-head wording tests on dispatch-prompt guidance, the prohibition arm produced clearly more of the unwanted content than the recipe arm (fully separated distributions), and trended worse than even the no-guidance control — micro-test your own case rather than assuming, but never reach for the prohibition by default. A recipe leaves nothing to negotiate: the output matches the stated shape or it doesn't.
**Rules for whichever form you pick:**
- **No nuance clauses.** "Don't X unless it matters" reopens the negotiation — appending a single nuance clause to a winning recipe degraded it from consistent to noisy in the same wording tests. Express a real exception as its own conditional on an observable predicate.
- **Exemption clauses don't scope.** "This limit doesn't apply to code blocks" still suppresses code blocks. If part of the output must be exempt, restructure so the rule can't reach it.
## Bulletproofing Skills Against Rationalization ## Bulletproofing Skills Against Rationalization
Skills that enforce discipline (like TDD) need to resist rationalization. Agents are smart and will find loopholes when under pressure. Skills that enforce discipline (like TDD) need to resist rationalization. Agents are smart and will find loopholes when under pressure.
**Scope:** this toolkit is for discipline failures — an agent that knows the rule and skips it under pressure. For wrong-shaped output or omitted elements, prohibition-based bulletproofing backfires; use the forms in Match the Form to the Failure instead.
**Psychology note:** Understanding WHY persuasion techniques work helps you apply them systematically. See persuasion-principles.md for research foundation (Cialdini, 2021; Meincke et al., 2025) on authority, commitment, scarcity, social proof, and unity principles. **Psychology note:** Understanding WHY persuasion techniques work helps you apply them systematically. See persuasion-principles.md for research foundation (Cialdini, 2021; Meincke et al., 2025) on authority, commitment, scarcity, social proof, and unity principles.
### Close Every Loophole Explicitly ### Close Every Loophole Explicitly
@@ -553,6 +572,18 @@ Run same scenarios WITH skill. Agent should now comply.
Agent found new rationalization? Add explicit counter. Re-test until bulletproof. Agent found new rationalization? Add explicit counter. Re-test until bulletproof.
### Micro-Test Wording Before Full Scenarios
Full pressure-scenario runs are the final gate, but they are slow and expensive per iteration. Verify the wording itself first with micro-tests:
1. **One fresh-context sample per call** — a raw API call, or a single-shot subagent if you don't have API access. System prompt = the realistic context the guidance will live in (the full skill or prompt template, not the guidance in isolation); user message = a task that tempts the failure.
2. **Always include a no-guidance control.** If the control doesn't exhibit the failure, there is nothing to fix — stop, don't author the guidance.
3. **5+ reps per variant.** Single samples lie.
4. **Manually read every flagged match.** Score programmatically if you like, but template echoes and quoted counter-examples masquerade as hits; automated counts alone overstate both failure and success.
5. **Variance is a metric.** When guidance lands, reps converge on the same shape. Five different interpretations across five reps means the wording isn't binding — tighten the form before adding words.
Micro-tests verify wording; they do not replace pressure scenarios for discipline skills.
**Testing methodology:** See [testing-skills-with-subagents.md](testing-skills-with-subagents.md) for the complete testing methodology: **Testing methodology:** See [testing-skills-with-subagents.md](testing-skills-with-subagents.md) for the complete testing methodology:
- How to write pressure scenarios - How to write pressure scenarios
- Pressure types (time, sunk cost, authority, exhaustion) - Pressure types (time, sunk cost, authority, exhaustion)
@@ -610,6 +641,8 @@ Deploying untested skills = deploying untested code. It's a violation of quality
- [ ] Keywords throughout for search (errors, symptoms, tools) - [ ] Keywords throughout for search (errors, symptoms, tools)
- [ ] Clear overview with core principle - [ ] Clear overview with core principle
- [ ] Address specific baseline failures identified in RED - [ ] Address specific baseline failures identified in RED
- [ ] Guidance form matches the failure type (see Match the Form to the Failure)
- [ ] For behavior-shaping guidance: wording micro-tested against a no-guidance control (5+ reps, every flagged match read manually) — N/A for pure reference skills
- [ ] Code inline OR link to separate file - [ ] Code inline OR link to separate file
- [ ] One excellent example (not multi-language) - [ ] One excellent example (not multi-language)
- [ ] Run scenarios WITH skill - verify agents now comply - [ ] Run scenarios WITH skill - verify agents now comply