Compare commits

..

3 Commits

Author SHA1 Message Date
Jesse Vincent
9887c17b31 fix(writing-skills): hang backfire mechanism on the separated prohibition-vs-recipe comparison (NEW-4); control comparison stated as trend 2026-06-11 11:30:31 -07:00
Jesse Vincent
391c33edf8 fix(writing-skills): scope empirical claims, honest noise reporting, conditionalize micro-test checklist line
Adversarial review findings 1/3/9: the head-to-head result is now scoped
to its context (dispatch-prompt guidance) with an explicit micro-test-your-
own-case instruction; the nuance-clause result is reported as
consistent->noisy rather than 'measurably dilutes'; the checklist line is
scoped to behavior-shaping guidance and the micro method no longer assumes
raw API access.
2026-06-11 11:10:33 -07:00
Jesse Vincent
95795c2e7b feat(writing-skills): form-selection table + micro-test wording method
RED battery (35 opus authoring samples against the current skill) showed
authors default to prohibition+rationalization-table for composition-
shaping problems (T1: 5/5), where that form measurably backfires
(prohibition 4.4 vs 3.6 no-guidance control vs 3.0 recipe restatement
errors), and design only full-subagent verification with no wording
micro-tests, no mandatory no-guidance control, no manual inspection of
automated matches, no variance signal (T7: 5/5).

Adds: Match the Form to the Failure (failure-type -> form table, nuance/
exemption rules), scope note on Bulletproofing, Micro-Test Wording
subsection, two checklist lines. Deliberately narrow: T3/T4/T5/T6 RED
samples showed Iron Law / elicit-first behavior already strong.
2026-06-11 10:20:24 -07:00
7 changed files with 40 additions and 17 deletions

View File

@@ -26,7 +26,7 @@ You MUST create a task for each of these items and complete them in order:
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
4. **Propose 2-3 approaches** — with trade-offs and your recommendation
5. **Present design** — in sections scaled to their complexity, get user approval after each section
6. **Write design doc** — save to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md` and commit (exactly this path — not `docs/specs/`)
6. **Write design doc** — save to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md` and commit
7. **Spec self-review** — quick inline check for placeholders, contradictions, ambiguity, scope (see below)
8. **User reviews written spec** — ask user to review the spec file before proceeding
9. **Transition to implementation** — invoke writing-plans skill to create implementation plan
@@ -109,7 +109,7 @@ digraph brainstorming {
**Documentation:**
- Write the validated design (spec) to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md`
- (An explicit user instruction overrides this default; an existing differently-named docs directory does not)
- (User preferences for spec location override this default)
- Use elements-of-style:writing-clearly-and-concisely skill if available
- Commit the design document to git

View File

@@ -16,7 +16,7 @@ Load plan, review critically, execute all tasks, report when complete.
## The Process
### Step 1: Load and Review Plan
1. Read plan file, and the spec it cites in its `**Spec:**` header (plans reference requirements rather than restating them)
1. Read plan file
2. Review critically - identify any questions or concerns about the plan
3. If concerns: Raise them with your human partner before starting
4. If no concerns: Create todos for the plan items and proceed

View File

@@ -86,10 +86,6 @@ digraph process {
}
```
## Spec Context
If the plan's header cites a spec (`**Spec:** <path>`), read it once during plan extraction. Plans reference requirements rather than restating them — when a task cites a spec section, paste that section's text into the implementer and spec-reviewer prompts along with the task text. Implementer subagents never read the spec file themselves; the spec reviewer may additionally read it at the cited path (its prompt says so).
## Model Selection
Use the least powerful model that can handle each role to conserve cost and increase speed.

View File

@@ -12,8 +12,6 @@ Subagent (general-purpose):
[FULL TEXT of task from plan - paste it here, don't make subagent read file]
[If the task cites spec sections, paste the cited sections' text here too]
## Context
[Scene-setting: where this fits, dependencies, architectural context]

View File

@@ -12,7 +12,7 @@ Subagent (general-purpose):
## What Was Requested
[FULL TEXT of task requirements, including the text of any spec sections the task cites]
[FULL TEXT of task requirements]
## What Implementer Claims They Built
@@ -28,7 +28,7 @@ Subagent (general-purpose):
git diff [BASE_SHA]..[HEAD_SHA]
```
Only read files in this diff. Do not crawl the broader codebase. (One exception: if the requirements cite a spec document, you may read that spec at its cited path.)
Only read files in this diff. Do not crawl the broader codebase.
## Read-Only Review

View File

@@ -7,18 +7,16 @@ description: Use when you have a spec or requirements for a multi-step task, bef
## Overview
Write comprehensive implementation plans assuming the engineer has zero context for our codebase and questionable taste. Document everything they need to execute: which files to touch for each task, code, testing, docs they might need to check, how to test it. Give them the whole plan as bite-sized tasks. DRY. YAGNI. TDD. Frequent commits.
Write comprehensive implementation plans assuming the engineer has zero context for our codebase and questionable taste. Document everything they need to know: which files to touch for each task, code, testing, docs they might need to check, how to test it. Give them the whole plan as bite-sized tasks. DRY. YAGNI. TDD. Frequent commits.
Assume they are a skilled developer, but know almost nothing about our toolset or problem domain. Assume they don't know good test design very well.
**Plans reference the spec; they never restate, paraphrase, or summarize it.** The spec owns the WHAT and WHY — requirements, acceptance criteria, design decisions; the plan owns the HOW — tasks, files, code, commands. Cite it by path in the header and by section where a task needs context. Reference discipline never means skipping the spec: if brainstorming produced one, it exists and the plan cites it. No Placeholders still requires repeating code and commands WITHIN the plan; copying FROM the spec is different: a step that needs a requirement's prose is under-specified — turn it into a concrete action. Snapshotting spec text into the plan hides drift, not prevents it. "Zero context" means each step is mechanically executable, not that the plan repeats the spec.
**Announce at start:** "I'm using the writing-plans skill to create the implementation plan."
**Context:** If working in an isolated worktree, it should have been created via the `superpowers:using-git-worktrees` skill at execution time.
**Save plans to:** `docs/superpowers/plans/YYYY-MM-DD-<feature-name>.md`
- (An explicit user instruction overrides this default; an existing differently-named docs directory does not)
- (User preferences for plan location override this default)
## Scope Check
@@ -55,8 +53,6 @@ This structure informs the task decomposition. Each task should produce self-con
**Goal:** [One sentence describing what this builds]
**Spec:** [Path to the spec doc, e.g. `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md` — requirements and design decisions live there; do not restate them here. Only if no spec doc exists (requirements arrived conversationally; brainstorming never ran): write "none — requirements:" and state them once here, not per task]
**Architecture:** [2-3 sentences about approach]
**Tech Stack:** [Key technologies/libraries]

View File

@@ -456,10 +456,29 @@ Different skill types need different test approaches:
**All of these mean: Test before deploying. No exceptions.**
## Match the Form to the Failure
Before writing guidance, classify the baseline failure. The form that bulletproofs one failure type measurably backfires on another.
| Baseline failure | Right form | Wrong form |
|---|---|---|
| Skips/violates a rule under pressure (knows better, does it anyway) | Prohibition + rationalization table + red flags (see Bulletproofing below) | Soft guidance ("prefer...", "consider...") |
| Complies, but output has the wrong shape (bloated prompt, buried verdict, restated spec) | Positive recipe or contract: state what the output IS — its parts, in order | Prohibition list ("don't restate", "never narrate") |
| Omits a required element from something they already produce | Structural: REQUIRED field or slot in the template they fill in | Prose reminders near the template |
| Behavior should depend on a condition | Conditional keyed to an observable predicate ("if the brief exists, reference it") | Unconditional rule + exemption clauses |
**Why prohibitions backfire on shaping problems:** under a competing incentive ("make the prompt self-contained"), agents negotiate with "don't X". In head-to-head wording tests on dispatch-prompt guidance, the prohibition arm produced clearly more of the unwanted content than the recipe arm (fully separated distributions), and trended worse than even the no-guidance control — micro-test your own case rather than assuming, but never reach for the prohibition by default. A recipe leaves nothing to negotiate: the output matches the stated shape or it doesn't.
**Rules for whichever form you pick:**
- **No nuance clauses.** "Don't X unless it matters" reopens the negotiation — appending a single nuance clause to a winning recipe degraded it from consistent to noisy in the same wording tests. Express a real exception as its own conditional on an observable predicate.
- **Exemption clauses don't scope.** "This limit doesn't apply to code blocks" still suppresses code blocks. If part of the output must be exempt, restructure so the rule can't reach it.
## Bulletproofing Skills Against Rationalization
Skills that enforce discipline (like TDD) need to resist rationalization. Agents are smart and will find loopholes when under pressure.
**Scope:** this toolkit is for discipline failures — an agent that knows the rule and skips it under pressure. For wrong-shaped output or omitted elements, prohibition-based bulletproofing backfires; use the forms in Match the Form to the Failure instead.
**Psychology note:** Understanding WHY persuasion techniques work helps you apply them systematically. See persuasion-principles.md for research foundation (Cialdini, 2021; Meincke et al., 2025) on authority, commitment, scarcity, social proof, and unity principles.
### Close Every Loophole Explicitly
@@ -553,6 +572,18 @@ Run same scenarios WITH skill. Agent should now comply.
Agent found new rationalization? Add explicit counter. Re-test until bulletproof.
### Micro-Test Wording Before Full Scenarios
Full pressure-scenario runs are the final gate, but they are slow and expensive per iteration. Verify the wording itself first with micro-tests:
1. **One fresh-context sample per call** — a raw API call, or a single-shot subagent if you don't have API access. System prompt = the realistic context the guidance will live in (the full skill or prompt template, not the guidance in isolation); user message = a task that tempts the failure.
2. **Always include a no-guidance control.** If the control doesn't exhibit the failure, there is nothing to fix — stop, don't author the guidance.
3. **5+ reps per variant.** Single samples lie.
4. **Manually read every flagged match.** Score programmatically if you like, but template echoes and quoted counter-examples masquerade as hits; automated counts alone overstate both failure and success.
5. **Variance is a metric.** When guidance lands, reps converge on the same shape. Five different interpretations across five reps means the wording isn't binding — tighten the form before adding words.
Micro-tests verify wording; they do not replace pressure scenarios for discipline skills.
**Testing methodology:** See [testing-skills-with-subagents.md](testing-skills-with-subagents.md) for the complete testing methodology:
- How to write pressure scenarios
- Pressure types (time, sunk cost, authority, exhaustion)
@@ -610,6 +641,8 @@ Deploying untested skills = deploying untested code. It's a violation of quality
- [ ] Keywords throughout for search (errors, symptoms, tools)
- [ ] Clear overview with core principle
- [ ] Address specific baseline failures identified in RED
- [ ] Guidance form matches the failure type (see Match the Form to the Failure)
- [ ] For behavior-shaping guidance: wording micro-tested against a no-guidance control (5+ reps, every flagged match read manually) — N/A for pure reference skills
- [ ] Code inline OR link to separate file
- [ ] One excellent example (not multi-language)
- [ ] Run scenarios WITH skill - verify agents now comply