Compare commits

...

7 Commits

Author SHA1 Message Date
Jesse Vincent
9887c17b31 fix(writing-skills): hang backfire mechanism on the separated prohibition-vs-recipe comparison (NEW-4); control comparison stated as trend 2026-06-11 11:30:31 -07:00
Jesse Vincent
391c33edf8 fix(writing-skills): scope empirical claims, honest noise reporting, conditionalize micro-test checklist line
Adversarial review findings 1/3/9: the head-to-head result is now scoped
to its context (dispatch-prompt guidance) with an explicit micro-test-your-
own-case instruction; the nuance-clause result is reported as
consistent->noisy rather than 'measurably dilutes'; the checklist line is
scoped to behavior-shaping guidance and the micro method no longer assumes
raw API access.
2026-06-11 11:10:33 -07:00
Jesse Vincent
95795c2e7b feat(writing-skills): form-selection table + micro-test wording method
RED battery (35 opus authoring samples against the current skill) showed
authors default to prohibition+rationalization-table for composition-
shaping problems (T1: 5/5), where that form measurably backfires
(prohibition 4.4 vs 3.6 no-guidance control vs 3.0 recipe restatement
errors), and design only full-subagent verification with no wording
micro-tests, no mandatory no-guidance control, no manual inspection of
automated matches, no variance signal (T7: 5/5).

Adds: Match the Form to the Failure (failure-type -> form table, nuance/
exemption rules), scope note on Bulletproofing, Micro-Test Wording
subsection, two checklist lines. Deliberately narrow: T3/T4/T5/T6 RED
samples showed Iron Law / elicit-first behavior already strong.
2026-06-11 10:20:24 -07:00
Drew Ritter
0cb1960068 chore(evals): bump submodule for Claude Haiku target 2026-06-10 16:31:16 -07:00
Jesse Vincent
f55642e0dd Require contributors to disclose authoring environment and target dev
Add a mandatory self-identification disclosure (model, harness, harness
version, all installed plugins) to the PR template and all three issue
templates, and document the requirement in the contributor guidelines.
We weigh contributions differently depending on what produced them:
content reasoned from documentation is held to a different bar than work
grounded in a real session.

Also state explicitly, in both CLAUDE.md and the PR template, that all
PRs must target the dev branch rather than main.
2026-06-08 22:14:34 -07:00
Drew Ritter
ae1eefb7f9 chore(evals): bump submodule to --scenarios filter (ff3ee83)
Adds `run-all --scenarios` for resuming a scenario subset across the Code
Assist rate-limit windows. Follows the agy rate-limit fix (79f9963).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 22:46:00 -07:00
Drew Ritter
617168aff5 chore(evals): bump submodule to antigravity rate-limit fix (79f9963)
Serialize antigravity against the Gemini Code Assist rate limit
(max_concurrency=1), diagnose 429/RESOURCE_EXHAUSTED honestly instead of as
auth, fail-fast on a latched window, and tolerant preflight OK match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 16:27:35 -07:00
7 changed files with 87 additions and 5 deletions

View File

@@ -12,14 +12,17 @@ add a comment or reaction to the existing one instead.
- [ ] I searched existing issues and this is not a duplicate
## Environment
## Environment (required)
<!-- Required. We assume an agent filed this report — tell us which one and
where it ran. We weigh reports by what produced them. -->
| Field | Value |
|-------|-------|
| Superpowers version | |
| Harness (Claude Code, Cursor, etc.) | |
| Harness version | |
| Model | |
| Your model + version | |
| All plugins installed | |
| OS + shell | |
## Is this a Superpowers issue or a platform issue?

View File

@@ -30,5 +30,18 @@ progress, and some were intentionally declined.
of project? If this is specific to your domain, workflow, or a
third-party tool, it may belong as its own plugin instead. -->
## Environment (required)
<!-- Required. We assume an agent wrote this request — tell us which one and
where it ran. We weigh proposals reasoned from documentation differently
than ones grounded in a real session where the problem actually came up. -->
| Field | Value |
|-------|-------|
| Superpowers version | |
| Harness (Claude Code, Cursor, etc.) | |
| Harness version | |
| Your model + version | |
| All plugins installed | |
## Context
<!-- Optional: version info, harness, model, workflow where you hit this. -->
<!-- Optional: the workflow where you hit this, links, transcripts. -->

View File

@@ -21,3 +21,14 @@ requested or discussed.
## Have you tried manual installation?
<!-- Many tools work with Superpowers through manual setup even without
official support. Did you try? What happened? -->
## Environment (required)
<!-- Required. We assume an agent wrote this request — tell us which one and
where it ran. -->
| Field | Value |
|-------|-------|
| Harness you currently use (Claude Code, Cursor, etc.) | |
| Harness version | |
| Your model + version | |
| All plugins installed | |

View File

@@ -4,6 +4,23 @@ sections blank, contain multiple unrelated changes, or show no evidence
of human involvement will be closed without review.
-->
> **This PR MUST target the `dev` branch, not `main`.** `main` is the
> released branch; active work lands on `dev` first. PRs opened against
> `main` will be asked to retarget `dev` before review.
## Who is submitting this PR? (required)
<!-- Required. PRs that omit this will be closed. We assume an agent wrote
this PR — tell us which one and where it ran. We weigh contributions by
what produced them: content reasoned from documentation is held to a
different bar than work grounded in a real session. -->
| Field | Value |
|-------|-------|
| Your model + version | |
| Harness + version | |
| All plugins installed | |
| Human partner who reviewed this diff | |
## What problem are you trying to solve?
<!-- Describe the specific problem you encountered. If this was a session
issue, include: what you were doing, what went wrong, the model's

View File

@@ -14,7 +14,8 @@ Before you open a PR against this repo, you MUST:
2. **Search for existing PRs** — open AND closed — that address the same problem. If duplicates exist, STOP and tell your human partner. Do not open another duplicate.
3. **Verify this is a real problem.** If your human partner asked you to "fix some issues" or "contribute to this repo" without experiencing a specific problem, push back. Ask them what broke, what failed, what the user experience was.
4. **Confirm the change belongs in core.** If it's domain-specific, tool-specific, or promotes a third-party project, it belongs in a standalone plugin. Tell your human partner.
5. **Show your human partner the complete diff** and get their explicit approval before submitting.
5. **Identify yourself.** Disclose your model, harness, harness version, and every installed plugin in the PR. Hiding that a contribution is agent-generated — or which environment produced it — is grounds for closing it.
6. **Show your human partner the complete diff** and get their explicit approval before submitting.
If any of these checks fail, do not open the PR. Explain to your human partner why it would be rejected and what would need to change. They will thank you for saving them the embarrassment.
@@ -26,6 +27,10 @@ If any of these checks fail, do not open the PR. Explain to your human partner w
**PRs that show no evidence of human involvement will be closed.** A human must review the complete proposed diff before submission.
**Submitters MUST identify themselves.** Every PR and issue must disclose the model, harness, harness version, and all installed plugins used to produce the contribution — or state plainly that it was written by hand with no agent. This is not optional. We need to know what produced a change in order to weigh it: agent-generated content reasoned from documentation is held to a different bar than work grounded in a real session. Contributions that hide their authoring environment will be closed.
**All PRs MUST target the `dev` branch, not `main`.** `main` is the released branch; active work lands on `dev` first. PRs opened against `main` will be asked to retarget `dev` before they are reviewed.
## What We Will Not Accept
### Third-party dependencies

2
evals

Submodule evals updated: e2b37138c8...f8e5a9949f

View File

@@ -456,10 +456,29 @@ Different skill types need different test approaches:
**All of these mean: Test before deploying. No exceptions.**
## Match the Form to the Failure
Before writing guidance, classify the baseline failure. The form that bulletproofs one failure type measurably backfires on another.
| Baseline failure | Right form | Wrong form |
|---|---|---|
| Skips/violates a rule under pressure (knows better, does it anyway) | Prohibition + rationalization table + red flags (see Bulletproofing below) | Soft guidance ("prefer...", "consider...") |
| Complies, but output has the wrong shape (bloated prompt, buried verdict, restated spec) | Positive recipe or contract: state what the output IS — its parts, in order | Prohibition list ("don't restate", "never narrate") |
| Omits a required element from something they already produce | Structural: REQUIRED field or slot in the template they fill in | Prose reminders near the template |
| Behavior should depend on a condition | Conditional keyed to an observable predicate ("if the brief exists, reference it") | Unconditional rule + exemption clauses |
**Why prohibitions backfire on shaping problems:** under a competing incentive ("make the prompt self-contained"), agents negotiate with "don't X". In head-to-head wording tests on dispatch-prompt guidance, the prohibition arm produced clearly more of the unwanted content than the recipe arm (fully separated distributions), and trended worse than even the no-guidance control — micro-test your own case rather than assuming, but never reach for the prohibition by default. A recipe leaves nothing to negotiate: the output matches the stated shape or it doesn't.
**Rules for whichever form you pick:**
- **No nuance clauses.** "Don't X unless it matters" reopens the negotiation — appending a single nuance clause to a winning recipe degraded it from consistent to noisy in the same wording tests. Express a real exception as its own conditional on an observable predicate.
- **Exemption clauses don't scope.** "This limit doesn't apply to code blocks" still suppresses code blocks. If part of the output must be exempt, restructure so the rule can't reach it.
## Bulletproofing Skills Against Rationalization
Skills that enforce discipline (like TDD) need to resist rationalization. Agents are smart and will find loopholes when under pressure.
**Scope:** this toolkit is for discipline failures — an agent that knows the rule and skips it under pressure. For wrong-shaped output or omitted elements, prohibition-based bulletproofing backfires; use the forms in Match the Form to the Failure instead.
**Psychology note:** Understanding WHY persuasion techniques work helps you apply them systematically. See persuasion-principles.md for research foundation (Cialdini, 2021; Meincke et al., 2025) on authority, commitment, scarcity, social proof, and unity principles.
### Close Every Loophole Explicitly
@@ -553,6 +572,18 @@ Run same scenarios WITH skill. Agent should now comply.
Agent found new rationalization? Add explicit counter. Re-test until bulletproof.
### Micro-Test Wording Before Full Scenarios
Full pressure-scenario runs are the final gate, but they are slow and expensive per iteration. Verify the wording itself first with micro-tests:
1. **One fresh-context sample per call** — a raw API call, or a single-shot subagent if you don't have API access. System prompt = the realistic context the guidance will live in (the full skill or prompt template, not the guidance in isolation); user message = a task that tempts the failure.
2. **Always include a no-guidance control.** If the control doesn't exhibit the failure, there is nothing to fix — stop, don't author the guidance.
3. **5+ reps per variant.** Single samples lie.
4. **Manually read every flagged match.** Score programmatically if you like, but template echoes and quoted counter-examples masquerade as hits; automated counts alone overstate both failure and success.
5. **Variance is a metric.** When guidance lands, reps converge on the same shape. Five different interpretations across five reps means the wording isn't binding — tighten the form before adding words.
Micro-tests verify wording; they do not replace pressure scenarios for discipline skills.
**Testing methodology:** See [testing-skills-with-subagents.md](testing-skills-with-subagents.md) for the complete testing methodology:
- How to write pressure scenarios
- Pressure types (time, sunk cost, authority, exhaustion)
@@ -610,6 +641,8 @@ Deploying untested skills = deploying untested code. It's a violation of quality
- [ ] Keywords throughout for search (errors, symptoms, tools)
- [ ] Clear overview with core principle
- [ ] Address specific baseline failures identified in RED
- [ ] Guidance form matches the failure type (see Match the Form to the Failure)
- [ ] For behavior-shaping guidance: wording micro-tested against a no-guidance control (5+ reps, every flagged match read manually) — N/A for pure reference skills
- [ ] Code inline OR link to separate file
- [ ] One excellent example (not multi-language)
- [ ] Run scenarios WITH skill - verify agents now comply