Compare commits

...

4 Commits

Author SHA1 Message Date
Jesse Vincent
08447d0d10 Bump evals submodule: escalation-aware planted-defect scenario (7dc0be7) 2026-06-11 15:39:04 -07:00
Jesse Vincent
c87f336cc0 E37: pre-flight plan review — surface plan conflicts as one batched question before Task 1 2026-06-11 15:31:01 -07:00
Jesse Vincent
a2a4190809 Spec: L2b tested — opus structural win, sonnet transmission+attention gap (E35/E36); bump evals to 9919b27 2026-06-11 14:53:31 -07:00
Jesse Vincent
8d354bb36b L2b: plan-mandated defects are findings the human adjudicates
Reviewer tripwire (Calibration): a plan-mandated defect IS a finding,
reported as Important and labeled plan-mandated — the plan's authorship
does not grade its own work.

Controller rule (review loop): a plan-mandated finding, or any finding
conflicting with the plan's text, escalates to the human like any plan
contradiction — never dismissed because the plan mandates it.

E35 micro (frozen 0a98 replay, sonnet reviewer, 6v6): without the
tripwire 0/6 reports give the controller anything to escalate on (all
Approved, defect endorsed as spec-required); with it 6/6 report the
defect as a labeled finding.
2026-06-11 13:41:21 -07:00
4 changed files with 48 additions and 3 deletions

View File

@@ -133,8 +133,29 @@ opus controller flagged it 5/5. Cheap controllers handle explicit
escalation; they absorb implicit authority-vs-quality adjudication.
A possible L2b (discrete rule: "a reviewer finding that conflicts with
the plan's text is the human's decision — escalate it") would route the
failing judgment through the escalation behavior that held; untested.
Original recon notes follow.
failing judgment through the escalation behavior that held.
**L2b tested 2026-06-11 (E35/E36, evals
`docs/experiments/2026-06-11-build-loop-autoresearch.md`): improves the
opus stack, does NOT rescue the sonnet rung.** Two rules: a reviewer
tripwire (a plan-mandated defect IS a finding — Important, labeled
plan-mandated; the human decides) and a controller escalation rule
(plan-mandated findings go to the human like any plan contradiction).
Micro on frozen sonnet-composed inputs: 0/6 → 6/6 labeled findings.
Full battery: opus controllers 2/2 internalized the rule, caught their
reviewer's miss as self-described backstop, and escalated for a
sanctioned fix (the 4241 ad-hoc behavior made structural); escalation
sanity 2/2 unbroken. Sonnet controllers: 1/5 full pass — paraphrase
drops the tripwire from dispatches (2/5 transmitted), transmission
alone doesn't fire it live (read-once dilution across the reviewer's
tool reads; placement within the dispatch refuted as the variable),
and no sonnet controller showed backstop behavior; 1/5 shipped the
defect. The L2b rules are a candidate commit for the opus stack.
A future L2c for the sonnet rung would pair the SKILL.md
constraints-recipe (the one channel sonnet transmits verbatim) with a
mandatory output-format slot for plan-mandated findings (the skeleton
survives every observed paraphrase and is consulted at composition
time); untested. Original recon notes follow.
**Recon (superseded):**
Sonnet-controller runs (claude-sonnet coding-agent): all gates green at

2
evals

Submodule evals updated: af05326467...7dc0be79bb

View File

@@ -82,6 +82,20 @@ digraph process {
}
```
## Pre-Flight Plan Review
Before dispatching Task 1, scan the plan once for conflicts:
- tasks that contradict each other or the plan's Global Constraints
- anything the plan explicitly mandates that the review rubric treats as a
defect (a test that asserts nothing, verbatim duplication of a logic block)
Present everything you find to your human partner as one batched question —
each finding beside the plan text that mandates it, asking which governs —
before execution begins, not one interrupt per discovery mid-plan. If the
scan is clean, proceed without comment. The review loop remains the net for
conflicts that only emerge from implementation.
## Model Selection
Use the least powerful model that can handle each role to conserve cost and increase speed.
@@ -181,6 +195,11 @@ final whole-branch review. When you fill a reviewer template:
findings in the progress ledger as you go, and point the final
whole-branch review at that list so it can triage which must be fixed
before merge. A roll-up nobody reads is a silent discard.
- A finding labeled plan-mandated — or any finding that conflicts with
what the plan's text requires — is the human's decision, like any plan
contradiction: present the finding and the plan text, ask which governs.
Do not dismiss the finding because the plan mandates it, and do not
dispatch a fix that contradicts the plan without asking.
- The final whole-branch review gets a package too: run
`scripts/review-package MERGE_BASE HEAD` (MERGE_BASE = the commit the
branch started from, e.g. `git merge-base main HEAD`) and include the

View File

@@ -128,6 +128,11 @@ Subagent (general-purpose):
would block a merge over — verbatim duplication of a logic block,
swallowed errors, tests that assert nothing. "Coverage could be broader"
and polish suggestions are Minor.
If the plan or brief explicitly mandates something this rubric calls a
defect (a test that asserts nothing, verbatim duplication of a logic
block), that IS a finding — report it as Important, labeled
plan-mandated. The plan's authorship does not grade its own work; the
human decides.
Acknowledge what was done well before listing issues — accurate praise
helps the implementer trust the rest of the feedback.