From e7b0761b64ec9d6eecf045c1f20914a904d3e22f Mon Sep 17 00:00:00 2001 From: Drew Ritter Date: Thu, 6 Aug 2026 16:24:30 -0700 Subject: [PATCH] docs(specs): tighten final-review correction Record Drew's approval of the final-review correction and reduce the amendment to the minimum distinct contracts.\n\nCombine proof-log and optional SSH capture into one evidence-pipeline scenario because they share the same pipefail ownership root cause. Remove repeated outcome text while retaining the test-first card lifecycle, bootstrap approval boundary, and required behavior evaluations. --- ...-23-agentic-e2e-safety-hardening-design.md | 60 ++++++------------- 1 file changed, 18 insertions(+), 42 deletions(-) diff --git a/docs/superpowers/specs/2026-07-23-agentic-e2e-safety-hardening-design.md b/docs/superpowers/specs/2026-07-23-agentic-e2e-safety-hardening-design.md index 363cee5a..7c863a9d 100644 --- a/docs/superpowers/specs/2026-07-23-agentic-e2e-safety-hardening-design.md +++ b/docs/superpowers/specs/2026-07-23-agentic-e2e-safety-hardening-design.md @@ -1,8 +1,8 @@ # Agentic E2E Safety Hardening — Design Date: 2026-07-23; amended 2026-08-06 -Status: initial design approved with Drew, 2026-07-23; final-review -correction approved in principle, with written amendment pending review +Status: approved with Drew (initial design 2026-07-23; final-review +correction 2026-08-06) Builds on: - `2026-07-04-agentic-end-to-end-testing-design.md` - `2026-07-04-spec-derived-scenario-cards-design.md` @@ -44,10 +44,9 @@ the workflow can produce misleading or incomplete evidence: whole-branch review, but the successful path does not require those durable files to be committed or reviewed before finishing. -Each finding has a direct reproduction. A gate exiting `7` becomes status -`0` through the documented outer pipeline; a four-marker fence containing a -three-marker literal can produce a false checker pass; and the successful E2E -path can reach finishing with its passing cards still untracked. +Direct reproductions show a gate exiting `7` becoming status `0`, a shorter +marker exposing fenced structure, and a successful E2E path reaching +finishing with passing cards still untracked. The review also identified that direct calls to a web application's internal JavaScript action can bypass its user-facing event wiring. That finding is @@ -62,11 +61,8 @@ valid, but Drew explicitly deferred the browser behavior change on than growing it into a general Markdown parser. - Ensure the tmux recipe cleans up only a session the scenario successfully created. -- Preserve failures from the real gate, SSH snapshot command, and `tee` in - evidence-capture recipes. -- Honor the opening fence's marker width when filtering structural Markdown. -- Ensure passing scenario cards are committed and focused-reviewed before - finishing, without committing them before they have been run. +- Close the final-review gaps in evidence-pipeline status, fence-width + tracking, and post-run card commit and review. - Add focused tests and behavior evidence for only the changed contracts. ## Non-goals @@ -75,11 +71,9 @@ valid, but Drew explicitly deferred the browser behavior change on - Supporting tables without leading and trailing outer pipes. - Building or importing a Markdown parser. - Changing the browser-driving recipe. -- Reordering the final whole-branch review or otherwise redesigning - subagent-driven development beyond finalizing its E2E artifacts. -- Committing scenario cards before they have passed their live run. -- Requiring an extra human pause for ordinary card authoring when the - governing spec already contains an approved scenario table. +- Redesigning subagent-driven development beyond finalizing its E2E + artifacts, committing cards before their live run passes, or adding a human + pause when the governing spec already has an approved scenario table. - Making remote-state snapshots mandatory for runs that do not touch remote or shared state. - Changing brainstorming, the card format, or the card-author role. @@ -193,25 +187,12 @@ cleanup all address the same session without repeating a literal name. ### 6. Evidence pipeline failure propagation -The shell that owns an evidence pipeline must also enable `pipefail`. Placing -`bash -o pipefail -c` around only the real gate and piping that child process -to `tee` in the caller is insufficient: the caller observes `tee` as the last -pipeline command and can return `0` after the gate failed. - -The proof-log recipe places the complete producer-and-`tee` pipeline inside -one Bash invocation with `pipefail` enabled. It retains live terminal output -and the saved log, while returning nonzero if either the real gate or `tee` -fails. The log still records the real gate's `EXIT_STATUS`; that marker does -not replace the shell's process status. - -The optional pre/post SSH snapshot recipe follows the same rule. The complete -`ssh`-and-`tee` pipeline runs under one pipefail-owning shell, so an SSH -transport or remote-command failure cannot leave an empty snapshot that the -workflow accepts as successful. A `tee` write failure also fails the capture. -The pre/post comparison runs only after both captures succeed. - -This design does not prescribe a reusable wrapper or add a dependency. It -corrects the executable command shape in the existing reference. +Both the proof-log and optional SSH snapshot recipes place their complete +producer-and-`tee` pipeline inside one `bash -o pipefail -c` invocation. Live +output and the saved log remain available, while a failed real gate, SSH +capture, or `tee` write makes the recipe return nonzero. The proof log still +records the gate's `EXIT_STATUS`, and snapshot comparison starts only after +both captures succeed. No reusable wrapper or dependency is added. ### 7. Proven scenario artifacts @@ -255,8 +236,7 @@ remains a separate delivery gate. | checker-reports-canonical-table-contract | Pipe-less tables remain unsupported with an explicit diagnostic | If a pipe-less table exits with a status other than 2 or its diagnostic does not say outer pipes are required, the scenario FAILS. | | checker-respects-opening-fence-width | Fenced examples remain excluded until a same-family marker run at least as long as the opener | If a shorter same-family run or an other-family run exposes example-only structure, the scenario FAILS. | | tmux-preserves-preexisting-session | A TUI scenario owns and cleans only the tmux session it creates | If a pre-existing tmux sentinel session is stopped or changed, or the scenario-owned session remains after cleanup, the scenario FAILS. | -| proof-log-preserves-command-failure | Live proof logging preserves failures from the real gate and from writing its log | If a failing gate or `tee` returns success from the documented proof-log pipeline, the scenario FAILS. | -| snapshot-log-preserves-ssh-failure | Optional remote-state capture fails when SSH or snapshot logging fails | If a failed SSH capture or `tee` write is accepted as a valid snapshot, the scenario FAILS. | +| evidence-pipelines-preserve-failures | Proof logging and optional SSH snapshots preserve producer and logging failures | If a failing gate, SSH capture, or `tee` returns success from the documented evidence pipeline, the scenario FAILS. | | passing-cards-land-before-finishing | Cards are run before commit, then committed and focused-reviewed before finishing | If finishing begins with a passing card untracked, modified, absent from `HEAD`, or unreviewed, the scenario FAILS. | ## Failure and exit behavior @@ -277,11 +257,7 @@ Specific outcomes: - an invalid card name exits `1` before path construction; - headings or falsification text found only in a card's fenced code do not count and therefore exit `1`; -- a shorter same-family marker run does not close a longer fence; -- a failed real gate, SSH capture, or `tee` write makes its evidence recipe - return nonzero; -- a successful live card run does not complete the E2E phase until the - unchanged passing card is committed and focused-reviewed. +- a shorter same-family marker run does not close a longer fence. Diagnostics identify the affected file and row or section. Tests assert only stable diagnostic contracts needed by a caller, not complete rendered error