rogee/superpowers
1
0
Fork 0
mirror of https://github.com/obra/superpowers.git synced 2026-07-07 18:49:05 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(skills): add agentic-end-to-end-testing decision core

Browse Source
This commit is contained in:
Jesse Vincent
2026-07-04 15:35:33 -07:00
parent 77f709aab3
commit c1a97b6b34
2 changed files with 108 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

107
skills/agentic-end-to-end-testing/SKILL.md Normal file
View File

@@ -0,0 +1,107 @@
---
name: agentic-end-to-end-testing
description: Use when verifying a running application end-to-end through its real interface (web UI, CLI/TUI, or desktop app), when asked to prove a feature works with evidence — "test it end to end", "prove it actually works", "make me a movie showing it off" — or after a change touches a user-facing surface that unit tests can't cover. Not for unit tests, code review, or API-only checks.
---
# Agentic End-to-End Testing
## Overview
Write a durable, falsifiable scenario; have an agent drive the live application through its real interface the way a user would; end with evidence that cannot be faked. The unit of work is a **scenario card** — a short markdown test written for an agent to execute, high-level enough that a small UI shuffle doesn't invalidate it, precise enough that two agents running it reach the same verdict. The run's product is a per-assertion pass/fail report backed by that evidence.
Two disciplines govern everything here. **Unfakeable evidence:** choose evidence a model cannot fabricate — a movie whose frames you extract and look at, an HTTP 401 that proves the server actually answered, a live third-party round-trip, a hash-sealed bundle. **Honest failure:** when the interface or evidence path breaks, report it, escalate, or pivot. NEVER weaken, skip, or reinterpret an assertion to make it pass.
## When to Use
- A feature touches a user-facing surface (button, palette command, status indicator, keybinding, rendered message) and you want proof it works live.
- The user asks to "test it end to end", "prove it actually works", or wants a demo they can watch.
- You changed a layer (projection, capability gate, renderer) whose effect is only observable in the assembled application.
A green unit test proves the wiring in isolation. A scenario proves the wiring *as assembled and rendered*. They catch different bugs — write the card even when the unit tests pass.
Don't use this for logic with no user-facing surface (unit-test that), or when a production gate makes the live path unreachable (see the over-specification trap below).
## The Scenario Card
One card = one `.md` file in `test/scenarios/`. Keep these sections; collapse any to one line when the scenario is simple. Don't pad.
```markdown
# <area>-<behavior>: one-line title
**What this covers**: the feature + the specific commits/IDs it exercises.
If something else breaks this, it should be caught here.
## Pre-state
What must be true before starting: a freshly built instance running, auth/creds
in place, a clean workdir. Give the exact commands to reach it.
## Steps
Numbered actions described by **intent**, each with the concrete command or
tool call and a real UI label (prefer labels the user sees over brittle
selectors like `#nav > li:nth-child(3)`).
## Expected
For each step, what you should observe — and the **falsification condition**:
"if you see X instead, the test fails." Silence is not success.
## Cleanup
Idempotent teardown so reruns are hermetic. Never touch state you didn't create.
## Sharp edges
Footguns, timing/ordering caveats, nondeterminism noted while recording.
```
## The Run Loop
1. **Preflight.** Build fresh from the code under test — the most common mistake is testing a stale binary. Rebuild every layer your change touches and confirm the running instance is the new one, not a process someone left up yesterday. Isolate hermetically: give the test instance its own HOME, port, and state directory so it can neither collide with nor pollute a real instance. Check credentials and models are in place. Run a minimal smoke check first — one where even a `401` is informative, because it means the server answered.
2. **Write or select the card.** New behavior gets a new card; a regression check reuses an existing one.
3. **Dispatch a disposable runner subagent** using [runner-prompt.md](runner-prompt.md). This is the default: a fresh context has no sunk-cost incentive to fudge the verdict. Running a card yourself in-session is the exception, reserved for a quick single-card check.
4. **Capture evidence** (see Pick Your Evidence below).
5. **Verify the evidence itself.** Extract a frame from the movie and read it. Re-read the capture file. Cross-check every rendered claim against on-disk ground truth — the UI can lie or lag; the log, database, or file is authoritative. Evidence you didn't inspect is evidence you don't have.
6. **Clean up, idempotently.** Shut down what you spawned, remove scratch dirs, leave pre-existing instances running and untouched. Never touch state you didn't create.
7. **Report per-assertion pass/fail with the concrete observation** — the rendered text, the on-disk value, the exit code. A vague "looks fine" is a failed report.
## Pick Your Interface
| Surface | Recipe |
| --- | --- |
| Web UI (browser) | [driving-web-browser.md](driving-web-browser.md) |
| CLI / TUI (terminal) | [driving-cli-tui.md](driving-cli-tui.md) |
| Desktop app | [driving-computer-use.md](driving-computer-use.md) |
## Pick Your Evidence
Ask one question: **what would be impossible to fabricate here?** Then capture that.
| Evidence | When to choose it |
| --- | --- |
| Captured real output / screenshot bundle | The cheap default: a terminal transcript or screenshots of the actual run, saved to files. |
| HTTP status / live third-party round-trip | When the claim is "the other end answered" — a real status code or a real external service response proves it. |
| Recorded movie | When the user wants to *watch* it work. See [recording-a-proof-movie.md](recording-a-proof-movie.md). |
| Rendered captioned demo | When the deliverable is a narrated showcase built from verified stills. See [rendering-a-demo-movie.md](rendering-a-demo-movie.md). |
| Hash-sealed bundle | When the artifact must not drift from the log it documents — seal both together. |
## Hard-Won Principles
- **Falsification, always.** Every assertion states what failure looks like. A step that can't fail proves nothing — make sure your check would fire on the failure path, not just the happy path.
- **Verify the right surface.** The same concept often exists at several layers: an internal capability vs. its REST projection, a model field vs. the rendered chip. Confirm your assertion reads the surface that carries the signal — a "missing" value is often present one layer over.
- **Present but not visible ≠ absent.** Scrollable bodies, virtualized lists, and auto-scroll-to-bottom routinely push a real element out of the capture window. Scroll or expand to where it should be before concluding it didn't render; confirm via a sibling read of the same state.
- **Executing the card tests the card.** Expect to find bugs in your own scenario — a wrong selector, a wrong layer, a vacuous assertion. Fix the card as you go; a card that passes because its check was vacuous is worse than none.
- **The over-specification trap.** A card can describe a path that production gating prevents (a keybind that's a no-op in the current mode). Confirm the gate in the source rather than fighting it through the UI; verify the underlying behavior with a unit test and note the gate in the card.
- **Cleanup is part of the test.** A half-shutdown fleet makes the next run's polling return false positives. Make teardown idempotent and scoped to what you created.
## Common Rationalizations
| Excuse | Reality |
| --- | --- |
| "The unit tests pass, so it works" | Unit tests prove the wiring in isolation; the bug class this skill exists for lives in the assembly. |
| "I read the code; the feature is clearly correct" | Reading is not running. Drive the real interface or report that you didn't. |
| "Screen recording is blocked, I'll ship what I have" | A blank or fabricated artifact is worse than none; pivot to evidence from the real run and say what you did. |
| "The assertion is too strict, I'll adjust it" | NEVER weaken, skip, or reinterpret an assertion to make it pass. |
| "I proved the backend, so the feature works" | Different claim. Say exactly what you exercised, then drive the real interface — or state that you didn't. |
| "My check passed" | A check that would also pass with the feature broken proves nothing — a broken detector and a clean run are indistinguishable. |
## Integration
- Runs after superpowers:subagent-driven-development completes a feature, and before superpowers:finishing-a-development-branch decides how the work lands.
- Complements superpowers:verification-before-completion: that skill gates any success claim on having run the checks; this one defines what counts as proof when the behavior under test is user-facing.