From 7965786d1bb448be3a22e965eda3559d589d8f4b Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Sat, 4 Jul 2026 16:12:56 -0700 Subject: [PATCH] feat(skills): add proof-movie and demo-movie evidence recipes --- .../recording-a-proof-movie.md | 137 ++++++++++++++++++ .../rendering-a-demo-movie.md | 133 +++++++++++++++++ 2 files changed, 270 insertions(+) create mode 100644 skills/agentic-end-to-end-testing/recording-a-proof-movie.md create mode 100644 skills/agentic-end-to-end-testing/rendering-a-demo-movie.md diff --git a/skills/agentic-end-to-end-testing/recording-a-proof-movie.md b/skills/agentic-end-to-end-testing/recording-a-proof-movie.md new file mode 100644 index 00000000..2441f51b --- /dev/null +++ b/skills/agentic-end-to-end-testing/recording-a-proof-movie.md @@ -0,0 +1,137 @@ +# Recording a Proof Movie (ffmpeg + avfoundation) + +Produce a watchable `.mp4`/`.mov` that proves an e2e run happened, that a +reviewer can audit and re-derive, and whose hashes match the raw artifacts it +renders. This is the fallback-that-is-actually-better when OS screen capture +is permission-blocked (macOS returns wallpaper-only frames): render the movie +from the real run's log instead of fighting the OS for pixels. + +## Try the real capture first — refuse to fake it + +```bash +# probe capture devices +/opt/homebrew/bin/ffmpeg -f avfoundation -list_devices true -i "" + +# short validation grab, then extract frame 1 and LOOK at it +/opt/homebrew/bin/ffmpeg -y -hide_banner -f avfoundation -framerate 15 -capture_cursor 1 \ + -t 2 -i ':none' -vf scale=1280:-2 -pix_fmt yuv420p /tmp/cap-validate.mp4 +/opt/homebrew/bin/ffmpeg -y -hide_banner -i /tmp/cap-validate.mp4 -frames:v 1 /tmp/cap-validate.png +``` + +If the frame is just wallpaper (app window missing), Screen Recording is +blocked for this process. **Do not ship it.** Say so explicitly and switch to +the rendered evidence reel below. `screencapture -x out.png` has the same +limitation; `screencapture -x -l out.png` can grab a single window +if you can resolve its CoreGraphics window id. + +## Run the real gate as the evidence source + +Wrap the actual e2e test/command so the log carries machine-checkable +markers. Use `bash`, not `zsh` — zsh's read-only `$status` injects a spurious +error *after* a passing run and pollutes the movie. + +```bash +bash -o pipefail -c ' + printf "MANUAL_E2E_KIND=\n"; + printf "STARTED_AT="; date -u +%Y-%m-%dT%H:%M:%SZ; + ; # e.g. xcodebuild test-without-building ... -resultBundlePath ... + rc=$?; + printf "FINISHED_AT="; date -u +%Y-%m-%dT%H:%M:%SZ; + printf "EXIT_STATUS=%s\n" "$rc"; exit "$rc" +' 2>&1 | tee /run.log +``` + +## Snapshot external state before and after + +If the run touches a remote host or a shared tmux, snapshot it identically +pre- and post-run and diff. Equal snapshots prove the run left no residue. + +```bash +ssh 'date -Is; tmux list-sessions -F "#{session_name}|#{session_windows}|attached=#{session_attached}"; \ + ps -eo pid=,args= | awk "// {print}"; find /tmp -maxdepth 1 -name "" | wc -l' \ + | tee /pre-snapshot.txt +# ... run gate ... then repeat with SNAPSHOT_KIND=post => post-snapshot.txt ; assert they match +``` + +## Render the reel from the log + +Draw 1920x1080 RGB frames from the log and snapshots (title / exact command +shape / result / before-after diff / evidence bundle) and stream +`img.tobytes()` into a single ffmpeg pipe. Keep it in a saved +`generate_*_movie.py` so it is re-runnable and auditable — don't leave it as a +one-shot heredoc for anything you'll repeat. + +```python +from PIL import Image, ImageDraw, ImageFont +import subprocess + +W, H, FPS = 1920, 1080, 15 +SANS = ImageFont.truetype('/System/Library/Fonts/Helvetica.ttc', 42) # macOS system fonts +MONO = ImageFont.truetype('/System/Library/Fonts/Menlo.ttc', 24) + +cmd = [ + '/opt/homebrew/bin/ffmpeg', '-y', '-hide_banner', + '-f', 'rawvideo', '-pix_fmt', 'rgb24', '-s', f'{W}x{H}', '-r', str(FPS), '-i', '-', + '-an', '-c:v', 'libx264', '-preset', 'medium', '-crf', '20', '-pix_fmt', 'yuv420p', + '-movflags', '+faststart', 'out.mov', +] +proc = subprocess.Popen(cmd, stdin=subprocess.PIPE) +for frame_count, render in scenes: # scenes = [(nframes, render_fn), ...] + denom = max(1, frame_count - 1) + for i in range(frame_count): + proc.stdin.write(render(i / denom).tobytes()) # render() -> PIL RGB Image, W x H +proc.stdin.close() +if proc.wait() != 0: + raise SystemExit('ffmpeg failed') +``` + +## Verify the encoding with ffprobe + +```bash +/opt/homebrew/bin/ffprobe -v error \ + -show_entries format=duration,size \ + -show_entries stream=codec_name,width,height,nb_frames \ + -of default=noprint_wrappers=1 out.mov +# expect e.g. codec_name=h264, width=1920, height=1080, real duration/nb_frames +``` + +## Extract frames, build a contact sheet, and look at it + +```bash +mkdir -p frame-checks +for t in 00:00:03 00:00:24 00:00:45 00:01:04; do + /opt/homebrew/bin/ffmpeg -y -hide_banner -ss "$t" -i out.mov \ + -frames:v 1 -update 1 "frame-checks/${t//:/-}.png" +done +# PIL: paste the extracted frames (resized) into a 2xN contact-sheet.png, labeled by timestamp +``` + +Then actually view `contact-sheet.png` (and any suspect full-size frame) to +confirm the text is legible. If a panel overflows or a frame is unreadable, +fix the generator and regenerate — do not ship an unreadable reel. + +## Hash the bundle + +```bash +shasum -a 256 out.mov frame-checks/contact-sheet.png run.log > SHA256SUMS +shasum -a 256 -c SHA256SUMS +``` + +If you later fix anything the movie renders (a wrong timestamp, a stale test +selector, a log line), **regenerate the movie and re-hash**. A hash that no +longer matches the log is a lie. + +## Non-negotiables + +- Never present a wallpaper-only or blank capture as evidence. Disclose the + OS limitation and render an auditable reel instead — say so plainly; that + pivot is the honest outcome, not a fallback to apologize for. +- The raw log and pre/post snapshots live *next to* the movie. The movie is + derived from them, not a substitute for them. +- `ffprobe` confirms the container is real; the contact sheet plus a human + view of it confirms it's legible. Neither alone is sufficient. +- `SHA256SUMS` covers the movie, the contact sheet, and the log — regenerate + it whenever any source artifact changes. +- Keep the working tree clean: isolate scratch paths, snapshot/clean external + state, and don't commit evidence artifacts unless the repo already tracks + that kind of evidence. diff --git a/skills/agentic-end-to-end-testing/rendering-a-demo-movie.md b/skills/agentic-end-to-end-testing/rendering-a-demo-movie.md new file mode 100644 index 00000000..b21fe545 --- /dev/null +++ b/skills/agentic-end-to-end-testing/rendering-a-demo-movie.md @@ -0,0 +1,133 @@ +# Rendering a Demo Movie (browser-composited) + +Turn a real, running app into a short titled/captioned demo `.mp4` whose +frames are genuine screenshots of the product — not mockups — and verify the +output is actually correct before handing it over. Needs a running instance +of the app, a browser-automation tool that can navigate, run JS (`eval`), set +a viewport, and screenshot to a path, plus `ffmpeg`/`ffprobe`, and a scratch +dir such as `/tmp/app-movie/`. + +## Step 1 — capture real scene frames from the live app + +Set a fixed viewport, then per scene: navigate/interact via JS to compose the +shot, screenshot to `frame-NN.png`, and **read the PNG back to confirm** the +shot is what you intended. No fixed fps — one deliberate screenshot per scene +beat. + +``` +use_browser: {"action":"navigate","payload":"http://localhost:/"} +use_browser: {"action":"screenshot","payload":{"path":"/tmp/app-movie/frame-01.png"}} +# ...navigate/eval to set up each subsequent scene, screenshot frame-02..frame-NN +``` + +## Step 2 — composite title/caption/end cards in the browser + +Prefer this over ffmpeg `drawtext`, which is fragile: on macOS-under-sandbox, +`textfile=` reliably fails with `Either text, a valid file, a timecode or +text source must be provided` (even with absolute paths), while a trivial +inline `text=Foo` may work. Don't fight it. Render cards as HTML and +screenshot them — you also get real fonts, `` accents, and CSS layout for +free. + +`card.html` (param-driven: title / end / image+caption-bar): + +```html + + + + +``` + +Drive it (name cards so a lexical glob orders them title → scenes → end: +`card-00` … `card-07` … `card-99`): + +``` +use_browser: {"action":"set_viewport","payload":{"width":1400,"height":960}} +use_browser: {"action":"navigate","payload":"file:///tmp/app-movie/card.html?mode=title"} +use_browser: {"action":"screenshot","payload":{"path":"/tmp/app-movie/card-00.png"}} +# per scene: define a helper once, then swap innerHTML and screenshot: +use_browser: {"action":"eval","payload":"window.__setCard=(img,cap)=>{document.body.innerHTML='
'+cap+'
';return img;}; __setCard('frame-01.png','The scene resolves — it lands in New state')"} +use_browser: {"action":"screenshot","payload":{"path":"/tmp/app-movie/card-01.png"}} +# ...repeat __setCard + screenshot for frame-02..frame-07 -> card-02..card-07 +use_browser: {"action":"navigate","payload":"file:///tmp/app-movie/card.html?mode=end"} +use_browser: {"action":"screenshot","payload":{"path":"/tmp/app-movie/card-99.png"}} +``` + +## Step 3 — concatenate the cards + +Pure image concat, no drawtext. `-framerate 1/3` holds each card 3 seconds; +the `card-*` glob orders them. + +```bash +cd /tmp/app-movie && \ +ffmpeg -y -loglevel error -framerate 1/3 -pattern_type glob -i 'card-*.png' \ + -vf "scale=1400:960" -r 30 -pix_fmt yuv420p ~/Desktop/app-demo.mp4 && \ +ffprobe -v error -show_entries format=duration -of csv=p=0 ~/Desktop/app-demo.mp4 +# 9 cards -> 27.000000 +``` + +## Step 4 — verify the artifact (do not skip) + +Extract a mid-movie frame and actually look at it; duration/size are +necessary but not sufficient. This is the step that catches a scene +screenshotted mid-scroll (half-blank) before it ships. + +```bash +ffmpeg -y -loglevel error -ss 13 -i ~/Desktop/app-demo.mp4 -frames:v 1 /tmp/app-movie/check.png +# then Read check.png; if a scene is wrong, re-capture just that frame-NN, +# recompose its card-NN.png, and re-run Step 3. +``` + +## If you must use ffmpeg drawtext (failed under sandbox — kept for reference) + +This is the approach that **FAILED** under macOS sandbox (`textfile=` +unreadable). Inline `text=` may still work for short labels; per-scene +captions letterbox the shot and draw text into the padding: + +```bash +FONT=/System/Library/Fonts/Helvetica.ttc +# title card (lavfi solid color + two inline drawtext) +ffmpeg -y -loglevel error -f lavfi -i "color=c=0xfaf8f4:s=1400x960:d=3" \ + -vf "drawtext=fontfile=$FONT:text='App Name':fontsize=110:fontcolor=0xb3422f:x=(w-text_w)/2:y=360,drawtext=fontfile=$FONT:text='one-line tagline':fontsize=42:fontcolor=0x44403a:x=(w-text_w)/2:y=510" \ + -r 30 -pix_fmt yuv420p seg-00.mp4 +# a captioned scene: scale to 1400x900, pad 60px dark bar, caption in the bar +ffmpeg -y -loglevel error -loop 1 -i frame-01.png -t 3 \ + -vf "scale=1400:900,pad=1400:960:0:0:color=0x2a2722,drawtext=fontfile=$FONT:text='caption text':fontsize=30:fontcolor=0xfaf8f4:x=(w-text_w)/2:y=918" \ + -r 30 -pix_fmt yuv420p seg-01.mp4 +# concat demuxer +for f in seg-*.mp4; do echo "file '$f'"; done > list.txt +ffmpeg -y -loglevel error -f concat -safe 0 -i list.txt -c copy ~/Desktop/app-demo.mp4 +``` + +## Why the browser-composited path wins + +- Real product screenshots as scenes are unfakeable — an honest "show it + off." +- No dependency on ffmpeg font rendering, the flaky part; cards get real + fonts, rich markup (`` accents), and CSS layout. +- Deterministic ordering via zero-padded `card-NN.png` filenames plus glob. +- The extract-a-frame-and-read-it check in Step 4 is the honesty gate: it is + how a bad frame gets caught instead of shipped.