rogee/superpowers
1
0
Fork 0
mirror of https://github.com/obra/superpowers.git synced 2026-07-08 19:19:04 +08:00
Code Issues Packages Projects Releases Wiki Activity

evals: drop drill source marker

Browse Source
This commit is contained in:
Drew Ritter
2026-05-06 14:55:14 -07:00
parent 74cddb5575
commit f7705f208e
3 changed files with 16 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/superpowers/specs/2026-05-06-lift-drill-into-evals-design.md
View File

@@ -106,7 +106,7 @@ Every change in the implementation plan gets cross-checked by an independent sub
| Change category | Subagent verification |
|----------------|----------------------|
| Each bash-test deletion | Dispatch a subagent with: (a) the bash test file content, (b) the candidate drill scenario YAML, (c) the prompt: *"List every assertion the bash test makes. List every verify entry in the drill scenario. For each bash assertion, find a matching drill check or report it as unmatched. Output a per-assertion table."* The subagent's output is the gate — only delete if every bash assertion has a match. |
| Initial `evals/` copy | Subagent verifies: (a) drill SHA being copied is recorded in commit message and `evals/.drill-source-sha` (a checked-in file) so divergence is detectable; (b) **per-file SHA-256 checksum** matches drill repo for every file (not just file count); (c) excluded paths (`.git/`, `.venv/`, `results/`, `.env`, `__pycache__/`, `*.egg-info/`, any `.private-journal/`) are absent from `evals/`; (d) all backend YAMLs reference paths that exist post-move; (e) `pyproject.toml`, `uv.lock`, `.gitignore` are intact. |
| Initial `evals/` copy | Subagent verifies: (a) drill SHA being copied is recorded in the lift commit message so provenance is auditable; (b) **per-file SHA-256 checksum** matches drill repo for every file (not just file count); (c) excluded paths (`.git/`, `.venv/`, `results/`, `.env`, `__pycache__/`, `*.egg-info/`, any `.private-journal/`) are absent from `evals/`; (d) all backend YAMLs reference paths that exist post-move; (e) `pyproject.toml`, `uv.lock`, `.gitignore` are intact. |
| Drill's own pytest suite | Subagent runs `cd evals && uv run pytest` after the path-default change. Drill ships its own pytest suite at `evals/tests/` including `test_backend.py` which exercises `SUPERPOWERS_ROOT` env-var behavior — these tests must update to match the helper and continue to pass. |
| Reference scrubbing after deletion | Subagent greps the entire superpowers tree (excluding `node_modules/`, `.venv/`, and `evals/`) for references to deleted bash test paths. Search targets: `docs/`, `docs/superpowers/plans/`, `RELEASE-NOTES.md`, `CLAUDE.md`, `GEMINI.md`, `AGENTS.md`, `README.md`, `.github/`, `scripts/`, `.opencode/INSTALL.md`, `.codex-plugin/INSTALL.md`, `lefthook.yml`. Any hit is either updated or surfaces a missed dependency. |
| Path defaults change (`SUPERPOWERS_ROOT` default) | Subagent runs at least one cheap drill scenario after the path changes (e.g., `triggering-test-driven-development`) and confirms it still passes. Real validation, not just code review. |
@@ -149,7 +149,7 @@ Each step is a separate commit (or small group of commits). Step 2 is the bigges
1. Branch off `dev` (f/evals-lift)
2. Copy drill repo into evals/ (single commit, easy to revert)
├─ Record drill SHA at copy time → commit message + evals/.drill-source-sha
├─ Record drill SHA at copy time → commit message
├─ Use `rsync -a --exclude=.git --exclude=.venv --exclude=results
│ --exclude=.env --exclude=__pycache__ --exclude='*.egg-info'
│ --exclude=.private-journal /path/to/drill/ evals/`
@@ -220,7 +220,7 @@ The implementation plan must show:
- All non-excluded drill source files present at `evals/` after step 2 (subagent **per-file SHA-256 checksum diff** vs `obra/drill@<recorded-sha>`).
- Excluded paths (`.git/`, `.venv/`, `results/`, `.env`, `__pycache__/`, `*.egg-info/`, `.private-journal/`) absent from `evals/`.
- `evals/.drill-source-sha` matches the SHA referenced in the step-2 commit message.
- The step-2 commit message records the drill source SHA.
- `cd evals && uv sync` succeeds without `SUPERPOWERS_ROOT` set.
- `cd evals && uv run pytest` passes (drill's own pytest suite).
- `cd evals && uv run drill list` returns the same scenario count as the standalone drill repo at the recorded SHA.