tests/claude-code/test-subagent-driven-development.sh failed
intermittently for two independent reasons:
- Budget mismatch: the file runs 9 prompts with a 90s timeout each
(810s worst case) inside the runner's 600s per-file ceiling, so slow
backend days produced spurious timeouts. Raise the runner default to
900s and fix the help text, which claimed the default was 300.
- Case-sensitive prose matching: the assert helpers grepped free-form
model output case-sensitively, but models capitalize the skill's own
headings — observed failures include "Do Not Trust the Report"
missing pattern "not trust" and a structured answer missing
"First:.*spec.*compliance". Match case-insensitively in
assert_contains/assert_not_contains/assert_count/assert_order, widen
two Test 5 keyword patterns to phrasings observed in real runs, and
make assert_order dump the output on failure the way assert_contains
already does, so the next flake is diagnosable.
Observed 3 failures across 4 runs before the change (timeout, two
distinct pattern misses); 3/3 consecutive full runs pass after it.
- Specs (brainstorming output) now go to docs/superpowers/specs/
- Plans (writing-plans output) now go to docs/superpowers/plans/
- User preferences for locations override these defaults
- Update all skill references and test files
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Claude Code headless mode requires --allowed-tools flag to actually
execute tool calls. Without it, Claude only responds as if it's doing
things but doesn't actually use tools.
Changes:
- Updated run_claude helper to accept allowed_tools parameter
- Updated integration test to use --allowed-tools=all
- This enables actual tool execution (Write, Task, Bash, etc.)
Now the integration test should actually execute the workflow instead
of just talking about it.
Created automated test suite for testing superpowers skills using
Claude Code CLI in headless mode.
New files:
- tests/claude-code/run-skill-tests.sh - Main test runner
- tests/claude-code/test-helpers.sh - Helper functions for testing
- tests/claude-code/test-subagent-driven-development.sh - First test
- tests/claude-code/README.md - Documentation
Test framework features:
- Run Claude Code with prompts and capture output
- Assertion helpers (contains, not_contains, count, order)
- Test project creation helpers
- Timeout support (default 5 minutes)
- Verbose mode for debugging
- Specific test selection
First test verifies subagent-driven-development skill:
- Skill loading
- Workflow ordering (spec compliance before code quality)
- Self-review requirements
- Plan reading efficiency (read once)
- Spec compliance reviewer skepticism
- Review loops
- Task context provision
Run with: cd tests/claude-code && ./run-skill-tests.sh
