mirror of
https://github.com/obra/superpowers.git
synced 2026-05-01 14:49:06 +08:00
The SDD integration test silently bailed before printing any verification results. Three independent bugs caused this: 1. `WORKING_DIR_ESCAPED` was computed from `$SCRIPT_DIR/../..` without resolving `..` segments. The resulting "directory" name contained literal `..` so `find` was looking in a path that doesn't exist. 2. With `set -euo pipefail`, the `find ... | sort -r | head -1` pipeline could exit non-zero (SIGPIPE on the producer when head closes early), killing the script silently before assertions ran. 3. The `claude -p` invocation never passed `--plugin-dir`, so it loaded the installed plugin instead of the working tree. Local edits to skills under test were not actually being tested. Other adjustments: - Run claude from inside the unique TEST_PROJECT directory instead of from the plugin root, so its session JSONL lives in its own `~/.claude/projects/` folder and doesn't race other concurrent claude sessions for "most recent file". - Use the same character-normalization claude does (every non-alphanumeric becomes `-`) when computing the session dir name; macOS-resolved `/private/var/...` paths and tmp dirs with `.`/`_` in their names need this to round-trip correctly. - Accept either `"name":"Agent"` or `"name":"Task"` in the subagent count — the harness renamed the tool but the test wasn't updated. Verified on this branch: all six verification tests now pass against a real end-to-end SDD run (skill invoked, 7 subagents dispatched, 6 TodoWrite calls, working code produced, tests pass, no extra features).
Claude Code Skills Tests
Automated tests for superpowers skills using Claude Code CLI.
Overview
This test suite verifies that skills are loaded correctly and Claude follows them as expected. Tests invoke Claude Code in headless mode (claude -p) and verify the behavior.
Requirements
- Claude Code CLI installed and in PATH (
claude --versionshould work) - Local superpowers plugin installed (see main README for installation)
Running Tests
Run all fast tests (recommended):
./run-skill-tests.sh
Run integration tests (slow, 10-30 minutes):
./run-skill-tests.sh --integration
Run specific test:
./run-skill-tests.sh --test test-subagent-driven-development.sh
Run with verbose output:
./run-skill-tests.sh --verbose
Set custom timeout:
./run-skill-tests.sh --timeout 1800 # 30 minutes for integration tests
Test Structure
test-helpers.sh
Common functions for skills testing:
run_claude "prompt" [timeout]- Run Claude with promptassert_contains output pattern name- Verify pattern existsassert_not_contains output pattern name- Verify pattern absentassert_count output pattern count name- Verify exact countassert_order output pattern_a pattern_b name- Verify ordercreate_test_project- Create temp test directorycreate_test_plan project_dir- Create sample plan file
Test Files
Each test file:
- Sources
test-helpers.sh - Runs Claude Code with specific prompts
- Verifies expected behavior using assertions
- Returns 0 on success, non-zero on failure
Example Test
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/test-helpers.sh"
echo "=== Test: My Skill ==="
# Ask Claude about the skill
output=$(run_claude "What does the my-skill skill do?" 30)
# Verify response
assert_contains "$output" "expected behavior" "Skill describes behavior"
echo "=== All tests passed ==="
Current Tests
Fast Tests (run by default)
test-subagent-driven-development.sh
Tests skill content and requirements (~2 minutes):
- Skill loading and accessibility
- Workflow ordering (spec compliance before code quality)
- Self-review requirements documented
- Plan reading efficiency documented
- Spec compliance reviewer skepticism documented
- Review loops documented
- Task context provision documented
Integration Tests (use --integration flag)
test-subagent-driven-development-integration.sh
Full workflow execution test (~10-30 minutes):
- Creates real test project with Node.js setup
- Creates implementation plan with 2 tasks
- Executes plan using subagent-driven-development
- Verifies actual behaviors:
- Plan read once at start (not per task)
- Full task text provided in subagent prompts
- Subagents perform self-review before reporting
- Spec compliance review happens before code quality
- Spec reviewer reads code independently
- Working implementation is produced
- Tests pass
- Proper git commits created
What it tests:
- The workflow actually works end-to-end
- Our improvements are actually applied
- Subagents follow the skill correctly
- Final code is functional and tested
Adding New Tests
- Create new test file:
test-<skill-name>.sh - Source test-helpers.sh
- Write tests using
run_claudeand assertions - Add to test list in
run-skill-tests.sh - Make executable:
chmod +x test-<skill-name>.sh
Timeout Considerations
- Default timeout: 5 minutes per test
- Claude Code may take time to respond
- Adjust with
--timeoutif needed - Tests should be focused to avoid long runs
Debugging Failed Tests
With --verbose, you'll see full Claude output:
./run-skill-tests.sh --verbose --test test-subagent-driven-development.sh
Without verbose, only failures show output.
CI/CD Integration
To run in CI:
# Run with explicit timeout for CI environments
./run-skill-tests.sh --timeout 900
# Exit code 0 = success, non-zero = failure
Notes
- Tests verify skill instructions, not full execution
- Full workflow tests would be very slow
- Focus on verifying key skill requirements
- Tests should be deterministic
- Avoid testing implementation details