mirror of
https://github.com/obra/superpowers.git
synced 2026-07-18 07:54:01 +08:00
fix(tests): stop the SDD skill test flaking on timing and prose case
tests/claude-code/test-subagent-driven-development.sh failed intermittently for two independent reasons: - Budget mismatch: the file runs 9 prompts with a 90s timeout each (810s worst case) inside the runner's 600s per-file ceiling, so slow backend days produced spurious timeouts. Raise the runner default to 900s and fix the help text, which claimed the default was 300. - Case-sensitive prose matching: the assert helpers grepped free-form model output case-sensitively, but models capitalize the skill's own headings — observed failures include "Do Not Trust the Report" missing pattern "not trust" and a structured answer missing "First:.*spec.*compliance". Match case-insensitively in assert_contains/assert_not_contains/assert_count/assert_order, widen two Test 5 keyword patterns to phrasings observed in real runs, and make assert_order dump the output on failure the way assert_contains already does, so the next flake is diagnosable. Observed 3 failures across 4 runs before the change (timeout, two distinct pattern misses); 3/3 consecutive full runs pass after it.
This commit is contained in:
@@ -25,7 +25,8 @@ fi
|
||||
# Parse command line arguments
|
||||
VERBOSE=false
|
||||
SPECIFIC_TEST=""
|
||||
TIMEOUT=600 # Default 10 minute timeout per test
|
||||
TIMEOUT=900 # Per-test-file budget; must exceed the file's worst case
|
||||
# (test-subagent-driven-development.sh: 9 prompts x 90s each)
|
||||
RUN_INTEGRATION=false
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
@@ -52,7 +53,7 @@ while [[ $# -gt 0 ]]; do
|
||||
echo "Options:"
|
||||
echo " --verbose, -v Show verbose output"
|
||||
echo " --test, -t NAME Run only the specified test"
|
||||
echo " --timeout SECONDS Set timeout per test (default: 300)"
|
||||
echo " --timeout SECONDS Set timeout per test (default: 900)"
|
||||
echo " --integration, -i Run integration tests (slow, 10-30 min)"
|
||||
echo " --help, -h Show this help"
|
||||
echo ""
|
||||
|
||||
Reference in New Issue
Block a user