Compare commits

...

5 Commits

Author SHA1 Message Date
Jesse Vincent
cda4b15fd6 fix(tests): stop the SDD skill test flaking on timing and prose case
tests/claude-code/test-subagent-driven-development.sh failed
intermittently for two independent reasons:

- Budget mismatch: the file runs 9 prompts with a 90s timeout each
  (810s worst case) inside the runner's 600s per-file ceiling, so slow
  backend days produced spurious timeouts. Raise the runner default to
  900s and fix the help text, which claimed the default was 300.
- Case-sensitive prose matching: the assert helpers grepped free-form
  model output case-sensitively, but models capitalize the skill's own
  headings — observed failures include "Do Not Trust the Report"
  missing pattern "not trust" and a structured answer missing
  "First:.*spec.*compliance". Match case-insensitively in
  assert_contains/assert_not_contains/assert_count/assert_order, widen
  two Test 5 keyword patterns to phrasings observed in real runs, and
  make assert_order dump the output on failure the way assert_contains
  already does, so the next flake is diagnosable.

Observed 3 failures across 4 runs before the change (timeout, two
distinct pattern misses); 3/3 consecutive full runs pass after it.
2026-07-16 04:00:15 +00:00
Jesse Vincent
93b250b155 fix(codex): make package script and its test portable beyond macOS/bsdtar
The packaging pipeline only worked on a Mac with default umask, for
three stacked reasons:

- The deterministic-metadata tar flags (--uid/--gid/--uname/--gname)
  are bsdtar spellings; GNU tar rejects them, so the tar.gz archive
  step died on Linux. Detect the tar flavor and use --owner=:0
  --group=:0 --numeric-owner on GNU tar, which writes byte-identical
  ustar headers (uid/gid 0, empty uname/gname).
- Staged file modes depended on two umasks canceling out: git archive
  masks entry modes with tar.umask (git default 0002 -> 775), and the
  unflagged tar extraction re-masked with the process umask (022 on
  macOS -> 755, but 002 elsewhere -> 775). Pin tar.umask=0022 on the
  archive call and extract with -p so staged modes are canonical
  755/644 on every machine.
- The test's timestamp assertion parsed bsdtar's -tv column layout and
  expected epoch 0 rendered in a US timezone ("Dec 31 1969"); GNU tar
  uses different columns and UTC hosts render "1970-01-01". Assert
  mtime == 0 via python3 tarfile instead, matching how the test
  already checks zip timestamps.

tests/codex/test-package-codex-plugin.sh now passes on Linux/GNU tar;
the bsdtar branch preserves the exact flags that passed on macOS.
2026-07-16 03:32:25 +00:00
Jesse Vincent
fb7b07088e docs: fix dead references to pruned claude-code-tools.md/copilot-tools.md
e7ddc25 deleted claude-code-tools.md and copilot-tools.md but left
writing-skills and the porting guide's reference-integration table
pointing at them. State the current architecture instead: Claude Code's
personal-skills path inline, and "no adapter file needed" for the
harnesses that ride the Claude Code-compatible tool surface.

Reported by @rasibintang (#1969, with a fix proposed in #1970).

Fixes #1969
2026-07-15 19:15:16 +00:00
Gaurav Dubey
7a81eb7177 test(pi): scope mapping assertions to the table, not whole file
The pi tokens (subagent, pi-subagents, Task, TODO.md) also appear in the
surrounding prose, so matching the whole file passed even with the mapping
table deleted — the exact regression this test exists to catch. Filter to
table rows (lines starting with '|') so the assertion fails when the table
is gone and passes on dev.

Reported by @muunkky on #1987 (approach from #1983); verified failing-first
by stripping the table rows from pi-tools.md.
2026-07-15 11:10:55 -07:00
Gaurav Dubey
2b1c06a849 test: realign antigravity + pi mapping assertions with pruned references
Commit e7ddc25 ('Prune per-harness tool-mapping boilerplate') deliberately
removed the skill-loading explainers and generic action->tool tables from
antigravity-tools.md and pi-tools.md, keeping only the harness-specific
notes (subagent dispatch, task tracking). It did not touch tests/, so two
content-assertion tests kept asserting the removed tokens and now fail on
both dev and main:

  - tests/antigravity/test-antigravity-tools.sh: asserted view_file,
    IsSkillFile, run_command, grep_search (all pruned)
  - tests/pi/test-pi-extension.mjs: asserted read/write/edit/bash (pruned)

Update both to assert only the surviving harness-specific mappings. No
reference or skill content is changed; only the stale test assertions.
2026-07-15 11:10:55 -07:00
9 changed files with 57 additions and 34 deletions

View File

@@ -784,10 +784,10 @@ Use this as the live index; when in doubt, read the files, not this table.
| Harness | Entry point | Bootstrap mechanism | Tool mapping | Tests | Distribution | | Harness | Entry point | Bootstrap mechanism | Tool mapping | Tests | Distribution |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| Claude Code | `.claude-plugin/plugin.json` + `hooks/hooks.json` | shell hook → `hooks/session-start` (`hookSpecificOutput.additionalContext`) | native `Skill` tool; `references/claude-code-tools.md` | `tests/hooks/` | marketplace | | Claude Code | `.claude-plugin/plugin.json` + `hooks/hooks.json` | shell hook → `hooks/session-start` (`hookSpecificOutput.additionalContext`) | native `Skill` tool; no adapter file needed | `tests/hooks/` | marketplace |
| Codex | `.codex-plugin/plugin.json` (declares empty `hooks`) | native skill discovery (no session-start hook) | `references/codex-tools.md` | `tests/codex/`, `tests/codex-plugin-sync/` | fork sync (`scripts/sync-to-codex-plugin.sh`) | | Codex | `.codex-plugin/plugin.json` (declares empty `hooks`) | native skill discovery (no session-start hook) | `references/codex-tools.md` | `tests/codex/`, `tests/codex-plugin-sync/` | fork sync (`scripts/sync-to-codex-plugin.sh`) |
| Cursor | `.cursor-plugin/plugin.json` + `hooks/hooks-cursor.json` | shell hook → `hooks/session-start` (`additional_context`) | `references/claude-code-tools.md` | `tests/hooks/` | hand-authored | | Cursor | `.cursor-plugin/plugin.json` + `hooks/hooks-cursor.json` | shell hook → `hooks/session-start` (`additional_context`) | none needed (Claude Codecompatible tool surface) | `tests/hooks/` | hand-authored |
| Copilot CLI | (shares Claude Code hook path; `COPILOT_CLI` env) | shell hook → `hooks/session-start` (`additionalContext`) | `references/copilot-tools.md` | `tests/hooks/` | — | | Copilot CLI | (shares Claude Code hook path; `COPILOT_CLI` env) | shell hook → `hooks/session-start` (`additionalContext`) | none needed (Claude Codecompatible tool surface) | `tests/hooks/` | — |
| Gemini CLI | `gemini-extension.json` + `GEMINI.md` | instructions file `@`-includes bootstrap + mapping | `references/gemini-tools.md` | — | `gemini extensions install` | | Gemini CLI | `gemini-extension.json` + `GEMINI.md` | instructions file `@`-includes bootstrap + mapping | `references/gemini-tools.md` | — | `gemini extensions install` |
| Kimi Code | `.kimi-plugin/plugin.json` | manifest `sessionStart.skill` loads `using-superpowers` | inline `skillInstructions` in manifest | `tests/kimi/` | marketplace or `/plugins install` GitHub URL | | Kimi Code | `.kimi-plugin/plugin.json` | manifest `sessionStart.skill` loads `using-superpowers` | inline `skillInstructions` in manifest | `tests/kimi/` | marketplace or `/plugins install` GitHub URL |
| OpenCode | `.opencode/plugins/superpowers.js` (declared via root `package.json` `main`) | in-process: `config` hook registers skills dir; `experimental.chat.messages.transform` injects user message | inline in `superpowers.js` | `tests/opencode/` | `opencode.json` plugin git URL | | OpenCode | `.opencode/plugins/superpowers.js` (declared via root `package.json` `main`) | in-process: `config` hook registers skills dir; `experimental.chat.messages.transform` injects user message | inline in `superpowers.js` | `tests/opencode/` | `opencode.json` plugin git URL |

View File

@@ -230,14 +230,16 @@ prepare_metadata_root() {
METADATA_ROOT="$(prepare_metadata_root "$METADATA_SOURCE")" METADATA_ROOT="$(prepare_metadata_root "$METADATA_SOURCE")"
git -C "$REPO_ROOT" archive --format=tar "$REF" -- \ # Pin tar.umask and extract with -p so staged modes are canonical 755/644
# regardless of the builder's git config or process umask.
git -C "$REPO_ROOT" -c tar.umask=0022 archive --format=tar "$REF" -- \
.codex-plugin \ .codex-plugin \
CODE_OF_CONDUCT.md \ CODE_OF_CONDUCT.md \
LICENSE \ LICENSE \
README.md \ README.md \
assets \ assets \
skills \ skills \
| tar -xf - -C "$STAGE" | tar -xpf - -C "$STAGE"
VERSION="$(jq -r '.version // empty' "$STAGE/.codex-plugin/plugin.json")" VERSION="$(jq -r '.version // empty' "$STAGE/.codex-plugin/plugin.json")"
[[ -n "$VERSION" ]] || die "could not read version from .codex-plugin/plugin.json" [[ -n "$VERSION" ]] || die "could not read version from .codex-plugin/plugin.json"
@@ -298,12 +300,19 @@ case "$FORMAT" in
) )
;; ;;
tar.gz) tar.gz)
# Match the prior official archive's deterministic tar entry metadata. # Match the prior official archive's deterministic tar entry metadata:
# ustar entries with uid/gid 0 and empty uname/gname. GNU tar and bsdtar
# (macOS) spell those flags differently.
if tar --version 2>/dev/null | grep -q 'GNU tar'; then
TAR_METADATA_FLAGS=(--owner=:0 --group=:0 --numeric-owner)
else
TAR_METADATA_FLAGS=(--uid 0 --gid 0 --uname '' --gname '')
fi
TZ=UTC find "$STAGE" -exec touch -t 197001010000 {} + TZ=UTC find "$STAGE" -exec touch -t 197001010000 {} +
( (
cd "$STAGE" cd "$STAGE"
rm -f "$OUTPUT" rm -f "$OUTPUT"
COPYFILE_DISABLE=1 tar -cf - --no-recursion --format ustar --uid 0 --gid 0 --uname '' --gname '' -T "$ARCHIVE_LIST" | COPYFILE_DISABLE=1 tar -cf - --no-recursion --format ustar "${TAR_METADATA_FLAGS[@]}" -T "$ARCHIVE_LIST" |
gzip -9n >"$OUTPUT" gzip -9n >"$OUTPUT"
) )
;; ;;

View File

@@ -9,7 +9,7 @@ description: Use when creating new skills, editing existing skills, or verifying
**Writing skills IS Test-Driven Development applied to process documentation.** **Writing skills IS Test-Driven Development applied to process documentation.**
**Personal skills live in your runtime's skills directory** — see [claude-code-tools.md](../using-superpowers/references/claude-code-tools.md), [codex-tools.md](../using-superpowers/references/codex-tools.md), [copilot-tools.md](../using-superpowers/references/copilot-tools.md), or [gemini-tools.md](../using-superpowers/references/gemini-tools.md) for the path on your runtime. Codex, Copilot CLI, and Gemini CLI all also recognize `~/.agents/skills/` as a cross-runtime alias. **Personal skills live in your runtime's skills directory** (`~/.claude/skills/` on Claude Code) — see [codex-tools.md](../using-superpowers/references/codex-tools.md) or [gemini-tools.md](../using-superpowers/references/gemini-tools.md) for the path on those runtimes. Codex, Copilot CLI, and Gemini CLI all also recognize `~/.agents/skills/` as a cross-runtime alias.
You write test cases (pressure scenarios with subagents), watch them fail (baseline behavior), write the skill (documentation), watch tests pass (agents comply), and refactor (close loopholes). You write test cases (pressure scenarios with subagents), watch them fail (baseline behavior), write the skill (documentation), watch tests pass (agents comply), and refactor (close loopholes).

View File

@@ -2,8 +2,9 @@
# Validate the Antigravity (agy) integration. agy installs the existing plugin # Validate the Antigravity (agy) integration. agy installs the existing plugin
# directly (`agy plugin install <repo-url>`): it loads the bundled skills and # directly (`agy plugin install <repo-url>`): it loads the bundled skills and
# runs the SessionStart hook for bootstrap, so there is no agy-specific scaffold # runs the SessionStart hook for bootstrap, so there is no agy-specific scaffold
# to test. What IS agy-specific is the tool mapping — agy has no `Skill` tool and # to test. What IS agy-specific is the tool mapping — subagent dispatch via
# loads skills by reading SKILL.md with view_file — and SKILL.md pointing at it. # invoke_subagent (self/research types) and task tracking via a task artifact —
# and SKILL.md pointing at it.
# #
# Mirrors tests/pi/test-pi-extension.mjs's "tools reference documents # Mirrors tests/pi/test-pi-extension.mjs's "tools reference documents
# harness-specific mappings" check. CI-safe: does not require `agy` installed. # harness-specific mappings" check. CI-safe: does not require `agy` installed.
@@ -22,16 +23,8 @@ echo "test-antigravity-tools: checking Antigravity tool mapping"
# --- Mapping exists --------------------------------------------------------- # --- Mapping exists ---------------------------------------------------------
[ -f "$MAPPING" ] || fail "tool mapping missing at $MAPPING" [ -f "$MAPPING" ] || fail "tool mapping missing at $MAPPING"
# --- Skill-load mechanism: view_file on SKILL.md (IsSkillFile), no Skill tool -
grep -qiE "view_file" "$MAPPING" \
|| fail "mapping does not document view_file as the file/skill-read tool"
grep -qiE "SKILL\.md" "$MAPPING" \
|| fail "mapping does not document reading SKILL.md as the skill-load path"
grep -q "IsSkillFile" "$MAPPING" \
|| fail "mapping does not document setting IsSkillFile when loading a skill"
# --- Core action→tool mappings are documented ------------------------------- # --- Core action→tool mappings are documented -------------------------------
for tool in write_to_file replace_file_content run_command grep_search invoke_subagent; do for tool in write_to_file replace_file_content invoke_subagent; do
grep -q "$tool" "$MAPPING" \ grep -q "$tool" "$MAPPING" \
|| fail "mapping does not document the '$tool' tool" || fail "mapping does not document the '$tool' tool"
done done
@@ -50,4 +43,4 @@ grep -qE 'ArtifactType.*task|task. artifact' "$MAPPING" \
grep -q "antigravity-tools.md" "$SKILL" \ grep -q "antigravity-tools.md" "$SKILL" \
|| fail "SKILL.md Platform Adaptation does not reference antigravity-tools.md" || fail "SKILL.md Platform Adaptation does not reference antigravity-tools.md"
echo "PASS: Antigravity tool mapping valid (view_file skill-load, agy tools, SKILL.md link)" echo "PASS: Antigravity tool mapping valid (subagent dispatch, task artifact, SKILL.md link)"

View File

@@ -25,7 +25,8 @@ fi
# Parse command line arguments # Parse command line arguments
VERBOSE=false VERBOSE=false
SPECIFIC_TEST="" SPECIFIC_TEST=""
TIMEOUT=600 # Default 10 minute timeout per test TIMEOUT=900 # Per-test-file budget; must exceed the file's worst case
# (test-subagent-driven-development.sh: 9 prompts x 90s each)
RUN_INTEGRATION=false RUN_INTEGRATION=false
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
@@ -52,7 +53,7 @@ while [[ $# -gt 0 ]]; do
echo "Options:" echo "Options:"
echo " --verbose, -v Show verbose output" echo " --verbose, -v Show verbose output"
echo " --test, -t NAME Run only the specified test" echo " --test, -t NAME Run only the specified test"
echo " --timeout SECONDS Set timeout per test (default: 300)" echo " --timeout SECONDS Set timeout per test (default: 900)"
echo " --integration, -i Run integration tests (slow, 10-30 min)" echo " --integration, -i Run integration tests (slow, 10-30 min)"
echo " --help, -h Show this help" echo " --help, -h Show this help"
echo "" echo ""

View File

@@ -30,12 +30,14 @@ run_claude() {
# Check if output contains a pattern # Check if output contains a pattern
# Usage: assert_contains "output" "pattern" "test name" # Usage: assert_contains "output" "pattern" "test name"
# Matching is case-insensitive: patterns are prose keywords, and models
# freely capitalize skill terms ("Do Not Trust", "Spec Compliance").
assert_contains() { assert_contains() {
local output="$1" local output="$1"
local pattern="$2" local pattern="$2"
local test_name="${3:-test}" local test_name="${3:-test}"
if echo "$output" | grep -q "$pattern"; then if echo "$output" | grep -qi "$pattern"; then
echo " [PASS] $test_name" echo " [PASS] $test_name"
return 0 return 0
else else
@@ -54,7 +56,7 @@ assert_not_contains() {
local pattern="$2" local pattern="$2"
local test_name="${3:-test}" local test_name="${3:-test}"
if echo "$output" | grep -q "$pattern"; then if echo "$output" | grep -qi "$pattern"; then
echo " [FAIL] $test_name" echo " [FAIL] $test_name"
echo " Did not expect to find: $pattern" echo " Did not expect to find: $pattern"
echo " In output:" echo " In output:"
@@ -74,7 +76,7 @@ assert_count() {
local expected="$3" local expected="$3"
local test_name="${4:-test}" local test_name="${4:-test}"
local actual=$(echo "$output" | grep -c "$pattern" || echo "0") local actual=$(echo "$output" | grep -ci "$pattern" || echo "0")
if [ "$actual" -eq "$expected" ]; then if [ "$actual" -eq "$expected" ]; then
echo " [PASS] $test_name (found $actual instances)" echo " [PASS] $test_name (found $actual instances)"
@@ -98,16 +100,20 @@ assert_order() {
local test_name="${4:-test}" local test_name="${4:-test}"
# Get line numbers where patterns appear # Get line numbers where patterns appear
local line_a=$(echo "$output" | grep -n "$pattern_a" | head -1 | cut -d: -f1) local line_a=$(echo "$output" | grep -ni "$pattern_a" | head -1 | cut -d: -f1)
local line_b=$(echo "$output" | grep -n "$pattern_b" | head -1 | cut -d: -f1) local line_b=$(echo "$output" | grep -ni "$pattern_b" | head -1 | cut -d: -f1)
if [ -z "$line_a" ]; then if [ -z "$line_a" ]; then
echo " [FAIL] $test_name: pattern A not found: $pattern_a" echo " [FAIL] $test_name: pattern A not found: $pattern_a"
echo " In output:"
echo "$output" | sed 's/^/ /'
return 1 return 1
fi fi
if [ -z "$line_b" ]; then if [ -z "$line_b" ]; then
echo " [FAIL] $test_name: pattern B not found: $pattern_b" echo " [FAIL] $test_name: pattern B not found: $pattern_b"
echo " In output:"
echo "$output" | sed 's/^/ /'
return 1 return 1
fi fi

View File

@@ -96,13 +96,13 @@ echo "Test 5: Spec compliance reviewer mindset..."
output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" "$CLAUDE_PROMPT_TIMEOUT") output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" "$CLAUDE_PROMPT_TIMEOUT")
if assert_contains "$output" "not trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then if assert_contains "$output" "not.*trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then
: # pass : # pass
else else
exit 1 exit 1
fi fi
if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code" "Reviewer reads code"; then if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code\|read.*diff\|trust.*diff" "Reviewer reads code"; then
: # pass : # pass
else else
exit 1 exit 1

View File

@@ -210,8 +210,13 @@ assert_equals "$tar_archive_paths" "$archive_paths" "zip and tar.gz archives con
tar_task_brief_mode="$(tar -tzvf "$tar_archive" skills/subagent-driven-development/scripts/task-brief | awk '{print $1}')" tar_task_brief_mode="$(tar -tzvf "$tar_archive" skills/subagent-driven-development/scripts/task-brief | awk '{print $1}')"
assert_equals "$tar_task_brief_mode" "-rwxr-xr-x" "tar.gz archive preserves executable script mode" assert_equals "$tar_task_brief_mode" "-rwxr-xr-x" "tar.gz archive preserves executable script mode"
tar_metadata_times="$(tar -tzvf "$tar_archive" | awk '{print $6, $7, $8}' | sort -u)" tar_metadata_times="$(python3 - "$tar_archive" <<'PY'
assert_equals "$tar_metadata_times" "Dec 31 1969" "tar.gz archive normalizes entry timestamps" import sys, tarfile
with tarfile.open(sys.argv[1]) as archive:
print(sorted({member.mtime for member in archive.getmembers()}))
PY
)"
assert_equals "$tar_metadata_times" "[0]" "tar.gz archive normalizes entry timestamps"
metadata_archive="$TEST_ROOT/metadata-source.tar.gz" metadata_archive="$TEST_ROOT/metadata-source.tar.gz"
metadata_zip="$TEST_ROOT/metadata-source.zip" metadata_zip="$TEST_ROOT/metadata-source.zip"

View File

@@ -122,7 +122,16 @@ test('pi tools reference documents pi-specific mappings', async () => {
assert.equal(existsSync(piToolsPath), true, 'pi-tools.md should exist'); assert.equal(existsSync(piToolsPath), true, 'pi-tools.md should exist');
const text = await readFile(piToolsPath, 'utf8'); const text = await readFile(piToolsPath, 'utf8');
for (const expected of ['Skill', 'Task', 'TodoWrite', 'read', 'write', 'edit', 'bash']) { // Assert against the mapping-table rows only. The surrounding prose mentions
assert.match(text, new RegExp(expected)); // these same tokens, so matching the whole file would still pass if the table
} // were deleted — the exact regression this test exists to catch.
const rows = text.split('\n').filter((line) => line.startsWith('|'));
assert.ok(
rows.some((row) => /subagent/i.test(row)),
'mapping table documents subagent dispatch',
);
assert.ok(
rows.some((row) => /todo|task/i.test(row)),
'mapping table documents task tracking',
);
}); });