mirror of
https://github.com/obra/superpowers.git
synced 2026-04-30 22:19:05 +08:00
Compare commits
5 Commits
codex/read
...
lift-code-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
45975ec695 | ||
|
|
e795530c23 | ||
|
|
28fd7a8192 | ||
|
|
831f6f977c | ||
|
|
5745f0ea99 |
@@ -1,48 +0,0 @@
|
||||
---
|
||||
name: code-reviewer
|
||||
description: |
|
||||
Use this agent when a major project step has been completed and needs to be reviewed against the original plan and coding standards. Examples: <example>Context: The user is creating a code-review agent that should be called after a logical chunk of code is written. user: "I've finished implementing the user authentication system as outlined in step 3 of our plan" assistant: "Great work! Now let me use the code-reviewer agent to review the implementation against our plan and coding standards" <commentary>Since a major project step has been completed, use the code-reviewer agent to validate the work against the plan and identify any issues.</commentary></example> <example>Context: User has completed a significant feature implementation. user: "The API endpoints for the task management system are now complete - that covers step 2 from our architecture document" assistant: "Excellent! Let me have the code-reviewer agent examine this implementation to ensure it aligns with our plan and follows best practices" <commentary>A numbered step from the planning document has been completed, so the code-reviewer agent should review the work.</commentary></example>
|
||||
model: inherit
|
||||
---
|
||||
|
||||
You are a Senior Code Reviewer with expertise in software architecture, design patterns, and best practices. Your role is to review completed project steps against original plans and ensure code quality standards are met.
|
||||
|
||||
When reviewing completed work, you will:
|
||||
|
||||
1. **Plan Alignment Analysis**:
|
||||
- Compare the implementation against the original planning document or step description
|
||||
- Identify any deviations from the planned approach, architecture, or requirements
|
||||
- Assess whether deviations are justified improvements or problematic departures
|
||||
- Verify that all planned functionality has been implemented
|
||||
|
||||
2. **Code Quality Assessment**:
|
||||
- Review code for adherence to established patterns and conventions
|
||||
- Check for proper error handling, type safety, and defensive programming
|
||||
- Evaluate code organization, naming conventions, and maintainability
|
||||
- Assess test coverage and quality of test implementations
|
||||
- Look for potential security vulnerabilities or performance issues
|
||||
|
||||
3. **Architecture and Design Review**:
|
||||
- Ensure the implementation follows SOLID principles and established architectural patterns
|
||||
- Check for proper separation of concerns and loose coupling
|
||||
- Verify that the code integrates well with existing systems
|
||||
- Assess scalability and extensibility considerations
|
||||
|
||||
4. **Documentation and Standards**:
|
||||
- Verify that code includes appropriate comments and documentation
|
||||
- Check that file headers, function documentation, and inline comments are present and accurate
|
||||
- Ensure adherence to project-specific coding standards and conventions
|
||||
|
||||
5. **Issue Identification and Recommendations**:
|
||||
- Clearly categorize issues as: Critical (must fix), Important (should fix), or Suggestions (nice to have)
|
||||
- For each issue, provide specific examples and actionable recommendations
|
||||
- When you identify plan deviations, explain whether they're problematic or beneficial
|
||||
- Suggest specific improvements with code examples when helpful
|
||||
|
||||
6. **Communication Protocol**:
|
||||
- If you find significant deviations from the plan, ask the coding agent to review and confirm the changes
|
||||
- If you identify issues with the original plan itself, recommend plan updates
|
||||
- For implementation problems, provide clear guidance on fixes needed
|
||||
- Always acknowledge what was done well before highlighting issues
|
||||
|
||||
Your output should be structured, actionable, and focused on helping maintain high code quality while ensuring project goals are met. Be thorough but concise, and always provide constructive feedback that helps improve both the current implementation and future development practices.
|
||||
@@ -3,7 +3,7 @@
|
||||
"hooks": {
|
||||
"sessionStart": [
|
||||
{
|
||||
"command": "./hooks/session-start"
|
||||
"command": "./hooks/run-hook.cmd session-start"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@ description: Use when completing tasks, implementing major features, or before m
|
||||
|
||||
# Requesting Code Review
|
||||
|
||||
Dispatch superpowers:code-reviewer subagent to catch issues before they cascade. The reviewer gets precisely crafted context for evaluation — never your session's history. This keeps the reviewer focused on the work product, not your thought process, and preserves your own context for continued work.
|
||||
Dispatch a code reviewer subagent to catch issues before they cascade. The reviewer gets precisely crafted context for evaluation — never your session's history. This keeps the reviewer focused on the work product, not your thought process, and preserves your own context for continued work.
|
||||
|
||||
**Core principle:** Review early, review often.
|
||||
|
||||
@@ -29,16 +29,15 @@ BASE_SHA=$(git rev-parse HEAD~1) # or origin/main
|
||||
HEAD_SHA=$(git rev-parse HEAD)
|
||||
```
|
||||
|
||||
**2. Dispatch code-reviewer subagent:**
|
||||
**2. Dispatch code reviewer subagent:**
|
||||
|
||||
Use Task tool with superpowers:code-reviewer type, fill template at `code-reviewer.md`
|
||||
Use Task tool with `general-purpose` type, fill template at `code-reviewer.md`
|
||||
|
||||
**Placeholders:**
|
||||
- `{WHAT_WAS_IMPLEMENTED}` - What you just built
|
||||
- `{DESCRIPTION}` - Brief summary of what you built
|
||||
- `{PLAN_OR_REQUIREMENTS}` - What it should do
|
||||
- `{BASE_SHA}` - Starting commit
|
||||
- `{HEAD_SHA}` - Ending commit
|
||||
- `{DESCRIPTION}` - Brief summary
|
||||
|
||||
**3. Act on feedback:**
|
||||
- Fix Critical issues immediately
|
||||
@@ -56,12 +55,11 @@ You: Let me request code review before proceeding.
|
||||
BASE_SHA=$(git log --oneline | grep "Task 1" | head -1 | awk '{print $1}')
|
||||
HEAD_SHA=$(git rev-parse HEAD)
|
||||
|
||||
[Dispatch superpowers:code-reviewer subagent]
|
||||
WHAT_WAS_IMPLEMENTED: Verification and repair functions for conversation index
|
||||
[Dispatch code reviewer subagent]
|
||||
DESCRIPTION: Added verifyIndex() and repairIndex() with 4 issue types
|
||||
PLAN_OR_REQUIREMENTS: Task 2 from docs/superpowers/plans/deployment-plan.md
|
||||
BASE_SHA: a7981ec
|
||||
HEAD_SHA: 3df7661
|
||||
DESCRIPTION: Added verifyIndex() and repairIndex() with 4 issue types
|
||||
|
||||
[Subagent returns]:
|
||||
Strengths: Clean architecture, real tests
|
||||
|
||||
@@ -1,111 +1,133 @@
|
||||
# Code Review Agent
|
||||
# Code Reviewer Prompt Template
|
||||
|
||||
You are reviewing code changes for production readiness.
|
||||
Use this template when dispatching a code reviewer subagent.
|
||||
|
||||
**Your task:**
|
||||
1. Review {WHAT_WAS_IMPLEMENTED}
|
||||
2. Compare against {PLAN_OR_REQUIREMENTS}
|
||||
3. Check code quality, architecture, testing
|
||||
4. Categorize issues by severity
|
||||
5. Assess production readiness
|
||||
**Purpose:** Review completed work against requirements and code quality standards before it cascades into more work.
|
||||
|
||||
## What Was Implemented
|
||||
```
|
||||
Task tool (general-purpose):
|
||||
description: "Review code changes"
|
||||
prompt: |
|
||||
You are a Senior Code Reviewer with expertise in software architecture,
|
||||
design patterns, and best practices. Your job is to review completed work
|
||||
against its plan or requirements and identify issues before they cascade.
|
||||
|
||||
{DESCRIPTION}
|
||||
## What Was Implemented
|
||||
|
||||
## Requirements/Plan
|
||||
{DESCRIPTION}
|
||||
|
||||
{PLAN_REFERENCE}
|
||||
## Requirements / Plan
|
||||
|
||||
## Git Range to Review
|
||||
{PLAN_OR_REQUIREMENTS}
|
||||
|
||||
**Base:** {BASE_SHA}
|
||||
**Head:** {HEAD_SHA}
|
||||
## Git Range to Review
|
||||
|
||||
```bash
|
||||
git diff --stat {BASE_SHA}..{HEAD_SHA}
|
||||
git diff {BASE_SHA}..{HEAD_SHA}
|
||||
**Base:** {BASE_SHA}
|
||||
**Head:** {HEAD_SHA}
|
||||
|
||||
```bash
|
||||
git diff --stat {BASE_SHA}..{HEAD_SHA}
|
||||
git diff {BASE_SHA}..{HEAD_SHA}
|
||||
```
|
||||
|
||||
## What to Check
|
||||
|
||||
**Plan alignment:**
|
||||
- Does the implementation match the plan / requirements?
|
||||
- Are deviations justified improvements, or problematic departures?
|
||||
- Is all planned functionality present?
|
||||
|
||||
**Code quality:**
|
||||
- Clean separation of concerns?
|
||||
- Proper error handling?
|
||||
- Type safety where applicable?
|
||||
- DRY without premature abstraction?
|
||||
- Edge cases handled?
|
||||
|
||||
**Architecture:**
|
||||
- Sound design decisions?
|
||||
- Reasonable scalability and performance?
|
||||
- Security concerns?
|
||||
- Integrates cleanly with surrounding code?
|
||||
|
||||
**Testing:**
|
||||
- Tests verify real behavior, not mocks?
|
||||
- Edge cases covered?
|
||||
- Integration tests where they matter?
|
||||
- All tests passing?
|
||||
|
||||
**Production readiness:**
|
||||
- Migration strategy if schema changed?
|
||||
- Backward compatibility considered?
|
||||
- Documentation complete?
|
||||
- No obvious bugs?
|
||||
|
||||
## Calibration
|
||||
|
||||
Categorize issues by actual severity. Not everything is Critical.
|
||||
Acknowledge what was done well before listing issues — accurate praise
|
||||
helps the implementer trust the rest of the feedback.
|
||||
|
||||
If you find significant deviations from the plan, flag them specifically
|
||||
so the implementer can confirm whether the deviation was intentional.
|
||||
If you find issues with the plan itself rather than the implementation,
|
||||
say so.
|
||||
|
||||
## Output Format
|
||||
|
||||
### Strengths
|
||||
[What's well done? Be specific.]
|
||||
|
||||
### Issues
|
||||
|
||||
#### Critical (Must Fix)
|
||||
[Bugs, security issues, data loss risks, broken functionality]
|
||||
|
||||
#### Important (Should Fix)
|
||||
[Architecture problems, missing features, poor error handling, test gaps]
|
||||
|
||||
#### Minor (Nice to Have)
|
||||
[Code style, optimization opportunities, documentation polish]
|
||||
|
||||
For each issue:
|
||||
- File:line reference
|
||||
- What's wrong
|
||||
- Why it matters
|
||||
- How to fix (if not obvious)
|
||||
|
||||
### Recommendations
|
||||
[Improvements for code quality, architecture, or process]
|
||||
|
||||
### Assessment
|
||||
|
||||
**Ready to merge?** [Yes | No | With fixes]
|
||||
|
||||
**Reasoning:** [1-2 sentence technical assessment]
|
||||
|
||||
## Critical Rules
|
||||
|
||||
**DO:**
|
||||
- Categorize by actual severity
|
||||
- Be specific (file:line, not vague)
|
||||
- Explain WHY each issue matters
|
||||
- Acknowledge strengths
|
||||
- Give a clear verdict
|
||||
|
||||
**DON'T:**
|
||||
- Say "looks good" without checking
|
||||
- Mark nitpicks as Critical
|
||||
- Give feedback on code you didn't actually read
|
||||
- Be vague ("improve error handling")
|
||||
- Avoid giving a clear verdict
|
||||
```
|
||||
|
||||
## Review Checklist
|
||||
**Placeholders:**
|
||||
- `{DESCRIPTION}` — brief summary of what was built
|
||||
- `{PLAN_OR_REQUIREMENTS}` — what it should do (plan file path, task text, or requirements)
|
||||
- `{BASE_SHA}` — starting commit
|
||||
- `{HEAD_SHA}` — ending commit
|
||||
|
||||
**Code Quality:**
|
||||
- Clean separation of concerns?
|
||||
- Proper error handling?
|
||||
- Type safety (if applicable)?
|
||||
- DRY principle followed?
|
||||
- Edge cases handled?
|
||||
|
||||
**Architecture:**
|
||||
- Sound design decisions?
|
||||
- Scalability considerations?
|
||||
- Performance implications?
|
||||
- Security concerns?
|
||||
|
||||
**Testing:**
|
||||
- Tests actually test logic (not mocks)?
|
||||
- Edge cases covered?
|
||||
- Integration tests where needed?
|
||||
- All tests passing?
|
||||
|
||||
**Requirements:**
|
||||
- All plan requirements met?
|
||||
- Implementation matches spec?
|
||||
- No scope creep?
|
||||
- Breaking changes documented?
|
||||
|
||||
**Production Readiness:**
|
||||
- Migration strategy (if schema changes)?
|
||||
- Backward compatibility considered?
|
||||
- Documentation complete?
|
||||
- No obvious bugs?
|
||||
|
||||
## Output Format
|
||||
|
||||
### Strengths
|
||||
[What's well done? Be specific.]
|
||||
|
||||
### Issues
|
||||
|
||||
#### Critical (Must Fix)
|
||||
[Bugs, security issues, data loss risks, broken functionality]
|
||||
|
||||
#### Important (Should Fix)
|
||||
[Architecture problems, missing features, poor error handling, test gaps]
|
||||
|
||||
#### Minor (Nice to Have)
|
||||
[Code style, optimization opportunities, documentation improvements]
|
||||
|
||||
**For each issue:**
|
||||
- File:line reference
|
||||
- What's wrong
|
||||
- Why it matters
|
||||
- How to fix (if not obvious)
|
||||
|
||||
### Recommendations
|
||||
[Improvements for code quality, architecture, or process]
|
||||
|
||||
### Assessment
|
||||
|
||||
**Ready to merge?** [Yes/No/With fixes]
|
||||
|
||||
**Reasoning:** [Technical assessment in 1-2 sentences]
|
||||
|
||||
## Critical Rules
|
||||
|
||||
**DO:**
|
||||
- Categorize by actual severity (not everything is Critical)
|
||||
- Be specific (file:line, not vague)
|
||||
- Explain WHY issues matter
|
||||
- Acknowledge strengths
|
||||
- Give clear verdict
|
||||
|
||||
**DON'T:**
|
||||
- Say "looks good" without checking
|
||||
- Mark nitpicks as Critical
|
||||
- Give feedback on code you didn't review
|
||||
- Be vague ("improve error handling")
|
||||
- Avoid giving a clear verdict
|
||||
**Reviewer returns:** Strengths, Issues (Critical / Important / Minor), Recommendations, Assessment
|
||||
|
||||
## Example Output
|
||||
|
||||
|
||||
@@ -7,14 +7,13 @@ Use this template when dispatching a code quality reviewer subagent.
|
||||
**Only dispatch after spec compliance review passes.**
|
||||
|
||||
```
|
||||
Task tool (superpowers:code-reviewer):
|
||||
Task tool (general-purpose):
|
||||
Use template at requesting-code-review/code-reviewer.md
|
||||
|
||||
WHAT_WAS_IMPLEMENTED: [from implementer's report]
|
||||
DESCRIPTION: [task summary, from implementer's report]
|
||||
PLAN_OR_REQUIREMENTS: Task N from [plan-file]
|
||||
BASE_SHA: [commit before task]
|
||||
HEAD_SHA: [current commit]
|
||||
DESCRIPTION: [task summary]
|
||||
```
|
||||
|
||||
**In addition to standard code quality concerns, the reviewer should check:**
|
||||
|
||||
@@ -4,9 +4,9 @@ Skills use Claude Code tool names. When you encounter these in a skill, use your
|
||||
|
||||
| Skill references | Codex equivalent |
|
||||
|-----------------|------------------|
|
||||
| `Task` tool (dispatch subagent) | `spawn_agent` (see [Named agent dispatch](#named-agent-dispatch)) |
|
||||
| `Task` tool (dispatch subagent) | `spawn_agent` (see [Subagent dispatch requires multi-agent support](#subagent-dispatch-requires-multi-agent-support)) |
|
||||
| Multiple `Task` calls (parallel) | Multiple `spawn_agent` calls |
|
||||
| Task returns result | `wait` |
|
||||
| Task returns result | `wait_agent` |
|
||||
| Task completes automatically | `close_agent` to free slot |
|
||||
| `TodoWrite` (task tracking) | `update_plan` |
|
||||
| `Skill` tool (invoke a skill) | Skills load natively — just follow the instructions |
|
||||
@@ -22,53 +22,12 @@ Add to your Codex config (`~/.codex/config.toml`):
|
||||
multi_agent = true
|
||||
```
|
||||
|
||||
This enables `spawn_agent`, `wait`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`.
|
||||
This enables `spawn_agent`, `wait_agent`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`.
|
||||
|
||||
## Named agent dispatch
|
||||
|
||||
Claude Code skills reference named agent types like `superpowers:code-reviewer`.
|
||||
Codex does not have a named agent registry — `spawn_agent` creates generic agents
|
||||
from built-in roles (`default`, `explorer`, `worker`).
|
||||
|
||||
When a skill says to dispatch a named agent type:
|
||||
|
||||
1. Find the agent's prompt file (e.g., `agents/code-reviewer.md` or the skill's
|
||||
local prompt template like `code-quality-reviewer-prompt.md`)
|
||||
2. Read the prompt content
|
||||
3. Fill any template placeholders (`{BASE_SHA}`, `{WHAT_WAS_IMPLEMENTED}`, etc.)
|
||||
4. Spawn a `worker` agent with the filled content as the `message`
|
||||
|
||||
| Skill instruction | Codex equivalent |
|
||||
|-------------------|------------------|
|
||||
| `Task tool (superpowers:code-reviewer)` | `spawn_agent(agent_type="worker", message=...)` with `code-reviewer.md` content |
|
||||
| `Task tool (general-purpose)` with inline prompt | `spawn_agent(message=...)` with the same prompt |
|
||||
|
||||
### Message framing
|
||||
|
||||
The `message` parameter is user-level input, not a system prompt. Structure it
|
||||
for maximum instruction adherence:
|
||||
|
||||
```
|
||||
Your task is to perform the following. Follow the instructions below exactly.
|
||||
|
||||
<agent-instructions>
|
||||
[filled prompt content from the agent's .md file]
|
||||
</agent-instructions>
|
||||
|
||||
Execute this now. Output ONLY the structured response following the format
|
||||
specified in the instructions above.
|
||||
```
|
||||
|
||||
- Use task-delegation framing ("Your task is...") rather than persona framing ("You are...")
|
||||
- Wrap instructions in XML tags — the model treats tagged blocks as authoritative
|
||||
- End with an explicit execution directive to prevent summarization of the instructions
|
||||
|
||||
### When this workaround can be removed
|
||||
|
||||
This approach compensates for Codex's plugin system not yet supporting an `agents`
|
||||
field in `plugin.json`. When `RawPluginManifest` gains an `agents` field, the
|
||||
plugin can symlink to `agents/` (mirroring the existing `skills/` symlink) and
|
||||
skills can dispatch named agent types directly.
|
||||
Legacy note: Codex builds before `rust-v0.115.0` exposed spawned-agent
|
||||
waiting as `wait`. Current Codex uses `wait_agent` for spawned agents. The
|
||||
`wait` name now belongs to code-mode `exec/wait`, which resumes a yielded exec
|
||||
cell by `cell_id`; it is not the spawned-agent result tool.
|
||||
|
||||
## Environment Detection
|
||||
|
||||
|
||||
@@ -12,23 +12,13 @@ Skills use Claude Code tool names. When you encounter these in a skill, use your
|
||||
| `Glob` (search files by name) | `glob` |
|
||||
| `Skill` tool (invoke a skill) | `skill` |
|
||||
| `WebFetch` | `web_fetch` |
|
||||
| `Task` tool (dispatch subagent) | `task` (see [Agent types](#agent-types)) |
|
||||
| `Task` tool (dispatch subagent) | `task` with `agent_type: "general-purpose"` or `"explore"` |
|
||||
| Multiple `Task` calls (parallel) | Multiple `task` calls |
|
||||
| Task status/output | `read_agent`, `list_agents` |
|
||||
| `TodoWrite` (task tracking) | `sql` with built-in `todos` table |
|
||||
| `WebSearch` | No equivalent — use `web_fetch` with a search engine URL |
|
||||
| `EnterPlanMode` / `ExitPlanMode` | No equivalent — stay in the main session |
|
||||
|
||||
## Agent types
|
||||
|
||||
Copilot CLI's `task` tool accepts an `agent_type` parameter:
|
||||
|
||||
| Claude Code agent | Copilot CLI equivalent |
|
||||
|-------------------|----------------------|
|
||||
| `general-purpose` | `"general-purpose"` |
|
||||
| `Explore` | `"explore"` |
|
||||
| Named plugin agents (e.g. `superpowers:code-reviewer`) | Discovered automatically from installed plugins |
|
||||
|
||||
## Async shell sessions
|
||||
|
||||
Copilot CLI supports persistent async shell sessions, which have no direct Claude Code equivalent:
|
||||
|
||||
@@ -115,6 +115,18 @@ Full workflow execution test (~10-30 minutes):
|
||||
- Subagents follow the skill correctly
|
||||
- Final code is functional and tested
|
||||
|
||||
#### test-requesting-code-review.sh
|
||||
Behavioral test for the code reviewer subagent (~5 minutes):
|
||||
- Builds a tiny project with a baseline commit
|
||||
- Adds a second commit that plants two real bugs (SQL injection, plaintext password handling)
|
||||
- Dispatches the code reviewer via the requesting-code-review skill
|
||||
- Verifies the reviewer flags the planted bugs at Critical/Important severity and refuses to approve
|
||||
|
||||
**What it tests:**
|
||||
- The skill actually dispatches a working code reviewer subagent
|
||||
- The reviewer template produces reviewers that catch obvious security bugs
|
||||
- The reviewer is not sycophantic — it does not approve a diff with planted Critical issues
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
1. Create new test file: `test-<skill-name>.sh`
|
||||
|
||||
@@ -79,6 +79,7 @@ tests=(
|
||||
# Integration tests (slow, full execution)
|
||||
integration_tests=(
|
||||
"test-subagent-driven-development-integration.sh"
|
||||
"test-requesting-code-review.sh"
|
||||
)
|
||||
|
||||
# Add integration tests if requested
|
||||
|
||||
214
tests/claude-code/test-requesting-code-review.sh
Executable file
214
tests/claude-code/test-requesting-code-review.sh
Executable file
@@ -0,0 +1,214 @@
|
||||
#!/usr/bin/env bash
|
||||
# Integration Test: requesting-code-review skill
|
||||
# Verifies the code reviewer dispatched via the skill catches a planted bug
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
source "$SCRIPT_DIR/test-helpers.sh"
|
||||
|
||||
echo "========================================"
|
||||
echo " Integration Test: requesting-code-review"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "This test verifies the code reviewer subagent by:"
|
||||
echo " 1. Setting up a tiny project with a baseline commit"
|
||||
echo " 2. Adding a second commit that plants an obvious bug"
|
||||
echo " 3. Dispatching the code reviewer via the requesting-code-review skill"
|
||||
echo " 4. Verifying the reviewer flags the planted bug as Critical/Important"
|
||||
echo ""
|
||||
|
||||
TEST_PROJECT=$(create_test_project)
|
||||
echo "Test project: $TEST_PROJECT"
|
||||
trap "cleanup_test_project $TEST_PROJECT" EXIT
|
||||
|
||||
cd "$TEST_PROJECT"
|
||||
|
||||
# Baseline: a small "safe" implementation
|
||||
mkdir -p src
|
||||
cat > src/db.js <<'EOF'
|
||||
import { Database } from "./database-driver.js";
|
||||
|
||||
const db = new Database();
|
||||
|
||||
export async function findUserByEmail(email) {
|
||||
if (typeof email !== "string" || !email) {
|
||||
throw new Error("email required");
|
||||
}
|
||||
return db.query(
|
||||
"SELECT id, email, created_at FROM users WHERE email = ?",
|
||||
[email],
|
||||
);
|
||||
}
|
||||
EOF
|
||||
|
||||
cat > package.json <<'EOF'
|
||||
{ "name": "test-codereview", "version": "1.0.0", "type": "module" }
|
||||
EOF
|
||||
|
||||
git init --quiet
|
||||
git config user.email "test@test.com"
|
||||
git config user.name "Test User"
|
||||
git add .
|
||||
git commit -m "Initial: parameterized findUserByEmail" --quiet
|
||||
BASE_SHA=$(git rev-parse HEAD)
|
||||
|
||||
# Second commit: plant two real bugs
|
||||
# 1. SQL injection — switch from parameterized to string concatenation
|
||||
# 2. Logs the user's password hash on every successful login
|
||||
cat > src/db.js <<'EOF'
|
||||
import { Database } from "./database-driver.js";
|
||||
|
||||
const db = new Database();
|
||||
|
||||
export async function findUserByEmail(email) {
|
||||
return db.query(
|
||||
"SELECT id, email, password_hash, created_at FROM users WHERE email = '" + email + "'",
|
||||
);
|
||||
}
|
||||
|
||||
export async function login(email, password) {
|
||||
const user = await findUserByEmail(email);
|
||||
if (user && user.password_hash === hash(password)) {
|
||||
console.log("login success", { email, password_hash: user.password_hash });
|
||||
return user;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function hash(s) { return s; }
|
||||
EOF
|
||||
|
||||
git add .
|
||||
git commit -m "Refactor user lookup, add login" --quiet
|
||||
HEAD_SHA=$(git rev-parse HEAD)
|
||||
|
||||
echo ""
|
||||
echo "Planted bugs in $BASE_SHA..$HEAD_SHA:"
|
||||
echo " - SQL injection (string concat instead of parameterized query)"
|
||||
echo " - Password hash logged in plaintext on every successful login"
|
||||
echo " - hash() is the identity function (passwords stored & compared in plaintext)"
|
||||
echo ""
|
||||
|
||||
OUTPUT_FILE="$TEST_PROJECT/claude-output.txt"
|
||||
|
||||
PROMPT="I just finished a refactor. The change is between commits $BASE_SHA and $HEAD_SHA on the current branch.
|
||||
|
||||
Use the superpowers:requesting-code-review skill to review these changes before I merge. Follow the skill exactly: dispatch the code reviewer subagent with the template, give the subagent the SHA range, and report back what it found.
|
||||
|
||||
Print the reviewer's full output."
|
||||
|
||||
# Run claude from inside the test project so its session JSONL lands in a
|
||||
# project-specific directory under ~/.claude/projects/, isolated from any
|
||||
# other concurrent claude sessions.
|
||||
echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..."
|
||||
echo "================================================================================"
|
||||
cd "$TEST_PROJECT" && timeout 600 claude -p "$PROMPT" \
|
||||
--plugin-dir "$PLUGIN_DIR" \
|
||||
--permission-mode bypassPermissions 2>&1 | tee "$OUTPUT_FILE" || {
|
||||
echo ""
|
||||
echo "================================================================================"
|
||||
echo "EXECUTION FAILED (exit code: $?)"
|
||||
exit 1
|
||||
}
|
||||
echo "================================================================================"
|
||||
|
||||
echo ""
|
||||
echo "Analyzing reviewer output..."
|
||||
echo ""
|
||||
|
||||
# Find the session transcript. Because we ran claude from $TEST_PROJECT (a
|
||||
# unique tmp dir), its sessions live in their own ~/.claude/projects/ folder.
|
||||
# Resolve the real path (macOS mktemp returns /var/... but claude normalizes
|
||||
# it to /private/var/...) and replicate claude's normalization (every
|
||||
# non-alphanumeric char becomes `-`).
|
||||
TEST_PROJECT_REAL=$(cd "$TEST_PROJECT" && pwd -P)
|
||||
SESSION_DIR="$HOME/.claude/projects/$(echo "$TEST_PROJECT_REAL" | sed 's|[^a-zA-Z0-9]|-|g')"
|
||||
# `|| true` prevents pipefail killing the script if ls gets SIGPIPE'd by head.
|
||||
SESSION_FILE=$(ls -t "$SESSION_DIR"/*.jsonl 2>/dev/null | head -1 || true)
|
||||
|
||||
FAILED=0
|
||||
|
||||
echo "=== Verification Tests ==="
|
||||
echo ""
|
||||
|
||||
# Test 1: Skill was actually invoked, and a subagent was actually dispatched
|
||||
echo "Test 1: requesting-code-review skill invoked + reviewer subagent dispatched..."
|
||||
if [ -z "$SESSION_FILE" ] || [ ! -f "$SESSION_FILE" ]; then
|
||||
echo " [FAIL] Could not locate session transcript in $SESSION_DIR"
|
||||
FAILED=$((FAILED + 1))
|
||||
elif ! grep -q '"skill":"superpowers:requesting-code-review"' "$SESSION_FILE"; then
|
||||
echo " [FAIL] requesting-code-review skill was not invoked"
|
||||
echo " Session: $SESSION_FILE"
|
||||
FAILED=$((FAILED + 1))
|
||||
elif ! grep -q '"name":"Agent"' "$SESSION_FILE"; then
|
||||
echo " [FAIL] Skill ran but no subagent was dispatched"
|
||||
FAILED=$((FAILED + 1))
|
||||
else
|
||||
echo " [PASS] Skill invoked and subagent dispatched"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 2: Reviewer caught the SQL injection
|
||||
echo "Test 2: SQL injection flagged..."
|
||||
if grep -qiE "sql injection|injection|string concat|parameterize|prepared statement|sanitiz" "$OUTPUT_FILE"; then
|
||||
echo " [PASS] Reviewer flagged the SQL injection vector"
|
||||
else
|
||||
echo " [FAIL] Reviewer missed the SQL injection — most obvious planted bug"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 3: Reviewer caught the credential / password issue (either logging or no real hashing)
|
||||
echo "Test 3: Credential handling issue flagged..."
|
||||
if grep -qiE "password|credential|secret|plaintext|log.*hash|hash.*log|sensitive" "$OUTPUT_FILE"; then
|
||||
echo " [PASS] Reviewer flagged a credential / password handling issue"
|
||||
else
|
||||
echo " [FAIL] Reviewer missed the password/credential issues"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 4: Reviewer marked at least one issue as Critical or Important (not just Minor)
|
||||
echo "Test 4: Severity classification..."
|
||||
if grep -qiE "critical|important|severe|high.*risk|security" "$OUTPUT_FILE"; then
|
||||
echo " [PASS] Reviewer classified findings at Critical/Important severity"
|
||||
else
|
||||
echo " [FAIL] Reviewer did not classify findings as Critical or Important"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 5: Reviewer did NOT approve the diff for merge
|
||||
echo "Test 5: Reviewer verdict..."
|
||||
# A correct reviewer says No or "With fixes". A broken/sycophantic reviewer says Yes/Ready.
|
||||
if grep -qiE "ready to merge.*yes|approved.*for merge|^\s*yes\s*$|safe to merge" "$OUTPUT_FILE" \
|
||||
&& ! grep -qiE "ready to merge.*no|with fixes|do not merge|not ready|block.*merge" "$OUTPUT_FILE"; then
|
||||
echo " [FAIL] Reviewer approved a diff with planted Critical bugs"
|
||||
FAILED=$((FAILED + 1))
|
||||
else
|
||||
echo " [PASS] Reviewer did not approve the diff"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "========================================"
|
||||
echo " Test Summary"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
|
||||
if [ $FAILED -eq 0 ]; then
|
||||
echo "STATUS: PASSED"
|
||||
echo "The code reviewer correctly:"
|
||||
echo " ✓ Was dispatched via the requesting-code-review skill"
|
||||
echo " ✓ Flagged the SQL injection"
|
||||
echo " ✓ Flagged the credential handling issues"
|
||||
echo " ✓ Classified findings at Critical/Important severity"
|
||||
echo " ✓ Did not approve the diff for merge"
|
||||
exit 0
|
||||
else
|
||||
echo "STATUS: FAILED"
|
||||
echo "Failed $FAILED verification tests"
|
||||
echo ""
|
||||
echo "Output saved to: $OUTPUT_FILE"
|
||||
exit 1
|
||||
fi
|
||||
@@ -135,8 +135,7 @@ EOF
|
||||
|
||||
# Note: We use a longer timeout since this is integration testing
|
||||
# Use --allowed-tools to enable tool usage in headless mode
|
||||
# IMPORTANT: Run from superpowers directory so local dev skills are available
|
||||
PROMPT="Change to directory $TEST_PROJECT and then execute the implementation plan at docs/superpowers/plans/implementation-plan.md using the subagent-driven-development skill.
|
||||
PROMPT="Execute the implementation plan at docs/superpowers/plans/implementation-plan.md using the subagent-driven-development skill.
|
||||
|
||||
IMPORTANT: Follow the skill exactly. I will be verifying that you:
|
||||
1. Read the plan once at the beginning
|
||||
@@ -147,9 +146,14 @@ IMPORTANT: Follow the skill exactly. I will be verifying that you:
|
||||
|
||||
Begin now. Execute the plan."
|
||||
|
||||
echo "Running Claude (output will be shown below and saved to $OUTPUT_FILE)..."
|
||||
PLUGIN_DIR=$(cd "$SCRIPT_DIR/../.." && pwd)
|
||||
|
||||
# Run claude from inside the test project so its session JSONL lands in a
|
||||
# project-specific directory under ~/.claude/projects/, isolated from any
|
||||
# other concurrent claude sessions.
|
||||
echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..."
|
||||
echo "================================================================================"
|
||||
cd "$SCRIPT_DIR/../.." && timeout 1800 claude -p "$PROMPT" --allowed-tools=all --add-dir "$TEST_PROJECT" --permission-mode bypassPermissions 2>&1 | tee "$OUTPUT_FILE" || {
|
||||
cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions 2>&1 | tee "$OUTPUT_FILE" || {
|
||||
echo ""
|
||||
echo "================================================================================"
|
||||
echo "EXECUTION FAILED (exit code: $?)"
|
||||
@@ -161,13 +165,17 @@ echo ""
|
||||
echo "Execution complete. Analyzing results..."
|
||||
echo ""
|
||||
|
||||
# Find the session transcript
|
||||
# Session files are in ~/.claude/projects/-<working-dir>/<session-id>.jsonl
|
||||
WORKING_DIR_ESCAPED=$(echo "$SCRIPT_DIR/../.." | sed 's/\//-/g' | sed 's/^-//')
|
||||
SESSION_DIR="$HOME/.claude/projects/$WORKING_DIR_ESCAPED"
|
||||
|
||||
# Find the most recent session file (created during this test run)
|
||||
SESSION_FILE=$(find "$SESSION_DIR" -name "*.jsonl" -type f -mmin -60 2>/dev/null | sort -r | head -1)
|
||||
# Find the session transcript. Because we ran claude from $TEST_PROJECT (a
|
||||
# unique tmp dir), its sessions live in their own ~/.claude/projects/ folder
|
||||
# and we can pick the most-recent one without racing other concurrent sessions.
|
||||
# Resolve the real path because macOS mktemp returns /var/... but claude
|
||||
# normalizes it to /private/var/... when naming the project dir.
|
||||
TEST_PROJECT_REAL=$(cd "$TEST_PROJECT" && pwd -P)
|
||||
# Claude normalizes the cwd to a directory name by replacing every non-alphanumeric
|
||||
# character with `-` (so `_`, `.`, `/` all become `-`).
|
||||
SESSION_DIR="$HOME/.claude/projects/$(echo "$TEST_PROJECT_REAL" | sed 's|[^a-zA-Z0-9]|-|g')"
|
||||
# `|| true` prevents pipefail killing the script if ls gets SIGPIPE'd by head.
|
||||
SESSION_FILE=$(ls -t "$SESSION_DIR"/*.jsonl 2>/dev/null | head -1 || true)
|
||||
|
||||
if [ -z "$SESSION_FILE" ]; then
|
||||
echo "ERROR: Could not find session transcript file"
|
||||
@@ -194,9 +202,9 @@ else
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 2: Subagents were used (Task tool)
|
||||
# Test 2: Subagents were used (Agent / Task tool — name varies by harness version)
|
||||
echo "Test 2: Subagents dispatched..."
|
||||
task_count=$(grep -c '"name":"Task"' "$SESSION_FILE" || echo "0")
|
||||
task_count=$(grep -cE '"name":"(Agent|Task)"' "$SESSION_FILE" || echo "0")
|
||||
if [ "$task_count" -ge 2 ]; then
|
||||
echo " [PASS] $task_count subagents dispatched"
|
||||
else
|
||||
|
||||
Reference in New Issue
Block a user