Agent skill

evaluate-attempt

This SOP evaluates a completed brazil-bench attempt against the spec.md requirements, capturing metrics for comparison across orchestration patterns. Supports Python and Swift/iOS projects.

View SKILL.md on GitHub Repository
Stars 163
Forks 31

Install this agent skill to your Project

npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/other/other/evaluate-attempt

SKILL.md

Evaluate Benchmark Attempt

Overview

This SOP evaluates a completed brazil-bench attempt against the spec.md requirements, capturing metrics for comparison across orchestration patterns. Supports both Python and Swift/iOS implementations.

Parameters

  • attempt_repo (required): Repository name (e.g., attempt-3)
  • output_dir (optional, default: ./results): Where to write evaluation results

Steps

0. Detect Project Language

Identify the primary language/platform of the implementation.

Detection Commands:

bash
cd ./reviews/{attempt_repo}

# Python indicators
ls pyproject.toml setup.py requirements.txt 2>/dev/null

# Swift/iOS indicators
ls Package.swift *.xcodeproj *.xcworkspace 2>/dev/null

# Check file extensions
find . -name "*.py" -not -path "./.venv/*" | head -5
find . -name "*.swift" | head -5

Language Detection Matrix:

Files Found Language Test Framework
pyproject.toml, *.py Python pytest
Package.swift, *.swift Swift Package swift test
*.xcodeproj, *.swift iOS/Xcode xcodebuild test
Both Python and Swift Multi-language Run both

Constraints:

  • You MUST detect the language before running tests
  • You MUST use appropriate commands for the detected language
  • You SHOULD note the detected language in the report

1. Clone Attempt

Fetch the attempt repository for local analysis.

Constraints:

  • You MUST clone into ./reviews/{attempt_repo}
  • You MUST verify the clone succeeded before proceeding
  • You MUST NOT modify any files in the cloned repo
bash
gh repo clone brazil-bench/{attempt_repo} ./reviews/{attempt_repo}

2. Verify Spec Integrity

Confirm the spec.md was not modified from the template.

Constraints:

  • You MUST compare spec.md against the template version
  • You MUST fail the evaluation if spec.md was modified
  • You SHOULD use a checksum comparison
bash
gh repo clone brazil-bench/benchmark-template ./reviews/_template --depth 1
diff ./reviews/{attempt_repo}/spec.md ./reviews/_template/spec.md

3. Run Conformance Tests

Execute the test suite defined in the spec against the implementation.

Constraints:

  • You MUST attempt to run all tests specified in spec.md
  • You MUST capture pass/fail counts and output
  • You SHOULD timeout tests after 60 seconds each
  • You MAY retry flaky tests once
  • If tests fail due to missing dependencies, follow the dependency resolution steps below

Python Test Commands

bash
cd ./reviews/{attempt_repo}

# Run pytest with verbose output
pytest --tb=short -v 2>&1 | tee test_output.log

# Get summary counts
pytest --tb=no -q 2>&1 | tail -5

Swift/iOS Test Commands

bash
cd ./reviews/{attempt_repo}

# Swift Package Manager
swift test 2>&1 | tee test_output.log

# Xcode project (iOS Simulator)
xcodebuild test \
    -project *.xcodeproj \
    -scheme "YourScheme" \
    -destination 'platform=iOS Simulator,name=iPhone 15' \
    2>&1 | tee test_output.log

# Parse xcodebuild results
grep -E "(Test Case|passed|failed)" test_output.log

# Using xcpretty for cleaner output (if available)
xcodebuild test -project *.xcodeproj -scheme "YourScheme" \
    -destination 'platform=iOS Simulator,name=iPhone 15' \
    | xcpretty --report junit

3a. Handle Missing Dependencies (Neo4j, etc.)

If tests fail due to missing external dependencies like Neo4j:

Step 1: Try to start the dependency via Docker

bash
# Check if Docker is available
docker --version

# Check for docker-compose files in the repo
ls ./reviews/{attempt_repo}/docker-compose*.yml

# If Neo4j docker-compose exists, start it
docker-compose -f ./reviews/{attempt_repo}/docker-compose.neo4j.yml up -d

# Or start Neo4j directly
docker run -d --name neo4j-eval -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/password \
  neo4j:5

# Wait for Neo4j to be ready
sleep 10
docker logs neo4j-eval 2>&1 | tail -5

Step 2: If Docker unavailable or fails, look for evidence of prior test runs

Check these sources for test results:

bash
# Check git history for test-related commits
git log --oneline --all | grep -iE "(test|pass|100%|fix.*test)"

# Check for CI/CD logs or badges
cat ./reviews/{attempt_repo}/README.md | grep -iE "(pass|badge|ci|test)"

# Check prompts.txt for test execution evidence
cat ./reviews/{attempt_repo}/prompts.txt 2>/dev/null | grep -iE "(pytest|test|pass|fail|scenario)"

# Check for pytest cache with results
ls -la ./reviews/{attempt_repo}/.pytest_cache/ 2>/dev/null

# Check for coverage reports
ls -la ./reviews/{attempt_repo}/htmlcov/ ./reviews/{attempt_repo}/coverage.xml 2>/dev/null

Step 3: Document findings in the report

If tests cannot be run directly, document:

  • Why tests couldn't run (missing Neo4j, etc.)
  • Evidence found of prior test runs (commit messages, prompts.txt entries)
  • Claimed test results from the attempt's documentation
  • Mark as "CANNOT VERIFY" with explanation

Constraints for dependency handling:

  • You MUST try Docker first if available
  • You MUST search for evidence if Docker fails
  • You MUST NOT claim tests pass without verification
  • You SHOULD note the source of any claimed test results
  • You SHOULD clean up Docker containers after evaluation: docker stop neo4j-eval && docker rm neo4j-eval

3b. Detect Skipped Tests

Skipped tests inflate test counts without providing actual verification. You MUST detect and report them separately.

Python: Detect Skipped Tests

Step 1: Run pytest with verbose output to capture skipped tests

bash
cd ./reviews/{attempt_repo}

# Run pytest and capture skip count
pytest --tb=no -v 2>&1 | grep -E "(PASSED|FAILED|SKIPPED|ERROR)" | head -100

# Get summary counts
pytest --tb=no -q 2>&1 | tail -5

# Look for skip patterns in test files
grep -r "pytest.skip\|@pytest.mark.skip\|skipif\|xfail" tests/ --include="*.py"

Step 2: Analyze test files for skip patterns

bash
# Count tests that call pytest.skip() inside the test body (worst pattern)
grep -r "pytest.skip(" tests/ --include="*.py" -l | wc -l

# Count tests with @pytest.mark.skip decorator
grep -r "@pytest.mark.skip" tests/ --include="*.py" | wc -l

# Count conditional skips (skipif)
grep -r "@pytest.mark.skipif" tests/ --include="*.py" | wc -l
Swift/iOS: Detect Skipped Tests

Step 1: Run swift test or xcodebuild and capture skipped tests

bash
cd ./reviews/{attempt_repo}

# Swift Package Manager - look for skipped in output
swift test 2>&1 | grep -E "(passed|failed|skipped)"

# Xcode - parse test results
xcodebuild test -project *.xcodeproj -scheme "YourScheme" \
    -destination 'platform=iOS Simulator,name=iPhone 15' \
    2>&1 | grep -E "Test Case.*passed|Test Case.*failed|skipped"

Step 2: Analyze test files for skip patterns

bash
# Count XCTSkip usage (explicit skips)
grep -r "XCTSkip\|throw XCTSkip" Tests/ --include="*.swift" | wc -l

# Count disabled tests (func name doesn't start with test)
grep -r "func disabled_test\|// func test" Tests/ --include="*.swift" | wc -l

# Count tests with availability checks that skip
grep -r "@available\|#available" Tests/ --include="*.swift" -A 2 | grep -i skip | wc -l

# Look for conditional test execution
grep -r "guard.*else.*return\|if.*XCTSkip" Tests/ --include="*.swift" | wc -l

Swift Skip Patterns:

Pattern Type Assessment
throw XCTSkip("reason") Explicit skip Acceptable if documented
#if !targetEnvironment(simulator) Conditional Acceptable for device-only
@available(iOS 16, *) Version skip Acceptable
Renamed to disabled_testFoo Hidden skip Should be penalized
Empty test body Stub Should be penalized

Step 3: Calculate effective test count

Metric How to Calculate
Total Tests Number of test functions defined
Passed Tests Tests that ran and passed
Skipped Tests Tests marked skip or calling pytest.skip()
Effective Tests Total - Skipped (tests that actually run)
Skip Ratio Skipped / Total (percentage of tests that skip)

Constraints for skipped test handling:

  • You MUST report skipped tests separately from passed tests
  • You MUST calculate the "effective test count" (passed + failed, excluding skipped)
  • You MUST flag ANY skipped tests for issue filing - zero tolerance for skips
  • You MUST distinguish between skip types for the issue description:
    • Conditional skips (@pytest.mark.skipif): Document reason in issue
    • Unconditional skips (pytest.skip() in body): Critical - tests never run
    • Decorator skips (@pytest.mark.skip): Document reason in issue
  • You MUST NOT count skipped tests toward the test score in rankings
  • You MUST file an issue for ANY skipped test (no acceptable skip threshold)

Example Analysis:

Total tests:     59
Passed:          44
Skipped:         15  (25% skip ratio - HIGH)
Failed:          0
Effective:       44  (use this for scoring, not 59)

Skip breakdown:
- pytest.skip() in body: 15 (integration tests that never run)
- @pytest.mark.skipif: 0
- @pytest.mark.skip: 0

Flag: INFLATED TEST COUNT - 15 tests skip unconditionally

Document in Report:

markdown
## Test Results

| Metric | Count |
|--------|-------|
| Total Tests | 59 |
| Passed | 44 |
| **Skipped** | **15** |
| Failed | 0 |
| **Effective Tests** | **44** |
| Skip Ratio | 25% |

⚠️ **Warning:** 15 tests (25%) are skipped and never execute.
These are integration tests that call `pytest.skip()` inside the test body.
The effective test count for scoring is 44, not 59.

3c. Self-Contained Integration Tests (REQUIRED)

Integration tests MUST be self-contained and actually run. Tests that skip because "Neo4j not available" or similar are not acceptable.

Requirement: Integration tests must start their own data stores as needed.

Detection Commands:

bash
cd ./reviews/{attempt_repo}

# Check for testcontainers usage (Python)
grep -r "testcontainers\|TestContainer\|DockerContainer" tests/ --include="*.py"

# Check for docker-compose in test setup
grep -r "docker-compose\|subprocess.*docker" tests/ --include="*.py"

# Check for pytest-docker fixture
grep -r "pytest-docker\|docker_compose" tests/ --include="*.py" pyproject.toml

# Check for in-memory alternatives (e.g., SQLite instead of Postgres)
grep -r "sqlite.*memory\|:memory:\|MockNeo4j\|FakeNeo4j" tests/ --include="*.py"

# Check for conftest fixtures that start services
grep -A 20 "@pytest.fixture" tests/conftest.py 2>/dev/null | grep -E "docker\|container\|start\|neo4j"

# Swift: Check for test containers
grep -r "Docker\|Container\|TestServer" Tests/ --include="*.swift"

Acceptable Patterns for Self-Contained Tests:

Pattern Example Assessment
testcontainers Neo4jContainer() in fixture ✓ Best - automatic lifecycle
pytest-docker docker_compose_file fixture ✓ Good - compose-based
conftest startup Fixture runs docker run neo4j ✓ Acceptable - manual but works
In-memory mock MockNeo4jClient class ✗ NOT acceptable - not persistent
External dependency pytest.skip("Neo4j not running") ✗ NOT acceptable
CI-only tests @pytest.mark.skipif(not CI) ✗ NOT acceptable
No integration tests No tests for data layer ✗ NOT acceptable

Example: testcontainers Pattern (Python)

python
# conftest.py
import pytest
from testcontainers.neo4j import Neo4jContainer

@pytest.fixture(scope="session")
def neo4j_container():
    """Start Neo4j container for integration tests."""
    with Neo4jContainer("neo4j:5") as neo4j:
        yield neo4j

@pytest.fixture
def neo4j_client(neo4j_container):
    """Get client connected to test container."""
    return Neo4jClient(
        uri=neo4j_container.get_connection_url(),
        auth=("neo4j", "password")
    )

Example: pytest-docker Pattern

python
# conftest.py
import pytest

@pytest.fixture(scope="session")
def docker_compose_file():
    return "docker-compose.test.yml"

@pytest.fixture(scope="session")
def neo4j_service(docker_services):
    """Wait for Neo4j to be ready."""
    docker_services.wait_until_responsive(
        timeout=30.0,
        pause=0.5,
        check=lambda: is_neo4j_ready()
    )

Scoring Impact:

Integration Test Quality Score Modifier
Self-contained (testcontainers/docker) No penalty
In-memory mock (not persistent) -10 points quality
Skips due to missing dependency -10 points quality
No integration tests at all -15 points quality

Constraints:

  • You MUST check if integration tests are self-contained
  • You MUST flag tests that skip due to external dependencies
  • You MUST NOT accept "works on CI" as justification for skipping locally
  • You SHOULD recommend testcontainers or pytest-docker patterns
  • You SHOULD verify integration tests actually execute (not just exist)

Document in Report:

markdown
## Integration Test Quality

| Aspect | Status |
|--------|--------|
| Self-contained | Yes/No |
| Data store management | testcontainers / docker-compose / mock / external |
| Integration tests run | X passed, Y skipped |

⚠️ **Issue:** Integration tests skip when Neo4j is not running.
Tests should use testcontainers or pytest-docker to manage dependencies.

3d. Context Header Blocks (REQUIRED)

Every source code file MUST have a context header comment block that documents:

  1. Purpose - What the file/module does
  2. Interfaces - Key classes, functions, or APIs exposed
  3. Change History - Record of modifications (updated on every change)

Detection Commands:

bash
cd ./reviews/{attempt_repo}

# Python: Check for docstrings or header comments in source files
for f in $(find src -name "*.py" -not -name "__init__.py"); do
  echo "=== $f ==="
  head -50 "$f" | grep -E '""".*|^#.*Purpose|^#.*Context|CONTEXT BLOCK|Change History|Interfaces'
done

# Swift: Check for header comments
for f in $(find Sources -name "*.swift" 2>/dev/null); do
  echo "=== $f ==="
  head -50 "$f" | grep -E '///|/\*\*|Purpose|Context|History'
done

# Count files with context headers vs total
total=$(find src -name "*.py" -not -name "__init__.py" | wc -l)
with_header=$(find src -name "*.py" -not -name "__init__.py" -exec head -30 {} \; -exec echo "---" \; | grep -l "CONTEXT\|Purpose\|Module:" | wc -l)
echo "Files with headers: $with_header / $total"

Required Header Format (Python):

python
"""
================================================================================
CONTEXT BLOCK
================================================================================
File: {filename}
Module: {module.path}
Purpose: {one-line description}

Description:
    {detailed description of what this module does}

Interfaces:
    - {ClassName}: {brief description}
    - {function_name}(): {brief description}

Dependencies:
    - {module}: {why needed}

Change History:
    - {date}: {description of change}
    - {date}: Initial creation
================================================================================
"""

Required Header Format (Swift):

swift
//
//  {FileName}.swift
//  {ProjectName}
//
//  Purpose: {one-line description}
//
//  Interfaces:
//    - {ClassName}: {brief description}
//    - {functionName}(): {brief description}
//
//  Change History:
//    - {date}: {description of change}
//    - {date}: Initial creation
//

Assessment Criteria:

Coverage Assessment Score Impact
100% files have headers Excellent No penalty
75-99% files have headers Good -2 quality
50-74% files have headers Partial -5 quality
<50% files have headers Poor -10 quality

Constraints:

  • You MUST check all source files for context headers
  • You MUST verify headers include purpose, interfaces, and change history
  • You MUST flag files missing headers for issue filing
  • You SHOULD note which files have incomplete headers (missing sections)

Document in Report:

markdown
## Context Header Compliance

| Metric | Count |
|--------|-------|
| Source files | X |
| With headers | Y |
| Coverage | Z% |

### Files Missing Headers
- `src/module.py` - No header
- `src/utils.py` - Missing change history

### Assessment
{Excellent/Good/Partial/Poor} - {X}% coverage

4. Measure Code Metrics

Collect quantitative data about the implementation.

Constraints:

  • You MUST capture: total lines of code, number of files, dependencies
  • You SHOULD capture: cyclomatic complexity, test coverage
  • You MAY capture: documentation coverage, type hint coverage

Python Metrics

bash
# Lines of code (excluding tests)
find ./reviews/{attempt_repo}/src -name "*.py" | xargs wc -l

# Dependencies
cat ./reviews/{attempt_repo}/pyproject.toml | grep dependencies -A 50

# File count
find ./reviews/{attempt_repo}/src -name "*.py" | wc -l

Swift/iOS Metrics

bash
# Lines of code (excluding tests)
find ./reviews/{attempt_repo}/Sources -name "*.swift" | xargs wc -l

# For Xcode projects
find ./reviews/{attempt_repo} -name "*.swift" -not -path "*/Tests/*" -not -path "*Test*" | xargs wc -l

# Dependencies (Swift Package Manager)
cat ./reviews/{attempt_repo}/Package.swift | grep -A 50 "dependencies:"

# Dependencies (CocoaPods)
cat ./reviews/{attempt_repo}/Podfile 2>/dev/null

# Dependencies (Xcode project - SPM)
grep -r "repositoryURL" ./reviews/{attempt_repo}/*.xcodeproj/project.pbxproj 2>/dev/null | head -20

# File count
find ./reviews/{attempt_repo}/Sources -name "*.swift" | wc -l

# Check for SwiftLint configuration
ls ./reviews/{attempt_repo}/.swiftlint.yml 2>/dev/null

5. Extract Git Metrics and Analyze Development Timeline

Analyze the development history to separate agent-driven work from human interactions.

Constraints:

  • You MUST capture: total commits, time from first to last commit
  • You MUST separate commits into agent-driven vs human-driven phases
  • You MUST calculate autonomous duration (agent work only)
  • You SHOULD capture: number of reverts, force pushes (if detectable)
  • You SHOULD extract commit messages mentioning "fix", "revert", "oops"

5a. Gather Raw Git Data

bash
cd ./reviews/{attempt_repo}

# Full commit history with timestamps and messages
git log --format="%ai | %H | %s" --reverse

# Count total commits
git log --oneline | wc -l

# Find fix/revert commits
git log --format="%H %s" | grep -iE "(fix|revert|oops|wrong)"

# Get first and last commit times
git log --format="%ai" --reverse | head -1  # First commit
git log --format="%ai" | head -1             # Last commit

5b. Identify Development Phases

Analyze commit timestamps and messages to identify distinct phases:

Phase 1: Setup (Human)

  • Initial commit, repo setup, file uploads
  • Typically first 1-3 commits before implementation starts
  • Look for: "Initial commit", "Add files", "upload", "setup"

Phase 2: Agent Implementation

  • Bulk implementation work by the agent
  • Characterized by:
    • Rapid succession of commits (minutes apart)
    • Large code changes
    • Messages like "Implement", "Add", "Create"
    • Consistent commit patterns (same author, similar timing)

Phase 3: Agent Test Iteration

  • Test fixing and iteration by the agent
  • Characterized by:
    • Commits mentioning "fix", "test", "pass"
    • Still rapid succession
    • Often shows progression: "Fix X" → "Fix Y" → "100% pass"

Phase 4: Human Intervention (Post-Completion)

  • Human-driven changes after agent work completes
  • Characterized by:
    • Time gaps (hours/days after previous commits)
    • Different commit patterns or author info
    • Messages about data, documentation, cleanup
    • Changes not required by the spec

5c. Heuristics for Identifying Agent vs Human Commits

Agent commits typically show:

  • Timestamps within minutes of each other
  • Consistent formatting in commit messages
  • Co-authored-by lines mentioning Claude/AI
  • Large, comprehensive changes
  • Focus on implementation and tests

Human commits typically show:

  • Time gaps of hours or days from previous work
  • Different commit message style
  • Focus on data, docs, or polish
  • Smaller, targeted changes
  • Work done after "100% tests pass" milestone
bash
# Look for time gaps > 1 hour between commits (potential phase boundaries)
git log --format="%ai" --reverse | while read ts; do echo "$ts"; done

# Check for co-author lines indicating AI
git log --format="%b" | grep -i "co-authored"

# Check prompts.txt for session boundaries
cat prompts.txt 2>/dev/null | grep -E "^(Done|Session|Agent)"

5d. Calculate Duration Metrics

Metric How to Calculate
Total Duration Last commit - First commit
Agent Duration Sum of time during agent phases only
Human Duration Sum of time during human phases
Autonomous Duration Phase 2 + Phase 3 (implementation + test fixing)

Example Timeline Analysis:

09:00:00 - Initial commit (Human Setup)
09:05:00 - Add spec file (Human Setup)
         --- Agent work begins ---
09:15:00 - Implement Phase 1 (Agent)
09:45:00 - Implement Phase 2 (Agent)
10:10:00 - Implement Phase 3 (Agent)
10:25:00 - Fix test issues (Agent)
10:40:00 - 100% tests pass (Agent)
         --- Agent work ends ---
         --- 2 day gap ---
Oct 3     - Add real data (Human)
Oct 3     - Update docs (Human)

Agent Duration: ~1h 25m (09:15 → 10:40)
Human Duration: ~5m setup + later changes
Autonomous Duration: ~1h 25m

5e. Document in Report

Include a Development Timeline section in the report:

markdown
## Development Duration Breakdown

| Phase | Duration | Description |
|-------|----------|-------------|
| **Setup (Human)** | ~5 min | Initial commit, file upload |
| **Phase 1: Implementation** | ~55 min | Agent implements all phases |
| **Phase 2: Test Fixing** | ~30 min | Agent iterates to 100% pass |
| **Total Autonomous** | **~1h 25m** | Agent work only |
| **Phase 3: Human Intervention** | 2 days later | Data and docs added |

### Commit Analysis
- Total commits: 15
- Agent commits: 10 (09:15 - 10:40 on Day 1)
- Human commits: 5 (setup + Day 3 changes)
- Fix commits: 3 (normal iteration, not rework)

6. Analyze Against Spec

Review implementation completeness against spec.md requirements.

Constraints:

  • You MUST evaluate against ALL 16 canonical requirements listed below
  • You MUST assess each as: implemented, partial, missing
  • You SHOULD note implementation approach for each
  • You MUST NOT make subjective quality judgments beyond spec compliance
  • You MUST use the exact requirement numbering for cross-attempt consistency

6.0 Canonical Requirements Checklist (16 Requirements)

All evaluations MUST use this exact checklist to ensure consistency across attempts.

Functional Requirements (6):

  1. [FR-1] Search and return match data from all CSV files
  2. [FR-2] Search and return player data
  3. [FR-3] Calculate basic statistics (wins, losses, goals)
  4. [FR-4] Compare teams head-to-head
  5. [FR-5] Handle team name variations correctly
  6. [FR-6] Return properly formatted responses

Query Performance (3): 7. [QP-1] Simple lookups respond in < 2 seconds 8. [QP-2] Aggregate queries respond in < 5 seconds 9. [QP-3] No timeout errors

Data Coverage (3): 10. [DC-1] All 6 CSV files are loadable and queryable 11. [DC-2] At least 20 sample questions can be answered 12. [DC-3] Cross-file queries work (player + match data)

Technical Requirements (4): 13. [TR-1] MCP server implementation with callable tools 14. [TR-2] BDD testing with Given-When-Then structure 15. [TR-3] UTF-8 encoding support (Portuguese characters: ã, ç, é, etc.) 16. [TR-4] Multiple date format handling (ISO, Brazilian DD/MM/YYYY, with time)

Report Format for Requirements:

markdown
## Requirements Checklist

### Functional Requirements (X/6)
- [x] [FR-1] Search and return match data from all CSV files
- [x] [FR-2] Search and return player data
- [ ] [FR-3] Calculate basic statistics (partial: missing draws)
...

### Query Performance (X/3)
- [x] [QP-1] Simple lookups respond in < 2 seconds
...

### Data Coverage (X/3)
- [x] [DC-1] All 6 CSV files are loadable and queryable
...

### Technical Requirements (X/4)
- [x] [TR-1] MCP server implementation with callable tools
- [x] [TR-2] BDD testing with Given-When-Then structure
...

**Total: X/16 requirements implemented**

6a. Real Data vs Simulated Data Assessment

Determine whether the implementation uses real external data or simulated/mock data.

Real Data Indicators:

  • Data loaders for external sources (Kaggle, APIs, etc.)
  • CSV/JSON files in data directory
  • API client code with authentication
  • Data normalization/mapping logic for external schemas

Simulated Data Indicators:

  • Hardcoded test fixtures
  • Factory/faker-generated data
  • Mock data in test files only
  • No external data loading code

Constraints for Real Data Implementations:

  • You MUST note which external data source is used
  • You MUST assess schema mapping quality (how well does the implementation adapt external schema to spec schema)
  • You MUST distinguish between:
    • Schema Implemented: The code defines models matching spec entities
    • Data Populated: The data loader can populate those fields from external source
    • Not Available in Source: Spec field cannot be populated because external data doesn't include it
  • You SHOULD credit implementations that adapt to real-world data constraints
  • You SHOULD note any enhancements beyond spec (e.g., additional fields from richer data sources)

Adjusted Compliance Scoring:

  • If real data is used and a spec field is "Not Available in Source", count it as:
    • Implemented if the model/schema supports the field
    • Note the data limitation separately
  • Example: If spec requires "attendance" but Kaggle data has no attendance:
    • Check if Match model has attendance field (schema compliance)
    • Note that field would be null with Kaggle data (data limitation)
    • This is NOT a failure - it's a data source constraint

6b. Documentation Quality Assessment

Evaluate the README.md for essential user documentation.

Required Elements:

  1. Setup Instructions: Prerequisites, installation steps, environment configuration
  2. MCP Server Setup: How to start the server, how to connect Claude
  3. Example Q&A: Sample questions and expected responses/output

Extraction Commands:

bash
# Check README content
head -100 ./reviews/{attempt_repo}/README.md

# Look for key documentation sections
grep -E "Quick Start|Installation|Setup|MCP|Example|Usage" ./reviews/{attempt_repo}/README.md

Documentation Quality Levels:

Level Criteria In Report
Excellent All 3 elements + extras (architecture, API ref, troubleshooting) "Comprehensive README"
Good All 3 required elements present "Good documentation"
Acceptable 2 of 3 elements "Partial documentation"
Poor 0-1 elements "Missing documentation"

Best Practice Reference:

  • 2025-10-30-python-hive: Excellent (Quick Start, MCP config, 15+ demo questions, architecture, troubleshooting)
  • 2025-12-15-python-claude-ruvector: Excellent (detailed setup, claude mcp add example, Q&A with output)

Include in Report:

markdown
## Documentation Quality

| Element | Present | Notes |
|---------|---------|-------|
| Setup Instructions | Yes/No | {details} |
| MCP Server Setup | Yes/No | {details} |
| Example Q&A | Yes/No | {details} |

**Assessment:** {Excellent/Good/Acceptable/Poor}

7. Generate Codebase Documentation

Generate comprehensive documentation for the implementation using the codebase-summary SOP.

Constraints:

  • You MUST run the codebase-summary skill on the cloned repository
  • You MUST output documentation to {output_dir}/{attempt_repo}-summary/
  • You SHOULD use the generated documentation to inform the final report
  • The documentation provides architecture, components, interfaces, and workflow analysis
summarize codebase reviews/{attempt_repo} to {output_dir}/{attempt_repo}-summary/

8. Generate Report

Produce structured evaluation output.

Constraints:

  • You MUST write results to {output_dir}/{attempt_repo}.md
  • You MUST include: attempt name, orchestration pattern, all metrics
  • You MUST use consistent format for cross-attempt comparison
  • You SHOULD include raw data as appendix

Output Format

markdown
# Evaluation: {attempt_repo}

## Summary
- **Pattern:** [swarm|hive|solo|...]
- **Spec Compliance:** X/Y requirements
- **Tests:** X passed, Y skipped, Z failed (X effective)
- **Autonomous Duration:** Xh Ym
- **Documentation:** See `{attempt_repo}-summary/`

## Metrics
| Metric | Value |
|--------|-------|
| Lines of Code | |
| Files | |
| Dependencies | |
| Commits (Total) | |
| Commits (Agent) | |
| Commits (Human) | |
| Fix Commits | |
| Tests (Total) | |
| Tests (Passed) | |
| Tests (Skipped) | |
| Tests (Effective) | |
| Skip Ratio | |

## Development Duration Breakdown

| Phase | Duration | Description |
|-------|----------|-------------|
| **Setup (Human)** | | Initial commit, file upload |
| **Agent Implementation** | | Core implementation work |
| **Agent Test Iteration** | | Test fixing to 100% pass |
| **Total Autonomous** | | Agent work only |
| **Human Intervention** | | Post-completion changes |

### Timeline

{timestamp} - {commit message} ({phase}) ...


### Commit Analysis
- Total commits: X
- Agent commits: X (timespan)
- Human commits: X (description)
- Fix commits: X (context: normal iteration vs rework)

## Requirements Checklist
- [x] Requirement 1
- [ ] Requirement 2 (partial: notes)
- [ ] Requirement 3 (missing)

## Architecture Summary
(Key insights from generated codebase documentation)

## Raw Data
...

Troubleshooting

Clone fails

  • Verify repo exists: gh repo view brazil-bench/{attempt_repo}
  • Check permissions: repo must be public or you need access

Tests won't run due to missing dependencies

  • Try starting Neo4j via Docker (see Step 3a above)
  • If Docker unavailable, search for evidence of prior test runs
  • Check git commits for "100% pass" or similar messages
  • Check prompts.txt for pytest output
  • Document as "CANNOT VERIFY" with evidence found

Neo4j connection errors

  • Verify Neo4j is running: docker ps | grep neo4j
  • Check credentials match: NEO4J_AUTH=neo4j/password
  • Wait for startup: Neo4j needs ~10-15 seconds to initialize
  • Check logs: docker logs neo4j-eval

Spec diff shows changes

  • Fail the evaluation
  • Note the changes in the report
  • This invalidates the benchmark comparison

Codebase documentation fails

  • Verify the codebase-summary skill is available
  • Check that the codebase-path exists and contains code
  • Ensure the output directory is writable
  • Try running the skill standalone first to debug

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

majiayu000/claude-skill-registry

agent-ops-spec

Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.

163 31
Explore
majiayu000/claude-skill-registry

agent-ops-state

Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.

163 31
Explore
majiayu000/claude-skill-registry

agent-ops-spec

Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.

163 31
Explore
majiayu000/claude-skill-registry

agent-ops-testing

Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.

163 31
Explore
majiayu000/claude-skill-registry

agent-ops-testing

Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.

163 31
Explore
majiayu000/claude-skill-registry

agent-ops-state

Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.

163 31
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results