Agent skill

code-sample-validator

Extract, validate, and test code samples in documentation. Verify syntax, execute samples, check outputs, validate imports, and ensure code samples are up-to-date with current APIs.

View SKILL.md on GitHub Repository
Stars 514
Forks 31

Install this agent skill to your Project

npx add-skill https://github.com/a5c-ai/babysitter/tree/main/library/specializations/technical-documentation/skills/code-sample-validator

Metadata

Additional technical details for this skill

author
babysitter-sdk
version
1.0.0

SKILL.md

Code Sample Validation Skill

Extract, validate, and test code samples in documentation.

Capabilities

  • Extract code blocks from Markdown/MDX
  • Syntax validation for multiple languages
  • Execute and verify code sample output
  • Snippet compilation testing
  • Version compatibility checking
  • Import/dependency verification
  • Code formatting validation (prettier, black)
  • Generate runnable code from documentation

Usage

Invoke this skill when you need to:

  • Validate code samples in documentation
  • Test code examples execute correctly
  • Check for outdated API usage
  • Verify import statements
  • Generate test files from documentation

Inputs

Parameter Type Required Description
inputPath string Yes Path to documentation file or directory
action string Yes extract, validate, execute, format
languages array No Filter by language (js, python, etc.)
outputDir string No Directory for extracted runnable code
timeout number No Execution timeout in seconds
config object No Language-specific configuration

Input Example

json
{
  "inputPath": "./docs",
  "action": "validate",
  "languages": ["javascript", "python"],
  "timeout": 30
}

Output Structure

Validation Report

json
{
  "summary": {
    "total": 45,
    "passed": 42,
    "failed": 2,
    "skipped": 1
  },
  "files": [
    {
      "file": "docs/quickstart.md",
      "samples": [
        {
          "language": "javascript",
          "line": 15,
          "status": "passed",
          "syntaxValid": true,
          "executionResult": {
            "success": true,
            "output": "Hello, World!",
            "duration": 125
          }
        },
        {
          "language": "python",
          "line": 45,
          "status": "failed",
          "syntaxValid": true,
          "executionResult": {
            "success": false,
            "error": "ModuleNotFoundError: No module named 'requests'",
            "suggestion": "Add 'requests' to test dependencies"
          }
        }
      ]
    }
  ],
  "issues": [
    {
      "file": "docs/api/users.md",
      "line": 78,
      "language": "typescript",
      "issue": "Type error: Property 'user' does not exist on type 'Response'",
      "code": "const name = response.user.name;",
      "suggestion": "Update to use 'response.data.user.name'"
    }
  ]
}

Code Extraction

Markdown Code Block Patterns

markdown
```javascript
// This block will be extracted and validated
const client = new Client({ apiKey: 'test' });
const result = await client.query('Hello');
console.log(result);
```

```python
# This block will be extracted and validated
from mypackage import Client

client = Client(api_key='test')
result = client.query('Hello')
print(result)
```

```bash skip-validation
# This block will be skipped due to directive
echo "Not validated"
```

```javascript title="example.js" runnable
// This block is marked as runnable
export function greet(name) {
  return `Hello, ${name}!`;
}
```

Extraction Configuration

json
{
  "extract": {
    "includeLanguages": ["javascript", "typescript", "python", "go"],
    "excludeLanguages": ["bash", "shell", "text"],
    "directives": {
      "skip": ["skip-validation", "no-test"],
      "runnable": ["runnable", "test"],
      "expectError": ["expect-error"]
    },
    "metaPatterns": {
      "title": "title=\"([^\"]+)\"",
      "filename": "filename=\"([^\"]+)\""
    }
  }
}

Language-Specific Validation

JavaScript/TypeScript

javascript
// validator-config.js
module.exports = {
  javascript: {
    parser: 'babel',
    parserOptions: {
      ecmaVersion: 2024,
      sourceType: 'module'
    },
    execute: {
      runtime: 'node',
      timeout: 10000,
      setup: `
        global.fetch = require('node-fetch');
        process.env.API_KEY = 'test-key';
      `
    },
    format: {
      tool: 'prettier',
      config: {
        semi: true,
        singleQuote: true,
        trailingComma: 'es5'
      }
    }
  },
  typescript: {
    compiler: 'tsc',
    compilerOptions: {
      target: 'ES2022',
      module: 'ESNext',
      strict: true
    },
    execute: {
      runtime: 'ts-node',
      timeout: 15000
    }
  }
};

Python

python
# validator_config.py
PYTHON_CONFIG = {
    'version': '3.11',
    'execute': {
        'timeout': 30,
        'setup': '''
import os
os.environ['API_KEY'] = 'test-key'
''',
        'virtualenv': '.venv'
    },
    'format': {
        'tool': 'black',
        'config': {
            'line_length': 88,
            'target_version': ['py311']
        }
    },
    'lint': {
        'tool': 'ruff',
        'rules': ['E', 'F', 'W']
    }
}

Go

go
// validator_config.go
package main

var GoConfig = Config{
    Version: "1.21",
    Execute: ExecuteConfig{
        Timeout: 30,
        Build:   true,
        Run:     true,
    },
    Format: FormatConfig{
        Tool: "gofmt",
    },
    Lint: LintConfig{
        Tool:  "golangci-lint",
        Rules: []string{"govet", "errcheck", "staticcheck"},
    },
}

Test Generation

Generate Test Files

javascript
// Generated from docs/quickstart.md

const { Client } = require('@example/sdk');

describe('Documentation Code Samples', () => {
  describe('quickstart.md', () => {
    test('Line 15: Basic client usage', async () => {
      const client = new Client({ apiKey: 'test' });
      const result = await client.query('Hello');
      expect(result).toBeDefined();
    });

    test('Line 45: Error handling', async () => {
      const client = new Client({ apiKey: 'invalid' });
      await expect(client.query('Hello')).rejects.toThrow('Invalid API key');
    });
  });
});

Test Configuration

json
{
  "testGeneration": {
    "framework": "jest",
    "outputDir": "tests/docs",
    "filePattern": "{docfile}.test.js",
    "imports": [
      "const { Client } = require('@example/sdk');",
      "const { mockServer } = require('./mocks');"
    ],
    "beforeAll": "await mockServer.start();",
    "afterAll": "await mockServer.stop();",
    "assertions": {
      "outputMatch": true,
      "noThrow": true,
      "typeCheck": true
    }
  }
}

CI/CD Integration

GitHub Actions

yaml
name: Validate Documentation

on:
  push:
    paths:
      - 'docs/**/*.md'
  pull_request:
    paths:
      - 'docs/**/*.md'

jobs:
  validate-code-samples:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          npm ci
          pip install -r requirements-docs.txt

      - name: Validate code samples
        run: |
          node scripts/validate-docs.js \
            --input docs/ \
            --languages javascript,python \
            --report validation-report.json

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: validation-report
          path: validation-report.json

      - name: Check for failures
        run: |
          if jq '.summary.failed > 0' validation-report.json | grep -q true; then
            echo "Code sample validation failed"
            jq '.issues' validation-report.json
            exit 1
          fi

Formatting Validation

Prettier Check

javascript
const prettier = require('prettier');

async function validateFormatting(code, language) {
  const options = {
    parser: getParser(language),
    semi: true,
    singleQuote: true,
    trailingComma: 'es5',
  };

  try {
    const formatted = await prettier.format(code, options);
    const isFormatted = code === formatted;

    return {
      valid: isFormatted,
      formatted: formatted,
      diff: isFormatted ? null : generateDiff(code, formatted),
    };
  } catch (error) {
    return {
      valid: false,
      error: error.message,
    };
  }
}

Black Check (Python)

python
import black
import difflib

def validate_formatting(code: str) -> dict:
    try:
        mode = black.Mode(
            target_versions={black.TargetVersion.PY311},
            line_length=88,
        )
        formatted = black.format_str(code, mode=mode)
        is_formatted = code == formatted

        return {
            'valid': is_formatted,
            'formatted': formatted,
            'diff': None if is_formatted else list(
                difflib.unified_diff(
                    code.splitlines(),
                    formatted.splitlines(),
                    lineterm=''
                )
            )
        }
    except black.InvalidInput as e:
        return {
            'valid': False,
            'error': str(e)
        }

Workflow

  1. Extract samples - Parse Markdown for code blocks
  2. Filter - Apply language and directive filters
  3. Validate syntax - Check for syntax errors
  4. Check imports - Verify dependencies exist
  5. Execute - Run samples in sandbox
  6. Verify output - Check expected results
  7. Check formatting - Validate code style
  8. Report - Generate validation report

Dependencies

json
{
  "devDependencies": {
    "@babel/parser": "^7.23.0",
    "prettier": "^3.0.0",
    "typescript": "^5.3.0",
    "ts-node": "^10.9.0",
    "jest": "^29.7.0",
    "gray-matter": "^4.0.0"
  }
}

CLI Commands

bash
# Validate all samples
node scripts/validate-docs.js --input docs/

# Extract and save runnable code
node scripts/validate-docs.js --input docs/ --action extract --output tests/samples/

# Generate test files
node scripts/validate-docs.js --input docs/ --action generate-tests --output tests/docs/

# Check formatting only
node scripts/validate-docs.js --input docs/ --action format-check

Best Practices Applied

  • Mark non-runnable samples with directives
  • Include expected output for verification
  • Use consistent formatting across samples
  • Pin dependency versions in samples
  • Test samples in CI/CD pipeline
  • Generate runnable tests from docs

References

Target Processes

  • docs-testing.js
  • api-reference-docs.js
  • sdk-doc-generation.js
  • interactive-tutorials.js

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

a5c-ai/babysitter

gsd-tools

Central utility skill for GSD operations. Provides config parsing, slug generation, timestamps, path operations, and orchestrates calls to other specialized skills. Acts as the unified entry point that the original gsd-tools.cjs provided via its lib/ modules (commands, config, core, init).

514 31
Explore
a5c-ai/babysitter

model-profile-resolution

Resolve model profile (quality/balanced/budget) at orchestration start and map agents to specific models. Enables cost/quality tradeoffs by selecting appropriate AI models for each agent role.

514 31
Explore
a5c-ai/babysitter

verification-suite

Plan structure validation, phase completeness checks, reference integrity verification, and artifact existence confirmation. Provides the structured verification layer ensuring GSD artifacts are well-formed and complete.

514 31
Explore
a5c-ai/babysitter

state-management

STATE.md reading, writing, and field-level updates. Provides cross-session state persistence via .planning/STATE.md with structured fields for current task, completed phases, blockers, decisions, and quick tasks.

514 31
Explore
a5c-ai/babysitter

git-integration

Git commit patterns, formats, and conventions for GSD methodology. Provides atomic commits per task, structured commit messages, planning file commits, branch management, and milestone tag operations.

514 31
Explore
a5c-ai/babysitter

frontmatter-parsing

YAML frontmatter parsing and manipulation for .planning/ documents. Provides read, write, update, query, and validation operations on frontmatter blocks in GSD markdown artifacts.

514 31
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results