Agent skill
promptfoo
Promptfoo evaluation framework for testing and comparing LLM outputs. Use when writing eval configs, creating test cases, debugging eval runs, or working with assertions.
Install this agent skill to your Project
npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/data/promptfoo
SKILL.md
Promptfoo
Promptfoo is a CLI tool for testing and comparing LLM outputs.
Config File
The CLI auto-discovers promptfooconfig.yaml in the current directory. Use -c path for other locations.
Supported extensions: .yaml, .json, .js
Configuration
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: "What this eval tests"
prompts:
- file://prompt.txt
- |
Inline prompt with {{variable}} substitution
providers:
- anthropic:messages:claude-sonnet-4-5-20250929
defaultTest:
options:
provider:
config:
temperature: 0.0
max_tokens: 4096
tests:
- description: "What this case tests"
vars:
variable: "value"
from_file: file://data/input.txt
assert:
- type: contains
value: "expected substring"
# Or load tests from files
tests: file://cases/all.yaml
outputPath: ./results.json
evaluateOptions:
maxConcurrency: 4
Provider IDs
| Model | ID |
|---|---|
| Opus 4.5 | anthropic:messages:claude-opus-4-5-20251101 |
| Sonnet 4.5 | anthropic:messages:claude-sonnet-4-5-20250929 |
| Haiku 4.5 | anthropic:messages:claude-haiku-4-5-20251001 |
Provider config: temperature, max_tokens, top_p, top_k, tools, tool_choice
Prompts
file://path.txt— load from file (path relative to config)- Inline string with
{{variable}}Nunjucks substitution - Chat format via JSON:
[{"role": "system", "content": "..."}, {"role": "user", "content": "{{input}}"}]
Assertion Types
| Type | Use | Value |
|---|---|---|
contains |
Substring match | "expected text" |
icontains |
Case-insensitive substring | "expected text" |
equals |
Exact match | "exact value" |
regex |
Pattern match | "\\d{4}-\\d{2}-\\d{2}" |
is-json |
Valid JSON output | — |
contains-json |
Output contains JSON | — |
starts-with |
Prefix match | "prefix" |
cost |
Max cost | threshold: 0.01 |
latency |
Max response time (ms) | threshold: 5000 |
javascript |
Custom JS expression | output.includes('x') |
python |
Custom Python | file://check.py:fn_name |
llm-rubric |
LLM-as-judge | rubric text |
similar |
Semantic similarity | value: "text", threshold: 0.8 |
model-graded-factuality |
Fact checking | — |
Prefix any assertion with not- to negate (e.g., not-contains).
llm-rubric
Uses an LLM to grade output against a rubric:
assert:
- type: llm-rubric
value: |
The response should:
- Mention at least 3 factors
- Include specific examples
threshold: 0.7
provider: anthropic:messages:claude-sonnet-4-5-20250929
javascript
Inline expressions or functions. Access output (string) and context (with vars, prompt):
assert:
- type: javascript
value: output.length > 100 && output.includes('route')
- type: javascript
value: |
const data = JSON.parse(output);
return data.calories >= 200 && data.calories <= 300;
Test Organization
Split cases into separate files and reference them:
tests:
- file://cases/basic.yaml
- file://cases/edge-cases.yaml
Each case file contains a YAML array of test objects.
CLI
npx promptfoo eval # Run with auto-discovered config
npx promptfoo eval -c path/to/config.yaml # Specific config
npx promptfoo eval --filter-metadata key=v # Filter tests
npx promptfoo view # Web UI for results
npx promptfoo cache clear # Clear result cache
References
Consult the configuration reference and Anthropic provider docs for full details.
Didn't find tool you were looking for?