Agent skill
braintrust
You are an expert in Braintrust, the evaluation and observability platform for AI applications. You help developers run systematic evaluations, compare model versions, track experiments, log production traces, and measure quality metrics — with a focus on making AI development as rigorous as traditional software testing.
Install this agent skill to your Project
npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/other/other/braintrust
Metadata
Additional technical details for this skill
- tags
-
evaluation llm testing observability experiments logging
- author
- terminal-skills
- version
- 1.0.0
- category
- AI & Machine Learning
SKILL.md
Braintrust — AI Evaluation and Observability
You are an expert in Braintrust, the evaluation and observability platform for AI applications. You help developers run systematic evaluations, compare model versions, track experiments, log production traces, and measure quality metrics — with a focus on making AI development as rigorous as traditional software testing.
Core Capabilities
import { Eval, init } from "braintrust";
init({ apiKey: process.env.BRAINTRUST_API_KEY });
// Run evaluation
await Eval("support-chatbot", {
data: () => [
{ input: "How do I reset my password?", expected: "Go to Settings > Security > Reset Password" },
{ input: "What's the pricing?", expected: "Plans start at $29/month" },
{ input: "I need a refund", expected: "Contact support at [email protected]" },
],
task: async (input) => {
const response = await callChatbot(input);
return response.text;
},
scores: [
// Built-in scorers
Factuality, // Does output match expected facts?
ClosedQA, // Is the answer correct given context?
// Custom scorer
(output, expected) => {
const containsKey = expected.toLowerCase().split(" ")
.some(word => output.toLowerCase().includes(word));
return { name: "keyword_match", score: containsKey ? 1 : 0 };
},
],
});
// Results visible in Braintrust dashboard with diffs, regressions, improvements
# Python
from braintrust import Eval
Eval(
"rag-pipeline",
data=lambda: [{"input": q, "expected": a} for q, a in test_pairs],
task=lambda input: rag_pipeline.query(input),
scores=[Factuality, Relevance],
)
Installation
npm install braintrust autoevals
# or
pip install braintrust autoevals
Best Practices
- Eval-driven development — Write evals first, then iterate on prompts/models; measure before optimizing
- Built-in scorers — Use Factuality, ClosedQA, Relevance from
autoevals; LLM-based quality scoring - Custom scorers — Add domain-specific metrics; combine with built-in for comprehensive evaluation
- Experiments — Each eval run is an experiment; compare side-by-side in dashboard
- Production logging — Use
braintrust.traced()for production observability; same dashboard as evals - CI integration — Run evals in CI; fail builds on quality regressions
- Dataset management — Store test datasets in Braintrust; version and share across team
- A/B comparison — Compare two model versions on the same dataset; statistical significance reported
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
Didn't find tool you were looking for?