AgentV Eval Analyzer

Analyze JSONL evaluation results and produce actionable suggestions for improving eval quality, reducing cost, and increasing reliability.

When to Use

• After running agentv eval and wanting to improve your evaluation config
• When evaluations are slow or expensive and you suspect LLM-judges are doing deterministic work
• When reviewing eval quality before sharing or publishing evaluation files
• When triaging flaky evaluations that produce inconsistent scores

Quick Start

# Find your most recent results file
agentv trace list --limit 5

# Dispatch the eval-analyzer agent with the results file
# Agent: eval-analyzer
# Parameters:
#   results-file: <path-to-results.jsonl>
#   eval-path: <path-to-eval.yaml>  (optional, for deeper analysis)

Dispatch the eval-analyzer agent with:

• results-file: Path to the JSONL results (from .agentv/results/ or agentv trace list)
• eval-path (optional): Path to the EVAL.yaml for assertion-level analysis

The agent produces a read-only report — it never modifies files.

What It Detects

1. Deterministic-Upgrade Candidates (highest value)

LLM-judge evaluators doing work that a deterministic assertion could handle — cheaper, faster, and more reliable.

2. Weak Assertion Detection

3. Cost/Quality Flags

4. Multi-Provider Variance

When results span multiple targets, flags evaluators with > 0.3 score variance across providers — indicating provider-sensitive assertions that may need tightening.

Applying Suggestions

The analyzer report includes concrete YAML snippets for each suggestion. To apply:

1. Open the EVAL.yaml referenced in the report
2. Find the assertions entry for the flagged evaluator (matched by name and test_id)
3. Replace or supplement the evaluator config with the suggested deterministic assertion
4. Re-run agentv eval to verify the change produces equivalent scores

Example upgrade:

Before (LLM-judge doing substring work):

assertions:
  - name: has-error-code
    type: llm-judge
    prompt: "Check if the response contains the error code 404"

After (deterministic, zero LLM cost):

assertions:
  - name: has-error-code
    type: contains
    value: "404"

Limitations

• Deterministic-upgrade detection is heuristic — review suggestions before applying
• Requires at least one completed eval run (needs JSONL results)
• Cannot detect all weak assertions without the EVAL.yaml file (results-only mode has reduced coverage)
• Multi-run comparison (flakiness across runs) is out of scope — use agentv compare for that