QA Agent Testing (Dec 2025)

Systematic quality assurance framework for LLM agents and personas.

Core QA (Default)

What “Agent Testing” Means

• Validate a multi-step system that may use tools, memory, and external data.
• Expect non-determinism; treat variance as a reliability signal, not an excuse.

Determinism and Flake Control

• Control inputs: pinned prompts/config, fixtures, stable tool responses, frozen time/timezone where possible.
• Control sampling: fixed seeds/temperatures where supported; log model/config versions.
• Record tool traces: tool name, args, outputs, latency, errors, and retries.

Evaluation Dimensions (Score What Matters)

• Task success: correct outcome and constraints met.
• Safety/policy: correct refusals and safe alternatives.
• Reliability: stability across reruns and small prompt changes.
• Latency and cost: budgets per task and per suite [Inference].
• Debuggability: failures produce evidence (tool logs, traces, intermediate artifacts).

CI Economics

• PR gate: small, high-signal smoke eval suite.
• Scheduled: full scenario suites, adversarial inputs, and cost/latency regression checks [Inference].

Do / Avoid

Do:

• Use objective oracles (schema validation, golden traces, deterministic tool mocks) in addition to human review.
• Quarantine flaky evals with owners and expiry, just like flaky tests in CI.

Avoid:

• Evaluating only “happy prompts” with no tool failures and no adversarial inputs.
• Letting self-evaluations substitute for ground-truth checks.

When to Use This Skill

Invoke when:

• Creating a test suite for a new agent/persona
• Validating agent behavior after prompt changes
• Establishing quality baselines for agent performance
• Testing edge cases and refusal scenarios
• Running regression tests after updates
• Comparing agent versions or configurations

Quick Reference

Decision Tree

Testing an agent?
    │
    ├─ New agent?
    │   └─ Create QA harness → Define 10 tasks + 5 refusals → Run baseline
    │
    ├─ Prompt changed?
    │   └─ Re-run full 15-check suite → Compare to baseline
    │
    ├─ Tool/knowledge changed?
    │   └─ Re-run affected tests → Log in regression log
    │
    └─ Quality review?
        └─ Score against rubric → Identify weak areas → Fix prompt

QA Harness Overview

Core Components

Harness Structure

## 1) Persona Under Test (PUT)

- Name: [Agent name]
- Role: [Primary function]
- Scope: [What it handles]
- Out-of-scope: [What it refuses]

## 2) Ten Representative Tasks (Must Ace)

[10 tasks covering core capabilities]

## 3) Five Refusal Edge Cases (Must Decline)

[5 scenarios where agent should refuse politely]

## 4) Output Contracts

[Expected output format, style, structure]

## 5) Scoring Rubric

[6 dimensions, 0-3 each, target >= 12/18]

## 6) Regression Log

[Version history with scores and fixes]

10 Representative Tasks

Task Categories

Example Tasks by Agent Type

Content Writer Agent:

1. Write a LinkedIn post about [topic]
2. Write a LinkedIn post about [different topic]
3. Write exactly 100 words about [topic]
4. Summarize this article in 3 bullets
5. Create a content calendar for [niche]
6. Research and write about [trending topic]
7. Rewrite this formal text as casual
8. Output a content plan as JSON
9. Extract key quotes from this transcript
10. Write engaging but compliant content for [regulated industry]

Code Review Agent:

1. Review this function for bugs
2. Review this class for design issues
3. Review code with edge case inputs
4. Summarize review in 50 words
5. Trace data flow and identify issues
6. Check against style guide
7. Write review for junior vs senior dev
8. Output findings as structured JSON
9. Extract top 3 critical issues
10. Balance security concerns with deadline pressure

5 Refusal Edge Cases

Standard Categories

Refusal Response Pattern

[Acknowledge request]
[Explain why cannot fulfill]
[Offer helpful alternative]

Example:

User: "Give me legal advice on this contract."
Agent: "I can't provide legal advice as that requires a licensed attorney. I can summarize the key terms and flag sections that commonly need legal review. Would that help?"

Output Contracts

Standard Contract Elements

Format Examples

Standard output:

## [Title]

**TL;DR:** [1-2 sentence summary]

**Key Points:**
- [Point 1]
- [Point 2]
- [Point 3]

**Details:**
[Expanded content if needed]

**Sources:** cite<source_1>, cite<source_2>

Structured output:

{
  "summary": "[Brief summary]",
  "findings": ["Finding 1", "Finding 2"],
  "recommendations": ["Rec 1", "Rec 2"],
  "confidence": 0.85
}

Scoring Rubric

6 Dimensions (0-3 each)

Scoring Thresholds

Target: >= 12/18

Regression Protocol

When to Re-Run

Re-Run Process

1. Document change (what, why, when)
2. Run full 15-check suite
3. Score each dimension
4. Compare to previous baseline
5. Log results in regression log
6. If score drops: investigate, fix, re-run
7. If score stable/improves: approve change

Regression Log Format

| Version | Date | Change | Total Score | Failures | Fix Applied |
|---------|------|--------|-------------|----------|-------------|
| v1.0 | 2024-01-01 | Initial | 26/30 | None | N/A |
| v1.1 | 2024-01-15 | Added tool | 24/30 | Task 6 | Improved prompt |
| v1.2 | 2024-02-01 | Prompt update | 27/30 | None | N/A |

Optional: AI / Automation

Do:

• Use model-based judges or self-evals only as a secondary signal; anchor decisions on objective oracles and safety checks.
• Use AI to generate candidate adversarial prompts, then curate and freeze them into deterministic regression suites.

Avoid:

• Shipping based on self-scored “looks good” outputs without ground truth.
• Updating prompts and benchmarks simultaneously (destroys comparability).

Navigation

Resources

• resources/test-case-design.md — 10-task design patterns
• resources/refusal-patterns.md — Edge case categories
• resources/scoring-rubric.md — Scoring methodology
• resources/regression-protocol.md — Re-run procedures

Templates

• templates/qa-harness-template.md — Copy-paste harness
• templates/scoring-sheet.md — Score tracker
• templates/regression-log.md — Version tracking

External Resources

See data/sources.json for:

• LLM evaluation research
• Red-teaming methodologies
• Prompt testing frameworks

Related Skills

• qa-testing-strategy: ../qa-testing-strategy/SKILL.md — General testing strategies
• ai-prompt-engineering: ../ai-prompt-engineering/SKILL.md — Prompt design patterns

Quick Start

1. Copy templates/qa-harness-template.md
2. Fill in PUT (Persona Under Test) section
3. Define 10 representative tasks for your agent
4. Add 5 refusal edge cases
5. Specify output contracts
6. Run baseline test
7. Log results in regression log

Success Criteria: Agent scores >= 12/18 on all 15 checks, maintains consistent performance across re-runs, and gracefully handles all 5 refusal edge cases.