Announce: "Using ds-validate (Phase 3.5) to validate analysis outputs against SPEC.md requirements."

Contents

• The Iron Law of Validation
• Red Flags - STOP Immediately
• Key Difference from Dev
• The Process
• Validation Levels
• Classification
• VALIDATION.md Template
• Gate
• Rationalization Prevention
• Drive-Aligned Framing
• Phase Transition

Output Validation Against SPEC.md

Phase 3.5 of the DS workflow (between implement and review). Maps every SPEC.md requirement to an output artifact and runs data quality checks.

NO REVIEW WITHOUT VALIDATION. This is not negotiable.

ds-review MUST NOT start until .planning/VALIDATION.md confirms all requirements have outputs. Validation is the DS equivalent of test coverage — without it, review is theater. </EXTREMELY-IMPORTANT>

Key Difference from Dev

DS validation does NOT auto-fill gaps. Dev's test-gap-auditor can write missing tests. DS gaps require human judgment — a wrong output means a wrong analysis, not just a missing test. When gaps are found, present them to the user and let the user decide: fix (return to implement) or accept (proceed to review).

Static Analysis (Constraint Check Scripts)

Before running runtime DQ checks, run the static analysis constraint check suite:

bash "${CLAUDE_SKILL_DIR}/../../scripts/check-all-ds.sh" "$(pwd)"

This runs all DS constraint check scripts (determinism, join audits, idempotency, error handling, schema contracts, standard errors, visualization integrity).

If any check FAILS: Report the failures in LEARNINGS.md. These are code quality issues in the analysis scripts that must be fixed before proceeding. Dispatch a fix subagent if needed.

If all checks PASS: Proceed to runtime DQ checks.

The Process

0. RUN static analysis check suite (check-all-ds.sh) — fix any failures first
1. READ .planning/SPEC.md requirements
2. READ .planning/PLAN.md task breakdown
3. READ .planning/LEARNINGS.md for pipeline row counts (DQ4 needs these)
4. DISCOVER and READ ds-checks.md via cache lookup
5. For each requirement: DISPATCH subagent to run DQ1-DQ5 + M1 on the output
6. WRITE .planning/VALIDATION.md

Step 1: Read Requirements

Read .planning/SPEC.md and extract every requirement:

For each requirement in SPEC.md:
  - Extract the requirement description
  - Note the success criteria
  - Note the expected output (table, figure, file, etc.)

Step 2: Read Plan

Read .planning/PLAN.md and extract:

• Task-to-requirement mapping
• Output file locations mentioned
• Key columns and data structure decisions

Step 3: Read Learnings

Read .planning/LEARNINGS.md and extract:

• Pipeline row counts at each stage (needed for DQ4 traceability)
• Data quality observations from implementation
• Any known issues or caveats

Step 4: Load DQ Check Definitions

Read ${CLAUDE_SKILL_DIR}/../../skills/ds-implement/references/ds-checks.md and follow its instructions.

Step 5: Dispatch Validation Subagents

For each SPEC.md requirement, spawn a subagent:

Agent prompt template:

You are a data quality validator. Your job is to verify that an analysis output
meets a specific requirement from SPEC.md.

REQUIREMENT: [requirement description from SPEC.md]
SUCCESS CRITERIA: [from SPEC.md]
EXPECTED OUTPUT: [file path or variable]
PIPELINE ROW COUNTS: [from LEARNINGS.md]

Run the following checks on the output:

DQ1: Empty/constant columns — flag columns with nunique() <= 1
DQ2: High-null columns — flag columns with >50% null values
DQ3: Duplicate rows — check for duplicates on key columns
DQ4: Row count traceability — verify final count matches LEARNINGS.md pipeline
DQ5: Cardinality check — flag categoricals with suspicious cardinality
M1: Spec compliance — does this output address the requirement?

For each check, report: PASS / WARN / FAIL with details.

RULES:
1. Do NOT modify any code or data files
2. Read and inspect outputs only
3. If an output file does not exist, report MISSING immediately
4. If checks reveal issues, report them — do NOT fix them

Step 6: Write VALIDATION.md

Compile all subagent results into .planning/VALIDATION.md using the template below.

Validation Levels

Each requirement is validated at four levels, in order:

Classification

For each requirement, assign a classification:

VALIDATION.md Template

---
status: validated | gaps_found
date: [ISO 8601]
requirements_total: N
covered: N
partial: N
missing: N
---
# Output Validation

## Requirements Map
| # | Requirement | Output | DQ1 | DQ2 | DQ3 | DQ4 | DQ5 | M1 | Classification |
|---|-------------|--------|-----|-----|-----|-----|-----|----|----------------|
| 1 | [from SPEC] | [path] | PASS | PASS | PASS | PASS | PASS | PASS | COVERED |
| 2 | [from SPEC] | [path] | PASS | WARN | PASS | PASS | PASS | PASS | PARTIAL |
| 3 | [from SPEC] | — | — | — | — | — | — | — | MISSING |

## DQ Details
[For any non-PASS check, include the specific finding]

## Summary
- Requirements: N total
- Covered: X
- Partial: Y
- Missing: Z

Status Rules

Visual Diagnostics for Decision Checkpoints

When presenting validation results to the user (especially gaps), generate diagnostic plots to accelerate the decision:

When to generate: Only at decision checkpoints where the user must choose fix vs accept. Do not generate plots for COVERED requirements (no decision needed).

Format: Inline matplotlib/seaborn plots in notebooks, or saved to scratch/diagnostics/ for script-based workflows.

Gate

Checkpoint type: human-verify (VALIDATION.md status is machine-verifiable)

.planning/VALIDATION.md must exist before proceeding.

• If status is validated: proceed to ds-review.
• If status is gaps_found: present gaps to user before proceeding.
  • User decides: fix (return to ds-implement) or accept (proceed to ds-review with known gaps).

This is the critical difference from dev-test-gaps. In dev, missing tests can be auto-generated. In DS, missing or wrong outputs mean the analysis itself may be wrong. Only the user can judge whether a gap is acceptable. </EXTREMELY-IMPORTANT>

Rationalization Prevention

Drive-Aligned Framing

The protocol is not overhead you pay. It is the safety net you provide. </EXTREMELY-IMPORTANT>

Phase Transition

After validation is complete, discover and read the ds-review skill: Read ${CLAUDE_SKILL_DIR}/../../skills/ds-review/SKILL.md and follow its instructions.