Data Validation Framework

Purpose

Comprehensive data validation framework for testing schema compliance, data quality, and referential integrity. Validates databases, APIs, data pipelines, and file formats. Generates data quality scorecards with anomaly detection.

Triggers

Use this skill when:

"validate data quality"
"check data integrity"
"schema validation"
"test data pipeline"
"data quality report"
"validate CSV"
"check for data anomalies"
"test ETL output"

When to Use

Data pipeline deployment
Database migration
API response validation
Report generation systems
Data warehouse testing
ML training data validation

When NOT to Use

API endpoint testing (use api-contract-validator)
Security testing (use security-test-suite)
Performance testing (use performance-benchmark)

Core Instructions

Data Quality Dimensions

Dimension	Description	Weight
Completeness	Missing values, required fields	25%
Accuracy	Type conformance, format validation	25%
Consistency	Cross-field rules, referential integrity	20%
Uniqueness	Duplicate detection, key uniqueness	15%
Freshness	Timestamp validation, staleness	10%
Anomaly	Statistical outlier detection	5%

Validation Categories

Category	Description	Severity
Schema	Structure and type compliance	Critical
Completeness	Missing/null value detection	High
Accuracy	Value correctness and format	High
Consistency	Cross-field/cross-table rules	Medium
Uniqueness	Duplicate detection	Medium
Freshness	Timeliness of data	Medium
Anomaly	Statistical outlier detection	Low

Schema Definition

yaml

schema:
  tables:
    transactions:
      columns:
        - name: transaction_id
          type: string
          required: true
          unique: true
          pattern: "^TXN-[A-Z0-9]{10}$"

        - name: amount
          type: float
          required: true
          min: 0.01
          max: 1000000

        - name: status
          type: string
          required: true
          enum: [pending, completed, failed]

Templates

Data Quality Report

markdown

# Data Quality Report

**Source:** {source_type}
**Table:** {table_name}
**Generated:** {timestamp}

## Quality Scorecard

**Overall Score:** {score}/100 ({grade})

| Dimension | Score | Status |
| --------- | ----- | ------ |
| Completeness | {completeness} | {status_icon} |
| Accuracy | {accuracy} | {status_icon} |
| Consistency | {consistency} | {status_icon} |
| Uniqueness | {uniqueness} | {status_icon} |
| Freshness | {freshness} | {status_icon} |

## Data Summary

| Metric | Value |
| ------ | ----- |
| Total Rows | {total_rows} |
| Valid Rows | {valid_rows} ({valid_percent}%) |
| Invalid Rows | {invalid_rows} ({invalid_percent}%) |

## Issue Details

### {category} Issues

**{issue_id}:** {message}

- Column: `{column}`
- Affected rows: {row_count}
- Sample values: `{samples}`

Example

Input: Validate transactions CSV against schema

Output:

markdown

## Quality Scorecard

**Overall Score:** 87.3/100 (B)

| Dimension | Score | Status |
| --------- | ----- | ------ |
| Completeness | 95.0 | Pass |
| Accuracy | 88.5 | Pass |
| Consistency | 82.0 | Pass |
| Uniqueness | 100.0 | Pass |
| Freshness | 75.0 | Warn |

## Issue Details

### Accuracy Issues

**TYPE-amount:** Expected float, got string

- Column: `amount`
- Affected rows: 45
- Sample values: `"N/A", "pending", "TBD"`

Validation Checklist

Schema definition matches expected structure
All required columns validated
Null thresholds appropriately set
Foreign key references checked (if applicable)
Anomaly detection parameters tuned
Sample data reviewed for false positives
Report includes actionable remediation

Related Skills

api-contract-validator - For API response validation
unit-test-generator - For data processing function tests
test-health-monitor - For tracking validation trends

Search AI Tools

data-validation

Install this agent skill to your Project

Metadata

SKILL.md