ReddRadar
Agent skill
etl-pipeline-builder
Build and manage ETL pipelines for data migration with transformation, CDC, and monitoring
Install this agent skill to your Project
npx add-skill https://github.com/a5c-ai/babysitter/tree/main/library/specializations/code-migration-modernization/skills/etl-pipeline-builder
SKILL.md
ETL Pipeline Builder Skill
Builds and manages ETL (Extract, Transform, Load) pipelines for data migration, supporting incremental loads, CDC, and comprehensive monitoring.
Purpose
Enable data pipeline creation for:
- Source-to-target mapping
- Transformation definition
- Incremental load setup
- CDC configuration
- Pipeline monitoring
Capabilities
1. Source-to-Target Mapping
- Define column mappings
- Handle schema differences
- Configure data type conversions
- Manage derived columns
2. Transformation Definition
- Data type transformations
- Value mappings
- Aggregations
- Lookups and enrichments
3. Incremental Load Setup
- Define watermarks
- Configure incremental columns
- Handle deletes
- Manage merge logic
4. CDC Configuration
- Log-based CDC
- Trigger-based CDC
- Timestamp-based CDC
- Full load comparison
5. Error Handling
- Define retry policies
- Configure dead letter queues
- Handle data quality issues
- Implement alerting
6. Pipeline Monitoring
- Track pipeline metrics
- Monitor data volumes
- Alert on failures
- Generate SLA reports
Tool Integrations
| Tool | Type | Integration Method |
|---|---|---|
| Apache Airflow | Orchestration | Python |
| dbt | Transformation | CLI |
| Airbyte | Data integration | API |
| Fivetran | SaaS ETL | API |
| AWS DMS | Cloud migration | CLI |
| Debezium | CDC | Config |
Output Schema
{
"pipelineId": "string",
"timestamp": "ISO8601",
"pipeline": {
"name": "string",
"source": {},
"target": {},
"mappings": [],
"transformations": [],
"schedule": "string"
},
"artifacts": {
"dagFile": "string",
"configFile": "string",
"sqlFiles": []
},
"deployment": {
"status": "string",
"url": "string"
}
}
Integration with Migration Processes
- database-schema-migration: Data movement
- cloud-migration: Cloud data pipelines
- data-format-migration: Format transformation
Related Skills
data-migration-validator: Validationschema-comparator: Schema mapping
Related Agents
database-migration-orchestrator: Pipeline orchestrationdata-architect-agent: Pipeline design
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
a5c-ai/babysitter
gsd-tools
Central utility skill for GSD operations. Provides config parsing, slug generation, timestamps, path operations, and orchestrates calls to other specialized skills. Acts as the unified entry point that the original gsd-tools.cjs provided via its lib/ modules (commands, config, core, init).
a5c-ai/babysitter
model-profile-resolution
Resolve model profile (quality/balanced/budget) at orchestration start and map agents to specific models. Enables cost/quality tradeoffs by selecting appropriate AI models for each agent role.
a5c-ai/babysitter
verification-suite
Plan structure validation, phase completeness checks, reference integrity verification, and artifact existence confirmation. Provides the structured verification layer ensuring GSD artifacts are well-formed and complete.
a5c-ai/babysitter
state-management
STATE.md reading, writing, and field-level updates. Provides cross-session state persistence via .planning/STATE.md with structured fields for current task, completed phases, blockers, decisions, and quick tasks.
a5c-ai/babysitter
git-integration
Git commit patterns, formats, and conventions for GSD methodology. Provides atomic commits per task, structured commit messages, planning file commits, branch management, and milestone tag operations.
a5c-ai/babysitter
frontmatter-parsing
YAML frontmatter parsing and manipulation for .planning/ documents. Provides read, write, update, query, and validation operations on frontmatter blocks in GSD markdown artifacts.
Didn't find tool you were looking for?