Agent skill
pipeline-design
Design ETL/ELT pipeline architectures with proper patterns for reliability and scalability.
Install this agent skill to your Project
npx add-skill https://github.com/timequity/plugins/tree/main/vibe-coder/skills/data/pipeline-design
SKILL.md
Pipeline Design
ETL vs ELT
| Approach | When to Use |
|---|---|
| ETL | Transform before load, limited warehouse compute |
| ELT | Modern warehouses (Snowflake, BigQuery, Redshift) |
Pipeline Patterns
Batch
Source → Extract → Stage → Transform → Load → Target
│ │
└── Checkpoint ──────┘
- Scheduled intervals (hourly, daily)
- Full or incremental loads
- Idempotent operations
Streaming
Source → Kafka/Kinesis → Process → Sink
│
└── State Store
- Real-time requirements
- Event-driven architecture
- Exactly-once semantics
Design Principles
- Idempotent - Safe to re-run
- Incremental - Process only new/changed data
- Observable - Metrics, logs, alerts
- Testable - Unit tests for transformations
- Recoverable - Checkpoints, retry logic
Staging Pattern
-- 1. Land raw data
COPY INTO raw.source_data FROM @stage;
-- 2. Deduplicate
CREATE TABLE staging.deduped AS
SELECT * FROM raw.source_data
QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY _loaded_at DESC) = 1;
-- 3. Transform to target
MERGE INTO target.dim_customer
USING staging.deduped
ON target.id = staging.id
WHEN MATCHED THEN UPDATE ...
WHEN NOT MATCHED THEN INSERT ...;
Error Handling
- Dead letter queues for failed records
- Retry with exponential backoff
- Alert on threshold breaches
- Quarantine bad data for review
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
claude-code-course
Learn Claude Code in 5 hands-on lessons. From basics to building applications. Use when: user wants to learn Claude Code, asks "how do I...", or is new. Triggers: "learn", "teach me", "how does this work", "I'm new", "course".
foundations
Claude Code basics for new users. Covers essential concepts and commands. Use when: user is new to Claude Code or routed here by learning-path assessment.
advanced
Claude Code mastery for power users. Build agents, MCP servers, publish plugins. Use when: user wants to create agents, build integrations, or publish plugins.
lessons
Lesson content for Claude Code Course. Use when: loading lesson content. Internal skill - contains lesson files.
intermediate
Claude Code customization for comfortable users. Covers skills, commands, MCP, hooks. Use when: user knows basics and wants to customize Claude Code.
course-help
Help and navigation for Claude Code Course. Use when: user asks about the course, lessons, or progress. Triggers: "help", "course help", "what lessons", "my progress".
Didn't find tool you were looking for?