Airflow

When to Use This Skill

USE when:

• Building complex data pipelines with task dependencies
• Orchestrating ETL/ELT workflows
• Scheduling recurring batch jobs
• Managing workflows with retries and error handling
• Coordinating tasks across multiple systems
• Need visibility into workflow execution history
• Requiring audit trails and lineage tracking
• Building ML pipeline orchestration

DON'T USE when:

• Real-time streaming data (use Kafka, Flink)
• Simple cron jobs (use systemd timers, crontab)
• CI/CD pipelines (use GitHub Actions, Jenkins)
• Low-latency requirements (Airflow has scheduler overhead)
• Simple single-task automation (overkill)
• Need visual workflow design for non-developers (use n8n)

Prerequisites

Installation Options

Option 1: pip (Development)

# Create virtual environment
python -m venv airflow-env
source airflow-env/bin/activate

# Set Airflow home
export AIRFLOW_HOME=~/airflow

# Install Airflow with constraints

*See sub-skills for full details.*
### Development Setup

```bash
# Install development dependencies
pip install apache-airflow[dev,postgres,celery,kubernetes]

# Install testing tools
pip install pytest pytest-airflow

# Install linting
pip install ruff

Version History

Resources

• Apache Airflow Documentation
• Airflow Best Practices
• Airflow Helm Chart
• Astronomer Guides
• Airflow Providers

This skill provides production-ready patterns for Apache Airflow workflow orchestration, tested across enterprise data pipelines.

Sub-Skills

• 1. Basic DAG Structure
• Integration with AWS Services
• 1. DAG Design Principles (+3)
• Common Issues (+1)

Sub-Skills

• 2. Advanced Operators (+6)