Workflow Optimizer

Description

Analyzes existing workflows and automation systems to identify bottlenecks, redundancies, and optimization opportunities. Specializes in N8N workflows, data pipelines, and multi-step automation processes. Use when you need to improve performance, reduce costs, or scale operations.

When to Use This Skill

Trigger this skill when the user:

• Says workflow is "slow" or "expensive"
• Wants to "optimize" or "improve" existing automation
• Asks "how can we make this faster/cheaper?"
• Reports bottlenecks or failures in pipelines
• Needs to scale from X to 10X volume
• Wants to reduce manual intervention
• Asks about parallel processing or batching

Core Capabilities

1. Bottleneck Identification

• Analyze workflow execution times
• Identify slowest steps
• Find sequential operations that could be parallel
• Detect rate limiting issues
• Spot unnecessary data transformations

2. Cost Optimization

• Calculate cost per operation
• Identify expensive API calls
• Recommend caching strategies
• Suggest batch processing
• Find redundant operations

3. Reliability Improvement

• Add error handling
• Implement retry logic
• Create fallback strategies
• Design circuit breakers
• Add monitoring and alerts

4. Scale Planning

• Identify scale bottlenecks
• Design for 10x/100x growth
• Plan infrastructure needs
• Optimize for throughput
• Balance cost vs speed

Analysis Framework

Step 1: Map Current State

## Workflow: [Name]

### Flow Steps
1. [Step 1] - Duration: X, Cost: $Y
2. [Step 2] - Duration: X, Cost: $Y
3. [Step 3] - Duration: X, Cost: $Y

### Metrics
- Total execution time: [X seconds]
- Success rate: [Y%]
- Cost per run: $[Z]
- Runs per day: [N]
- Daily cost: $[N × Z]

### Pain Points
- [ ] Slow steps
- [ ] High failure rate
- [ ] Manual intervention needed
- [ ] High cost
- [ ] Can't scale

Step 2: Identify Optimizations

For each step, evaluate:

Speed Optimizations:

• Can this run in parallel?
• Can we batch operations?
• Can we cache results?
• Can we reduce data transferred?
• Can we use a faster API/tool?

Cost Optimizations:

• Is this API call necessary?
• Can we use a cheaper alternative?
• Can we reduce call frequency?
• Can we negotiate volume pricing?
• Can we self-host?

Reliability Optimizations:

• What if this step fails?
• Do we have retry logic?
• Is there a fallback?
• Are we monitoring this?
• Can we recover automatically?

Step 3: Prioritize Improvements

Use Impact × Ease matrix:

Priority Formula: Impact (1-10) × Ease (1-10) = Score

• Score >50: Do immediately
• Score 25-50: Plan for this month
• Score <25: Nice to have

Common Optimization Patterns

Pattern 1: Parallel Processing

Before:

Step 1 → Step 2 → Step 3 → Step 4 → Step 5
Total: 50 seconds

After:

Step 1 → [Step 2, Step 3, Step 4 in parallel] → Step 5
Total: 20 seconds

When to Use:

• Steps don't depend on each other
• API allows concurrent requests
• System can handle parallel load

Pattern 2: Batch Operations

Before:

For each item (1000 items):
  - API call (100ms)
Total: 100 seconds

After:

Batch items into groups of 100:
  - Batch API call (500ms)
Total: 5 seconds

When to Use:

• API supports batch requests
• Many small operations
• Rate limits allow batching

Pattern 3: Smart Caching

Before:

Every request:
  - Fetch data from API (1s)
  - Transform data (0.1s)
Total: 1.1s per request

After:

First request:
  - Fetch + cache (1s)
  - Transform (0.1s)
Subsequent requests:
  - Read from cache (0.01s)
  - Transform (0.1s)
Total: 0.11s per request

When to Use:

• Data changes infrequently
• Same data accessed multiple times
• Cache invalidation is manageable

Pattern 4: Smart Retries

Before:

API call fails → Workflow fails
Success rate: 85%

After:

API call fails → Wait 1s → Retry (3x max)
Success rate: 99%

When to Use:

• Transient failures are common
• Retry doesn't cause problems
• Cost of retry is acceptable

Pattern 5: Staged Processing

Before:

Process all 10,000 items → 2 hours
(Can't stop, can't resume)

After:

Process in batches of 100:
  - Batch 1 → Store progress
  - Batch 2 → Store progress
  - ...
(Can stop/resume anytime)

When to Use:

• Long-running operations
• Risk of failure mid-process
• Need progress tracking

N8N-Specific Optimizations

1. Use Function Nodes for Simple Transforms

Why: Faster than HTTP nodes for basic operations

Before: HTTP node to external API for simple math After: Function node with JavaScript

2. Minimize Data Between Nodes

Why: Large payloads slow execution

Before: Pass entire 10MB dataset After: Pass only IDs, fetch details when needed

3. Use Split In Batches Node

Why: Process large datasets efficiently

[Split In Batches: 100 items]
  ↓
[Process Batch]
  ↓
[Merge Results]

4. Set Appropriate Timeouts

Why: Don't wait forever for failing operations

• Fast APIs: 5-10s timeout
• Slow APIs: 30-60s timeout
• Scraping: 60-120s timeout

5. Use Execute Once Per Item Wisely

Why: Can create unnecessary loops

Slow: Execute once per item for 1000 items = 1000 executions Fast: Process array in single function = 1 execution

Workflow Design Principles

1. Fail Fast

Validate inputs early, fail immediately on bad data.

[Validate Input] → [Expensive Operation]
Not: [Expensive Operation] → [Discover Bad Input] → [Fail]

2. Idempotent Operations

Design steps that can be retried safely.

Good: "Set user status to X" (same result if run twice) Bad: "Increment counter by 1" (different result if run twice)

3. Observable Systems

Add logging and metrics at key points.

[Step 1] → [Log: "Step 1 complete"]
         → [Metric: duration_step1]
         → [Step 2]

4. Graceful Degradation

Design fallbacks for non-critical steps.

[Try: Enrich with API data]
  ↓ (If fails)
[Fallback: Use basic data]
  ↓ (Continue)
[Next Step]

Cost Reduction Checklist

For each workflow, check:

• Are we caching API results?
• Are we batching requests?
• Are we using the cheapest API tier?
• Do we have unnecessary transformations?
• Are we filtering data early?
• Are we using free alternatives?
• Do we have redundant API calls?
• Are we processing duplicates?
• Can we reduce polling frequency?
• Are we storing unnecessary data?

Performance Optimization Checklist

For each workflow, check:

• Can steps run in parallel?
• Are we using the fastest APIs?
• Do we have unnecessary waits?
• Are we transferring minimal data?
• Are we using efficient data formats?
• Do we have database indexes?
• Are we using connection pooling?
• Can we pre-compute results?
• Are we streaming large datasets?
• Do we have proper caching?

Example Optimization Report

## Optimization Report: Lead Hunter Pipeline

### Current Performance
- Execution time: 45 seconds per lead
- Cost: $0.50 per lead
- Success rate: 85%
- Throughput: 100 leads/hour

### Bottlenecks Identified
1. **Sequential skip tracing** (30s)
   - FastPeopleSearch: 15s
   - TruePeopleSearch: 15s
   - Running sequentially

2. **No caching** (Cost: +$0.20/lead)
   - Re-validating known emails
   - Re-checking business patterns

3. **No retry logic** (Success: 85%)
   - API failures not retried
   - Lost 15% of leads

### Recommended Optimizations

#### Quick Wins (Implement Today)

1. **Parallel Skip Tracing**
   - Impact: -20s execution time
   - Effort: 1 hour
   - Savings: 44% time reduction

2. **Add Validation Cache**
   - Impact: -$0.20 per lead
   - Effort: 2 hours
   - Savings: $200/month (1000 leads)

3. **Retry Failed API Calls**
   - Impact: 85% → 98% success
   - Effort: 1 hour
   - Gains: +15% more leads

#### 30-Day Improvements

4. **Batch Property Lookups**
   - Impact: -10s execution time
   - Effort: 1 day
   - Savings: 22% time reduction

5. **Smart Lead Scoring**
   - Impact: Process high-value first
   - Effort: 2 days
   - Business value: +30% conversion

#### 90-Day Vision

6. **Predictive Lead Filtering**
   - Impact: Reduce low-value processing
   - Effort: 1 week
   - Savings: 40% cost reduction

### Projected Results

**After Quick Wins:**
- Execution time: 25s (-44%)
- Cost: $0.30 (-40%)
- Success rate: 98% (+15%)
- Throughput: 144 leads/hour (+44%)

**ROI**: 6 hours effort → $200/month savings = Break-even in 1 day

### Implementation Plan

**Week 1:**
- [ ] Parallel skip tracing (Day 1)
- [ ] Validation cache (Day 2-3)
- [ ] Retry logic (Day 4)
- [ ] Test and deploy (Day 5)

**Weeks 2-4:**
- [ ] Batch property lookups
- [ ] Smart lead scoring

**Month 2-3:**
- [ ] Predictive filtering
- [ ] A/B test optimization

Integration with Other Skills

• data-strategy-architect: For high-level architecture decisions
• error-annihilator: When optimizations cause bugs
• lead-hunter: For domain-specific Lead Hunter optimizations
• roadmap-builder: To prioritize optimization work

Key Metrics to Track

Performance Metrics

• Execution time (p50, p95, p99)
• Throughput (ops/second, ops/hour)
• Latency per step
• Queue depth

Reliability Metrics

• Success rate
• Error rate by type
• Retry frequency
• Recovery time

Cost Metrics

• Cost per operation
• API calls per operation
• Data transfer costs
• Infrastructure costs

Business Metrics

• Cost per lead
• Time to result
• Lead quality score
• Conversion rate

Last Updated: November 21, 2025 Version: 1.0