Tradeoff Decision Workflow

Purpose

Make informed ship/no-ship/iterate decisions when A/B test results are mixed (some metrics up, others down) or when a launched feature shows conflicting performance. Avoids binary thinking by exploring nuanced approaches that preserve gains while addressing losses.

When to Use This Workflow

Use this workflow when:

• A/B test shows some metrics increasing, others decreasing
• Feature launched with mixed performance across different metrics
• Stakeholders disagree on whether to ship or rollback
• Redesign shifted user behavior in unexpected ways
• Need to choose between two good but mutually exclusive options
• Optimizing for one metric clearly harms another

Skills Sequence

This workflow applies four skills systematically:

1. North Star Alignment
   ↓ (Determine which metric matters more strategically)
2. Trade-off Evaluation (Short vs. Long-term)
   ↓ (Assess temporal trade-offs)
3. Trade-off Evaluation (Mitigation)
   ↓ (Explore "best of both worlds" approaches)
4. Funnel-Based Metric Mapping
   ↓ (Identify where in user journey problem occurs)
   
OUTPUT: Assessment (net positive/negative/depends), 
        Recommended action (ship/rollback/iterate),
        Mitigation strategies, Risks

Required Inputs

Gather this information before starting:

Metric Changes

• Metrics that increased

  • Which metrics went up?
  • By how much (absolute and %)?
  • Statistical significance?

• Metrics that decreased

  • Which metrics went down?
  • By how much (absolute and %)?
  • Statistical significance?

Context

• Change description

  • What was tested or launched?
  • What was the hypothesis?
  • How long has it been running?

• Company strategic priorities

  • What's most important to company now?
  • Growth, monetization, retention, competitive positioning?

• User journey context

  • Where in the funnel does change occur?
  • Which user behaviors shifted?

Current State

• Rollback feasibility

  • Can we easily revert?
  • What's the cost of reverting?
  • How many users affected?

• Iteration options

  • Can we modify the change?
  • What variations could we test?
  • Timeline for iteration?

Workflow Steps

Step 1: Strategic Alignment Assessment (15 minutes)

Use the north-star-alignment skill

Determine which metric matters more to company strategy:

Questions to answer:

1. Which metrics align with North Star?

   • Do positive metrics connect to company-level goals?
   • Do negative metrics impact North Star metrics?

2. Which metrics align with mission?

   • Does change serve company mission?
   • Does it strengthen core value proposition?

3. Which metrics drive long-term competitive advantage?

   • Does change improve differentiation?
   • Does it strengthen market position?

4. Which metrics feed important flywheels?

   • Do positive metrics create virtuous cycles?
   • Do negative metrics break retention loops?

Output:

## Strategic Alignment Assessment

**Metrics Aligned with Strategy:**
- [Metric that went up/down]: [Why strategically important]
- [Metric that went up/down]: [Why strategically important]

**Strategic Priority Ranking:**
1. [Most important metric]: [Rationale]
2. [Second priority]: [Rationale]
3. [Lower priority]: [Rationale]

**Mission Alignment:**
- Change serves mission: [Yes/No/Partially]
- Rationale: [Explanation]

**Competitive Positioning:**
- Strengthens positioning: [Yes/No/Mixed]
- Rationale: [Explanation]

Step 2: Temporal Trade-off Analysis (15 minutes)

Use the tradeoff-evaluation skill - Short vs. Long-term Framework

Evaluate whether this is short-term gain/long-term loss or vice versa:

Framework questions:

1. Short-term Impact (this quarter)

   • Immediate revenue effect?
   • This quarter's growth?
   • Current user satisfaction?

2. Long-term Impact (12+ months)

   • User retention trajectory?
   • Flywheel effects?
   • Sustainable competitive advantage?
   • Brand and market position?

3. Irreversibility Assessment

   • Can we reverse course later if needed?
   • What becomes locked in?
   • What habits are we building?

Apply Snapchat test:

• Does this improve vanity metrics (time-on-site) while breaking engagement flywheel (messages/stories)?
• Are we optimizing for wrong thing?

Output:

## Temporal Analysis

**Short-term (0-3 months):**
- Revenue impact: [Positive/Negative/Neutral - magnitude]
- Growth impact: [Positive/Negative/Neutral - magnitude]
- User satisfaction: [Positive/Negative/Neutral]

**Long-term (12+ months):**
- Retention impact: [Projected effect]
- Flywheel effects: [Strengthened/Weakened/Neutral]
- Competitive position: [Stronger/Weaker/Same]
- Strategic alignment: [High/Medium/Low]

**Trade-off Classification:**
- [Short-term gain, long-term loss] → Risky
- [Short-term loss, long-term gain] → Consider if affordable
- [Short-term gain, long-term gain] → Ship
- [Short-term loss, long-term loss] → Don't ship

**Recommendation based on temporal analysis:**
[Ship/Don't ship/Depends on mitigation]

Step 3: Mitigation Strategy Exploration (20-30 minutes)

Use the tradeoff-evaluation skill - Mitigation Framework

Before deciding to rollback or ship as-is, explore "best of both worlds" options:

Mitigation strategy categories:

Strategy 1: Segmented Rollout

When to use: Impact varies by user segment

Analysis:

• Which segments show net positive?
• Which segments show net negative?
• Can we ship to positive segments only?

Implementation:

• Define segment criteria
• Rollout plan for positive segments
• Iteration plan for negative segments

Strategy 2: Feature Modification

When to use: Specific element causing harm

Analysis:

• Which specific change caused negative impact?
• Can we keep beneficial parts, modify harmful parts?
• What variations could preserve gains while fixing losses?

Implementation:

• Identify problematic component
• Design modified version
• A/B test: Original vs. Modified vs. Current

Strategy 3: Compensation Mechanisms

When to use: Trade-off is necessary but painful

Analysis:

• What complementary features could compensate?
• Can we create new paths to value?
• Can we boost other areas to offset loss?

Implementation:

• Identify compensation opportunities
• Parallel feature development
• Monitor net effect

Strategy 4: Gradual Rollout with Monitoring

When to use: Uncertain about long-term effects

Analysis:

• What could we learn from gradual rollout?
• What delayed effects might emerge?
• What monitoring would de-risk?

Implementation:

• 10% → 25% → 50% over weeks
• Maintain holdback group
• Set rollback trigger criteria

Strategy 5: Threshold-Based Approach

When to use: Acceptable within certain bounds

Analysis:

• What magnitude of negative impact is acceptable?
• What thresholds trigger concerns?
• Can we monitor and revert if crossed?

Implementation:

• Define acceptable thresholds
• Set up automated alerts
• Establish rollback criteria

Output:

## Mitigation Strategies

### Evaluated Strategies:

**Strategy 1: [Name]**
- Approach: [Description]
- Preserves: [Positive metrics]
- Addresses: [Negative metrics]
- Feasibility: [Easy/Medium/Hard]
- Timeline: [How long to implement]
- Risks: [What could go wrong]

**Strategy 2: [Name]**
[Same structure]

**Strategy 3: [Name]**
[Same structure]

### Recommended Mitigation:
**[Strategy name]**
- Why: [Rationale]
- Implementation: [Specific steps]
- Success criteria: [How to know it worked]
- Fallback plan: [If mitigation fails]

Step 4: Funnel Location Analysis (10 minutes)

Use the funnel-metric-mapping skill

Identify where in the user journey the trade-off occurs:

Questions:

1. Which funnel stage is affected?

   • Reach, Activation, Engagement, Retention?
   • Multiple stages?

2. Is impact concentrated or distributed?

   • Single point of friction?
   • Systemic shift across journey?

3. Do metrics at different stages conflict?

   • Earlier stage improved, later stage harmed?
   • Suggests optimization for wrong outcome

4. What transitions are affected?

   • Conversion rates between stages changed?
   • Drop-off points shifted?

Output:

## Funnel Impact Analysis

**Affected stages:**
- [Stage name]: [Metric change description]
- [Stage name]: [Metric change description]

**Transition impacts:**
- [Stage A → Stage B]: [Conversion rate change]

**Concentration:**
- [Single point / Distributed across journey]

**Insights:**
- [What funnel analysis reveals about trade-off]

Step 5: Make Recommendation (10 minutes)

Synthesize all analyses into clear recommendation:

Decision options:

1. Ship Fully

   • When: Net positive across segments, strategic alignment high
   • Action: Roll out to 100%
   • Monitoring: Track counter-metrics

2. Ship to Segments

   • When: Positive for some users, negative for others
   • Action: Roll out to net-positive segments
   • Plan: Iterate for problematic segments

3. Iterate First

   • When: Promising but needs modification
   • Action: Implement mitigation strategy
   • Timeline: Test modified version before full ship

4. Gradual Rollout

   • When: Uncertain about long-term effects
   • Action: Staged rollout with monitoring
   • Triggers: Rollback criteria defined

5. Rollback

   • When: Net negative, no clear mitigation, high risk
   • Action: Revert to previous state
   • Learning: Document why for future

Recommendation document:

# Trade-off Decision: [Change Name]

## Executive Summary
[One paragraph: Metrics moved, strategic assessment, recommendation]

## Metric Changes
**Positive:**
- [Metric]: +X% ([why this matters])
- [Metric]: +X% ([why this matters])

**Negative:**
- [Metric]: -X% ([why this matters])
- [Metric]: -X% ([why this matters])

## Strategic Assessment (Step 1)
**Most important metric:** [Name]
**Rationale:** [Why strategically critical]
**Alignment:** [High/Medium/Low]

## Temporal Analysis (Step 2)
**Short-term:** [Net positive/negative]
**Long-term:** [Net positive/negative]
**Trade-off type:** [Classification]

## Mitigation Options (Step 3)
**Explored:**
1. [Strategy name]: [Feasibility]
2. [Strategy name]: [Feasibility]

**Recommended:** [Strategy name]

## Funnel Impact (Step 4)
**Affected stage:** [Stage name]
**Insight:** [What this reveals]

## Recommendation

**Decision:** [Ship Fully / Ship to Segments / Iterate First / Gradual / Rollback]

**Rationale:**
- [Point 1]
- [Point 2]
- [Point 3]

**Implementation:**
- Action: [Specific steps]
- Owner: [Name/team]
- Timeline: [When]
- Success criteria: [How to measure]

**Risks:**
- Risk 1: [Description + mitigation]
- Risk 2: [Description + mitigation]

**Monitoring Plan:**
- Metrics to track: [List]
- Review frequency: [Daily/Weekly]
- Rollback triggers: [Conditions that require revert]
- Responsible: [Name/team]

## Appendices
- A/B test results: [Detailed data]
- Segment analysis: [Breakdown]
- Stakeholder input: [Summary]

Decision Matrix

Use this matrix for quick assessment:

Common Mistakes

Success Criteria

Tradeoff decision succeeds when:

• Strategic alignment assessed (which metric matters more)
• Temporal trade-offs evaluated (short vs. long-term)
• 3-5 mitigation strategies explored
• Funnel impact analyzed
• Clear recommendation made with rationale
• Implementation plan defined
• Risks identified with mitigation plans
• Monitoring plan established
• Stakeholders aligned on decision

Real-World Example: Snapchat Redesign

Context

• Redesign launched: Separate social from media
• Time on site: ↑ 15%
• Messages sent: ↓ 12%
• Stories shared: ↓ 8%

Step 1: Strategic Alignment (15 min)

Questions:

• Which metric aligns with Snapchat's strategic positioning?
• Camera/social app or content consumption app?

Analysis:

• Content app positioning (time-on-site):

  • Market saturated (TikTok, YouTube, Instagram)
  • Not core competency
  • Short-term thinking

• Social/camera app positioning (messages/stories):

  • Unique value proposition
  • Core competency
  • Sustainable competitive advantage

Conclusion: Messages and stories strategically more important than time-on-site

Step 2: Temporal Analysis (15 min)

Short-term (this quarter):

• Revenue: Positive (more ads shown with higher time-on-site)
• Growth: Neutral
• User satisfaction: Mixed (some like creators, some miss social)

Long-term (12 months):

• Retention: Negative risk (broken social flywheel)
  • Fewer stories → Fewer notifications → Less app opening
  • Fewer messages → Less reciprocal engagement
• Flywheel: Weakened critical loop
• Competitive position: Confused (competing with TikTok in saturated market)

Classification: Short-term gain (time/ads), long-term loss (retention/flywheel)

Assessment: Net negative

Step 3: Mitigation Strategies (25 min)

Strategy 1: Immediate Rollback

• Pros: Restores social flywheel quickly
• Cons: Loses time-on-site gains, admits failure
• Feasibility: Easy

Strategy 2: Targeted Modifications

• Approach: Keep creator content but re-integrate friend stories
• Implementation:
  • Show friend stories in creator tab
  • Add "message friend" prompts after content viewing
  • Balance algorithm between creator and friend content
• Preserves: Time-on-site gains
• Addresses: Social engagement losses
• Feasibility: Medium complexity, 4-6 weeks

Strategy 3: Segmented Experience

• Heavy social users: Friend-focused interface
• Content consumers: Creator-focused interface
• Feasibility: Complex, 8-12 weeks

Recommended: Strategy 2 (Targeted Modifications)

• Best balance of preserving gains and fixing losses
• Reasonable timeline
• Testable before full rollout

Step 4: Funnel Analysis (10 min)

Affected stages:

• Engagement (Depth): Messages and stories down (core actions)
• Engagement (Breadth): Time on site up (passive consumption)

Insight:

• Optimizing breadth at expense of depth
• Depth drives retention flywheel, breadth doesn't
• Classic mistake: Vanity metric over strategic metric

Step 5: Recommendation (10 min)

Decision: Iterate First (Don't immediately rollback, don't keep as-is)

Rationale:

1. Clear strategic importance of social metrics
2. Long-term retention risk outweighs short-term revenue gain
3. Feasible mitigation strategy exists
4. Can preserve some gains while fixing critical issue

Implementation:

• Action: Implement targeted modifications (Strategy 2)
• Owner: Product team + Design
• Timeline: 4-6 weeks for modified version
• Success criteria:
  • Messages sent: Recover to within 5% of baseline
  • Stories shared: Recover to within 5% of baseline
  • Time on site: Maintain at least 50% of gains

Risks:

• Risk: Modifications don't fix social engagement
  • Mitigation: Full rollback if metrics don't improve in 30 days
• Risk: Lose all time-on-site gains
  • Mitigation: Acceptable trade-off for strategic alignment

Monitoring:

• Daily: Messages, stories, time-on-site
• Weekly: Full funnel review
• Rollback trigger: If messages/stories drop further 5%

Timeline: 90 minutes for full analysis

Related Skills

This workflow orchestrates these skills:

• north-star-alignment (Step 1)
• tradeoff-evaluation (Steps 2-3)
• funnel-metric-mapping (Step 4)

Related Workflows

• metric-diagnosis: Use first if root cause of metric changes unclear
• metrics-definition: Defines counter-metrics to detect trade-offs early
• dashboard-design: Monitors trade-offs through balanced metric sets

Time Estimate

Total: 70-90 minutes

• Step 1 (Strategic alignment): 15 min
• Step 2 (Temporal analysis): 15 min
• Step 3 (Mitigation): 20-30 min
• Step 4 (Funnel analysis): 10 min
• Step 5 (Recommendation): 10 min