When to Use (+4)

When to Use

Use hypothesis testing when you need to determine whether an observed difference is likely real or could be due to random chance. Common scenarios:

• A/B test results: Is variant B actually better than A?
• Before/after comparison: Did the product change actually move the metric?
• Segment comparison: Do enterprise customers really have higher retention?

The Framework

1. Null hypothesis (H0): There is no difference (the default assumption)
2. Alternative hypothesis (H1): There is a difference
3. Choose significance level (alpha): Typically 0.05 (5% chance of false positive)
4. Compute test statistic and p-value
5. Interpret: If p < alpha, reject H0 (evidence of a real difference)

Common Tests

Practical Significance vs. Statistical Significance

Statistical significance means the difference is unlikely due to chance.

Practical significance means the difference is large enough to matter for business decisions.

A difference can be statistically significant but practically meaningless (common with large samples). Always report:

• Effect size: How big is the difference? (e.g., "Variant B improved conversion by 0.3 percentage points")
• Confidence interval: What's the range of plausible true effects?
• Business impact: What does this translate to in revenue, users, or other business terms?

Sample Size Considerations

• Small samples produce unreliable results, even with significant p-values
• Rule of thumb for proportions: Need at least 30 events per group for basic reliability
• For detecting small effects (e.g., 1% conversion rate change), you may need thousands of observations per group
• If your sample is small, say so: "With only 200 observations per group, we have limited power to detect effects smaller than X%"