Magnitude Checks (+2)

Magnitude Checks

For any key number in your analysis, verify it passes the "smell test":

Cross-Validation Techniques

1. Calculate the same metric two different ways and verify they match
2. Spot-check individual records -- pick a few specific entities and trace their data manually
3. Compare to known benchmarks -- match against published dashboards, finance reports, or prior analyses
4. Reverse engineer -- if total revenue is X, does per-user revenue times user count approximately equal X?
5. Boundary checks -- what happens when you filter to a single day, a single user, or a single category? Are those micro-results sensible?

Red Flags That Warrant Investigation

• Any metric that changed by more than 50% period-over-period without an obvious cause
• Counts or sums that are exact round numbers (suggests a filter or default value issue)
• Rates exactly at 0% or 100% (may indicate incomplete data)
• Results that perfectly confirm the hypothesis (reality is usually messier)
• Identical values across time periods or segments (suggests the query is ignoring a dimension)