The Leap in Measurement: Challenging Common Industry Mistakes
Most measurement mistakes aren’t about bad tools or bad data. They’re the same handful of reasoning errors, wearing different disguises depending on which channel or platform is being measured that week.
Look across enough measurement failures, in enough channels, and a pattern emerges that has nothing to do with which platform or methodology was involved. Almost every one of them is the same underlying error: treating something that correlates with an outcome as though it caused that outcome, without ever actually testing the difference.
The one mistake wearing four different disguises
Correlation-versus-causation is a first-year statistics lesson, and yet it’s the single most common failure mode running through marketing measurement at every level, from a single campaign report to an enterprise attribution model. The reason it persists isn’t ignorance, it’s that testing for genuine causation, via a holdout group or a proper incrementality test, is harder and slower than simply reporting what happened to a group that saw an ad. The correlation-based number is always available. The causal one has to be deliberately built.
There’s a career incentive tangled up in this too, worth naming honestly. A correlation-based report almost always looks better than an incrementality-tested one, because a proper test frequently reveals that a channel is driving less genuine lift than its raw numbers implied. Nobody is rewarded for presenting a smaller, more honest number over a larger, more flattering one, which means the bias toward correlation isn’t purely a knowledge gap. It’s also, quietly, an incentive problem.
The four disguises, and where each one shows up
| Where it shows up | What the mistake looks like |
|---|---|
| Multi-touch attribution | Crediting every touchpoint a converting customer encountered, without testing whether removing any single one would have changed the outcome, the exact caveat covered in Multi-Touch Attribution |
| Retail media reporting | Reporting that a customer saw an ad and later bought the product, without a holdout group proving the purchase wouldn’t have happened anyway, covered in Retail Media Networks and Their Impact on Attribution |
| Offline attribution | Matching an online exposure to an offline purchase and treating the match itself as proof of impact, rather than one input into a proper test |
| Finance conversations | Presenting engagement or awareness metrics as if they translate directly into revenue, without the incrementality step covered in Articulating Marketing’s Value to Finance |
Need to put a number on your next media decision?
Model the impact of a media or marketing decision on your own numbers, browse the full library of strategic calculators and decision tools, and get definitions straight on the industry terms that come up along the way, three free resources, ready whenever you need them.
The leap that actually fixes it
The leap isn’t a new tool or a smarter dashboard. It’s a single standing question added to every measurement report before it’s trusted: what would have happened without this. Answering that honestly requires a genuine comparison, a holdout group, a geographic test, a time-based control, not a more detailed breakdown of what already happened to the group that was exposed. Most organisations have the technical capability to run this kind of test already. What’s usually missing is the habit of asking for it before believing a number.
Making incrementality thinking a standing habit, not a special project
Incrementality testing gets treated as a special, occasional initiative, something run once a year on the biggest channel, rather than a standing question applied to ordinary reporting. The more durable fix is smaller and less glamorous: build the “what would have happened without this” question into the default reporting template for every major channel, so the correlation-only version of a number is never the only version anyone sees.
Free Playbook
The Retention Economics Playbook covers building measurement practices grounded in genuine incrementality, the same discipline that separates a defensible number from a comfortable one.
Get the Retention Economics PlaybookThe Leap in Measurement: FAQ
Is correlation-based measurement always wrong?
Not wrong exactly, but incomplete. Correlation-based numbers are genuinely useful as a starting signal, the mistake is treating them as proof of causation without ever testing the difference with a proper holdout or control group.
Do smaller organisations have the resources to run genuine incrementality tests?
Often more than they assume. A basic geographic or time-based holdout test doesn’t require enterprise-scale infrastructure, just a deliberate decision to withhold spend from part of the audience specifically to measure the difference.
How often should incrementality testing happen?
Ideally as a standing element of major channel reporting rather than a once-a-year special project, so correlation-only numbers are never presented as if they were causal ones by default.
