What Is Your Paid Media Actually Worth? Lessons From an 8-Week Incrementality Study
A national consumer brand spending ~$300K a month wanted a real answer — not the platform-reported one. We ran an 8-week matched-market geo-holdout across 32 DMAs to find out. Here is what the test revealed about Google, Meta, Pinterest, and how the platform numbers compare to true incremental return.— The CMO, before a Q2 board presentation
01 · The ProblemStrong reported numbers. A nagging question.
The client — a national manufacturer of residential plumbing fixtures, ~$180M annual retail sell-through, top-5 share in its category — was spending $280–320K a month across four paid media channels. Platform-reported numbers looked good.
| Channel | Monthly Spend | Platform-Reported ROAS |
|---|---|---|
| Google Ads (Search + Shopping + PMax) | $135,000 | 6.8x |
| Meta Ads (Prospecting + Retargeting) | $88,000 | 4.2x |
| Pinterest Ads (Awareness + Catalog) | $32,000 | 2.1x |
| Retail Media (sponsored products) | $45,000 | 8.4x |
| Total | ~$300,000 | 5.3x blended |
But a 5.3x blended ROAS was suspect. The brand had strong organic search, a high repeat purchase rate, and big in-store shelf presence. Platform-reported ROAS counts every conversion an ad touches — not every conversion the ad caused. Google’s PMax campaigns were the loudest tell: more than 40% of PMax conversions came from branded search terms, where the buyer was almost certainly going to convert anyway.
The CMO needed a defensible answer for the board: How much of this revenue is really incremental?
02 · ApproachA geo-holdout, because the answer had to include retail.
We chose a matched-market geo-holdout over user-level holdouts or platform-native lift studies, for three reasons:
- The client sells through retail. We needed to measure offline lift, not just DTC conversions.
- A geo test captures the full-funnel effect — including people who see an ad and then buy in-store.
- It avoids the platform-specific limitations of conversion lift studies, which only measure inside one platform’s walls.
How we ran the test
From 210 US DMAs, we narrowed to 68 candidates (excluding the top-15 metros, markets with active retail co-op, distribution gaps, or known competitor activity). We propensity-matched the remaining DMAs on sales volume, trend, density, income, category and brand development indices, paid media spend per capita, and seasonality — then split the matched pairs into a 16-DMA test group (~12.6M households) and a 16-DMA control group (~12.8M households, within 1.3% of test on baseline sales).
For 8 weeks (January 13 – March 9, 2025), we suppressed Google, Meta, and Pinterest spend in the control DMAs while continuing to spend normally in test markets. A 6-week pre-period established the baseline; a 3-week post-period confirmed the rebound when media was restored. Retail media stayed on in both groups (the platforms don’t support geographic suppression) — so the study measures incrementality of Google + Meta + Pinterest specifically, not all paid media.
Methodology detail (matching, statistical framework, limitations)
Estimator. Primary: paired-difference test on matched-pair sales differences (test minus control, post minus pre). Robustness: pooled difference-in-differences regression with DMA and week fixed effects, standard errors clustered at the DMA level using wild cluster bootstrap (Cameron, Gelbach, Miller 2008) to address the small-cluster problem (n=32 DMAs).
Power. Pre-study analysis indicated MDE of 8% at 80% power (alpha = 0.10, one-sided). We chose alpha = 0.10 over 0.05 deliberately — with 16 DMAs per cell, 95% confidence is unrealistic.
Data sources. Weekly retailer POS data from two national chains (covering ~65% of unit volume); GA4 for DTC revenue by metro; platform ad servers for spend, impressions, and reported conversions; Dynata survey panel (400 respondents per cell, pre- and post-period waves) for awareness; SEMrush + Pathmatics for competitive monitoring.
Channel decomposition. Because all three channels were suppressed simultaneously, channel-level iROAS estimates are modeled, not directly measured — derived from time-series correlation analysis plus a supplementary 2-week Google-only holdout in 4 DMAs during weeks 5–6. The total-media lift is statistically grounded; the channel-level shares should inform directional budget decisions, not precise allocation targets.
03 · The FindingsReal lift. And a chart that landed in the boardroom.
Sales lift (test vs. control, difference-in-differences)
| Metric | Test | Control | Lift | 90% CI |
|---|---|---|---|---|
| Retail POS sales (indexed, per-store avg) | 114.5 | 100.0 | +14.5% | [9.2%, 19.8%] |
| DTC eCommerce revenue (indexed, per-DMA avg) | 124.2 | 100.0 | +24.2% | [11.7%, 36.8%] |
| Combined sales (indexed) | 115.2 | 100.0 | +15.2% | [10.1%, 20.3%] |
Combined lift of 15.2%, statistically significant at the 90% confidence level. DTC lifted more (digital ads → online conversion is a shorter path), but retail POS also moved meaningfully — confirming that digital media drives in-store purchases, not just web sessions. Pre-period parallel trends held tight: test and control DMAs tracked within 1.3% of each other for 6 weeks before the test began (matched-pair correlation 0.94).
Platform-reported ROAS vs. incremental ROAS
This is the chart that changed the conversation:
| Channel | Platform-Reported ROAS | Incremental ROAS | Reported vs. Incremental |
|---|---|---|---|
| Google Ads | 6.8x | 3.1x | 2.2x over-reported |
| Meta Ads | 4.2x | 4.4x | ~ accurate |
| Pinterest Ads | 2.1x | 6.2x | 3x under-reported |
| Blended | 5.3x | 3.8x | 1.4x over-reported |
Google’s reported 6.8x collapsed to a true incremental 3.1x — more than half of Google’s claimed conversions would have happened anyway, mostly via branded search and remarketing pathways. Meta was honest with itself: 4.2x reported vs. 4.4x incremental (likely slightly under-reported, since Meta drives awareness that converts via organic and direct). Pinterest was dramatically undervalued — 2.1x reported, but 6.2x incremental. The blended truth: $3.80 of incremental revenue per $1 spent, vs. the $5.30 the platforms claimed.
04 · What Was SurprisingFour findings the team didn’t expect.
Google PMax was the biggest over-reporter
PMax showed a platform-reported ROAS of 9.1x but contributed the least incremental lift per dollar within Google. Standard Shopping and non-brand Search were materially more efficient. This matched industry-wide concerns about PMax claiming credit for branded search and remarketing.
Pinterest was severely undervalued
The client had been considering cutting Pinterest entirely on the strength of its 2.1x platform ROAS. The geo-test showed it was the most incrementally efficient channel in the mix. Working hypothesis: Pinterest reaches users earlier in the home-renovation consideration cycle, where competition for attention is lower.
Retail lift kept pace with DTC lift
We expected DTC to dominate — ads point directly to the website. The fact that retail POS sales lifted nearly as much (+14.5% vs. +24.2%) suggests substantial cross-channel impact: people see ads online and buy in-store. This is invisible to most attribution stacks.
Control markets didn’t crater
Sales in control markets (with all three digital channels off) only declined ~3% vs. pre-period baseline. The brand has strong organic demand. But the 15% gap between test and control proves paid media is additive, not just redistributing demand the brand would have captured anyway.
05 · What the Client DidReallocation, not retreat.
The findings drove an immediate Q2 budget reallocation. Total spend stayed roughly flat (~$305K vs. ~$300K), but the mix shifted toward incrementally efficient channels.
| Channel | Q1 Spend | Q2 Spend | Change | Restructure |
|---|---|---|---|---|
| Google Ads | ~$135K | ~$118K | −12% | PMax cut 30%, shifted to standard Shopping + non-brand Search; stricter brand exclusions |
| Meta Ads | ~$88K | ~$94K | +8% | Prospecting +15%, retargeting frequency capped; new Advantage+ Shopping campaign |
| Pinterest Ads | ~$32K | ~$48K | +40% | Added video and carousel formats; expanded into upper-funnel awareness |
| Retail Media | ~$45K | ~$45K | — | Unchanged (not part of test); separate study planned for H2 |
What changed in how the client measures performance
- Retired platform-reported ROAS as the primary KPI. Still tracked for campaign management; board reporting now uses incremental ROAS benchmarks calibrated from the geo-test.
- Quarterly incrementality pulse. A 3-week lightweight holdout in 4–6 DMAs every quarter, to re-validate that the ratios haven’t shifted. Calibration, not full re-study.
- Bayesian MMM, calibrated to geo-test ground truth. Built in Meridian (Google’s open-source MMM tool), with the geo-test serving as the truth source that keeps the model honest between formal lift studies.
Our paid media is generating $3.80 in incremental revenue for every dollar invested. That’s lower than the $5.30 the platforms report, but it’s real and it’s profitable. We’re now reallocating toward the channels where the incremental return is highest.
The CMO · Q2 board presentation
The board approved a 10% paid media budget increase for H2 2025, contingent on continued incrementality measurement.
06 · LessonsWhat we’d take into the next study.
- DMA matching is the make-or-break step. A clean 6-week pre-period and tight propensity matching gave us much narrower confidence intervals than we’d have gotten from a simple test-vs-control split. If the matching is sloppy, the rest of the study doesn’t matter.
- Run channel-specific holdouts sequentially next time. Suppressing all three channels at once gave us a clean total-media number but made channel decomposition a modeling exercise. A staggered design (Google alone for 4 weeks, then Google + Meta for 4 weeks) would produce directly measured channel-level estimates — at the cost of a longer overall test.
- Retailer POS access is the unlock. Without weekly sell-through data from the client’s two largest retail partners, the study would have been DTC-only — and DTC is 8% of revenue. Always confirm retail data access before scoping the engagement.
- 90% confidence is the realistic target for studies with 15–20 DMAs per cell. Don’t promise 95% unless the brand has 30+ DMAs per group. Setting the right expectation up-front is what kept the client from killing the study at the midpoint when the confidence interval was still wide.
