How to Design a Geo Holdout Test for Meta Ads

Proving Meta's actual value with a geo holdout

Meta tells you your campaigns generate a 3.5x ROAS. Your attribution tool says 2.8x. But neither number answers the real question: if you turned off Meta entirely, how much revenue would you lose?

A geo holdout test answers that question directly. You pause Meta Ads in some regions, keep them running in others, and measure the revenue difference. The result is Meta's true incremental impact -- the revenue that wouldn't exist without Meta.

This guide walks through the exact process for designing and running a geo holdout test specifically for Meta Ads.

Why Meta specifically needs incrementality testing

Meta's self-reported numbers are among the most inflated in digital advertising. Three structural factors drive the inflation:

View-through attribution. Meta's default 1-day view window claims conversions from people who saw (but never clicked) your ad. For brands with high organic traffic, this can add 15-25% phantom conversions.

Post-iOS 14 modeling. With 75%+ of iOS users opted out, Meta models a significant portion of conversions. These models lean optimistic because conservative modeling would reduce reported ROAS and Meta's ad revenue.

Cross-platform credit-taking. Someone discovers your brand through a friend's recommendation, sees a Meta ad while scrolling, and then Googles your brand. Meta claims the conversion. Google claims the conversion. Neither actually caused it.

A geo holdout test cuts through all of this by measuring what actually happens when Meta ads are absent.

Step-by-step test design

Step 1: Choose your test type

Full holdout (recommended for first test): Completely pause Meta Ads in control regions. This gives you the cleanest measurement of Meta's total incremental impact.

Spend reduction holdout: Reduce Meta spend by 50% in control regions instead of fully pausing. This is less disruptive but produces a smaller signal that's harder to detect statistically.

Channel-specific holdout: Pause only specific campaign types (e.g., prospecting only, while keeping retargeting active). This measures the incrementality of specific campaign strategies.

For your first test, go with a full holdout. It produces the clearest, most actionable signal.

Step 2: Select geographic regions

Use DMAs (Designated Market Areas) as your geographic unit. DMAs align with how Meta delivers ads and how most brands analyze regional performance.

Region selection criteria:

• Similar historical revenue per capita
• Similar Meta ad delivery and performance metrics
• Similar demographics and market characteristics
• No major local factors that would skew results (new store openings, competitor activity)

How many regions? Aim for 10-15 control regions (where Meta is paused) and 10-15 test regions (where Meta continues). More regions = more statistical power = more reliable results.

Allocation tip: Don't make your top-performing regions the control group. Pausing Meta in your highest-revenue markets maximizes your financial risk. Instead, distribute your best and worst regions evenly across test and control groups.

Step 3: Establish baselines

Pull 8-12 weeks of historical data for each region:

• Total revenue by region by week
• Meta-attributed revenue by region by week
• Conversion counts by region by week
• Traffic by source by region by week

Calculate the historical ratio between test and control groups. If the test group has historically generated 1.8x the revenue of the control group, that ratio is your baseline. During the test, you'll measure whether the ratio changes.

Step 4: Configure the holdout in Meta

In Meta Ads Manager:

1. Create a location exclusion list containing all your control DMAs/regions
2. Apply this exclusion to all active Meta campaigns
3. Verify that ad delivery stops in control regions (check delivery by region in reporting)

Important details:

• Exclude control regions at the campaign level, not the ad set level, to prevent Meta from reallocating budget
• If you use automated rules or CBO, ensure the budget doesn't shift to compensate for the excluded regions
• Keep all other campaign settings identical in test regions

Step 5: Determine test duration

For Meta Ads, a geo holdout typically needs 3-4 weeks to produce reliable results. The calculation depends on:

• Your weekly conversion volume: 500+ conversions/week across all regions = 3 weeks is usually sufficient. Under 200/week = you may need 5-6 weeks.
• Expected incremental impact: If Meta drives a large share of your revenue, the signal will be strong and easier to detect. If Meta drives a small share, you need a longer test.
• Day-of-week effects: Run the test in full-week increments to avoid day-of-week bias.

Pre-test power analysis: Use your historical data to simulate the test. If you expect Meta to drive a 20% lift in test regions, calculate how many weeks you need for your specific conversion volumes to detect that lift at 90% confidence.

Step 6: Run the test

During the test period:

Do:

• Monitor daily for anomalies (but don't make changes)
• Track revenue by region from your backend (Shopify, CRM, etc.) -- not from ad platforms
• Continue running all non-Meta campaigns identically in both groups
• Keep creative and offer schedules the same

Don't:

• Adjust Meta campaigns during the test
• Launch new channels or major promotions in test regions only
• Change your website, pricing, or offers during the test
• React to early data

Step 7: Analyze results

After the test period, calculate:

Incremental revenue from Meta:

1. Calculate the actual revenue ratio between test and control during the test
2. Compare to the historical baseline ratio
3. The difference in ratios, applied to total revenue, is Meta's incremental contribution

Example:

• Historical ratio (test:control): 1.80
• During test ratio (test:control): 2.15
• The increase of 0.35 means test regions (with Meta running) outperformed by 19.4% more than historical baseline
• If control regions generated $200K during the test, the incremental Meta revenue in test regions is approximately $39K

Incremental ROAS:

• Incremental revenue / Meta ad spend during the test period
• This is your true Meta ROAS -- not platform-reported, not modeled, but measured

Statistical significance:

• Run a difference-in-differences analysis
• Check that the result is significant at 90-95% confidence
• If not significant, the test was either too short or Meta's incremental impact is too small to detect at your current scale

What to do with the results

If incremental ROAS is strong (above target)

Meta is genuinely driving revenue. Your attribution model may be undercounting Meta's true impact (if the geo test shows higher incrementality than your model). Consider increasing Meta spend, especially on campaign types that were active during the test.

If incremental ROAS is weak (below target)

Meta's reported numbers are significantly inflated beyond what you already suspected. This doesn't mean cut Meta entirely -- it means recalibrate your expectations and targets. A 2x incremental ROAS might be perfectly profitable even if Meta reports 4x.

If results are inconclusive

Either your test lacked statistical power (too few regions, too short duration) or Meta's incremental impact is genuinely small. Design a longer or larger test, or consider whether your Meta budget is primarily capturing demand that would exist without it.

Common mistakes specific to Meta geo holdouts

Forgetting organic social. Pausing Meta Ads doesn't pause your organic Facebook/Instagram content. If organic posts drive traffic in control regions, you're measuring paid incrementality, not total Meta incrementality. This is fine for most purposes, but be clear about what you're measuring.

Ignoring halo effects. Meta awareness campaigns in test regions might drive Google searches that happen in control regions (e.g., someone sees your ad in a test region and tells a friend in a control region). This cross-region contamination understates Meta's true impact.

Using Meta's own conversion data. Measure results using backend revenue data (Shopify, bank deposits), not Meta's reported conversions. Using Meta's data to evaluate Meta defeats the purpose.

FAQ

Will pausing Meta hurt my account's learning?

Yes, temporarily. When you restart Meta in control regions after the test, expect 1-2 weeks for the algorithm to re-learn. Budget for this re-learning period. The insight gained from the test is worth the temporary efficiency dip.

Can I run a geo holdout on a $20K/month Meta budget?

It's challenging. At $20K/month, splitting across 20+ regions means each region gets $1K/month in spend. The signal may be too small to detect reliably. Geo holdouts work best at $50K+/month in Meta spend. Below that, consider Meta's native Conversion Lift study instead.

How does this compare to Meta's own lift studies?

Meta offers Conversion Lift studies that use a similar methodology but are run within Meta's infrastructure. The advantage of running your own geo holdout: you control the design, you use your own backend data, and there's no platform bias in the measurement. The disadvantage: more work to set up and analyze.

---

Go Funnel uses server-side tracking and multi-touch attribution to show you which ads actually drive revenue. Book a call to see your real numbers.