How to Run Your First Incrementality Test in 7 Days

Stop guessing whether your ads work. Test it in a week.

You've seen the platform ROAS numbers. You've wondered whether they're real. An incrementality test gives you the answer -- and you can set one up in a single day, get results in a week, and make budget decisions with actual causal data instead of correlation.

This is the fastest path from "I think my ads work" to "I know my ads generate X incremental conversions per dollar."

Day 1: Choose your test and set the parameters

Pick the right campaign to test

Start with your largest single campaign or channel. The reason is statistical: you need enough conversions in both the test and holdout groups to reach significance. A campaign generating 10 conversions per day will give you reliable data in 7 days. A campaign generating 2 conversions per day will need 30+ days.

The best first test candidates:

• Retargeting campaigns -- they typically show the biggest gap between platform-reported and incremental ROAS
• Your highest-spend prospecting campaign -- if it's not incremental, that's your biggest savings opportunity
• Branded search -- quick to test and the results are often eye-opening

Calculate your required sample size

You need a minimum detectable effect (MDE) before you start. Here's the shortcut:

• If your campaign drives 50+ conversions/day, a 10% holdout for 7 days gives you enough data to detect a 20% lift
• If your campaign drives 20-50 conversions/day, use a 15% holdout for 7 days to detect a 25% lift
• If your campaign drives 10-20 conversions/day, use a 20% holdout for 14 days to detect a 30% lift

Below 10 conversions per day, a 7-day test won't work. Extend to 21-28 days or pick a higher-volume campaign.

Set your holdout group

On Meta: Use the built-in Conversion Lift test. Go to Experiments in Ads Manager, select "Conversion Lift," and Meta will randomly split your audience into test (sees ads) and holdout (doesn't see ads). Meta handles the randomization, which eliminates selection bias.

On Google: Google's Conversion Lift is available through your Google rep. For self-serve, use a geographic holdout -- pause ads in 2-3 DMAs that match your active markets in demographics and historical conversion rates.

Cross-platform: If you want to test your entire paid media portfolio, geographic holdouts are the only option. Select holdout markets representing 10-15% of your revenue, pause all paid media in those markets, and compare to active markets.

Day 2: Launch and verify the test is running

Pre-launch checklist

Before flipping the switch:

• [ ] Confirm your holdout group size matches your sample size calculation
• [ ] Set up tracking for the primary conversion event (purchase, not clicks or add-to-carts)
• [ ] Record baseline conversion rates for both groups from the prior 7-day period
• [ ] Document any other marketing activities running during the test period (email campaigns, influencer launches, PR) that could affect results
• [ ] Ensure no other tests or bid changes overlap with the test period

Launch-day verification

After starting the test:

• Confirm the holdout group is actually being excluded (check impression delivery reports)
• Verify conversion tracking is firing for both groups
• Set a calendar reminder for Day 7 to pull results

Do not touch the campaign during the test. No bid changes. No budget adjustments. No new creative. Any mid-test changes invalidate the results.

Days 3-6: Let the data accumulate

This is the hardest part. Do nothing.

Resist the urge to check results daily. Daily fluctuations will mislead you. E-commerce conversion rates vary by day of week, and a Monday-only snapshot will look very different from a full-week view.

Use this time to prepare your analysis framework:

Set up your results spreadsheet

Create columns for:

| Metric | Treatment Group | Holdout Group | Incremental | |--------|----------------|---------------|-------------| | Users | | | | | Conversions | | | | | Conversion Rate | | | | | Revenue | | | | | Incremental Lift % | | | | | Incremental CPA | | | | | Incremental ROAS | | | |

Know your significance threshold

For business decisions, you need 90% confidence minimum, 95% preferred. Calculate statistical significance using a two-proportion z-test or an online calculator (Evan Miller's is the standard).

If your p-value is above 0.10, the test is inconclusive. Don't treat it as "no lift" -- it means you need more data. Extend the test or increase the holdout percentage.

Day 7: Pull and analyze results

Step 1: Extract the raw numbers

From Meta's Conversion Lift tool, the results are pre-calculated. For geographic tests, pull conversion data for your holdout markets and treatment markets separately.

Critical: use a consistent conversion window. If you're comparing 7 days of ad exposure, count conversions that occurred within those 7 days plus your typical conversion delay (usually 1-3 days for ecommerce, 7-14 days for higher-consideration purchases).

Step 2: Calculate incremental lift

Incremental Lift = (Treatment CR - Holdout CR) / Holdout CR

Example:

• Treatment group: 1,200 conversions from 250,000 users = 0.48% CR
• Holdout group: 38 conversions from 12,500 users = 0.30% CR
• Incremental lift: (0.48% - 0.30%) / 0.30% = 60%

This means your ads caused a 60% increase in conversions above the organic baseline.

Step 3: Calculate incremental conversions

Incremental conversions = Total treatment conversions x (1 - Holdout CR / Treatment CR)

Using the example above: 1,200 x (1 - 0.30/0.48) = 1,200 x 0.375 = 450 incremental conversions

Of the 1,200 conversions your platform claims, only 450 were actually caused by your ads. The other 750 would have happened anyway.

Step 4: Calculate your true metrics

Incremental CPA = Total spend / Incremental conversions

If you spent $45,000 during the test: $45,000 / 450 = $100 incremental CPA

Compare this to your platform-reported CPA of $45,000 / 1,200 = $37.50. Your true cost per acquisition is 2.7x higher than what the platform reports.

Incremental ROAS = Revenue from incremental conversions / Total spend

If your AOV is $85: (450 x $85) / $45,000 = 0.85x incremental ROAS

In this example, the campaign is actually unprofitable on an incremental basis, despite platform-reported ROAS showing a healthy return.

What to do with your results

If incremental ROAS is above your target

The campaign works. You now have a baseline for incremental performance. Test scaling budget by 20-30% and re-run the incrementality test in 4-6 weeks to see if the lift holds at higher spend.

If incremental ROAS is positive but below target

Optimize before cutting. The campaign drives real conversions but at a higher true cost than you thought. Tighten targeting, refresh creative, or reduce frequency caps. Re-test in 30 days.

If incremental ROAS is near zero or negative

Reduce spend by 50% and monitor overall business metrics for 2-3 weeks. If total conversions don't drop meaningfully, you've confirmed the campaign was taking credit for organic conversions. Reallocate the budget to channels with proven incrementality.

The 7-day test is just the beginning

One test gives you one data point. To build a reliable incrementality measurement practice, run tests quarterly on each major channel, test new channels before scaling, and use the results to calibrate your attribution model.

Agencies that run incrementality tests for clients retain those clients 40% longer than those that don't. When you can prove causal impact, the client conversation shifts from "justify your spend" to "where should we invest next."

Frequently Asked Questions

What if my test shows zero incremental lift?

A zero-lift result doesn't necessarily mean the channel has no value. It could mean your current targeting is reaching people who would convert organically, your frequency is too high (wasting impressions on already-decided buyers), or the test duration was too short to capture the full conversion window. Before cutting the channel entirely, adjust targeting to focus on new customer acquisition, reduce frequency caps, and re-test. If the second test also shows zero lift, confidently reallocate the budget.

Can I run incrementality tests on multiple campaigns simultaneously?

Yes, but only if the campaigns target non-overlapping audiences. If Campaign A and Campaign B both target the same demographic, a holdout from one campaign may still see ads from the other, contaminating your results. The safest approach is to test one campaign at a time, or use platform-native tools like Meta's Conversion Lift that handle audience separation automatically. For cross-platform tests, geographic holdouts are the cleanest method since you can pause all paid media in holdout markets simultaneously.

How do I handle seasonality in a 7-day incrementality test?

Compare your test period to the same days in prior weeks, not to the week before. If you're testing during a promotional period, holiday, or seasonal peak, the test is still valid -- it measures lift during that specific period. The risk is extrapolating results to non-seasonal periods. A campaign that shows 40% incremental lift during Black Friday may show 15% lift in February. Run separate tests during peak and off-peak periods to get a full picture.

---

Go Funnel uses server-side tracking and multi-touch attribution to show you which ads actually drive revenue. Book a call to see your real numbers.