When to Use Incrementality Testing vs Media Mix Modeling

Two approaches. Different questions. Neither works alone.

Incrementality testing and media mix modeling (MMM) are the two most talked-about alternatives to attribution modeling. Both aim to measure marketing effectiveness more accurately than platform-reported numbers. But they take fundamentally different approaches, work on different timescales, and answer different questions.

Getting them confused -- or using the wrong one for a given decision -- leads to expensive mistakes.

What each method actually does

Incrementality testing: controlled experiments

Incrementality testing uses holdout groups (randomized or geographic) to measure the causal effect of a specific marketing activity. You turn off ads for a subset of your audience, compare conversion rates to the group that sees ads, and the difference is your incremental lift.

Strengths:

• Measures true causal impact (the gold standard for a single-channel question)
• Works for any channel, including offline
• Results are straightforward to interpret
• No historical data required -- just run the experiment

Weaknesses:

• Tests one thing at a time (one channel, one campaign, one time period)
• Requires sacrificing revenue in the holdout group
• Takes 2-6 weeks per test
• Results are point-in-time -- they tell you what happened, not what will happen at different spend levels

Media mix modeling: statistical regression

MMM uses historical data (typically 2-3 years) to build a statistical model that estimates how much each marketing channel contributes to business outcomes. It regresses sales or conversions against marketing spend, controlling for external factors like seasonality, pricing, economic conditions, and competitive activity.

Strengths:

• Models all channels simultaneously
• Provides continuous measurement (not point-in-time)
• Estimates response curves showing diminishing returns at different spend levels
• Doesn't require any tracking or cookies -- works entirely on aggregate data
• Can incorporate offline channels (TV, radio, print, OOH) alongside digital

Weaknesses:

• Requires 2-3 years of historical data at minimum
• Results are estimates based on correlation patterns, not controlled experiments
• Slow to update -- models are typically refreshed monthly or quarterly
• Struggles to distinguish between channels that move together (if you always increase Meta and Google simultaneously, the model can't separate their effects)
• Assumes past relationships between spend and outcomes will hold in the future

Decision framework: which to use when

Use incrementality testing when:

You need to validate a specific channel. "Is retargeting actually working?" is an incrementality question. A holdout test gives you a direct yes/no with a causal lift estimate. MMM can give you a modeled estimate, but it won't have the experimental certainty.

You're considering killing or dramatically changing a channel. Before cutting $100K/month from a channel, run a holdout test. The cost of the test ($10-15K in forgone revenue) is a small price for certainty.

You're entering a new channel. When you launch TikTok ads or start a podcast sponsorship, there's no historical data for MMM to work with. Run an incrementality test in the first 60 days to understand the channel's causal impact before committing long-term budget.

You need to convince a skeptic. CFOs, board members, and private equity investors trust controlled experiments. Showing a holdout test result -- "we turned off ads in these markets and conversions dropped X%" -- is more persuasive than a regression coefficient from an MMM model.

Your media mix hasn't changed much. MMM needs variation in spend levels to estimate channel effects. If you've spent roughly the same amount on each channel for the past two years, the model can't distinguish channel contributions from the baseline. Incrementality tests create the variation artificially.

Use media mix modeling when:

You need a total-portfolio view. How much should you spend on Meta vs. Google vs. TikTok vs. TV? MMM provides estimates for all channels simultaneously with response curves that show diminishing returns.

You're doing annual or quarterly planning. MMM is built for strategic budget allocation. It answers: "If I move $200K from Google to Meta, what happens to total revenue?" Incrementality tests can't answer hypothetical reallocation questions.

You need to account for offline media. TV, radio, print, and out-of-home advertising can't be measured with user-level holdout tests. MMM handles these channels naturally because it works with aggregate spend and outcome data.

You want to understand external factors. MMM models seasonality, economic trends, competitor activity, and pricing effects alongside marketing spend. This context helps you understand what portion of your results comes from marketing versus external factors.

You need ongoing measurement without continuous testing. Incrementality tests are periodic snapshots. Between tests, you're flying blind. MMM provides a continuous estimate of channel effectiveness that updates with each new data point.

Use both when:

You can afford to. The ideal measurement stack uses incrementality tests to calibrate MMM. Run holdout tests on 2-3 channels per quarter. Feed the results into your MMM as "ground truth" data points. The model's estimates for untested channels benefit from the calibration, and you have experimental validation for the channels that matter most.

Your MMM results seem suspicious. If the model says branded search drives 30% of total revenue, that's a signal to run an incrementality test on branded search. The test either validates the model's estimate or reveals a miscalibration that affects all the model's outputs.

You're making a major strategic shift. Changing your channel mix by more than 20%? Run incrementality tests on the channels you're scaling up and the channels you're scaling down. Use MMM to forecast the combined effect of the changes. Compare actual results to the forecast after 90 days.

How to combine them effectively

The calibration loop

1. Build your MMM using 2+ years of historical data. The model estimates contribution and response curves for each channel.

2. Run incrementality tests on 2-3 channels per quarter, prioritizing channels where the MMM estimate seems high or low relative to your intuition.

3. Compare results. If the MMM says Meta drives $500K/month in revenue and the incrementality test says it drives $380K/month, you have a 24% over-estimation in the model.

4. Calibrate the model. Use the incrementality results as prior constraints in the next MMM refresh. This forces the model to align with experimental evidence while still providing estimates for untested channels.

5. Repeat quarterly. Each round of testing improves the model's accuracy, and the model improves your test prioritization (by highlighting channels with the most uncertainty).

Google, Meta, and Measured (the incrementality platform) all recommend this combined approach. Google's Meridian MMM framework explicitly accepts incrementality test results as calibration inputs.

What the combined approach catches that neither alone would

A brand running both discovered:

• Their MMM said YouTube contributed $200K/month. An incrementality test showed $310K/month. The MMM was undercounting because YouTube's effect showed up through branded search (YouTube exposure led to Google searches), and the model was attributing that revenue to Google.

• Their incrementality test on Meta prospecting showed strong lift during the test period. But the MMM showed that Meta's marginal return had been declining over 6 months -- meaning the test result was valid for current spend but wouldn't hold if they scaled further. The test alone would have led them to scale aggressively into diminishing returns.

Neither measurement alone would have surfaced these insights.

The cost reality

Incrementality testing: $5K-$15K per test in forgone revenue (depending on holdout size and duration) plus internal analyst time. No software cost if using platform-native tools. Third-party platforms (Measured, Haus) charge $30K-$100K annually.

Media mix modeling: In-house builds require a data scientist and 3-6 months of development. Off-the-shelf solutions (Robyn by Meta, Meridian by Google) are free but require technical implementation. Managed services from agencies or vendors run $50K-$200K annually.

Combined approach: $80K-$250K annually when including both incrementality tests and MMM. For brands spending $1M+/month on media, this represents less than 2% of total spend -- and typically identifies 15-25% in budget optimization opportunities.

Frequently Asked Questions

Can I use open-source MMM tools like Meta's Robyn instead of paying for a vendor?

Yes, and many brands do. Robyn is a well-built, free MMM tool that runs in R and produces channel decomposition and budget optimization recommendations. The catch is implementation: you need a data scientist or analyst who can wrangle your data, configure the model, validate results, and re-run it regularly. If you have that talent in-house, Robyn or Google's Meridian are excellent starting points. If you don't, the cost of hiring a contractor ($5K-$15K for initial setup plus $2K-$5K for quarterly refreshes) is still significantly cheaper than managed services.

How accurate is MMM for digital-heavy brands with limited offline spend?

MMM was originally built for CPG brands with heavy TV and print spend, but modern implementations work well for digital-heavy brands. The key requirement is spend variation: you need periods where you spent more and less on each channel so the model can estimate the relationship between spend and outcomes. Digital brands often have this variation naturally through seasonal spending, channel tests, and budget shifts. Where digital-only MMM struggles is distinguishing between platforms that always scale together. If you always increase Meta and TikTok simultaneously, consider running them on alternating budgets for a few months to give the model better data.

How frequently should I refresh my MMM?

At minimum, quarterly. Each refresh incorporates the latest data and adjusts estimates for any changes in market conditions, brand awareness, or competitive dynamics. Monthly refreshes are ideal if you have the data pipeline automated. The most common mistake is building an MMM once and treating its recommendations as permanent. Markets change, response curves shift, and channels evolve. A model built on 2024 data may not accurately predict 2026 channel performance. Quarterly refreshes with incrementality calibration keep the model current and reliable.

---

Go Funnel uses server-side tracking and multi-touch attribution to show you which ads actually drive revenue. Book a call to see your real numbers.