How To Run Incrementality Experiments?

Incrementality experiments are structured tests conducted in a controlled setting to help validate marketing hypotheses. Essentially, they're about testing key questions: Is a specific marketing channel really driving the key performance indicators we care about? Is a particular ad creative effective? By running these experiments, you can confidently identify which parts of your marketing plan are actually contributing to your goals.

Advertisers who consistently conduct experiments throughout the year experience significant long-term benefits. Specifically, they can achieve a 30% improvement in ad performance within the same year and expect an increased performance by an impressive 45% in the following year.

Google recommends using incrementality experiments to add an extra level of awareness to their attribution and MMM efforts. This can enhance the attribution models and provide a clearer picture of your marketing efforts' effectiveness. Before discussing the steps of running a successful incrementality experiment, let's understand why you should run it first.

Why do you need incrementality experiments?

Let's think about why experiments are so valuable in marketing. Essentially, they help you see clearly what's actually working. It's like having a reliable map while trying to navigate through a thick fog. Experiments help you separate the genuine effects of your marketing efforts from all the noise caused by outside factors. This way, you know for sure which channel is pulling its weight.

This clarity is essential when you're deciding where to put your money. By understanding the real impact of each channel, you can spend more wisely, focusing on what truly drives results. This approach also helps you steer clear of common mistakes, like confusing a coincidental uptick in sales with the effectiveness of a particular campaign.

Finally, this kind of precise information is gold when it comes to fine-tuning your marketing measurement tools, like marketing mix models. By feeding these tools data from experiments, they become sharper and more aligned with reality. This isn't just about making minor adjustments, it's about making your whole marketing approach smarter, ensuring that you're not just guessing, but making decisions based on solid evidence.

Four steps to run successful incrementality experiments

Now, let's dive into how to run these experiments effectively.

1) Define your hypothesis

Successfully executing incrementality experiments hinges on crafting a well-defined and targeted hypothesis. Start by pinpointing precisely what you want to test. For example, you might hypothesize that escalating your ad spend on a specific platform will enhance conversion rates, or that introducing a new social campaign could amplify your brand's visibility.

Integrating your experiment with other sophisticated measurement methods, such as multi-touch attribution (MTA) or marketing mix modeling (MMM), can enrich your approach. Utilize insights from MTA to identify potentially underrepresented channels in your attribution model, thereby focusing your experimental efforts on verifying their true value. Similarly, insights from MMM can guide you in designing your experiments, particularly in testing channels that exhibit significant variances in previous analyses.

Here's how to frame a hypothesis with precision and relevance: Suppose you decide to focus on digital advertising's impact, specifically testing Facebook video ads. Your hypothesis might state:

"Deploying Facebook video ads will achieve an incremental Return on Ad Spend (iROAS) of 4 over a six-week campaign with a budget of $100,000. Achieving or surpassing this iROAS target will prompt us to adopt an always-on advertising approach, aiming to maximize efficient, incremental revenue. Conversely, if the campaign falls short of the iROAS goal, we will refine our tactics and conduct additional tests to enhance the campaign's effectiveness before increasing our financial commitment."

This approach not only clarifies what you are testing and why but also sets clear benchmarks for evaluating success and making informed decisions on future marketing strategies.

2) Identify key success metrics

This crucial step not only defines what success looks like for your initiative but also ensures that the insights derived are actionable and relevant to your broader business goals.

Begin by pinpointing the specific outcomes your marketing efforts aim to enhance. These could vary from boosting sales and increasing conversion rates to strengthening customer engagement or elevating the quality of leads. For instance, if your experiment is centered around a new social media strategy, appropriate key success metrics might be engagement rate, click-through rate, and conversion rate specific to those platforms.

For scalability in incrementality tests, Google advises setting targets based on incrementality at the channel level. This strategic focus facilitates more informed planning and optimization of your media spending, directing resources toward channels that demonstrate substantial growth.

Additionally, consider using these incrementality-based Key Performance Indicators (KPIs) as secondary metrics. This helps you assess potential shifts in decision-making when incrementality is prioritized over traditional attribution metrics.

It's also vital to ensure that the metrics chosen are suitable for the channel under scrutiny and that the tools used to measure these metrics across various platforms are consistent. This uniformity allows for more accurate comparisons and calibration of results, providing a clearer view of each channel's performance and its impact on your marketing objectives.

3) Design your incrementality experiment

Designing your incrementality experiment involves selecting the appropriate methodology to answer the specific question you want to explore. You must ensure the experiment is sufficiently powered to detect any statistically significant changes the interventions might cause.

This involves establishing a robust framework, ensuring sufficient investment in the tested channel, and confirming that the sales volume can demonstrate a clear result.

Start by selecting the type of incrementality test that best suits your hypothesis. Here are two common approaches:

Geo experiments

This method effectively measures incrementality by comparing the performance between a test group (exposed to the campaign) and a control group (not exposed) within specific geographic areas. This setup helps isolate the campaign's impact from other variables that could affect the outcomes.

Audience split test

Similar to A/B testing, this method involves dividing your audience into different segments based on various criteria, such as demographic information, psychographic profiles, or behavioral patterns. Segmentation allows you to assess how different groups respond to changes in your marketing strategy, providing clear insights into the effectiveness of specific changes.

Additionally, it's crucial to maintain a clean holdout group to avoid contamination from overlapping audiences. This control group should not be exposed to the campaign. It should mirror the test group as closely as possible in all other respects to ensure that any observed differences in performance are attributable solely to the marketing intervention.

Check out how Incrementality Tests and Geo-Experiments empowered a Personal Care Brand to confidently increase its advertising budget with a $500,000 monthly ad spend.

4) Acting on incrementality results

Once you have the results from your incrementality experiments, acting on them is imperative to enhance your marketing strategy. Positive results that align with your hypotheses confirm the efficacy of your campaigns, allowing you to continue or even scale these initiatives confidently.

On the other hand, unexpected outcomes should not be disregarded as failures; instead, we should view them as invaluable insights that can guide future adjustments and improvements. Let's explore a few ways you can act on the incrementality results:

Calibrating attribution based on incrementality:

Calibration involves assessing the discrepancies between the incremental impact measured by controlled experiments and the attributed impact, typically reported by standard attribution models. These figures often vary due to inherent biases in traditional attribution methods.

To address this, calculate an incrementality factor, which is the ratio of the incremental ROI to the attributed ROI. This multiplier adjusts the attributed values to more accurately reflect the true Incremental performance of your marketing efforts across your campaigns, ad groups, and ads.

Example of incrementality factorin action:

Consider three hypothetical marketing channels:

• Channel A is reported to have an attributed Return on Ad Spend (ROAS) of $10. However, incrementality testing reveals a true ROAS of $7, resulting in a calibration multiplier of 0.7 (i.e., $7/$10).
• Channel B shows an attributed ROAS of $8, while the incrementality experiment indicates an ROAS of $4, leading to a multiplier of 0.5 ($4/$8).
• Channel C reports an attributed ROAS of $5, but the incrementality results show a significantly higher iROAS of $15, which gives a multiplier of 3 ($15/$5).

These multipliers help refine the attributed results, ensuring that your ongoing and future marketing decisions are grounded in a more realistic understanding of each channel's contribution to your overall marketing success.

Using these multipliers, you can recalibrate past and future attribution results to better reflect the true effectiveness of each channel, ensuring that your marketing budget is allocated more efficiently.

Validate whether optimizations are improving incrementality over time

To effectively validate that marketing optimizations are enhancing incrementality, employing a systematic approach using AI-powered tools and detailed incrementality testing is crucial. Initially, develop a hypothesis about how certain optimizations could improve the incremental return on ad spend (iROAS).

Let's consider a hypothesis centered on the impact of targeted advertising during a high-profile sports event. The hypothesis might be: “Implementing personalized ads during the Super Bowl, aimed at sports enthusiasts between the ages of 25 and 40, will enhance the TV campaign's incremental return on ad spend (iROAS).”

Here's a structured approach to test this hypothesis:

• Initial measurement: Begin with a geo-experiment to measure the iROAS of your current TV campaign, which broadly targets a general audience. Assume the campaign initially achieved an iROAS of $1.50 and an attributed ROAS of $5.00. This test period should last 2-3 weeks.

• Implement optimization: Then, optimize by specifically targeting viewer segments likely more receptive to the ads, such as those identified through viewer behavior analysis. This implementation should continue for the remainder of the quarter, approximately 9-10 weeks.

• Post-optimization: Compare the results with the historical performance of the same quarter last year. The expected outcome is that both iROAS and attributed ROAS will have improved, demonstrating the success of the optimization. For example, you might find that the iROAS has increased to $2.25 from the initial $1.50, indicating the effectiveness of the targeted ad placements.

Final thoughts

Whether adjusting creative content, fine-tuning audience segments, or shifting ad schedules, the insights gained from these experiments provide a clear roadmap for enhancing performance. Implementing optimizations based on solid data increases the efficiency of your marketing spend and maximizes the overall return on investment.

Remember, each experiment, successful or not, offers valuable lessons that can refine your marketing approach. This ensures that your strategies are responsive and proactive in driving your business forward. Embrace incrementality experiments as a critical component of your marketing toolbox to stay ahead in a competitive landscape and achieve sustainable success.

Book a demo now to try how Lifesight revolutionizes incrementality experiments.