Your friend’s birthday is coming up, and you decided to bake a delicious chocolate cake. Since weighing ingredients is critical to ensuring the best outcome, you have taken out a kitchen scale that, unfortunately, you have not used for years.
To ensure your scale’s accuracy, you first zero your scale and put a penny on it. It should be exactly 2.5 grams, but the weight is off. You bring another one and place it on, and again, the number reads 2.4 grams. To restore your scale’s accuracy, you replace batteries and reset the scale, and finally, the scale registers the correct 2.5 grams for a single penny.
This procedure of testing a scale with a known weight to maintain its accuracy is known as calibration. In fact, calibration also plays an important role in marketing measurement, specifically marketing mix modeling (MMM). In this post, we will look into how you should work with calibration for the best insights.
Before moving on, check out the below posts to know MMM better:
👉 Your All-In-One Guide to MMM
👉 The Evolution of MMM: A Pathway to Privacy-First Measurement
👉 MMM vs. MTA: Which One Is Right For You?
How does calibration work on MMM?
MMM uses aggregated observational data, meaning that the model analyzes historical data and assumes that past patterns will repeat in the future. This is why MMM requires regular calibration. The model can reveal correlation but not causation, and the model might have been right with the past but be wrong with the future due to its negligence of changes.
As a measure to accurately reflect real-world relationships between different variables, the model is calibrated by comparing its predictions to actual outcome data from experiments, known as the "ground truth.”
A common way to determine the ground truth for the effectiveness of a marketing campaign is to conduct a conversion lift study or a holdout test. Once you have identified the ground truth, you can compare it to the projections made by the model. If there is a significant discrepancy between the model's predictions and the ground truth, you can adjust the model's parameters or variables accordingly.
How do you find the ground truth?
There are several tools that can be used to conduct an experiment to retrieve the ground truth. The most reliable technique is the randomized controlled trial (RCT), which involves exposing a randomly selected group of users to an ad and comparing the conversion rate of that group to a control group that was not exposed to the ad.
However, conducting an RCT at scale takes work because it requires access to a large audience and the ability to randomly select a control group from that audience. This is something of which only the biggest ad platforms in the world are capable. Meta's Lift Study and Google's Conversion Lift are two RCT tools that can measure the incremental effectiveness of channels and campaigns.
Although an RCT is considered the gold standard in the hierarchy of evidence, there is an alternative: geo experimentation, also known as GeoLift, which tests the effectiveness of an ad in a specific geographic location and effectively measures incrementality.
Yet, geo experimentation does have some limitations. Most notably, it assumes that the impact of a marketing campaign is consistent across the chosen geographic location, which may not always be the case. Another limitation is that it may be difficult for smaller companies to collect data from dozens of cities or states across the nation and find out the optimum number of test locations or the ideal control and test locations.
How often should you calibrate?
Calibration should be performed regularly to ensure that the models remain accurate over time. The frequency can vary, but when you choose to calibrate in shorter periods, your model is less likely to operate outside acceptable parameters.
Periodic calibration is essential in a rapidly changing marketing landscape where the relationships between different variables can change quickly. However, conducting actual experiments can take time. Typically, RCTs can take anywhere from two to three weeks to complete. For this reason, it's important to strike a balance between the need for regular calibration and the time and resources required to conduct experiments.
Ultimately, the key is finding a calibration schedule that works for your organization. The idea is to calibrate often enough to ensure that your models remain accurate but not so often that it becomes a burden on your resources and budget.
How Airbridge uses calibration for the best insights
The Airbridge MMM, powered by advanced machine learning and statistical techniques, provides actionable insights at a lower cost and a faster speed so that marketers around the world can effectively find and scale their true sources of growth.
Airbridge allows its users to use experiment results for calibration, as many other marketing analytics companies recommend. These results can be compared with MMM results to identify how overestimated or underestimated the impact of different marketing activities is. In other words, you can get a more aligned and consistent understanding of your advertising impact.
However, we also know that some app developers, especially those with smaller marketing budgets, cannot afford the time and cost required to conduct experiments. Airbridge helps its users in such cases use multi-touch attribution (MTA) results, found to resemble real-world experiment results, as an alternative. Nonetheless, for the highest possible accuracy, we advise relying on experimentation.
Ready to power up your marketing measurement? Request a demo to learn more about calibration, MMM, and your mobile app success.
