main

ARTICLE

Four Fatal Flaws of Digital Attribution and How to Address Them: Part IV

August 23, 2016 — by Ari Buchalter    

In this 4-part series, we’ve examined the four fatal flaws of attribution in today’s digital marketing landscape. Last time we discussed the third fatal flaw of Missing the Offline View and now tackle the fourth and final fatal flaw:

Fatal Flaw #4: Failing to Measure Incremental Impact

The Problem. Consumers encounter a wide array of advertising. They move across online and offline media, across devices and across contexts, often seeing multiple ads from a given advertiser over time as well as from competitors. And advertising is, of course, just one of several factors that play a role in impacting consumer decisions. While many marketers focus on an absolute view of performance (often denominated in metrics such as aggregate CPA or ROI), sophisticated marketers are starting to hone in on an incremental view of performance (measured as incremental return on ad spend, or “incremental ROAS”). They want to understand not just whether a consumer action took place after exposure to advertising but whether it took place because of the exposure to advertising, by measuring the likelihood that action would have taken place in the absence of that exposure. In the very low percentage of cases where consumers actually clicked on an ad prior to some action, it’s plausible to assume the ad had some direct impact. But advertising can generate impact in many other ways besides a click, both at the top of the funnel (e.g., generating awareness, prompting increased interest and consideration that can lead to later action) and at the bottom of the funnel (e.g., triggering consumers to conduct a search or directly navigate to an advertiser’s site to take immediate action). Understanding the true difference caused by the advertising exposure (sometimes referred to as “lift”) is critical to accurate attribution and making the right marketing investment decisions.

The Complication. Numerous methods exist that attempt to measure the true lift of digital marketing efforts. These can be classified as either “holdout-based” methods, which separate real-world audiences into test and control groups for exposure to different marketing treatments, or “modeling-based” methods, which use advanced analytics on historical data, combined with some assumptions, to tease results out from the data. Both try to establish a baseline level of conversion activity that would have happened in the absence of a particular marketing program, against which the results of the program can be compared. However, both sets of methods are highly prone to a variety of contamination issues that mix up users assigned to different measurement groups, making it extremely difficult to obtain clean results. For example, cookies are often used to assign users to test and control groups, but if cookies are deleted (or not accepted in the first place), the same user can easily be assigned to both. The same problem exists with geo-based measurement groups, where inaccuracies in geo-IP mapping can contaminate populations. Contamination also arises from cross-device usage (or usage of multiple browsers on the same device) where, for a single consumer, one device (or browser) is assigned to the test group and the other to the control. The inability to track populations within certain walled-garden media environments can also lead to contamination due to untracked media exposure. Finally, and perhaps least well understood, there are strong sampling biases that come into play in biddable auction environments, which introduce massive contamination into these kinds of lift measurements. Collectively, these issues introduce so much noise as to effectively drown out the signal in both holdout-based and modeling-based approaches, often resulting in measurements with little lift, no lift, or even negative lift! Not to mention both methods entail significant excess workflow and cost. Holdout-based methods often require additional cost to serve PSA ads to a control group and involve significant effort to setup and manage “mirror” campaigns. Modeling-based methods involve outsourced analytics that take many quarters to set up, can be very sensitive to the modeling assumptions,  are difficult to understand & audit, and hard to validate in practice (which is perhaps why these engagements are often not renewed). With both methods there can be a variety of different approaches to the actual calculation of lift, which are often opaque and can lead to ambiguous conclusions if not well understood.

The Resolution. Accurate measurement of incremental impact requires that marketers bring to bear sophisticated new methodologies that can ensure statistically identical test and control populations and address all the various sources of contamination, including countering the bias effects associated with bidded auction dynamics. It should make both the data and methodology – from the input to the output and everything in between – fully transparent and auditable, in order to validate, compare, and ultimately trust the results. An ideal solution would also be automated – requiring little or no workflow to set up, maintain, or analyze – and avoid the need to spend precious marketing dollars on “placebo” ads. It’s a tall order indeed, and very few companies have figured out how to successfully tackle all of these challenges, but the ones that have are able to consistently demonstrate incredibly strong lift with extremely high significance, completely reframing how advertisers think about marketing effectiveness. Even beyond measuring lift, the most forward-thinking marketers are starting to look for solutions that actually optimize to it, leveraging algorithms that automatically reallocate spend in the real-time buying systems so as to maximize not aggregate performance, but incremental performance. Perhaps one day soon, pricing models might even evolve such that marketers only pay for measurably incremental results.

Conclusion

Taking a step back and considering all four of the attribution flaws we’ve discussed, the good news is these are only fatal if left unchecked. Fortunately, the market has developed a solid understanding of both the limitations inherent to current attribution methodologies and of the value in addressing them. The challenging part is driving to the solutions. These are fairly tough problems, where sophisticated analytics are necessary but not by any means sufficient. Integrating impression-level predictions into media buying, developing robust device identification and cross-device association, driving up offline-to-online match rates, and delivering accurate incremental impact measurement, each represents a complex problem in its own right, and the market is developing powerful solutions to address each one. Tackling attribution successfully, however, requires addressing all of them simultaneously, in a manner that is seamlessly unified with real-time execution, and that’s ahead of where most of the market is today.

Marketers should talk about these challenges with their technology partners to understand whether and how their solutions tackle each of them, and whether they have the scale, sophistication, ecosystem connectivity, and proven know-how to make them work. They should select partners who have not only thought deeply about these issues, but have designed and implemented solutions that can demonstrably deliver the cure to all of these fatal flaws.

Ari Buchalter

Ari is the President, Technology, at MediaMath. Ari oversees MediaMath’s Product & Engineering teams, focused on designing, developing, and supporting best-in-class product solutions and platform capabilities for MediaMath’s customers. Ari brings a unique combination of business leadership, strategic vision, practical marketing knowledge, and quantitative expertise. Whether it’s designing MediaMath’s proprietary Brain optimization algorithm, unifying marketing approaches across digital channels, innovating new targeting technologies, or building the open marketing platform of the future, Ari is impassioned by the chance to work on industry-transforming solutions and humbled by the opportunity to lead the best technology team in the business.