How Marketers Can Start Integrating AI in Their Work

June 7, 2018 — by MediaMath0


This article originally appears in Harvard Business Review as a byline by our CMO/CSO Dan Rosenberg. 

According to Constellation Research, businesses across all sectors will spend more than $100 billion per year on Artificial Intelligence (AI) technologies by 2025, up from a mere $2 billion in 2015. The marketing industry will be no exception.

AI holds great promise for making marketing more intelligent, efficient, consumer-friendly, and, ultimately, more effective. Perhaps more pointedly, though, AI will soon move from being a “nice-to-have” capability to a “have-to-have.” AI is simply a requirement for making sense of the vast arrays of data — both structured and unstructured — being generated from an explosion of digital touchpoints to extract actionable insights at speeds no human could ever replicate in order to deliver the personalized service consumers now demand.

Interestingly, in many cases, the sophistication of AI technologies has already advanced further and faster than most marketers’ ability to actually make use of them. On the one hand, there are the technical challenges of gathering and normalizing data inputs — the act of making different types of data comparable — connecting them to a unified view of the customer, and then aligning the AI-driven decisions to real-world actions. On the other hand, there are also real philosophical, ethical, or at least policy decisions to be made on the value exchange between marketers and consumers when data is shared and used to optimize marketing experiences.

The good news is that, as an industry, we are starting to see meaningful progress on both fronts. For businesses looking to keep pace with innovation and leverage AI, there are steps they can take today. But first, what are some examples of how AI can help make marketing more effective?

Read the rest of the article here


Identity and Measurement: What is Next for Mobile and Omnichannel Advertising?

June 5, 2018 — by Floriana Nicastro0


No matter how many times an advertiser hears that they must run on mobile to reach their audiences, if they can’t measure the output of their investment, it’s a lost cause. In fact, many advertisers are refusing to invest another dime in mobile until the industry finds a common ground to solve for mobile measurement at scale to help quantify real mobile ROI.

The gap between advertiser spend and user time spent

More than 62 percent of user time online is spent on mobile. But a big part of that time seems unable to be monetized; the gap between advertising investment and user behavior still represents a $7 billion opportunity according to Mary Meeker’s latest Internet Trends report. Advertisers don’t have the right tools to quantify mobile impact and justify investment due to several issues:

  • Accuracy vs scale: the cross-device solution conundrum

As we started discussing last year, with user identity being fragmented between cookies and device IDs. It is hard for advertisers to have a holistic view of customers and execute on a real omnichannel strategy. It is still a real challenge to offer both accuracy and scale. Only 14 percent of marketers can track cross-channel and act on data, according to a report by L2inc.

  • Higher walls in a fragmented identity ecosystem

The online identity ecosystem is highly dependent on Apple and Google, with US mobile browser share of voice a split between 51 percent Safari and 42 percent Chrome, and app browsers split 45 percent iOS and 53 percent Android. These challenges come by way of limited tracking ability to understand how users are engaging in these environments and other channels.

Further, Apple’s Limit Ad Tracking (iOS 10),  Intelligent Tracking Prevention (iOS 11) and SDK app networks and now the ITP update for social and fingerprinting (ios 12) are adding additional restrictions that advertisers need to navigate through to measure the impact of their advertising spend.

  • Attribution models

Thirty-four percent of B2B marketers have no attribution model, and it’s easy to see why. Attribution models and technology are complex. Numerous internal “organizational roadblocks” can make it very hard to convince senior executives of the long-term gains of attribution over the short-term high costs.

Mobile is a key point on your users’ road to conversion, but it is not necessarily the last signpost they hit. You need a multi-touch attribution model to account for all touchpoints before a user converts. Multi-touch attribution is a more ideal model than last-touch because it enables you to better understand true marketing impact. By letting you see exactly what led a customer to convert at all points—both online and off—on the path to purchase, you can immediately act upon what’s working and what’s not and give appropriate credit to advertising partners.

There are steps you can take to shift your business’s mindset on attribution and justify the ROI against the investment in cost, time and expertise. Read our attribution playbook for specific tactics on how to get there.

 What’s next?

If measurement is a market challenge, then the ad tech industry is poised to develop new tools that provide the solution. MediaMath is leading the charge by prioritizing the following areas:

  • Enriched identity

We believe that for advertisers to identify their best customers at scale, they need a neutral, shared device namespace that enables global reach and proprietary value-add. We believe DigiTrust represents this opportunity, which is why we joined their consortium late last quarter.

This neutral, standardized device ID will improve consumer experience by supporting privacy, reducing page load time, increasing the relevance of marketing messages and enabling the diverse ecosystem of publishers and online platforms on which they rely. We also have our own proprietary cross-device and cookieless identity solution, ConnectedID, for which we have long been sourcing deterministic signal directly from advertisers, and will start working with our industry partners to further activate.

  • Cross-device analytics reporting

It is crucial for advertisers to be armed with knowledge related to cross-device multitouch measurement and understand the impact of mobile on their omnichannel outcomes. Demonstrating mobile performance for your campaigns has often been a challenge in digital marketing. MediaMath has been developing several new reports to provide marketers with concrete data and help them with their investment decisions. As trusted advisors for our clients, we wanted this data approach to be consultative and incorporate our analytics service to benefit our clients—we know you still need a human touch.


Greetings from the Front Lines of the Marketing Revolution!

June 4, 2018 — by Joe Zawadzki0


It has been more than 100 years since the marketing pioneer John Wanamaker said, “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”  For almost a century, that statement defined marketing as a profession.

Things have changed.  Now, with programmatic marketing and a mobile computer in your pocket, marketing and the technology it uses to connect people to messages is a force so effective and powerful that the public, policymakers, and regulators have taken an intensified interest in what we do and how we do it.  That’s an opportunity.  It requires us to come together to accelerate our maturation in the industry—providing more and more control, transparency, and accountability in our practices even as we continue to innovate in pursuit of outcomes.

MediaMath has come a long way since the birth of the category more than a decade ago.  The vision was to create a single software layer, with data and insights mixed in to make high-quality marketing decisions—which ad to buy, how much to pay for it, what to show people. And then if we could bring in AI, the math part of media, and machine learning to bear to automate those high-quality marketing decisions in real time, in runtime, we could make marketing better for everyone involved.  And we did it.

Here we are, 11 years later, having made digital marketing and programmatic a reality, one in which a CMO can sit in an office anywhere in the world, push a button, and, like magic, immediately change how his or her message and brand are displayed on billions of screens. It’s amazing. And there is so much more to come as we ask this machinery to drive more sophisticated goals, as we infuse creativity and storytelling (back) into platforms.

The moment has arrived for the industry to mature beyond its gangly adolescence, to focus on the consumer experience and respect for digital dignity, to match strength for strength. The stakes are high. In Europe, regulators have adopted data protection and privacy regulation intended to give control over personal data back to citizens. Elsewhere, regulators and legislators are considering whether and how to address these issues. Without the right guardrails on use, we’ve seen marketing technology weaponized to inflame schisms in society, swing elections, reshape economic and trading blocks. This is our opportunity to meet the rising expectations of the society we live and work in to ensure that the engine of the digital economy, data-driven marketing, runs smoothly and takes us to a place we all want to go.

MediaMath was there at the beginning, and we intend to lead the next phase with clear intent and direction, focused on our mission—to make marketing everyone loves. Consumers want to understand and control how their data is being used and why, as well as be able to trust the companies that are using it. Marketers want to know exactly who they are reaching and why, and want trusted partners along every step of the value chain. Everyone agrees that, as an industry, we can do better. It all starts with remembering the human being on the other side of the screen.

That’s why in 2018, we will focus on continuing to further build the infrastructure and software that connect consumers with the brands and companies they love in a way that they love.  Our mandate for this year and beyond is continued transformation. Our product offering will evolve along three pillars: supply chain refactoring, next-generation identity / audience capabilities, and augmented AI. For our clients, this work will manifest in better outcomes—marketing dollars spent smarter, with decisioning powered by unique data sets and AI. For consumers, these initiatives will provide a better experience—brands that consumers know and trust, in an environment that respects them, bringing the whole industry toward a more idealized vision of what advertising can and should be.

We appreciate you joining us on this journey. Together, we are going to make marketing everyone loves. We invite you to challenge our ideas, inject your own, and evangelize with us to make this a reality in 2018 and beyond.


GDPR Is A Force For Good: MediaMath’s Zawadzki

February 2, 2018 — by Amarita Bansal0


This video originally appears on Beet.TV.

Looming new European legislation governing how global companies track and process consumer personal data seem to pose a big challenge to the new-wave of pumped-up ad-tech practitioners.

But one boss at the centre of the ad targeting boom thinks the so-called “GDPR”, whose final compliance deadline comes this May, is a force for good that will help clean up ad practices and put consumers in a better relationship with marketers.

“Advertising has not done a very good job of advertising itself,” MediaMath CEO Joe Zawadzki says in this video interview with Beet.TV. “I think GDPR is this wonderful opportunity for the industry to basically say, ‘Let’s make all of those things that we’re doing explicit’.”

New measures in the GDPR, which passed in 2016, include:

  • tighter consent conditions for the collection of citizens’ data.
  • consumers can instruct companies to stop processing their data.
  • automated decision-making and profiling decisions must be made clear.
  • consumers can request decisioning by automated processes be stopped and handled by a human instead.
  • they have the right to request an explanation of automated decision-making.
  • they can request free access, rectification and deletion of data.

And the rules must be followed by any global company processing EU citizens’ data, with penalties of up to 4% of global turnover.

But Zawadzki sees the positives. “What is exciting about it, I think, is having an explicit relationship with the end-consumer,” he says.

“Let’s have a consumer Bill of Rights. Let’s be true consumer advocates and let’s use this as some mode of force to not just do it for the EU, but to use this and decide that what makes sense for a global business is to have a global set of standards.”

Views of executives interviewed for Beet.TV’s GDPR series range everywhere from “not much” to “world-changing”.

Almost two years after GDPR was implemented, we have variously heard views that many businesses remain underprepared, many ad-tech investors remain in the dark, that GDPR could have little impact and that it will fundamentally re-shape digital advertising.

There is one consensus – that GDPR is coming at the same time as a general movement toward people-based marketing, a tactic in which advertisers develop real, consensual relationships with consumers, rather that simply watching them from afar.

All that may be true, but GDPR is a policy instrument. Whilst Zawadzki is eager to adhere to it, he thinks a common technology infrastructure may be required, to underpin an ecosystem in which everyone sings from the same hymnsheet.

“Some of the things that we are missing are some true identity standards – in terms of the use of consumer data, what’s PI, what’s anonymized, what is the role of synonymous in these things,” he adds. “There’s some definitions that continue, I think, to require clarity. That may not, in fact, come pre-May.

“To actually create advertising that works, we have to create those technical specifications and maybe even those companies in order to manage that.”

To read the full article, click here. 


GDPR is Coming: Marketers Must Prioritize Transparency

February 1, 2018 — by Amarita Bansal0


This article originally appears on MarTech Series here.

We spoke to executives across the globe to identify the top challenges that marketers would face in 2018. The common message was — “It’s time to delight the customers. Dare or risk dropping out of the league.”

GDPR is Coming: Marketers Must Prioritize Transparency

Top business leaders from the data management industry have spoken to us on how GDPR would impact the ecosystem and customer experiences along the buyer’s journey. GDPR is definitely one of the top challenges for marketers in 2018.

Alice Lincoln, Vice President, Data Policy and Governance, MediaMath, said, “With GDPR set to take effect in May 2018, many companies are preparing to adjust their business processes and technology accordingly – especially marketers. In the new year, we’ll see the increased scrutiny of major 1P players’ privacy, security, anti-fraud, brand safety, and election-related practices (Facebook, Google, etc.)”

Alice continued, “This level of scrutiny will also extend to 3P companies, and improved industry standards will emerge to proactively address these concerns. There will also be continued momentum in terms of walled gardens’ evolution to provide marketers with transparency that resembles that of 3P companies. Further, we’ll see ongoing developments in improved industry-wide standards to accommodate emerging technologies, including connected TV and IoT.”

Read the full article here.


What is Customer-Centric Marketing?

January 29, 2018 — by Laura Carrier0


It’s seems like obvious advice to tell marketers to focus on the customer. After all, the whole idea behind marketing is to influence the customer’s actions and decisions.

In practice though, many marketers aren’t executing customer-centric marketing, they’re employing campaign-focused marketing or maybe marketer-focused marketing. What’s the difference? In customer-centric marketing, you are first and foremost focused on the customer, their behaviors, their experiences, their needs. With campaign-focused marketing, the focus is on you, the marketer, and your business outcomes. It’s about what you did and what results you got.

That may seem like a subtle difference, but right now there’s a split between the two approaches. In his 2016 Letter to Shareholders, Amazon CEO Jeff Bezos dismissed product-focused and technology-focused businesses in favor of “obsessive customer focus.” That focus has certainly helped Amazon shake up the market. Customer-centric marketing doesn’t mean forgetting about your business outcomes. But it does mean first focusing on the customer such that you satisfy the customer first, which will lead to better outcomes for your business. Here’s a look at what you need to bring that kind of customer-centricity to your business.

The Strategy for Customer-Centric Marketing

Customer-focused is the second of our six pillars of effective marketing (the six in order are data-driven, customer-focused, omnichannel, integrated customer engagement, optimized messaging and comprehensive understanding). Of this half dozen, customer focus demands the most major mind shift for marketers.

In short, the idea is to take a step back from standard marketing processes. In terms of planning, optimization and measurement, consider the customer first. For example, stop comparing how this year’s Halloween campaign did versus last year’s or worrying about how much to spend on mobile versus display on Labor Day. Those are the wrong questions since they are campaign-focused. Make planning & optimization decisions at the customer level (and in real time) — should Jane Doe receive a display ad or a mobile ad when she is researching Halloween costumes? Should she be served an ad for BOGO or for free shipping? Instead of looking at campaigns for June and July, for instance, run a longitudinal analysis over a six-month period to see how all of your 18 campaigns performed in the aggregate against your customers’ behaviors. Did you positively influence your customers’ behaviors during that period? And how much did you influence them — did your marketing lead to incremental impact? That will tell you if your marketing was effective.

In general, customer centricity means thinking about the products and solutions you are using in terms of their customer-actionability — for instance cross-device implementation, sequential marketing, predictive analytics, incremental lift, multi-touch attribution. You are looking at the success of your overall marketing engagements, not a snapshot of how one campaign or one aspect of a campaign performed. The goal is to optimize customer behavior.

How to Implement Customer-Centric Marketing

One primary difference between marketer-focused marketing and customer-centric marketing is the metrics you’re tracking. There are three ways to track changes in customer behavior. On the upper funnel, you’re looking at incremental brand perception. Tools for measuring brand perception include brand perception surveys, which should be focused on the perception change, not awareness and consideration changes, as perception change is what drives behavioral change.

On the lower funnel, the relentless focus should be on incremental return on ad spend (ROAS). Such analyses are based on randomized control trials (CRTs) like the ones used in medical testing. A CRT measures the effects of exposure by including a holdout group (often around 20% of the potential audience) to see if the ad actually influenced a desired behavior.

In the mid funnel, customer-centric marketers are looking for key performance indicators (KPIs) that correlate to incremental ROAS. For instance, clicking on a display ad actually has a low or negative correlation to making a purchase. That’s because display clicks are usually fraudulent, accidental or made by serial clickers. Mid-funnel indicators are specific to your business and your customers, so running correlation analysis to understand which customer behaviors/signals lead to incremental ROAS is imperative. And then run campaigns that focus around driving those particular customer behaviors. Opening emails, reading a blog post, visiting a product page and signing up for a loyalty program are all examples of behaviors that could have a positive correlation for your business.

Note that these metrics are very different from the typical KPIs involved in a marketer-focused marketing campaign, where marketers choose to optimize and measure their campaigns against customer behaviors that aren’t driving true business outcomes like click-throughs, engagement and impressions. Focusing on these not only means that you are not driving business outcomes, but it also leads to frustration for your customers as you try to drive them to behaviors that are irrelevant to the intent of their engagement with your business.

Reorienting marketing around this customer-centric mission requires a radical shift in planning, execution and management. It also means more cross-team collaboration than you usually find in marketing departments organized around functions, channels or the funnel.

The benefits are tremendous in terms of less waste, enhanced customer relationships and, ultimately, strong business growth. If you’re frustrated with your current marketing, then maybe it’s time to try a new approach and shift the focus to the customer first.


How to Tell AI Hype from Reality

January 11, 2018 — by Todd Wasserman0


In 2017 it was hard to escape talk of artificial intelligence. Gartner claimed that AI had reached the peak of its hype cycle and deep learning and machine learning were at the “peak of inflated expectation.”

Such excitement is par for the course in tech. Some have compared AI in 2017 to cloud computing a decade or so before. Though the cloud was hyped relentlessly, actual adoption was slow. AI may be on a similar path.  A 2017 SAS survey of European business execs, for instance, showed just 24% had the right infrastructure in place for AI.

Is the market ready for AI? That’s a moot question because marketers are already using many AI technologies. Some have been using for years without calling it AI. But other AI applications are still years or even decades away. Here’s how to tell the difference:

AI reality

Most AI applications are improved versions of what we would have called “data mining” a few years ago. The difference is that there is often a predictive layer. Marketers are now using these technologies in varying degrees:

• Content curation/recommendations: IBM Watson teamed up with UnderArmour in 2016 to create a personal health consultant that uses a particular consumer’s data to craft a personalized training routine. For instance, a 45 year-old man training for a 10K could use UA’s Cognitive Coaching system to get advice on diet and exercise goals, which are based on data from other consumers in the same demo. More recently, IBM conducted a pilot that involved integrating its Watson Cognitive Bidder with MediaMath’s DSP to incorporate more unstructured data like the weather or location to fine-tune real-time buying decisions that address consumers’ current states of mind.

• Natural language processing: Consumers telegraph their needs via search queries and social media chatter. But because of the way we express ourselves, translating that language into data can be tricky. For instance, as Stanford computer scientist Chris Manning recently pointed out, “terrific” used to mean “to terrify” and has only fairly recently taken on a positive meaning. Thanks to wide daily exposure to natural language, machines are getting much better at picking up on such nuances. Some companies even analyze incoming sales calls to unlock patterns and opportunities.

• Image recognition: This is another AI technology that has become so common that it’s mundane. We take it for granted, for example, that Facebook will be able to “tag” people in photos we post. For marketers, image recognition provides analysis of visual imagery. U.S. consumers now post some 3.25 billion photos a day. Marketers can analyze such images to get clues about behavior and brand affinity.

• Ad targeting. Of course, one of the most popular deep learning applications right now is ad targeting. By analyzing data, machines can optimize ad performance via targeting. Marketers use this tool to improve their ROAS. In theory at least, your ad targeting keeps improving over time.

• Customer service/chatbots. One bad customer service experience can undermine millions in advertising. That’s why some brands are turning to chatbots to handle such inquiries. Experiments with voice-based chatbots have found that they can solve consumers’ queries 82% of the time and leave humans to handle the more complex queries. In addition, brands like 1-800 Flowers and Whole Foods have used chatbots to establish a brand presence on Facebook’s Messenger app.

AI hype

While those technologies are in use right now, some AI being discussed won’t be of use for years — maybe even decades. Those include:

• General intelligence: This is the type of AI usually portrayed in Hollywood movies and sci-fi novels. In reality, today’s AI is “narrow AI,” meaning the intelligence is focused on a single task, like playing chess or understanding spoken language. Exhibiting a well-rounded intelligence — one that can make quick connections and exhibit wisdom — is still a long way away, if it is ever possible.

• Complete automation: Another fear about AI is that it will monopolize all jobs handled by human workers. While some tasks — like customer service — are increasingly going this way, they are often shifting the work of human workers, rather than replacing it completely. As a corollary, look at the spread of ATMs. While you might think that they would have replaced human tellers, instead there were more because ATMs made it cheaper to open new branches and more branches meant more jobs for human tellers.

• Creativity: In advertising, AI can be used for “dynamic creative,” which swaps in marketing messages based on available data. (For instance, you might see an ad for a sunny vacation in Rio if the temperature has just plummeted near you.) But what machines are really doing is grabbing elements that a human has created. While AI systems can now compose music and even write Harry Potter fan-fiction and create movie trailers though, humans are still required to exercise judgment. An AI system is great at presenting random combinations based on exposure, but a human is still needed to make sense of it. Making unusual connections (like Jay-Z’s sampling of “It’s a Hard Knocks Life” from Annie) is still beyond even the most advanced AI system.

As we move on from a year in which AI hype has peaked, 2018 will likely be a building year in which businesses and marketers pragmatically apply the technologies that affect real-world outcomes. Meanwhile, AI itself will keep improving, expanding the palette of what’s possible and what will be possible in decades to come.


MathCapital: A Venture Capital Fund to Support the Next Wave of Marketing Innovation

January 8, 2018 — by Eric Franchi0


The year 2018 finds us at the beginning of the next wave of innovation in digital. Billions of connected devices, the emergence of identity-based targeting and measurement, new consumer interfaces such as AR, VR and voice and all forms of Artificial Intelligence will make digital marketing more effective and digital media more engaging than ever.

It’s an exciting time to be a marketer, media creator and consumer. But it’s never been a more exciting time to be a startup focused on addressing the needs of this giant, and growing, market. That’s why we are thrilled to announce the launch of MathCapital, an early-stage venture capital firm focused on the digital transformation of marketing and media. We created MathCapital to help identify and support the startups that will become the next generation of industry leaders.

As long-time entrepreneurs ourselves — Joe Zawadzki is founder and CEO of MediaMath, and founder of [x+1], and I previously co-founded Undertone – our team is often approached for advice and investment by startups. This led to notable personal angel investments in names many are familiar with, such as AppNexus, Moat, Integral Ad Science, BounceX and mParticle.

We’re also excited to have the support of MediaMath. While MathCapital is a separate entity, MediaMath has committed resources via its in-house innovation group that has incubated a number of profitable agencies. This group will serve as an access point to MediaMath’s 4,500+ clients, 350+ partners and 600+ employees, helping to accelerate all facets of our portfolio companies’ businesses out of the gate.

We’re a startup of four partners ourselves, so we can’t wait to roll up our sleeves and get to work. Our first investments will be announced here soon, so watch this space. And if you’re a funder who would like to learn more about working with us, please contact us here.


Incrementality is the Best Way to Prove Your Advertising is Working. Here’s How to Measure It

December 27, 2017 — by Natraj Ramachandran0


The number one question advertisers are asking now is, “How can I measure incrementality?” In other words, “Did my ad campaign cause the consumer to convert?”

There have been many methods for measuring incrementality, including looking at conversions pre-and post your digital marketing efforts or using a complicated model. But in recent years, marketers have understood that the only true way to measure causality is to use a scientific approach.

In public health, the gold standard in measuring the efficacy of any intervention is a randomized control trial (RCT). For example, if we take two equal populations who have similar behavior and give one population a pill and another a placebo we can then measure the effectiveness of the pill we administered.

The same can be done in marketing. In Incrementality Testing, users are separated into two equal populations. One group (test) receives an ad and another group (control) doesn’t. Once we observe the conversion rates of both these populations we can measure their incremental conversions. More importantly, we can accurately measure the cause and effect of our marketing efforts.

In order to create solid lift measurement tests, you need to follow an experimental design approach:

Setting up your test

Measuring incrementality is simple if you follow these steps:

1) Randomization
Randomization ensures that the populations in test and control are statistically equivalent. This allows us to compare behaviors of the groups.

2) Hypothesis
The key to a strong Lift Measurement test is creating a hypothesis. This can be created by looking at previous observational data. Without a strong hypothesis, the results of your test will be pointless.

3) Primary outcome
Decide the behavior you are trying to observe. For example, if we show an ad for pants, we hope that the desired behavior would be the customer would then buy pants. In your test that would be the conversion event.

4) Length
Select a start and end date of a reporting cycle to ensure that reporting results align to the hypothesis of the test (minimizing distortions due to influences of time).

5) Expected use of outcome
The results of your study should contribute to the advancement of knowledge of your marketing campaign. You will use results to make strategy, budgetary and/or optimizing results.

Understanding your results

1) Lift, incrementality, incremental conversion, confidence, confidence intervals what do they all mean?
A common confusion with any test are the metrics you receive. It is essential that you have defined your results before you receive them.

a) Lift
This will give you the likelihood to convert. For example, let’s say your lift is 30%, this would mean that if we show your ad to a consumer they are 30% more likely to convert.

b) Incrementality

The percentage of conversions you received because of your ad. Let’s say we run our test and get a 20% incrementality, this means we received 20% more conversions because we showed an ad. Or alternatively we would have lost 20% had we not shown this ad.


c) Incremental Conversions/Revenue

Generally, a portion of sales would have occurred despite the campaign. Your incremental conversions/revenue are conversions that occurred as a result of your campaign. These conversions wouldn’t have occurred if your ad was not shown.

1) Confidence %

Your confidence is the % of distribution that sits above zero. For example, if we get a 90% confidence We are 90% sure that the lift is > 0.  This is a good, positive signal we look for to validate expected results.

2) Confidence Intervals

This provides more detail into the health of a measurement. A confidence interval is a range of values that you can be certain contains the true mean of the population. If for example you use a 95% confidence interval, we are saying that we are 95% certain that interval we calculated has the true lift of the population, the narrow the confidence interval the better. The confidence interval is a measure of how precise your lift/incrementality are in your test.

Guardrails/Mitigating factors

1) Why do my lift/incrementality numbers vary so much?

a) Sensitivity — Lift/Incrementality are calculated based on small conversion rates. In a recent campaign the measured conversion rate for the test group was .03% and .02% for the control group. This would equate to a 33% incrementality.

If the test group conversion rate increased to .04% this equates to a incrementality of 50%.

a) Seasonality – The control group response rate can vary drastically depending on different buying seasons. For example, if we ran a test during Black Friday, the control group response rate may be higher than other times of the year as users are more likely to purchase during this time. This would in turn dampen the influence of your ad campaigns.

b) Brand Awareness – Some products are more familiar to consumers than others. If we ran a test for Snapple and another test for Honest Tea, we should expect larger lift/incrementality for Honest Tea vs Snapple. Snapple is more recognized by consumers than Honest Tea. This means that Snapple consumers are more likely to convert even without the influence of an ad campaign. This means the ad campaign run for an Honest Tea would have a larger impact on consumer behavior.

1) Can you tell me results at the strategy or tactical level?

This is an almost impossible question to answer unless your test is set up appropriately and you have a well-defined hypothesis. Take for example a prospecting and remarketing campaign that are run in tandem. In this scenario, there will be a natural flow of users from the prospecting campaign to the remarketing campaign, causing overlapping populations. As a result, there may be some users who receive two ads then convert (see below).

It becomes almost impossible to understand and attribute the conversion to the appropriate ad. Additionally, this type of behavior isn’t seen on the control side as they are held out to all ads.


In order to understand tactic/strategy level results it is best to test each strategy one at a time to avoid overlapping audiences and audience flow.


Incrementality testing can be an incredibly powerful tool if implemented correctly. Many public health organizations continue to use it to measure causality for clinical interventions. Testing in marketing is still nascent and still needs more exploration.

In order to have an effective test you need to:
1. Ensure you determine what you would like to learn from the test before campaign and incremental lift test(s) are set up.
2. Determine the primary goal you are testing (Visits, Checkouts, Signups).
3. Ensure the results of your study contribute to the advancement of knowledge of your marketing campaign.
4. Make sure the results will be used to make strategy, budgetary and/or optimization decisions.

Understand that your results will continue to change because the behaviors of your clients are also constantly changing. In order to make your tests useful have a test plan at the beginning of ever year and continue to iterate on your hypothesis. Running one or two tests won’t provide you with the answers you need. Measurement for measurement’s sake is doesn’t advance learning.  Don’t just measure to say you did it, and then let the results linger in a PPT print out on your desk, measurement should be actioned on.  If you can’t make a decision or take an action based on what you have learned, then that measurement isn’t useful.


In 2018, Marketers Will Discover More AI Applications in Programmatic Advertising

December 14, 2017 — by Amarita Bansal0


This Q&A with Chris Victory, VP Partnerships at MediaMath, originally appeared on eMarketer.

Even when digital ads are highly targeted down to individual users, it’s hard to get consumers to pay attention to them — let alone interact with and enjoy them. Using artificial intelligence (AI) to further optimize digital ad spending is the next step marketers are taking to make ads a more welcome part of consumers’ digital lives. eMarketer’s Tricia Carr spoke with Chris Victory, vice president of partnerships at demand-side platform MediaMath, about the demand for AI in the programmatic advertising space and the core elements of his company’s partnership with IBM’s Watson.

eMarketer: Are marketers typically aware the benefits of AI in programmatic settings, or are you doing a lot of education on this front?

Chris Victory: Marketers who use programmatic advertising have seen the benefits of implementing machine learning into their decisioning and spend of their marketing dollars. AI is the next step in this journey, and everybody is super excited about it. But the challenge lies in delivering something that goes beyond what machine learning does today and is truly artificial intelligence. Those in the programmatic setting are perfect candidates to use the next generation of AI, because machine learning is core to what they do and it’s much more palatable to them.

eMarketer: How are you helping to propel the use of AI in this sense through your partnership with IBM Watson?

Victory: There are three main areas where we focus on the power of AI. The first is infusing AI into the decisioning process on how to better spend your money. Today that’s based on machine learning, but think about all the additional data that can be ingested, like sentiment and mood of the user, and a deeper contextual analysis of a page. These types of things are driven by AI and can be infused into the bidding process to make better decisions.

The second is AI-driven ad creative — it’s like the next-gen version of dynamic creative optimization — to deliver things like interactive ad units. For example, you can interact with an ad by asking a question, and it will give users different answers based on the question asked.

The third is using AI across customer analytics and marketing insights. Today you can look at analytics through a number of different technologies that give insights into your customer base. But the next generation of that is using AI for more predictive analytics. For example, don’t just tell me who my lapsed customers are, but show me who my potential lapsed customers are based off of behaviors. Then you can create user segments to target with specific messages related to their potential of becoming a lapsed customer.

eMarketer: Are there any roadblocks in the industry preventing marketers from experimenting with AI to optimize their programmatic advertising?

Victory: There’s a lot of noise out there — this doesn’t just apply to AI, but marketing technology in general. AI is the buzzword now, and there are a lot of companies getting money from venture capitalists because they have AI in their strategy. But it’s getting harder for a marketer to cut through the noise and understand exactly what they can do with AI — what’s real and what’s not.

eMarketer: Will consolidation happen in 2018, or is it a long way off?

Victory: I think there will be a proliferation of companies before there is consolidation. Next year will be interesting. You’ll see a lot of companies coming in.

There will be a real need for agencies to play a role here and become experts in the use and the application of AI, because agencies are engaged with their clients. It’s very similar to the evolution of the agency role in the programmatic world.

eMarketer: What else do you expect to happen in this landscape in 2018?

Victory: There will be a lot more marketers and advertisers jumping in headfirst to embrace and adopt AI. Once they start seeing the results of some of the applications of AI, there will be a snowball effect and people will want to do more.

There will also be continued proliferation of companies in this space and in the different applications of AI. Virtual reality has been the hot topic for the last year and a half, but now AI is. It’ll be interesting to see where venture capitalists make their nest here, and where that directs the market.