How to know if your DSP is really working

By: Ari Buchalter  Published: October 18, 2011

Link to original article

Depending on who you ask, there are anywhere from 10-20 billion display impressions available globally each day for demand-side platforms (DSPs) and their clients to purchase on real-time bidding (RTB) sources like ad exchanges and supply side platforms (SSPs). On top of that, some DSPs have now enabled buying of other types of media, including premium display, social, video, and mobile, bringing the total addressable ad inventory closer to something like 30 billion daily impressions that can be serviced through the execution and decisioning layer of a DSP.

Let's put that in the context of a single advertiser. For easy math, let's say a display campaign spends $100,000 per month, or a little over $3,000 per day. At an eCPM of say $2.00, that's a little over 1.5 million impressions per day, or roughly 0.01 percent of the total available supply. Even if you tweak the assumptions above to be 10 times higher or lower, the bottom line is that a typical campaign is buying an incredibly small fraction of what's out there. But with such a small fraction to play with, how do you ensure that you're getting only the very best-performing impressions and audiences for your budget, and not simply an average cross-section of the total, or worse, a below-average one?

You can start with retargeting, which we all know works, but has limited scale. But what happens after you've remarketed the consumers who are already engaged with your brand and products? How do you bring new customers into the conversation? How do you get new prospects to convert? In short, how do you decide which 0.01 percent of the unwashed mass of impressions is the right 0.01 percent for your campaign?

This article introduces four important concepts and folds them together into a framework designed to pinpoint that 0.01 percent. As any data modeler knows, being able to single out the top decile, or in this case the top hundredth of a percentile, generates massively disproportionate returns. It's no different here.

Impression Quality is a measure of how effectively an impression delivers the action desired by a particular buyer, be it a positive brand response, a click, a purchase, or even ROI. When I say "impression" I don't just mean the blank space on a website waiting for an ad to show up. I mean that blank space, combined with the creative that is served into it, combined with the media in which it sits, combined with the user who is going to see it. The sum of all those things — media + creative + user — is an impression. Mathematically, the impression quality is an algorithmic prediction of how likely the user is to take the action desired by that buyer on that impression, relative to all other impressions.

For example, if a buyer's goal is to drive purchases, and a particular impression is predicted to have a response rate of 10 percent per 1,000 impressions, versus a campaign average of 2 percent, that impression has an impression quality of 5.0. The impression quality prediction is based on dozens if not hundreds of variables describing the creative (offer, message, image, etc.), the media (site, content category, ad size, ad placement, time of day, day of week, etc.), and the user (via any number of cookie-based and non-cookie-based user variables).

Impression quality is incredibly important because it separates the wheat from the chaff — the quality impressions that drive high user action rates, from the low-quality impressions that don't. And since it depends not just on the media, but on the user who is seeing the ad and on the creative served, the very same impression could have high quality to buyer "A" (because that user is buyer A's ideal target consumer and that publisher has relevant content to buyer A) and low quality to buyer "B" (because that user is not buyer B's target, nor is that publisher relevant to buyer B).

The same impression could even have different quality to a single buyer, depending on which creative the buyer chooses to serve (e.g., because the user is more likely to take action on creative 1 versus creative 2).

High impression quality often comes at a high price, but not always. Imagine you place a certain ad on the homepage of a certain website and show it to a certain user at a certain time. That ad will have a certain click-through rate or response rate, or whatever "action rate" the buyer cares about, regardless of what the publisher happened to charge for it. Whether it was a $20 CPM, $2 CPM, or a free impression that was part of a "make good," the intrinsic quality and performance of that impression is actually the same. Price doesn't cause impression quality, but it does correlate because some of the media characteristics that drive high ipression qality happen to be the same ones that publishers charge higher prices for. That said, much if not most of the information that goes into calculating impression quality is only known by the buyer. Just as beauty is in the eye of the beholder, impression quality is specific to the buyer, which is why it's critical for buyers to have the algorithmic capability to calculate it, for billions of impressions a day, in real-time.

Buyer Value is the dollar value that a specific impression is worth to a specific buyer, and it's a function of both the impression quality and the buyer's goals. In essence, it translates the impression quality into a CPM. Mathematically, buyer value is simply equal to the predicted user action rate multiplied by the buyer's goal value.

For example, if an impression has a predicted 2 percent response rate per 1,000 impressions, and the buyer's goal is a $75 CPA, that impression has a buyer value of $1.50 CPM. A different buyer, running a different creative, will have a different predicted action rate and a different goal value, resulting in perhaps a $0.10 CPM buyer value, while a third buyer with their creative, predicted action rate, and goal value may have a $7.75 CPM buyer value — all for the same impression. That's why publishers actually have very little insight into buyer value — like impression quality, it's primarily a function of factors that are specific to each advertiser.  And since it's related to impression quality, a sophisticated algorithm is required to accurately determine buyer value.

Market Value is the predicted price at which an impression is likely to clear in an RTB auction. Unlike Buyer Value, it isn't dependent on any single buyer, but rather on the interaction of all participants in the RTB marketplace. Mathematically, it's a prediction of the price at which an impression will clear, with a certain probability. For example, for a certain impression, on a certain site, a $5.00 bid will win 94 percent of the time, a $3.00 bid will win 68 percent of the time, a $1.00 bid will win 40 percent of the time, etc. Of course, it's not just a function of the site, but of many other variables like ad size, position on the page, time of day, and dozens of other variables. In contrast to Buyer Value, publishers do know the Market Value, because it quantifies the price at which their impressions clear, and at which they get paid. Buyers can easily know it too, down to the impression level, if they leverage an algorithm that can accurately predict it based on the myriad variables that characterize every impression.

And finally, Relative Value quantifies the gap between what a single buyer thinks an impression is worth (which governs the bid price), and what the rest of the market thinks it's worth (which governs the clearing price). Mathematically, it's simply a ratio of the buyer value to the market value. In other words, the larger the relative value, the more that particular buyer values the impression over the market, and hence the greater the value to the buyer. It's important to note that the relative value of an impression is not necessarily a reflection of price; it's the relative comparison of the buyer value to the market value that matters. A relative value of 3.0 could mean a buyer value of a $0.30 CPM versus a market value of a $0.10 CPM, or a buyer value of a $12 CPM versus a market value of a $4 CPM. As we'll see in a moment, the latter is much more interesting.

So what does this all mean? Let's take a look at the graph. It's a simplified 2×2 representation of impression quality versus relative value. Think of it as effectiveness (impression quality) versus efficiency (relative value).

The shading indicates how the universe of available RTB impressions populates the graph. Darker areas contain more available impressions in the RTB landscape, while lighter areas contain fewer impressions. Let's take a look at each quadrant of the graph:

Quadrant 1: Non-performance
The impressions here are of low quality, meaning they aren't very good at generating the actions the buyer cares about. Moreover, these impressions have low relative value, meaning they are "expensive" (i.e., the market value is high) relative to what they are worth to the buyer. They may only cost $0.50 (not expensive in an absolute sense), but they are too expensive relative to the awful performance they generate. Unfortunately, this is where the vast majority of RTB impressions are found — roughly 50-70 percent. Nearly all of these impressions won't be effective for the buyer's campaign, and the cost of winning them exceeds what they are worth to the buyer.

But perhaps the worst news is that this is also where most DSPs spend their clients' dollars. DSPs that don't have the algorithmic capability to find the high-quality and high-value impressions typically end up guessing at who their clients' audiences should be with pre-defined "audience buys," often incurring incremental data costs to do so. While a segment here or there may pop, the result on average is usually poor. In the world of SEM, search bidders tend to know in advance what keywords work, how well, and also what to bid. But in display, it's usually not obvious what impressions and user characteristics make a campaign work. You need an algorithm to determine which impressions and users to buy and at what price. You simply can't "audience guess" your way through 15 billion daily impressions, especially not when you have large budgets to spend. If you do, you'll wallow in this pit of non-performance.

Quadrant 2: Cost-driven performance
As with non-performance, the impression quality here is low. But the relative value is actually high, meaning that even though these impressions don't perform very well, they are dirt cheap enough to squeak by and meet the buyer's goals. It's a bit like fast food — you're getting a pretty lousy burger, but it only costs $1, so it works for you. But just as that isn't a great long-term strategy for your health, it's also not a great long-term strategy for you campaign's health, or for the ecosystem at large, for that matter.

From the advertiser's standpoint, you may be technically hitting your goal, but you're doing it in a cheap, low-quality environment. These impressions are most likely on long-tail sites, below the fold, deep in the frequency curve, or all of the above. Not the best use of the buyer's dollar. From the publisher's standpoint, the eCPMs are low, and while that may be OK for some, it's not going to encourage the quality publishers to show up in force.

Unfortunately, this is also where a lot of impressions live in the RTB world — about 20-50 percent. And sadly, as the ad exchanges and SSPs know, a lot of DSPs service their clients by simply "spraying and praying" low bids. They are bidding low and winning low-quality impressions that are cheap enough to technically meet the buyer's goal, but actually failing to maximize total value, which is what brings us to the next quadrants.

Quadrant 3: Quality-driven performance
Now we're dealing with high-quality impressions that deliver high-action rates as defined by the buyer: strong brand lift, strong engagement and click-through, high response and conversion, or high ROI. However, the relative value here is low, meaning you are going to pay full price for that quality. And that's fine. A high-quality impression worth $6.50 to the buyer, who wins it for $6, likely results in a happy buyer and a happy seller.

For a given campaign, less than 10 percent of RTB impressions are typically in this quadrant. The good news is that's billions of daily impressions. The bad news is that if your DSP isn't capable of modeling these impressions out and quantitatively defining their characteristics, you'll only buy them about 10 percent of the time (or less actually, if you're just spraying low bids that have a lower chance of winning these quality impressions), and that means you'll also perform 10 times worse.

Quadrant 4: Value-driven performance
This is where RTB realizes its full potential. Here, buyers enjoy the best of both worlds: not only is the impression quality is high, resulting in high effectiveness, but so is the relative value, resulting in high efficiency as well. These impressions are gold for the buyer, but like gold, they are hard to come by. For any given campaign, only a small percent of impressions will be in this quadrant.

What makes them even more interesting is that they are also gold for the publisher. Just because their relative value is high doesn't necessarily mean they are cheap. In fact, cheap impressions often don't make it into this quadrant because many cheap impressions happen to not have very high impression quality (some do, but most do not). It simply means they cost less than what they are worth to the buyer. A typical scenario here would be where the impression quality is so high that it has a buyer value of $12, and a Market Value of $4. So the buyer is getting high performance at a great relative price, and the seller is realizing high absolute eCPMs. Great, right?

But imagine if the buyer and seller could then take this to next level. Imagine the buyer saying, "Hey, your inventory is working great for me in RTB — I'd be willing to pay even more than a $4 CPM to get more of it," while the seller says, "I've only been putting a trickle of that inventory into RTB because I was worried about channel conflict, price erosion, and advertiser quality. But for an $8 CPM, you can have all of it." Deal.

So, having explored each of the four quadrants, how should a buyer spend their monthly display budget to find the best 0.01 percent of the RTB universe? By picking off the impressions and users tucked into very top-right corner of the graph (labeled "maximum performance"), representing the highest-quality and highest-value impressions for that campaign. As budgets increase and more impressions are needed, larger scale buyers should expand out from that corner, carving out larger portions of that upper-right section of the graph to meet their budget needs (as the concentric circles in the graph suggest).

Ironically, most DSPs actually spend their clients' budgets in the lower left portion of the graph (non-performance) because (a) that's where most of the impressions are, and (b) they lack the ability to quantify and model things like impression quality, buyer value, or market value. Instead, they are making manual, time-consuming, but inaccurate guesses at what to buy, or simply just bidding low and praying. That underscores the fact that none of this smart buying is possible without the underlying algorithms to determine these quantities, and to do so fast, accurately, and often in a data-rich and highly dynamic environment. That's the area where most DSPs fall down.

How do you know if your DSP is doing this? A good way to tell is to ask for log-level data on the bid price for each impression they deliver for your campaign. If all their bids are simply falling within some limited range (e.g., $1.25 plus or minus $0.25), then you know you've got an algorithm-less DSP. Alternatively, one can ask: "If I were to give you an impression and all its associated characteristics, could you tell me the predicted user action rate (and hence the impression quality), as well as the correct buyer value, market value, and relative value?" (Your DSP may not use these terms, but the underlying principles should be the same).  If they say yes, then ask for this data on all the impressions for your campaign.

If they can't, then you have to wonder how they are spending the budget. Is it on the right 0.01 percent? If they are just guessing at audiences or bidding low, the odds of that are about the same as finding a four-leaf clover on the first try. Good luck!

Ari Buchalter oversees MediaMath's product team, designing and delivering best-in-class solutions to the market via the TerminalOne platform, as well as MediaMath's business operations group focused on delivering exceptional results to clients.