Overcounting Web Audiences
When you hear a web site's audience size, it's most often stated in unique visitors. And those unique visitor counts are usually generated by analytics packages that use cookies to determine unique visitors (browsers). There are other methods (beacons and ip addresses) but those methods are not as popular as cookies because they suffer from a number of problems.
Well today comScore released the results of a study on the role of cookie deletion in web audience reporting that they've been working on for months. And they determined that using cookies to determine web audience will on average overcount your audience by 2.5x. That's right - 150% higher than it actually is.
And it's worse if you have an audience of hard core cookie deleters. Seven percent of the web audience (as measured by comScore's panel), deletes cookies more often than four times a month and on average they will be overcounted by 12.5x a month.
It's been a rule of thumb that third party cookies (set by ad servers and ad networks) are the primary target of cookie deletion software so if you set first party cookies, you don't have to worry so much. Wrong. It turns out cookie deleters make almost no distinction between third party cookies and first party cookies and delete them with about equal frequency.
I got a preview of comScore's press release and wrote this post on Sunday because I am flying most of the day on Monday. I'll link to it when I land but it should be linked to on this page. It's a good release, filled with lots of numbers and quotes from academics, ad agency execs, and industry insiders who all agree that this is important research.
It proves something I've always said. You cannot rely on your own analytics data. You need third party data as well. That's not to say that third party data (primarily panel data) is perfect either. You have to triangulate between all the numbers to get a decent view of what's actually going on.
People often say that comScore's numbers (and it's competitors numbers too) are too low. It's true that panel data is generally a lot lower than your own server logs. But that doesn't mean your server logs are right.
My roommate works at an ad agency tracking ads and is always showing off their latest ads. I'm a heavy firefox user and many times the ads she shows me don't show up because I've already been to the site earlier. We just open up IE and sure enough, their big fancy ad shows up. I'm sure there aren't many dual browser users, but that could also add to cookie inflation.
Posted by: Yaacov | April 16, 2007 at 11:17 AM
It's worth mentioning that comScore's products include online audience measurement (http://www.comscore.com/metrix/default.asp).
Thus, it stands to gain financially from discrediting competing methods.
They recently filed to go public (http://tinyurl.com/2x9rdy), so spreading FUD about competitors can't hurt...
Posted by: Elad Kehat | April 16, 2007 at 03:27 PM
What's news to me is... that this is still news!
By gosh, it seems literally nothing has changed since day one of the web. And in a medium that constantly shouts from the rooftops about its exactitude and precision and measurability, and in a world of VCs and inventors spending non-stops resources innovating, that is stunning. How is it that audience measurement is in exactly the same state it has been in since 1995?!?
Posted by: Grand Egress | April 16, 2007 at 06:03 PM
i'll wait until i can see the internals.
Posted by: jim | April 16, 2007 at 08:07 PM
Grand Egress, I completely agree with you.
The idea of measuring unique visitors is pretty useless. Pageviews and Visits (sessions) are much more accurate and standardized.
Advertising is usually priced on a CPM or CPC basis. CPM meaning Cost per thousand impressions and CPC meaning Cost per click. In other words, measuring unique visitors doesn't matter for online advertising.
The WORST part of the press release is the "industry experts" section. Vice Presidents and CEOs are not experts in web measurement. How about asking real experts at web analytics vendors. They would tell you that cookies are currently the best method available. Panel measurement is highly inaccurate except for the most popular websites.
Posted by: Ben | April 16, 2007 at 08:39 PM
I dunno, Fred. This isn't such big news, really. Articles such as these about Jupiter and Atlas - each of whom, admittedly, have different motivations behind their data - have been circulating for a long time. And while cookie deletion exists, the key is ensuring that your measurements track against a consistent baseline. If your revenue model depends upon advertising, you probably should use an independent party for tracking in order to minimize bias and provide a basis for setting competitive ad rates. If your revenue depends on e-commerce, use a tool that you're comfortable with that shows solid behavioral information. In either case, what matters most is how effectively you're able to action the data, even if it's imprecise.
Posted by: Tim Peter | April 16, 2007 at 11:03 PM
You may be interested in this recent article on YGG, it's created quite a stir (hey, Seth Godin blogged it and he's a genius!)
The Devil & Online Advertising
Posted by: Eric | April 16, 2007 at 11:32 PM
Ben, it's true that the CPM and CPC models are important components of Internet advertising, but to ignore reach/frequency metrics (i.e. unique visitors or unique recipients of ads served) is to deny the importance of Internet advertising as a brand-building medium. Reach/frequency metrics are the cornerstone of virtually any offline media plan -- so why wouldn't they be fundamental to online advertising?
Posted by: Andrew Lipsman | April 17, 2007 at 11:59 AM