comScore Pokes Stick at Stats

One of the issues raging within the online world is the accuracy of statistics. There are so many ways to slice and dice the numbers, that it’s difficult to know whether any of them are on the mark. (See my Apples vs. Oranges vs. Plums vs. Pears post). comScore, which makes its living from providing online statistics, wades into the debate today with a study/marketing brochure looking at whether cookies are a valid tool to measure unique visitors to Web sites or the number of unique visitors that were served an ad by an ad server. comScore “discovered” that Web site server logs that use first-party cookies (which are left on a computer after it visits a Web site) can exaggerate the number of unique visitors by as much as 2.5X (or 150%). Question #1: what is comScore suggesting: that cookies are a bad way to measure unique visitors, or that anyone not using comScore’s service is making a mistake?

Subscribe to "This Week in Canadian Startups"

A weekly newsletter featuring news about the Canadian startup scene, as well as great startup content. We'll also send you an e-book featuring more than 100 of the leading digital marketing services.

This entry was posted in Aside. Bookmark the permalink.

http://gostats.com Richard

Allow me to add a few relevant points:
-Specific metrics like “sessions” or “daily uniques” (available from GoStats for example) are much more reliable and accurate than any generalized monthly summation. The reason for this is the multi-session lifespan of the average cookie: While 7% Internet users may clear all their cookies each month only a fringe will clear all their cookies daily (or hourly).
-The article was intentionally vague on the specifics of the “deletion/reset” of the cookie. Often a cookie “reset” is really a cookie “refresh”. It’s not difficult to determine by observation if a cookie is deleted or refreshed. It is actually irresponsible and misleading to group those two data parts together since for the purpose of tracking, a cookie can be refreshed without losing data.
-Determining monthly visitors becomes less accurate than a daily visitor total regardless of which methodology is used (comscore, web logs, GoStats, etc). On top of that, the mobility of today’s internet connections allow people to access their favorite site from different computers/networks through the month even further skewing the results. So if a company is try to sell you on the accuracy of the â€œmonthly unique visitorsâ€, you should question their expertise in understanding exactly what is important and relevant in stats data.
-Most basically: For monthly data, “pageviews” are your best metric of volume. While augmentation for unique visitors are best counted from a daily average.

(sorry for filling your post up with my massive comment)
http://www.comscore.com Magid Abraham

Mark:
Mark:
What is it about this study that raises your question? This study analyzes the dynamics of cookie counting and how people can be double and triple counted when they delete their cookies. I would assume you understand how the double and triple counting happens. If you believe cookie deletion happens, and many other studies support what we found in terms of rates of deletion, then the over-counting logically follows. If a 2.5x overstatement by counting cookies is not enough to question its measurement validity, then go ahead and say so.
As far as comScore’s own measurement, it stands on its own merit for many more other reasons than problems with cookie counting. The only thing this study suggests is that if you find third party UV measurement to be less than half what you ‘internal’ data suggests, you should not be surprised or automatically assume your internal data is right. Chances are the opposite is true.
Dale Hannon

The question I found myself asking when I read this study was are all these people manually managing (read deleting) their cookies, or are they running anti-spyware utils that do this for them?

If the latter, I wonder if there’s any good data regarding the default behavior of the most prevelant anti-spyware applications.

It’s germane, I think, because the default behavior of these applications may be a contributing factor to the parity in 1P/3P cookie mortality. In which case, we can’t necessarily assume that the users themselves aren’t making a distinction.

The end result is the same, I suppose.
http://gostats.com Richard

Magid, it seems to me that Mark is not convinced by the logic and reasoning of the conclusions from comscore over GoStats. If you really do think that stats services are off by that much, I am available for consulting to help you understand how the traffic is actually determined by GoStats.
-Oh, I should also mention that stats services (like GoStats) track both UV via cookies and via host IP addresses. So, the comscore arguments are really moot. Further comscore can only make estimations based on the data sample available to their partners. *I can see a huge margin of error for comscore right there* Specific Accuracy should not be a selling point for comscore over GoStats for obvious reasons.

Comscore is a great system in it’s own right. However, it’s definitely not a drop in substitute for GoStats in any way.

Magid, send me an email, I’d like to chat more with you about this.

inbox

comScore Pokes Stick at Stats