Online Audience Measurement: Cross Check Your Instruments

BtoB’s 2006 Interactive Marketing Guide just came to my office today. (Yes, Virginia, people do still read print magazines.) Of course the Online Publishers section caught my eye and I immediately checked the chart of online publishers on page 33 to see how Penton was represented. Only two Penton titles made the chart, Windows IT Pro and IndustryWeek, but the traffic shown for these two sites wasn’t even in the ballpark.

BtoB reported Windows IT Pro as having 166,000 unique visitors in March 2006, but Windows actually had nearly 2.8 million unique visitors in March according to WebSideStory’s HBX, our industry-recognized, third-party, independent and audited metrics system. BtoB also reported IndustryWeek as having only 34,000 unique visitors in March while HBX reported over 104,000 unique visitors. I also noticed that the numbers for EETimes and CNET seemed really off as well. Only 404,000 unique visitors to CNET in March? No way.

So what’s going on here? The answer is in the fine print at the bottom of the chart: “Audience measurement from comScore Media Metrix, March 2006. (Some) audience measurement figures self-reported.” Ah … it’s all becoming clear now. From what I understand, BtoB used comScore’s Media Metrix 2.0 as the source for many of the traffic numbers in its chart. For other sites, it used the publisher’s own reported numbers.

I find it odd that BtoB would mix two different methodologies in the same chart. That’s usually not a good way to draw accurate comparisons. Even then, why didn’t BtoB confirm these numbers with the publishers? I realize that there are a lot of companies to call, but this is a pretty important chart and is only published once per year.

An even bigger issue is the fundamental limitations of Media Metrix — and other — audience and advertising measurement systems. Media Metrix 2.0 uses a panel of around 100,000 U.S. Internet users. Through this panel, comScore attempts to get a large enough sample to statistically project a site’s overall traffic. They claim to be able to report traffic numbers for over 8000 web sites. For large consumer sites, portals, search engines, and even some of the very largest B2B tech sites (multi-million visitors per month), they can provide relatively good data. But for the vast majority of B2B and niche specialty sites, they simply don’t report traffic or advertising market share reliably … assuming you even show up in the samples.

Here are the problems with relying only on these kinds of systems to measure a site’s overall traffic:

• Media Metrix 2.0 is U.S focused. Many B2B web sites have significant global traffic … sometime up to 50% can be from outside the United States. Europe and Asia are critical to many markets and we cannot just ignore them in reporting traffic. It is the World Wide Web, after all.
• Many B2B web sites or niche consumer sites simply aren’t big enough to measure accurately. With only 100,000 users across the United States, it’s incredibly difficult if not impossible to get a statistically relevant and projectable sample for these smaller sites. Although one would think that with 2.8 million unique visitors per month, Windows IT Pro would qualify as a relatively large B2B site and one where you could get a pretty good sample.
• These measurement service don’t accurately take into account networks. Windows IT Pro, for example, isn’t a single site, but rather a network of several different domains and subdomains. EETimes, CNET, and many others are similar in this regard. I’ve seen significant discrepancies in online measurement tools because they don’t have a brand’s network properly organized in their hierarchy.
• Often these measurement services don’t take into account alternate domains or brand identities. Through it’s lifetime, Windows IT Pro has been called Windows NT Magazine, Windows 2000 Magazine, and Windows & .NET Magazine. Each iteration had its own URL and brand recognition which has been kept active as a courtesy to long-time readers. Online measurement tools have a difficult time accounting for situations like this.

comScore has an expanded version called Media Metrix XPC that has over 2 million global Internet users. It is possible that XPC could have reported traffic numbers more in line with reality, but many of the same issues I’ve outlined here still apply and cause the data for many B2B and nich consumer sites to be questionable. BTW, this isn’t just a comScore issue. Similar problems exist with Nielsen//NetRatings AdRelevance and @Plan … both services that I have personally used before here at Penton.

As an instrument-rated pilot, I have been trained not to trust any single instrument. Rather, it has been drilled into my brain to cross-check every instrument against another instrument to make sure the two are in agreement. You can bet that cruising along at 10,000 feet while not being able to see my own propeller in front of me that I’m constantly cross-checking my instruments. If they agree, all’s fine. But if they don’t, I need to quickly figure out which instrument is not giving me accurate information so I can react accordingly.

Online audience and advertising measurement systems are simply one tool for publishers to measure themselves versus their competition and for marketers to determine where to spend their limited ad budgets. But they should not be the only tool that is used. I applaud companies like comScore and Nielsen//NetRatings for coming up with systems to address the very difficult process of measuring audience and advertising across multiple sites. They will continue to improve over time. But even as sophisticated as these systems are, they can be inaccurate and we need to take them with a grain of salt. 

Publishers: Make sure you know HOW your company and sites appear in these systems. As marketers get more sophisticated, they will begin to rely upon them to make purchasing decisions and will shape their perception of your site.

Marketers: Don’t rely on these systems too much when making buying decisions. They can be good tools in the right situations and for the right kind of sites, but realize their strengths, weaknesses, and limitations. Always ask for an audited traffic report from any publisher you plan to do business with.

Measurement Companies: Keep on refining your systems. You’ve made great in-roads, but we’ve still got a long way to go. Make sure you educate clients and prospective customers on the strengths, weaknesses, and limitations of your systems.

And for all of us, make sure to constantly cross-check those instruments!