Pondering Panels

ComScore, Nielsen report dissimilar numbers due to methodology differences

In the world of online user measurement, two companies are competing to serve as the equivalent of television’s Nielsen ratings service.

One is NetRatings Inc., whose controlling ownership is VNU NV, parent company of the Nielsen Media Research television ratings company. (The NetRatings service is branded as Nielsen//NetRatings.) The other measurement firm is comScore Networks Inc., which includes the company once known as Media Metrix.

Both companies measure audiences by recruiting people willing to install monitoring software on their computers. But Nielsen and comScore come up with different numbers for the sites they measure – sometimes dramatically different. Newspaper online directors find themselves in the uncomfortable position of having to pay substantial amounts for access to data they believe is unreliable, while knowing that their revenues may rise and fall based on decisions by advertisers who use the comScore and NetRatings numbers.

As an illustration, here is one month of 2004 data for two regional newspapers run by the same parent company (the company asked that the papers not be named):

Why do two companies using similar measurement approaches come up with substantially different counts of unique visitors and page views? Ultimately, the reasons lie in the different techniques the companies employ to create their user panels.

Panel Methodology

Nielsen recruits its NetRatings panel via telephone using random digit dialing (RDD). Incentives offered include $50 and $100 savings bonds. Historically, RDD has been the most reliable way to recruit a representative sample for any kind of survey or poll – by dialing random digits, even people with unlisted numbers can be included. Researchers are finding, however, that it is increasingly difficult to get a good sample through telephone surveys because of factors including cellular phone use and antipathy to telemarketers.

Historically, comScore has recruited its panel members through MarketScore, a Web site that offers free software in return for participating in the panel. The software offerings include an “Internet accelerator” (which speeds up downloads for dial-up users) and Symantec e-mail virus scanning.

Both companies have weighted their data to reflect the demographics of Internet users as estimated through traditional market research. But through 2002, the panel recruitment strategies significantly differentiated the two services. Nielsen had a panel recruited through a more scientifically valid sampling technique, but its panel was too small to be useful for understanding all but the largest Web sites. ComScore had a much larger panel that could produce data for smaller sites.

Since comScore acquired Media Metrix in 2002, the services have become more similar. Media Metrix was, and continues to be, a service based on a panel recruited through RDD. So now comScore offers both an RDD-recruited Media Metrix panel and a Web-recruited XPC panel.

The Web-recruited panel is much larger: More than a million people in the U.S., compared with about 50,000 recruited through RDD, according to comScore. It enabled the company to measure traffic to thousands of sites that Nielsen//NetRatings couldn’t. The size of the Web-recruited panel also enabled comScore to offer LocalScore, a service providing traffic data for people living in 100 local markets. (NAA has licensed the LocalScore service and used it to produce “Identifying your Online Audience”.

In the face of the competition, Nielsen//NetRatings has added a Web-recruited “megapanel” and has announced that it, too, will be able to offer local market measurement services.

The two companies, arch competitors, are known for sniping at each other. In early 2004, Nielsen adjusted its estimate of the size of the total U.S. Internet population upward. “We can’t figure out how they came up with such a steep rise in Internet users in over one year,” comScore CEO Magid Abraham told Advertising Age.

Michael Saxon, vice president for media products for Nielsen//NetRatings, said the adjustment was driven by a real increase in the number of Web users and by new data on the size of the U.S. population. “We are very honest about our methodology,” Saxon said. “Our clients find it very difficult to say the same about comScore.”

Researchers for newspaper companies believe both services underestimate at-work usage because so many corporations won’t let employees install non-approved software, including the monitoring software used by comScore and Nielsen//NetRatings. Representatives of both services say they have been working hard to increase their access to at-work users, but both maintain they account fairly for at-work usage already.

Major newspaper companies, which subscribe to both services, find that NetRatings tends to show higher usage for newspaper Web sites. In part, newspaper researchers believe this is because comScore has a smaller proportion of at-work users because of its online panel recruitment. But this argument is contradicted by the fact that the national comScore data, just like NetRatings’, comes from an RDD-recruited panel.

Difficult Decision

So, confronted with conflicting data from companies using different approaches, what’s a site manager to do?

“You buy the best research you can, use the best stuff you can, and basically just punt,” said Tim Stehle, who oversees Web measurement for Knight Ridder Digital. “It’s an insoluble problem. Broadcasters face the same issue.”

While the Internet is potentially the most measurable medium ever, Stehle said, audience reach and frequency will never be known with certainty unless a Web publisher requires registration to use an entire site – and even then, unless the user has to provide accurate personal data such as a credit card number, there is the possibility of multiple registrations by the same person.

Steve Yelvington, vice president of strategy and content for Morris Digital Works, argues that too much energy is being wasted worrying about the differences between the data provided by the competing panel-based services.

“Mass numbers are a holdover from the old century, an anachronism in an age of precision marketing, targeted advertising, and sophisticated audience management technology,” Yelvington wrote in the E-Media Tidbits Weblog. “Audience qualities and behaviors are the meat and potatoes of the new analytics."

Digital Links:

comScore
Knight Ridder Digital
Morris Digital Works
NetRatings Inc.

By Rich Gordon, Chair - Newspapers & New Media, Medill School of Journalism, Northwestern University

First Published:
June 8, 2007