Sizing Up Online Audience Measurement Services

Greg Swanson, director of Internet sales for Lee Enterprises, just wanted to know two things. How many people were visiting the Web sites of his company’s 44 newspapers? And how often did they visit?

The problem was, the answers Swanson got didn’t make sense to him. In Montana, for instance, the Billings Gazette had a Sunday readership of about 52,000 people, but the Web site had almost 10 times that many visitors in a month. And the average site user was visiting the site just one or two times per month.

“We had unbelievably large reach – which in some months exceeded the entire population of the state of Montana. Huge reach, but incredibly low frequency,” Swanson said. “For advertisers, it was as if we were putting a message on the side of a bus and driving though a crowded downtown area once a month.”

Swanson would ultimately conclude that the problem stemmed from the inaccuracy of the tools his company was using to measure Web site usage. At newspaper Web sites, he is one of many people who have been frustrated by the difficulty in getting answers to seemingly simple questions about their online audiences.

Since the beginning of the Web, publishers have made the case that the Internet is the most measurable medium ever – one that allows accurate measurement of site usage and advertising effectiveness. But while site administrators are awash in data, many have found that the numbers generate as much confusion as understanding.

It’s not because of a lack of measurement tools. A recent report, Marketing Sherpa’s “Buyer’s Guide to Web Analytics,” described more than 50 different tools for measuring Web site traffic – and said there were at least 50 others. And the report didn’t include companies that provide traffic measurement services using panels of Web users (such as comScore Media Metrix and Nielsen//NetRatings) or those that collect behavioral information from users via telephone or online surveys.

The central problem is this: Every measurement tool has limitations. Asked a straightforward question, such as how many visitors a site has in a month, each of the tools will generate a different answer. What’s a site manager to do?

The Audience Development Committee for NAA’s New Media Federation decided that one place to start would be a guide to the different types of measurement tools. In this article, you’ll find a list of the kinds of tools available, an explanation of how they work and an overview of the advantages and limitations of each. The committee hopes to increase the level of understanding of Web analytics – and better equip newspaper Web site leaders to discuss these issues with advertisers and with companies that provide audience measurement tools and services.

This package also includes articles that address two of the most vexing issues involved in measuring Web usage:

• “The Trouble With Cookies” explains why – as Greg Swanson discovered – many of the most popular usage measurement tools overcount the number of visitors and undercount the frequency with which they visit.
• “Pondering Panels” addresses the advantages and limitations of the two primary services – comScore and Nielsen//NetRatings – that use panels of Internet users to measure traffic and demographics for multiple sites.

Types of Tools Available

A newspaper Web site manager interested in measuring audience size and online behavior can choose from several different kinds of measurement systems:

• Server log analysis software
• Page-tagging (or “beacon”) systems
• Panel-based measurement
• Consumer behavior surveys conducted by phone
• Consumer behavior surveys conducted the Web
• Other tools

Server Log Analysis Software

Summary: These were the first analytical tools. They analyze data that Web servers store automatically every time someone downloads a page or graphic file. Because of their low cost, they are the standard user-measurement tools for small Web sites.

Examples: Urchin, WebTrends

Description: Every time a user views a Web page, the site’s Web server delivers an HTML page plus embedded graphics or multimedia files. In a “log file,” the server stores information about each file, including:

• the URL of the file
• the date and time when it was delivered
• the Internet address, operating system and browser software of the user
• the URL of the “referring” page – the page where the user clicked to arrive at the current one

Log analysis software uses the data stored in server logs to produce reports that display counts of page views, visits, visitors, time spent on the site, etc. The simplest software – some of which is free to download – does nothing more than organize and count the information stored in the log file. More sophisticated tools also use cookies to recognize users who visit the site more than once in a given time period.

Problems presented by log analysis tools ultimately stem from the fact that the Web server cannot distinguish one user from another. This can produce distorted counts of users.

The most basic tools (without cookies) attempt to identify visits and unique users by looking at IP address, browser type, operating system and time of visit. But visitors that come through proxy servers run by corporations, institutions or Internet providers such as America Online are not assigned unique IP addresses. So these tools will, for instance, consider two different AOL users running Internet Explorer on Windows XP to be the same person. This means the number of unique visitors will be undercounted and the frequency of visits by a typical user will be overcounted.

More sophisticated log analysis tools, which use cookies to identify unique users, present a different problem. Unless a site requires registration, a single user who visits a site using two different computers will be considered two different people because he or she will be issued two different cookies. Belden Associates has found that a significant number of users of newspaper sites visit from computers at home and at work – and some use multiple computers in one or both locations.

Furthermore, Belden has found that an increasing number of online users routinely delete their cookie files as a security measure. A single user who visits before and after she deletes her cookie file would be recognized by the server log analysis software as two different people.

Taken together, usage of multiple computers and deletion of cookie files cause log analysis tools to overestimate the number of unique individuals visiting a Web site – and to underestimate how frequently each person visits. (See “The Trouble With Cookies” for more detail.)

Advantages:

• No additional work or cost involved to collect data (servers collect it automatically).
• All site users are monitored.
• Software is relatively inexpensive – or even free.

Limitations:

• Proxy servers and ISP cache servers can distort counts of page views, visits and visitors.
• Unless personally identifiable information is captured for all users via a separate registration system, more sophisticated log analysis tools overcount the number of unique visitors and underestimate the frequency of their visits.
• If your site’s traffic is spread across multiple servers (say, a newspaper site, an entertainment guide and a classified partner), it is cumbersome to combine the data from multiple log files. And because it is difficult to track individual users across the multiple domains, unique visitor counts become even more problematic.

Page-Tagging (“Beacon”) Systems

Summary: These systems require a site to add JavaScript code to every Web page (or to the templates used to generate those pages). Vendors of these systems run massive databases that collect data from the JavaScript tags and allow site managers to run queries based on that data. These systems provide deep, rich data. But because these systems are complex and relatively expensive, they appeal mostly to large companies with staff dedicated to analyzing Web metrics.

Examples: Omniture SiteCatalyst, SageMetrics SageAnalyst, WebSideStory HBX Analytics

Description: Tag-based systems are the current state of the art for internal user measurement, and they have significant advantages over server log analysis tools. For one thing, these systems allow a site to customize the information gathered each time a page is viewed. The JavaScript code is commonly attached to an invisible single-pixel image delivered with every page; these images are sometimes called “beacons.” The JavaScript can collect and transmit a variety of information about the page, including the section of the site where it is located, the time the page was delivered, even details about which specific link is clicked or, for an e-commerce site, how much a user has placed in his shopping cart. These tools also make it possible to see the paths that users take through a site – including the pages they start on and the ones they were viewing when they decide to leave.Another significant benefit is that the JavaScript code can be included on any page on any server – even on pages served through partners. This means site usage data can be easily aggregated and analyzed.As a tool for counting a Web site’s audience reach and frequency, though, tag-based systems suffer from the same basic limitation as server log analysis tools: They rely on cookies to identify unique users. If a user visits from more than one computer (for instance, one at home and one at work), he will be counted as two separate users. Newspaper sites are increasingly addressing the cookie problem with mandatory registration, but this is not a perfect solution. Even registration-based sites make a number of high-traffic pages (including the home page) available to all users, so it’s possible for an infrequent, unregistered visitor to be counted more than once.Page-tagging systems are expensive, but the leading vendors take care of some of the complexity and technology challenges by hosting the data on their own servers and providing sophisticated query interfaces. (These kinds of vendors are commonly termed Application Service Providers [ASP]).

Advantages:

• All site users are monitored.
• Page-tagging systems generate deep, rich data that allow analysts to ask detailed questions about site usage, including the paths that users follow through the site.
• Leading vendors maintain the data warehouse for these systems, which means they require no oversight by your IT staff.
• By changing the information transmitted via the JavaScript tag, sites can significantly customize the data that’s collected.
• The JavaScript tags can be attached to content on multiple servers, allowing sites to aggregate user measurements from many sites, including those of partner sites under different ownership.

Limitations:

• Like server log analysis tools, they rely on cookies to identify unique users, so without user registration they will count someone who visits from two computers as two different people.
• Expensive to operate, usually with an ongoing monthly fee.
• Can be time-consuming to implement effectively, requiring staff to customize JavaScript parameters and ensure that the tags are placed properly across pages and templates on multiple servers.
• To get full value out of the investment, site managers generally find they need to dedicate one or more staff members to running queries and analyzing the results.

Panel-Based Measurement

Summary: These are the Web equivalents to services that produce TV ratings. The companies sign up users who agree to install software that monitors and stores information about the Web pages they view. The main value of these systems is to enable comparisons of your site to others.

Examples: comScore, Nielsen//NetRatings Description: Panel-based measurement companies try to assemble a representative cross-section of Internet users who are willing to have their Web usage monitored. From the behavior of the user panel, the ratings companies project Web usage data for the Internet-enabled population as a whole.

As with political polls and market research surveys, the key to panel-based measurement is making sure that the group being monitored is a representative sample of the people – in this case, all Internet users – whose behavior you want to understand. The two leading U.S. services, comScore and Nielsen//NetRatings, use very different methods to recruit their panels, so they have some different advantages and limitations. (See “Pondering Panels” for more details.) Officials with both companies acknowledge a common challenge of measuring at-work usage. The problem: Many corporate IT departments prohibit employees from installing unauthorized software on their computers. The ratings companies say they take steps to address this problem, but if at-work usage is underrepresented, it will be a particular issue for sites such as those run by newspapers, which receive the bulk of their traffic during the business day.As a practical matter, panel-based measurement is only useful for relatively large sites. Small sites will not generate enough traffic among panel members to be useful for analytical purposes.A particular problem for panel-based measurement is the fact that many media companies receive Web traffic through multiple domains. If your company subscribes to one of the services, you can work with them to define a “rollup” – in which the measurement company will aggregate user data for multiple servers in a single site report. But if you don’t subscribe – and pay – the company, your site’s user data can be distributed across multiple URL’s. Even with a rollup implemented, it may be difficult or impossible to include the appropriate traffic to a partner such as an online classified site.

Advantages:

• For traffic to be monitored, a site doesn’t need to do anything.
• Enables comparison of reach and frequency to other sites.
• Because the identity of panel members is known, the cookie problem associated with server log and page tagging systems does not exist.

Limitations:

• Not useful for small sites because traffic needs to reach certain thresholds for usage by panel members to be statistically significant.
• Measurement for local markets is limited to large metropolitan areas.
• Likely to underrepresent at-work usage because corporate IT departments won’t allow monitoring software to be installed (the ratings services say they account for this through weighting and are working hard to improve their at-work monitoring).
• Data may not be aggregated for traffic to multiple domains.

Consumer Surveys Conducted by Phone

Summary: Telephone surveys are commonly used by newspapers (and other companies) to gauge consumer attitudes and behavior. Before the Web, questions focused on readership of the newspaper, content interests and patterns of usage of other forms of media. Now questions are also routinely asked about Web site usage as well.

Examples: In-house research, surveys by companies such as Belden Associates or Minnesota Opinion Research Inc. (MORI), audience measurement services such as The Media Audit from International Demographics Inc. and Scarborough Research.

Description: Telephone surveys, using random-digit dialing techniques, have been the standard methodology for market research for decades. By using random-digit dialing (RDD), researchers have been able to reach even those people who have unlisted phone numbers.It is now common for newspaper telephone surveys to include questions about usage of the paper’s Web site and, in many cases, competing Web sites as well. Phone surveys offer one significant advantage to systems based on monitoring Web usage: In surveys, the questionnaire can ask about attitudes and perceptions as well as usage. In other words, a survey might find out why someone uses (or doesn’t use) a site, or what they like (or don’t like) about it.

Phone surveys can also ask about shopping and buying behavior, which can be extremely useful in ad sales efforts. If a survey indicates that a third of a newspaper’s Web users have shopped for a new car in the past year, that fact might help persuade a car dealer to advertise on the site. But phone surveys also have significant limitations. The most fundamental is that they rely on respondents to remember previous behavior – and report it accurately. If someone is asked whether they have visited a Web site in the previous month, they may not recall for sure – or they may say they did when they really didn’t. Market researchers say that users tend to over-report usage of news Web sites, and to say they are familiar with Web sites maintained by well-known traditional media companies even if they haven’t visited.“Audience surveys, in general, measure awareness and perception,” said Andy Merits, manager of interactive research for Tribune Interactive. “When people answer, they think where I should go and where I might have gone.”

Research companies also report that it is increasingly difficult to reach a valid sample of the population through telephone surveys, especially in hard-to-reach audience categories such as those under 25. The growing trend to use cellular phones as a replacement to wired lines is exacerbating the problem, as is the number of people who object to being called at home by businesses for any reason. By making repeated call-backs to numbers that are not answered, however, researchers say they are still able to reach a representative sample. But it’s getting harder, and more expensive, every day.Another limitation of telephone surveys is that people won’t stay on the line with an interviewer indefinitely. So the number of questions that can be asked is limited. Some companies that conduct phone surveys, such as Scarborough Research, supplement their telephone questionnaire with a paper questionnaire that elicits more detail about media usage, shopping habits and purchase behavior.

Advantages:

• Unlike tools that monitor Web usage, telephone surveys can generate attitudinal and behavioral information.
• Surveys can ask about shopping and buying behavior – information that is particularly valuable to support advertising sales.
• Surveys can be used to compare awareness and recalled usage of competing sites as well as your own (but only those sites that are asked about).

Limitations:

• Data is all self-reported; survey respondents may have faulty memories or be untruthful.
• Validity of data is dependent on how representative the population sample is – and constructing a sample is getting increasingly difficult.
• The number of questions that can be asked is limited by people’s patience.

Consumer Surveys Conducted on the Web

Summary: In part because of the limitations of phone surveys, it is increasingly common to conduct surveys online. Respondents can fill out the survey by clicking on buttons or checkboxes or filling out a form on a Web page.

Examples: In-house research, market research firms. Many companies that historically conducted market research for newspapers via telephone now also provide online survey capabilities.

Description: Typically survey respondents are recruited through a pop-up message that offers a chance at a prize (iPods have been popular recently). Respondents may also be recruited by e-mail or by phone and asked to go online to fill out the survey.

The fundamental problem with online surveys is that they will only reach people who already visit your site or whose contact information you already have. As a result, online surveys are particularly valuable to get feedback about your site from people who use it now. If your goal is to understand how to reach new audiences, online surveys won’t help much.Another potential problem is that users who are willing to take the time to fill out a survey may be different than those who aren’t. In particular, a survey that pops up on a newspaper Web site might generate more response from people who like the paper than those who are hostile or ambivalent. (This problem also exists for telephone surveys, but researchers can account for it more easily in phone survey work through weighting results to match known demographics of the population.)Newspaper sites often use online surveys to inquire about patterns of usage, about use of competing sites and about demographics of site users. This information can be useful in plotting content strategy and in supporting ad sales efforts.

Advantages:

• Data collection is much cheaper than telephone surveys.
• If deployed on a particular site, respondents are known to be users of that site.
• Can generate attitudinal and behavioral information.
• Can ask about shopping and buying behavior – information that is particularly valuable to support advertising sales.
• Can be used to compare awareness and recalled usage of competing sites as well as your own (but only those sites that are asked about).

Limitations:

• Sample will be limited to people who already use your site – or whose contact information you already have for other reasons.
• Data is all self-reported; survey respondents may have faulty memories or be untruthful.
• The number of questions that can be asked is limited by people’s patience.

Other Tools

There are other tools available to a Web site manager that may help them measure and understand online audiences. These are not, strictly speaking, audience measurement systems. But each can generate metrics that are useful for content strategy, marketing campaigns or advertising sales.

• Network traffic monitoring systems rely on reported traffic from Internet service providers. The leading firm in this space is Hitwise, which reports that it tracks the behavior of more than seven million U.S. Internet users (without collecting personally identifiable information) through agreements with ISPs. Using this data, Hitwise reports market share for different sites in 160 site categories. Hitwise does not try to count the number of users, but it can show how many page views or visits your site gets compared to your competitors. Hitwise can also report demographics of site visitors through a separate panel of recruited volunteers.
  
  Another company that uses ISP usage data is Compete Inc., but it focuses its research on specific industry categories – at the time of this writing, automotive, banking and wireless telephony.
  
• Registration systems allow a site to register users and track their behavior while using the site. Like page tagging and server log tools, they rely on cookies to recognize unique visitors. But because the users provide identification when they register – and must log in if there is no cookie on the machine they are using – registration provides more accurate user counts. In combination with page-tagging systems, registration enables detailed understanding of what kinds of people use what parts of your site.
  
• Behavioral targeting systems, such as those provided by Tacoda Systems and Revenue Science Inc., allow you to see the pages viewed by a user and the online activities they engage in. These systems do rely on cookies to recognize repeat visitors on their return, but they do not collect personally identifiable information. Their value is in targeting advertising based on user behavior. Someone who visits a site’s travel section on Monday, for instance, can be shown travel ads on Tuesday when she visits the sports section. These systems can also report a number of “targetable customers” – those who return to the site and whose patterns of usage can be categorized. The number of targetable customers will be lower than total users measured by other systems.

Digital Links:

Billings Gazette
Belden Associates
Compete Inc.
comScore Media Metrix
Hitwise
Lee Enterprises
The Media Audit
Minnesota Opinion Research Inc.
Nielsen//NetRatings
Omniture SiteCatalyst
Revenue Science Inc.
SageMetrics SageAnalyst
Scarborough Research
Tacoda Systems
Urchin
WebTrends

By Rich Gordon, Chair - Newspapers & New Media, Medill School of Journalism, Northwestern University 

First Published:
June 8, 2007