Forbes.com

Survey Sabotage: Is Your Customer Data Misleading You?

Does your data match what you see at your business? If not, bias may to be blame.

By Fred Reichheld and Rob Markey

min read


This article originally appeared on Forbes.com. In this age of Yelp, Facebook and Twitter, consumer opinions have never been so powerful. While it’s easier than ever to find reviews and ratings, companies still grapple with how to best gauge their relationships with their clients. Many top companies are using or have adopted a Net Promoter System® to learn more about the quality of their relationships. A closed-loop feedback process can help businesses identify key clients, product features customers like or dislike, service opportunities and problems that need immediate attention. But these efforts can only go so far if the survey data is flawed by biased answers. Like a virus that yields no obvious symptoms, bias can creep into survey data all sorts of ways. Here are some of the most common factors that can contaminate your data, and how to avoid them: Risk: Fear and embarrassment. Consumers may avoid giving negative scores out of embarrassment, particularly when surveyed face to face or on the phone. In business-to-business (B2B) situations, smaller customers may avoid negative reviews when surveyed by a large or critically important vendor, for fear of being cut off. Prevention: To coax out candid feedback, leading NPS® practitioners find ways to offer each customer appropriate levels of confidentiality. In B2B settings, for instance, companies can maintain transparency by reporting average scores for an account while keeping individual ratings confidential. Risk: Employees who try to game the system. Sales and service representatives at auto dealerships are notorious for begging customers for top survey scores. Employees of other companies may engage in shenanigans, such as altering the phone numbers of unsatisfied customers so surveyors can’t reach them. Prevention: Companies with robust Net Promoter Systems pay close attention to potential gaming of this sort. At the car rental firm Enterprise, which uses a customer loyalty system called ESQi, gaming the system is called “speeding” and is grounds for dismissal. Risk: Responder bias. This occurs when the individuals who respond to a survey offer a different view than the overall population being surveyed. Imagine that a company administers a survey with a 20% response rate. It indicates that it’s Net Promoter Score® is 50% (60% promoters minus 10% detractors). Now imagine that the firm studies the behavior of nonresponders by looking at whether these customers made repeat purchases or spent more over time. The additional research determines that the mix of that group is 10% promoter, 40% passive and 50% detractor. That means 80% of customers who ignored the survey had an NPS of negative 40%. So the true NPS for the company—the weighted average of the two groups—is negative 22%, not 50%. The company’s relationship with its customers isn’t as rosy as it seemed. Prevention: If you can’t study nonresponders, consider scoring all of them as detractors (probably not too far off in business-to-business settings) or as a 50-50 mix of passives and detractors (a reasonable estimate for many consumer businesses). Naturally, a high response rate reduces the risk of responder bias. There is only one sure way to check whether your system has effectively defused the land mines of low response rates and bias: You must regularly validate the link between individual customers’ scores and those customers’ behavior over time. Does your research match what you see at your business? If not, bias may to be blame. Fred Reichheld is a Fellow and Rob Markey a partner at Bain & Company. They are authors of the bestseller The Ultimate Question 2.0: How Net Promoter Companies Thrive in a Customer-Driven World. Net Promoter^®, Net Promoter System^®, Net Promoter Score^® and NPS^® are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.

This article originally appeared on Forbes.com.

In this age of Yelp, Facebook and Twitter, consumer opinions have never been so powerful. While it’s easier than ever to find reviews and ratings, companies still grapple with how to best gauge their relationships with their clients.

Many top companies are using or have adopted a Net Promoter System® to learn more about the quality of their relationships. A closed-loop feedback process can help businesses identify key clients, product features customers like or dislike, service opportunities and problems that need immediate attention. But these efforts can only go so far if the survey data is flawed by biased answers.

Like a virus that yields no obvious symptoms, bias can creep into survey data all sorts of ways. Here are some of the most common factors that can contaminate your data, and how to avoid them:

Risk: Fear and embarrassment. Consumers may avoid giving negative scores out of embarrassment, particularly when surveyed face to face or on the phone. In business-to-business (B2B) situations, smaller customers may avoid negative reviews when surveyed by a large or critically important vendor, for fear of being cut off.

Prevention: To coax out candid feedback, leading NPS® practitioners find ways to offer each customer appropriate levels of confidentiality. In B2B settings, for instance, companies can maintain transparency by reporting average scores for an account while keeping individual ratings confidential.

Risk: Employees who try to game the system. Sales and service representatives at auto dealerships are notorious for begging customers for top survey scores. Employees of other companies may engage in shenanigans, such as altering the phone numbers of unsatisfied customers so surveyors can’t reach them.

Prevention: Companies with robust Net Promoter Systems pay close attention to potential gaming of this sort. At the car rental firm Enterprise, which uses a customer loyalty system called ESQi, gaming the system is called “speeding” and is grounds for dismissal.

Risk: Responder bias. This occurs when the individuals who respond to a survey offer a different view than the overall population being surveyed. Imagine that a company administers a survey with a 20% response rate. It indicates that it’s Net Promoter Score® is 50% (60% promoters minus 10% detractors). Now imagine that the firm studies the behavior of nonresponders by looking at whether these customers made repeat purchases or spent more over time. The additional research determines that the mix of that group is 10% promoter, 40% passive and 50% detractor. That means 80% of customers who ignored the survey had an NPS of negative 40%. So the true NPS for the company—the weighted average of the two groups—is negative 22%, not 50%. The company’s relationship with its customers isn’t as rosy as it seemed.

Prevention: If you can’t study nonresponders, consider scoring all of them as detractors (probably not too far off in business-to-business settings) or as a 50-50 mix of passives and detractors (a reasonable estimate for many consumer businesses). Naturally, a high response rate reduces the risk of responder bias.

There is only one sure way to check whether your system has effectively defused the land mines of low response rates and bias: You must regularly validate the link between individual customers’ scores and those customers’ behavior over time.

Does your research match what you see at your business? If not, bias may to be blame.

Fred Reichheld is a Fellow and Rob Markey a partner at Bain & Company. They are authors of the bestseller The Ultimate Question 2.0: How Net Promoter Companies Thrive in a Customer-Driven World.

Net Promoter^®, Net Promoter System^®, Net Promoter Score^® and NPS^® are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.