Why focusing too much on the numbers can be a bad thing

I recently visited the store of a well-known telecommunications provider to make a change to my account. The store assistant who dealt with me was excellent – he walked me through the problem and found me a swift solution.

As I was about to leave the store, he mentioned that I would soon receive a customer experience survey by text.

So far, so normal.

Then he lowered his voice. “Just so you know, only a 9 or 10 is a good score, so it would be good if you could remember that if you respond. I need a few high scores to keep my bonus.”

I was probably going to give a high score anyway, so why does this matter? For starters, what if I had had a negative experience, but decided to provide a high rating because I wanted to be a good human being and help the assistant to get their bonus?

That story is an example of where measurement can go wrong. Rather than allowing the provider to get an accurate measure of customer experience, it has encouraged employees to play the game and manipulate the results.

However, this issue isn’t limited to the measurement of customer experience.

Take timesheets, which many professional services organizations, including research agencies, often use.

In theory, timesheets allow managers to monitor their most precious resource – employees’ time – to understand which projects and clients are profitable. It can also help them to identify individuals who have availability when a new project is starting.

The reality can be different. Many organizations with timesheets use the data to inform decisions about employees. For example, those who have worked enough billable hours become eligible for a bonus, while those who haven’t become candidates for redundancy.

Therefore, employees are ‘incentivized’ to manipulate their timesheets to make it look like they’re working more than they are. Even in quiet periods, everyone seems to be busy, and there is limited capacity in the business.

That isn’t to say that measurement is terrible. Far from it. It is a critical tool in the management of any business.

But measurement needs to be done with care. In the case of measuring customer experience, here’s what we suggest:

• Keep it confidential – it encourages honesty and reduces situations the likelihood of a store assistant following up with a customer to ask why they gave a specific score
• Ensure you don’t just rely on ‘transactional’ measures – while it’s good to get a measure of satisfaction with each transaction, these measures are the easiest to manipulate. More strategic scores that look at performance across a specific time period can be captured without involving store assistants or Account Managers. Therefore, the totality of the service experience drives the ratings, and there is less incentive to manipulate them. As a result, strategic measures are an excellent complement to transactional ones – they can identify trends that may be masked by transactional studies due to manipulation
• Prevent manipulation – include a question in the survey to ensure participants haven’t been ‘primed’ and occasionally ‘mystery shop’ surveys to explore if manipulation is standard
• Re-define reward – financial incentives are powerful motivators, but if the distribution of the incentive is solely linked to your experience score, you will drive the manipulation of scores. It’s better if experience scores aren’t the sole component