Why focusing too much on the numbers can be a bad thing

Why focusing too much on the numbers can be a bad thing

I recently visited the store of a well-known telecommunications provider to make a change to my account. The store assistant who dealt with me was excellent – he walked me through the problem and found me a swift solution.

As I was about to leave the store, he mentioned that I would soon receive a customer experience survey by text.

So far, so normal.

Then he lowered his voice. “Just so you know, only a 9 or 10 is a good score, so it would be good if you could remember that if you respond. I need a few high scores to keep my bonus.”

I was probably going to give a high score anyway, so why does this matter? For starters, what if I had had a negative experience, but decided to provide a high rating because I wanted to be a good human being and help the assistant to get their bonus?

That story is an example of where measurement can go wrong. Rather than allowing the provider to get an accurate measure of customer experience, it has encouraged employees to play the game and manipulate the results.

However, this issue isn’t limited to the measurement of customer experience.

Take timesheets, which many professional services organizations, including research agencies, often use.

In theory, timesheets allow managers to monitor their most precious resource – employees’ time – to understand which projects and clients are profitable. It can also help them to identify individuals who have availability when a new project is starting.

The reality can be different. Many organizations with timesheets use the data to inform decisions about employees. For example, those who have worked enough billable hours become eligible for a bonus, while those who haven’t become candidates for redundancy.

Therefore, employees are ‘incentivized’ to manipulate their timesheets to make it look like they’re working more than they are. Even in quiet periods, everyone seems to be busy, and there is limited capacity in the business.

That isn’t to say that measurement is terrible. Far from it. It is a critical tool in the management of any business.

But measurement needs to be done with care. In the case of measuring customer experience, here’s what we suggest:

  • Keep it confidential – it encourages honesty and reduces situations the likelihood of a store assistant following up with a customer to ask why they gave a specific score
  • Ensure you don’t just rely on ‘transactional’ measures – while it’s good to get a measure of satisfaction with each transaction, these measures are the easiest to manipulate. More strategic scores that look at performance across a specific time period can be captured without involving store assistants or Account Managers. Therefore, the totality of the service experience drives the ratings, and there is less incentive to manipulate them. As a result, strategic measures are an excellent complement to transactional ones – they can identify trends that may be masked by transactional studies due to manipulation
  • Prevent manipulation – include a question in the survey to ensure participants haven’t been ‘primed’ and occasionally ‘mystery shop’ surveys to explore if manipulation is standard
  • Re-define reward – financial incentives are powerful motivators, but if the distribution of the incentive is solely linked to your experience score, you will drive the manipulation of scores. It’s better if experience scores aren’t the sole component

 

Summary

Measurement is a good thing. However, companies often set up systems that incentivize data manipulation. If you’re going to measure, it’s essential to do it well.

In the case of measuring customer experience, we suggest: keeping it confidential; ensuring you don’t just rely on ‘transactional measures’; taking steps to prevent manipulation; re-defining rewards.

Chris Wells
Share:

Got a B2B market research project
you’d like to discuss?

Contact us

More from the blog

The challenges of using marketing focus groups in B2B research

How to

February 28, 2023

The challenges of using marketing focus groups in B2B research

Focus groups are a viable option for consumer research, but challenging in B2B. We explain why and explore the alternatives.

The problem with B2B research panels: understanding the limitations and challenges

Insights

February 2, 2023

The problem with B2B research panels: understanding the limitations and challenges

Panels are useful for consumer research but a much-discussed topic in B2B research. We reveal why and explain how to recruit genuine B2B decision-makers.

The importance of ad testing in B2B markets

How to

January 23, 2023

The importance of ad testing in B2B markets

How to run B2B ad testing and optimize your creative for maximum impact. There are some traps to avoid too - we share our best practices for success.