How to improve the quality and reliability of B2B research panel participants

Can you use panels in B2B research?

When people talk about research panels, they generally mean one of two things:

• Buying sample from a third-party panel
• Managing your own panel

You can certainly use the latter in B2B market research. Building a panel makes sense when your target audience is niche or senior, or when you conduct a lot of research, because you can:

• Control respondent quality
• Build rich profiles about each panel member
• Deliver insights more quickly and more frequently
• Save costs in the long-term
• Use your panelists for any research methodology (whereas third party panels are typically only used for quantitative surveys)

There are also drawbacks to managing your own panel:

• It usually requires a lot of up-front investment. While this investment tends to pay off in the long-term, it is often challenging to convince budget-holders of the value of spending such a significant amount
• Once a panel is ‘live,’ it requires a significant amount of time to manage. Specifically, it requires time to nurture panelists to ensure they remain engaged

In general, the benefits of managing your own panel outweigh the drawbacks. To discuss the best way to approach this, contact us.

A more difficult question to answer is whether you should use third-party panels in B2B research.

Third-party panels are a controversial topic in B2B.

They are certainly tempting to use. Panels offer instant access to a pool of decision-makers who have been pre-qualified and who have expressed interest in responding to surveys online. As a result, they allow you to conduct interviews quickly and cost-effectively.

Unfortunately, the model used by third-party research panels often makes them unsuitable for B2B research.

1. Incentives are too low for real business decision-makers.

Consumer research panels tend to pay survey respondents a small amount of money for taking part in research (e.g., $1 or £1).

Of course, B2B research requires higher incentives:

• The target audience is much smaller than consumer research. You need to offer higher incentives to ensure that the small number of eligible individuals are persuaded to take part in the research
• Business decision-makers are time-poor, and persuading them to spend precious time on a survey requires a significant amount of money. In general, you have to pay them more than what they would typically receive in exchange for their time. For example, if you’re interviewing a consultant who bills $100 by the hour, the incentive for an hour-long interview should probably be more than $200

While research panels do offer incentives higher than $1/£1 when conducting B2B surveys, the incentives they offer are still significantly lower than you would expect a business decision-maker to accept in return for their time.

Are time-poor, well-paid executives realistically going to spend their time filling out a 15-minute survey, for random companies, for less than $10? A small number might, most won’t.

2. Many panels sell consumer sample as B2B sample.

The economics of research panels are not suited to B2B sample.

It costs more to onboard business decision-makers to a panel than it does consumers. It also costs more to administer a B2B panel. Business decision-makers change roles and companies frequently, and there are costs associated with updating panelists’ information (e.g., contact information, role).

These higher costs are often hard to justify, as they can’t necessarily be recouped. For example, there are limits on how frequently decision-makers can be surveyed (and, therefore, how often panel companies can make money from their survey responses).

As a result, many panel companies do not recruit real B2B panelists. They recruit consumer research panelists. Many of these consumers have jobs, and can answer questions about those jobs.

However, the sorts of consumers who sign up for research panels are unlikely to be senior decision-makers. Therefore, you may be paying the research panel company high rates to interview an individual who will complete a survey for a very low incentive. It’s not always clear whether the higher price you’re paying is being passed onto the decision-maker or kept by the panel company.

3. Panels have a ‘professional respondent’ problem.

Research panels tend to be full of ‘professional respondents’ who complete multiple surveys in a day.

Some of these respondents are fraudsters. They are trying to earn money by pretending to be someone they are not. B2B research surveys, which offer higher incentives, are particularly attractive targets.

Professional respondents are not necessarily all fraudsters. Some are genuine, and academic studies have shown that professional respondents can be of higher quality than non-professional ones.

But even ‘genuine’ professional respondents pose significant challenges.

First, they introduce a bias to the research – the sort of person who takes surveys professionally is probably not representative of the broader target audience.

Second, they may be tempted to exaggerate their seniority, or the extent of their role, so that they can be eligible for a specific survey.

4. Most panel companies are ‘fishing in the same pool.’

There are many third-party panel providers out there, but many of them rely on the same sources of sample.

In other words, you’re not paying to access their proprietary panel. You’re paying for them to ‘mark-up’ someone else’s panel.

Wasting money like this is frustrating, but there is a broader issue here. If you are working with multiple panels to achieve your target number of interviews, it’s highly likely that they are using the same sources for their sample.

That raises an important question. If the first panel couldn’t find enough people to take part in the research, why would a second panel, which probably uses the same sources, be able to? There are a few possible explanations, only one of which is reassuring:

• Despite your best efforts and the panel’s claims, individuals have taken part in the survey twice – once on each panel
• The second panel has looser quality control measures, and has let through people who shouldn’t be let through
• The second panel has some unique advantages in terms of how it can source research respondents. In other words, it either has access to an exclusive source, or it is particularly good at targeting niche audiences with existing panels

5. Some panel companies are bad actors.

There are several good panel companies in the industry. Unfortunately, several bad actors give the industry a bad name.

As a result, some claims made by panel companies aren’t always accurate. For example, researchers who investigated third-party panels discovered panels that allow individuals to complete as many questionnaires as they want, as well as panels that don’t prevent panelists from using bots to automate responses.

Both of these policies run counter to what panel providers claim on their websites

Bringing it together.

Our view is that third-party research panels are only appropriate in specific scenarios – for example, if you wanted to interview individuals who are either in junior or prevalent roles, e.g., bar managers.

Even in those scenarios, we suggest using other recruitment techniques, either instead of, or in addition to, a panel. For ideas on how to recruit decision-makers for B2B research – click here.

And if you have to use a third-party panel, you should be skeptical. We assume ‘guilt’ rather than ‘innocence.’ In other words, we assume everyone is a fraudster unless they prove otherwise.

Ultimately, you cannot assure 100% quality, but you can take some steps to get as close to 100% as possible.

 

How do you maximize the quality and reliability of B2B research panel participants?

Having a skeptical attitude is critical. But you also need to take the correct steps and ask the right questions. We suggest the following five steps to maximize the quality and reliability of research panel participants.

1. Find which panels that you can trust (as well as which you cannot trust).

A good starting point is to review past articles on panel quality, e.g., the Dirty Little Secrets project.

We also recommend speaking to industry peers and colleagues to see who they will or won’t work with.

These two actions will provide you with a good baseline, but they don’t guarantee good quality. That is why it is critical to ask the panel providers about their general processes.

Here are the sorts of questions that you should be asking:

• The size of the active panel:
  how many panelists do they have? How many of them are B2B panelists? How long do people typically last on the panel?
• Where they source/recruit respondents from: do they buy sample from other panels, or partner with low-quality aggregator sites? If they have a proprietary panel, you should check they obtain panelists from sources that have many business professionals – e.g., trade press, business-focused loyalty programs
• How new panelists are vetted: once an individual has been recruited, how do they vet them to ensure that they are genuine? For example: 1) Does the panel look at the person’s LinkedIn to check that it matches what they claimed in the onboarding process? 2) Does the panel look at the individual’s IP address to check it matches the location that they’ve provided? 3) Does the panel use a CAPTCHA-style test to verify that the responses are entered by a human being and not a bot? 4) Did the panel check whether the individual is a member of another panel?
• The profiling information that exists for each business decision-maker: this is usually a good test of whether a panel is just selling consumer sample as B2B sample. Some panels only have demographic profiling (e.g., age, gender), with nothing related to the individual’s job title or employer. That is usually evidence that this is not a B2B panel
• How frequently decision-makers are asked to take part in research: panel providers have to strike a balance between contacting a panelist too little and too much. Over-contacting is a particular issue as it can lead to ‘respondent fatigue.’ You should ask panel providers if they restrict how often individuals can be invited to take part in research. You should also explore whether they prevent panel members from participating in surveys on the same subject in a specific period. Often these limits exceed what a business decision-makers would credibly accept
• How people are incentivized to take part in research: Panels tend to compensate survey respondents with cash or ‘points.’ It is essential to try to understand more about how this incentivization works. For example, what is the panel’s average incentive for B2B panelists? Are individuals rewarded if they don’t qualify for the survey? If they are not, how does the panel ensure their continued engagement and interest in future studies?
• The mechanisms that are in place to identify professional respondents or fraudsters:’ It is crucial to ask providers to outline their policies for identifying and removing ‘bad actors.’ For example: 1) How do they prevent automated responses? 2) How do they prevent someone from completing the survey multiple times? 3) How do they identify less engaged respondents?
• How they respond to ‘bad actors’: Understandably, a panel might be unwilling to remove someone from the panel if they have been flagged for an issue on a survey. After all, a genuine respondent can mistakenly be flagged. However, it is worth asking the panel provider what their policies are for ‘bad actors.’ For example, how many ‘strikes’ before a panelist is permanently removed? And are panelists prevented from completing surveys for a certain period following a ‘strike’?

There is an alternative to the above questions. You can ask providers for their responses to ESOMAR’s ’28 Questions (to help research buyers of online samples)’.

Most panels will have their answers to the 28 questions immediately ready, which is helpful when time is limited. There is a lot of overlap between our list and ESOMAR’s. Still, we prefer our list because it is a bit less theoretical, and forces the panel provider to answer direct questions rather than responding with answers that have been approved by the legal team but are ambiguous.

Of course, there’s no guarantee that a panel’s responses are accurate, so it is worth checking what they tell you by signing up for their panel as a respondent.

Signing up to panels will help you to understand:

• How easy it is to join, including whether the panel’s quality control measures are as claimed (e.g., if you lied about your role, did they spot it?)
• How surveys are positioned to respondents – for example, are they doing anything that might bias the research?
• What are the incentives for a B2B respondent? Are they high enough for the target audience?

2. Each time you ask a trusted provider for a quote, ask for specifics.

It is important to repeat some of the above questions each time you ask a panel provider for a quote. For example, we recommend asking the following each time you ask for a quote:

• Where they are sourcing respondents – see above for details
• How much respondents are being compensated – see above for details

We also tend to ask providers if they plan to pre-screen panelists, or use a ‘portal’ in which they ask a few questions before your questionnaire. Both can bias the sample, and aren’t good practice.

3. When writing the questionnaire, use a variety of techniques to spot ‘bad actors’ and maximize quality.

There are several techniques that we suggest using when designing a questionnaire to ensure that panel respondents are high quality:

• First, build a screening questionnaire that removes anyone who is not relevant to the survey. That just helps to check that you are speaking to the right decision-makers. For more detail on how to write a screening questionnaire, click here
• When designing the screening questions, use a lot of ‘balanced scales.’ Doing so makes it difficult for fraudsters to realize what target audience you are looking for, and to tailor their responses to pass the screener. For example, if you are going ask how many employees a company has, and you are only interested in speaking to businesses with fewer than 250 employees (i.e., SMBs), it is tempting for the final answer option to be ‘250 or more employees’. However, you should include several options (e.g., 250-499, 500-999, etc.) so that it is less clear that you want SMBs
• We recommend including at least 1-2 open-end questions that force interviewees to go into detail on a specific topic. Open-ends easily identify individuals who do not know what they are talking about
• We also suggest including ‘red herrings’ throughout the survey. These are trick questions that help us to identify people who are just going through the study as quickly as possible without thinking. To give two examples: 1) If you have a question that shows a list of brands and asks respondents which brands they are aware of, you can include a fake brand. Anyone saying they are aware of that brand is likely to be a bad actor. 2) If you have a series of statements that you are asking survey participants to agree or disagree with, you can include contradictory statements. If someone agrees with (or disagrees with) both statements, it suggests they have not considered the question before responding

4. Before and during the survey being live, insist on complete transparency.

Before launching the survey, ask the panel provider to confirm how they are introducing the study to potential respondents:

• What language are they using to explain what the survey is about?
• Are the client, or the product category, mentioned?
• Is there anything that reveals what sort of responses to give to pass the screening questions?
• Do respondents see any adverts before taking the survey that might bias their responses?

Once the survey is live, ask providers for information that proves the fieldwork is being properly conducted. For example, ask for:

• The distribution of non-completes, as well as completes, to check that sampling was fair
• How many invitations were sent
• How many people responded to invitations
• How many individuals were removed from the survey because they were ‘bad actors.’ If there are relatively few, this suggests that quality control isn’t good

5. Monitor fieldwork progress and remove any ‘bad actors’.

While panel providers are supposed to be monitoring respondent quality, you shouldn’t wholly step back. We suggest periodically checking the raw data to identify any ‘bad actors’ that the panel provider has not removed. Here is what we look for:

• Speeding: When someone completes a survey too quickly, it indicates they may not have paid attention to the questions
• Flat-lining: If a respondent is shown a question with multiple rating scales (e.g., ‘rate the following brands on a scale of 1-10…’), and selects the same answer for most, or all, of the answer options, it suggests that they did not read the question properly
• Failing red herrings: This is a simple test of whether a survey participant was answering the survey properly. For example, if wanted to ask a question to gauge customers’ awareness of specific B2B SaaS brands, you could include a fake brand. If a respondent says they are aware of this brand, it suggests they’re not paying attention or lying
• Duplicates: Identifying someone who has completed the survey more than once is challenging when doing panel research. Especially now that GDPR limits your ability to collect IP addresses. However, we suggest looking for any indisputable evidence of duplications (e.g., identical open-end responses, suggesting they copy-and-pasted responses)
• Inadequate open-ended responses: Any individual who has typed gibberish (e.g. ‘jfdfd’), or has given an answer that doesn’t make sense in the context of the question, probably didn’t understand or read the question

Many of these issues are not a problem in isolation. For example, it’s feasible that someone is a speed-reader and can quickly complete surveys to a high quality. We tend to look for individuals who have multiple ‘red flags.’

 

The Adience blog was recently featured in Feedspot’s list of top 100 B2B marketing blogs.