Market research without data quality control isn’t necessarily producing reliable consumer insights. And when it comes to survey research, gaps in quality control can severely muddy the waters. So how can you ensure your respondents are providing accurate and thoughtful responses? One widely-accepted technique is to use targeted data quality control questions throughout your survey.
Today, we’re excited to explore why and how to implement quality control questions in online surveys. We’ll go over their basic functions, their best practices, and some of the benefits surrounding the different kinds you can deploy in your survey. By the end, you’ll understand how these questions can significantly improve the accuracy of your survey results. Ready to learn how a few data quality control questions can provide pathways to better survey data? Let’s hop right in.
What are data quality control questions?
Data quality control questions go by several names—attention checks, quality checks (QCs), or simply quality control questions. Whatever you call them, their purpose is more or less implied in the name—they’re going to help you check attention and ensure quality responses. The idea is to insert them strategically throughout a survey so you can confirm your respondents are reading the questions and answer options carefully.
Quality control questions should be simple and straightforward for an attentive respondent to answer correctly. They are not designed to trick participants, but rather to filter out those just clicking or speeding through without regard to the survey content. These are the kind of low-quality responses that can distort results. By including one or two very obvious “test” questions, you’ll be able to validate the accuracy of responses.
So what happens if a respondent fails to answer correctly? If done properly, it should catch the respondents who are intentionally not reading the questions thoroughly. This calls all of their other answers into question, and could mean you scrap their responses from your dataset. But before we get into that, let’s look at some best ways to ensure your data quality control questions provide a pathway to cleaner, higher quality data.
Do’s and don'ts for quality control questions
Crafting effective quality control questions is a balancing act. You want them to blend into the survey without calling too much attention to themselves, but you also don’t want to fool your respondents—that’s not helping anyone. Here are some best practices to follow:
DO:
- Keep your questions simple and make sure they have but one (very obvious) correct answer. For this reason, single choice questions work the best.
- Let them know that this is for quality control or data validation purposes only. No need for them to overthink it, tell them in the question text.
- Use culturally universal options and make sure translations are clear when fielding an international study.
- Spread quality control questions out, using one or two per survey. Don't overdo it.
DON'T:
- Add quality control questions to the start of a survey or screening section. This is when attention is highest, so it’ll have a reduced impact.
- Use too many quality checks in a short survey—it’s a waste of everyone’s time to have five attention checks in a 15-question survey.
- Tie the question to survey subject matter, because it may confuse your respondents and impact results.
- Include math questions, logic puzzles, or fake brands. These can all seem like tricks, which may frustrate your respondents. Just keep it simple.
Examples of quality control questions
Next, let’s take a look at some different examples of quality control questions. Feel free to use any of these for your own survey!
Question #1
This simple, single choice question validates respondents are paying attention by having them select a specific option. Here, using a food item should avoid any relation to the survey topic—so if you’re surveying about fruit, you may want to consider changing to something non-related.
For data quality control purposes, please select 'Strawberry' from the list below.
1. Apple
2. Banana
3. Peach
4. Orange
5. Strawberry
Question #2
Here we explicitly thank them for thoughtful responses and state clearly that this is a quality check. It’s also worth noting that getting your respondents to choose a letter makes it clear there is only one right answer.
We very much appreciate your thoughtful responses to this survey so far. The following question has been added for data quality control purposes only. Please select “B” below:
1. A
2. B
3. C
4. D
Question #3
It’s hard not to giggle looking for the right answer on these, but by randomizing options and asking respondents to identify the color, this question confirms attention in a very straightforward way.
Please select the only answer option that is a color. This question is for data quality purposes only.
1. Picture
2. Shirt
3. Purple
4. Boat
5. Dancer
Question #4
This one is all about having one clearly incorrect choice. Again, this validates if the respondent is actually reading carefully before selecting an answer. It’s also ok to laugh at the concept of a basketball flavored ice cream.
The following question is for data quality control purposes only. Please select the item that is not an ice cream flavor below:
1. Strawberry
2. Cookie Dough
3. Mint Chocolate Chips
4. Basketball
5. French Vanilla
Key takeaways
Again, feel free to use those questions as templates in your own attention checks. And now that we’re wrapping things up, let's quickly recap the key points we covered today:
- High-quality data is essential for producing meaningful insights from market research. Low-quality data only works against you here.
- Data quality control questions help validate response accuracy by filtering out inattentive respondents who aren’t giving you thoughtful responses.
- These attention checks should be implemented with care, making sure to keep the survey experience positive.
- Better data quality leads to better insights that inform smarter business decisions. Just a few well-placed quality checks can significantly improve data accuracy.
A note on respondents: Just because a respondent fails a single attention check doesn’t necessarily mean that they’re a bad respondent. That’s why it's important to use quality control questions in combination with other quality measures to identify speeding, straightlining, and more. That way you ensure you’re catching true bad actors rather than respondents who simply made a single mistake.
Looking ahead
As we enter into the age of artificial intelligence, the advent of large language models like ChatGPT will become a bigger and bigger concern for quality control. The reality is that generative AI and semantic search may be able to pass some of the questions we’ve outlined above. So if you’re concerned about the use of AI by bad actors, you’re not alone.
One potential way to meet the challenge of AI when it comes to data quality control is to incorporate two questions or statements on a grid that are in direct opposition to each other. For example, one question can ask if a respondent completely agrees with a statement that they would buy a product, and then a contradictory question agreeing that they would NOT buy a product. These objectively opposed questions may be understood by AI but not answered consistently.
So to wrap up, like most things in insights technology, data quality control questions are in an interesting state of evolution, yet still serve as effective techniques for improving response quality in surveys. As we look towards the future of market research, we hope this article has helped you understand why and how data quality control questions can be applied appropriately.
.png){kind=tracking}
.jpeg)
