Survey Bias Can Destroy the Value of Your Data

When you go to a dinner party, you don’t complain about the décor, the food and the other guests. You sit and enjoy the food, compliment the home, and enjoy the pleasant dinner conversation. Well, pleasant conversation is fine for dinner parties, but doesn’t make for very helpful survey results. Politeness is just one of the factors that can create survey bias.

survey bias

To avoid survey bias, you’ll need to be prepared to hear “bad news” about your company. If you want results that will help you improve, you’ll need your respondents to be honest, concise, and forthcoming.

Want Honesty? Be Honest

The best way to get honest answers is to be honest yourself. Be completely upfront with your respondents. Tell them that you are looking for genuine information to improve your company, and that you value their honest opinions and candid feedback. They need to know that you’re not afraid of negative comments. It can also help to remind them that their answers are anonymous, so they know there will be no repercussions for negative feedback.

Cut Down on Agree Scales

Questions with answer options such as: “strongly agree, agree, disagree, strongly disagree,” are great in many situations. However, using them too often can potentially lead to skewed data. Respondents feel the need to either be polite or answer out of convenience. Either way, when you make your question into a statement and then ask people if they agree, they are more likely to agree than they would be if the question were phrased differently. Think about using importance scales or a different question format, if possible. It’s still okay to use agree scales sometimes, but just try not to rely on them too often.

Avoid Leading Questions

A surefire way to get a polite answer from your respondents is to tell them what answers you want to hear. When a wife asks her husband, “does this dress make me look fat?” he knows exactly what he wants to hear and will most likely answer accordingly, no matter if he actually agrees or not. In surveys, questions like: “Does this new layout make more sense than the old one?” make respondents think you’re just looking for a polite response. So even if they don’t think the new layout makes sense, they’ll tell you it does, leaving your answers distorted and useless.

Randomize Answers

When your question includes a list of answers, presented in the same order to each and every respondent, they tend to focus more on the answers toward the top of the list than those at the bottom. Fortunately, the solution to avoiding this type of bias is very simple: randomize the order of your list. Of course, you still should keep certain options at the bottom of the list where necessary, such as “all of the above.”

Minimize Survey Bias

Getting honest data from your survey respondents can seem like a daunting task. But it is possible if you make it easy and comfortable for your respondents to give you candid feedback. Don’t be afraid of getting “bad answers” about your company, because in the long run it will help you improve. Leave the pleasant conversation for dinner parties and embrace honest answers from your survey respondents.

Photo Credit: Test Room from Flickr

ABOUT THE AUTHOR: Lev Mazin, Co-founder & CEO
Lev is the CEO and Co-founder of - Ask Your Target Market, the leading innovator in DIY online market research. Having a graphic design background, Lev is an expert in UI/UX with over 15 years of experience. Prior to AYTM, Lev worked with such clients as Oracle, Tiffany & Co, Maserati, Harry Winston, Whole Foods Market, and Jamba Juice helping them build their brands and multimedia/web applications.