This is a Test of your Online Survey Critiquing Skills. This is Only a Test.

Consider this the equivalent of the written test you must undergo to get your driver’s license. Treat this series of questions as a mock test in acquiring your survey design license!

Below are 3 example questions. Can you spot the errors?


Designing a survey?

Create a new market research study, and get the valuable answers you need, with beautiful survey design built in automatically. As a market research company, we at AYTM are here to help you quickly and easily test ideas for your new company, product, service, or homework assignment. Learn more


Example 1:

“Do you currently own a digital camera?”
Yes
No

What’s wrong?

There is a glaring issue. We need to define what we mean by digital camera. Do we want them to include phones with cameras? Or video cameras that also do stills? We need to be crystal clear in what we are asking.

As well, if this question is for a study on purchase plans or brand preferences, hopefully it was given extra context by also including other questions about whether the respondent received the camera as a gift (in which case they are less likely to have thought carefully about features, brand and price) versus buying for themselves, and whether they have more than one “digital camera.”

Example 2:

“How often do you buy a new computer? Pleases select one answer below.”
A. Only when needed
B. Every year
C. Every 2 years
D. Every 3 to 5 years
E. Whenever cool new features come out

What’s wrong?

First, we are mixing time and drivers. As a respondent, I might want to answer A and D, but if I have to pick one, I feel I am being forced to pick one of possibly two correct answers. The question should be split into two; one on time frames, and one on purchase “triggers.”

And second, there is not a longer time frame option. We need to offer a top range, which in this case would be “More than 5 years.” Is this unlikely to be common? Yes, but it still needs to be included for the outliers.

Example 3:

I have seen many variations of this question over the years:
“How likely are you to purchase any products from the following brands?”
A. Gatorade
B. Vitamin Water
C. Rockstar
D. Monster
Followed by a scale from “Very Unlikely” to “Very Likely” for each.

What’s wrong?

Ummm…when? As in, ever? Without a specified time frame, this question is so hypothetical that the results will be useless. “The brands I am likely to buy from within the next 3 weeks?” -Maybe I will select one. If I think you mean “ever,” I may select all.

We are giving the respondents too much room for interpretation. Some may assume we mean to ask in the next few weeks, and others will be more literal and think long-term. Also, there is the option for “None”.

My other concern with this question is that some respondents may select a neutral or “somewhat likely” answer options for all 4 brands. Then, what have we learned? This is a case where the easy default is a 5-point Likert scale, but perhaps an alternate approach would give us a clearer sense of brand preference.

Conclusion:

It is always easier to spot errors in other peoples’ surveys than in your own. So always take the time to have someone with a fresh set of eyes review your surveys before you start data collection. Even experienced drivers sometimes need a co-pilot to help read the directions and look out for potholes.


Still confused about how to design your survey?

Easily create a new market research study right now on our Target Market page to test your concept(s), where your survey design will already be optimized for online and mobile surveytaking. If you have any questions about how to construct your study, email [email protected] or call us at +1 (415) 364-8601 (8am-6pm Pacific Time).

ABOUT THE AUTHOR: Kathryn Korostoff
Kathryn Korostoff taught market research best practices at Ask Your Target Market, and is the president of Research Rockstar, delivering market research training and support services. She can be reached at KKorostoff AT ResearchRockstar DOT com.