In those cases where online survey projects go wrong, it is commonly because too many people got involved in the survey design process. Perhaps it started with a small team making rational, focused decisions. But then word gets out that you’re planning to do a survey project, and it seems everybody wants to add “just” two or three questions to the questionnaire: “As long as you’re talking to our customers, can you ask them about their retail preferences?” or, “Can you find out if they’ve seen the new ad we ran?”
Before you know it, a very precise, focused questionnaire that can be done with excellence has now become a questionnaire with an illogical flow that doesn’t make any sense to the people who are taking the survey. Now, instead of being a nice 10-minute questionnaire, it’s a 30-minute monster.
These survey hitchhikers don’t mean to make your life difficult, but they don’t understand that what they’re asking for may be unreasonable.
Grab Your Badge: You Are The Survey Cop
So it is your job to be the survey cop. Someone has to define, protect, and uphold the agreed upon project priorities. Does that mean you should just say no to these requests? Of course not. The easiest strategy is to offer the hitchhikers some options:
• Help them find their information elsewhere (such as in secondary research reports). In their excitement to add on to a new survey project, perhaps they have forgotten to look for existing information. A little guidance about where the answers may possibly exist could solve the conflict.
• Help them scope a second, separate study. If truly no readily available answers exist, it may simply be time to scope a new online survey to address their specific needs.
Why Is Hitchhiking Illegal?
If you decide to add questions to your instrument that are not specifically aligned with your primary goals, it can cause two major issues. The obvious one? Survey duration. For surveys, we all know that shorter durations mean respondents pay more attention to the wording of your questions and answer options. And that helps maximize your data quality. But there is another issue, which may not be quite so obvious: fit.
Are the people you’re going to recruit for the primary goals also the right “fit” to answer these new questions? If not, you might need to revamp the original sampling plan, which can drive up costs and introduce delays.
Most studies require very specific samples (the people who will be qualified to take your survey). For example, let’s say the primary goal of this project is to measure awareness of your brand. You are likely interested in measuring awareness only among people who actually buy your product category (you might not want to measure awareness of your frozen pizza brand among people who never buy frozen pizza). You’re going after a very specific audience.
If you want to add questions outside of the original scope, you have to decide if the same population can answer the additional questions. Perhaps, instead, they would be best answered by a differently qualified group. Maybe now you would have to change your recruiting parameters to include both people who already buy your product category and those who do not. In the pizza example, perhaps your company is now thinking about extending its brand into frozen hamburgers; so now you want to find out if your brand has permission to enter that category. It may be valid to test your brand’s permissions among people who do and do not currently buy frozen pizza.
The bottom line: Unnecessarily long or complicated questionnaires can drive up costs and drive down data quality—and do so very quickly.
Ignorance of the Law Is No Excuse
Ignorance of the law is no excuse, but it doesn’t hurt to raise awareness of the law. Or in this case, best practices.
To do this, it is best to set expectations about questionnaire length early. Let your internal colleagues know you have a firm maximum duration and a firm scope. If they won’t listen to the logic, make them accountable with money: If you plan a project that was specified as 15 minutes with 2 screening criteria, and it becomes 30 minutes with 10 screeners, you will incur fee increases. Are they willing to pay that fine? After all, hitchhiking is illegal.