As you work through a research project, there arises this slight temptation to forego testing your online survey before launching it—is it really worth the effort? Yes. It is. In fact, many of us have experienced the horror of rushing to launch a survey only to realize mistakes that ultimately make our results unreliable.
In this article, we’ll help you plan your survey’s voyage, check for leaks in your logic, and go over all the best ways to test your study before you launch. After this, you’ll be able to spot any potential dangers lurking in the depths waiting to sink your survey, and confidently collect a treasure trove of powerful insights. All aboard—exciting new discoveries await!
Plan properly
Know the purpose of your voyage
Before sending your survey out into the world, you first must make some decisions: Where do you want to sail and what treasure do you seek? You need to chart your course carefully in order to arrive where x marks the spot—so be certain of your target audience, the questions you’re looking to ask, and how you intend to use the data you’ve obtained. Only by achieving this certainty can you move on to considering the resources and time you’ll need to make available for your expedition.
Crosscheck all maps and schedules
Chart a comprehensive course by assessing the resources and budget you have available. Be sure to build in contingencies for potential technical difficulties or unforeseen challenges. Here, it’s helpful to establish a timeline that allocates an achievable schedule—especially if you plan on conducting an exploratory audit of any data issues before the big launch.
But for now it’s time to carefully anticipate any obstacles that may hinder participation, like survey fatigue or a lack of respondent motivation. This will help you plot a route that avoids potential hazards such as ambiguous wording, complex logic, or misleading assumptions. Build a schedule that will help you arrive at your destination and yield meaningful insights—including a buffer (we suggest at least a few extra days) to detour around any unforeseen storms. Arranging for ample time to test your survey will protect you from potential data errors and missed deadlines.
Supply your ship
A strategic plan can weather any storm. But you also need to be properly equipped to realize your vision. Determine the right combination of questions and visual elements to engage your audience. Do you have the proper stimuli? Are you in need of translations to reach all the respondents in your target audience? Once you’re fully stocked for the voyage ahead, you can move onto the next part of your preparations.
Check survey design and logic
Know what’s holding your ship together
Begin your survey with screening questions designed to ensure that you’re gathering data from the correct audience. These questions shouldn’t be yes/no questions; but rather, offer a range of options including some options that you can select as termination points so that respondents who select them will not advance. Also, keep your language and syntax simple and to the point. It is better to have three different screening questions than to combine three different requirements into a single question.
Then, you want to move on to the the body of your survey. This includes all the questions designed to gather all the insights that you need from the survey respondents. The design options that you have are really infinite, but keep in mind that a real human being is going to be reading them, considering them, and providing answers. Design your questions so that they are simple to understand and difficult to misinterpret. Avoid long and repetitive grids. Generally, design a survey that you would be interested in and wouldn’t mind investing your time in completing.
Survey logic provides the final framework for how respondents will travel through the questions. So it’s critical that you carefully inspect the routing architecture to confirm it’s watertight and not in danger of springing a leak. Types of logic like branching, looping, and skip patterns can help guide respondents down specific paths and customize their survey-taking experience.
Make sure you’re seaworthy
When testing the survey, we’d recommend following every route respondents may take so you can accurately diagnose any gaps, traps, or dead ends that could steer your target audience off course. Think about how you plan to analyze the data and ensure that you have included all the data points that you will need to complete your analysis. You can even analyze test data to detect any weaknesses or inconsistencies in the logic that could sink your whole survey (more on this later). Just remember: The logic framework must be robust enough to channel all respondents, however diverse, to the end of the survey or to an intended termination point.
Consider user interface
Check the rigging
Now let’s direct our attention to the more elemental aspects of your survey. This includes confirming how it looks to your audience. You need to ensure the interface fills your sails on any device or browser, and keeps your respondents smoothly onboard no matter where the winds may blow. For example, do your scales render up and down or left to right? Do your images scale to the appropriate size regardless of whether you are on a 24” desktop or the latest iphone. Check that all interface elements display as intended on mobile and desktop, in every browser, to avoid inconsistencies leaving respondents adrift.
With sails trimmed for all platforms, your interface can harness even the gustiest of respondent groups without overturning. So adjust these elements as needed until you’re set to catch every breeze.
Swab the decks
Maintaining a seaworthy ship requires vigilance and care for details of the deck itself—same goes for your survey. Check the survey for any “splinters” in the interface that could puncture the user experience or confuse your respondents. This includes issues with fonts, image sizing, or spacing that could disrupt an otherwise pleasant voyage. Ensure that any audio, video, or image elements are rendering quickly and in the desired size and composition. With a sound deck, your respondents stay happily onboard. So inspect each board and bolt, making repairs until the deck proves watertight—then you'll sail steady through the toughest of storms!
Test your survey
All Hands on deck
Test your survey with colleagues, friends, family and other casual acquaintances. Ask for input on question wording, flow, logic and overall coherence. One recommended approach is to consider that your grandmother was taking the survey. Would she understand your questions? Would your intentions be clear? Would she be terminated by your screening logic or would she pass through to the end? Look for issues that could frustrate respondents or pose problems when launched. An hour of thoughtful review and editing is worth 10 hours of frenzied corrections later on. Revise as needed based on feedback. Outside input can be instrumental in identifying blind spots and will ensure your survey resonates with audiences beyond your team. Consult others to check you have not gone off course before setting sail.
Consult the navigator
Discuss the survey objectives, strategic approach, and the analysis it will empower with the consulting stakeholders eager for results. Even better, invite them to preview the survey and provide pre-launch feedback. Explain to them how their feedback helps verify the survey and yield the insights they seek before launch. Ask them to evaluate flow, question coherence, and relevance to goals. Incorporate the feedback that keeps the survey on course and headed towards the objective you planned for. Stakeholders can spot omissions or oversights, but may suggest additions that push the survey off course. Consult them as navigators with a vested interest in your successful voyage. If you are really an ambitious explorer, you can even start to build things like banners, cross-tabs, and preliminary presentation slides with the test data. This can be a great way to show your stakeholders what they are going to see (and shine a lot on what might be missing) with the final data set.
Bring in outside experts
For an objective “pre-flight check” of your survey's seaworthiness, consider consulting with research experts. They can evaluate screening question design, logic, question flow and the overall respondent experience with a perspective free of assumptions or familiarity. Experts can spot potential issues that often evade both researchers’ and stakeholders’ perview.
Here, tools like aytm’s Survey Editor allow you to share a link for the experts to preview and assess your survey. Our experts can also provide this kind of feedback—identifying any areas that could cause problems and suggesting the necessary revisions before launch. After a pre-flight check, you can move forward with confidence having the all clear from the experts, you sail confident of blue skies and minimal turbulence ahead.
Prepare to launch
Testing the waters
It might be a good idea to launch your survey to 10% of your sample in order to check for leaks before embarking fully. We touched on this earlier, but a soft launch will give you a chance to analyze initial responses to see that questions sail as intended, logic is sound, and there are no drop-offs, confused sailors or unrealistic outliers that could maroon data quality. With a soft launch you can mitigate risk and make repairs as needed before launching to the larger sample. Reword unclear questions, adjust logic to avoid trapping respondents in loops. This is an especially important step if you are on a tight budget (don’t want to have to restart at additional cost) or if you have a limited number of survey respondents. You don’t want your survey quotas to fall short because you have exhausted your potential respondent pool with a failed first launch. Your test sail highlights holes to patch before difficulty arises. And it reassures your voyage will yield insights the insights you seek!
Look towards the horizon
When the time comes to launch, understand that you’ll need to watch the horizon carefully—even after launch. Check early responses for signs that everything is headed in the right direction. The aytm platform allows you an excellent vantage point to watch responses in close to real time. You can see the data update and parts of your analysis populate with live data. Think about if the data is coming in as you expected. If not, are there problems causing this or is this just unexpected learnings? Look for technical issues preventing some from boarding, questions that seem to be frustrating sailors into abandoning ship, or icebergs that could puncture data integrity. Adjust as needed to avoid foundering. No sea is safe without scouting, no matter how calm the day may seem. Check the skies, note dangers that mark your way.
It’s time to set sail
Thank you for following this map to insight and discovery. With planning, preparation and vigilance for any troubles that could run your survey aground, the seas of data collection will yield bright horizons of understanding rather than more storms to weather. If additional charts or compass checks would help guide your course, don't hesitate to reach out to our crew of research navigators.
To further equip you for the voyage, we built you a jumpstart guide. It provides an overview of the research process from objectives to insights, with strategies for smooth sailing and tools to help lead you to new discoveries. Use it as a map to navigate the waters of your quest for insight.
