Helping the Cream to Rise: Picking Variables to Focus On When Analyzing Online Survey Results

Whenever you’re doing a survey project, you’re going to have objectives. Otherwise why would you do the survey? You might be testing one or more hypotheses, looking for surprise discoveries, or hoping to confirm what you think you already know. But the process is reminiscent of the way cream is separated from the milk — it rises to the top. Given a little effort and time, the process of data analysis works the same way, and you as the researcher can help the useful information rise — if you go about it correctly.

Let’s say you’ve just completed data collection with your amazing online survey. There are an almost endless number of ways that you can analyze that fresh data, an intoxicating list of possibilities, but this nearly unlimited supply of options doesn’t come with an infinite amount of time. Your goal now must be to avoid endless analysis or the so-called “analysis paralysis.” For your analysis to be meaningful and effective, you need a data analysis plan. You need to ask, “What variables should my analysis focus on?” so that you can help bring that informational “cream” to the surface.

With AYTM you’ve got some great, easy analysis options. When you log into your AYTM dashboard and look at your project’s final results, you see the left-hand side provides buttons to easily filter data by gender, income, etc. It’s a powerful tool that you will enjoy using. Still, in the interest of time, you will need to manage this “skimming” process. To help you, here are three important tips for deciding what variables you’re going to spend your limited time analyzing:

1. Testing your Theories — Do you have any specific hypotheses about your target market? Test them first. Let’s imagine that this online survey you’ve done was on feature requirements of a new iPhone application. And let’s say you started this project with a hypothesis that interest in specific features will vary by gender, income level and geography. Great. Start there. With your hypotheses as a guide, you can avoid the trap of attempting to analyze everything “just in case”. You can focus.

2. Keeping it actionable — What type of analysis will you actually use? This is really important. If your analysis suggested that interest in a specific feature varied by income level and education level, would you be likely to act on both? Probably not, so think of the part you’re not likely to use as a #2 priority (more on that in a moment). Don’t waste your time if it’s not actionable.

Similarly, let’s suppose you’ve found that a feature you expected to do very well was unexpectedly very unpopular. You’re really shocked, but once you start looking at it, you notice that there’s more interest from one sub group than another. That can be really helpful, but don’t fall so in love with your features that you beat the data to death. You can slice and dice and chop and stir until you find a sub group where interest in that feature is stronger, but you’ve narrowed the focus so far that the information isn’t actionable any more. Chase your pet hypothesis, yes, but recognize that there are diminishing returns, and stop before you get there.

3. Prioritizing and eliminating — Start with a documented plan. This can be as simple as a list where you write down the variables you’re interested in analyzing, but do it BEFORE you start analyzing any data. Now prioritize each element of your list for relevance to your goals.

• Priority 1 should be key to the objectives that caused you to do the survey in the first place, or relate to a primary hypothesis that you’re looking to test.

• Priority 2 could be some good supporting contextual information or analysis—analysis you would definitely use to inform key decisions.

• Priority 3 is “nice to know” but you’re never going to use it. Eliminate it.

Over-stirring milk will keep the cream from rising, and over-analyzing your data can hide the really useful, actionable information behind a flood of minutia. Prioritize, eliminate the extraneous, test your hypotheses, and focus on your goals. That’ll let the good stuff rise to the top.

ABOUT THE AUTHOR: Kathryn Korostoff
Kathryn Korostoff taught market research best practices at Ask Your Target Market, and is the president of Research Rockstar, delivering market research training and support services. She can be reached at KKorostoff AT ResearchRockstar DOT com.