.jpeg)
Designing an online survey takes real effort, careful planning, and thoughtful decisions. A lot goes into clarifying goals, crafting question and answer options, selecting the population parameters and so on. Still, one of the most important questions to answer is, “how many completes do I need?” Or in market research speak, “what is my minimum sample size?” To answer that, let’s look at three versions of that question.
What is the minimum sample size to have “reliable data”?
Clearly, the larger the sample size, the more reliable your data is, but there are points of diminishing returns. The key point is all about extrapolation. In market research, we want to collect data such that it’s representative of a larger target market so that we can extrapolate our findings to that broader market. It’s not practical to survey everyone, like a census, so instead we survey a “sample”.
- How big a sample is big enough? In consumer research, a sample size of 300 to 400 completes is often sufficient for general research purposes, and there are many online sample size calculators available to help. By plugging in a few assumptions, you can get a more precise sample size goal for your particular project. In business-to-business (B-to-B) markets you can usually get away with even smaller sample sizes, as few as 200 completes is often fine.
- Why the difference? Because in many B-to-B studies, your target market is more homogeneous. For example if you survey HR managers about payroll software, they will have differences of opinion but they won’t be that diverse. In contrast, imagine surveying consumers about their preferences in frozen dinner entrées—where you can see huge differences by budget, geography, ethnicity, household type, and so on. Clearly a diverse population demands a larger sample size than a more homogeneous one.
What is the minimum sample size if I want to do sub-group analysis?
This is an important question, because you do need to plan ahead for any subgroup analysis. For example, in a survey of both men and women you may want to look at results in aggregate and by gender. Or you could analyze those same results by three categories of income, (below $50,000, $50,000-$100,000, and above$100,000). In the first case, you have two subgroups (male and female), and in the income example we have three. Note that to do meaningful subgroup analysis you need a minimum number of responses per subgroup — typically 50 to 100 — though for B-to-B studies it’s not uncommon to have 30 to 50 per group.
What is the minimum sample size to satisfy my colleagues or managers?
Many people who are not trained market researchers assume that there are “magic” numbers when it comes to sample size, and “1,000” is one of those numbers. Of course it’s great to have more data to work with and if budget isn’t a concern, by all means get 1,000 completes for every project you do. In many cases, though, that’s overkill. Here are the facts:
- The needed sample size is not based on the size of the target market, but rather on its variability (though many people, and online calculators, do use assumptions about population size to make generalizations). Some markets are simply more homogeneous than others, and in those cases you can have smaller sample sizes and still have good data. Of course, planned subgroup analysis can increase that level.
- When management says, “but we need 1,000 to be reliable”, treat it as a teachable moment. Explain that there’s a formula that’s used to calculate the required sample size, and the inputs to that formula have to do with how much reliability you want and the variability of that population—though you will be happy to spend more of their money if they’d like!
In the final analysis, sample size is more complicated than a rule-of-thumb, but it’s not rocket science, either. Some commonsense guidelines can lead you to the sample size that is “just right” for each online survey project. Answer these three questions, and you’re on your way to a happy ending.
Image: MGM_Photos
.png){kind=tracking}
