Top 10 Ways to Align Your Survey Sample Size with Your Goal

What’s the magic sample size number? Every researcher is asked this question at one point or another in their career. And rightfully so. Going too small puts you at risk of losing data significance. Going too big puts your budget at risk. While identifying the magic number is never a perfect science, there are certainly guidelines that can help you get there.

For your next research project, consider these 10 factors when determining your magic number.

1. What are you trying to achieve?

Why are you conducting a survey? How will you use the data that you collect? If you’re planning to use the data to make a critical business decision, then you need to be certain that the data is reliable and accurate. However, if you’re just trying to get a general understanding of how a specific population feels or thinks, you have some wiggle room. In other words, the use and intended goal of the data plays an important role in determining how big your survey sample size needs to be.

2. How much time do you have?

Sometimes, there isn’t enough time to survey a large sample size. If you need your results quickly, then you might have to reduce the size of your sample to get data in time to meet your deadline. Of course, this is a decision you’ll need to make with the understanding that your results might not be as representative of the full population if the sample is too small. Only you and your team can determine if it’s necessary to reduce the sample size in order to meet a tight deadline and at what point the sample size gets small enough that the deadline must be changed or the survey isn’t worth it.

Depending on the market research provider you’re working with, you might be able to get results to your survey sooner without sacrificing your sample size. For example, AYTM’s Blended Panel can expedite the completion of your survey when you need results sooner.

3. How many qualified respondents are in your pool?

Your survey population is made up of the total number of people who fit the audience profile you’re trying to connect with. This profile could include demographic, psychographic, and behavioral traits. For example, your population size could be all adults in the United States or all men between the ages of 18 and 35 who own a motorcycle. Your sample is a selection of people within that total population who have been chosen in a manner that deems them to be most representative of the total population. While you don’t need to know the exact number of people in your total population, you can approximate it.

Next, you need to determine how difficult it will be to connect with people in your population. How many can you find who will actually take your survey? The harder it is to find people in your population, the longer it will take to complete your market research study, and the more expensive it will be. With that said, it’s common for sample sizes to be small for surveys with a limited number of qualified respondents.

4. What’s your variability?

Standard deviation is used to determine how much variability you expect in response to your survey questions. For example, if you’re surveying a broad population on a broad topic, you’re likely to get a lot of variability in responses. However, if your population is narrow and homogeneous, responses will vary less. That means your sample size has to be larger for broad populations and broad topics. A standard deviation of .5 (50% variability) is typically safe.

5. What’s your statistical accuracy?

When you present your survey results report, you’ll probably get a lot of questions. One will likely be, “Is the data statistically significant?” Statistical significance is a measurement that tells you whether the results you obtained were caused by a factor of interest or by chance. Unfortunately, sampling errors can cause problems with survey data, so statistical significance tells you when a finding is real or not. In other words, if data is statistically significant, you can feel confident making decisions based on that data.

The two primary factors that affect the statistical accuracy of the data you collect are sample size and variability. Both problems can be mitigated by choosing the right sample size, so consider how statistically accurate you need your data to be before you finalize your sample size.

6. What’s your acceptable margin of error?

All surveys deliver responses with errors, and you need to consider the margin of error you’re willing to accept in order to determine your sample size. This is also referred to as the confidence interval. It shows the threshold of response accuracy in your results. For example, if you’re willing to accept a 5% confidence interval, then you would report a specific finding with that interval included saying, “90% of respondents like the new logo with a margin of error of +/- 5%.”The lower the margin of error you’re willing to accept, the larger your sample size must be. Remember, a larger sample size will reduce sampling errors and the confidence interval, but it will also add time and money to your research project. There is an inverse relationship between sample size and margin of error, but at a certain point, continuing to increase the sample size won’t improve the accuracy of your results by much. This is part of the law of diminishing returns, so don’t feel like you need a huge sample size to get reliable results.

7. What is your acceptable confidence level?

The margin of error tells you the extent to which the results of your survey reflect the overall population, so a smaller margin of error means responses are closer to the overall population’s responses at a specific confidence level. Stated another way, confidence level tells you how often the percentage of respondents who answer a survey question in a certain way fall into the confidence interval. In research, confidence represents not just how confident you want to be in your data but also how much risk you’re willing to accept by collecting data that isn’t perfect.

Remember, you’re collecting data from a sample that is representative of your total population, and you need to project that data onto the full population. Ask yourself how confident you need to feel in your results in order to reach your goals. How much risk are you willing to accept if you can only be 90% confident in your data? What if your confidence level is 95% or 99%? The higher the confidence level, the larger the sample size must be.

8. Who are your audiences?

Is your population made up of a variety of subgroups whose opinions you need to understand? For example, do you need to analyze your population by gender, age, income level, or marital status? If so, your sample size needs to accomplish two things. It must be large enough to give you acceptable data related to your total population, and it must be large enough that each subgroup is adequately represented.

You can meet your subgroup requirements by increasing your overall sample size or by seeking responses specifically from subgroups until you reach your sample size. This can be done using online survey tools such as the balancing feature within the demographic targeting options of AYTM’s survey builder.

9. What is your budget?

How much money do you have to spend on your survey? Quantitative market research is affordable thanks to online survey providers that enable you to quickly survey consumer panels. However, depending on the accessibility of qualified respondents in your population, the length of your survey, the types of questions included in your survey, and how quickly you need results, the price can vary quite a bit.

As mentioned earlier in this article, the larger your sample size, the more expensive your survey will be. That’s because it takes more time to find qualified respondents and gather the data you’re looking for. Don’t forget to think about your budget when you determine your sample size. With AYTM’s survey tool, you can see the price of your survey in real-time as you build it, so you always know what it will cost to get the data you need.

10. How will you make the most of your data?

Of course the strength in your data relies on how you break it down, analyze it, and ultimately take action on it. Do you know how to slice and dice the data collected in your surveys to make decisions based on that data? Are you prepared to take action on that data and take the necessary steps to use it for your business? Your answers to these questions will affect the size of your sample.

If you’re not familiar with statistical analysis and don’t know how to interpret the data you collect in order to act on it, then you’ll need a sample size that gives you complete confidence in the reliability of your data. Yes, your sample size will need to be larger, which means your study will be more expensive. Keep in mind, survey providers like AYTM make it easy to choose a sample size, collect data, and analyze that data in reports you can use to make decisions with confidence. The best part is, they’re extremely affordable when compared to traditional market research providers.

The Takeaway

If you don’t survey a large enough sample of your target population, then your market research results might not be reliable. Don’t make business decisions based on questionable data. Instead, ensure you’re surveying enough people. Otherwise, your efforts and research investment could be worthless. Or worse, you could do more harm than good to your business.