Rank Order Questions in Online Surveys: A Question of Priority

Imagine you are surveying a customer list about possible new product improvements. You could simply display a list of possible options and ask them to select all they would like to see offered in the next twelve months. Simple, right? But what if they select many of them? How would you know which ones are really of the most interest?

Or imagine you are surveying your target market about new customer support ideas.  You might show a list of ideas and then ask them to rate each one on a scale from “Not at all” to “very” important. A common approach, but what if they say all of the items are “very” important? Is that going to be useful? Probably not.

Instead of a check all or 5-point scales question, you can consider a rank order approach. In this approach, you ask respondents to order things from high to low. This forces them to prioritize; they can’t say everything is equally important.  Even still, should you go with this approach, you have some wording options to consider. Here are three common alternatives, each with specific nuances.

Consider the following three rank order questions for a survey on digital camera product requirements.

‍What features are important to you when evaluating digital cameras? Please order the following items from most important to least important.  This is best asked if you have determined, in a previous question, that they are currently or planning to evaluate digital cameras within three months. Otherwise the question is way too hypothetical.

‍Considering the digital cameras you currently own, which of the following features do you like best? Order the features from highest (the one you like best) to lowest (the one you like least).  If you are looking to offer a new, differentiated product, a good first step is to know what the “table stakes” are. If you know certain features are well-liked in current products, than those may be the items you need to match (at minimum). Then, knowing the aspects that are least well-liked gives you input on what possible differentiators you may be able to seize. So everyone likes the current zoom levels offered? Great. So that is the base. But nobody, hypothetically, likes the ease of downloading? Now you have a potential differentiator.

‍When buying your next digital camera, what features are most likely to be tie-breakers? That is, if you are deciding between products, which feature is most likely to drive your final choice?  Please order the items from most likely to be a tie-breaker at the top, to least likely at the bottom.  While this approach is wordier, it reflects a reality: choosing between products in many product categories is often ultimately driven by one or two items, and all other features are not all that differentiated.

Ranking Order

Ranking items is a great way to get a  prioritized list, which is often important when informing new product decisions. However, these questions do take more respondent effort, so use them judiciously (one per survey is best). Also be very careful about what you include in those lists; you don't want to force people to prioritize a list of things if it is likely that nothing on that list is of greater value.  Do so and you may end up erroneously concluding that Feature X is going to thrill your target market—when really it is just the least bad of several terrible ideas.

‍Bottom line: if you find yourself over-relying on 5-point scales and “check all that apply” questions in your online surveys, the remedy may be a rank order question.  In AYTM, this is called the reorder format.