Know When to Hold ‘em, Know when to Fold ‘em: Open Ended Questions & Online Surveys

Online surveys often include something called the “open-ended question”. An example of an open-ended question might be, “Is there anything else you'd like to share about your recent experiences?”, placed at the end of a customer feedback survey. Or when testing new product features, we might ask, “Are there any other features you would like us to add?” These questions are called open ended because you're soliciting unstructured responses to your question; there is no 5-point scale or list of pre-scripted answers from which to select. To the researcher, this is both the curse and the value of the open-ended question.  The responses are messy and sometimes confusing, and the data collected will never fit neatly into your results without a lot of processing. Still, you can gather information here that you won’t access in any other way.

Think of it as a wild card, the Ace-in-the-hole of the survey world. Played skillfully, it can be a real winner, but there are some issues that have to be considered in applying the technique.

The Power of Words

Open-ended questions can be very powerful in research because you get to see people’s top‑of‑mind perceptions and attitudes, and you're getting that in their own words. It can be very enlightening to see what word choices people make. When they're describing their brand preferences, purchase plans, unmet needs, what words do they use?

Step 1: Know When to hold ‘em, Know when to fold ‘em

If you plan to use the results of open-ended questions, take the time to read and clean them first. A small number of people will write-in bizarre and obviously unrelated comments. Remove the obviously extraneous comments before you start your analysis.  And always clean your open-ends before you share them with a wider audience; trust me, it will save you a lot of aggravation.

Filtering the Responses

So let’s say you do a survey and you end up getting lots of responses to your open-ended questions. This is where the work begins.  How do you analyze these results?  Here are a few strategies.

       
• Visual tools — one option is to export your open-ended responses to a word cloud tool like Wordle or Tagxedo. These tools will then create a visually appealing graphic, where the size of each word corresponds to how frequently that word appears in your responses. This creates a very compelling visual display that tells a story; what words are coming up the most versus coming up the least? Here, I’ve used Wordle to create a word cloud from the text you’ve read so far.
       
• Theme scanning — for a more robust approach, scan through all the responses first and identify eight to ten recurring themes. Examples of themes might include technology, customer service, logo, product problems, etc. You may also have a “catch-all” category for positive and negative themes that don’t fit a specific group. Once you've identified these themes, you can go through and tally them up to see if there are any patterns. Which of those themes actually turn out to be most prevalent versus least prevalent in your survey results?
       
• From the Horse’s Mouth — A third and favorite strategy is to insert some of the text responses as quotes into your report.  People love to see how customers express their brand preferences, product needs and do on in their own words. It’s a guaranteed crowd-pleaser.
       
• Tools you can use – Budget permitting, there are some great text analytics tools for deeper analysis. These tools do the heavy lifting of finding themes and sizing them up for you. Attensity, Clarabridge and IBM’s SPSS text analytics module are some of the currently available options.

Keep in mind that although open-ended responses are generally qualitative rather than quantitative data, I promise that your audience is going to want to see counts. They want to see which type of comment was more common and which was less so.

Use Sparingly to Win

Open-ended questions are a powerful tool for online surveys, and in most projects it makes sense to have one or two. Because they require more from the respondents, you do need to avoid over-use, or your completion rate will suffer. Play that card only when it is most likely to win, though, and you’ll have the satisfaction of a straight flush.

Image: Helga Weber