Matrix Questions in Online Surveys: Satisfying But Addictive

Online surveys can be designed with many different types of questions. Indeed, online survey platforms such as Ask Your Target Market give survey writers many different options beyond the basic “select one” from a list of answer options. One of the popular options is the so‑called matrix questions (which uses an interactive “sliders” interface in the AYTM platform).Matrix questions are a favorite of survey creators because they are very space efficient; you can collect a lot of data in a single question. See below, created using AYTM.

As you can see in this example, the matrix question looks like a table. Each row is a particular variable and the columns represent answer options (in this case a 5 point satisfaction scale). In this example we are asking iPad owners to rate their satisfaction along a number of specific attributes. The good news?

• In a single question, this six-row matrix collects a lot of information.
• It is easy for the respondent because they have one common structure, the satisfaction scale, so it’s very simple to read down the different answer options and indicate answers.

But while the matrix question is a favorite of market researchers, there is a catch.Online Survey Design CrutchSo what’s the problem? Unfortunately it’s very easy to get addicted to matrix questions. Many survey designers end up over-using them to the point where two problems are created:

1. Survey writers apply a selected scale even when it is not the best choice. Not every question is best asked in the form of a Satisfaction scale, Agreement scale and so on. Too many survey writers adopt a scale for a matrix and force their items into that structure. For example, even in the satisfaction survey, not everything needs to be asked along the scale of “not at all satisfied” to “extremely satisfied.”
2. Survey writers often create matrix questions that have too many rows. A matrix with more than 5 rows tends to turn thoughtful respondents into survey zombies: they start to select one answer and just use it the whole way down that long list. We have effectively created “speeders”—people who scan down that list and answer as quickly as possible, without thinking about each item.

Survey Design Best Practice: Everything in ModerationMatrix questions are a very efficient way to collect a lot of information, but we have to be cautious.

1. We don’t want to use them to the point where we’re encouraging respondents to speed through long, onerous matrices. If you use them, try to keep them to five or fewer rows.
2. Matrices, and a given scale used in them, may not always be the best choice for what we’re specifically trying to accomplish. Be sure to think carefully about what scale is really best for the items you want to measure.

Use them. Don’t abuse them.