How to: Using Advanced MaxDiff in Your Market Research Survey

MaxDiff is an approach for obtaining preference/importance for multiple attributes (brand preferences, brand images, product features, advertising claims, etc.). An alternative to standard rating question type it forces respondents to make tradeoffs between attributes. This avoids the issue of each attribute possibly being equally important. If you have between 7 and 40 attributes to test keep reading to find out how to use AYTM’s Advanced MaxDiff in your survey!

When to use Advanced Max Diff

The Advanced MaxDiff test is a great way to compare many alternatives without overwhelming respondents by asking them to read and consider all items at once. It takes a list of your items to be compared, and shows them in a balanced order to each respondent 4 at a time.  Imagine you are working on launching a new product, you will likely have a list of features, claims and descriptions. Whether you are using them for brand packaging or marketing you want to find out what resonates with your consumer. What is important to them, what do they expect of this product.  An Advanced MaxDiff experiment will allow you to compile all of the possible features, claims or descriptions your team has imagined and find the hierarchy and distance between each. Ultimately allowing you to make data driven decisions on how to develop and market your new product.

 

Adding Advanced MaxDiff to Your Survey


Incorporating an Advanced MaxDiff experiment into your survey is a simple drag and drop!

Locate the Advanced MaxDiff icon and click to drag and drop anywhere in your survey. Once added to your survey you will only need to input the list of attributes you want to test and any directions you want to provide in the question text area. We recommend including a short instruction for it, such as “Please rank the following items in the order of your preference: from the most preferred on top, to the least preferred on the bottom.”

Once you have your list of attributes add them to the experiment. An advanced AYTM programming tip, if your list of attributes is compiled somewhere like a document simply copy the list and paste into the first attribute space, the system will fill in the list for you.  The number of attributes you have to test will determine the total questions respondents will be asked for this experiment and how many total completes we recommend for statistical stability. As you add attributes the system will update the total questions in the experiment under the last attribute in your list.  

Advanced MaxDiff: Express vs HB (Hierarchical Bayesian)

We now offer two options when using our Advanced MaxDiff research test, Express or Hierarchical Bayesian (HB).

Express

When you add an Advanced MaxDiff experiment to your survey the default will be Express mode as show in the example below. As you fill in your attributes the system will update the line at the bottom of the experiment with how many questions each respondent will see for this exercise.

This method is focused on collecting general aggregate information, without the intention to obtain individual-level estimates. In a typical setting respondents would see 3-5 screens. 

Maximum Likelihood Multinomial Logit model performs the core analysis of respondents’ preferences. 

 

Hierarchical Bayesian or Advanced MaxDiff HB

When you add an Advanced MaxDiff experiment to your survey click the dropdown outlined below to switch to HB mode. As you fill in your attributes the system will update the line at the bottom of the experiment with how many questions each respondent will see for this exercise.

This method is focused on collecting high-resolution individual-level data to be analyzed by Hierarchical Bayesian model. In a typical setting respondents would see 10-20 screens. 

The core analysis of respondents’ preferences is performed with help of Hierarchical Bayesian Multinomial Logit model. Bayesian model is estimated with Hybrid Gibbs Sampler with a random Metropolis step MCMC. The number of burn-in iterations is determined automatically when there’s enough evidence for convergence.

The model considers properties of other items presented in a task when respondent makes a choice. The best/worst probabilities correspond to the logit transformation of the linear combination of utility scores of the packages in the task. Respondents are analyzed individually, with their preference scores being a realization of pooled “average” opinion which follows a Normal distribution, at the same time reflecting their individual preferences. As a result, raw Logit coefficients are available for every respondent. 

 

Launching an Advanced MaxDiff Experiment

Respondents taking the survey will see 4 features in each question and see each feature twice. They will be asked to rank each set of 4 and will repeat this for the number of question listed as you program your experiment. This experiment is an improvement on the classic symmetrical tables allowing respondents to focus on ranking the winner and loser with minimal effort. This helps to keep respondents engaged, reducing the dropout rate and ensuring the highest quality data for you!





While you are building your survey keep in mind how many questions the Advanced MaxDiff will require. For example if you have 7 attributes using Express will add 3 questions to your total survey while HB will add 6, if you programmed 10 question including a MaxDiff it would be 13 total questions and 16 total questions respectively.

 

How to Analyze Your MaxDiff Results

Now that you have successfully programmed and launched your Advanced MaxDiff experiment you can view the results. When you navigate to your statistics page and scroll to your Advanced MaxDiff experiment you will first see graphic and the menu to choose which analysis option you want to review. We have the same three options, maxdiff scale percentages, average probabilities, and raw coefficients whether you use Express or HB as show in the examples below. The differences between Express and HB are how they are analyzed on the back end, which is explained in more detail above.

Express

The statistics page has three display modes: 

  • Raw coefficients mode – aggregation of zero-minimized raw Logit coefficients. Score of 0 means the item was least preferred compared to all other items. The larger the score, the more preferable the item was.
  • Average probabilities mode – aka Probability of Choice – probability of this item being selected in an exercise with two items, the second item being an average item. 
  • MaxDiff Scale % mode – is the average probabilities mode scaled down so that sum of probabilities equals 100%. 


Export includes:

  • Raw coefficients export – zero-centered Logit coefficients for each respondent 
  • Raw data Export –  data on what each respondent saw, and what decision was made on each task

HB (Hierarchical Bayesian)

The statistics page has three display modes: 

  • Raw coefficients mode – aggregation of zero-minimized raw Logit coefficients. Score of 0 means the item was least preferred compared to all other items. The larger the score, the preferable the item was.
  • Average probabilities mode – aka Probability of Choice – probability of this item being selected in an exercise with two items, the second item being an average item. This statistic is calculated individually for each respondent, then aggregated using simple mean.
  • MaxDiff Scale % mode – is the average probabilities mode scaled down so that sum of probabilities equals 100%. This statistic is calculated individually for each respondent, then aggregated using simple mean.

When applying filters to a survey, Advanced MaxDiff HB questions will show aggregate statistics for the current subset of respondents. 

Export includes:

  • Raw coefficients export –  zero-centered Logit coefficients for each respondent 
  • Raw data Export – data on what each respondent saw, and what decision was made on each task


The AYTM Difference

AYTM’s Advanced MaxDiff test is more accurate than the classical MaxDiff test, because it uses an adaptive real-time randomization algorithm that works while the survey is being fulfilled to provide the greatest possible efficiency of item distribution per quads and per respondent, rather than relying on a predetermined map of the entire test.

All of this heavy lifting is done by us seamlessly in the background. If you are interested in incorporating Advanced MaxDiff in your next survey don’t hesitate to reach out to support@aytm.com for help!