The Evolution of Market Research: Automation Improves Efficiency

My first post on the evolution of Market Research to the digital world described online research being a game changer in terms of saving time and money vs “traditional” methods, particularly in the realm of quantitative research. But even as online research became a $3 billion industry, and overtook all other modes of quantitative research within only 10 years after the introduction of web-based surveys, it was really only the beginning of the impact this new technology would eventually have.

Early versions of web-based survey platforms were fairly simple and were meant to do fairly simple types of research, like concept tests and other programmatic, CPG-research staples.

Two early pioneers in the online research space (Zoomerang and SurveyMonkey both debuted in 1999) offered self-service platforms for non-researchers to do very simple research on their own. They were so simple, in fact, that a user didn’t need to know programming languages or analytics packages to conduct research. These platforms included real-time access to results.

This took the promise of saving time and money to a new level. A user would no longer have to spend a week or more soliciting proposals from market research firms, get into their programming and data processing queues and wait for the results. They could launch a quick survey and get feedback within just a few days.

Of course, leveraging these early self-service platforms did require having bandwidth within an organization to manage and execute the project, and there weren’t really guardrails in place to ensure the research was being conducted with any recognized research standards.

So while both Zoomerang and SurveyMonkey boasted millions of users within just a few years of their introductions, they were generally used for very simple feedback, not for more strategic work.  

Meanwhile, as online research became more prevalent, there was a push to get more types of research online. While developers continued to improve their web-based platforms incrementally, much of the efficiency gains for doing more sophisticated research online was accomplished by building up off-system capabilities:

  • Programmers with knowledge of advanced scripting allowed for more sophisticated survey programming
  • Data scientists using statistical packages like Sawtooth or SPSS to apply the appropriate analytics to the data
  • Data processors creating tables which could be used to populate PowerPoint charts
  • Project management teams to facilitate these cross-functional handoffs and keep projects on schedule

For this type of research, web-based platforms were primarily used simply for data collection. Pretty much everything else was handled off-system. Efficiencies were gained through better labor utilization more so that improvements in technology or more automation.

Given the amount of manual work still required to do this type of research, and high overhead rates resulting from years of Marketing & Automation activity in the industry and investments in platforms and organizational infrastructure, there was clearly room for more disruption.

The late 2000s saw another economic downturn, this time the crash of the financial sector. The Great Recession had global reach and again impacted research budgets putting pressure on the industry to innovate and automate further.

In the ten years that have followed, we have really seen some incredible advances in the sophistication of self-service research platforms, as evidenced by AYTM’s offering (as a single, excellent example among several others, like Qualtrics).

These are extremely efficient platforms that offer sophisticated pre-programmed questionnaire templates, embedded analytics, and data visualization tools baked in. (In AYTM’s case, it even includes an integrated consumer panel.) The need to move off-system for additional customization is becoming rarer and rarer.

At the same time, there is also more affordable guidance available than ever before to give self-service users confidence. Self-Service platforms almost universally offer assisted services from survey design to reporting as an optional add on. As suppliers continue to get squeezed, more researchers are branching out on their own as consultants to help self-service researchers design and execute studies using the same standards Market Research vendors use.

So what’s next? There are some really interesting developments around data collection and analytics which seem poised to be part of the next big breakthrough in marketing research. I’ll discuss a few of these in part three of the evolution of Market Research series.

Find Part One here