The Evolution of Market Research: Market Research in the New Age

Posted Apr 15, 2019

Larry Praml

As we’ve discussed in part one, The Evolution of Market Research: The Dawning of the Digital Age, and part two, The Evolution of Market Research: Automation Improves Efficiency of this series, the digital age has changed market research data collection dramatically, effectively eliminating the need for interviewers and data entry resources from the process in most quantitative survey research, saving both time and money.

But it is still evolving and the next wave of changes may even be more radical. Consumers leave data trails everywhere they go without answering surveys.

Social media offers a raw, unfettered, unscripted and voluntary source of consumer interests, attitudes, and beliefs on a wide range of topics. And consumers generate geolocation data, transaction logs, web activity and more on a nearly continuous basis in our digital world, passively, without prompting.

This digital diary that every one of us leaves behind is a potential gold mine. For instance, your digital record can show when you were exposed to a digital ad, what websites you visited to do research on a product, what search terms you used, when you purchased an item and how much you paid for it.

Your path to purchase is laid bare. Along with millions of others’ paths, across millions of transactions and millions of transactions that were researched but never converted into a sale. This is just a portion of the “big data” that exists and is growing exponentially. And the value of being able to tap that data for insight is clear. This may be the marketer’s Holy Grail of data. There is little doubt that passive digital data panels will be the next big thing in data collection.

But adding more data sources usually introduces more complexity for analysis. A few years back when surveys were first designed with smartphone users in mind, leading platforms developed “in the moment” approaches which incorporated video open ends, image uploads and other rich media inputs from respondents as part of the data collection.

It was a huge advancement in data quality and depth. But it also created a real conundrum on the back end. Who was going to review, code and analyze all of those videos? And who had the budget to pay for that analysis?

On the data operations side, artificial intelligence (AI) is being seen as a potential game-changer for making sense out of all this data. Algorithms have been developed, and are being refined further, to analyze vast amounts of unstructured data (like open-ended text and video) for themes and sentiment in a tiny fraction of the time it would take a human to code and analyze them.

AI will eventually help marketers, forecasters and strategists tame the “big data” monster.

But there is lower hanging fruit than that for AI. Most quantitative survey data is structured and pre-coded (a rating scale, for instance). Structured data is much easier to analyze, so while AI experts work on the hard task of cracking the big data nut, along the way they have developed algorithms for analyzing structured survey data, as well, effectively automating analysis and reporting.

While still relatively new, automated insight development and reporting should relatively quickly become a standard feature on survey platforms.

As insights become more data agnostic, and as machines do more of the heavy lifting, operational expertise in market research is less valuable and strategy consultation more valuable. Management consultants, like Deloitte and others, are likely to benefit from the cost savings on the operational side of market research. And eventually, management consultants are likely to give way to strategy algorithms, as well.

AI isn’t just a phenomenon impacting the world of market research. There is a lot of work being done in this area across many industries where decisions are made based on large amounts of available data, including healthcare (e.g., patient diagnostics), finance (e.g., fraud protection and cybersecurity) and automotive (e.g., self-driving cars). The heavy investment in this technology is likely to accelerate its development and amplify its potential.

So while the dawning of the digital age has been the biggest catalyst for market research evolution to date, mostly with how we collect data, AI is likely to be an even more disruptive change-agent. It will help us organize, interpret and act on all of that data, giving us the ability to process far more data in a fraction of the time and at a fraction of the cost of what we can achieve today.

I’m not sure what the pioneers of market research envisioned for the future of market research a century ago, but thanks to some major technological breakthroughs and economic motivation, the industry’s evolution from being highly labor-intensive and specialized to leveraging technology and automation across the board has been nothing short of remarkable.