Market research has been around for centuries but has been commercially used to understand how to motivate consumers since early in the 20th century. Some pioneers of the industry, like Daniel Starch and George Gallop, came from academia, while others, like A.C. Nielsen, had backgrounds in other industries.

These pioneers developed methodologies and sampling techniques to create more effective advertising, test products prior to mass production, estimate market shares and measure media use and public opinion. For decades, though, collecting quantifiable data with a reliable sample was an expensive, labor-intensive proposition. As the industry grew, data collection techniques evolved to bring costs down. Door-to-door interviewing gave way to central location interviews and telephone interviewing. Mail surveys brought costs down further but were more time consuming and difficult to control for a reliable sample.

Technology improvements in the 1970s and 1980s further improved timing and costs as more automation was introduced. Computer Aided Telephone Interviewing (CATI) systems made data entry instantaneous, replacing the need for punch cards or key entry data from pencil and paper surveys, and predictive dialers improved interviewer productivity. Desktop computers and commercial software packages made complex statistical modeling like Conjoint Analysis accessible to more data scientists. These technological improvements helped increase speed and lower costs, but the biggest impact came in the mid-’90s as the fledgling World Wide Web began to scale. Visionaries saw the potential of an almost instantaneous reach to a broad consumer base. And web-based surveys allowed rich media (clear images, videos, audio) to be used in surveys to better describe concepts.

Early online surveys were criticized for having a non-representative sample. But as online panels improved (and as response rates for telephone surveys declined, calling the reliability of phone sampling into question), the massive savings of time and costs were impossible to ignore.  Companies like General Mills helped fuel the burgeoning industry claiming time savings of up to 75% over traditional phone interview methodologies with cost savings of up to 50%. Other respected giants like P&G and Unilever joined announcing that the majority of their massive research budgets were going to be spent using online methods. The established industry was quick to respond with more investment in technology and survey platforms, conducting research to improve online research standards. Meanwhile, following the dot-com bust in 2001, there was a flood of talent on the market that also saw an opportunity in a growing space, leading to new entrants and accelerating development.

In 2006, roughly a decade after the introduction of web-based surveys, online research was a $3 billion industry, according to the European Society for Opinion and Market Research (ESOMAR), and had overtaken both phone and in-person interviews in terms of overall revenues.

The Takeaways

The digital age has arrived, but the evolution continues. Improvements in technology and platform capabilities have taken these tools to a new level. My next post will take a deeper look at the emergence of highly automated DIY platforms, the impact they are having on the industry and the promise they show for the future.