Making numbers count


Alison Starr

Director, London

Categories Blog

The Cambridge Analytica scandal and Ticketmaster data breach have brought into sharp focus the impact that new technologies are having on our data-driven society.

Likewise, a flurry of GDPR-related emails is waking us to the sheer amount of companies that have an interest in us.

As we develop and refine our quantitative approaches at The Big Picture, we’ve been thinking about just that. How can we use the tension between an increasing thirst for data and the responsibilities of the industry, as an opportunity to take stock and change? How can we make data more transparent for participants and clients alike?

Image: Dennis Kummer

We recently ran an online poll with our clients. It had a simple question – ‘How much do you trust your quant data?’ The views you – our clients – shared, echoed our ambitions.

“Too many professional respondents.”

“Agencies are dishonest. Data can be interpreted in too many different ways.”

“Often questions are asked in a leading manner”

“More mixed methodologies to back up results from each other; quant data can never stand alone – it needs to be rooted in qual insights”

These comments were pretty typical. So what’s gone awry? Historically quantitative data was seen as the preserve of robustness, giving black and white illumination on marketing issues.

It seems to us that as researchers, we’ve lost the connection between the number and the person. We forget that data reflects behaviour, and that it’s not just about numbers in their own right. To quote The Prisoner, our participants would be right to shout “I am not a number!”

At The Big Picture we place great emphasis on getting to the heart of real behaviour. We try, as much as we can, not to ask people what they do, (or why they do things), but to observe what they do. And our approach focuses on eliciting and capturing immediate, intuitive responses to new ideas, packs or brands.

Image: Jannes Glas

Within the industry there’s still a great reliance on direct questioning. There’s a tendency to ask people what they do, and to rank their emotional responses using rational, often numeric scales. Even the use of emoticons in surveys still masks the numbered scale behind the different smiley faces.

Why, when the market research world has been embracing the thinking of Daniel Kahnemann and his ilk for the last decade, are quantitative surveys still based around explicit questioning, when the drive from clients is all about uncovering (and measuring) the implicit?

To do so, is reducing the people we’re talking with to numbers. It doesn’t actually value their behaviour, or their emotional responses, but aims to reduce them into nice, neat numbers (preferably percentages to show to the board!) And in so doing, risks losing all the nuances and insights that can truly make the difference in launching successful brands or designs, because you’re failing to appreciate or capture the reality of the consumer’s world.

Image: David Werbrouck

Jeff Bezos, founder and CEO of Amazon would seem to share our views:

“I’ve noticed when the anecdotes and the metrics disagree, the anecdotes are usually right … that’s why it’s so important to check that data with your intuition and instincts and you need to teach that to executives and junior executives.”

We’re not sure that answering our clients’ problems using our intuition and instincts is quite enough, but we do share the view that quant on its own will tell you only part of the story. And quant which is based on explicit questioning is even less likely to get you to the real story.

In developing our own quantitative products, we’ve used the same philosophy and approach that we employ with our qualitative work – to use approaches and questioning techniques to uncover real behaviour and elicit a more System 1 response. In addition, we always include other data sources, be that a qualitative component, or eye-tracking for example.

Image: Yeshi Kangrang

We’ve already seen the benefit of this: in the redesign of Genius gluten-free bread, the mixture of qual insights and behavioural quant helped us reach a much deeper and strategic understanding of design performance. And eye-tracking interviews showed the difference that a change of colour made to shoppers’ engagement when we explored the new packaging for Jus-Rol’s Bake-it-Fresh range.

Don’t get me wrong. Numbers are great. We love numbers – as long as they’re based on behaviour. And as researchers, we need to be able to turn these numbers into a compelling story for our clients, for the success of their brands. But numbers on their own? They just don’t add up.