What's wrong with focus group interviews?

#1: You're talking to System 2 while real-world decisions are mostly made by System 1

I may be repeating myself but raising awareness of the existence of two information processing systems — automatic, habitual and impulsive System 1 and rational and slow System 2 — and translating this knowledge into everyday work are crucial for the issues at hand here.

Assuming you're not in the business of selling apartments or similar products that we buy once, maybe twice in a lifetime and on which we spend most of our life savings, you can safely assume that your consumers make decisions about your products using System 1. System 1 is automatic, intuitive, and habitual. It's the hidden, unconscious part of your mind. System 2 — analytical and rational thinking — is rarely involved in everyday decisions. Yet, System 2 believes that it is very wise (it is) and that it knows everything (it doesn't). It thinks it understands System 1, its motives, and actions. On top of that, it likes and is good at creating stories, so it quickly makes up explanations as to why a person bought your product or why they're using your services.

What does this all mean? It means that when you ask consumers why they bought your product, or what is important to them when choosing products in a given category, you get an incorrect answer. The answer to the questions you ask during FGIs are provided by System 2 while purchasing decisions are made by System 1. People fill in the gaps in their consciousness but (so) the answers you receive have nothing to do with reality. And not because people lie or have bad intentions: They really do think they are telling the truth because their System 2 believes it... and you believe them and base your marketing strategy and tactics on their declarations.

#2: Conversation changes perception

The distinction between System 1 and System 2 is only the tip of an iceberg. As behavioral research shows, the very fact that you ask a consumer to explain their choices and motives results in a change in preferences, and, consequently, in a change in behavior.

A few years ago, a simple study was carried out that illustrates this mechanism well. Students at a university were given the opportunity to choose one of two posters and to take home the one they liked more. Half of them chose a poster, assessed how much they liked it, and took it home. The other half were additionally asked to explain why they chose this poster. A few weeks later, all the students were asked to assess again how much they liked the posters they had selected.

What was the result?

Those who were asked to explain the reasons behind their choice liked their posters less than those who didn't have to justify their decisions. Ask your consumers “why” and their response may shift their preferences, without you or them noticing it. You will get your answer during the interviews, but this answer will stop being true a second later, yet you may end up basing your marketing decisions on it for months or years to come.

#3: Choice blindness

Another significant cognitive bias at play here is choice blindness. As behavioral science research shows, we don't pay as much attention to the decisions we make as we think: We choose A, someone gives us B and we not only don’t notice that we were given the option we did not want, but we're also able to easily justify “our” choice without any problems, easily giving numerous arguments for the superiority of B over A. Moreover, as a result, our preferences change and from this moment onwards we actually end up preferring B to A.

As you can see in the TED Talk below, this mechanism is entirely unconscious and concerns not only such trivial decisions as choosing a poster or a face of a random person, but even matters as important as our political preferences.

#4: Humans are social beings and other cognitive biases

Focus group interviews are a magnet for cognitive biases. If you look at the complete list of biases with FGIs in mind, you will suddenly realize that at least half of them can appear during a focus group interview. The most notable ones are:

• Social influences, in the form of messenger effect and social proof. Opinions of FGI participants are shaped by what other participants are saying, especially if these other participants are similar to them, have authority (e.g., a dominant personality) or constitute the majority of the group.
• Choice confirmation bias is our tendency to search for, favor, and only to recall information that supports our existing beliefs. In an FGI setting, this bias leads to participants selectively recalling situations or examples that support an opinion another participant just expressed an opinion, forgoing or forgetting examples that may contradict it.
• Hindsight bias (AKA the “I-knew-it-all-along” effect) is our tendency to perceive events as predictable after they happen. Hindsight bias also leads us to retrospectively justify our decisions. In an FGI setting, this leads to participants thinking they had and knew the reasons for their choice at the time of making it, while these reasons may not have been there at all. They only make sense now, after the decision has been made. In other words, they're just a made-up story.

#5: Stop drawing quantitative conclusions from qualitative research

My nearly 10-year-long experience working in marketing shows that, unfortunately, marketers often treat qualitative research as quantitative research. Of course, their System 2 knows that one should not generalize the opinions of 20 or 30 people to the entire population, but System 1 does its job, and the conclusions are generalized.

A big no-no.

The opinion of Mrs. X or Mr. Y is an opinion of one person. Sure, it may turn out that it's a great insight, which indeed resonates with your entire target group, but to know this, you need to conduct a quantitative study on a sufficiently large sample. Insights and conclusions that come from focus group interviews conducted with three or four dozen people can't be generalized to entire populations.

Why marketers rely on FGIs so much, even when they know better

Reason #1: Status quo bias

People don’t like change. We like what we already know and what is familiar. Familiar things are safe, while all that's new is risky and costly. There may be a financial cost, but there definitely is a mental cost and a time cost caused by the need to come up with a new methodology, design a new procedure, spend time figuring out how to do things better. All this leads marketers to procrastinate. After all, if focus group research has been used in marketing for 40 years, why not continue doing it? After all, one more research project won't hurt, right?

Reason #2: Choice confirmation bias

Back in my marketing days, we did a lot of market research, including FGIs, of course. According to the studies we ran, it was our brand that had the best image on the market. At least we did until I spoke to a colleague who worked at a research agency that worked for our competitor... and I learned that, according to my colleague's company's research, it was that competitor who had the best image on the market. How is this possible? 

Experience tells me that such results have nothing to do with the lack of competence of the moderators responsible for the focus group interviews. We worked with the best. My colleague was one of the best as well and worked at a great agency who knew their methodology and what they were doing. The problem is that qualitative research is an ideal ground for the aforementioned choice confirmation bias, i.e., our tendency to only take note of information that confirms our opinions or assumptions. Moderators may try their best, but they are only human too — they may (and apparently do?) unconsciously select the information they perceive and recall from a study, resulting in their conclusions being biased and unreliable.

Reason #3: A safety net

I don’t know if there’s an actual cognitive bias to describe this phenomenon, so let's call things for what they are. Marketers often work in big corporations, with hierarchies and politics. They have business objectives, budgets, competitors, and bosses. They need to cover their butts.

It seems that quite often the main reason for marketers investing so much money into FGIs is wanting to have a safety net, and a "proof" for a supervisor, a CMO or a CEO, in case a new product, service or marketing campaign isn't as successful as everyone may have wanted it to be. This leads to a (funny?) phenomenon whereas everyone inside the company does know that information gathered at focus group interviews isn't very reliable or representative of the entire population, but they still use it because it's relatively cheap and quick to obtain and, most importantly, it provides a safety net. You can always say that consumers said they liked it.