Blog

Quantitative Context for Qualitative Research

The General Idea

Qualitative research is a set of tools made for the purpose of discovery. Quantitative research is a set of tools made for the purpose of impact and scale. While there is a clear line differentiating the two, it is a mistake to use either without the other to add the proper context.

Market Research can be thought of as a two party system; while both working to uncover insights to drive business forward, lacking the support of the other party impedes the ability of successful execution.

Thankfully the party lines are not as deeply entrenched as others which allow us to use the strengths of both qualitative and quantitative tools to accomplish what is necessary. The formulas below are an example of how a minimal amount of quantitative context can drastically improve the strength of any qualitative project.

Imagine you are starting some preliminary advertising research with a small series of in-depth interviews; we’ll say 15 interviews in total. When you are done, you have a list of findings about what people thought about the ad.

We are going to look at one way to determine if you have a list of discoveries that can be deemed ‘complete’. The goal of any qualitative research is to identify and understand peoples’ reaction to a given stimulus; in our example an ad. We know that we will never be able to uncover every possible reaction that anybody could have; but with a little math we can see the odds of our ability to have uncovered the vast majority of reactions.

The Math

So let’s continue our example to put a number on how complete our list of findings is from our 15 IDI’s. For our purposes, we’ll say that we are looking for issues experienced by at least 14% of our customers.

What follows is an explanation on how to determine what your list of qualitative findings represent in context of your total population.

P – Percent of Issues Discovered

i – Incidence of Issues (percentage of customers that will experience the issue)

n – Sample Size

P = 1 – (1 – i)ⁿ

1 – (1 – 0.14)¹⁵ = 1 – (0.86)¹⁵ = 1 – 0.1041 = 0.8959 = 90%

If there is a negative reaction by 14% of our customers concerning the language used in the advertisement, there is a 90% chance that it was discovered in our 15 completed interviews.

Say we have not yet started our interviews and are in the planning phase. If we are looking to calculate how many completes we need to discover issues that would be experienced by a certain percentage of customers, the following derived formula is used:

n = log( – (P – 1)) / log( 1 – i )

For example, if we want to have a 90% chance to capture any issue that is experienced by as few as 10% of customers, we would need 22 completed interviews.

log( – (0.9 – 1)) / log( 1 – 0.1) = log( 1 ) / log (0.9) = -1 / -0.0458 = 21.83 = 22

The Context

Our purpose in completing the exercise is to help explain how our list of findings from the qualitative research can be understood in the context of releasing our ad to the general population. It is always going to be challenging to stress the importance of research conducted with a small sample size (as most qualitative is) but having a figure that allows the audience to relate the findings to the larger population helps with digestion.

The original formula was cited in “A mathematical model of the finding of usability problems” by J Nielsen and T.K. Landauer. The derived formula for n was cited in “Deriving a Problem Discovery Sample Size” by Jeff Sauro.

Neuroscience and Market Research

Market Research is a slow moving industry. Every now and then there is a big breakthrough that changes the face of the industry but even those revolutionary changes are minor when you consider the shifts experienced in other professions. Case in point: the biggest thing to happen to research in the past 20-25 years is that surveys are now online as opposed to on paper. Granted, the shift in medium has created an exponentially increasing number of twists on respondent interactions but the basic ideas remain the same.

However, slowly but surely researchers have started to embrace different forms of biometric measurement that may truly revolutionize our industry. It started with some crude eye-tracking studies done as much for the pretty pictures as for the insight; but ever since Paco Underhill detailed his use of Neuroscience in mapping consumer shopping habits, Researchers have started to wander into what is truly a new world of consumer insight.

On the surface, it looks like the absolute magic bullet of research. We spend countless hours honing our methodologies to find ways of uncovering the ‘real’ answers from respondents. All of a sudden we are presented with a tool that no longer requires all that effort; we’ll simply read their minds!

Miss Cleo is Unavailable

Neuroscience is the study of the nervous system’s response to different stimuli; which due to its intimidating nature, is poorly understood as a market research methodology. Part of the issue is that it is still very expensive so most experiences with these studies are from third hand accounts which in some cases are as much myth as fact. All the same, it is a very cool method to see leveraged but like all other research tools, it has its limitations.

My intent is to profile some of the different Neuroscience methodologies so that some of the mystery can be cleared away. The end goal is to help remind everybody that like all other tools, they only work when applied correctly. Neuroscience is definitely one of the most intimidating set of tools we have but when fit with the right methodology, it can get you insights easily overlooked by other research.

Neuroscience in the Ivory Tower

If budget and respondent experience were non-issues, Neuroscience in the context of market research would utilize the very same tools that are used in hospitals. For the few where budget really is a non-issue, fMRI’s and MEG’s are the considered to be the best “mind readers” available. Below I have given a very brief description of the tools that inspire what is more commonly available within the reach of market research:

MRI (magnetic resonance imaging) is a large magnet that takes static pictures of the brain. Essentially a magnetic field is ‘pulsed’ which ‘resonates’ water molecules; think of it as strumming a guitar. Overall, it’s not terribly useful for market research but useful in understanding the fMRI.

fMRI (functional magnetic resonance imaging) is very similar to an MRI except that it can be used to image blood flow. This is a key difference because it allows you to see where the brain is active over a 3-5 second interval. That may sound fast but count out 5 seconds and imagine how many split second thoughts, emotions, or reactions can occur over that time.

MEG (magneto encephalography) is the most similar to what is commonly used in the field. This is a device that measures the electrical activity of the brain at over 1000 times per second. A key difference from the fMRI is that the MEG does not necessarily show exact brain function but that the brain is active. Sample output below:

Neuroscience in the Field

EEG (electroencephalography) is similar to the MEG in that it also measures electrical activity in the brain. However, it is not nearly as sensitive and has significantly fewer receptors to sense any activity. That being said, it is portable which the MEG certainly is not.

The respondent experience is still a little rough but there have been ongoing improvements that can make it less intimidating. Pictured below is an example of what the sensor looks like and why respondent experience is such an issue with this methodology:

NOTE: Some companies have developed proprietary alternatives that are supposedly less intrusive though it is best to speak with them directly about any impact to the results

How it’s Used

Since the EEG is measuring electrical activity in the brain, determining what “active” vs. “non-active” looks like requires a baseline to be established. That is important because it means that the measurements only mean something when considered over a period of time, that is to say that “active” is relative and is not based on an absolute scale.

Remember to keep this in mind because it impacts the types of stimuli you are able to test; something like a commercial which is experienced by a respondent over 30 seconds, is more interpretable than a print ad that is experienced all at once.

Reliable Results

While Neuroscience is clearly different than our more standard qualitative or quantitative methods, it has helped me to position it more closely to qualitative when developing project plans. These studies are typically less about confirmation or definition (goals more typical of quant studies) and more about clarification and deeper understanding (goals more typical of qual studies).

That being said, just because something has been “typically” used for one purpose, does not mean it cannot be leveraged for something else. It is certainly possible to use Neuroscience methods in a quantitative study; though it will likely be extremely expensive. If you are still interested, make sure to be very clear as to why you want a larger sample size. Remember that by itself, this method tells you a respondent brain is active or it’s not when responding to your test; understanding the whys still requires the tried and true respondent Q&A.

In the End

Neuroscience is still a very new approach within market research and creative methodologies and new approaches can turn any limitation into a strength. My intention has been for this to help clear away the mystery and make it easier to have discussions if/when Neuroscience becomes an option for you and your brand.

If this is a type of study that you are currently interested in pursuing, I currently do not provide Neuroscience services but would be very happy to direct you to a couple of firms that are experts in this space.

Creating Your Value or V-Curve

Building and maintaining a competitive advantage in any industry is difficult.

I want to make that clear so that what I am about to say is not misinterpreted. If you are charged with building and maintaining a competitive advantage in a mature industry…it is extraordinarily more difficult to be unique than in other industries. The reason is that in mature industries, everybody tends to know what everybody else knows. Insights that uncover strategic advantages happen though they usually lead to gains that are short lived and more time is spent figuring out how to implement than there is time enjoying the advantaged.

It is no wonder than that many of insights uncovered are not leveraged with much urgency…9 out of 10 times, it’s simply not worth the cost.

So, what if we could identify the 1 out of 10 insights that actually are worth the cost? From a research perspective it means forecasting the benefits of what we learn, from a management perspective it means breaking out of the shackles and thinking like an entrepreneur.

Mature Industries

Let me start with the catch. The process and assumptions that I am about to explain only seem to apply to mature industries. That being the case let me lay out some characteristics of ‘mature’.

Success is defined as protecting and/or stealing market share…no more adopters
Competitive advantage is built by matching and edging the competition
Consumers tend to focus on differences between brands more than benefits of the product

As a result of some of these market constrictions, often firms have a tendency to extrapolate those restrictions onto themselves. Granted, this is not always true but it is common. Firms start to operate under the belief that industry structure is now ‘given’ and that it constrains their actions:

Customer segments are fixed and differences have to be leveraged
Products and benefits are changed incrementally
Increasing functionality is almost immediately followed by decreasing prices

Entrepreneurial Thinking

There is risk involved when any brand tries to change the game. Think about the airline industry. A while back, Southwest very successfully changed the way the industry operated and became wildly successful (within the context of the industry). Then we look at JetBlue, a company that was started with the intentions of changing the industry based on some of the very same principals and they just couldn’t make it work the same way.

On the off chance that you do not have 10 ideas handy that will change the face of your industry, a useful tool to help brainstorm and operationalize ideas is the V-Curve, or value curve. Here are the guiding principles:

Think like you’re new to the industry (this can be really difficult so brainstorm with an intern if necessary)
Industry conditions can be shaped (worry about how when you have an idea worth the effort)
Do not measure against competition (worry about them when you have something worth comparing)
Do not constrain yourself with current capabilities (worry about it when you have to)

FOCUS ON DETERMINING THE IDEAL CUSTOMER PRODUCT
FOCUS ON COMMON ATTRIBUTES CUSTOMERS VALUE…NOT DIFFERNCES

Reality

It’s fun to have maxims like that. Here is what that looks like in the real world, or at least these are the steps to help turn lofty ideas into something that may yield something tangible.

1. Select a MASS Customer Market

A little counter-intuitive maybe but its best to start with your largest customer groups; after all if you’re going to really do something big, you want to appeal to a big group. So start with your largest 2 customer segments.

It really is best to use at least 2 segments so that your ideas do not turn into just the best solutions for a single group of people…our goal is to create a solution that works for many different types of people.

2. Understand Buyer Experience and Cycles

This is where research is going to come in very handy. You want to make sure that you understand how the segments are interacting with your product/service through the full lifecycle of their engagement.

What does their interaction look like?
Describe each portion of their brand interaction.

Qualitative focus groups or IDI’s are a common and very useful tool here that will let you define the specifics on each of the following customer interactions:

Purchase
Delivery
Use
Supplements
Maintenance
Disposal

Out of all the steps this is the one the trips up the most people. As managers in these types of industries, you know your businesses better than anyone. However, the questions that need to be

answered are not really about your business but about how your customers perceive your business. There can be a thousand reasons that giving consumer digital rights to music can’t work, but if consumers don’t understand that…none of those reasons really matter.

3. Identify STANDARD EXPERIENCE Attributes

When you are done with your focus groups, you should walk away with a maximum of 10-14 ‘macro factors’ for each customer segment. This part can be a little tricky too because ‘macro factors’ are attributes used to articulate customer experiences and are not inherently important or unimportant. So while some of these factors will be things that are critical to the segment, many may not be important at all.

For example, an airline may walk away with the following for a single segment:

Over the course of brainstorming and research you will probably have a list of close to 100 attributes for each segment that you simply ‘must’ consider. This will not be helpful to you because we are trying to find big ideas …it is way too early to be dealing with specifics.

After you have your list of ‘macro factors’ for each segment, start to compare your factors between segments to see which they have in common. This is the part that makes it important to have at least 2 segments participating so that you have multiple customer perspectives.

Your list of factors that apply across segments should have about 7-8 attributes; once you have them, you are ready to start some basic measuring.

4. Find Your Starting Point

Once you have your list of factors that apply across segments, you are ready to start some basic measuring. While this is a separate step, it is nothing elaborate; in fact, the more basic you make this, the more helpful it will be.

Take all of the factors that you identified in step three and place them across the x-axis of a simple x/y graph. Now, rate yourself on the y-axis on how well you do on each factor. Remember, this is supposed to be simple so ‘good’, ‘OK’, and ‘bad’ ratings work just fine.

After you have rated yourself, rate your top 2 or 3 competitors for the sake of comparison. To continue the airline example I have plotted 8 of the attributes listed above in the graph below. For the purposes of explanation, only one competitor will be compared.

5. Create New V-Curve

After you have your basic ratings plotted, you are ready to start thinking about your new Value Curve. In a perfect world, you would start this by running a quantitative study to understand the importance of each of your factors within all of your segments.

Interjection

I can already hear the protests and see the cost and time cutting ideas popping into your heads. Here is the thing, focus groups are not measurement tools…they are focusing tools. It does not do you much good to measure importance among such a small group. The goal of this quantitative portion is to correctly identify the mutually important attributes among your segments. This is not the same as finding your ‘macro factors’. Where before we were looking for common but neutral factors that contribute to customer experience, now we need to know which of those factors are more important than the others.

Again, some of you probably think you already know what matters and maybe you do; though in an effort to dissuade you from strategy by ‘gut feeling’ I will offer up that while most of these studies come back making intuitive sense, the outcomes are almost never predicted correctly. This is probably an argument heard over and over so I’ll leave it at that.

Also, measuring importance has its own set of assumptions and best practices. I’ll have to leave most of it for another time though I will say that asking a thousand people to rank the importance of the attributes is not the most reliable way to get this information.

Back to the Process

Once you determine what is important to your segments as a whole, rate them on the same chart that you had plotted your performance on the same factors. Connect the plots and take a look at your new Value Curve.

Now you have something real that lets you focus on the things that matter and gives you some permission to walk away from the money pits that don’t matter at all. For example, look at ‘Food Price’. It’s wonderful that our airline rates highly but it is not impacting the customers’ experience…so why waste the money? Or how about out ‘Waiting Lounge Area’? We do an OK job with it but our customers don’t care so why spend that money?

If we focus all of our energy delivering on the things that matter and start ignoring the things that don’t we put ourselves in a position to deliver the best possible product for our customers with less effort.

Initiating Execution

The practical implication of this tool is that allows you to identify all of the things that are distracting your brand from delivering on the few things that matter. In its purest form, this tool says to take all the resources being used to be OK at a lot of things and just focus on being excellent at a few things.

Like many good ideas, it is easier to say than it is do, so the trick is knowing what few things are worth that effort.

Organically Grown Brand Personality

Every brand has a personality. Even if the personality has not been defined by the brand itself it has been defined by your customers. Additionally, your non-customers have defined your brand; by their perception of the traits shared by your customers, non-customers decide how your brand relates to them.

Marketers have accepted this very idea when they noticed that one of their most powerful marketing tools was word-of-mouth. When friends recommend a product or service it is expected to be more influential than nearly any advertisement. At the same time, when a friend tells you about a bad experience with a brand, it is accepted that the criticism will be more damaging than reading a bad review online.

This understanding has done great things for the re-energized the importance of customer satisfaction studies. “Are they going to recommend us? Why? How are they different from these other people who are not recommending us?” These are really important questions…it’s just that this is only a portion of what is going on.

Relying on Wild Grown Brand Personalities

In practice, we know that the positive or negative impact on a brand recommendation or complaint depends on the perception of the person giving endorsement. The reason we know this is because it is seen over and over again in segmentation studies.

A brand will realize that it has grown to a point of having multiple sets of customers defined by different needs; and in the course of developing a segmentation strategy it will stumble across the group of people that influence the people around them. These are the ones that are typically labeled some version of “aspirationals”.

It’s not bad to be in this position but when these people are simply “found” it means that there was no strategy to grow this particular segment. This is a very valuable customer group to simply hope will spring up in your back yard. If you want influencers you need to attract them to your brand which means understanding more about how your customers’ personalities impact your brand image.

Growing Organic Brand Personality

Different brands attract different people. Typically successful brands go out of their way to attract different and specific people. These people relate to the brand and continue to be loyal customers as long as the brand reflects whatever it was that brought them to the store originally. That’s wonderful.

These people are also the true and organic definition of your brand. More than any messaging the marketing team can put out into the universe, these customers walk around wearing, using, and either recommending or criticizing whatever it is your brand stands for.

The question now is which of these customers are other people paying attention to?

In a recent study, we analyzed the personality profiles of customers between different brands to try and determine if brand personality was actually impacting the consumer personalities attracted to the brand. Additionally, we wanted to know how customer personality impacted the brand.

We have been able to show that brand personalities draw in specific customer personalities that do influence the brand image.

There is sample of our findings embedded below but I want to share one of the examples:

What you see in the image below is a comparison of the iPhone and Android brands completely defined by the personality profiles of their customers.

Take a look at the sample below for more information and things like sample size and methodology.