I’m sure you’ll agree:
At first sight, user research seems like child’s play.
You just need to ask users a couple of questions,or observe how they use the product…and BAM! As if by magic, you know exactly how to knock their socks off next.
It’s just ….
… it never works like that, does it?
You end up with only basic insight about your product’s features. Receive conflicting design feedback. And get confused about what your users really want.
If you wonder why, keep on reading.
In this post I’ll show you the 5 most common reasons your research might be flawed.
Reason #1. You Ask Users What They Want
Asking users what features they prefer or need seems like the best way to gain customer insight, doesn’t it? After all, how else could you find out about it other than by asking directly?
In fact, this lack of insight is one of the key traits of our human behavior.
According to Emily Pronin’s “Introspection Illusion” theory, for instance:
“[…] people wrongly think they have direct insight into the origins of their mental states, while treating others’ introspections as unreliable. In certain situations, this illusion leads people to make confident but false explanations of their own behavior or inaccurate predictions of their future mental states.” (source)
A theory of “casual theories” confirms that “there are very few genetically driven causes for behavior for humans in general and none for individual traits”. It states that a person typically doesn’t notice the real reasons for their behavior. We are however good at trying to provide explanations.
And lastly, in their 1977 paper, “Telling More that We Can Know” (PDF version) Richard E. Nisbett and Timothy DeCamp Wilson described a series of studies on cognitive processes and confirmed that:
“When people attempt to report on their cognitive processes, that is on the processes mediating the effect of a stimulus on a response, they do not do so on the basis of any true introspection. Instead, their reports are based on priori, implicit casual theories or judgments about the extent to which a particular stimulus is a plausible cause for a given response.”
In other words:
We humans have absolutely no understanding of our mental processes. And thus, there’s no point in asking us about what we want.
We simply don’t know it.
O.K. Pawel, I get your point. But how else can I gain customer insight since my users can’t reveal it themselves?
One word answer: Observe.
Listen to what users say, then dig deeper into their motives, ask them questions to slowly uncover the truth hidden in between the lines.
Reason #2. Your Research is Biased
If there’s one thing you fear as a researcher, it’s hearing what you really don’t want to know about.
After all, who wants the research to confirm that users don’t need a particular feature you’ve spent months and months developing? Or that the layout you feel so proud of is actually unusable?
And so, it’s tempting to construct the research to direct participants to give answers that obstruct the truth.
Look, it happens in user research all the time. Just consider these examples:
You Ask Leading Questions
Leading questions suggest a particular answer, or imply what kind of information you expect to confirm. For example:
- “What do you dislike about feature X?” –prompts the subject to think negatively about a particular feature and thus include only negatives in their answer.
- “How good is the feature X” – prods subject to respond only in relation to the feature’s usefulness.
A non leading question on the same topic would be asked this way:
- “What do you think about feature X?” – doesn’t suggest anything about the feature or prompt the subject to think about it in any particular way.
You Suggest the Answer
Including your point of view in a question leaves a user with two options:
They can either agree with you or face having to defend their ideas.
For instance, I once saw a website quality survey featuring this question:
“We love our site. Do you?”
The survey allowed me to reply either yes or no (and only the latter included a text field where I could explain my choice).
Now think about it:
My involvement with the site was minimal. I landed on it to find information or satisfy a particular need, but my attitude towards it was neutral at best.
Forced to respond to a survey, I preferred to agree rather than face having to explain why I disagreed with the company. And I’m sure I wasn’t the only one leaving the company with completely useless feedback.
You Preselect the Answer
We humans are awful at decision making.
To most of us, the process of making a decision is often too much effort, and for that reason, we often skip doing it and go with the default option presented.
Here’s a great TEDtalk by Dan Ariely where he explains this behavior:[ted id=548]
Pre selecting or suggesting ananswer in any way skims the research. Simply because many users will naturally go for the default option… if only to skip that torturous task of making a decision.
Reason #3. You’re Using the Wrong Medium
You launch your favorite app, one you just can’t live without… only to be greeted with a survey request. You shrug at it but since you use the app so much, you decide to help. But what seemed like a simple survey turns out to be pages upon pages of questions…leaving you stuck and frustrated instead of working on your stuff.
Sounds familiar, right?
And tell me, at what point do you start picking responses at random to get to the end quicker?
For instance, users are more prone to filling in a quick survey while using an app, but they’d be more likely to provide long and in-depth answers outside of it.
Last year, Christian Rohrer described a 3-dimensional framework for identifying when to use different user research methods.
The 3 dimensions include:
- Attitudinal vs. Behavioral
- Qualitative vs. Quantitative
- Context of Use
And include the following contextual methods to use:
- Natural use of product,
- Scripted (lab-based) use of product,
- De-contextualization (not using the product), and
- A hybrid of them all.
Check out his thorough explanation for the framework, it’ll help you decide when to use different research methods.
Reason #4. You Reuse Old Ideas in New Research
I agree, starting every research project from scratch could be a pain.
After all, who wants to write the objectives again, then follow up with hypotheses and devise research methods…Every…single…time.
It’s much easier to just tweak old research and make it relevant to a new problem, right?
But here is the catch:
The purpose of user research is to learn; to find out what works, what to iterate on, and then, move to a new problem.
Reusing old ideas isn’t conducive to learning, at least not anything new.
Reason #5. You Don’t Test the Research Method
Has this ever happened to you?
You launch a new research project, put up a survey in the app for all users to see and… a day later discover that the script is broken…or that your question confused most users…
You received no feedback, or whatever you got was unusable.
And all that for just one simple reason – you didn’t run a pilot test to assess if your survey works. A pilot test can help establish whether your research will generate viable feedback.
Whenever you launch a new research project, send it to a single participant first. If they get confused or can’t complete it for any other reason, revise it. Make sure that users can understand your tasks or questions and that you receive their feedback. hen, rinse and repeat the test until you’re 100% sure your research is going to produce meaningful results.