“I want to improve the user experience of my product. Where do I start?”
This is a question we hear (and ask ourselves!) often, but there doesn’t seem to be consensus on how to go about it. Sure, there are lots of methods for improving UX – and more than enough advice on how to use those methods. The web is littered with guides on usability testing, contextual interviews, web analytics, etc. but each of those methods have different outcomes, and I don’t often see the guides answer the most important question when it comes to improving UX: “Where do I start?”
This article is my attempt to answer that question.
How to Improve User Experience with Data
The first thing to clear up is this: in order to know where to start to improve the user experience of your product, you’re going to need more than a single data source. No single method can tell you the whole story you need to make good decisions. Usability testing will show you where there are interaction problems, but it won’t show you the magnitude of those problems. Web analytics will show you lots of numbers, but it can’t tell you how to fix issues you encounter. Even worse, it’s not always clear how to interpret analytics in isolation, so you might not even be able to tell if some things really are issues in the first place.
There is hope, though. It’s in the combination of these two methods that we find the elusive answer to our question. Combining usability testing with web analytics gives us the insight we need to make the right UX decisions about our products.
There are two approaches for using usability testing and web analytics to figure out where to start improving user experience, depending on the situation you find yourself in. In the first approach you start with usability testing first, and then look at web analytics. In the second approach you flip the order around. Let’s look at each approach in turn.
Approach 1: Usability Testing First, Web Analytics Second
One of the major problems with the Big Data movement is that it can be really difficult to figure out where to look to find the best insights. If every possible data point is being recorded, finding the useful data in a sea of numbers is not easy. You might find yourself in a situation where analytics guide you to a usability issue that looks important, but is actually dwarfed by a much larger problem that you didn’t see, simply because you didn’t know where to look.
So if you’re in a situation where every page and every event is tagged and recorded into your data warehouse or analytics software, doing usability testing first will give you the direction you need to focus your web analytics efforts.
In this scenario, the focus of usability testing is as broad as possible. Tasks are fairly generic: ask participants to go through the process of purchasing an item they’re interested in, sign up for a service and go through onboarding, etc. The goal of usability testing is twofold in this scenario:
- To get an initial sense for the biggest usability issues in the experience.
- To find out enough about the problems to serve as a starting point for solutions once prioritization has happened.
An initial round of 5 usability tests will give you enough information to go to your web analytics tool and find out how big each of the problems are. Did a few participants struggle with a credit card field? Use analytics to find out what the actual drop-off rate on that field is. Did some participants struggle with the photo upload flow? Use analytics to see how often people start uploading a photo and then abandon the flow.
As an example of this, I once did a round of usability testing for a mobile classifieds site. We saw during the testing that some users added a $ sign to the price (instead of just a number), and that it would result in a price of $0 on the listing. This isn’t something the product team was aware of, so they went to their analytics to find out the magnitude of the problem. And of course, they found that a massive amount of users fail to enter just a number, and that the validation on the field was broken, so it resulted in a $0 listing. This was an easy bug to fix, but usability testing guided the way to where we needed to look.
By starting with a round of usability testing, and then moving to web analytics, you’ll be able to figure out which of the problems you uncovered are the biggest problems to solve – without having to go through days and weeks of data, looking for patterns that might not even exist. The added bonus is that the qualitative nature of usability testing will give you a good starting point for why users get stuck, which will help inform the solution.
Approach 2: Web Analytics First, Usability Testing Second
If you have good, interaction-focused reporting set up for your product, starting with some of those reports can help you figure out where to focus your UX efforts – and reduce some of the ambiguity that often exists in data.
Let’s say that in a recent redesign of your iPhone app, you noticed that time spent in the app reduced significantly. Is this a good thing or a bad thing? Does less time spent mean people give up faster, or that they are simply accomplishing their goals much faster?
Here’s another scenario: let’s say you release a new feature and your analytics tells you that very few people are using the feature. Is that because users can’t find the feature, or because they don’t find it useful? The answer to this question means the difference between moving a feature and killing it, so you better get it right!
These types of questions are hard to answer from analytics alone. But they’re questions that usability testing are great at answering.
In this approach, compile a list of biggest (and most ambiguous) issues you see in web analytics. Follow this up with usability testing sessions that have a much narrower focus than the previous approach. Tasks are shorter, and presented to participants in rapid succession. For example, ask participants:
- How would you upload a photo to this group?
- How would you invite a new team member to this IM conversation?
- How would you find this feature?
This approach accomplishes a few important goals:
- Determine whether or not specific data points represent a real user experience issue or not.
- Collect insights on how to solve the issues that are worth fixing.
As an example of this, at a previous company we noticed, from our analytics, that most users failed to upload a photo of their listings when they tried to sell something. We wanted to understand the impact of this, so we ran two rounds of usability testing. One study focused on buyers, and it showed us how buyers simply skip listings without a photo. And usability testing on sellers helped us uncover why users weren’t uploading photos. Armed with these data points we were able to solve the problem and provide a major win for the business.
Starting your process with web analytics gives you a head start on prioritization, and the follow-up usability testing will weed out the false positives, while at the same time help inform possible solutions to the real issue.
The TL;DR: Start in the Right Place With the Right Data
The main point I’m hoping to communicate is this: it’s really dangerous to use a single data source to make decisions about user experience improvements. With qualitative methods (like usability testing) alone you might spend a lot of time solving a problem that a negligible number of users have issues with. With quantitative methods (like web analytics) alone you might “solve” a problem that isn’t even a problem at all.
There are huge blind spots in data, and using more than one source drastically improves the chances of avoiding those blind spots. And that’s the reason for the formula that makes the title of this post. Want to know how to improve user experience? Find out what users are doing, as well as why they’re doing those things. It’s the only way to avoid spending time and money on the wrong efforts.