In the art and science of Product Management, there is a constant struggle to make sense of how the user is interacting with the product and how the product is interacting with the business. The struggle is in tying these together without getting flooded in unused or difficult to track metrics and feedback. This article will examine Google’s HEART metrics (Happiness, Engagement, Adoption, Retention, Task Success) and their Goals – Signals – Metrics process, which includes effective and practical tools for making product decisions.
UX Metrics Are Product Management Metrics
User experience (or “UX”) encompasses many aspects of how a user experiences a product – from how they feel emotionally about the interactions to how they judge the value, utility, and ease of use of the product experience. UX is reflected in all factors in the success of a product, including adoption, engagement, retention, and monetization. User experience is also both subjective and dynamic, which makes it difficult to measure, particularly for metric-driven Product Managers.
Product Managers balance stakeholder needs and product success, and therefore have to balance business (and often vanity) metrics with user-centered metrics. UserVoice’s Heather McCloskey points out these different approaches in her article Focus On the Metrics The Matter: Identifying Your Product’s Key Metrics and KPIs.Traditionally, Product Managers have zoomed in on specific data points like“PULSE” metrics (Page views, Uptime, Latency, Seven-day active users, Earnings), or have taken a sales view of the business, focusing on conversion rates or customer acquisition costs. These metrics may be important, but are they really the metrics that should change your product decision-making? How do you know? These metrics only indirectly assess a critical factor for the success of the product — the user experience.
Product Managers struggle to find clear and decision-critical metrics for making product UX decisions and tend to view UX as overly qualitative and difficult to measure. I have written about the complexity of Product Managers working with UX. Thankfully, the team at Google have developed the HEART framework and Goals-Signals-Metrics process to tackle this problem with a clear methodology for planning and tracking product user experience metrics.
Google’s HEART UX Metrics and the Goals-Signals-Metrics Process
In 2010, the team of Kerry Rodden, Hilary Hutchinson, and Xin Fu at Google published a research paper on Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications. In the paper, they outlined the now-popular HEART framework for assessing user experience in products.
The Goals-Signals-Metrics Process
Just as importantly, the team at Google outlined the Goals-Signals-Metrics process for selecting practically measurable metrics that actually tie back to product goals aligned on by the team. This process is powerful, since it can help teams plan out any product assessment. The Google team explained why they developed the Goals-Signals-Metrics process:
“No matter how user-centered a metric is, it is unlikely to be useful in practice unless it explicitly relates to a goal, and can be used to track progress towards that goal.”
By articulating the team’s product goals, identifying the appropriate signals for success or failure, then measuring the relevant metrics from those signals, Product teams can consistently and effectively assess and make decisions on UX.
Before considering metrics, the team should align on product or feature goals. Frameworks like HEART, below, help prompt the team on which areas they should focus on when selecting goals.
Discussing and aligning on goals for a project before diving into signals and metrics is important to build buy-in and to avoid irrelevant metric-tracking or debate down the line. As a Product Manager, I make a point to align on project goals at the beginning of any product discussion. Different team members will often disagree on what the goals should be, but if you do not align on goals first, you will end up with a 10-times longer list of irrelevant metrics the team will want to track. This leads to unclear decisions and endless debate.
Once a goal is set, think about how the failure or success of that goal might bubble up directly or indirectly in user behaviors or attitudes. First, use user-centered language to describe what attitude or behavior would signal success or failure. Try to use signals that are sensitive to the goal. For example,i.e. the signal should only change if the goal is successfully met or not, and not because of some other interfering factor.
This is the time to consider both quantitative and qualitative data sources to track these signals. User actions, such as a successful purchase, could be tracked in activity logs or behavior analytics software. User attitudes, such as trust, delight, or perceived ease-of-use in an interaction may need to be proactively measured through user surveys or through analyzing customer support requests. Do not worry about the specific metrics at this stage, but focus on the signals and their possible qualitative and quantitative data sources.
Finally, the chosen signals should be converted into specific metrics.
Techniques for effective metrics selection:
- Cohort analyses: Instead of “Increase weekly average posts by 10,” try, “The cohort of users who sign up the week of March 1st should post 50% more in their first week than the cohort for the week of February 1st.” (Check out UX Magazine’s article on “Using Cohort Analysis to Optimize Customer Experience”)
- Relative % change: Instead of “Increase monthly page views by 100K,” try “Increase monthly page views by 10%.”
- Averages per user instead of totals: Instead of “Increase by 1MM posts,” try “Increase by 10 posts per user.”
- Unique users completing actions instead of totals: Instead of “Increase monthly projects by 30,” consider “Increase unique users posting a project by 10.”
Make sure your data is valid: The biggest issue after selecting the right metric is to ensure the data is valid. User logs, event tracking, and survey results could all be skewed by several factors. Web traffic can come from bots, scrapers, or irrelevant sources (e.g. countries you do not serve). Event tracking code could be misfiring or not tracking at all. Survey questions could be biased. Spend the time to ensure data collection is working or you will spend a lot more time collecting the data from scratch.
[cta id=”2003897″ align=”none”]
Google’s HEART Framework
The HEART framework includes 5 metrics: Happiness, Engagement, Adoption, Retention, and Task Success. Every product, feature, and user is different, so a PM should include only the relevant parts of the framework for the project at hand.
The happiness metric is probably most foreign to a data-minded PM, as it covers a user’s qualitative attitudes to your product, interaction, or feature. These qualitative attitudes can be turned into measurable metrics. Often, happiness can be reflected in passively observing user actions, such as the uncued sharing of the product with friends or repeated interactions without extrinsic rewards. Actively collected feedback can include customer satisfaction surveys, perceived ease of use, or visual appeal for your feature (a certain in-app customer feedback tool might do the trick), or asking how likely a user would be to recommend your product to calculate your net promoter score (NPS). It can also be collected by implementing a User Feedback Solution (hint: you might want to check out UserVoice’s Feedback Solution).
Nir Eyal’s suggests in his book Hooked (one of my recommended 15 Books for Product Managers to Learn UX) that variable rewards can build engagement habits. If we were Tinder assessing the swipeable cards feature, the happiness goal of swipable cards might be to provide Tinder users with an enjoyable swiping experience, the signal might be users continuing swiping with no other extrinsic feedback or “good contenders” expressed with a lack of right swipes/likes. The metric could then be average swipes per match or per right swipe, or % of users without a match who return for another session. Tinder would see from users’ continual swiping of cards with potential matches, even in the absence of any actual matches or other extrinsic benefits, that the users are “happy” with the experience, partly because of the unpredictability of what is in the next card or if a match will occur. A proper follow-up usability interview where one observes users swiping on Tinder would likely reveal either the enjoyment or frustration of the user.
As you can see, starting with a goal can lead you to the one or two metrics you need, as opposed to a laundry list of metrics or feedback to collect.
Starting with a goal can lead you to the one or two metrics you actually need.Tweet This
Engagement measures the frequency, intensity, or depth of a user’s involvement with your product. If you were the Gmail team at Google, you might just take the PULSE engagement metric of “seven-day active users” (how many users visited the product in the last 7 days). If you start by first discussing the engagement goal, which might be for users to use Gmail as their main communication platform in their daily routine, then the signal could be the number of times users check their Gmail in the week. A metric that uses the signal to identify if the goal of “Gmail as a daily routine” might be “% of active users who visited the product on five or more days during the last week,” which is precisely the metric that the Gmail team used. For the Tinder swiping feature example, the goal could be related to habits, while the signal could be swipes or time in the app, and the metric would be # of swipes or # of minutes per app session or per day.
Adoption & Retention
Adoption is fairly straightforward, typically related to the number of users who use a feature or product for the first time. Retention measures the continued repeat engagement of those users with the feature or product over time.
If we take the Tinder example again, happiness was measured by swiping regardless of benefits. Adoption might be more straightforward, such as % of users who signed up who swiped at least one card (Note: I suggested % conversion instead of # of users who swipe a card, because the latter would scale with user acquisition, not with UX improvement).
Goal-setting for Retention would be more involved, as it could be measured in % of users who swipe after 5 hours, 5 days, or 5 weeks, and these would also depend on the goals of the product team at different stages of growth. Tinder is usually used while dating, so the team would assume that users could be retained for a few months at a time depending on the culture and dating habits of the region. There is also “re-engagement” which is retention after drop-off, and that could be because a user found a partner, then later became single again, and decided to return to the app or not. This is something the Tinder team would want to monitor as well, for the health of it’s platform. This is all important discussion before discussing final metrics.
This final metric in the HEART framework, task success, relates to the efficiency, effectiveness, and error rate of a user completing a task with your product’s workflow. A task is more efficient if it takes a smaller amount of time to complete it effectively. It is effective if it either completes the task with greater quality or a greater hit-rate. For example, if users use Google Maps to search for locations, effectiveness could be both the % of successful searches and the accuracy of the exact pin location for the results. A task’s error rate is the inverse, and is typically the frequency that the task is unsuccessful, for example, the % of map searches that produce incorrect place results, the correct result with the wrong pin location on the map, or simply that produce an error due to a technical bug.
Task goals are very important, as the options often vary widely in scope and type, and will determine how you define your signals and metrics. For Google Maps, the goal could be finding users the right location, or it could be as broad as getting users to the right location. The former requires successful search results, while the latter requires accurate search results, directions, and metrics around successful arrivals.
To get the hang of this, pick a product or product feature that you think you understand fairly well and that is not overly complicated. Use the table template below to determine your goals, signals, and metrics. Consider which parts of the HEART framework would be relevant to your UX decisions.
Once you are comfortable with the framework, try it with one of your own products and share the table with 2 or 3 members of your team or discuss it on a whiteboard in a meeting, asking them to fill in what they believe is relevant. You will quickly see how quickly the team diverges on different aspects of the table.
After your team has defined your goals, signals, and metrics, you may want a framework for actually producing product UX optimizations to meet your goals. Cleverism’s article on The Art and Science of User Experience (UX) Optimization suggests some approaches to optimizing UX to drive your product metrics, and is a good place to start, along with my UX book list for Product Managers.
Google’s HEART framework is a useful tool that helps frame discussions for product teams around the otherwise hard-to-quantify realm of UX decisions. The Goals-Signals-Metrics process is an even broader-reaching tool that can be used in any decision-making endeavor. These tools have been used and tested by Google for several years across many projects, and now are being adopted by other product teams. If you have examples or comments you would like to share about the HEART framework or other metric-finding methodologies, please feel free to leave a comment below.