4 Data Pitfalls Product Managers Need to Watch Out For

Intuition, hunches, and gut feels have no place in the modern product manager’s data-savvy toolbox… right?

‍There’s no arguing that decision makers now have more information than ever to help them make those decisions, but that’s not always a good thing. Here are the four downsides of data that you’re likely to encounter as you try and actually bring a great product or feature to market. We've also got some expert advice on how to deal with them.

Getting Past the Gatekeepers

These days, if someone comes to a meeting and expresses an opinion without a piece of data to back up their case, it's hard to take their argument seriously. So the diligent product manager knows they must provide supporting evidence when fighting for resources, prioritization, or strategy shifts. Hopefully, a well-argued case with a few data points to hammer things home is enough to get a go/no-go from the powers that be. And just like in sales, a quick no can be as good as a yes when you’re a product manager trying to keep your team on target. Unfortunately, there’s a third response you can receive that means more work and more time spent on the issue you were all set to put to bed -“I’d like to see more data.” This dreaded phrase means you’ve got to dig back into the issue. You can try to uncover new nuggets from your internal metrics or find new third-party sources. A worse scenario? Having to go BACK to the customers, users, and prospects you’ve already surveyed to ask them even MORE questions.How can you avoid this fate? There’s two reliable methods for avoiding the not-enough-data indecision scenario:

Previewing—lay out the basics for the stakeholders you’ll be pitching and ask them what information they’ll need to make their eventual decision. Do this BEFORE you start doing your research and digging through the dashboard for data. If you know what metrics matter, you can avoid any stalling when you’re ready to ask for their vote.‍

Clarification—In those cases when you are sent back to the data mines to come up with more evidence, make sure you ask the team exactly what information they’re looking for to make the decision. This ensures you won’t waste time compiling new stats that don’t get them over the decision-making hump.Requesting more data shouldn’t be an acceptable stalling tactic, so providing gatekeepers with exactly what they’ve asked for as quickly as possible can cut short delays and force a decision.

Vanity Metrics

While your CEO might love talking about them to investors and marketing clinks drinks over them when they reach a new peak, there are plenty of metrics that are essentially meaningless when it comes to product decision making. Vanity metrics are fine for press releases and sales pitches, but the metrics that matter actually help you do your job.So how can you avoid letting these less important data points infiltrate your stakeholder meetings? Point out their futility when it comes to impacting the nuts and bolts of the business.It really comes down to goals, and measuring what’s going to help you meet them. The numbers you should care about are ones that can be directly tied back to revenue or adoption.

If you’re an ad-based service, impressions might be the stat that matters, and you can try to determine if each move you might make will increase or decrease total impressions. For a freemium company focused on conversions, data showing Behavior X is more likely to lead to a subscription vs. Behavior Y is probably far more useful than how many people visited your home page.

So before you let a stakeholder—or yourself—get distracted by the shiny metrics getting the headlines, make sure you’re making decisions to move the needles that matter. If you’re not going to change your business or take action based on a metric going up or down, then that metric's not really very important.

Data Knife Fights

Now that everyone has yesterday’s numbers at their fingertips (or via a quick trip to Google Analytics), you’re probably not the only person armed with data. That’s not necessarily a bad thing; transparency and appreciation for a business’s metrics are healthy habits of every employee. The problems start when people start picking favorites.How can this happen? Why wouldn’t everyone in the company be aligned that Metric X is the one true North?Well, we already know about the danger of Vanity Metrics. But beyond that there are two other types of data favoritism that can wreck a productive conversation:

‍Personally-relevant metrics—Even at companies where everyone’s on the same team, we still have individual goals and objectives we’re trying to reach. That’s why customer service cares about how many support tickets are being generated and closed, while operations worries about uptime, and marketing frets about conversions…and of course sales is laser-focused on hitting their quotas. The metrics tying back to these individual areas of concern will always be more important to the affected individuals than everyone else, so be respectful of their concerns while keeping the focus on the big picture numbers driving the company’s overall goals.

‍Argument-specific metrics—In a data-driven world, metrics are ammunition in the battles we wage internally. Everyone knows they can’t get their way without being able to throw out some numbers to make their case, which leads some stakeholders to cherry-pick the stats that support their argument. This can lead to different sides making seemingly reasonable pitches with data evidence that directly conflicts with other data. In the latter case, it’s essential to acknowledge that the other side is presenting a valid case (since they’re using accurate data, too). Next you need to design a quick test or experiment that can prove your data-backed case is stronger. Hopefully a quick A/B test can do the trick. If not, it’s back to the cave to come up with more evidence that strengthens your case; perhaps you can add some qualitative, customer interview-driven color to the numbers.

Analysis Paralysis

Dueling data sets are one cause of analysis paralysis, but decision-making delays don’t always require an ornery adversary or even another human being at all. Sometimes, the bottleneck is simply you and those piles and piles of data points.

‍This is why you need to set some boundaries around your own data-driven decision making. Here are some best practices:

‍Set a time limit—There’s always more data to be found, more cross-sections to be scrutinized, more slicing and dicing you can do. Don’t let yourself get sucked into an endless research exercise and give yourself a predetermined number of hours or days to get it done.

‍Limit the scope—Don’t allow for an unlimited set of options, set up a decision between two, three or four options and prepare the data analysis accordingly. Make it a project you can wrap your arms around that can’t go down too many rabbit holes.Decide what’s important before you begin—Having clear goals means a clear finish line for your analysis. Bound your analysis by the preset objectives and cut down on the noodling.

‍Weigh your sources—Not all data is created equal. The metrics that matter and reputable sources of information should be more important to some random survey you found on Google.

‍Just pick something already—Even if you’re not 100% confident you’ve landed on the perfect answer, sometimes you just need to move forward. If you’re still feeling unsure, include caveats when discussing your ideas with others and build lightweight MVP experiments into the process so you don’t put all your eggs in a basket you’re not so sure of.Remember, data is your friend and a valuable resource, but it can’t do your job for you. You still need to ask the questions, create the context, define the experiments, and balance it all with what you’re seeing and hearing from actual human beings.