Stop tracking dashboard metrics and go talk to real users

Your product team should prioritize collecting qualitative data (like customer interviews and user testing sessions) to inform and improve your user experience. In a land of uncertainty about where your product should go, qualitative data is the guiding light for your product, and it allows your team to focus on creating features users need.

‍

Unfortunately, when building new products or features, teams over-emphasize quantitative data (like surveys and app analytics) for decision-making. This approach often results in feature-laden dashboards but a misguided product.

‍

“People who are really excited about being data-driven, to me that is oftentimes a red flag for their product thinking, especially if it's an executive who's saying things like, oh, so we make all our decisions with our dashboards. To me that says that the team is over-emphasizing quantitative data at the expense of qualitative data and they're not using good judgment.”
– Maggie Crowley

‍

If qualitative research is prioritized (the way it should be):

• Your team will understand why users are struggling with your product
• Your team can collect data much earlier in your product’s lifecycle.
• Your team will have more confidence they are making the right decisions.

‍

‍

Qualitative data tells you why your users struggle.

Product teams develop empathy and understand why users struggle with the product through qualitative testing. Each conversation with a user reveals pain points the product team might have overlooked. Not only that, but the user can share how they expected the product to function.

‍

Quantitative data doesn’t give the team that luxury. With data collected from surveys, the team must infer why the user struggled in a specific area. In order to confirm they solved the UX problem, the dev team has to build and launch the adjusted feature. 

‍

You might be thinking, “It’s too difficult to talk with 100+ users to uncover the root of the issue. I’d be better off guessing based on survey data and app analytics.” That might be true if you had to speak with 100+ users, but you don’t. Maggie Crowley nails it when she says, 

‍

“[with] most PMs, most jobs, most products, you're going to be better off talking to 10 users and you'll get more and better insights out of why things are happening than you would with any dashboard.”
– Maggie Crowley

‍

‍

Qualitative data can be collected much earlier in the product lifecycle

Your product can be validated by users before writing a single line of code. Using design prototypes, user researchers can uncover critical UX problems by testing with your target audience. By validating your product in its earliest stages, you can minimize tech debt, save engineering hours, and alter the project’s trajectory to cater to your users. 

‍

On the flip side, teams that rely on quantitative data have to wait for feedback. Often, there is a 2-3 week lag before the sample size is large enough to make decisions. These precious weeks really start to add up if your product team wants to validate multiple iterations of a product or feature.

‍

‍

Your team will have more confidence they are making the right decisions.

Nothing brings clarity to a product team like hearing customers say, “Yes, this makes sense,” or “I have no idea what to do next”. Whether the feedback is good news or not, teams can optimize for solving the user’s problem – the confidence that comes from talking with users allows teams to move faster. Product teams are able to point to user feedback as the north star and align in that direction.

‍

Quantitative data sometimes requires so much interpretation that the entire team doesn’t agree on the next step. 

‍

A cautionary tale: be wary of letting 1 or 2 users direct your product. This can lead your team down to niche problems that very few people need to be solved.

‍

‍

Don’t rely only on qualitative research

Diversifying your research will often lead to much better results. Maggie hits the nail on the head when she says, 

“I am not saying that you don't need data. You absolutely do need to instrument your products and understand if it's working at scale, but you can't forget that, you need to know why. If you don't understand why it's happening, I don't think you can come up with good insights about what you should do next.”
– Maggie Crowley

‍

While product teams should find a balance between qualitative and quantitative research, that doesn’t mean the two are equal.

Your team should prioritize qualitative research, especially in the earlier stages of your product.

‍

Not convinced qualitative research is more impactful?

🧪 Conduct one user testing session and see for yourself. 

⭐ Need some guidance for your first testing session? → check out our free 6-step guide

‍