Data Driven Decisions vs. Product Sense
How do you make decisions?
“We are a data-driven company” is something many pride themselves on. This is why PMs often feel as if they must decide between data-driven decision-making and applying intuitive product sense, which is the core of many strategic discussions. Today's newsletter dives into these contrasting approaches to understand how the most successful product managers use both.
Understanding the Spectrum
At one end of the spectrum, quantitative analysis or data-driven decision-making relies on empirical evidence to guide product decisions. This evidence can be quantitative—like user analytics, conversion rates, and A/B testing results. This approach reduces risks and ensures that decisions are anchored in reality, rather than personal biases.
On the other end, product sense, or intuition, is built from years of experience and a deep understanding of the market the company operates in as well as it’s users. It is the ability to anticipate needs that data might not yet reveal. This gut feeling often guides decisions in scenarios where data is incomplete or absent.
In the middle of this spectrum, qualitative analysis is bridging empirical evidence with intuition. It involves gathering insights through user interviews, surveys, and open-ended feedback to uncover underlying motivations, behaviors, and pain points that quantitative data alone cannot capture. This enables you to provide a nuanced perspective that complements both data-driven strategies and intuitive judgment.
Quantitative Analysis
Data is democratic. It does not favor intuition or bias (Attention: One can still be biased in interpreting data). It represents the voice of the user at scale. For instance, Whenever we are looking at usage statistics or results from an A/B test, it is done via quantitative analysis. Facts in the end beat opinions if they can be backed with data.
When to consider using quantitative analysis to guide your decision:
Optimizing existing features: Data can show how users interact with your product, helping identify which features are hits and which are misses.
Measuring Impact: Analysing how users behave before and after a change allows you to assess their impact.
The issue with working data-driven is that data is not always readily available. There are many scenarios in which the data is either “not enough” or “not representative”. This is where quantitative analysis falls short and one should resort to other ways of guiding their decision.
Qualitative Analysis
Quantitative data provides a broad view, but qualitative analysis offers a closer, more nuanced perspective. By understanding the 'why' behind user behavior, qualitative insights enrich data-driven strategies, revealing motivations and emotions. For example, user interviews and usability testing provide direct feedback from users without the need to build and deploy a product to gather data over long periods.
When to consider qualitative analysis to guide your decision:
Direct conversations with users can uncover information quickly
Qualitative insights provide the context needed to understand the motivations behind the numbers.
The challenge with qualitative analysis is that it often requires significant time and resources to gather and interpret data. Moreover, findings can be subjective, demanding careful consideration to ensure they're representative and not influenced by personal biases.
Product Sense
While data tells you what is happening and sometimes why, think of product sense as the feeling in your stomach that something should or could happen. But are people born with product sense, is it a talent some have and some don’t? Short answer: No.
Product sense is the result of working in product or industry for a long time, consciously focusing on developing it. It is a combination of different aspects one can know about the industry and customer they serve with their product.
When to consider using product sense to guide your decision:
A product is very early in its life and data is not yet available
When creating something entirely new, you won’t always have the possibility to get qualitative or quantitative insights from your customers
Your unique experience and insights on your market or user allow you to innovate beyond what is currently possible/available.
The issue with product sense is that applying it requires one to have it. Especially more junior product managers will struggle with making good intuition-based decisions as they may not yet have strong knowledge of the market, customers, or competition. In addition, a decision based purely on product sense is often a risky bet that needs careful consideration
When to Lean on Data vs. Intuition:
Use data-driven strategies (quantitative or qualitative) when you have enough information to form reliable predictions.
Trust product sense when venturing into uncharted territories or when data contradicts your core understanding of your target users. (Yet you might just be biased, so triple-check why the data is telling a different story)
Conclusion
The most important thing you should not forget is, that most of the time the world is not black or white. You don’t have to be in camp data or camp intuition. Mixing multiple sources to find the best way forward is a good strategy in most cases.
VERSUS Series
The VERSUS series is a set of 8 newsletter articles (2 per week) that answer questions PMs ask themselves all the time. Check out the other articles below!