Probabilistic Thinking in Problem Statements

Product management is fundamentally a discipline of decision-making. Which investments to make, which problems to solve, which customers to serve, etc. The approach we take to decisions is fraught with peril, and we benefit from removing unconscious biases – improving our ability to elegantly make decisions to improve and advance our products.

Thinking in Absolutes

n the previous article about developing economic framing through the impact section of a problem statement, I introduced the following example problem statement.

The Problem of… We cannot market to the 20% of repeat customers abandon the new product registration process
Affects Whom… 200,000 existing customers making additional purchases each year
The Impact of Which is… We lose $400K – $500K in incremental attached-services sales per year
The Benefits of a Solution are… We increase revenue by at least $400K per year in incremental attached-services sales

In order to keep the topic focused on articulating economic outcomes, I avoided the topic of discussing how to describe the impact. Usually, people make the mistake of estimating a number instead of a range. Commonly, the impact section would read as follows:

The Impact of Which is… We lose $450K in incremental attached-services sales per year

Estimation is necessary, but estimating with absolute number has consequences which affect both your current thinking and the pursuant collaboration. The number generates a false sense of precision and misleads everyone who uses the problem statement to inform future decisions. Product management, and problem framing in general, happens in an environment of uncertainty. The only thing certain about creating an absolute estimate is that you can be certain that your estimate is wrong.

A False Sense of Certainty

When people read a single number, they read it with an unwarranted sense of certainty. The single number does not provide enough information and introduces ambiguity in pursuit of clarity. In this example – is the impact expected to be $450K +/- $5K? +/- $50K, or +/- $500K? The individual value does not shape the conversation correctly – making it impossible to prioritize effectively.

Even intellectually appreciating that the single number is meant to represents a range of possible values, it cannot convey enough information about that range. Is the single value meant to be a maximum impact? A minimum? The mean, median, or mode impact of all the expected values? Without understanding something more the spread of possible values, you will struggle to make good prioritization decisions.

Consider two problems you are considering solving – where they have identical estimated costs and estimated opportunity costs, everything is effectively same, the only difference is in the estimated impact.

Impact A: $450K
Impact B: $500K

With the limited information presented, you would choose to invest to solve problem B – because you expect your investment to be more impactful.

In 2010 when Dr. Jose Briones introduced me to the concept of probabilistic decision analysis in the context of estimation and forecasting, a seed of an idea was planted which would lie dormant until I combined it with the work of Doug Hubbard while I was helping a mutual client. Doug wrote How to Measure Anything, where he really leans into the science of decision-making, the value of information, and again – using probability distributions to inform decision making in an uncertain environment. Those two moments, for me, established a more elegant view of estimation than the underpowered “expected value.”

Learning how to quantify your current uncertainty about any unknown quantity is an important step in determining how to measure something in a way that is relevant to your needs.

Doug Hubbard, How to Measure Anything

When you compare two ranges – you may make a different decision between them than you would when comparing two discrete values. This is subtle, but very important. You already know that the single value actually represents a range – but viewing the range as if it were a value will bias you towards a choice which eliminates your ability to choose.

The essence of shaping your product comes from choosing which problems to solve, which problems to solve first, and the degree to which you will aspire to solve those problems. You want to eliminate the biases which affect your ability to decide.

When comparing ranges, you want to have a sense for the shape of the distribution – is it uniform (all values are equally likely), or skewed to the left of right (very high or very low values within the range are more likely than the opposite), or a distribution like the normal distribution (values in the middle are more likely than values at the extremes). When your decision-making is nuanced enough to require you to know the shape, but you don’t know the shape yet, research or experiment to better improve your beliefs. In practice, I almost never see product-teams doing this.

A simplified approach is to use the same thinking which “won the battle” in the science of estimating work-effort. Coopt the simplicity of a PERT analysis. Ideally, like a true PERT estimate, you will identify upper and lower bounds for your range using the following mental model: You believe there is no more than a 5% chance the actual number is lower than your lower bound, and no more than a 5% chance the actual number is above your upper bound. You are expressing that you believe the likelihood of the actual value being between your upper and lower numbers is 90%.

Consider how you make approach the decision when presented with “the same information” using a value versus a range.

Impact A: $450K
Impact B: $500K

The Impact of Which is… We lose $450K in incremental attached-services sales per year
The Impact of Which is… We lose $500K in incremental attached-services sales per year

When all else is equal, the clear answer is to try and make things better for Impact B.

Impact A: Between $350K and $550K
Impact B: Between $495K and $505K

The Impact of Which is… We lose between $350K and $550K in incremental attached-services sales per year
The Impact of Which is… We lose between $495K and $505K in incremental attached-services sales per year

Now, with ranges, you are presented with a different choice – one which at a minimum allows you to incorporate the characteristics of your risk tolerance into your decision. Do you pursue the problem which is assured to have the highest possible minimum impact (Impact B), or the one which has the highest possible maximum impact (Impact A)? In game theory, these are known as maximin and maximax strategies. John von Neumann referred to these as the conservative and aggressive strategies when he described them almost a hundred years ago (1928).

You may still choose option B, of course – but now you are making a different decision entirely. Much more elegant than “big number better.”

The weird thing about decision-making – even if you make the same choice (Impact B in both cases), the second decision is actually a better, higher quality, decision than the first one.

Conclusion

You undermine your ability to make good prioritization decisions when you compare estimated values and ignore that those estimates really represent ranges. We operate in an uncertain environment – not only are many things rapidly changing, but our degree of understanding of those changing things is changing. Articulating your estimates as ranges will help you make better, and less-biased decisions.