When deciding what to measure, we often choose metrics which sound good or metrics which are easy. These mistakes can make a product strategy incoherent, excessively expensive, and ineffective. How we talk about what we choose to do sets our teams up for success. Or failure.
Unpacking Three Ideas
There are two broad categories of measures – economic measures and proxies for economic measures. We operate with proxy variables all the time, sometimes for good reason and sometimes with hidden consequences. Leading indicators are proxy variables which provide rapid feedback when economic measures are delayed. Vanity metrics are the proxy variables which make you feel good, in spite of their questionable connection to economic outcomes.
Three key areas where incorrectly using proxy variables in your problem statements are worth exploring in depth. The measure you use to describe the impact of a problem and the potential benefit of solving it matters. I’ll review each of the three areas, and then dive into the first one in this article, leaving the others for follow-up articles. The right answer is to use a measure of economic benefit to describe the problem. The following dives into the reasons why this matters.
The final two sections in the problem statement template, The Impact of Which is…, and The Benefits of a Solution are… must use the same measure – a common denomination. Even if you cannot shift to an economic measure, the two fields must at least use the same measure. This is important to support the process of deciding how much of the problem you want to address, or how much of the problem you want to address right now. When developing a product strategy, we are not only activating our teams to go “in that direction” but also establishing “how far to go.” When they don’t share a common denomination, the problem statement is incoherent.
**Three issues you introduce when you focus on proxy variables instead of economic value when describing problems:
- You constrain the product strategy shaping activities to be purely based on intuition, in a situation where this approach is expected to be ineffective. By asking leaders to make choices among problems to be solved without a common denomination for comparison you undermine their ability to reason when making the decision.
- You hamper collaboration on achieving outcomes. When teams are disconnected from the development of outcome hypotheses, they are disempowered. By scoping their objectives to achieving a change in a proxy variable, you remove their ability to contribute to assuring their work will be consequential.
- You eliminate your ability to make rational choices about how much to invest to solve a particular problem. It is inappropriate to spend $1,000 to solve a $20 problem; there is a necessary level of business acumen necessary to make these decisions responsibly.
Value Seeking
In a recent webinar on developing hypotheses and measurement, I taught folks how to use the mechanisms of value-realization to define their approach to observation. The theme of my talk was measure what matters and the feedback from attendees was positive. When you’re writing problem statements, you are describing today and you are communicating a vision for how you will find value in a better tomorrow. Here we need to shift to an earlier stage of the process, not talking about how to discern value in what you’ve built, but rather how to think about value before you decide what to build.
When you describe your understanding of value, there is a best practice and there are consequences for doing it other ways.
A major element of prioritization is making comparisons – should you solve one problem ahead of another or instead of another? The problems you are considering solving will all look very different. You’ll have exploitable market opportunities, internal capability shortcomings, product improvements to name a few. Each problem could be described in terms which make sense within those local contexts (grow share of wallet, shorten a cycle time, increase usage of a new feature). Using those terms prevents you from making comparisons and runs the very real risk of misdirecting you towards creating something which doesn’t matter.
The local context of executing on something can be effectively influenced by measuring a vanity metric, and you may never know that it led you astray. The best way to douse a forest fire is to prevent it from starting. The best time to stop wasting time and energy build the wrong things is before you get started, otherwise you’re faced with the challenge of un-ringing the bell which originally placed you on a bad path.
When you start describing problems in terms of proxy variables is where you run into your first issue. Proxy variables exist because they are easy to measure, at the expense of being impossible to compare. By displacing economic measures with proxy variables, you eliminate a shared framework for making decisions.
The lack of an economic framework does not prevent people from making economic decisions. It simply causes these decisions to be driven underground and to be based on intuition instead of analysis.
Donald Reinertsen, The Principles of Product Development Flow
The Apples and Oranges of Proxy Variables
Imagine your product is a mobile app which helps people to reserve parking spots ahead of time near entertainment venues. This allows people to avoid allocating extra time before an event to try and find parking, frees up congestion on the streets near venues, and helps people to better enjoy their experience. You build the app, set up partnerships with parking space providers, and train up people in your call center to field support calls for users of the app / service.
Your team comes to you with an update – things are not going well, and there are three key problems which have been identified. You only have enough budget to address one of them. Or your team only has the available capacity to solve one of them. How do you choose which of the three situations to address?
Situation A: Only 2% of people are still using our app after their first experience.
Situation B: We spend an average of 20 minutes per support call.
Situation C: Our call center employee turnover rate is 40% per year.
These all represent genuine problems worth solving, yet you cannot rationally choose to prioritize one of them ahead of the others. You can only apply intuition. Based on your past experiences, one of those situations may sound worse than the others. If you elaborated each of these situations into problems using the problem statement template, but retaining the proxy variables, it becomes easier to use intuition, but still no easier to make a rational choice about which problem to solve.
The Problem of… 98% of new customers stop using our app because they aren’t able to find and reserve good parking spaces
Affects Whom… Drivers looking for parking solutions
The Impact of Which is… Only 2% of people are still using our app after their second experience
The Benefits of a Solution are… We could increase our total active user population by 10x
OK, great – now you have enough context to inform an intuitive decision – because you don’t actually help with the job to be done, people to drop your product. If you fix the problem, you could increase your user base dramatically. Sounds important and valuable. But you don’t have the information needed to know if you should prefer to solve this problem at the expense of not solving the other problems.
The Problem of… Our customers call us for help when the parking space they reserved is already occupied when they arrive
Affects Whom… Drivers who reserved parking through our app prior to attending an event
The Impact of Which is… We spend an average of 20 minutes per support call.
The Benefits of a Solution are… We could reduce the duration of these calls to under 2 minutes
Again – you have a sense of a real problem, reservations are not being honored and our customers call the support staff. Your customers scramble to find parking while likely risking being late for whatever event they are attending. The value proposition for your customer is in not having to allocate time to find parking before an event, because parking is there waiting for them, making for a hassle-free experience of enjoying a night out. It feels like this sort of problem needs to be addressed.
The Problem of… Our call center agents are unsupported in solving the problems customers ask them to solve
Affects Whom… Call center agents
The Impact of Which is… Our employee turnover rate is 40% per year.
The Benefits of a Solution are… Our call center employee turnover rate drops to 20% per year.
This problem looks inward – where you discover some assumptions about how to deliver a solution have proven to be wrong. When modeling the costs associated with providing good service to your customers, you did not account for the high expense of continuously training new call center employees to become genuinely helpful as well as good brand ambassadors. The cost of training the volume of agents is much higher than it should be for the size of the customer base. It feels like your business model may proved to be unsustainable.
Each of these problems could be critical to address. And each of them “sounds like” or “feels like” a big deal. My friend Paul Young, VP at Pragmatic Institute,
delivered a presentation on the product management “X-Factor” at one of the early Product Camps in Austin. Part of the X-Factor is the ability to be persuasive within your organization. A good product manager could pick any one of the three problems and convince the leaders to invest to solve it. When people describe the responsibilities of a good product manager, they often list the activities – including persuasion. Being good at product management also requires us to develop the skills and practices which enable us to avoid bias in the decisions which define and create our products.
Just because a product manager could convince leaders to invest in any of the three does not mean they should convince them. The product manager is also subject to bias. When asking product folks and leaders to compare apples and oranges like this, you are forcing them to make an intuition-only decision. It is almost impossible to avoid bias in these decisions. In Strategic decisions: When can you trust your gut?, Gary Klein and Daniel Kahneman explore the situations where it can make sense to rely on intuition in decision making. The article is noteworthy in that Klein and Kahneman generally approach decision making from ‘different camps of psychology’, and the article focuses on areas of shared understanding.
First, there needs to be a certain structure to a situation, a certain predictability that allows you to have a basis for the intuition. If a situation is very, very turbulent, we say it has low validity, and there’s no basis for intuition. For example, you shouldn’t trust the judgments of stock brokers picking individual stocks. The second factor is whether decision makers have a chance to get feedback on their judgments, so that they can strengthen them and gain expertise. If those criteria aren’t met, then intuitions aren’t going to be trustworthy.
Gary Klein, Strategic decisions: When can you trust your gut?
This example is definitely not a situation which clears the high validity hurdle. Any given problem a product manager is considering is unlikely to meet Klein’s criteria. The consequence of using intuition exclusively to make the decision is that bias will run unfettered and decision quality will be degraded.
The Solution
The solution is pretty simple, but not necessarily easy. Describe each of the problems in terms of a shared measure of economic benefit.
This is hard, because it requires you to describe your outcome hypotheses. Each of the proxy variables from above is believed to lead to a potential benefit. You should write out the hypotheses associated with problem statement, in order to reframe the problem in terms of economic benefits.
The Problem of… 98% of new customers stop using our app because they aren’t able to find and reserve good parking spaces
Affects Whom… Drivers looking for parking solutions
The Impact of Which is… Only 2% of people are still using our app after their second experience
The Benefits of a Solution are… We could increase our total active user population by 10x
What is implied when the problem statement uses proxy-variables is that building whatever needs to be built to cause 20% of users to continue to use the app, instead of 2% will result in some sort of benefit. What you need to do is describe how much benefit you expect, in order to know if this is the right problem to solve. An outcome hypothesis is the tool to use.
We believe if we increase the new customer retention rate from 2% to 20% it will result in a tenfold increase in billings in the second month. We will know we are right if we see at least a tenfold increase in receivables at the end of the second month. We will know we are wrong if see an increase of less than 1,000%.
The problem statement can be updated.
The Problem of… 98% of new customers stop using our app because they aren’t able to find and reserve good parking spaces
Affects Whom… Drivers looking for parking solutions
The Impact of Which is… 90% of potential revenue from motivated customers is lost
The Benefits of a Solution are… We could increase our revenue by 10x
With almost all of the clients I’ve helped, using revenue as the common denomination of economic value is sufficient. Revenue is not the only choice – the right choice for a given company may not be the right choice for another. You may need to describe economic impact in terms of profits. You may need to describe it in terms of billings. You may get the best insight by looking at problems in terms of per-customer profitability, because that correlates best with enterprise valuation in your unique situation.
The trick is to find something broadly-enough applicable (like revenue or profitability) to apply to both capability improvements and product or service improvements. Customer lifetime value (CLV) is a metric you often see, and this helps with comparisons associated with increasing share-of-wallet, or improving loyalty, or contrasting market segments. CLV is harder to connect to process improvements, as an example. You have to make a couple assumptions of causality – improvement of a process results in a change in customer behavior because… Whereas if you’re looking at profitability, you can more easily isolate investments which reduce your cost to serve a customer.
I’ve been asked before – what do you do when profit is not the motivation? If you are a non-profit, or a public-good company, or otherwise driven by some other purpose? Find the common denomination – which problem has the greatest impact on reducing child hunger? Which has the greatest impact on coral reef bleaching? There is a context within which you’re making decisions – find the lowest common denomination which allows you to make those decisions.
Conclusion
Use a common denomination of economic value for all of the problem statements you use to shape a product strategy.
One thought on “The Wrong Measure Will Misdirect You”