Prioritization With ROI and Utility

Prioritization with ROI is generally thought of as a quantitative analysis. For hard and soft ROI, that is true. But benefits can not always be quantified. Economists get around this with the notion of utility. You have to make a prediction of the utility of the requirement or feature. How do you do that?

Calculating ROI

Calculating the return on an investment (ROI) is hard, even when reviewing the past. The benefit divided by the cost is the ROI. We have two seperate challenges – calculating costs and calculating benefits. Management accounting provides us with standard approaches to calculating costs. The problem is that you have to measure benefits. And benefits are made up of both hard and soft benefits. Those hard and soft benefits are combined with cost data to determine hard and soft ROI.

Calculating Hard ROI Benefits

To forecast hard ROI benefits, you have to formulate a measure of risk. No one can predict the future with certainty. By adjusting for risk, you can calculate the expected value of a requirement or feature. That risk-adjusted value can be converted into a net present value.

Hard ROI benefits are benefits that are directly attributable to the item being measured. For example, if creating a website for selling your products generates $1,000 in profits from those sales, then the hard ROI is $1,000.

Calculating Soft ROI Benefits

Soft ROI benefits are those benefits that are indirectly attributable to the item being measured. After creating your website, perhaps you add a feature that allows people to check inventory levels on the items. Profits go up by 10%. How much of that change can be attributed to the “check inventory” feature, and how much is attributable to other factors? The same problem applies to site redesigns, marketing campaigns, etc. Benefits are indirect, or hard to quantify.

Tom Pisello offers a good explanation and some soft ROI examples in his searchCRM.com article.

He suggests that we discount the soft ROI numbers by 60% to 90%. In other words, if we forecast/predict/guess that the “check inventory” feature will generate an additional $100 in profit, we should only use $10 to $40 in our calculations. The heavy discounting is an implicit reflection of the risk that our hypothesis is wrong. We’re predicting, presuming, or modeling a connection between a measurable factor (the ability to check inventory levels) and an indirect measurement (the number of sales we get).

You can use techniques like A/B testing to improve your ability to track this after the fact. For example, you can modify the website so that some randomly selected customers get the inventory data, and others don’t. Then you can compare the statistics of sales for the two groups. If both groups rose by 10%, the “check inventory” feature was not likely to have caused the rise. If the control group rose by 5% and the inventory-checking group rose by 20%. then the “check inventory” feature may have been responsible for the extra 15% growth.

You can’t use A/B testing before the implementation, however. To compensate for this, you should discount the anticipated benefit as Pisello suggests.

What About Utility?

Utility is the measure of satisfaction that someone gets from something. When prioritizing requirements, we will narrow down the definition to focus on the satisfaction that a user gets from a requirement or feature.

Agile software proponent Kent Beck argues that people don’t know what they want when you ask them. There may be some scientific support for his argument based on some research cited by Barry Schwartz in The Paradox of Choice: Why More is Less.

Schwartz references psychologist Daniel Kahneman’s studies of how people remember events, in an explanation of how people measure and assess utility. To sum up, here’s the flow of events:

• A person has an experience, and recognizes some satisfaction from it. This is known as experienced utility. This is the function that economists reference when defining the indifference curve.
• When remembering the event later, a person will remember incorrectly. Their memory of their level of satisfaction is their remembered utility. The problem is that the remembered utility never precisely matches the experienced utility.

This poses an interesting problem. Even if we find people who have experienced exactly what we’re trying to analyze, they will be unable to correctly tell us how much satisfaction they derived in the past.

At some level, this translates into “Users can’t tell us what they liked in the past.” At least not with precision.

Schwartz goes on to make the point that when people are asked how much satisfaction they will get from something in the future, they will still get it wrong. A prediction of satisfaction is known as expected utility. His argument is that those predictions are based upon past experiences, but the extrapolation is error-prone.

People have some actual experienced utility, which they remember imprecisely as remembered utility. They base predictions of expected utility upon the remembered utility, but introduce errors in the prediction process. We’re two steps away from the data we actually need – with no way to compensate for the error. And that only applies when we have someone who can apply past experiences when predicting satisfaction levels for the requirement we are evaluating. When we want to introduce something new, the interviewee has no framework from which to build their predictions accurately.

All of this data supports the premise that people can’t tell us what they want (precisely) before they review what we offer them. What they can do, however, is provide feedback in a review of a delivery, or ideally a prototype.

Conclusion

• The value of a requirement or feature is a function of user satisfaction and the financial benefit of having that requirement implemented.
• The user satisfaction can not be quantified in advance because we have no framework for reliably predicting satisfaction – even based on past experiences.
• Financial benefit is what we have to base our decisions on – both hard ROI and soft ROI data.
• We can measure cost data, and even forecast it – taking into account both fixed and variable costs. But if we use the wrong cost allocation strategy, we will make suboptimal decisions.
• The success of a requirement is based upon the ratio of value to costs.