Using ROI For Requirements Is A Risky Business

We must allow for risk when calculating ROI

We’ve talked repeatedly about using ROI to drive prioritization of requirements based upon value. ROI can be used as the basis for prioritization for all decision making.

If we fail to take risk into account, our calculations will certainly be wrong, and we may make a poor decision. When we talk about accounting for risk in this context, we mean that we are accounting for the unlikely, undesired, or unintentional outcomes. We use the term expected value to refer to the risk adjusted approximation of the outcome. In financial circles, this is also called discounting.

The most common mistake people make when calculating ROI is failing to take into account the expected value of the return or the expected value of the cost of a project.

Expected value is easy to overlook

Imagine a project proposal for reducing costs for a software development team. The proposal is for the installation of requirements management software like Caliber RM from Borland. The cost of the system is $150,000* plus another $100,000 to cover 3 years of maintenance costs. Free installation and training sessions are bundled with the deal. The entire payment is required up front. The Borland sales rep proposes that the system will save $100,000 per year in reduced time creating software.

The total up front cost is therefore $250,000. [Update: Previously, there was a typo, where the upfront cost was $200,000 – thanks Paul for catching it and commenting!]

The first step in calculating ROI is to calculate the net present value (NPV) of both the investment and the savings.

How do we calculate NPV (in general)
The NPV of money that is spent (or earned) right now is the exact amount of money. If you promise to give me $100 right now, the NPV of that promise is $100. If we are determining the NPV of money that we will get in the future, we have to adjust the amount based upon an interest rate. Essentially, we are calculating the amount of money that we would give right now in exchange for another amount of money at a predetermined time in the future.

For example, we can put $100 in the bank now and receive $105 a year from now. We know this because the bank has promised us a 5% interest rate. We can work the math backwards to calculate the NPV of $105 that would be given to us a year from now. The bank promises us $105 a year from now, if we give them $100 today. With a 5% interest rate, we can divide the “end amount” by (1 + the interest rate) to determine the NPV of that $105. ($105 / 1.05) = $100. To calculate the NPV of an amount of money more than one year from now, we just keep dividing by the annual interest rate plus one.

If the bank offered us $110 two years from now, we would compute the NPV as ($110/1.05/1.05) = $99.77. Since the NPV of getting $110 in two years is lower than getting $105 in one year, it is not as valuable, and therefore it isn’t as good of an investment.

Calculating the NPV of our purchase decision

The net present value of the software expense is (negative) -$250,000 because we have to spend $250,000 right now.

To simplify the math, assume the $100,000 per year savings occurs at the end of each year. The NPV of our savings is ($100,000/1.05) + ($100,000/1.05^2) + ($100,000/1.05^3) = $272,256.

Now we can calculate the ROI for our software purchase

The ROI would therefore be ($272,256 – $250,000)/$250,000 = 9%.

Looks pretty good. Not a home run, but a sound investment. At 9%, this is a better return than we could get by just putting the money in the bank. The problem is that we’ve based this calculation on the assumption that we absolutely will save $100,000 per year. There’s a chance we’ll save more, and a very good chance that we’ll save less. How do we account for the risk?

Now we account for risk

We can quantify this risk as an expected value by identifying the possible benefits, and our belief of the probability of each one happening. See the linked definition for an explanation of how and why this works.

We believe the following probabilities are true:

1. In the first year, there is a 75% chance that we will only save $50,000 and a 25% chance that we will save the full $100,000.
2. In the second year, we believe there is a 50% chance that we will save $50,000 and a 50% chance that we will save $100,000.
3. In the third year, we believe that there is a 50% chance that we will save $50,000 and a 50% chance that we will save $100,000.

For the first year, there is a 75% chance that we will only save $50,000 and a 25% chance that we will save the full $100,000. The expected value of the first year’s savings is (0.75 * $50,000 + 0.25 * $100,000) = $62,500. By quantifying the risk, we have identified that the expected value is a lot lower (in our opinion) than the proposed value.

There is a 50% chance of saving $50,000 and a 50% chance of saving $50,000 in each of the second and third years. The expected values for each of those years are therefore (0.5 * $50,000 + 0.5 * $100,000) = $75,000 per year. These expected values are also lower than the proposed values.

We now recalculate the NPV of our savings using the expected values: ($62,500/1.05) + ($75,000/1.05^2) + ($75,000/1.05^3) = $192,339.

Conclusion
Calculating the ROI with expected values shows a loss instead of a savings: ($192,339 – $250,000)/$250,000 = -23%.

By including a representation of risk, we have calculated a more realistic ROI. In our example we have identified that the project is most likely to cause us to lose money instead of saving us money.

This demonstrates the imporatance of including a risk analysis in our ROI calculations.

– – –

Minor tangent
Ultimately, decisions will be subject to other constraints, such as availability of people or resources. On some teams, the constraints are so restrictive that ROI calculations are almost irrelevant – one reader commented privately that his teams were so restricted by these constraints that they never had the opportunity to use ROI for their decision making.

*This is a representative price for use in our example – it isn’t a quote for Caliber RM.