Writing Verifiable Requirements should be a rule that does not need to be written. Everyone reading this has seen or created requirements that can not be verified. The primary reason for writing requirements is to communicate to the team what they need to accomplish. If you can’t verify that what the team delivered is acceptable, neither can the team. This may be the most obvious of the rules of writing requirements – but it is ignored every day.
Verifiable Requirements – Revisiting
In 2006, I first looked at how and why writing verifiable requirements is important – as part of the ongoing series on the rules of writing good requirements. The focus of that article was on testability, as is this one. When visiting verifiability again, however, I will add that not only must specifications be objectively measurable, but so must goals be written so that you can identify when a solution has met the objectives that justified its creation.
Directly Verifiable Requirements
Most of the requirements we come across can be directly verified. The only problem with these requirements is that they lack specificity. The following example is almost verifiable, and only needs a little help:
- The web site must make it easier for users who use site search to find the products they want to buy.
This requirement presents the challenge of resolving the ambiguity of the requirement. This lack of specificity prevents the requirement from being verifiable. You have to identify how much easier the site search process must become for users. Findability is a more is better characteristic in the Kano Analysis model for defining requirements.
You have to acknowledge that making it easier to find what you’re looking for is better for users than making it harder – there is an upward slope to the function. However, there are also diminishing returns.
[- A search that includes “the perfect item” is massively better than a search that does not include what you are looking for.
- A search that includes the perfect item on the first page of results is significantly better than one that returns the perfect item on the second page of results.
- A search that includes the perfect item as the first result is only marginally better than a search that returns that perfect item as the second or third result.
Improving findability as articulated above, from a user perspective, manifests in value to your company in two ways: immediate revenue impact and indirect revenue impact.
The immediate impact can be measured in terms of increases in commerce. The indirect impact results from improving the user experience. These improvements in experience result in increased word of mouth, as users altruistically encourage others to visit your website, and also result in an increase in satisfaction causing users to return to your site more frequently and make more purchases from you.
You can measure these impacts in terms of
- Conversion percentage (percentage of people who search and then purchase the products found in the results)
- Revenue attributed to users who search (an absolute measurement of purchases of products found via search)
- Site traffic levels (the number of people that visit your site over time)
- Visitor-recency statistics (the amount of time that elapses between return visits for returning visitors)
Conversion percentage, for users who search, normalized against users who do not search, is the most isolated (from other variables and noise) and fastest responding (as a delayed measurement of impact) measurement of the value of making it “easier” for users to search for the products they want to buy.
Your team will design different approaches to achieving these improvements. You have to estimate both the cost and the potential benefit of each approach. Balancing cost estimates with potential benefits will yield the ideal requirement – perhaps a 10% improvement. [Note: I’ve collapsed at least another article’s worth of balancing cost versus benefit, and multiple articles of “understanding and measuring site search” into one paragraph here, in hopes of staying on task with writing verifiable requirements.]
Rewriting the requirement as follows makes the requirement verifiable:
- Before: “The web site must make it easier for users who use site search to find the products they want to buy.”
- After: “Users who search on the site will be at least 10% more successful at finding the products they want to buy when using site search.”
Impossible to Verify Requirements
Often, stakeholders will present us with requirements that are impossible to verify (as requested).
- The home page needs to load fast.
At first blush, this looks just like the previous requirement (easier search is similar to faster page load). You can use the same techniques to determine a measurable “requirement” like “the home page needs to load in under 1 second.” However, that’s not a good requirement, because it is not an implicitly valuable requirement. This statement provides no insight into why a fast-loading home page might be valuable to a particular user or why it would be valuable to the company.
To address this issue, you have to meet with the stakeholder(s) and understand the actual underlying requirements. You will ask why, determine motivations and underlying problems, and apply active listening skills to discover the underlying requirement.
The problem is that the statement, “the home page needs to load fast” is a specification. In Requirements Documents – One Man’s Trash…, I first used the following diagram to show how different people in the process of designing software view their piece of providing a solution.
A developer may receive a spec – “home page must load in under 5 seconds” and feel like the why component is perfectly well defined – it is a matter of context and perspective. Another way to think about this is that the problem is with the problem statement.
A product manager, however, needs to do research that follows a path like the following:
Given that we are building an eCommerce website, we know that people have expectations around page load times (per Forrester / Akamai report (requires registration)).
[Further, those expectations manifest as people abandoning slow-loading websites (from the same report).
[larger image].
We know from this market research that page-load times (for websites) represent another more is better Kano attribute, but one that has must be characteristics at the extremes.
[While more speed is better, what the market research reveals is that for any given person, there is a minimum-speed threshold, at which speed is a must-be characteristic – too slow, and you lose customers (immediately).
A combination of market research and elicitation will reveal that there is a true underlying goal of being fast enough to satisfy as many users as possible. Combining this with practicalities of scaling your solution, and the associated costs; in the context of a particular solution design (an eCommerce website), you will arrive at a goal that allows you to rework your requirement so that it is verifiable.
Note: The requirement in this example is a subordinate requirement to a goal focused on maximizing conversion percentage (the percentage of customers who visit the site making purchases) – with a recognition that a major source of non-conversion is abandonment of the site, and that a large contributor to visitor abandonment is page load times. An Ishikawa (or other model) will help you articulate this complexity and formulate requirements at a level of abstraction that is both valuable and actionable.
Rewriting the requirement as follows makes the requirement verifiable:
- Before: “The home page needs to load fast.”
- After: “No more than 10% of potential site visitors will abandon our site before viewing the page.”
Indirectly Verifiable Requirements
Some requirements are impossible to literally verify, and must be inferentially verified.
- The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.
Models are Models, Not Reality.
Imagine trying to create a test by organizing a million visitors to hit your SaaS web solution on the same day, with at least 10,000 of them hitting the site simultaneously. That is completely impractical. That doesn’t however, invalidate the requirement or make it untestable. The combination of modeling, hypothesis formulation, and extrapolation gives you a reasonable way to verify that your solution probably meets this requirement.
You’re already used to the concept that models are representations of reality – think about the maps you use – a map may have a scale like one inch equals one mile. The map is a model of reality. A map where one inch equals one inch would be impractical.
An effective approach to measuring this type of scalability requirement is to simulate users (with load testing and realistic user-behavior scripts) with automation. That takes care of most of the coordination problem. However, the cost of simulating a million users and 10,000 simultaneous users is high. You can, more realistically, measure the site’s performance when 10, 100, and 1,000 simulated users simultaneously access a server. You can extrapolate from those results to estimate how the system is likely to perform under 10,000 user loads.
Note: Extrapolation is dangerous (follow the Elvis link below)- you have to justify why extrapolation holds true, and identify when it does not, to minimize the risk of invalidating your hypothesis. Make sure you have confidence in the engineering prowess of whoever is doing your extrapolation and modeling.
[see tip #5 of these ROI calculation tips for the Infinite Elvis anti-pattern]
While this requirement does not to be rewritten in order to be verifiable, it does need to be augmented:
- Original: “The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.”
- Additional: “Testing of the site must show that 500 simultaneous users following <user script X> will yield no more than 1% page load times over 200 ms.”
Conclusion
Every requirement you write must be verifiable. When you can’t verify something, and can’t rewrite it into a verifiable form, that should be a sign that either it is a vision statement or a red herring. Vision statements guide how we approach creating products and engage markets – they are valuable, but they are not requirements. Red herrings are well-meaning but ill-advised inputs into the process that need to be culled – they are neither valuable nor requirements.
Make sure all your requirements are verifiable.
13 thoughts on “Verifiable Requirements”