In his post, Roger uses an example of an untestable requirement “We might specify that the car should last seven years without repairs as long as the owner maintains the car according to a certain maintenance schedule and doesn’t have a collision.” He makes a great point, that just because you can’t directly test the requirement, you should not ignore the requirement. And I agree.
The premise behind the rule that requirements must be testable is driven by the goal of avoiding ambiguous language in your requirements. Statements like “the application must have a clean user interface” or “search response times must be fast” are also untestable, but more because of language than anything else.
You can rewrite these hypothetical ambiguous requirements in a testable way –
“The application will meet the (attached) user interface guidelines.” Where the UI-guidelines describe detailed inspectable criteria (sharing a common navigation bar at the top of a page, no horizontal scrolling on an 800×600 interface, controls must be at least 10 pixels apart, etc).”
“Search results must return the first page of results within 2 seconds, when the user is connected to the same LAN as the server. If there are multiple pages, each additional page must be presented to the same user within 2 seconds of selection.”
Back to Roger’s example…
While you can’t wait 7 years to test the car before you decide to build it, you can rewrite the requirement to make it testable.
First, I would point out that the example requirement is ambiguous. Do they mean that none of the cars will have a warranty-repair? Or no more than 1% of the cars? Greater specificity should be included. Let’s add the 1% number. We will also want to specify “normal usage patterns” – which can mean no off-road driving for sedans, a specified temperature range, maximum miles per month, etcetera.
We still can’t directly test the requirement. And it’s not actionable – you haven’t told the engineers how to know when they’ve completed the design of the car.
How do car manufacturers build quality cars today? They test components and assemblies of components, and characterize their failure rates statistically. Then they combine that empirical data with a statistical model of the expected wear and tear of the vehicle over time. The result is a statistical prediction of when the car is likely to have it’s first warranty repair. And that statistical prediction is a continuum. But it’s testable, if you rewrite the requirement:
“The results of running our existing lifetime-quality-test* for sedans on the vehicle design will predict fewer than 1% of cars will have a warranty repair during their first 7 years of usage, with a 90% confidence level.” The lifetime-quality-test is a referenced document in the requirements, and it describes how components are tested.
Anyone have an “untestable” example for me?