In his post, Roger uses an example of an untestable requirement â€œWe might specify that the car should last seven years without repairs as long as the owner maintains the car according to a certain maintenance schedule and doesnâ€™t have a collision.â€ He makes a great point, that just because you canâ€™t directly test the requirement, you should not ignore the requirement. And I agree.
The premise behind the rule that requirements must be testable is driven by the goal of avoiding ambiguous language in your requirements. Statements like â€œthe application must have a clean user interfaceâ€ or â€œsearch response times must be fastâ€ are also untestable, but more because of language than anything else.
You can rewrite these hypothetical ambiguous requirements in a testable way –
â€œThe application will meet the (attached) user interface guidelines.â€ Where the UI-guidelines describe detailed inspectable criteria (sharing a common navigation bar at the top of a page, no horizontal scrolling on an 800Ã—600 interface, controls must be at least 10 pixels apart, etc).â€
â€œSearch results must return the first page of results within 2 seconds, when the user is connected to the same LAN as the server. If there are multiple pages, each additional page must be presented to the same user within 2 seconds of selection.â€
Back to Rogerâ€™s exampleâ€¦
While you canâ€™t wait 7 years to test the car before you decide to build it, you can rewrite the requirement to make it testable.
First, I would point out that the example requirement is ambiguous. Do they mean that none of the cars will have a warranty-repair? Or no more than 1% of the cars? Greater specificity should be included. Letâ€™s add the 1% number. We will also want to specify â€œnormal usage patternsâ€ – which can mean no off-road driving for sedans, a specified temperature range, maximum miles per month, etcetera.
We still canâ€™t directly test the requirement. And itâ€™s not actionable – you havenâ€™t told the engineers how to know when theyâ€™ve completed the design of the car.
How do car manufacturers build quality cars today? They test components and assemblies of components, and characterize their failure rates statistically. Then they combine that empirical data with a statistical model of the expected wear and tear of the vehicle over time. The result is a statistical prediction of when the car is likely to have itâ€™s first warranty repair. And that statistical prediction is a continuum. But itâ€™s testable, if you rewrite the requirement:
â€œThe results of running our existing lifetime-quality-test* for sedans on the vehicle design will predict fewer than 1% of cars will have a warranty repair during their first 7 years of usage, with a 90% confidence level.â€ The lifetime-quality-test is a referenced document in the requirements, and it describes how components are tested.
Anyone have an â€œuntestableâ€ example for me?