When I was still leading implementation teams, I had a lot of success with having team members design the tests (including the data for the scenarios, like 1499, 1500, 1501, -1, 99999, etc) before designing the code. It is easier to discover when a test is leaky (e.g. has incomplete coverage) this way. Your example, of testing 1600 to confirm “not more than 1500″ is a good one. In evaluating the test-design, the question of “what about 1501?” it is obvious that the test-design is not rigorous enough.

Thanks again

When I read your sources of bugs … I interpreted them as examples of how people misinterpret requirements in the first place.So main cause of all bugs is people interpreting the requirements of stakeholders according to their belief, biases and understanding.

If we can fix that … I can say 90% of bugs are gone.

I wrote about requirements here

http://shrinik.blogspot.com/2008/02/software-requirement-heuristic-all.html

I am following your blog … you are on my google reader feed … Keep up the good work

Shrini Kulkarni

Shrini

Thank you for the feedback and the encouragement. My first line was meant to get people’s attention, but you are right — it does seem like spam! I have just started this campaign and am making continuous improvements. I have thought about making categories for the bugs, but for now I just think I need to start populating the site and spreading the word!

Thanks again for your encouragement (and feedback).

Your diagram is very interesting. I would add another error (E) to the ‘developers create solution design’ task. I would define the error as “that is not technically possible”. Too many marketing/sales start projects and write requirements before a technical feasibility process is performed. The developers are handed a set of requirements that sometimes aren’t even feasible — therefore they ‘improvise’. Just my opinion :)

BTW, great site. I particularly like the focus on the entire software development process — not just the design/development itself. Good job!

Interesting idea – a free-form “post a bug about (any) software here” blog, allowing for discussion. I actually don’t think it will work, but I could be wrong.

I’m leaving your comment up, in hopes that it is a great idea, and it won’t annoy our readers to see the comment.

As a point of feedback, it read like a generic spam comment, and _almost_ got deleted by me when managing the site. If I hadn’t been really paying attention, I would have deleted as soon as I read “Are you tired of crappy software?”

Good luck with your effort.

I have started a campaign to improve software quality worldwide. Please help!

I want to force rich development companies to take responsibility for their poor development efforts!

Let’s expose software bugs to the public, so that development companies are forced to release better software!

The exact same question/critique was made last week on the requirements engineering yahoo group. Here’s part of my response within that thread. I’ve edited to include only the parts of that response relevant to your question:

You’re exactly right. I lumped design and implementation together, which I should not have done. And I completely overlooked deployment. I really should have caught both of those. Thank you to you, and to everyone who’s stimulated this discussion. Now I have an excuse to re-write the article and make it much better :).

I would replace the entire “E3″ section with two sections:

Errors in design – fundamentally, there are two types. The first comes from misinterpretation of the requirements (solving the wrong problem) and the second comes from inadequate solutions (not solving the problem). Both can be tested (at the design stage) with a Gedanken experiment (an imperfect test). Both can also be tested by creating “requirements tests” of the implementation. Those tests will not isolate design problems from implementation problems, but they will (when the tests are good) identify “failure to meet the requirements.” Root cause analysis will determine if it was a misinterpretation (a risk that can be mitigated with communication) or a mis-application (preventable only with skill).

Although I don’t have data [...], my intuition is that this is where most non-functional requirements are violated.

Implementation errors can be failure to achieve the desired design, either through misunderstanding or inadequacy. Inadequacy can be captured with unit tests created by the developers. Misunderstanding can also be captured with unit tests, but only when the test suite is validated by the designer as “testing the design.” Incorrect execution of these tests is error source E4 in the diagram.

Deployment errors are another beast entirely. I’m embarrassed that I didn’t talk about them at all in the original article. To Scott Preece’s point – yup, there’s a whole lot of work that happens “on the job site” in all of the engineering disciplines. The permutations of environmental variables, and the need to expect / prevent bugs “in other software” that only appear when your “perfect” software co-exists can be overwhelming. That’s why creating an operating system for an open hardware platform is so hard. That’s also why massively appealing consumer software is hard (so many different, but simple, environments). And it explains why enterprise software is hard (few, but complex, environments). Paul Graham wrote an essay once about the liberation that comes from developing server-side software where you control “the job site.”