No matter how good your quality process is, you are introducing bugs. This article reviews the places where bugs are introduced in the software development process (from stakeholders to users), and reviews ways to address those bugs.
One way to identify the sources of bugs is by listening to how people tell you about the bugs.
- “That is not what I want”
- “That is not what I said”
- “That is not what I meant”
- “That is not how it is supposed to work”
- “That is not what they meant”
- “That is not what I want it to do”
Visually, you can think of the development process (regardless of methodology) like the following flow
Each of the six sources of bugs is labeled in the above flow. The article, Where Bugs Come From, goes into a detailed explanation of each source of bugs. This article looks at what you can do to improve the ability of your process to reduce these bugs. I have been a stakeholder, a product manager, a business analyst, a developer and a tester. And I’ve been guilty of all of these offenses at one time or another. When I say you, I also mean me. So chill.
“That is not what I want”
Stakeholders introduce bugs into the process by being unable to define what they want. Some agile methodology proponents argue that as a stakeholder, you can’t know what you want until you have something you don’t want in front of you. In other words, those agilists are giving up on trying to prevent these errors – they claim it is futile.
You can change your mind. The team could be dealing with “That is not what I want now” – and change is something that can not be prevented – although the impact of change can be mitigated. Iterative development and course-correction through getting immediate feedback is one way to do that. Prototyping is another method of course correction. Prototypes can also be used to elicit implicit requirements.
But that still avoids trying to directly address the issue of you not effectively describing what you actually want. There are ways to improve your team’s engagement with you, so that you actually say what you want. First, an emphasis on valuable requirements drives critical thinking about why you need a particular capability or feature. One approach to achieve valuable requirements is to question their justification.
The key to making this work is to make sure you are approving your requirements. And watch out for the anti-patterns in the article.
“That is not what I said”
Business analysts, product managers, and requirements analysts introduce bugs into the process through misunderstanding the stakeholders. Using the techniques above to stay on top of changes in stakeholder needs is not enough. Even when you effectively engage your stakeholders so that they are self-aware and can articulate exactly what they need and why, you still have a problem.
You can improve how you listen, and you can improve how you document what you heard. Make sure that you write complete, consistent and unambiguous requirements. While you’re at it – make sure you are writing your requirements correctly.
“That is not what I meant”
Developers can misinterpret requirements. While there are situations where a requirement is adequately expressed, but the developer is not capable of understanding, that is not the norm. Developers are smart. The onus is on the person writing requirements to make sure that they are written for the audience. So, even though the source of these bugs appears to be the developers, it is actually your documentation. If your documents need to span cultural chasms, then that’s your reality.
Make sure you write your requirements unambiguously, so that they are not misinterpreted. Make sure that they are feasible requirements, and that they do not specify design. The requirements also need to be concise without being overly terse.
“That is not how it is supposed to work””
Developers should test what they create. Given an understanding of what you believe the code is supposed to do, you should test that it actually does it. This is the stereotypical bug – we asked for X, and it doesn’t work. Continuous integration is the most effective technique for addressing bugs that are introduced at this point in the process.
In more complex situations, you can apply more advanced techniques to assure that your testing is complete.
“That is not what they meant”
The testing team can also misinterpret requirements. This is a source of false positive bugs – even when the developers properly interpret the requirements, a bug may be lodged against their code when there is no bug. That’s why we have the ability to mark issues as “not a bug” when closing them. Apply the same rules for well-written requirements and you will also address this issue. Further, make sure that your requirements can be measurable, or verifiable. We recently wrote more about why you should make requirements testable.
“That is not what I want it to do”
User education is the key here. Users should be encouraged to provide feedback about your product. You don’t want to discourage feedback, you want to minimize the number of false positives. False positives come from users misinterpreting how the application is behaving – a sign of poor design. False positives also come from users not understanding what the application can (or should) do.
You can address design issues by focusing on design with the user in mind. And you can educate the users through a combination of training and with a novel approach to documentation. Document the product in terms of what users are trying to accomplish, in the context of how they are trying to do it.
Conclusion
There are a lot of people involved in the software development lifecycle. As people, we introduce bugs. As informed software professionals, we also know how to address many of them.
Scott, thanks for giving a novel presentation to the problem of bug introduction and possible mitigation strategies!
Can’t bugs be introduced during the following steps of the development process?
1) Developers create solution design
2) Software is deployed
Bugs added in the above steps might exist because of technological reasons (and could impact the satisfaction of non-functional requirements e.g. performance, maintainability, availability)
Thanks, Inder, and welcome to Tyner Blain!
The exact same question/critique was made last week on the requirements engineering yahoo group. Here’s part of my response within that thread. I’ve edited to include only the parts of that response relevant to your question:
You’re exactly right. I lumped design and implementation together, which I should not have done. And I completely overlooked deployment. I really should have caught both of those. Thank you to you, and to everyone who’s stimulated this discussion. Now I have an excuse to re-write the article and make it much better :).
I would replace the entire “E3” section with two sections:
Errors in design – fundamentally, there are two types. The first comes from misinterpretation of the requirements (solving the wrong problem) and the second comes from inadequate solutions (not solving the problem). Both can be tested (at the design stage) with a Gedanken experiment (an imperfect test). Both can also be tested by creating “requirements tests” of the implementation. Those tests will not isolate design problems from implementation problems, but they will (when the tests are good) identify “failure to meet the requirements.” Root cause analysis will determine if it was a misinterpretation (a risk that can be mitigated with communication) or a mis-application (preventable only with skill).
Although I don’t have data […], my intuition is that this is where most non-functional requirements are violated.
Implementation errors can be failure to achieve the desired design, either through misunderstanding or inadequacy. Inadequacy can be captured with unit tests created by the developers. Misunderstanding can also be captured with unit tests, but only when the test suite is validated by the designer as “testing the design.” Incorrect execution of these tests is error source E4 in the diagram.
Deployment errors are another beast entirely. I’m embarrassed that I didn’t talk about them at all in the original article. To Scott Preece’s point – yup, there’s a whole lot of work that happens “on the job site” in all of the engineering disciplines. The permutations of environmental variables, and the need to expect / prevent bugs “in other software” that only appear when your “perfect” software co-exists can be overwhelming. That’s why creating an operating system for an open hardware platform is so hard. That’s also why massively appealing consumer software is hard (so many different, but simple, environments). And it explains why enterprise software is hard (few, but complex, environments). Paul Graham wrote an essay once about the liberation that comes from developing server-side software where you control “the job site.”
Are you tired of crappy software? I am… and I have decided to do something about it!
I have started a campaign to improve software quality worldwide. Please help!
I want to force rich development companies to take responsibility for their poor development efforts!
Let’s expose software bugs to the public, so that development companies are forced to release better software!
Achilla,
Interesting idea – a free-form “post a bug about (any) software here” blog, allowing for discussion. I actually don’t think it will work, but I could be wrong.
I’m leaving your comment up, in hopes that it is a great idea, and it won’t annoy our readers to see the comment.
As a point of feedback, it read like a generic spam comment, and _almost_ got deleted by me when managing the site. If I hadn’t been really paying attention, I would have deleted as soon as I read “Are you tired of crappy software?”
Good luck with your effort.
Hi Scott,
Thank you for the feedback and the encouragement. My first line was meant to get people’s attention, but you are right — it does seem like spam! I have just started this campaign and am making continuous improvements. I have thought about making categories for the bugs, but for now I just think I need to start populating the site and spreading the word!
Thanks again for your encouragement (and feedback).
Your diagram is very interesting. I would add another error (E) to the ‘developers create solution design’ task. I would define the error as “that is not technically possible”. Too many marketing/sales start projects and write requirements before a technical feasibility process is performed. The developers are handed a set of requirements that sometimes aren’t even feasible — therefore they ‘improvise’. Just my opinion :)
BTW, great site. I particularly like the focus on the entire software development process — not just the design/development itself. Good job!
Scott,
There is a serious bug in your blog … kindly fix it …
After entering a long comment, I proceed to fill name, email etc but pressed “enter” key accidently before completing required fields. I was then shown error page stating that data for “required field” is missing. There is no way to go back the post draft… that is bug no 1. I had to use browser back button to reload the post …Now entire draft post is gone and I can not recover it. Why is the when the post is reloaded, comment form content gets cleaned up?
Shrini
Shrini – welcome to Tyner Blain, and thanks for pointing this out. The comment functionality was just changed in the latest upgrade to the blogging software (a couple days ago). I was able to duplicate exactly what you saw, but I don’t know how to change it yet. I will look into it – thanks for taking the time to point it out!
Scott,
This was the content that I wished to post but lost most of by accidentally pressing enter key in “mail” field without entering the data there.
When I read your sources of bugs … I interpreted them as examples of how people misinterpret requirements in the first place.So main cause of all bugs is people interpreting the requirements of stakeholders according to their belief, biases and understanding.
If we can fix that … I can say 90% of bugs are gone.
I wrote about requirements here
http://shrinik.blogspot.com/2008/02/software-requirement-heuristic-all.html
I am following your blog … you are on my google reader feed … Keep up the good work
Shrini Kulkarni
Hi Scott,
While the approach outlined is nice in my experience I have seen that many time testers say that they have covered the scenarios yet they miss defect. It’s mainly because the real time user scenarios was not considered or correct test data was not used. E.g. for testing an ATM scenarios where the max withdrawal limit for a day is $1500 then testing 1400, 1500 and 1600 is not sufficient even though they cover all classes. Ideally it should be 1499, 1500 and 1501
Otherwise great write up!!
Thanks
Janhavi
Thanks, Janhavi!
When I was still leading implementation teams, I had a lot of success with having team members design the tests (including the data for the scenarios, like 1499, 1500, 1501, -1, 99999, etc) before designing the code. It is easier to discover when a test is leaky (e.g. has incomplete coverage) this way. Your example, of testing 1600 to confirm “not more than 1500” is a good one. In evaluating the test-design, the question of “what about 1501?” it is obvious that the test-design is not rigorous enough.
Thanks again