Last week, Ivan Chalif, a product manager / blogger, tapped into a thread criticising product managers for not adopting and espousing agile, or at least rapid-release techniques. In this article we look at Ivan’s comments and one of the articles that he referenced. We also share our own perspective and an alternative analysis of what may have happened.
Short Release Cycles are Bad
At the end of his article, Ivan references a well presented position on why short release cycles are bad that was published by Justin James on TechRepublic last June.
Ivan’s article basically says that agile approaches are fine for non-mission critical software. Justin’s article includes criticism of software he has used, and implies that the problems are due to the development methodologies used.
In a nutshell, they are both saying that short release cycles are bad because they result in bad software being released. Ivan qualifies his perspective by saying that for some applications, like “consumer web applications” that’s ok – because being bad doesn’t hurt as much.
Our perspective is a little bit different – short release cycles are the best thing ever. Immature or incomplete releases are bad. We define incomplete as meaning “not all of the must be requirements have been satisfied.” We define immature as meaning “quality, usability, and performance are not where they need to be for the audience.”
Ignorance is No Excuse
On the point of requirements, we’ll add that failure to implement an undiscovered / undocumented must-be requirement still qualifies as incompleteness. A product manager’s failure to document a critical requirement may absolve the implementation team from addressing and testing it – but the “entire team” is still on the hook for it.
In short, we don’t think agile approaches need to be forced to live in the “software that isn’t important to users” box. Agile techniques and approaches can be used on any project. Any project can succeed or fail, and any process can succeed or fail on a particular project. What is more important is the team. And unfortunately, the environment – people trump process, but politics trumps people. If your organization can not or will not support an iterative development cycle, your project will fail if you use an agile process.
Mission critical and non-mission critical software have mandatory requirements that must be satisfied. Must-be requirements, by definition, must be satisfied. A build of the software that does not satisfy those requirements is one that is incomplete, and should not be released for production use.
Note that we specified “production use.”
Anti-Agile Argument
We’ll take a point-counterpoint approach to one of the arguments raised by Justin. Our position is that his criticism, while anecdotally valid, should not be a criticism of the process, but of its poor execution.
Here is an example of what I mean: today, I made hotel reservations with a Hyatt hotel on their Web site.[...]Except there is one huge flaw: it does not use HTTPS! So not only is my credit card number being sent in plain text (along with my first and last name), but it is going to be stored in my browser’s auto-fill system in plain text!
Justin James
Is that a problem? Absolutely. Is secure transmission of user financial data a must-be requirement? Unquestionably. I explained to my mom that her credit card number is safer on the Amazon form than it is on the receipt at a restaurant. The whole ecommerce world is dependant on that assumption. The team that implemented the website Justin was using screwed up big time. But how did they screw up?
The team that built this Hyatt website screwed up in one of four ways:
- They did not identify the security requirement.
- They did not write code to satisfy the security requirement.
- The code they wrote did not satisfy the security requirement.
- They wrote code that broke a previously working solution to the security requirement.
This clearly can be blamed on the “entire team” but can not be clearly blamed on agile software development.
Is Agile To Blame?
Let’s look at each of the possible explanations from above in more detail…
They did not identify the security requirement
The first possibility – failure to identify the requirement, is a risk with every development process. Identification of the critical requirements is a matter of product manager competence (or product champion, or whoever on the team is gathering the requirements). More rigorous requirements gathering will identify more requirements. You don’t neccessarily have to have a big up-front requirements (BUFR) exercise before writing the first line of code. You do have to have “enough” requirements gathering. A BUFR process does not assure a complete set of requirements any more than an incremental process prevents it.
All teams will reach a point when they decide they have “enough” to risk developing against an incomplete spec. A product manager has to develop a sufficient breadth of understanding to make an informed decision about when “enough” is enough. That’s why we encourage defining the space with use case names and brief descriptions of use cases. Understanding the domain is the goal of this exercise, and it is not the same as defining all of the requirements up front.
This team failed to get an understanding of the domain, and overlooked something as obvious as the need for customers to have secure transmission of their credit card numbers. That is a failing of the product manager, not an agile approach. Note: we dismiss any process that says “don’t develop an understanding of the domain” as a bad one, agile or otherwise.
They did not write code to satisfy the security requirement.
Was the requirement mis-prioritized as being less than critically important? If so, blame the product manager again.
What if the requirement was defined, identified as a must-be requirement, and still wasn’t implemented? Who’s fault is it then? Did the team have poor execution, or did they follow a process that says “define the requirements, but don’t address them if you don’t feel like it?”
I’m not aware of an agile process that says you should release the software to production without implementing a solution to a known critical requirement. Definitely can’t blame this on agile. You could potentially blame it on an executive or project manager who said “release on date X, I don’t care if it’s done!” In which case, Justin just had bad timing, because any agile process would put this at the top of the list for the next release. If not, it isn’t agile.
The code they wrote did not satisfy the security requirement.
OK, now we’re looking at a team that 1) identified the requirement, and 2) attempted to implement it in their release, but 3) failed to test it (or released with a known but uncommunicated bug). I say uncommunicated because Justin didn’t get the message, even if someone on the team tried to send it.
Lack of testing and failure to communicate are not characteristics of agile processes. Just because you’re delivering code rapidly, managing the amount of work to be delivered in a timebox, does not mean that quality is ignored. That is certainly a choice that any foolish team (or irresponsible team) can make. But it is not unique to agile development, or even characteristic of agile development.
The agile projects that we’ve seen tend to have better quality than waterfall projects. Not because they are agile, but because the developers who are following an agile process have tended to (this is anecdotal data) have a greater appreciation for the benefits of continuous integration and quality in general. They live by the test cases. If it isn’t tested, it hasn’t been completed. And if it fails the tests, it hasn’t been completed.
There is a greater possibility that a waterfall project will allow developers to write code for months on end without creating tests – with the expectation that it will get tested in the end. We’ve worked with a client who’s timeline for a release of a key set of features included 3 months for development and one week for testing. When we asked “Where’s the time allocated to fix the code when you discover that it is broken?” we got the strangest looks. As an aside, in the year after we helped that team implement a continuous integration process, they’ve reduced their outstanding bug list by almost 80% and the team lead expects it to be cut in half again in the next month. They’re making great progress just by adding one agile technique (continuous integration). The team has to create, review, and pass tests before they check code into the trunk for the current release cycle.
Absolutely not a failing of agile software development – this would be a failing of the team to deliver quality product.
They wrote code that broke a previously working solution to the security requirement.
This is another variation of the previous situation. 1) They identified, scheduled, implemented and tested the functionality. 2) The later broke it, but didn’t retest it. If they discovered that they broke it, and didn’t fix it, that falls in the previous bucket. More likely, they didn’t discover that they broke it.
Regression testing is an important element of continuous integration, and from a cost perspective, the primary reason to automate testing as much as possible. Technically, as much as practical – if it is cheaper to test manually, test manually. As the code base builds, so does the test suite. And a simple addition to the check-in policy is all that is needed.
Without regression testing: If it doesn’t work, don’t check it in (and therefore don’t release it).
With regression testing: Also, if it breaks something else, don’t check it in without fixing the other thing (and therefore don’t release it).
The team we worked with before was working on a mammoth code base with over 100 developer-years and hundreds of thousands of lines of code. They had such a huge backlog of outstanding bugs because 1) there were another 5 to 10 developer-years of new functionality being added to the code every year, and 2) they had no practical way to discover or prevent regression bugs. The developers rarely checked in code that didn’t work, but regularly introduced errors or unearthed latent errors in the existing code. And effort spent fixing those errors (after they were discovered) introduced more errors. It was a vicious and never-ending cycle. [Ed: Here's the case study we published on integrating testing automation.]
When they adopted the continuous integration philosophy, they included regression testing as part of their required process for release – no existing tests could be broken. And they’ve been able to start dedicating time to addressing old bugs – some of them years old with confidence. They’ve also been able to refactor the existing code so that it reduced the cost of ongoing work, allowing them to accelerate the improvement of their quality levels. It is a great story.
Failure to regression test is not a characteristic of an agile process.
Conclusion
There’s no disputing Justin’s bad experience. We believe that a poorly delivered product is a result of poor team execution, not poor process decisions.
Latest Articles
Comments on all Articles
