Foundation Series: Software Process (Waterfall Process versus Incremental Process)

A software process is the set of activities required to create software. This process can be defined with very precise steps, roles and responsibilities. The process can also be defined with a more fluid set activities in pursuit of concrete, high level objectives. Or software can be created without explicitly defining or following any process at all. The steps are always present, even if not conciously managed or defined.

A common framework is needed to compare processes and their merits. All software development includes the following three steps in some combination:

1. Decide what to do.
2. Do it.
3. Deliver it.

It sounds like an overly simplified list; Decide, Develop, Deliver. We’ll see that it isn’t.

Different process styles are like different dance styles.

Different processes will put different levels of emphasis and effort into each step. Each processes may repeat steps, or take them in a different order.

We can think of each different process as a different style of dance. They are alike and unalike. There are people who prefer one style to another, and there are specialists in one style and generalists across styles.

For any particular song, some styles of dance are intuitive and comfortable while others are awkward. The same is true of applying a software process to a given development scenario.

What works for a global team on a multi-year enterprise software project may not work well for a start-up creating a consumer website. The process that works best for an engineering firm developing embedded software for a pace maker may be a nightmare to apply to an open-source project for creating an RSS reader.

Software processes are usually categorized as waterfall or iterative – although I first heard these categories from proponents of iterative processes (the “new” way) when they were getting a lot of press in the late 90’s. Bias aside, the analogy works.

Waterfall process
A waterfall process is one where all of the work is done by the team, and when it is complete, it is delivered to their customer(s). With a waterfall, the water doesn’t do anything useful until after it falls off the end of the cliff – and by then it’s too late to climb out of the barrel if you change your mind. You don’t know if you’ll hit a rock until you reach the bottom of the waterfall. The larger the waterfall, the larger the risk, and the bigger the mess if you do hit a rock.

The same is true for waterfall software processes – we decide what to do, we develop the solution, and then we deliver it. We don’t revisit our decisions. Once the decisions are made, we’re in the barrel and falling, and we won’t know if our software will succeed until after we deliver it. The risk of failure is proportional to the amount of money and time spent before delivering the software to our customer.

Incremental process
An incremental process is one where we will deliver portions of the software in the smallest reasonable subsets of the ultimate functionality. We will have multiple deliveries to our customer. We will revisit the assumptions and prioritizations for each “next” delivery when we complete each “previous” delivery.

A sailing analogy works well for this approach. To sail in a straight line from start to finish, we set a heading, H1, and begin sailing. After a period of time, we check to see where we are (invariably, we’ve drifted off course a little), correct our course and set a new heading, H2, and continue sailing. We keep doing this (checking position, adjusting heading, sailing some more) incrementally until we reach our destination.

With incremental software development, we do the same thing as when sailing. First we decide on what to do and what to do first. Then we develop and deliver the first set of functionality. We then revisit our decisions (including getting feedback on what we’ve just delivered) and decide what to do next. We then develop and deliver the next set of functionality. We repeat until all of the desired functionality has been delivered.

Which process is best?
Picking the best process, like picking the best dance depends on circumstances. Let’s look at a few “real world” factors that can drive our decisions.

Budgeting and planning (winner: waterfall process)
With a waterfall process, we decide at the start exactly what we intend to accomplish. We can therefore scope and schedule the project. We can also determine staffing and costs. Budget decisions are easy – IRR and ROI can be calculated – we can calculate expected values for both costs and (forecasted) benefits. With an iterative process, we’re saying “I reserve the right to change my mind later.” We fully expect to change the scope of delivery mid-project. We know we will learn more about how to forecast the benefits after each incremental release. Because we know about, and even desire future changes, we can’t reasonably estimate ROI. We can “fix” the budget (aka timebox the project), but we can’t predict the value we will achive within the time allotted.

Return on investment (ROI) (winner: incremental process)
Incremental processes get the nod for return on investment because they start creating value earlier. Imagine software that can be delivered in two releases, where each release provides features that generate $10,000 per month in value. Each release takes 6 months. With waterfall delivery, there will be no value creation until after 12 months – and then it will generate $20,000 per month. With incremental delivery, the first release will generate $10,000 per month from month 6 to month 12, and then it will generate $20,000 per month for the next 12 months. The extra value from incremental delivery is the $60,000 earned from the first release while the second release is being developed.

Delivering a subset of functionality (winner: it depends)
Some projects can’t tolerate partial delivery. A pace maker that only regulates pulse, but can’t be recharged is not useful. However, an RSS reader without archiving can be released now with feed-reading, and updated in the next release to include archiving. Many times we run into users who say things like “Until it does everything, I won’t use it.” This is most common when replacing a legacy application. Users are accustomed to a particular workflow, and have expectations about what they should be able to do. Change management is the discipline that focuses specifically on helping users manage changes in the way they do their work, and the tools they use to do it. We can often help users make a transition, even from a legacy system, in incremental steps. When we can’t, a waterfall style delivery is usually better. The key thing to remember about incremental delivery is that the increments are as small as possible – which may mean “deliver it all in the first release.”

Are there other factors that we’ve overlooked? Add a comment and tell us about them.

Check out the index of the Foundation Series posts for other introductory articles.