Barrier To Agile Development

Why don’t more companies and teams use agile development techniques? We know some teams just aren’t aware of them – although that list is getting shorter every year. The benefits of iterative development over waterfall development are pretty well established. I don’t believe I’ve seen a study that shows that waterfall is more effective. Do people refuse to believe in the data? Or maybe they are unable to believe.

Inspiring This Article

Mishkin Berteig just wrote another article inspiring this one. He’s done it before, when talking about code debt and quality, and we’ve highlighted his article on the core elements of an agile approach. Generally speaking, you should just read his blog.

His most recent article talks about the non-negotiability of software quality when pressed for time. Although the comments thread seems to be going off on a tangent about the iron triangle of project negotiations (scope/content, cost, and time), the article is pushing a different idea. The idea that time already spent is a sunk cost.

The idea of sunk costs, also know as “don’t throw good money after bad” is difficult for most people at a psychological level. We often feel like making just a little more investment of time, work or cash will turn an ailing effort around. Unfortunately, a traditional waterfall or phased-based approach to project work increases the psychological pressure to continue working, even when the project is “bad”.

Mishkin Berteig

The point that Mishkin seems to be driving at is that by using an agile process, you dramatically reduce the sunk cost psychological issue. One thing Alan Cooper said about agile processes that really stuck with me is that agile processes are optimized to deal with change. His comment was in the context of a left-handed compliment, that agilists presume that change is inevitable because it is impossible to know in advance what needs to be done. That is an interesting debate, and even mentioning it here is likely to cause our discussion to go off on a tangent.

Taking those two ideas together – it makes sense that people who anticipate and plan for change will not incur a psychological penalty for changing their solution in response to changes in requirements. It makes for good support to Mishkin’s argument. Waterfall processes make change harder because of sunk costs. But there’s more.

Cognitive Dissonance

OK, now the picture of a brain scan might make more sense for this article.

Cognitive dissonance is a psychological term which describes the uncomfortable tension that may result from having two conflicting thoughts at the same time, or from engaging in behavior that conflicts with one’s beliefs.

Wikipedia

The reason that people feel uncomfortable about sunk costs is rational. Making decisions based on sunk costs is irrational. Here’s how cognitive dissonance can play out in a waterfall project:

1. We know we’re good at what we do. We gathered requirements and we’re implementing a solution based on those requirements – and we’ve developed a test plan to validate that we actually did what we set out to do. In short – we rock.
2. We’ve invested a lot of time implementing what we’ve done so far.
3. We’ve just been told that what the customer really needs is something else.
4. If we accept that the customer needs “something else”, then that means that we wasted a bunch of time and money doing the wrong thing. The delays (time is not negotiable – see Mishkin, we get it) may doom this product to failure even if we do fix it. This of course means that we are a lousy product team.

Cognitive dissonance is introduced between items 1 and 4 in the list above. We can not both rock and be lousy. These two ideas are mutually exclusive. In Mistakes Were Made, But Not By Me (linked in the ‘Recommended Reading’ box at the end of this article), Carol Tavris and Elliot Aronson describe this situation and its global ramifications in a way that will enlighten and frighten you. A quick quote from Dr Cathy Goodwin that sums up the central idea behind the book:

Why do people refuse to admit mistakes – so deeply that they transform their own brains? They’re not kidding themselves: they really believe what they have to believe to justify their original thought.

And that’s what we have here. Cognitive dissonance prevents people from accepting that change is important in a waterfall project, because they have conspicuously avoided change for the life of the project. It almost seems like they go out of their way to make sure that they don’t accidentally uncover changes until the project is complete. And the higher the waterfall, the more vested interest, the more likely it is that change is needed, and the greater the cognitive dissonance.

Cognitive Dissonance Prevents Adoption of Agile Development

Not only does the notion of cognitive dissonance explain why people freak out about sunk costs, but it explains why waterfall projects are so terrifically bad when they fail.

Now take this tactical assessment and apply it to the longer term.

1. We’re good at software development.
2. We run waterfall projects, we have check-gates, and other processes to assure that we get the job done right.
3. Sure, many of our projects fail. And maybe our “successes” are not as big as we initially planned them to be.
4. Agile development would make this better. The CHAOS report, many other companies, and many thought leaders in the space all seem to agree. We don’t do agile. We’re really not good at software development.

We’re introducing cognitive dissonance again. We can’t be good at software development if our process choice was the wrong one.

Catch and Release

There’s a catch to cognitive dissonance. And this is the striking part. It isn’t the case that rational people see the data and don’t like the conclusions to which it leads. Cognitive dissonance is a subconscious process. People are not aware. In an interview on NPR’s Science Friday podcast, Dr. Aronson referenced studies with brain scans that demonstrated the dissonance affect.

How big is this affect? In the podcast, Dr. Aronson refers to multiple cases where prosecutors discover that innocent people are incarcerated (and will continue to be incarcerated). They have, for example, DNA evidence that the convicted person could not have committed the crime. And those prosecutors would rather (subconsciously) let the innocent person rot in jail for the rest of his sentence than accept that he was wrongly convicted. That’s powerful stuff!

The pleasure centers of the brain of an affected individual are rewarded when they resolve the dissonance by ignoring or discounting the “conflicting idea.” We subvert the possibility of being wrong, at a subconscious level, and our brains reward us for it. That’s the catch.

So what do we do? Our only hope is to consciously accept the premise that our old ideas might be bad. That we might in fact, not be good at software development (or whatever). By making the change to acceptance of the new idea – and perhaps even experiencing contrition about it – we also will satisfy the pleasure centers of our brains. That’s the release.

Summary

Cognitive dissonance explains why people feel bad about sunk costs. The effect is amplified with waterfall processes.

Cognitive dissonance also exists as a barrier to adopting agile processes. People experience cognitive dissonance when exposed to data that what they’ve been doing with waterfall processes would be improved by changing to agile processes. These people resist changing how they run their projects as much as they resist the acknowledgment of changing needs during their projects.