Learn to Fly with Software Process Automation

We can reach the next step in our software process evolution by automating much of our process. Flying squirrels evolved a technique* to quickly move from one tree to another without all the tedious climbing and dangerous running. Software teams that automate their processes achieve similar benefits. Automation allows us to increase efficiency while improving quality. And we spend less time on tedious and mundane tasks.

Benefits of process automation

Tim Kitchens has a great article at developer.* where he highlights the benefits of process automation. Here are our thoughts on the benefits he lists:

• Repeatability. The first step in debugging software is the isolation of variables. A repeatable build process eliminates many variables, and likely many hours of wasted effort.
• Reliability. A repeatable process eliminates the possibility of us introducing errors into our software by messing up a step in the build.
• Efficiency. An automated task is faster than a manual task.
• Testing. Reductions in overhead of building and testing allow us to test more frequently.
• Versioning. The scripts that drive our build process are essentially self-documenting process documents. And tracking versions of the scripts provides us with precise records of the process used for prior builds. This documentation, and re-use of it can reduce the cost of running our projects at any CMMI level.
• Leverage. We get much more efficient use of our experts’ time – they spend less effort on turn-the-crank processes and more effort on writing great software.

What and when should we automate?

The short answer is automate everything, unless there’s not enough ROI. We have to examine each process that we use to make a final decision – some automation will not make sense due to uncommon situations. Also, if we’re nearing the end of an existing project, there is less time to enjoy the benefits of automation, so we may not be able to justify the costs. We may be under pressure to deliver ROI in a short payback period. We would suggest exploring the automation of the following activities:

Automate the build process

Most people underestimate the benefits of an automated build. The obvious benefit is time savings during the normal build cycle. Imagine the build takes an hour, happens monthly, and usually happens twice per month. Two hours per month doesn’t seem like a lot of savings. However, chasing down a bug caused by the build process is at best expensive, and at worst nightmarishly expensive (because we aren’t looking in the right place to find the problem). Use an estimate of the probability of this happening to the expected value calculation for the savings.

The largest potential benefit of an automated build is changing the way we support our customers. Monthly builds aren’t scheduled because the business only wants updates once per month. They are scheduled at a monthly rate because that’s a balance someone has achieved between the cost-of-delivering and the cost-of-delaying a delivery. When we automate our delivery process, we dramatically reduce the cost of delivery, and can explore more frequent release schedules.

Automate unit testing

We significantly improve the efficiency of our team at delivering by shortening the feedback loop for developers. On a Utopian dev team, we would run our test suite as often as we compiled our code. Realistically, developers should run relevant automated whitebox tests every time they compile. They should run the suite of whitebox tests every time they promote code. And an automated process should run the full suite against the latest tip on a nightly basis (to catch oversights). It would be great if the check-in process initiated an automated test run and only allowed a promotion if all the tests passed.

Automate system and functional testing

End to end and blackbox tests should be automated next. These are the big picture tests, and should be run nightly on a dedicated box against the latest code base. We’ve had the most success with teams that used a nightly testing process, which sent an email with test results to the entire team whenever results changed. We’ve had the pleasure of working with a team that included performance testing on the nightly runs, and reported statistically significant improvement or degradation of performance.

Documentation

Generate tactical documentation whenever possible. Use javadoc or the equivalent to automatically generate well formatted and organized reference materials for future developers.

Marginally relevant reporting

If our team is asked to report metrics like lines of code, cyclomatic complexity, code coverage, etc. We should automate this. This work is the definition of tedium, while presenting tenuous value to the manager who requested it. If we can’t convince someone that they don’t want this data, we should at least eliminate the pain of creating it.

Code coverage statistics can provide better than nothing insight into how much testing is being done, or how much functionality is exercised by the test suite. But code coverage metrics have the danger of false precision. There’s no way to say that a project with 90% code coverage has higher quality than a project with 80% coverage.

Conclusion

Automation makes sense. We save time, increase quality, and ensure a more robust process. We also spend less time on turn-the-crank activities and more time creating differentiated software.

*Technically, they don’t fly – they fall. With style.