Archive of Testing Articles

Martin Fowler has identified the key process elements of making Continuous Integration work. You could even argue that they are the elements that define Continuous Integration (done correctly). We include his list and our thoughts below:

Continuous Integration is the software development and quality process where all team members merge their code and verifies it frequently - at least daily. This verification project includes both an automated build process and automated testing. The main benefits of continuous integration come from risk-reduction and cost-reduction.

Most teams think about testing in terms of code coverage - what % of the lines of code are covered? What matters to our stakeholders is how well the software works. More precisely, how well does the software let the users work? We should be targeting our quality message in terms of use cases, because that matches their perspective and context.

A grass roots campaign has been started by Peter Provost to get Microsoft to include unit testing support included with all versions of Visual Studio 2005 (VS). Currently, Microsoft is only including it with Visual Studio Team System (VSTS) versions of VS. This looks to be a great example of a killer feature in a product providing so much surprise and delight that people are demanding that it be universally available. This is also a great example of market segmentation by Microsoft. The irony is that there is an open source alternative that makes the opportunity cost very low, and yet people are still clamoring. Let’s see why.

We can reach the next step in our software process evolution by automating much of our process. Flying squirrels evolved a technique* to quickly move from one tree to another without all the tedious climbing and dangerous running. Software teams that automate their processes achieve similar benefits. Automation allows us to increase efficiency while improving quality. And we spend less time on tedious and mundane tasks.

We’ve talked about the value of using whitebox testing in our Software testing series post on whitebox testing. What we haven’t explored is how to make sure we are creating the right tests. We have to validate our tests against the requirements. This post shows where the flaw is in the typical whitebox testing process, and how to fix it.

A reader emailed us with the comment, “It’s been my experience that developers can’t test their own code.” Their problem was that they were missing a link in the software development chain (missing a step in the process).

Very large and complex systems can be very difficult and expensive to test. Often, we inherit legacy systems with multiple man-years of development effort already in place, in the field and of unknown quality. With these systems, there are frequently huge gaps in the requirements documentation. Pairwise testing provides a way to test these large, existing systems. And on many projects, we’re called in because there is a quality problem.

The software development process for most companies has a flow - gather requirements, design, implement, test, release. There can be feedback loops, iterative cycles, spirals or waterfalls, but they all have these steps. When teams “freeze the code” and submit to test, they are creating their own mini-ice age and dooming themselves to extinction.

Should we test our software? Should we test it more?

The answer to the first question is almost invariably yes. The answer to the second question is usually “I don’t know.”

We write a lot about the importance of testing. We have several other posts in our series on software testing. How do we know when we should do more automated testing?

We’ve presented an example of the software development process across several posts over the last two weeks. In this post we tie them all together, showing the steps in process order.