Tyner Blain » Test Automation

You Are Creating Bugs In Your Software

Scott Sehlhorst — Tue, 15 Jan 2008 03:19:16 +0000

No matter how good your quality process is, you are introducing bugs. This article reviews the places where bugs are introduced in the software development process (from stakeholders to users), and reviews ways to address those bugs.

One way to identify the sources of bugs is by listening to how people tell you about the bugs.

“That is not what I want”
“That is not what I said”
“That is not what I meant”
“That is not how it is supposed to work”
“That is not what they meant”
“That is not what I want it to do”

Visually, you can think of the development process (regardless of methodology) like the following flow

Each of the six sources of bugs is labeled in the above flow. The article, Where Bugs Come From, goes into a detailed explanation of each source of bugs. This article looks at what you can do to improve the ability of your process to reduce these bugs. I have been a stakeholder, a product manager, a business analyst, a developer and a tester. And I’ve been guilty of all of these offenses at one time or another. When I say you, I also mean me. So chill.

“That is not what I want”

Stakeholders introduce bugs into the process by being unable to define what they want. Some agile methodology proponents argue that as a stakeholder, you can’t know what you want until you have something you don’t want in front of you. In other words, those agilists are giving up on trying to prevent these errors – they claim it is futile.

You can change your mind. The team could be dealing with “That is not what I want now” – and change is something that can not be prevented – although the impact of change can be mitigated. Iterative development and course-correction through getting immediate feedback is one way to do that. Prototyping is another method of course correction. Prototypes can also be used to elicit implicit requirements.

But that still avoids trying to directly address the issue of you not effectively describing what you actually want. There are ways to improve your team’s engagement with you, so that you actually say what you want. First, an emphasis on valuable requirements drives critical thinking about why you need a particular capability or feature. One approach to achieve valuable requirements is to question their justification.

The key to making this work is to make sure you are approving your requirements. And watch out for the anti-patterns in the article.

“That is not what I said”

Business analysts, product managers, and requirements analysts introduce bugs into the process through misunderstanding the stakeholders. Using the techniques above to stay on top of changes in stakeholder needs is not enough. Even when you effectively engage your stakeholders so that they are self-aware and can articulate exactly what they need and why, you still have a problem.

You can improve how you listen, and you can improve how you document what you heard. Make sure that you write complete, consistent and unambiguous requirements. While you’re at it – make sure you are writing your requirements correctly.

“That is not what I meant”

Developers can misinterpret requirements. While there are situations where a requirement is adequately expressed, but the developer is not capable of understanding, that is not the norm. Developers are smart. The onus is on the person writing requirements to make sure that they are written for the audience. So, even though the source of these bugs appears to be the developers, it is actually your documentation. If your documents need to span cultural chasms, then that’s your reality.

Make sure you write your requirements unambiguously, so that they are not misinterpreted. Make sure that they are feasible requirements, and that they do not specify design. The requirements also need to be concise without being overly terse.

“That is not how it is supposed to work”"

Developers should test what they create. Given an understanding of what you believe the code is supposed to do, you should test that it actually does it. This is the stereotypical bug – we asked for X, and it doesn’t work. Continuous integration is the most effective technique for addressing bugs that are introduced at this point in the process.

In more complex situations, you can apply more advanced techniques to assure that your testing is complete.

“That is not what they meant”

The testing team can also misinterpret requirements. This is a source of false positive bugs – even when the developers properly interpret the requirements, a bug may be lodged against their code when there is no bug. That’s why we have the ability to mark issues as “not a bug” when closing them. Apply the same rules for well-written requirements and you will also address this issue. Further, make sure that your requirements can be measurable, or verifiable. We recently wrote more about why you should make requirements testable.

“That is not what I want it to do”

User education is the key here. Users should be encouraged to provide feedback about your product. You don’t want to discourage feedback, you want to minimize the number of false positives. False positives come from users misinterpreting how the application is behaving – a sign of poor design. False positives also come from users not understanding what the application can (or should) do.

You can address design issues by focusing on design with the user in mind. And you can educate the users through a combination of training and with a novel approach to documentation. Document the product in terms of what users are trying to accomplish, in the context of how they are trying to do it.

Conclusion

There are a lot of people involved in the software development lifecycle. As people, we introduce bugs. As informed software professionals, we also know how to address many of them.

Test Smarter, Not Harder – Part 3

Scott Sehlhorst — Thu, 03 Jan 2008 04:00:06 +0000

This series is a reprint of an article by Scott Sehlhorst, written for developer.* in March 2006. A recent article on dailytech about “new” methods for software testing points to some very interesting research by the NIST (the National Institute of Standards and Technology) Information Technology Lab. We’ve split the original article into three articles to be more consistent in length with other Tyner Blain articles.

This is part 3 of a 3 part series.

Article Overview

Part 1 of this article explores the problem of getting good testing coverage of complex software.
Part 2 of this article discusses solution approaches (including those identified in the NIST research).
Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

How to make it even better

When we don’t know anything, or don’t apply any knowledge about our application to our testing strategy, we end up with far too many tests. By applying knowledge of the application to our test design we can greatly reduce the size of our test suite. Tests that incorporate knowledge of the application being tested are known as whitebox tests.

Map out the control dependencies

In our previous examples, we applied no knowledge of the interactions of controls (or the interactions within the program of having made selections in the controls. If we consider a visual map of the controls and their possible relationships, it would look like the following diagram.

There is a possibly-relevant connection between the selections in every pair of controls. We have designed our testing around the lack of knowledge that is clearly visible in the diagram.

It is likely that we can rule out some of the dependencies, but possibly not all of them. Our approach should be conservative – only remove those dependencies that we know don’t exist. This knowledge comes from an understanding of the underlying application. Once we remove these links the diagram will look like this:

This clarified mapping allows us to reduce the size of our test suite dramatically, because we’ve identified the independence of many controls. In an ideal case, the result will be two or more completely disconnected graphs, and we can build a set of tests for our suite around each separate graph. As the diagram above shows, we do not have two completely independent graphs. We can take a testing approach as shown in the following diagram:

We’ve grouped all of the controls on the left in a blue box – these controls will be used with the N-wise generation tool to create a set of tests. The grouping of controls on the right will also be used to generate a set of tests.

In this example, we reduce the number of tests required by a significant amount when order matters.

Also note that we increase the number of tests required when order doesn’t matter, if we have any overlapping controls (if the graphs can’t be separated). When the graphs can be separated, this reduces the amount of testing even if order is irrelevant.

The key to separating the graphs is to make sure that all controls only connect to other controls within their region (including the overlapping region).

Eliminate equivalent values from the inputs

When we know how the code is implemented, or have insights into the requirements, we can further reduce the scope of testing by eliminating equivalent values.

Consider the following example requirements for an application:

And two variables that we are evaluating in our testing – imagine that they are controls in a user interface (or values imported from an external system).

Which we can collapse for testing purposes into:

This consolidation of equivalent values reduces the number of tests we need to run. For our simple pairwise test, we reduce the number from 18 to 12. When there are more controls involved, and when we are doing N-wise testing with N=3, the impact is much more significant.

Conclusion

We can test very complex software without doing exhaustive testing.

Random sampling is a common technique, but falls short of high quality goals – very good quality requires very high quantities of tests.

Pairwise testing allows us to test very complex software with a small number of tests, and reasonable (on the order of 90%) code coverage. This also falls short of high-quality goals, but is very effective for lower expectations.

N-wise testing with N=3 provides high quality capable test suites, but at the expense of larger suites. When the order of inputs into the software matters, N-wise approaches become limited in the number of variables they can support (fewer than 10), due to limitations of test-generation tools available today.

We can apply knowledge of the underlying software and requirements to improve our testing strategy. None of the previous techniques require knowledge of the application, and thus rely on brute force to assure coverage. This approach results in conceptually redundant tests in the suite. By mapping out the grid of interdependency between inputs and subdividing the testing into multiple areas we reduce the number of tests in our suite. By removing redundant or equivalent values from the test suite we also reduce the number of tests required to achieve high quality.

Article Overview

Part 1 of this article explores the problem of getting good testing coverage of complex software.
Part 2 of this article discusses solution approaches (including those identified in the NIST research).
Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

One of the interesting points, not addressed by this article, and not

Test Smarter, Not Harder – Part 2

Scott Sehlhorst — Fri, 28 Dec 2007 04:00:03 +0000

This is part 2 of a 3 part series.

Article Overview

Part 1 of this article explores the problem of getting good testing coverage of complex software.
Part 2 of this article discusses solution approaches (including those identified in the NIST research).
Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

Solving the problem

There are a lot of applications that have millions or billons of combinations of inputs. They have automated testing. They have solutions to this problem. We just finished discussing how impractical it is to test exhaustively, so how do companies test their complex software?

Random sampling

Early on in the software testing world, someone realized that by randomly checking different combinations of inputs, they would eventually find the bugs. Imagine software that has one million possible combinations of inputs (half as complex as our previous example). Each random sample would give us 0.000001% coverage of all possible user sessions. If we run 1000 tests, we would still only have 0.001% coverage of the application.

Thankfully, statistics can help us make statements about our quality levels. But we can’t use “coverage” as our key measurement of quality. We have to think about things a little bit differently. What we want to do is express a level of confidence about a level of quality. We need to determine the sample size, or number of tests, that we need to run to make a statistical statement about the quality of the application.

First we define a quality goal: we want to assure that our software is 99% bug free. That means that up to 1% of the user sessions would exhibit a bug. To be 100% confident that this statement is true, we would need to test at least 99% of the possible user sessions, or over 990,000 tests.

By adding a level of confidence to our analysis, we can use sampling (selecting a subset of the whole, and extrapolating those results as being characteristic of the whole) to describe the quality of our software. We will leverage the mathematical work that has been developed to determine how to run polls.

We define our goal to be that we have 99% confidence that the software is 99% bug free. The 99% level of confidence means that if we ran our sample repeatedly, 99% of the time, the results would be within the margin of error. Since our goal is 99% bug free code, we will test for 100% passing of tests, with a 1% margin of error.

How many samples do we need, if there are one million combinations, to identify the level of quality with a 99% confidence, and a 1% margin of error? The math for this is readily available, and calculators for determining sample size are online and free. Using this polling approach, we find that the number of samples we require to determine the quality level with a 1% error and 99% confidence is 16,369.

If we test 16,369 user sessions and find 100% success, we have established a 99% confidence that our quality is at least at a 99% level. We only have 99% quality, because we have found 100% quality in our tests, with a 1% margin of error.

This approach scales for very large numbers of combinations. Consider the following table, where our goal is to establish 99% confidence in a 99% quality level. Each row in the following table represents an increasingly complex software application. Complexity is defined as the number of unique combinations of possible inputs).

We can see that the very few additional tests have to be run to achieve the same level of quality for increasingly complex software. When we have a modest quality goal, such as 99/99 (99% confidence in 99% quality), this approach is very effective.

Where this approach doesn’t scale well is with increasing levels of quality. Consider the quest for “five nines” (99.999% bug free code). With each increase in the desired level of quality, the number of tests we have to run grows. It quickly becomes an almost exhaustive test suite.

Each row in the following table represents an increasingly stringent quality requirement, with the complexity of the software staying constant at one million possible input combinations.

The random sampling approach does not provide a benefit over exhaustive testing when our quality goals are high.

Pairwise testing of input variables

Studies have shown that bugs in software tend to be the results of the combination of variables, not individual variables.

Consider a very simple laptop configuration page, having three selectable controls: CPU, Memory, and Storage. Each control has three possible values as shown in the table below.

We successfully pass tests of each of the different values available in the CPU control. However, we discover that the software test fails if the user selects a CPU value of “Consumer” and selects a Storage value of “Huge.” This highlights an unknown dependency between the CPI and Storage controls.

Pair-wise testing is designed to get coverage of every possible combination of two variables, without testing every possible combination of all the variables. For this example, there are 27 unique combinations of all of the selections. The following table shows the first 9 combinations. An additional 9 combinations are needed for each of the other CPU selections.

Exhaustive pair-wise testing will make sure that every unique combination of any two variables will be covered. The next table shows the combinations for this example.

With just 9 tests, we are able to exhaustively cover every unique pair of CPU and Memory, CPU and Storage, and Memory and Storage. Pair-wise testing allows us to get full coverage of the combinations of every two variables, with a minimal number of tests.

Pair-wise testing not only gives us full coverage of every pair of values, it also gives us (redundant) coverage of every single value for each control.

If we look back at our previous examples of laptop-configuration, we can calculate the numbers of tests required to get full pair-wise coverage. For the entry level laptop configurator, there are 32,256 possible unique combinations of inputs. We can test every unique combination of two variables with 31 tests. For the high-end laptop configurator, there are 2,322,432 unique combinations of inputs. We can test every unique combination of two variables with 36 tests.

N-wise testing

The concept of pair-wise testing can be extended to N-wise testing – looking at every combination of N possible inputs.

The following table shows how many tests are required to get full coverage of each N-wise combination of inputs for both the low-end and high-end laptops configurators.

This is a much more manageable situation. Using N=3, we get 179 tests versus 2.3 million tests for exhaustive coverage! Existing studies have consistently shown that N=3 creates on the order of 90% code coverage with test suites, although the number will vary from application to application. We will use N=3, based on practical experience that N=4 tests rarely uncover bugs that were missed with N=3.

This approach only works when users are forced to enter values in a proscribed sequence, and in cases where the sequence of entry is irrelevant. This set of tests won’t give us representative coverage of what the users will do when they are allowed to make selections in arbitrary but relevant order.

Order relevance and statistical testing

There’s been an assumption implicit in all of our calculations so far – that the order of selection in the controls is irrelevant. The available N-wise test calculation tools do not incorporate order of selection in their permutations – explicitly, they assume a fixed order of operations. When we test an API we have control over the order of processing – there are a fixed number of arguments, in a fixed order. N-wise testing People, however, do not always interact with the controls in a fixed order. And web service architectures may not be able to depend upon a predetermined sequence of events.

With 5 controls in an interface, we have 5! (factorial – see definition) or 120 possible sequences in which selections can be made by a person. Although the user interface may incorporate dynamic filtering that prevents some subsets of out-of-sequence selection, N-wise testing is blackbox testing, and will not have access to that information.

For an interface with M possible controls, each script created by an N-wise test generator will have to be tested in M! sequences to get exhaustive coverage. If the controls are split across multiple screens, then we can reduce the number of sequences. For example, if there are 5 controls on the first screen, and five controls on the next screen, instead of considering 10! (3.6 million sequences), we can consider all first-screen-sequences in combination with all second-screen sequences (5! * 5! = 120 * 120 = 14,400 script sequences).

In our example laptop configurators, there are 11 and 13 controls (all on the same page) for the low-end and high-end laptops respectively. This would imply 11! and 13! possible sequences (40 million and 6 billion).

We do not need to do exhaustive coverage of the sequencing permutations. An N-wise test is specifically analyzing the interdependence of any combination of N controls. As a lower-bound, we would only need N! sequences for each generated script. So our 179-script suite for the high-end laptop (with N=3) would need 3! (6) * 179 = 1,074 scripts to cover the product.

Here’s the table for the lower-bound of scripts required to account for different values of N for both laptop-configurators.

This is a lower bound, because it assumes a perfect efficiency in combining unique sequences of each group of N controls. Existing N-wise testing tools do not (to the author’s knowledge) take order of operations into account. For N=2, this is trivial – just duplicate the set of tests, in the exact reverse order.

We can take order of operations into account by treating the sequence as an additional input. We use the mathematical formula “X choose Y” which tells us the number of different combinations of Y values from a set of X values. The formula for calculating “X choose Y” is X!/(Y!*(X-Y)!) where X is the number of inputs and Y is the dimension of the desired N-wise test.

Here’s the table of the number of combinations for each N, for both the low-end and high-end laptop configuration screens we’ve been discussing.

Here are the values, generally, for varying numbers of inputs.

(click for larger version)

We would then calculate the N-wise testing using a value of N+1 as an input to the test-generation tool, and include the number of unique sequences as if it were a control input.

Unfortunately, we don’t have a solver capable of handling single dimensions larger than 52. This limits our ability to create a test suite for N=3 to a maximum of 7 controls.

To show the impact of sequencing on the test suite, consider an interface with 7 controls, each having 5 possible values. N=3 would require 236 tests if order is irrelevant. We then include sequence of selection as a parameter (by adding an 8^th control with 35 possible values, and testing for N=4), In this case, N=3 (with sequencing) requires 8,442 scripts. Our theoretical lower bound would be 236 * 35 = 8260.

Article Overview

Part 1 of this article explores the problem of getting good testing coverage of complex software.
Part 2 of this article discusses solution approaches (including those identified in the NIST research).
Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

Test Smarter, Not Harder – Part 1

Scott Sehlhorst — Wed, 26 Dec 2007 04:00:05 +0000

This is part 1 of a 3 part series.

Article Overview

Part 1 of this article explores the problem of getting good testing coverage of complex software.
Part 2 of this article discusses solution approaches (including those identified in the NIST research).
Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

Introduction

In this article we focus on a method to minimize the cost of quality. Regardless of the method you use for calculating the cost of poor quality, minimizing the cost of assuring good quality is relevant to you. We will discuss techniques for minimizing the cost of creating and maintaining an automated regression testing suite for a software application.

Automated testing changed the software development process as much as the assembly line changed the manufacturing industry. We are assuming in this article that we already have automated testing as a part of our development process.

We open with a discussion of the problem. We will then discuss approaches to automated testing that are progressively better at minimizing costs. We hope you enjoy the article and would love to hear back from you with feedback and contributions.

The problems

Here is the sound bite version of the problems this article is designed to address

We aren’t meeting our quality goals.
We have a development team, regularly releasing new versions of our software.
We have a suite of automated tests we run on our software.
Our automated test suite may be large, but out product is complex. There’s no way to test every possible scenario. Bugs are still getting released to the field.

Not solving the problems

Two solutions that we have to consider are to test nothing, and to test everything. We would consider testing nothing if we can’t afford to test the software. When people don’t appreciate the complexities of testing or the limitations of automated testing, they are inclined to want to test everything. Testing everything is much easier to said than done.

We can’t afford to test it

I was able to attend a training session with Kent Beck a few years ago. I was also honored to be able to enjoy a large steak and some cold beer with him that night after the training. When asked how he responds to people who complain about the cost of quality, Kent told us he has a very simple answer, “If testing costs more than not testing then don’t do it.” I agree.

There are few situations where the cost of quality exceeded the cost of poor quality. These are situations where the needed infrastructure, test-development time, and maintenance costs outweighed the expected cost of having a bug. The expected cost is the likelihood (as a percentage) of the bug manifesting in the field, multiplied by the cost of dealing with the bug.

The techniques described in this article are designed to reduce the cost of quality, to make it even less likely that not testing is the best answer.

Just test everything – it’s automated

We’ve all been on at least one project or team where our manager has said

“I demand full testing coverage of the software. Our policy is zero tolerance. We won’t have bad quality on my watch.”

What we struggle with here is the lack of appreciation for what it means to have full coverage or any other guarantee of a particular defect rate.

There are no absolutes in a sufficiently complex system – but that’s ok. There are statistics, confidence levels, and risk-management plans. As engineers and software developers, our brains are wired to deal with the expected, likely, and probable futures. We have to help our less-technical brethren understand these concepts – or at least put them in perspective.

We may get asked “Why can’t we just test every combination of inputs to make sure we get the right outputs? We have an automated test suite – just fill it up and run it!” And we need to resist the urge to respond by saying “Monkeys with typewriters will have completed the works of Shakespeare before we finish a single run of our test suite!”

Complexity leads to futility

Consider a web page for customizing a laptop purchase. Take a look at Dell’s customize it page for an entry level laptop, if you’ve never configured a laptop online before.

The web page presents eleven questions to the user that have from two to seven responses each. Specifically, each decision the user has to make presents (2,2,2,2,2,3,2,2,3,4,7) choices. This is a simple configuration problem. The number of possible laptop configurations that could be requested by the user is the product of all of the choices. In this very simple page, there are 32,256 possibilities. The page for customizing Dell’s high end laptop at the time of this writing has a not-dissimilar set of controls with more choices in each control – (3,3,3,2,4,2,4,2,2,3,7,4,4). The user of this page can request any of 2,322,432 different laptop configurations! If Dell were to add one more control presenting five different choices, there would be over ten million possible combinations!

Creating a test suite that tries all two million combinations for their high end laptop could be automated, but even if every test took one tenth of second to run, the suite would take over 64 hours to run! And Dell changes their product offering in less time than that.

If we use a server farm to distribute the test suite across ten machines we could run it in about 6 hours. Ignoring the fact that we would be running this type of test for each customization page Dell has, 6 hours is not unreasonable.

Validating the two million results is where the really big problem is waiting for us. We can’t rely on people to manually validate all of the outputs – it is just too expensive. We could write another program, which inspects those outputs and evaluates them using a rules-based system (“If the user selects 1GB of RAM, then the configuration must include 1GB of RAM” and “The price for the final system must be adjusted by the price-impact of 1GB of RAM relative to the base system price for this model.”)

There are some good rules-based validation tools out there, but they are either custom software, or so general as to require a large investment to make them applicable to a particular customer. With a rules-based inspection system, we have the cost of maintaining the rules.

The validation rules are going to have to be updated regularly, as Dell changes the way they position, configure, and price their laptops regularly.

Since we aren’t Dell, we don’t have the scale (tens of billions of dollars of revenue) to justify this level of investment. The bottom line for us is that we can’t afford to exhaustively test every combination.

Article Overview

Part 1 of this article explores the problem of getting good testing coverage of complex software.
Part 2 of this article discusses solution approaches (including those identified in the NIST research).
Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

Is Agile Bad For Software Development?

Scott Sehlhorst — Tue, 17 Apr 2007 03:00:26 +0000

Last week, Ivan Chalif, a product manager / blogger, tapped into a thread criticising product managers for not adopting and espousing agile, or at least rapid-release techniques. In this article we look at Ivan’s comments and one of the articles that he referenced. We also share our own perspective and an alternative analysis of what may have happened.

Short Release Cycles are Bad

At the end of his article, Ivan references a well presented position on why short release cycles are bad that was published by Justin James on TechRepublic last June.

Ivan’s article basically says that agile approaches are fine for non-mission critical software. Justin’s article includes criticism of software he has used, and implies that the problems are due to the development methodologies used.

In a nutshell, they are both saying that short release cycles are bad because they result in bad software being released. Ivan qualifies his perspective by saying that for some applications, like “consumer web applications” that’s ok – because being bad doesn’t hurt as much.

Our perspective is a little bit different – short release cycles are the best thing ever. Immature or incomplete releases are bad. We define incomplete as meaning “not all of the must be requirements have been satisfied.” We define immature as meaning “quality, usability, and performance are not where they need to be for the audience.”

Ignorance is No Excuse

On the point of requirements, we’ll add that failure to implement an undiscovered / undocumented must-be requirement still qualifies as incompleteness. A product manager’s failure to document a critical requirement may absolve the implementation team from addressing and testing it – but the “entire team” is still on the hook for it.

In short, we don’t think agile approaches need to be forced to live in the “software that isn’t important to users” box. Agile techniques and approaches can be used on any project. Any project can succeed or fail, and any process can succeed or fail on a particular project. What is more important is the team. And unfortunately, the environment – people trump process, but politics trumps people. If your organization can not or will not support an iterative development cycle, your project will fail if you use an agile process.

Mission critical and non-mission critical software have mandatory requirements that must be satisfied. Must-be requirements, by definition, must be satisfied. A build of the software that does not satisfy those requirements is one that is incomplete, and should not be released for production use.

Note that we specified “production use.”

Anti-Agile Argument

We’ll take a point-counterpoint approach to one of the arguments raised by Justin. Our position is that his criticism, while anecdotally valid, should not be a criticism of the process, but of its poor execution.

Here is an example of what I mean: today, I made hotel reservations with a Hyatt hotel on their Web site.[...]Except there is one huge flaw: it does not use HTTPS! So not only is my credit card number being sent in plain text (along with my first and last name), but it is going to be stored in my browser’s auto-fill system in plain text!

Justin James

Is that a problem? Absolutely. Is secure transmission of user financial data a must-be requirement? Unquestionably. I explained to my mom that her credit card number is safer on the Amazon form than it is on the receipt at a restaurant. The whole ecommerce world is dependant on that assumption. The team that implemented the website Justin was using screwed up big time. But how did they screw up?

The team that built this Hyatt website screwed up in one of four ways:

They did not identify the security requirement.
They did not write code to satisfy the security requirement.
The code they wrote did not satisfy the security requirement.
They wrote code that broke a previously working solution to the security requirement.

This clearly can be blamed on the “entire team” but can not be clearly blamed on agile software development.

Is Agile To Blame?

Let’s look at each of the possible explanations from above in more detail…

They did not identify the security requirement

The first possibility – failure to identify the requirement, is a risk with every development process. Identification of the critical requirements is a matter of product manager competence (or product champion, or whoever on the team is gathering the requirements). More rigorous requirements gathering will identify more requirements. You don’t neccessarily have to have a big up-front requirements (BUFR) exercise before writing the first line of code. You do have to have “enough” requirements gathering. A BUFR process does not assure a complete set of requirements any more than an incremental process prevents it.

All teams will reach a point when they decide they have “enough” to risk developing against an incomplete spec. A product manager has to develop a sufficient breadth of understanding to make an informed decision about when “enough” is enough. That’s why we encourage defining the space with use case names and brief descriptions of use cases. Understanding the domain is the goal of this exercise, and it is not the same as defining all of the requirements up front.

This team failed to get an understanding of the domain, and overlooked something as obvious as the need for customers to have secure transmission of their credit card numbers. That is a failing of the product manager, not an agile approach. Note: we dismiss any process that says “don’t develop an understanding of the domain” as a bad one, agile or otherwise.

They did not write code to satisfy the security requirement.

Was the requirement mis-prioritized as being less than critically important? If so, blame the product manager again.

What if the requirement was defined, identified as a must-be requirement, and still wasn’t implemented? Who’s fault is it then? Did the team have poor execution, or did they follow a process that says “define the requirements, but don’t address them if you don’t feel like it?”

I’m not aware of an agile process that says you should release the software to production without implementing a solution to a known critical requirement. Definitely can’t blame this on agile. You could potentially blame it on an executive or project manager who said “release on date X, I don’t care if it’s done!” In which case, Justin just had bad timing, because any agile process would put this at the top of the list for the next release. If not, it isn’t agile.

The code they wrote did not satisfy the security requirement.

OK, now we’re looking at a team that 1) identified the requirement, and 2) attempted to implement it in their release, but 3) failed to test it (or released with a known but uncommunicated bug). I say uncommunicated because Justin didn’t get the message, even if someone on the team tried to send it.

Lack of testing and failure to communicate are not characteristics of agile processes. Just because you’re delivering code rapidly, managing the amount of work to be delivered in a timebox, does not mean that quality is ignored. That is certainly a choice that any foolish team (or irresponsible team) can make. But it is not unique to agile development, or even characteristic of agile development.

The agile projects that we’ve seen tend to have better quality than waterfall projects. Not because they are agile, but because the developers who are following an agile process have tended to (this is anecdotal data) have a greater appreciation for the benefits of continuous integration and quality in general. They live by the test cases. If it isn’t tested, it hasn’t been completed. And if it fails the tests, it hasn’t been completed.

There is a greater possibility that a waterfall project will allow developers to write code for months on end without creating tests – with the expectation that it will get tested in the end. We’ve worked with a client who’s timeline for a release of a key set of features included 3 months for development and one week for testing. When we asked “Where’s the time allocated to fix the code when you discover that it is broken?” we got the strangest looks. As an aside, in the year after we helped that team implement a continuous integration process, they’ve reduced their outstanding bug list by almost 80% and the team lead expects it to be cut in half again in the next month. They’re making great progress just by adding one agile technique (continuous integration). The team has to create, review, and pass tests before they check code into the trunk for the current release cycle.

Absolutely not a failing of agile software development – this would be a failing of the team to deliver quality product.

They wrote code that broke a previously working solution to the security requirement.

This is another variation of the previous situation. 1) They identified, scheduled, implemented and tested the functionality. 2) The later broke it, but didn’t retest it. If they discovered that they broke it, and didn’t fix it, that falls in the previous bucket. More likely, they didn’t discover that they broke it.

Regression testing is an important element of continuous integration, and from a cost perspective, the primary reason to automate testing as much as possible. Technically, as much as practical – if it is cheaper to test manually, test manually. As the code base builds, so does the test suite. And a simple addition to the check-in policy is all that is needed.

Without regression testing: If it doesn’t work, don’t check it in (and therefore don’t release it).
With regression testing: Also, if it breaks something else, don’t check it in without fixing the other thing (and therefore don’t release it).

The team we worked with before was working on a mammoth code base with over 100 developer-years and hundreds of thousands of lines of code. They had such a huge backlog of outstanding bugs because 1) there were another 5 to 10 developer-years of new functionality being added to the code every year, and 2) they had no practical way to discover or prevent regression bugs. The developers rarely checked in code that didn’t work, but regularly introduced errors or unearthed latent errors in the existing code. And effort spent fixing those errors (after they were discovered) introduced more errors. It was a vicious and never-ending cycle. [Ed: Here's the case study we published on integrating testing automation.]

When they adopted the continuous integration philosophy, they included regression testing as part of their required process for release – no existing tests could be broken. And they’ve been able to start dedicating time to addressing old bugs – some of them years old with confidence. They’ve also been able to refactor the existing code so that it reduced the cost of ongoing work, allowing them to accelerate the improvement of their quality levels. It is a great story.

Failure to regression test is not a characteristic of an agile process.

Conclusion

There’s no disputing Justin’s bad experience. We believe that a poorly delivered product is a result of poor team execution, not poor process decisions.

Insight Into Test Driven Development

Scott Sehlhorst — Wed, 13 Sep 2006 04:55:20 +0000

James Kovacs shares a great insight on software testing and the software testing process. His epiphany about test driven development makes it obvious to all of us why this technique is so powerful.

TDD

Test driven development is an approach to software development where you write the tests first, and the code second. This sounds ludicrous at first, but it really does work. Here’s James’ epiphany on the subject (emphasis added).

I didn’t understand why TDD practitioner were so zealous about writing the tests first. Why did it matter? I thought it was to ensure that some project manager doesn’t try to shave some time off the project by cutting the unit tests. Then I started doing some reading about TDD, poking around, asking questions, and trying it out myself. I discovered the real reason for writing tests first is that TDD isn’t about testing code, it’s about designing code.

A Tale of Two Epiphanies: TDD and Mocking, James Kovacs

These tests are unit tests, and they are black-box tests (or at least should be black-box tests – read James’ article).

Works Great With Structured Requirements

Not only is this a great way to approach design, but it is a great way to coordinate software development with structured requirements. A well-written requirement describes a need without specifying design. This description then works like a black-box contract. The implementer then must do something that meets the specification given. As James points out, TDD is very effective at creating black-box tests.

This makes sense, when you consider the outside-in approach that comes from starting with requirements. This outside in approach leads to API decisions (for a program or module or method). The API has to support all of the required possible inputs and outputs. Once an API is written, tests can be written.

Best Applied As Part Of Continuous Integration For Larger Teams

All of the tests will fail (because the code hasn’t been written). Then the implementer will write code until the tests pass. Using a continuous integration approach is a very effective way for entire teams to do this simultaneously on larger projects.

Faster Processes

We can also validate the design for completeness prior to creating the implementation. We can compare the generated APIs (and their rationalizations) with the software specification (or application requirements, or FRS… see alphabet soup article). This allows us to address some refactoring needs before the code has been implemented – saving time and money.

Conclusion

Use TDD as an outside-in approach that assures that the delivered software will meet the objectives of the requirements specification. It also saves time and money by supporting validation in advance of implementation.

John Henry, Manual Tester

Scott Sehlhorst — Fri, 18 Aug 2006 02:13:44 +0000

There’s a piece of North American folklore about John Henry, who was a manual laborer during the expansion of the railroads in our country. His job was being replaced by steam-driven heavy equipment, as the railroad industry applied technology to become more efficient. The same dynamics are happening today with manual testers. We need to make sure that manual testers avoid John Henry’s fate – read on to see why.

Manual and Automated Testing

Software can not be tested efficiently with only manual testing. And it can not be tested completely with only automated testing. James Bach writes an insightful article about how he worked his way to enlightenment over several years, and how he has incorporated this perspective into his training classes.

My understanding of cognitive skills of testing and my understanding of test automation are linked, so it was some years before I came to understand what I now propose as the first rule of test automation:

Test Automation Rule #1: A good manual test cannot be automated.

James goes on to explain that if a test can be automated, it is not a good manual test.

We’ve discussed repeatedly the benefits of automated testing, one example is The Code Freeze is Killing the Dinosaurs. We’ve also stressed the importance of doing functional testing where we discuss the dynamics and tradeoffs of using manual and automated approaches to test software.

Generally, we’ve approached the problem from the perspective of blackbox versus whitebox testing (and more details), and thought about our development process from the perspective of continuous integration as a means to deliver more efficiently.

We’ve not thought about it from James’ perspective before. Even with our improved approaches to automated testing, we are still no different than the inventor of the steam-hammer in Henry’s fable.

Enlightenment

James puts things in perspective:

Rather than banishing human qualities, another approach to process improvement is to harness them. I train testers to take control of their mental models and devise powerful questions to probe the technology in front of them. This is a process of self-programming. In this way of working, test automation is seen as an extension of the human mind, not a substitute.

He highlights a very interesting point – we are missing a key benefit of having manual testers – the ability to gather feedback on intangibles, heuristics, usability, affordances, and other elements that we might classify as design bugs.

We talk about the importance of usable software, and different approaches to creating great software. But we’ve never really talked about how to test for software greatness.

A Broken Model

In the 1970′s, when the American automotive manufacturers were getting their clocks cleaned by the newly exporting Japanese companies like Honda, Datsun (now Nissan), and Toyota, a lot of people thought they understood the problem. Many more people pretended there wasn’t a problem, until the US government bailed out Chrysler.

The experts decided that the main problem was that quality was not “Job One”, that American manufacturers ignored the advice of W. Edwards Demming, and the Japanese did not. Toyota’s lean manufacturing philosophy is credited with much of the success.

It is true that the oil crisis of the time gave the Japanese companies an opportunity to penetrate the US market with smaller, more efficient cars. But that crisis ended (about the time that the American companies killed the muscle car and began building more efficient cars). But the Japanese cars didn’t go away. Once gas prices dropped, they lost a differentiating element – but they maintained two others. Cost and Quality.

Target of Opportunity Enables Strategic Advantage

Cost matters. But it matters tactically, as in “I can afford $X right now – what are my choices?” Quality creates Loyalty. And loyalty is strategic. The Japanese manufacturers gain and continue to gain success after success and award after award in the automotive industry. Why? Because they have good quality.

Automated and Manual Testing

As an engineer, I know that we can specify tolerances, inspect components, and test assemblies to make sure that products are “within specification.” And all of this testing can be automated – just like automated software testing. But passing these tests doesn’t make a product good, it merely indicates that the product is “within specification.” Did the Japanese manufacturers have tighter tolerances? Yes. But did they have better designs? Yes. And those better designs were about more than miles-per-gallon and horsepower and torque.

They were about qualitative items that many engineers struggle to absorb. “Feels Good” and “The controls are where I expect them to be” and “Makes sense” and “I like it.” Software developers often struggle with the same issues.

And this is where manual testing matters. Getting qualitative feedback on your product designs is the only way to improve qualitative elements of those designs. The American manufacturers showed their disdain and hubris in allowing their customers to provide that feedback. The Japanese companies got feedback before they sold their products.

We can use manual testing to provide us with the kinds of feedback that can’t be automated. Things like “This UI is confusing” and “Why won’t it let me…?”

Summary

Don’t give up on manual testing as we strive to achieve software product success. Automate everything we can, and use people to test what we can’t. We need to make sure that we don’t lose the ability (or make sure that we find the ability) to test manually, for qualitative insight into our products. We can’t let the testers burst their hearts like poor John Henry, trying to manually perform an automatable test.
Thanks again, James, for starting us down this path!

Test Smarter, Not Harder – A Detailed Article

Scott Sehlhorst — Tue, 30 May 2006 03:18:32 +0000

developer.* has just published a 15-page article, Test Smarter, Not Harder by Scott Sehlhorst on test automation. We present the background for that article here (why you should read it) as well as a summary of what is covered in the article. Check it out at developer.*, and thanks to Dan Read, both for starting developer.* (which is a GREAT site) and for publishing the article. Skip to the end of this post to see how you can help.

Automated Testing Must Be a Core Competence

Automated testing has become a critical component of software product success. Processes like continuous integration require test automation to be effective. Unit testing requires automated testing to be effective as a rapid-development tool. Whitebox testing is almost always done with automation. More and more blackbox testing is being done with automation. A team without a good approach to automated testing is working at a severe disadvantage, and is arguably doomed to extinction.

Functional Testing (or system testing) is where a team can achieve process differentiation with automation. 80% of teams rely on manual functional testing today. These teams argue that automating functional tests is too expensive. The reasons most cited are complexity and rapid change in the underlying product.

Testing Complex Software

Complex software presents very challenging problems for test automation. First – how do we deal with the extreme complexity of many of today’s software applications? Billions or trillions of combinations (scripts) of user actions can be made. And software that works with one combination may be buggy in hundreds of others.

We can’t realistically test all of the combinations. Even if we did have the hardware needed to run them we would not be able to validate the results of all of those tests. So we can’t achieve exhaustive test coverage.

Without exhaustive test coverage, how do we know when we’ve tested enough?

Test Smarter Not Harder

The article starts with an analysis of the complexity of modern software and a brief discussion of the organizational realities and typical responses. Then the article explores common approaches and techniques in place today:

Random Sampling
Pairwise Testing
N-wise Testing

The article explores the math, benefits, and limitations of each approach. While N-wise testing is the most effective approach in common use today, it has a limitation – it assumes that the user’s actions (input variables) happen in a prescribed sequence or are irrelevant. In most complex software this assumption is invalid. The article presents a technique for incorporating order-dependence into the statistical approaches for developing test coverage.

The article also demonstrates techniques for simplifying the testing representation using whitebox testing techniques to redefine the problem space, and then applying blackbox testing (statistical) approaches to execution of the resultant test parameters.

By combining the statistical techniques explained in the article with a communication plan for sharing the appropriate quality metrics with stakeholders, we can deliver higher quality software, at a lower cost, and with improved organizational support and visibility.

Why developer.*?

While many Tyner Blain readers are interested in test automation, most of our readers would not want to go into this much depth. The audience at developer.* has a more technical focus, and their site is a more natural place for this article. Personally, I am honored to be included as an article author there – and all of you who want more technical depth than we go into at Tyner Blain should really check out the stuff Dan Read has put together. They’ve also just published their first indie-publisher ‘real book‘, which you can get via their site. My copy is on the way as I type.

How can you help?

This is a great opportunity for PR for Tyner Blain – please share the article with your associates because we want our community to grow. Also, it would be great if you ‘digg’ the article at digg (just follow the link, and ‘digg it’), del.icio.us, blink, or any other networking site. In addition to being good content, I hope to make this an experiment in viral marketing. Lets see if we can collectively reach the tipping point that brings this article, and the Tyner Blain community to the next group of people.

[update: the article just made the front page at digg and has 374 diggs and 32 brutal comments :). Welcome to all diggers, mobile, agile, or hostile!]

Foundation Series: Functional Testing of Software

Scott Sehlhorst — Thu, 18 May 2006 03:55:35 +0000

Functional Testing, also referred to as System Testing of software is the practice of testing the completed software to confirm that it meets the requirements defined for the software. A functional test is typically a test of user interactions, but can also involve communication with external systems. We contrast functional testing with unit testing. We also show how functional testing provides different benefits than unit testing.

This is a relatively long post for a Foundation Series post, so sit back with some coffee and relax. This primer will be worth it if its a new topic. If you know this stuff already, check out the links to other articles that go into more depth on points we make.

An Application is a Series of Flows

We can think of an application from the perspective of a user, as a series of interactions, or flows through the user interface.

People are not usually forced to follow a fixed set of instructions, or a predefined sequence of actions in an application. They can interact with controls in a random order, skip controls entirely, or otherwise do stuff that developers don’t expect.

Unit Tests are Whitebox Tests

Unit testing, as we detailed in our telephone example, provides targeted test coverage of specific areas of the code inside the application. Unit tests are written by developers, to allow them to test that the implementation that they created is behaving as they intended. Unit tests don’t implicitly provide the start-to-finish coverage that functional tests usually provide. Unit tests are whitebox tests that assure that a specific behavior intended by the developer is happening. A weakness of using unit tests alone is that they will not identify when the developer misinterpreted the requirements.

Functional Tests are Blackbox Tests

A functional test, however, is designed without insight into how the implementation works. It is a blackbox test. A functional test represents a set of user interactions with the application. The concept behind a functional test is to validate something about the state of the application after a series of events. According to Aberro Software, 80% of all functional tests are performed manually. That means that the most common functional test involves a tester making selections in a series of controls, and then evaluating a condition. This evaluation is called an assertion. The tester asserts that the software is in a specific state (an output is created, a control is filtered in a specific way, a control is enabled, a navigation option is disabled, etc).

Good functional requirements are written as concisely as possible. A requirement that supports a particular use case might state that the user specifies A, B, and C, and the application responds with D. A functional test designed to validate that requirement will almost always mimic this most common flow of events. The script that the tester follows will be to specify A, then B, then C. The tester will then evaluate the assertion that D is true. If D is false, then the test has failed.

A functional test may not cover the entire set of likely user interactions, but rather a subset of them.

One problem with this approach is that it does not account for a user specifying (A, B, X, C) or (A, C, B). These variations in order of operations might cause the underlying code to execute differently, and might uncover a bug. For a tester to get complete coverage of the requirement (A + B + C => D), he would have to create multiple scripts. This is expensive, tedious, and often redundant. But a tester has no way to know if the multiple scripts are redundant, or required.

Combining Unit Tests and Functional Tests

When we combine both unit testing and functional testing approaches, we are implementing what is called graybox testing (greybox testing). This is also referred to as layered testing. Graybox testing provides two types of feedback into the software development process. The unit tests provide feedback to the developer that her implementation is working as designed. The functional tests provide feedback to the tester that the application is working as required.

Graybox testing is the ideal approach for any software project, and is a key component of any continuous integration strategy. Continuous integration is a process where the software is compiled and tested every day throughout the release cycle – instead of waiting until the end of the cycle to test. Read this plan for implementing continuous integration if you want more details.

Automating Functional Tests

Automating unit testing is both straightforward, and relatively inexpensive. Automating functional testing is more expensive to set up, and much more expensive to maintain. Each functional test represents a script of specific actions. A tester (with programming skills) can utilize software packages like WinRunner to create scripts of actions followed by assertions. This represents an upfront cost of programming a script to match the application, in parallel with the development of the application – and it requires a tester with specialized skills to program the script.

The maintenance cost of automating functional tests is magnified in the early development stages of any application, and throughout the life of any application developed with an agile process. Whenever an element of the user interface is changed, every script that interacts with that element can be broken (depending on the nature of the change). These broken scripts have to be manually updated to reflect these ongoing changes. In periods of heavy interface churn, the cost of maintaining the test suite can quickly become overwhelming.

In the real world, apparently 80% of teams find that this overwhelming cost of automated testing outweighs even the high cost of manual functional testing.

Improved Automation of Functional Tests

We can reduce the maintenance cost of keeping automated scripts current with the user interface by abstracting the script-coding from the script-definition. This is referred to as keyword and table scripting. A set of objects are coded by the tester and given keywords. Each object represents an element in the user interface. Script behavior (sequence of interaction) is defined in terms of these keywords. Now, when a UI element is changed, the keyword-object is updated and all of the scripts that reference it are repaired.

This, however, does not address issues where one control is refactored into two controls, the adding or removing of controls, or changes in the desired flow of interaction. There is still a very large (albeit smaller) maintenance burden. And the applications that use this approach (such as QTP) can cost in the tens of thousands of dollars. Another reason to do functional testing manually.

Conclusion

Functional testing is important to validating requirements. It is an important element of assuring a level of software quality. And it is still expensive with the best of today’s proven solutions. Even with the high cost, it is much cheaper than the risk of delivering a solution with poor quality. Plan on having functional testing as a component of any process to achieve software product success.

- – -

Check out the index of the Foundation series posts which will be updated whenever new posts are added.

Ten Essential Practices of Continuous Integration

Scott Sehlhorst — Wed, 10 May 2006 04:01:14 +0000

Martin Fowler has identified the key process elements of making Continuous Integration work. You could even argue that they are the elements that define Continuous Integration (done correctly). We include his list and our thoughts below:

Maintain a Single Source Repository
Automate the Build
Make Your Build Self-Testing
Everyone Commits Every Day
Every Commit Should Build the Mainline on an Integration Machine
Keep the Build Fast
Test in a Clone of the Production Environment
Make it Easy for Anyone to Get the Latest Executable
Everyone can see what’s happening
Automate Deployment

For background information, check out the Foundation Series on Continuous Integration article.

1. Maintain a Single Source Repository

The smartest people I know use subversion when they have been able to make the choice themselves. Aside from being open source, it provides two key differentiated benefits relative to “everything else”.

Atomic commits: The ability to check in everything or nothing, so you don’t risk breaking the build with a partial check-in.
Overall project versioning: Allows you to track changes in source file directory hierarchies, file renaming, etc. Each version is of the entire project, not of a single file.

2. Automate the Build

Fowler sums it up perfectly:

“…anyone should be able to bring in a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine”

3. Make Your Build Self-Testing

Include automated testing as part of the build.

4. Everyone Commits Every Day

More frequently when possible. This is the minimum.

5. Every Commit Should Build the Mainline on an Integration Machine

A seperate, dedicated machine does a daily (or more frequent) build and full test suite run autonomously. The “build and test” model above relies on people to kick off the build when they commit. A scheduled task on a seperate machine provides a safety net for human error (oversight). Alternately, companies like Calavista can make this foolproof by automatically triggering an automated build as part of every commit. With Calavista’s devEdge, if the developer “commits”, what really happens is that the automated build/test cycle is triggered with his new code, and it gets promoted only if all the tests pass.

6. Keep the Build Fast

Tests can take a long time, builds should only take ten minutes. Fowler suggests a strategy of staged builds to address the 10-minute threshold. Run unit tests against the 10-minute build, and run the full suite in parallel or series.

Another option is to use statistical sampling of tests to get a “10 minute answer” while the full suite is kicked off in parallel.

7. Test in a Clone of the Production Environment

Eliminate even more variables. Make sure the tests are running against a clone of the production environment. Teams that are pushing the envelope today use virtual machines (VMs) to quickly create cloned production environments, install the software and run the tests.

8. Make it Easy for Anyone to Get the Latest Executable

Make sure everyone knows where the latest build can be found. Probably a good idea to keep recent builds in the same place too, in case a problem sneaks through the process temporarily (like a memory leak or other obscure, not-yet-tested situation).

9. Everyone can see what’s happening

Visibility! eMail the team when builds start/finish, including success/failure information. Put a rubber chicken on the desk of the person currently running the build (don’t ask – just read Fowler’s post). Ring a desk bell when the build passes. Have fun with it.

10. Automate Deployment

Make deployment into production as easy as running the build. Since tests are run against a production clone, and are already automated, this presents minor incremental effort.

Conclusion

Martin presents a great list. In addition to the above, we would suggest

Generate test-results documents per requirement. For each build, identify which requirements pass, fail, or are untested. The most relevant information for communicating outside of the team is status of previous requirements (did our regression tests pass?) and current requirements (are we almost done with this timebox?).

Foundation Series: Continuous Integration

Scott Sehlhorst — Tue, 09 May 2006 04:55:40 +0000

Continuous Integration

Continuous Integration is the software development and quality process where all team members merge their code and verify it frequently – at least daily. This verification project includes both an automated build process and automated testing. The main benefits of continuous integration come from risk-reduction and cost-reduction.

Integration has to happen. Making it continuous reduces its cost. There are also efficiencies for developers who can write better code faster when they are writing it in the context of the latest (most up-to-date) version of the code.

Risk is reduced in two ways. First, continuous integration causes fewer bugs to be created, thereby reducing risk. Second, when bugs are created, they are identified at the earliest possible moment (same day as their creation). This maximizes the time available to resolve them. No surprises at the end of the release cycle.

Merging Code

When a single software developer is writing code, she writes her code, saving frequently, and archiving it. But she is the only person working on the code. Other than the developer, no one cares how often she checks it in, as long as she can deliver the software on the release date.

When multiple developers work together, they depend upon each other. On any given day, different developers are writing different pieces of software – usually objects with today’s languages. These objects talk to each other, depend upon each other, or at least co-exist with each other in the completed software. This stuff is all “under the hood” for users, but imperative to understand when trying to manage a development process or team.

Why Merging Matters

After the developers go off into their offices and create their objects independently, some or all of the team members have to stop what they are doing, and integrate all of those objects back into a common code-base (aka ‘the software). When a developer fixes a bug in one or more objects, those fixes need to be incorporated back into the common code-base. With multiple developers, there are multiple elements that all have to be rolled back together again into the code. Developers often refer to this as merging or promoting into the trunk, or tip, or main branch.

Each change has a set of predicted effects on the rest of the software. These changes can be tested by the developer before integrating her code into the trunk. Each change also has a set of unpredicted effects on the rest of the software. And combinations of changes can ‘create’ effects that did not occur with either change individually. ‘Unpredicted effects’ is fancy-talk for ‘bugs’. The more changes we integrate into the trunk at a time, the more bugs we create. And this is an accelerating effect – the complexity of integration increases faster than the number of changes being integrated.

Increased Complexity Drives Higher Costs

As we increase the frequency of integration, we decrease the quantity of changes per integration. This decreases both the cost per integration and the total cost of integration. It is much cheaper to integrate five changes 100 times than to integrate 100 changes five times.

Each object and each connection in these diagrams represent a potential source of error. With 4 objects, we have 10 sources. With 7 objects, we have 30 sources of error. This represents an accelerating increase in the cost of integration.

To minimize the costs, we need to minimize the number of objects being integrated. We do that by minimizing the time between integrations.

Overhead

The main resistance to frequent merging is the cost of testing. Testing involves building the merged code, running the tests, and evaluating the results. When the building and testing (the integrating) are automated, the cost of evaluating test results can be minimized. When test-evaluation is also automated, we can bypass test-result evaluation except when the system notifies us that something is broken.

Continuous integration is only feasible when the overhead of integrating (merging and verifying) is trivialized through automation.

Conclusion

Agile processes depend upon continuous integration, but any software development process is improved with continuous integration. This is one of the enablers of iterative development processes. It reduces the cost of quality (or allows us to achieve higher quality levels at the same cost). It also makes development more enjoyable because developers spend less time on fixing bugs and more time implementing solutions.

- – -

Check out the index of the Foundation series posts for other introductory articles.

Market Segmentation or Senseless Mistake?

Scott Sehlhorst — Wed, 26 Apr 2006 04:52:12 +0000

A grass roots campaign has been started by Peter Provost to get Microsoft to include unit testing support included with all versions of Visual Studio 2005 (VS). Currently, Microsoft is only including it with Visual Studio Team System (VSTS) versions of Visual Studio. This looks to be a great example of a killer feature in a product providing so much surprise and delight that people are demanding that it be universally available. This is also a great example of market segmentation by Microsoft. The irony is that there is an open source alternative that makes the opportunity cost very low, and yet people are still clamoring. Let’s see why.

Background

Visual Studio 2005 is a development environment for developing .NET applications. Microsoft offers several versions of the software – 8 in the 2005 packaging. Even for people familiar with the product, the market segmentation strategy can be pretty confusing. To oversimplify, each version offers more capability than the less-expensive version below it. Rob Caron provides the best explanation of the product definition strategy in his Hitchhiker’s guide to VSTS. He starts by explaining the Visual Studio 2003 packages, and then shows the evolution to the VS2005 appraoch, including the Team System versions.

Unit testing support is offered only in the four most expensive, most capable versions – the Team System versions. The petitioners argue that unit testing is critical to all developers, and should be included in every version of the product. Unit testing is a form of whitebox testing where developers create automated tests of their code.
Microsoft is implementing a classic market segmentation strategy with this approach.

Market Segments

Markets are not homogenous – we don’t sell products to clones. Everyone has a different set of criteria for purchasing software. They make different tradeoffs in terms of price versus performance, or cost versus capability. Imagine a market roughly divided into two populations – price sensitive people, and feature-driven people.

The price-sensitive people like getting extra features, but will only pay marginally more for them. The feature-driven population is willing to pay a higher premium for added capabilities.

If we treat our potential customers as a homogenous market, we will make one of three mistakes:

Price the product so that everyone buys it. If we set the price based on the price-sensitive population, we are leaving money on the table. The feature-driven people would gladly pay more for the features.
Price the product so that only feature-driven people will buy it. We lose out on sales to the price-sensitive population, who won’t pay for the extra capabilities.
Try and compromise. Nobody wins. We won’t get enough of the price-sensitive customers, and we’ll leave money on the table with the feature-driven customers.

The Good, The Better, and The Best

One way to serve all of the customers is with multiple products. As an example, imagine three versions of a washing machine. They all basically do the same thing, wash clothes. The manufacturers can put a stronger motor, fancier control panel, or better sound insulation on some versions of the same product, and sell them to different people for different prices.

Most of the engineering costs apply to all three versions of the same product. The same is even more applicable to software. An easy way to do it would be to write the “best” software, and then disable some features to create the “better” and “good” versions. Many small software companies do this today, offering free and paid versions of their software. The free versions usually are missing features of the paid versions.

Microsoft has presumably identified several user profiles, and tailored a specific version of the software for each profile. Each version has different capabilities, and a different price.

Product Differentiation

Unit testing support, within the Visual Studio development environment, is absolutely a valuable capability. The growing response to the petition proves it. This is a great example of a surprise and delight feature (in Kano terms). In fact, some users find it to be so compelling that they want all users to get it “for free” as part of purchasing any version of Visual Studio.

This is one way that Microsoft is providing differentiation of the Team System versions of Visual Studio. There are other tools that may provide even more compelling reasons to get the Team System version.

Opportunity Cost

The odd thing is that NUnit is an open-source unit testing tool that can be plugged-in to all versions of Visual Studio. This means that there is a free tool for doing exactly what the petition is asking Microsoft to do. The cost of using NUnit is the time spent setting it up – I would imagine a few hours to figure it out and create an install document for the rest of the team. This is a surprisingly low-cost alternative. And it may even be the better alternative, as NUnit has a very active community, and there are many areas to find free support and help. The opportunity-cost logic applies to this situation (but in reverse). There is a low-cost alternative, so why spend the money on the extra features?

The other capabilities available in Team System provide much better differentiation, as they don’t have low-cost alternatives like NUnit.

Conclusion

This is a great example of using market segmentation to sell more software for more profit. Feature-driven people who want unit testing will pay more for it. People who are more price sensitive will still buy the versions without unit testing baked in, and will hopefully know about NUnit and bolt it on.

Good job Microsoft marketing.

Learn to Fly with Software Process Automation

Scott Sehlhorst — Wed, 05 Apr 2006 04:49:18 +0000

We can reach the next step in our software process evolution by automating much of our process. Flying squirrels evolved a technique* to quickly move from one tree to another without all the tedious climbing and dangerous running. Software teams that automate their processes achieve similar benefits. Automation allows us to increase efficiency while improving quality. And we spend less time on tedious and mundane tasks.

Benefits of process automation

Tim Kitchens has a great article at developer.* where he highlights the benefits of process automation. Here are our thoughts on the benefits he lists:

Repeatability. The first step in debugging software is the isolation of variables. A repeatable build process eliminates many variables, and likely many hours of wasted effort.
Reliability. A repeatable process eliminates the possibility of us introducing errors into our software by messing up a step in the build.
Efficiency. An automated task is faster than a manual task.
Testing. Reductions in overhead of building and testing allow us to test more frequently.
Versioning. The scripts that drive our build process are essentially self-documenting process documents. And tracking versions of the scripts provides us with precise records of the process used for prior builds. This documentation, and re-use of it can reduce the cost of running our projects at any CMMI level.
Leverage. We get much more efficient use of our experts’ time – they spend less effort on turn-the-crank processes and more effort on writing great software.

What and when should we automate?

The short answer is automate everything, unless there’s not enough ROI. We have to examine each process that we use to make a final decision – some automation will not make sense due to uncommon situations. Also, if we’re nearing the end of an existing project, there is less time to enjoy the benefits of automation, so we may not be able to justify the costs. We may be under pressure to deliver ROI in a short payback period. We would suggest exploring the automation of the following activities:

Automate the build process

Most people underestimate the benefits of an automated build. The obvious benefit is time savings during the normal build cycle. Imagine the build takes an hour, happens monthly, and usually happens twice per month. Two hours per month doesn’t seem like a lot of savings. However, chasing down a bug caused by the build process is at best expensive, and at worst nightmarishly expensive (because we aren’t looking in the right place to find the problem). Use an estimate of the probability of this happening to the expected value calculation for the savings.

The largest potential benefit of an automated build is changing the way we support our customers. Monthly builds aren’t scheduled because the business only wants updates once per month. They are scheduled at a monthly rate because that’s a balance someone has achieved between the cost-of-delivering and the cost-of-delaying a delivery. When we automate our delivery process, we dramatically reduce the cost of delivery, and can explore more frequent release schedules.

Automate unit testing

We significantly improve the efficiency of our team at delivering by shortening the feedback loop for developers. On a Utopian dev team, we would run our test suite as often as we compiled our code. Realistically, developers should run relevant automated whitebox tests every time they compile. They should run the suite of whitebox tests every time they promote code. And an automated process should run the full suite against the latest tip on a nightly basis (to catch oversights). It would be great if the check-in process initiated an automated test run and only allowed a promotion if all the tests passed.

Automate system and functional testing

End to end and blackbox tests should be automated next. These are the big picture tests, and should be run nightly on a dedicated box against the latest code base. We’ve had the most success with teams that used a nightly testing process, which sent an email with test results to the entire team whenever results changed. We’ve had the pleasure of working with a team that included performance testing on the nightly runs, and reported statistically significant improvement or degradation of performance.

Documentation

Generate tactical documentation whenever possible. Use javadoc or the equivalent to automatically generate well formatted and organized reference materials for future developers.

Marginally relevant reporting

If our team is asked to report metrics like lines of code, cyclomatic complexity, code coverage, etc. We should automate this. This work is the definition of tedium, while presenting tenuous value to the manager who requested it. If we can’t convince someone that they don’t want this data, we should at least eliminate the pain of creating it.

Code coverage statistics can provide better than nothing insight into how much testing is being done, or how much functionality is exercised by the test suite. But code coverage metrics have the danger of false precision. There’s no way to say that a project with 90% code coverage has higher quality than a project with 80% coverage.

Conclusion

Automation makes sense. We save time, increase quality, and ensure a more robust process. We also spend less time on turn-the-crank activities and more time creating differentiated software.

*Technically, they don’t fly – they fall. With style.

Software testing series: Pairwise testing

Scott Sehlhorst — Sun, 19 Mar 2006 03:19:51 +0000

Before we explain pairwise testing, let’s describe the problem it solves

Very large and complex systems can be very difficult and expensive to test. We inherit legacy systems with multiple man-years of development effort already in place. These systems are in the field and of unknown quality. With these systems, there are frequently huge gaps in the requirements documentation. Pairwise testing provides a way to test these large, existing systems. And on many projects, we’re called in because there is a quality problem.

We are faced with the challenge of quickly improving, or at least quickly demonstrating momentum and improvement in the quality of this existing software. We may not have the time to go re-gather the requirements, document them, and validate them through testing before our sponsor pulls the plug (or gets fired). We’re therefore faced with the need to approach the problem with blackbox (or black box) testing techniques.

For a complex system, the amount of testing required can be overwhelming. Imaging a product with 20 controls in the user interface, each of which has 5 possible values. We would have to test 5^20 different combinations (95,367,431,640,625) to cover every possible set of user inputs.

The power of pairwise

With pairwise programming, we can achieve on the order of 90% coverage of our code in this example with 54 tests! The exact amount of coverage will vary from application to application, but analysis consistently puts the value in the neighborhood of 90%. The following are some results from pairwise.org.

We measured the coverage of combinatorial design test sets for 10 Unix commands: basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc. […] The pairwise tests gave over 90 percent block coverage.

Our initial trial of this was on a subset Nortel’s internal e-mail system where we able cover 97% of branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.

[…] a set of 29 pair-wise AETG tests gave 90% block coverage for the UNIX sort command. We also compared pair-wise testing with random input testing and found that pair-wise testing gave better coverage.

Got our attention!

How does pairwise testing work?

Pairwise testing builds upon an understanding of the way bugs manifest in software. Usually, a bug is caused not by a single variable causing a bug, but by the unique combination of two variables causing a bug. For example, imagine a control that calculates and displays shipping charges in an eCommerce website. The website also calculates taxes for shipped products (when there is a store in the same state as the recipient, sales taxes are charged, otherwise, they are not). Both controls were implemented and tested and work great. However, when shipping to a customer in a state that charges taxes, the shipping calculation is incorrect. It is the interplay of the two variables that causes the bug to manifest.

If we test every unique combination of every pair of variables in the application, we will uncover all of these bugs. Studies have shown that the overwhelming majority of bugs are caused by the interplay of two variables. We can increase the number of combinations to look at every three, four, or more variables as well – this is called N-wise testing. Pairwise testing is N-wise testing where N=2.

How do we determine the set of tests to run?

There are several commercial and free software packages that will calculate the required pairwise test suite for a given set of variables, and some that will calculate N-wise tests as well. Our favorite is a public domain (free) software package called jenny, written by Bob Jenkins. jenny will calculate N-wise test suites, and its default mode is to calculate pairwise tests. jenny is a command line tool, written in C, and is very easy to use. To calculate the pairwise tests for our example (20 controls, each with 5 possible inputs), we simply type the following:

jenny 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 > output.txt

And jenny generates results that look like the following:

1a 2d 3c 4d 5c 6b 7c 8c 9a 10c 11b 12e 13b 14d 15a 16c 17a 18d 19a 20e
1b 2e 3a 4a 5d 6c 7b 8e 9d 10a 11e 12d 13c 14c 15c 16e 17c 18a 19d 20d
1c 2b 3e 4b 5e 6a 7a 8d 9e 10d 11d 12a 13e 14e 15b 16b 17e 18e 19b 20c
1d 2a 3d 4c 5a 6d 7d 8b 9b 10e 11c 12b 13d 14b 15d 16d 17d 18b 19e 20a
1e 2c 3b 4e 5b 6e 7e 8a 9c 10b 11a 12c 13a 14a 15e 16a 17b 18c 19c 20b
1a 2a 3c 4e 5e 6a 7b 8c 9d 10b 11b 12b 13e 14a 15d 16d 17c 18c 19b 20d [...]

Where the numbers represent each of the 20 controls, and the letters represent each of the five possible selections.

What’s the catch?

There are two obvious catches. First, when you use a tool like jenny, we must run all of the tests that it identifies, we can’t pick and choose. Second, pairwise testing doesn’t find everything. What if our example bug before about taxes and shipping only manifested when the user is a first time customer? Pairwise testing would not catch it. We would need to use N-wise testing with N >= 3. Our experience has been that N=3 is effective for almost all bugs.

There is also a sneaky catch – test generators like jenny assume that the order of variables is irrelevant. Sometimes we are testing dynamic user interfaces, where the order of value selection in controls is relevant. There is a solution to this, and we will update this post with a link to that solution when it is available.

- – -

Check out the index of software testing series posts for more testing articles.

Measuring the Cost of Quality: Software Testing Series

Scott Sehlhorst — Wed, 22 Feb 2006 15:04:56 +0000

Should we test our software? Should we test it more?

The answer to the first question is almost invariably yes. The answer to the second question is usually “I don’t know.”

We write a lot about the importance of testing. We have several other posts in our series on software testing. How do we know when we should do more automated testing?

Determining the costs is an ROI analysis. Kent Beck has a great position -

If testing costs more than not testing, then don’t test.

At first glance, the statement sounds trite, but it really is the right answer. If we don’t increase our profits by adding more testing, we shouldn’t do it. Kent is suggesting that we only increase the costs and overhead of testing to the point that there are offsetting benefits.

We need to compare the costs and benefits on both sides of the equation. We’ll start with a baseline of the status quo (keeping our current level of testing), and identify the benefits and costs of additional testing, relative to our current levels.

We should do more automated testing when the benefits outweigh the costs

We’ll limit our analysis to increasing the amount of automated testing, and exclude manual testing from our analysis. We will use the assumption that more testing now will reduce the number of introduced bugs in the future. This assumption will only hold true when developers have the ability to run the automated tests as part of their personal development process. We’ve written before about the sources of bugs in the software development process, and in other posts in this series we show how automated testing can prevent future bugs (unlike manual testing, which can only identify current bugs).

We are also assuming that developers are running whitebox unit tests and the testing team is running blackbox tests. We don’t believe that has an impact on this analysis, but it may be skewing our perspective.

Benefits

Reduced costs of bugs in the field. Bugs in the field can cause us to have “emergency releases” to fix them. They can increase the costs of (internal teams) using our software and working around the bugs. They can cause delayed sales. Bugs cause lost customers.
Reduced costs of catching future bugs. When developers can run a regression suite to validate that their code didn’t break anything before asking the testing team to test it, they can prevent introducing regression bugs. And thereby prevent the costs of finding, triaging, and managing those bugs.
Reduced costs of developing around existing bugs. Developers can debug new code faster when they can isolate it’s effects from other (buggy) code.
Reduced costs of testing around existing bugs. There is a saying – “What’s the bug behind the bug?” we’ve heard when testers are trying to validate a release. A bug is discovered, and the slack time in the schedule is used fixing that bug – then the code is resubmitted to test to confirm that the bug was fixed. Another bug was hiding behind it, and untestable because the first bug obfuscated the second bug. Addressing the second bug introduces unplanned testing costs. Preventing the first bug will reduce the costs of testing the latent bug.

Costs

Most of these increased costs are easy to measure once they are identified – they are straightforward tasks that can be measured as labor costs.

Cost of time spent creating additional tests.
Cost of time spent waiting for test results.
Cost of time spent analyzing test results.
Cost of time spent fixing discovered bugs.
Cost of incremental testing infrastructure. If we are in a situation where we have to increase our level of assets dedicated to testing (new server, database license, testing software licenses, etc) in order to increase the amount of automated testing, then this cost should be captured.

Conclusion

This is a good framework for making the decision to increase automated testing. By focusing on the efficiencies of our testing approaches and tools, we can reduce the costs of automated testing. This ultimately allows us to do more automated testing – shifting the pareto optimal point such that we can increase our incremental benefits by reducing our incremental costs.

Software development process example

Scott Sehlhorst — Mon, 20 Feb 2006 12:21:16 +0000

We’ve presented an example of the software development process across several posts over the last two weeks. In this post we tie them all together, showing the steps in process order.

A discussion of the concept of tagging. Context and background on tagging as a technology, with pros and cons.
The top five problems with test automation suites. We’ve talked repeatedly about how test automation suites are better than purely manual testing. Here we look at the “second order” problems. What are the main problems with unit test automation? These represent our market requirements for the creation of a software product designed to improve unit test automation suite usage.
Converting from MRD requirements to PRD requirements. The ideation / triage process of determining which requirements should be addressed in software.
Creating informal use cases to support our requirements. We define the use cases that support the high-level requirements.
Writing functional requirements to support the use cases. With a user-centric approach to software development, it is imperative that we build out our functional requirements in the context of use cases – we keep our eye on the ball with this focus on the user.
Design elements that support our functional requirements. Without going into esoteric details about how to design test automation software, we discuss the elements of the design that relate to the application of tagging to addressing some of the larger market opportunities.
Iterating and prototyping. We show the iterative process from PRD to design to users and back again. [Update 21 Feb - added this step. Thanks again, Deepak]

Let us know if you’d like to see a discussion of any of these or other steps in more detail by leaving a comment on this post. Thanks in advance!

Software Testing Series: Organizing a Test Suite with Tags Part Three

Scott Sehlhorst — Sun, 19 Feb 2006 04:08:58 +0000

Organizing a test suite with tags (part 3)

This is the third in a three-part post about using tags as a means to organize an automated unit test suite.

Part 3 of this post can be read as a standalone article. If it were, it would be titled Design elements of an automated unit test framework using tags. If you’re only reading this post and not parts 1 and 2, pretend that this is the title.

In part one of this post we developed an understanding of tagging as a technology, including its pros and cons.
In part two of this post we defined the top five opportunities and problems inherent in the organization of automated test suites.
In part three of this post we will consider the key design associated with using tags as a mechanism for organization.

Setting expectations

We designed a custom unit test automation system based on the use of tags to organize automated tests for a client. That system isn’t the subject of this post, but it did provide us with context and insight into the problem we’ve addressed. In this post we won’t be presenting a completed design – there are many good tools out there already for automating unit tests. We will be talking about a subset of the design decisions that are associated with the use of tags as a mechanism to organize the unit testing within the suite. In Marc Clifton’s advanced unit testing articles, he walks readers through the creation of a test automation tool for C#. The concepts presented here can be incorporated into a design relatively easily, but the details of doing that are a little too off-topic for most of our readers.

We are writing about designing software that tests other software. To keep our language consistent and easy to follow in this post, we will use two terms. Tool represents the test automation software that we are designing. Application represents software being tested with the tool, or more specifically, with tests maintained within the tool.

Design approach

After identifying the use cases and functional requirements for the tool, we began iterating on screen designs, business (requirement) object models and architectural (implementation) object models. We created an object oriented analysis (OOA) diagram to represent the concept concisely.
OOA diagram

An object oriented analysis diagram of the key relationships between scripts, inspections and tags.
A script is the embodiment of a user session in the application – it represents a set of actions that a user of the application would take. The user of the tool will create a script (as a separate file), and will create a reference to that script in the tool.

An inspection is an unit test of the application. The inspection evaluates a particular condition or makes a specific assertion about the state of the application (a properly filled out order is submitted when the user clicks “submit”) . The code that executes that inspection is represented outside of the tool. The user of the tool will create a reference to the inspection within the tool.

Inspections can be associated explicitly with any number of scripts (including none). An association between inspection 1 and script A is an instruction to the tool to run script A within the application, and evaluate inspection 1 against the script.

The processing of scripts and inspections is outside the scope of this document, but is covered in many other references, including Marc Clifton’s.

Any number of tags can be associated with each script. Tags could be used to represent different user actions (like deleting items from a shopping cart), different specific selections (user adds 1000 of an item to the shopping cart), different situations (shipping address does not match billing address), or any other relevant descriptor of the user session. A single script could have multiple tags.

Each inspection can be transitively associated with a set of scripts by explicitly associating it with one or more tags. By associating an inspection with a tag, we are instructing the tool to dynamically associate the inspection with all scripts that are associated with all of the identified tags. There are two benefits to this approach. First, this approach reduces the amount of time that a user of the tool must spend to associate an inspection with a set of relevant existing scripts. Second, this indirect mapping is utilized to assure that an inspection that is mapped to a tag will also automatically become associated with any future scripts that are added – as long as they have associations with the same tag or tags as the script. This reduces the cost of creating future mappings when scripts are added to the suite in the future.

We expect this design approach to provide significant labor savings in maintaining a test suite. We built our business case for this project upon that assumption. We also expect that this design approach will result in better testing coverage of the application by users of the tool. We did not incorporate that expectation into our cost benefit analysis when calculating the ROI of this project.

Followup

We will follow-up some months from now when we can evaluate data and draw conclusions from use of a tool built along similar lines for a client. Until then, we have confidence that it will work very well, but no tangible data.

Summary

In part one of this post we developed an understanding of tagging as a technology, including its pros and cons.
In part two of this post we defined the top five opportunities and problems inherent in the organization of automated test suites.
In part three of this post we considered the key design elements associated with using tags as a mechanism for organization.

- – -Check out the index of software testing series posts for more articles.

Sample Use Case Examples

Scott Sehlhorst — Fri, 10 Feb 2006 00:03:43 +0000

Sample informal use case examples

We talked about informal use cases a while ago in our use case series. Over a series of posts, we are demonstrating the process of defining a software product. The next step, and subject of this post, is the creation of informal use cases to support the defined goals for the software.

In part two of our post on using tags to organize automated tests, we identified several market requirements, and identified the ones that we would incorporate as software requirements.

These then become our goals in MRD to PRD requirements conversion- and in this post, we will show how we can create informal use cases to support those goals (aka high level requirements).

Where do use cases belong?

In our introduction to structured requirements, we showed a basic model for describing requirements. Once goals are identified, we articulate the use cases that enable those goals.

Our product requirements (goals) were defined to be

Minimize time spent identifying broken and obsolete scripts.
Minimize time spent removing obsolete scripts from the suite.
Minimize the time spent managing the mapping of inspections to scripts.

The first step in creating informal use cases

The easiest way to start writing the use cases is to write the names of the use cases. Just write out the names (on paper, whiteboard, one-note – whatever works for you), using semi-descriptive prose. For example, for the third goal, we could identify the following two use cases.

Developer adds a new script and maps to existing inspections
Developer adds a new inspection and maps to existing scripts

Notice that in the names we have an actor and an action – this is a good consistent naming convention that makes it easy to understand exactly what we’re talking about. Now we can write the informal use case details for each of these.

Developer adds a new script and maps to existing inspections

The developer’s goal is to add her recently created script to the suite and associate it with a set of specific inspections, producing a set of test outputs. She may already have a set of inspections that she knows she wants to associate with the script. She is also opportunistic about leveraging other existing scripts to create new tests when they are relevant. She creates a reference to the script in the test framework. She then identifies the set of inspections that she wants to associate with the script and creates the associations. She reviews the set of associations, and either needs to change the set of associations, or is satisfied with the mappings. If she needs to add or remove some of her newly created mappings, she will identify the inspections that are to be modified and create those associations as well.

Developer adds a new inspection and maps to existing scripts

The developer’s goal is to add his recently created inspection to the suite and associate it with a set of existing scripts, producing a set of test outputs. He already knows, generally, what types of scripts he wants to map his inspection to. He does not have a specific set of scripts in mind for the mapping. The developer creates a reference to the inspection in the test framework. Then he reviews the existing scripts, and identifies the scripts to be mapped to his new inspection and creates the associations. He reviews the set of associations and either decides that he has too many, too few, or has the right amount. If he needs to modify the associations he does so and re-reviews, repeating until satisfied.

Two things to note about the process of creating the informal use cases above:

This is very fast – these took about 10 minutes each to create. It is a very small amount of time that has to be invested prior to validating the use cases with the users. During and after validation, these use cases can be modified, eliminated or replaced with other use cases. Having something concrete with a minimum investment of time both helps drive good decisions and saves costs.
These use cases are the result of iteration. We did not just type for ten minutes and move on. While writing the second use case, we noticed that we were not handling the possibility that the developer would initially create too many associations and need to go back and remove some of them. We updated both use cases to reflect this possibility.

More Example Use Cases

We occasionally add other example use cases, usually focused on a particular element of use case writing. Here are some that we’ve added since this article was originally published:

Summary

Our next step is to validate these use cases with the actors identified in the use cases (developers). Once that is complete, we will define the functional specs for the software that enable these use cases.

Software Testing Series: Organizing a Test Suite with Tags Part Two

Scott Sehlhorst — Wed, 08 Feb 2006 13:46:41 +0000

Organizing a test suite with tags (part 2)

This is the second in a three-part post about using tags as a means to organize an automated test suite.

Part 2 of this post can be read as a standalone article. If it were, it would be titled Top five problems with test automation suites. If you’re only reading this post and not parts 1 and 3, pretend that this is the title.

In part one of this post we developed an understanding of tagging as a technology, including it’s pros and cons.
In part two of this post we will define the top five opportunities and problems inherent in the organization of automated test suites.
In part three of this post we will explore an approach to combining tagging with test suite organization.

What are the problem areas inherent in managing automated tests?

We start with identification of the problems or opportunities, before defining what the requirements will be. This is the same process we discussed in From MRD to PRD, applied to the test-automation space. The following are the top five problem areas we can identify about test automation suites.

Maintaining the suite becomes too expensive. Once we have a suite in place, we have to maintain it. As the size of the suite grows, the amount of maintenance of existing test grows. It grows in proportion to the number of tests in the suite and the rate of change of the underlying software being tested. As the amount of maintenance grows, so does its expense.
Developers will never start using the suite. Change is bad. Well, for many people it is. Asking someone with a full time, salaried job to take on additional responsibilities has to be done correctly. There is absolutely a risk that people won’t start using the suite. Since this project is focusing on iterative development of an already deployed tool, already in use, this problem really doesn’t apply.
Developers will stop using the suite. Developers avoid tedium. They’re smart. They want to avoid unneccesary work, menial work, and irrelevant work. If the developers perceive the test suite in any of these ways, we’re doomed – they will stop using it.
Not testing the right stuff. A test suite that doesn’t test the right areas of the software is worse than not having one at all – because it gives you a false sense of confidence.
Test suite becomes less effective over time. An initially effective suite can grow less effective over time as the underlying software changes. Individual tests become irrelevant as they become impossible to reproduce with the application – perhaps the user interface has changed. If test design was linked to the heaviest usage patterns, and those patterns change, then coverage of the new heavy usage parts of the suite will be reduced – and the effectiveness of the suite will be reduced.

Which problems should we address with software?

With limited resources, we need to make sure that we focus our software efforts on those problems where software can have the most impact on solving the problem. We’ll start by identifying

Maintaining the suite becomes too expensive. There are three approaches to solving this problem – reduce the required maintenance, make the required maintenance more efficient, and reduce the cost of the labor that maintains the solution. Labor cost reductions may very well be the most effective general way to solve this problem, but given the real world project constraints for the project behind this post, we aren’t exploring that option. This is a candidate for the software solution.
Developers will never start using the suite. Make them want to use it, or make them use it. We believe you want to make them want to use it – both by evangelizing the benefits and by quickly crossing the suck threshold so that users get positive feedback. For this project, we have taken that approach, although it’s true that there is also a mandate from the dev team’s managers that we must make sure they use it. With process and education approaches that have proven effective, this is not a target of the current software solution.
Developers will stop using the suite. The looming mandate will assure that developers won’t go AWOL on the suite. But if they can present a compelling reason to their managers, there is a risk that they will decide to stop using it. This is a candidate for the software solution.
Not testing the right stuff. Test suite planning is a science unto itself. We will keep in mind “ways to make test suite planning easier” as a candidate for the software solution, but we aren’t otherwise targeting this for the current software solution.
Test suite becomes less effective over time. Tests can grow irrelevant over time when the software they test is constantly changing (as in this project). This problem has been addressed to a large extent by using whitebox unit tests in the test suite. We are not targeting this as part of the current software solution.

Reminder

In part one of this post we developed an understanding of tagging as a technology, including it’s pros and cons.
In part two of this post we defined the top five opportunities and problems inherent in the organization of automated test suites.
In part three of this post we will explore an approach to combining tagging with test suite organization.

- – -

Check out the index of software testing series posts for more articles.

Software Testing Series: Organizing a Test Suite with Tags Part One

Scott Sehlhorst — Tue, 07 Feb 2006 03:04:27 +0000

Organizing a test suite with tags

Tagging is a method of organizing information that is pushing into the mainstream now through the success of sites like Flickr and Del.icio.us, and blogging software like WordPress. We can apply this idea to managing our automated test suites. An automated test suite is a critical component of any continuous integration process.
First steps first…

This post is a follow-up to our previous case study on incorporating unit testing into an existing team’s development environment. The case study is based on a real solution that has already started reaping rewards for our client, and is gaining momentum. We’re now looking at making it easier for the development team to maintain this test suite, and proposing some extensions – including a form of tagging.

In part one of this post we develop an understanding of tagging as a technology, including it’s pros and cons.
In part two of this post we will define the top five opportunities and problems inherent in the organization of automated test suites.
In part three of this post we will explore an approach to combining tagging with test suite organization.

Understanding tagging

Tagging allows users to define their own categories for describing the items that they care about. The technical term for a free-form approach to labeling items is folksonomy as opposed to taxonomy, which is a classical categorization approach.

The Wikipedia presents the following definitions for folksonomy and taxonomy:

Folksonomy, a portmanteau word combining “folk” and “taxonomy,” refers to the collaborative but unsophisticated way in which information is being categorized on the web. Instead of using a centralized form of classification, users are encouraged to assign freely chosen keywords (called tags) to pieces of information or data, a process known as tagging. Examples of web services that use tagging include those designed to allow users to publish and share photographs, personal libraries, bookmarks, social software generally, and most blog software, which permits authors to assign tags to each entry.

Taxonomy (from Greek verb tassein = “to classify” and nomos = law, science, cf “economy”) may refer to:

the science of classifying living things (see alpha taxonomy)

a classification

Initially taxonomy was only the science of classifying living organisms, but later the word was applied in a wider sense, and may also refer to either a classification of things, or the principles underlying the classification. Almost anything, animate objects, inanimate objects, places, and events, may be classified according to some taxonomic scheme.

There is debate about the value of tags

Several people have voiced concerns that tagging is simply a bad idea, with some compelling arguments. The Net Takeaway has a post and links to previous posts.

Look, if I am looking for something specific, then I type those terms in. Say I use a search engine. If I am looking for a phrase, I use quotes and type in all the words (up to 10 for most engines) and I get hits with that phrase.

But usually, I want stuff “like” or “similar” to my words. [...]

But that’s now how tagging systems work. Instead, you have to know the terms up front to find anything.[...] Note that every popular “tagging” system, to date, has been for consumer fun stuff (flickr, etc.) and not for real knowledge management.

There are some good rebuttals in the comment thread as well, providing insight into what is intended by tagging, suggestions on how to use it, and alternative comparisons with search. If you only read one, read comment #3. Then go back and read the rest of them.

Benjamin Booth writes, in The Present Failure of Tagging, about the challenges. Benjamin’s approach is “I like tagging, how do we make it work the way we want?”

The general problem can be seen as the task of 1) externalizing knowledge-retrieval ‘landmarks’ when encountering information you want to store (in some context) and then, 2) being able to quickly find these landmarks when trying to recall the information later on, potentially in a completely different context from the one in which you created the landmark in the first place.

Near the end of his post he begins:

We need refactoring for tagging.[...]

Benjamin makes reference to the concept mapping tool from IHMC that we talked about previously.

Rashmi Sinha has an outsanding article where she applies the science of cognitive psychology to the issues with tagging usability.

Cognitively, we are equipped to handle making category decisions. So, why do we find this so difficult, especially in the digital realm – to put our email into folders, categorize our bookmarks, sort our documents. Here are some factors that lead to what I call the “post-activation analysis paralysis”.

Our conclusions about tagging

After reading the linked posts and their discussion threads, we are pretty well versed about the pros and cons of tagging.

Pros

Tagging eliminates the “How do I organize this?” analysis paralysis that happens when trying to start organizing
Tagging allows for a dynamic classification system that grows over time. If we make a bad decision early, we can grow out of it.
The “free association” approach to tagging that exists in the digital world today is consistent with the way our brains function when storing information.
There are a bunch of smart people working on tagging right now, so there is plenty of opportunity to leverage their good ideas.

Cons

Tagging makes retrieval of information difficult. If we don’t know how we previously tagged something, it can be hard to find it later.
The existing (digital) approaches to tagging don’t provide an analog for the way our brains function when continuously updating and refining our “free associations” as we learn.
People rooted in traditional taxonomy-based classification systems struggle with the concept of tagging. This probably characterizes 90% of people, so gaining mindshare outside of the technorati will be difficult, and user adoption could be a challenge.

Reminder:

In part one of this post we develop an understanding of tagging as a technology, including it’s pros and cons.
In part two of this post we will define the top five opportunities and problems inherent in the organization of automated test suites.
In part three of this post we will explore an approach to combining tagging with test suite organization.

- – -

Check out the index of software testing series posts for more articles.