Category Archives: Test Automation

Automated testing is critical to software development. Efficient development relies upon automated tests for speed of delivery, preventing regression bugs, and mitigating risks. These articles look specifically at how automation can be improved, and how test automation can be applied to improve the overall process.

Test Smarter, Not Harder – Part 3

interconnected controls

This series is a reprint of an article by Scott Sehlhorst, written for developer.* in March 2006. A recent article on dailytech about “new” methods for software testing points to some very interesting research by the NIST (the National Institute of Standards and Technology) Information Technology Lab. We’ve split the original article into three articles to be more consistent in length with other Tyner Blain articles.

This is part 3 of a 3 part series.

Continue reading Test Smarter, Not Harder – Part 3

Test Smarter, Not Harder – Part 2

interconnected controls

This series is a reprint of an article by Scott Sehlhorst, written for developer.* in March 2006. A recent article on dailytech about “new” methods for software testing points to some very interesting research by the NIST (the National Institute of Standards and Technology) Information Technology Lab. We’ve split the original article into three articles to be more consistent in length with other Tyner Blain articles.

This is part 2 of a 3 part series.
Continue reading Test Smarter, Not Harder – Part 2

Test Smarter, Not Harder – Part 1

interconnected controls

This series is a reprint of an article by Scott Sehlhorst, written for developer.* in March 2006. A recent article on dailytech about “new” methods for software testing points to some very interesting research by the NIST (the National Institute of Standards and Technology) Information Technology Lab. We’ve split the original article into three articles to be more consistent in length with other Tyner Blain articles.

This is part 1 of a 3 part series.

Continue reading Test Smarter, Not Harder – Part 1

Is Agile Bad For Software Development?

broken software

Last week, Ivan Chalif, a product manager / blogger, tapped into a thread criticising product managers for not adopting and espousing agile, or at least rapid-release techniques. In this article we look at Ivan’s comments and one of the articles that he referenced. We also share our own perspective and an alternative analysis of what may have happened.

Continue reading Is Agile Bad For Software Development?

Insight Into Test Driven Development


James Kovacs shares a great insight on software testing and the software testing process. His epiphany about test driven development makes it obvious to all of us why this technique is so powerful.


Test driven development is an approach to software development where you write the tests first, and the code second. This sounds ludicrous at first, but it really does work. Here’s James’ epiphany on the subject (emphasis added).

I didn’t understand why TDD practitioner were so zealous about writing the tests first. Why did it matter? I thought it was to ensure that some project manager doesn’t try to shave some time off the project by cutting the unit tests. Then I started doing some reading about TDD, poking around, asking questions, and trying it out myself. I discovered the real reason for writing tests first is that TDD isn’t about testing code, it’s about designing code.

A Tale of Two Epiphanies: TDD and Mocking, James Kovacs

These tests are unit tests, and they are black-box tests (or at least should be black-box tests – read James’ article).

Works Great With Structured Requirements

Not only is this a great way to approach design, but it is a great way to coordinate software development with structured requirements. A well-written requirement describes a need without specifying design. This description then works like a black-box contract. The implementer then must do something that meets the specification given. As James points out, TDD is very effective at creating black-box tests.

This makes sense, when you consider the outside-in approach that comes from starting with requirements. This outside in approach leads to API decisions (for a program or module or method). The API has to support all of the required possible inputs and outputs. Once an API is written, tests can be written.

Best Applied As Part Of Continuous Integration For Larger Teams

All of the tests will fail (because the code hasn’t been written). Then the implementer will write code until the tests pass. Using a continuous integration approach is a very effective way for entire teams to do this simultaneously on larger projects.

Faster Processes

We can also validate the design for completeness prior to creating the implementation. We can compare the generated APIs (and their rationalizations) with the software specification (or application requirements, or FRS… see alphabet soup article). This allows us to address some refactoring needs before the code has been implemented – saving time and money.


Use TDD as an outside-in approach that assures that the delivered software will meet the objectives of the requirements specification. It also saves time and money by supporting validation in advance of implementation.

John Henry, Manual Tester

John Henry

There’s a piece of North American folklore about John Henry, who was a manual laborer during the expansion of the railroads in our country. His job was being replaced by steam-driven heavy equipment, as the railroad industry applied technology to become more efficient. The same dynamics are happening today with manual testers. We need to make sure that manual testers avoid John Henry’s fate – read on to see why.

Manual and Automated Testing

Software can not be tested efficiently with only manual testing. And it can not be tested completely with only automated testing. James Bach writes an insightful article about how he worked his way to enlightenment over several years, and how he has incorporated this perspective into his training classes.

My understanding of cognitive skills of testing and my understanding of test automation are linked, so it was some years before I came to understand what I now propose as the first rule of test automation:

Test Automation Rule #1: A good manual test cannot be automated.

James goes on to explain that if a test can be automated, it is not a good manual test.

We’ve discussed repeatedly the benefits of automated testing, one example is The Code Freeze is Killing the Dinosaurs. We’ve also stressed the importance of doing functional testing where we discuss the dynamics and tradeoffs of using manual and automated approaches to test software.

Generally, we’ve approached the problem from the perspective of blackbox versus whitebox testing (and more details), and thought about our development process from the perspective of continuous integration as a means to deliver more efficiently.

We’ve not thought about it from James’ perspective before. Even with our improved approaches to automated testing, we are still no different than the inventor of the steam-hammer in Henry’s fable.


James puts things in perspective:

Rather than banishing human qualities, another approach to process improvement is to harness them. I train testers to take control of their mental models and devise powerful questions to probe the technology in front of them. This is a process of self-programming. In this way of working, test automation is seen as an extension of the human mind, not a substitute.

He highlights a very interesting point – we are missing a key benefit of having manual testers – the ability to gather feedback on intangibles, heuristics, usability, affordances, and other elements that we might classify as design bugs.

We talk about the importance of usable software, and different approaches to creating great software. But we’ve never really talked about how to test for software greatness.

A Broken Model

In the 1970’s, when the American automotive manufacturers were getting their clocks cleaned by the newly exporting Japanese companies like Honda, Datsun (now Nissan), and Toyota, a lot of people thought they understood the problem. Many more people pretended there wasn’t a problem, until the US government bailed out Chrysler.

The experts decided that the main problem was that quality was not “Job One”, that American manufacturers ignored the advice of W. Edwards Demming, and the Japanese did not. Toyota’s lean manufacturing philosophy is credited with much of the success.

It is true that the oil crisis of the time gave the Japanese companies an opportunity to penetrate the US market with smaller, more efficient cars. But that crisis ended (about the time that the American companies killed the muscle car and began building more efficient cars). But the Japanese cars didn’t go away. Once gas prices dropped, they lost a differentiating element – but they maintained two others. Cost and Quality.

Target of Opportunity Enables Strategic Advantage

Cost matters. But it matters tactically, as in “I can afford $X right now – what are my choices?” Quality creates Loyalty. And loyalty is strategic. The Japanese manufacturers gain and continue to gain success after success and award after award in the automotive industry. Why? Because they have good quality.

Automated and Manual Testing

As an engineer, I know that we can specify tolerances, inspect components, and test assemblies to make sure that products are “within specification.” And all of this testing can be automated – just like automated software testing. But passing these tests doesn’t make a product good, it merely indicates that the product is “within specification.” Did the Japanese manufacturers have tighter tolerances? Yes. But did they have better designs? Yes. And those better designs were about more than miles-per-gallon and horsepower and torque.

They were about qualitative items that many engineers struggle to absorb. “Feels Good” and “The controls are where I expect them to be” and “Makes sense” and “I like it.” Software developers often struggle with the same issues.

And this is where manual testing matters. Getting qualitative feedback on your product designs is the only way to improve qualitative elements of those designs. The American manufacturers showed their disdain and hubris in allowing their customers to provide that feedback. The Japanese companies got feedback before they sold their products.

We can use manual testing to provide us with the kinds of feedback that can’t be automated. Things like “This UI is confusing” and “Why won’t it let me…?”


Don’t give up on manual testing as we strive to achieve software product success. Automate everything we can, and use people to test what we can’t.  We need to make sure that we don’t lose the ability (or make sure that we find the ability) to test manually, for qualitative insight into our products.  We can’t let the testers burst their hearts like poor John Henry, trying to manually perform an automatable test.
Thanks again, James, for starting us down this path!

Test Smarter, Not Harder – A Detailed Article

Scott Sehlhorst

developer.* has just published a 15-page article, Test Smarter, Not Harder by Scott Sehlhorst on test automation. We present the background for that article here (why you should read it) as well as a summary of what is covered in the article. Check it out at developer.*, and thanks to Dan Read, both for starting developer.* (which is a GREAT site) and for publishing the article. Skip to the end of this post to see how you can help.

Automated Testing Must Be a Core Competence

Automated testing has become a critical component of software product success. Processes like continuous integration require test automation to be effective. Unit testing requires automated testing to be effective as a rapid-development tool. Whitebox testing is almost always done with automation. More and more blackbox testing is being done with automation. A team without a good approach to automated testing is working at a severe disadvantage, and is arguably doomed to extinction.

Functional Testing (or system testing) is where a team can achieve process differentiation with automation. 80% of teams rely on manual functional testing today. These teams argue that automating functional tests is too expensive. The reasons most cited are complexity and rapid change in the underlying product.

Testing Complex Software

Complex software presents very challenging problems for test automation. First – how do we deal with the extreme complexity of many of today’s software applications? Billions or trillions of combinations (scripts) of user actions can be made. And software that works with one combination may be buggy in hundreds of others.

We can’t realistically test all of the combinations. Even if we did have the hardware needed to run them we would not be able to validate the results of all of those tests. So we can’t achieve exhaustive test coverage.

Without exhaustive test coverage, how do we know when we’ve tested enough?

Test Smarter Not Harder

The article starts with an analysis of the complexity of modern software and a brief discussion of the organizational realities and typical responses. Then the article explores common approaches and techniques in place today:

  • Random Sampling
  • Pairwise Testing
  • N-wise Testing

The article explores the math, benefits, and limitations of each approach. While N-wise testing is the most effective approach in common use today, it has a limitation – it assumes that the user’s actions (input variables) happen in a prescribed sequence or are irrelevant. In most complex software this assumption is invalid. The article presents a technique for incorporating order-dependence into the statistical approaches for developing test coverage.

The article also demonstrates techniques for simplifying the testing representation using whitebox testing techniques to redefine the problem space, and then applying blackbox testing (statistical) approaches to execution of the resultant test parameters.

By combining the statistical techniques explained in the article with a communication plan for sharing the appropriate quality metrics with stakeholders, we can deliver higher quality software, at a lower cost, and with improved organizational support and visibility.

Why developer.*?

While many Tyner Blain readers are interested in test automation, most of our readers would not want to go into this much depth. The audience at developer.* has a more technical focus, and their site is a more natural place for this article. Personally, I am honored to be included as an article author there – and all of you who want more technical depth than we go into at Tyner Blain should really check out the stuff Dan Read has put together. They’ve also just published their first indie-publisher ‘real book‘, which you can get via their site. My copy is on the way as I type.

How can you help?

This is a great opportunity for PR for Tyner Blain – please share the article with your associates because we want our community to grow. Also, it would be great if you ‘digg’ the article at digg (just follow the link, and ‘digg it’),, blink, or any other networking site. In addition to being good content, I hope to make this an experiment in viral marketing. Lets see if we can collectively reach the tipping point that brings this article, and the Tyner Blain community to the next group of people.

[update: the article just made the front page at digg and has 374 diggs and 32 brutal comments :). Welcome to all diggers, mobile, agile, or hostile!]

Foundation Series: Functional Testing of Software

Functional testing class

Functional Testing, also referred to as System Testing of software is the practice of testing the completed software to confirm that it meets the requirements defined for the software. A functional test is typically a test of user interactions, but can also involve communication with external systems. We contrast functional testing with unit testing. We also show how functional testing provides different benefits than unit testing.

This is a relatively long post for a Foundation Series post, so sit back with some coffee and relax. This primer will be worth it if its a new topic. If you know this stuff already, check out the links to other articles that go into more depth on points we make.

An Application is a Series of Flows

We can think of an application from the perspective of a user, as a series of interactions, or flows through the user interface.

Application flow

People are not usually forced to follow a fixed set of instructions, or a predefined sequence of actions in an application. They can interact with controls in a random order, skip controls entirely, or otherwise do stuff that developers don’t expect.

Unit Tests are Whitebox Tests

Unit testing, as we detailed in our telephone example, provides targeted test coverage of specific areas of the code inside the application. Unit tests are written by developers, to allow them to test that the implementation that they created is behaving as they intended. Unit tests don’t implicitly provide the start-to-finish coverage that functional tests usually provide. Unit tests are whitebox tests that assure that a specific behavior intended by the developer is happening. A weakness of using unit tests alone is that they will not identify when the developer misinterpreted the requirements.

unit testing

Functional Tests are Blackbox Tests

A functional test, however, is designed without insight into how the implementation works. It is a blackbox test. A functional test represents a set of user interactions with the application. The concept behind a functional test is to validate something about the state of the application after a series of events. According to Aberro Software, 80% of all functional tests are performed manually. That means that the most common functional test involves a tester making selections in a series of controls, and then evaluating a condition. This evaluation is called an assertion. The tester asserts that the software is in a specific state (an output is created, a control is filtered in a specific way, a control is enabled, a navigation option is disabled, etc).

full functional testing

Good functional requirements are written as concisely as possible. A requirement that supports a particular use case might state that the user specifies A, B, and C, and the application responds with D. A functional test designed to validate that requirement will almost always mimic this most common flow of events. The script that the tester follows will be to specify A, then B, then C. The tester will then evaluate the assertion that D is true. If D is false, then the test has failed.

A functional test may not cover the entire set of likely user interactions, but rather a subset of them.

Targeted functional testing

One problem with this approach is that it does not account for a user specifying (A, B, X, C) or (A, C, B). These variations in order of operations might cause the underlying code to execute differently, and might uncover a bug. For a tester to get complete coverage of the requirement (A + B + C => D), he would have to create multiple scripts. This is expensive, tedious, and often redundant. But a tester has no way to know if the multiple scripts are redundant, or required.

Combining Unit Tests and Functional Tests

When we combine both unit testing and functional testing approaches, we are implementing what is called graybox testing (greybox testing). This is also referred to as layered testing. Graybox testing provides two types of feedback into the software development process. The unit tests provide feedback to the developer that her implementation is working as designed. The functional tests provide feedback to the tester that the application is working as required.

Layered testing

Graybox testing is the ideal approach for any software project, and is a key component of any continuous integration strategy. Continuous integration is a process where the software is compiled and tested every day throughout the release cycle – instead of waiting until the end of the cycle to test. Read this plan for implementing continuous integration if you want more details.

Automating Functional Tests

Automating unit testing is both straightforward, and relatively inexpensive. Automating functional testing is more expensive to set up, and much more expensive to maintain. Each functional test represents a script of specific actions. A tester (with programming skills) can utilize software packages like WinRunner to create scripts of actions followed by assertions. This represents an upfront cost of programming a script to match the application, in parallel with the development of the application – and it requires a tester with specialized skills to program the script.

The maintenance cost of automating functional tests is magnified in the early development stages of any application, and throughout the life of any application developed with an agile process. Whenever an element of the user interface is changed, every script that interacts with that element can be broken (depending on the nature of the change). These broken scripts have to be manually updated to reflect these ongoing changes. In periods of heavy interface churn, the cost of maintaining the test suite can quickly become overwhelming.

In the real world, apparently 80% of teams find that this overwhelming cost of automated testing outweighs even the high cost of manual functional testing.

Improved Automation of Functional Tests

We can reduce the maintenance cost of keeping automated scripts current with the user interface by abstracting the script-coding from the script-definition. This is referred to as keyword and table scripting. A set of objects are coded by the tester and given keywords. Each object represents an element in the user interface. Script behavior (sequence of interaction) is defined in terms of these keywords. Now, when a UI element is changed, the keyword-object is updated and all of the scripts that reference it are repaired.

This, however, does not address issues where one control is refactored into two controls, the adding or removing of controls, or changes in the desired flow of interaction. There is still a very large (albeit smaller) maintenance burden. And the applications that use this approach (such as QTP) can cost in the tens of thousands of dollars. Another reason to do functional testing manually.


Functional testing is important to validating requirements. It is an important element of assuring a level of software quality. And it is still expensive with the best of today’s proven solutions. Even with the high cost, it is much cheaper than the risk of delivering a solution with poor quality. Plan on having functional testing as a component of any process to achieve software product success.

– – –

Check out the index of the Foundation series posts which will be updated whenever new posts are added.

Ten Essential Practices of Continuous Integration

Rubber chicken

Martin Fowler has identified the key process elements of making Continuous Integration work. You could even argue that they are the elements that define Continuous Integration (done correctly). We include his list and our thoughts below:

  1. Maintain a Single Source Repository
  2. Automate the Build
  3. Make Your Build Self-Testing
  4. Everyone Commits Every Day
  5. Every Commit Should Build the Mainline on an Integration Machine
  6. Keep the Build Fast
  7. Test in a Clone of the Production Environment
  8. Make it Easy for Anyone to Get the Latest Executable
  9. Everyone can see what’s happening
  10. Automate Deployment

For background information, check out the Foundation Series on Continuous Integration article.

1. Maintain a Single Source Repository

The smartest people I know use subversion when they have been able to make the choice themselves. Aside from being open source, it provides two key differentiated benefits relative to “everything else”.

  • Atomic commits: The ability to check in everything or nothing, so you don’t risk breaking the build with a partial check-in.
  • Overall project versioning: Allows you to track changes in source file directory hierarchies, file renaming, etc. Each version is of the entire project, not of a single file.

2. Automate the Build

Fowler sums it up perfectly:

“…anyone should be able to bring in a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine”

3. Make Your Build Self-Testing

Include automated testing as part of the build.

4. Everyone Commits Every Day

More frequently when possible. This is the minimum.

5. Every Commit Should Build the Mainline on an Integration Machine

A seperate, dedicated machine does a daily (or more frequent) build and full test suite run autonomously. The “build and test” model above relies on people to kick off the build when they commit. A scheduled task on a seperate machine provides a safety net for human error (oversight). Alternately, companies like Calavista can make this foolproof by automatically triggering an automated build as part of every commit. With Calavista’s devEdge, if the developer “commits”, what really happens is that the automated build/test cycle is triggered with his new code, and it gets promoted only if all the tests pass.

6. Keep the Build Fast

Tests can take a long time, builds should only take ten minutes. Fowler suggests a strategy of staged builds to address the 10-minute threshold. Run unit tests against the 10-minute build, and run the full suite in parallel or series.

Another option is to use statistical sampling of tests to get a “10 minute answer” while the full suite is kicked off in parallel.

7. Test in a Clone of the Production Environment

Eliminate even more variables. Make sure the tests are running against a clone of the production environment. Teams that are pushing the envelope today use virtual machines (VMs) to quickly create cloned production environments, install the software and run the tests.

8. Make it Easy for Anyone to Get the Latest Executable

Make sure everyone knows where the latest build can be found. Probably a good idea to keep recent builds in the same place too, in case a problem sneaks through the process temporarily (like a memory leak or other obscure, not-yet-tested situation).

9. Everyone can see what’s happening

Visibility! eMail the team when builds start/finish, including success/failure information. Put a rubber chicken on the desk of the person currently running the build (don’t ask – just read Fowler’s post). Ring a desk bell when the build passes. Have fun with it.

10. Automate Deployment

Make deployment into production as easy as running the build. Since tests are run against a production clone, and are already automated, this presents minor incremental effort.


Martin presents a great list. In addition to the above, we would suggest

  • Generate test-results documents per requirement. For each build, identify which requirements pass, fail, or are untested. The most relevant information for communicating outside of the team is status of previous requirements (did our regression tests pass?) and current requirements (are we almost done with this timebox?).