Category Archives: Testing

Testing can be automated or manual. Articles on testing at Tyner Blain involve tracing tests to other artifacts (code, requirements, etc). They also look at process approaches like continuous integration as a framework for testing, and occasionally go into the details of automated testing, or discuss the tradeoffs between blackbox and whitebox testing.

APR: Infrastructure Setup

under the hood

This morning I finished up the infrastructure setup for our project. A bunch of under the hood work. Definitely required some propeller-head skills. The goal of this work is to get us to a working prototype as soon as possible. There are a couple links to some good agile testing articles, and the rest of this one is a quick list of what we did on Friday and today.

Continue reading APR: Infrastructure Setup

Is Agile Bad For Software Development?

broken software

Last week, Ivan Chalif, a product manager / blogger, tapped into a thread criticising product managers for not adopting and espousing agile, or at least rapid-release techniques. In this article we look at Ivan’s comments and one of the articles that he referenced. We also share our own perspective and an alternative analysis of what may have happened.

Continue reading Is Agile Bad For Software Development?

The Difference Between Use Cases and Test Cases

praying mantis

People who are new to software, requirements, or testing often ask “What’s the difference between a use case and a test case?” This article answers that question, by building on earlier articles about use cases and use case scenarios. At the soundbite level, each use case has one or more scenarios, and each use case scenario would lead to the creation of one or more test cases.

Continue reading The Difference Between Use Cases and Test Cases

Crossing The Desert With Bad Project Planning

Johanna Rothman recently wrote an article with a poignant introduction: “A project team focuses on an interim milestone, works like the devil to meet that milestone. They meet the milestone, look up, and realize they’re not at the end of the project–they still have to finish the darn thing. They’re living the Crossing the Desert syndrome.” Fixing it isn’t enough – how do we prevent it from happening?

Unrealistic Scheduling

Johanna points out that when a team has to put in overtime (what we might call heroic efforts) to achieve an interim milestone in a project, the remaining project schedule is suspect. Johanna reccommends revisiting the remaining schedule, and we agree.


Johanna also highlights the way to recover – give the people on the project a break. Have them move back to 40 hour weeks if they were working overtime. Force them to take the weekends off if they weren’t already. In short, restore a rational schedule. If people worked through vacations or holidays, require them to take the time off.

Its been said that it takes two weeks of down time to recover from work-burnout. Extending the desert analogy that Johanna credits to Jack Nevison of Oak Associates, we’ve reached the oasis in the middle of the desert, and we need to stay there long enough to rest and rehydrate.

Unfortunately, we’re still in the middle of the desert, and the rush to reach the oasis presumably had merit. If a project schedule is so intense that we have created the need for recovery in the middle of the project, the likelihood of having two weeks to recover is low. Whatever drove us to push to reach the oasis is still driving us, so resting for too long will only make it worse. And we’re still in the middle of the desert. We need to focus on prevention.


irrigation to prevent desert

We can’t always prevent the impetus to work long hours on a project. We can manage our execution in a way that reduces the chances of feeling like we’re crossing a desert.

  • Improved estimation of tasks. Entire books are written about this topic, we won’t try and summarize ways to do this in a single blog post.
  • Realistic effort allocation. When scheduling how many hours a day a developer can be “on task”, our experience has been that 5 to 6 hours is the maximum (when working full time or a little more). For requirements work, this might even be a little bit aggressive.
  • Writing verifiable requirements. We need requirements that specify (or at least allow us to identify) when we’re done. Scope creep isn’t always implementing more features, it can be over-implementing features. With test-driven development (TDD) processes, the tests are written first (they fail), and the developer writes code until the tests pass. Once the tests pass, the code is done. Doing this requires us to write testable requirements. Practically speaking, this may only be realistic when using a continuous integration approach to code development.
  • Managing smaller work chunks. Timeboxing (more details) is very effective for controlling the size of iterations. Iterations are important not only for the feedback cycle, but because they reduce the size of the “desert patches.”
  • Feedback into the estimation cycle. Timeboxes become even more effective when we keep track of our estimation accuracy, and refine those estimates on the fly to help in replanning future releases. This is a key step in the maturation of a process or company from a CMMI perspective.
  • Better communication of release content. Planning releases based on use cases / use case versions is a great way to target communication for external (to the team) stakeholders. It can also be really effective for helping people avoid the desert-crossing syndrome. Essentially you’re providing a map (“The next watering hole is just over that sand dune”) that helps keep efforts in context. One reason the desert image is so powerful is that we imagine being lost in a sea of sand – the hopelessness is a function of both the reality and the perception. Better communication helps address perceptions, while other elements help with the reality.


Overworking the team is bad. Burning them out in the middle of the project is worse. Prevention is the solution, through iterative development, better communication, and improved estimation/planning skills.

Rinse, Lather, Repeat – 10 Reasons to Repeat Tests


Why run a test more than once? If it passed the first time, we don’t need to run it again – or do we? James Bach provides ten good reasons to run the same test more than once.

Common Sense

Common sense dictates that we don’t need to run the same test twice – once we’ve run a test, we know the results – and we therefore don’t need to run it again.

The definition of insanity is doing the same thing over and over again and inspecting different results.

Benjamin Franklin

James Bach writes a great article that points out that we can never do exactly the same thng every time, so running the same test again is not insane. He presents the following ten reasons to run the same test more than once. The commentary here is ours – check out his original post for more details, and other good writing on the topic.

  1. Recharge – By changing the code that is being tested, we recharge a test making it valuable again.
  2. Intermittence – Some bugs repeat as a function of otherwise uncontrolled variables.
  3. Retry – Sometimes we don’t run a test correctly, or can’t be certain that we ran it correctly.
  4. Mutation – By creating a test that is a variation on an existing test, we are repeating that subset of the test that is unchanged, but in a different context, established by the mutated version of the test.
  5. Benchmark – Performance measurement is most easily understood by establishing a baseline performance with a set of tests – then repeating those tests as the code is modified (or the platform it runs on is modified).
  6. Inexpensive – As long as there is some value to a test, and the cost of running it is low – what does it hurt?
  7. Importance – “This will never happen again” – best prevented with a regression test.
  8. Enough – James uses virus scanning as an example – honestly, this one doesn’t click for us. If you understand it, please add a comment.
  9. Mandated – The man always puttin’ me down, makin’ me run the same test over and over again.
  10. Indifference / Avoidance – What if your goal isn’t to find bugs, but to avoid them? What better way than to run the same test over again?

Sightings in the Real World
Continuous integration is an approach to testing/development that explicitly calls for the repeated running of tests as part of software development. In James’ framework, continuous integrations uses recharge, intermittence, and mutation as the drivers for repeating tests.

Combining unit tests and functional tests is an effective way to approach system testing. This takes advantage of mutation to provide effective coverage.

By contrast, pairwise testing is designed to achieve enough without repeating any tests.

Insight Into Test Driven Development


James Kovacs shares a great insight on software testing and the software testing process. His epiphany about test driven development makes it obvious to all of us why this technique is so powerful.


Test driven development is an approach to software development where you write the tests first, and the code second. This sounds ludicrous at first, but it really does work. Here’s James’ epiphany on the subject (emphasis added).

I didn’t understand why TDD practitioner were so zealous about writing the tests first. Why did it matter? I thought it was to ensure that some project manager doesn’t try to shave some time off the project by cutting the unit tests. Then I started doing some reading about TDD, poking around, asking questions, and trying it out myself. I discovered the real reason for writing tests first is that TDD isn’t about testing code, it’s about designing code.

A Tale of Two Epiphanies: TDD and Mocking, James Kovacs

These tests are unit tests, and they are black-box tests (or at least should be black-box tests – read James’ article).

Works Great With Structured Requirements

Not only is this a great way to approach design, but it is a great way to coordinate software development with structured requirements. A well-written requirement describes a need without specifying design. This description then works like a black-box contract. The implementer then must do something that meets the specification given. As James points out, TDD is very effective at creating black-box tests.

This makes sense, when you consider the outside-in approach that comes from starting with requirements. This outside in approach leads to API decisions (for a program or module or method). The API has to support all of the required possible inputs and outputs. Once an API is written, tests can be written.

Best Applied As Part Of Continuous Integration For Larger Teams

All of the tests will fail (because the code hasn’t been written). Then the implementer will write code until the tests pass. Using a continuous integration approach is a very effective way for entire teams to do this simultaneously on larger projects.

Faster Processes

We can also validate the design for completeness prior to creating the implementation. We can compare the generated APIs (and their rationalizations) with the software specification (or application requirements, or FRS… see alphabet soup article). This allows us to address some refactoring needs before the code has been implemented – saving time and money.


Use TDD as an outside-in approach that assures that the delivered software will meet the objectives of the requirements specification. It also saves time and money by supporting validation in advance of implementation.

John Henry, Manual Tester

John Henry

There’s a piece of North American folklore about John Henry, who was a manual laborer during the expansion of the railroads in our country. His job was being replaced by steam-driven heavy equipment, as the railroad industry applied technology to become more efficient. The same dynamics are happening today with manual testers. We need to make sure that manual testers avoid John Henry’s fate – read on to see why.

Manual and Automated Testing

Software can not be tested efficiently with only manual testing. And it can not be tested completely with only automated testing. James Bach writes an insightful article about how he worked his way to enlightenment over several years, and how he has incorporated this perspective into his training classes.

My understanding of cognitive skills of testing and my understanding of test automation are linked, so it was some years before I came to understand what I now propose as the first rule of test automation:

Test Automation Rule #1: A good manual test cannot be automated.

James goes on to explain that if a test can be automated, it is not a good manual test.

We’ve discussed repeatedly the benefits of automated testing, one example is The Code Freeze is Killing the Dinosaurs. We’ve also stressed the importance of doing functional testing where we discuss the dynamics and tradeoffs of using manual and automated approaches to test software.

Generally, we’ve approached the problem from the perspective of blackbox versus whitebox testing (and more details), and thought about our development process from the perspective of continuous integration as a means to deliver more efficiently.

We’ve not thought about it from James’ perspective before. Even with our improved approaches to automated testing, we are still no different than the inventor of the steam-hammer in Henry’s fable.


James puts things in perspective:

Rather than banishing human qualities, another approach to process improvement is to harness them. I train testers to take control of their mental models and devise powerful questions to probe the technology in front of them. This is a process of self-programming. In this way of working, test automation is seen as an extension of the human mind, not a substitute.

He highlights a very interesting point – we are missing a key benefit of having manual testers – the ability to gather feedback on intangibles, heuristics, usability, affordances, and other elements that we might classify as design bugs.

We talk about the importance of usable software, and different approaches to creating great software. But we’ve never really talked about how to test for software greatness.

A Broken Model

In the 1970’s, when the American automotive manufacturers were getting their clocks cleaned by the newly exporting Japanese companies like Honda, Datsun (now Nissan), and Toyota, a lot of people thought they understood the problem. Many more people pretended there wasn’t a problem, until the US government bailed out Chrysler.

The experts decided that the main problem was that quality was not “Job One”, that American manufacturers ignored the advice of W. Edwards Demming, and the Japanese did not. Toyota’s lean manufacturing philosophy is credited with much of the success.

It is true that the oil crisis of the time gave the Japanese companies an opportunity to penetrate the US market with smaller, more efficient cars. But that crisis ended (about the time that the American companies killed the muscle car and began building more efficient cars). But the Japanese cars didn’t go away. Once gas prices dropped, they lost a differentiating element – but they maintained two others. Cost and Quality.

Target of Opportunity Enables Strategic Advantage

Cost matters. But it matters tactically, as in “I can afford $X right now – what are my choices?” Quality creates Loyalty. And loyalty is strategic. The Japanese manufacturers gain and continue to gain success after success and award after award in the automotive industry. Why? Because they have good quality.

Automated and Manual Testing

As an engineer, I know that we can specify tolerances, inspect components, and test assemblies to make sure that products are “within specification.” And all of this testing can be automated – just like automated software testing. But passing these tests doesn’t make a product good, it merely indicates that the product is “within specification.” Did the Japanese manufacturers have tighter tolerances? Yes. But did they have better designs? Yes. And those better designs were about more than miles-per-gallon and horsepower and torque.

They were about qualitative items that many engineers struggle to absorb. “Feels Good” and “The controls are where I expect them to be” and “Makes sense” and “I like it.” Software developers often struggle with the same issues.

And this is where manual testing matters. Getting qualitative feedback on your product designs is the only way to improve qualitative elements of those designs. The American manufacturers showed their disdain and hubris in allowing their customers to provide that feedback. The Japanese companies got feedback before they sold their products.

We can use manual testing to provide us with the kinds of feedback that can’t be automated. Things like “This UI is confusing” and “Why won’t it let me…?”


Don’t give up on manual testing as we strive to achieve software product success. Automate everything we can, and use people to test what we can’t.  We need to make sure that we don’t lose the ability (or make sure that we find the ability) to test manually, for qualitative insight into our products.  We can’t let the testers burst their hearts like poor John Henry, trying to manually perform an automatable test.
Thanks again, James, for starting us down this path!

Quality and Requirements – You Got Chocolate In My Peanut Butter

Reeses Cup

Quality writers are writing about requirements. Requirements writers are writing about quality. Just like the Reese’s Peanut Butter Cup – Two Great Tastes that Taste Great Together. You can’t have one without the other.


The Business Analyst Blog writes about speeding up testing:

Everyone has suggestions about how to improve your testing—implement a testing process or methodology, utilize IEEE standards, work towards CMMI compliance, etc. No one mentions that improving requirements will improve testing! […] Why does it take so long? I would argue that one of the main reasons is that poor (or missing) requirements slow down the process as much as any other issue.

Well said. If we’re testing for the wrong things, we’re wasting time and resources.


The Software Quality Blog writes about requirements best practices:

No process is more fundamental than the process of defining and managing business and technical requirements. It’s no surprise that studies cite inaccurate, incomplete, and mismanaged requirements as the primary reason for project failure.

Well said again. The Standish Group estimated that 40% to 60% of defects were due to errors in requirements, not code.


In Where Bugs Come From, we showed that having the requirements errors happen throughout the process, not just at the beginning of the process. Numbers 3 and 4 of the following list are the painful and visible impacts of the problems both of our authors above are addressing. And those errors are introduced in numbers 1 and 2:

Requirements errors can happen

  1. When stakeholders incorrectly define their goals. A focus on ROI is the best way to assure that our stakeholders are focusing on the right part of the business. Elicitation techniques help us to drive this process in a way that tends to focus on the right requirements. The centerpiece of elicitation is interviewing stakeholders – both users and beneficiaries of the system.
  2. When product managers / business analysts incorrectly document stakeholder goals. Writing good requirements is an absolutely masterable skill. Check out these top ten tips for writing good requirements.
  3. When developers write unit tests of the wrong implementation. Even with a great continuous integration strategy, all we end up doing is efficiently testing the wrong stuff.
  4. When test teams assure that the wrong requirements have been implemented. We waste time testing requirements that aren’t really requirements. And we waste time tracking down false positive defects.


When we’re selling the value of requirements engineering, use this argument to explain the benefits (or impacts) of good (or bad) requirements. When we’re responsible for implementation (dev and test) on a project, remember why we care about requirements. It isn’t enough to verify that requirements are implementable. We have a vested interest in validating that the requirements are also complete and valuable.


[disclaimer: I am a bit of a dork.] I cracked myself up today when I typed “errors of omision”.

Writing Verifiable Requirements

big ten logo

One of the ten big rules of writing a good MRD is writing verifiable requirements. Verification is both a function of having a precise goal, and having the ability to affordably measure the requirement. A precise goal is a verifiable requirement if we can clearly answer “yes” or “no” when asked if the requirement has been implemented. We also face the practical realities of being able to measure the results profitably.

The Big Rule of Writing Verifiable Requirements

From our previous article, Writing Good Requirements – Big Ten Rules:

We use a process that starts with market requirements, and then decomposes them into software requirement specifications. the market requirements must be written in a way that we can verify that the associated requirements specification will meet the market need.


We wrote previously that if a requirement can’t be tested, it should be rewritten. Roger Cauvin has recently reposted on testable requirements. Perhaps in anticipation of this article? :).

In the world of test driven design (TDD), the philosophy is to write the test first, and then write the code. We use a continuous integration approach to check in code regularly and run all of the tests as we go. When the test for a particular requirement passes, the code is done for that requirement.

We also get a big benefit in validating that we have delivered the right functionality to the customer. When we agree on the language of the requirement, we are also agreeing on the language of its verifiability.

Affordably Verifiable

With this big rule, we are assuming that we have already eliminated impractical or unattainable requirements.

Some tests are very expensive, or nigh on impossible. Roger alludes to this as testable in principle. Requirements should be testable in practice whenever possible. If a requirement can not be tested (“Must last 10 years”) in the delivery timeframe, we need to at least define a specific acceptance criteria – (“95% confidence in a cycle life of 10,000 cycles per procedure XYZ.doc”). This is the approach that Underwriter Labs uses for awarding UL certification to electrical appliances for safety.

Semantically, we can argue that this proxy criteria is in fact the requirement. It is the criteria by which acceptance is determined.

If the test is possible, but impossibly expensive, we shouldn’t run the test. If we can not agree on a practical proxy criteria, we are left with two choices. We can choose to not implement the requirement. Or we can choose to agree that the requirement is delivered when we say it is. Neither is a particularly good option, but the former is better than the latter, when it comes to maintaining a positive relationship with our customer.

Writing requirements without acceptance criteria introduces risk into our project. The last thing we want is to enter into a nitpicking discussion about what we have and have not delivered. We would much rather spend that time discussing what we can deliver next.


Write requirements that can be verified given the constraints of the project. Common constraints are time, money, equipment and expertise. Use language that makes the verification process explicit. The process of determining verifiability also dramatically helps eliminate ambiguity in our requirements.

Test Smarter, Not Harder – A Detailed Article

Scott Sehlhorst

developer.* has just published a 15-page article, Test Smarter, Not Harder by Scott Sehlhorst on test automation. We present the background for that article here (why you should read it) as well as a summary of what is covered in the article. Check it out at developer.*, and thanks to Dan Read, both for starting developer.* (which is a GREAT site) and for publishing the article. Skip to the end of this post to see how you can help.

Automated Testing Must Be a Core Competence

Automated testing has become a critical component of software product success. Processes like continuous integration require test automation to be effective. Unit testing requires automated testing to be effective as a rapid-development tool. Whitebox testing is almost always done with automation. More and more blackbox testing is being done with automation. A team without a good approach to automated testing is working at a severe disadvantage, and is arguably doomed to extinction.

Functional Testing (or system testing) is where a team can achieve process differentiation with automation. 80% of teams rely on manual functional testing today. These teams argue that automating functional tests is too expensive. The reasons most cited are complexity and rapid change in the underlying product.

Testing Complex Software

Complex software presents very challenging problems for test automation. First – how do we deal with the extreme complexity of many of today’s software applications? Billions or trillions of combinations (scripts) of user actions can be made. And software that works with one combination may be buggy in hundreds of others.

We can’t realistically test all of the combinations. Even if we did have the hardware needed to run them we would not be able to validate the results of all of those tests. So we can’t achieve exhaustive test coverage.

Without exhaustive test coverage, how do we know when we’ve tested enough?

Test Smarter Not Harder

The article starts with an analysis of the complexity of modern software and a brief discussion of the organizational realities and typical responses. Then the article explores common approaches and techniques in place today:

  • Random Sampling
  • Pairwise Testing
  • N-wise Testing

The article explores the math, benefits, and limitations of each approach. While N-wise testing is the most effective approach in common use today, it has a limitation – it assumes that the user’s actions (input variables) happen in a prescribed sequence or are irrelevant. In most complex software this assumption is invalid. The article presents a technique for incorporating order-dependence into the statistical approaches for developing test coverage.

The article also demonstrates techniques for simplifying the testing representation using whitebox testing techniques to redefine the problem space, and then applying blackbox testing (statistical) approaches to execution of the resultant test parameters.

By combining the statistical techniques explained in the article with a communication plan for sharing the appropriate quality metrics with stakeholders, we can deliver higher quality software, at a lower cost, and with improved organizational support and visibility.

Why developer.*?

While many Tyner Blain readers are interested in test automation, most of our readers would not want to go into this much depth. The audience at developer.* has a more technical focus, and their site is a more natural place for this article. Personally, I am honored to be included as an article author there – and all of you who want more technical depth than we go into at Tyner Blain should really check out the stuff Dan Read has put together. They’ve also just published their first indie-publisher ‘real book‘, which you can get via their site. My copy is on the way as I type.

How can you help?

This is a great opportunity for PR for Tyner Blain – please share the article with your associates because we want our community to grow. Also, it would be great if you ‘digg’ the article at digg (just follow the link, and ‘digg it’),, blink, or any other networking site. In addition to being good content, I hope to make this an experiment in viral marketing. Lets see if we can collectively reach the tipping point that brings this article, and the Tyner Blain community to the next group of people.

[update: the article just made the front page at digg and has 374 diggs and 32 brutal comments :). Welcome to all diggers, mobile, agile, or hostile!]