John Henry, Manual Tester

There’s a piece of North American folklore about John Henry, who was a manual laborer during the expansion of the railroads in our country. His job was being replaced by steam-driven heavy equipment, as the railroad industry applied technology to become more efficient. The same dynamics are happening today with manual testers. We need to make sure that manual testers avoid John Henry’s fate – read on to see why.

Manual and Automated Testing

Software can not be tested efficiently with only manual testing. And it can not be tested completely with only automated testing. James Bach writes an insightful article about how he worked his way to enlightenment over several years, and how he has incorporated this perspective into his training classes.

My understanding of cognitive skills of testing and my understanding of test automation are linked, so it was some years before I came to understand what I now propose as the first rule of test automation:

Test Automation Rule #1: A good manual test cannot be automated.

James goes on to explain that if a test can be automated, it is not a good manual test.

We’ve discussed repeatedly the benefits of automated testing, one example is The Code Freeze is Killing the Dinosaurs. We’ve also stressed the importance of doing functional testing where we discuss the dynamics and tradeoffs of using manual and automated approaches to test software.

Generally, we’ve approached the problem from the perspective of blackbox versus whitebox testing (and more details), and thought about our development process from the perspective of continuous integration as a means to deliver more efficiently.

We’ve not thought about it from James’ perspective before. Even with our improved approaches to automated testing, we are still no different than the inventor of the steam-hammer in Henry’s fable.

Enlightenment

James puts things in perspective:

Rather than banishing human qualities, another approach to process improvement is to harness them. I train testers to take control of their mental models and devise powerful questions to probe the technology in front of them. This is a process of self-programming. In this way of working, test automation is seen as an extension of the human mind, not a substitute.

He highlights a very interesting point – we are missing a key benefit of having manual testers – the ability to gather feedback on intangibles, heuristics, usability, affordances, and other elements that we might classify as design bugs.

We talk about the importance of usable software, and different approaches to creating great software. But we’ve never really talked about how to test for software greatness.

A Broken Model

In the 1970’s, when the American automotive manufacturers were getting their clocks cleaned by the newly exporting Japanese companies like Honda, Datsun (now Nissan), and Toyota, a lot of people thought they understood the problem. Many more people pretended there wasn’t a problem, until the US government bailed out Chrysler.

The experts decided that the main problem was that quality was not “Job One”, that American manufacturers ignored the advice of W. Edwards Demming, and the Japanese did not. Toyota’s lean manufacturing philosophy is credited with much of the success.

It is true that the oil crisis of the time gave the Japanese companies an opportunity to penetrate the US market with smaller, more efficient cars. But that crisis ended (about the time that the American companies killed the muscle car and began building more efficient cars). But the Japanese cars didn’t go away. Once gas prices dropped, they lost a differentiating element – but they maintained two others. Cost and Quality.

Target of Opportunity Enables Strategic Advantage

Cost matters. But it matters tactically, as in “I can afford $X right now – what are my choices?” Quality creates Loyalty. And loyalty is strategic. The Japanese manufacturers gain and continue to gain success after success and award after award in the automotive industry. Why? Because they have good quality.

Automated and Manual Testing

As an engineer, I know that we can specify tolerances, inspect components, and test assemblies to make sure that products are “within specification.” And all of this testing can be automated – just like automated software testing. But passing these tests doesn’t make a product good, it merely indicates that the product is “within specification.” Did the Japanese manufacturers have tighter tolerances? Yes. But did they have better designs? Yes. And those better designs were about more than miles-per-gallon and horsepower and torque.

They were about qualitative items that many engineers struggle to absorb. “Feels Good” and “The controls are where I expect them to be” and “Makes sense” and “I like it.” Software developers often struggle with the same issues.

And this is where manual testing matters. Getting qualitative feedback on your product designs is the only way to improve qualitative elements of those designs. The American manufacturers showed their disdain and hubris in allowing their customers to provide that feedback. The Japanese companies got feedback before they sold their products.

We can use manual testing to provide us with the kinds of feedback that can’t be automated. Things like “This UI is confusing” and “Why won’t it let me…?”

Summary

Don’t give up on manual testing as we strive to achieve software product success. Automate everything we can, and use people to test what we can’t. We need to make sure that we don’t lose the ability (or make sure that we find the ability) to test manually, for qualitative insight into our products. We can’t let the testers burst their hearts like poor John Henry, trying to manually perform an automatable test.
Thanks again, James, for starting us down this path!