People who already use Scrum will only find one new thing in this article – a way to communicate what happens inside a sprint that has proven effective for me.  People who are new to Scrum who wonder “how do things work inside a sprint?” will see how things work in a way that avoids hyperbole and is easy to map to what they already understand from traditional software development processes.

Two Teams Separated by a Common Process

George Bernard Shaw was an Irish author in London who memorably said “England and America are two countries separated by a common language.”  The same can be said about teams developing in agile and waterfall processes.

The story of agile adoption is one that has many colorful examples of adoption and of spreading the gospel.  Some practitioners traveled across the ocean from the agile continent to the lands of agile and opportunity to reap the rewards.  A few of those, after realizing the benefits of agile, invade our consciousness with a hellfire and brimstone tale of doom for anyone who doesn’t convert to the new religion.

Others seem hell-bent on kidnapping entire development teams, smuggling them across the ocean in the belly of steamer ships and unceremoniously dumping them in the land of agile, ready or not.  Scrum is but one branch of the agile movement.

Stalwarts of the old way, which has worked fine for me, thank you very much, refuse to leave their comfortable lives to step into the unknown wilds of agile.  Open-minded but responsible potential converts ask questions to gain an understanding of what life in the new world might be like.  What will life be like if I join this Scrum congregation?

How Does Scrum Really Work?

The best explanations of how scrum works that I’ve read come from Mike Cohn and Mountain Goat Software.  The training, videos, and explanations they share provide fantastic top-down introductions (as well as guidance after adopting Scrum).  His book, User Stories Applied, is the ultimate reference when gaining an understanding of the mechanics of using user stories as the central artifacts for developing software with scrum.

[larger image]

The process diagram above provides a good outside-in, top-down view of how “this newfangled process” results in shippable product, incrementally.  It is a great way to present the concept of Scrum, and incremental development in general.

When people I’ve worked with first gain this understanding, they often ask, what are the artifacts that live in these backlogs?  User stories live there.  Sometimes use cases make more sense than user stories for Scrum- it depends (read the article to see which works best in your circumstance).  The common element is an outside-in, user-centric approach to describing which problems the software is intended to solve.  This is also known as goal-driven development.

Once people understand the top-down view of the process, and have an idea of what user stories are, the next question they tend to ask is “how do stories move through the Scrum process?”

The first attempt at answering this question leans on the process diagram above:

1. The product owner takes the most important stories from the product backlog and puts them into the sprint backlog – collaborating with the team to agree on the scope of work for this sprint (based on the amount of work, capacity of the team, real-world constraints, etc).
2. The team “works the stories“, meeting every day to communicate effectively and provide progress updates – best visualized with a burndown chart.  This chart says “we started with X work to do, and every day, we track how much of X remains to be done.”
3. At the end of the sprint, the work is done, and the software (or an update to the software) is ready to deliver.
4. Sometimes, multiple sprints happen between releases – launches of the updated software, because customers (and the rest of your organization) incur a cost when changing to the latest version.  But the output of each sprint is deliverable – that’s a key concept – even if you choose not to deliver it just yet.

What Does it Mean to Work The Stories?

I’ve repeatedly had people ask, after absorbing the above explanation, “what does it mean to work the stories?”

I’ve had consistent success (in bridging the language divide) using the following diagrams (drawn on a whiteboard) to explain how stories flow through the sprint process in Scrum.

The first diagram shows that user stories have a structure – the story itself, and the acceptance criteria for the story. I also establish that the story is going to go through a life cycle within the sprint.

[larger image]

Next I remind folks that the user stories make it into the sprint backlog because they are the most important (highest priority) not-yet-implemented stories in the product backlog.

[larger image]

Developers on the team pull stories from the sprint backlog and begin working on them.  These stories are work in progress, and are owned by individuals.  Some teams maintain that priority order within the sprint backlog (e.g. implement the most important story in the sprint first, etc.), while other teams use a more coarse-grained approach, relying on the product backlog prioritization to assure that everything in this sprint is the next most important stuff, and the order that the team implements, within the sprint, is up to the team.  Some folks like to strenuously debate this decision (don’t prioritize within sprints vs. do prioritize within sprints).  Both approaches are valid, and you should choose the right one for your team.

[larger image]

As each user story owner believes that her work is complete (all unit tests pass, and the developer believes that the acceptance criteria have been met), the story is ready to be tested by QA.  This step, in particular, helps people who are new to scrum and who think of things as “code and test”, to make some sense of the inner workings of a sprint.  When working with a team that was under-staffed in the QA department, I discovered that calling out this queue of user stories waiting to be tested helped convince management to increase QA investment in the team.  When too many stories pile up here, you know you have a problem.

[larger image]

The scrum team member responsible for testing the user story has the responsibility of assuring that it meets all of the acceptance criteria.  He pulls the story from the queue of “done” stories, and tests.

[larger image]

If the story fails to meet any of the acceptance criteria, it moves back into either the sprint backlog or the being developed column, where a member of the team resumes ownership of the user story and works to resolve the issue.  Once the issue is resolved, the user story is re-queued for testing.  When the user story meets the acceptance criteria, it is moved into the done done column.

[larger image]

Conclusion

If you are new to Scrum, and wondered how the sausage is made inside a sprint, this gives you a framework for understanding what is going on, and how the team delivers.  If you already know or use Scrum, you may have learned a couple things.

• Using this framework to describe how a sprint works is effective when explaining to people who do not have any experience with agile.
• Laying things out this way visually, will give you an easy-to-see signal if your team is unbalanced between developers and testers.

I’ve been involved with agile teams and processes for just over 11 years now, and incremental development is burned into my brain.  If you are new to scrum, please let me know if this seemed like a good way for you to consume this material.  If you’re an old hat, and have other suggestions or presentations, please add them in the comments below, because future readers will benefit from the additional ideas.

Jump forward in time to the day of your next big product launch (first release, new features, new market segment, etc).  And your site/application crashes due to the “unexpected” demand.  All you can do now is look for a bucket of water to put out the fire.  What could you have done to prevent this disaster?  Jump back to today and start doing it!

Backwards Planning

Depending on how you look at things, this is a backwards planning exercise, or a variation of the  remember the future innovation game, or risk management, or proactive product management.  You can avoid a disaster by imagining what might happen, then hypothetically figuring out why it (would have) happened.  That leads to planning how you could prevent it.  And now you’ve left the dream world of a Gedanken experiment and returned to the real world of product management.

Problem Triage

The way to approach this is straightforward.  Imagine some failure scenarios and the importance of preventing them:

1. Imagine a failure scenario.
2. “Predict” the likelihood of the failure.
3. “Estimate the impact of the failure.
4. Repeat for each scenario

You can prioritize your failure scenarios by multiplying the likelihood of each with the impact of each, and sorting them from largest to smallest.  Then determine which ones you’re willing to address, and which ones you’re willing to risk.  You may not be able to predict the likelihood of some failures (at least until you do a root cause analysis).  Take each of these and put them directly above the scenario with the next highest impact.  The rationalle is that these are so bad, that you really want to find out how likely they are to happen.  Once you predict likelihood (see below) you can reprioritize.

Root Cause Analysis

For the failure scenarios you choose to address, the next step is to do a root cause analysis that identifies why it might have happened.  The best tool for capturing this analysis is an Ishikawa diagram.  Consider that one problem you might face is your website crashing.

[click for larger version]

Essentially, you can crash your site by having too many users, too many concurrent users, or too many concurrent sign-ups.  Developing a cause-and-effect diagram (another name for an Ishikawa diagram) is usually an iterative and exploratory process.  You probably won’t create the simple version above first.  You may ask your implementation team “What can cause the website to crash?”  For each of their answers, you identify when that situation can happen.  Or you start top down.  Most likely, a mix of the two.  Your completed root cause analysis may look like the following:

[click for larger version]

At this point, your team can probably predict many (maybe all) of the root causes of a website crash.  The predictions may be conditional – “we can handle 10 concurrent users, but 20 probably kill us, and 100 definitely would.”  Developers are notoriously good at answering questions with conditional statements that reveal the nuances of their thinking.

Remember that you’re looking back from the future.  At product launch, what are you hoping for / reasonably expecting?  For this example, assume it is 10,000 total users, with 100 concurrent users (normally) and 500 concurrent signups.  You determine these numbers by working with your PR, marketing, or mar-com people (or wearing those hats, when it is all you).  Your plan is to do a big launch with a demo and a promo code for signup.  You know your audience will have internet connections, and will have twitter running at the time of your presentation.  You expect/dream of an immediate burst of signups, followed by tweets and word of mouth, and eventually blog articles causing additional growth over the next couple of weeks.

Use this data to feed back into the developer’s conditional responses.  If you’re like me, you will have found “absolute certainty of failure” from something.  And you may have even identified the thresholds for each element.  For example, database loading can handle 75 concurrent users, but with the current implementation, you only have enough database connections available to support 25 concurrent users.

Jumping back to the present, you now have some very discrete, and very important things to do before your launch.  If you need to, revisit the prioritized list of failure scenarios.  By looking at the next level of detail, have you found that the order of importance (to fix) has changed?  What about the “must fix” versus “willing to risk” line?  Has it moved?

Fold the “must fix” items into your backlog, and prioritize them relative to the other capabilities on your roadmap.  As a side note – make sure you’ve built in some testing to make sure you actually prevent the problems.  This might even be a great opportunity to implement “performance regression tests” – it is not enough to prevent bugs, you have to prevent slowdowns.

Rethinking the Problem

Without going into details on how the team will solve each problem, make sure that together you keep the Ishikawa diagrams in mind, and see how any proposed solutions might “reappear” on the diagram.  For example, rewriting your database connections to use asynchronous processes and a set of pooled connections may prevent a crash, but it may really hurt performance.  You may not have time to find an elegant solution.  So stop and rethink the problem.

At this point, you’ve said

1. Given a marketing plan / launch strategy, we would crash the website.
2. We can make changes between now and the launch that will double the number of concurrent users we can support (or whatever), but that is not enough to support the launch strategy.
3. Solution: Change the launch strategy.

Maybe you can’t support a wide-open promo-code based signup.  You should modify your launch so that it can only create as much demand as your product (including pending improvements) can support.  Maybe you limit it to the first 1,000 new users (probably more code to write to enforce the limit).  Maybe you launch with per-user invitations, where you can control the speed of propagation of invites (start with 100, when those have been sent, make another 100 available, etc).

Entire Team Problem

This is a problem that is solved collaboratively, by the entire team.  It is not just a “go write the code” problem.  What your product can support at a launch should drive how you choose to launch, just as how you choose to launch should drive what you want your product to support.  

You may have to delay a key capability in order to scale.  Does your marketing team know this?  Slightly less bad than crashing would be announcing a feature that is disabled.  Still need to announce the feature?  Pre-announce it: “Coming in a month…”

This stuff is important for every company and product, but it is especially critical for start-ups.  As a start-up, you have limited opportunities to grow, and a limited safety-net to catch you when you fail to capitalize on those opportunities.  So make sure everyone (not just the development team) is aligned to make the best use of each opportunity.

Conclusion

You have an opportunity to prevent problems.  All you have to do is imagine that they have happened in the future, figure out why they would have happened, then do what it takes (in software, or organizationally) to prevent them.

When companies first start off-shoring, they usually send the “low level” implementation work overseas first, to work out the process kinks and manage risk. Over time, your valued, domain-aware developers will perceive a lack of career opportunities with this limited role. Naturally, you will want to consider sending design work offshore too. You can make it work. If you do it wrong, you’re toast.

Different Models for Offshore Development

There are many ways to organize software-creation teams (where creation includes product management, design, development and testing). When developing an organizational design that incorporates elements of off-shoring, there are four primary models for outsourced software creation.

1. No offshore development at all. Occasionally referred to as insourcing, this is the traditional “everyone in one place” model – or at least “everyone in similar time zones.”
2. Low-level outsourcing. The implementation teams (both coding and testing) are located offshore, with design and product management staying local.
3. High-level outsourcing. The focus of this article. Keeping product management “local” but moving design and development responsibilities offshore.
4. Complete technical outsourcing. Everyone except the product manager is offshore.

The models can be most easily compared when the same process is compared in each – just with different locations. Co-location of team members has an impact when comparing face-to-face communication with online / phone / remote communication. But this factor is not a primary one in influencing team-decisions – it is no different than having someone who works from home. The key element in team dynamics is a dramatic shift in timezones.

This timezone shift causes a latency in the communication process, illustrated by the following example:

Imagine the following expensive question and answer session:

• Person A asks person B a question.
• 12 hours later, person B responds with a request for a clarification.
• 12 hours later, Person A clarifies the question.
• 12 hours later, Person B responds with an answer.
• 12 hours later, Person A acknowledges the answer.

When this communication channel happens between an onshore person and an offshore person, it takes 48 hours instead of 48 minutes. The more this happens, the more expensive it is to outsource. The key to making low-level outsourcing work cost effectively is to minimize the impact of this communication latency, while realizing the benefits of lower salaries in the offshore location.

Making Offshore Development Work

This is why proper communication design is the make-or-break element of making collaborative teams work when using an outsourcing model that involves anyone being offshore.

High-Level Outsourcing

Consider the following software development process diagram (from Four Application Development Outsourcing Models):

In this off-shoring process flow diagram, the shaded areas represent the activities that happen offshore. Note that this is the exact same process flow that is used to describe the other outsourcing models. The difference from other models is primarily where resources are located, but also the relevant scope of responsibility. In comparison with low-level outsourcing, the only difference in the process flow is that responsibility for test design and solution design is transitioned to the offshore development team.

Artificial Boundary

This model creates a bit of an artificial boundary between the “interpret requirement” step and the “design solution” and “design tests” steps. This artificial boundary creates a potentially odd dynamic within the team.

To make reading easier, we’ll talk about the development side of the process flow only, but the same ideas apply on the testing side.

This model has one developer interpreting requirements, and a different developer doing the design work. In an insourcing model, those two developers are usually the same person. In large development teams, a common breakout here is architect versus senior designer. The architect (the figure on top) would be responsible for those design considerations that span the enterprise and the senior designer would be responsible for those design considerations that affect a single application. Here’s a good background article by Scott Ambler on approaching enterprise architecture.

Why have an artificial boundary? Because this model is an emergent design.

1. A company starts with low-level outsourcing.
2. The offshore developers complain about a lack of growth opportunities (both career and skill).
3. Executives are not prepared to completely outsource all technical work (risk aversion), or the team is not ready to succeed with that approach.

Splitting the “interpret requirements” task from the “design a solution” task is a compromise. It minimizes the risk associated with providing growth opportunities to offshore team members. That risk is minimized by avoiding the high-latency communication channel (from onshore to offshore) when communicating requirements. Instead, the interpretation of the requirements is done onshore, and that interpretation is communicated.

You can argue that this model is the offspring of a trust issue within the organization. A company can absolutely say “We want growth opportunities for our offshore team members” and at the same time say “We are not ready to relinquish complete technical control.” This is definitely a trust issue, and therefore a perceived risk mitigation strategy. The risk may be very real, or non-existent. In either case, it is emotionally present.

Vague Scope and Role Conflict

If, while reading this, the notion of splitting interpretation from design feels uncomfortable, that’s because it is. For many teams that are evolving their outsourcing model, it feels uncomfortable, but it feels less uncomfortable than releasing complete technical control.

One problem, which I completely grok as a former developer, is that it is almost impossible to interpret requirements without imagining designs. But this model has different people performing the different tasks. So should the interpreter just discard those designs? Presumably the interpreter is more experienced, certainly more knowledgeable about the domain. It would be a shame to discard those design insights.

Robbing Peter to Pay Paul

This approach has clear benefits for the offshore developers who are now presented with growth opportunities. Unfortunately, you run the risk of sucking the fun out of the role of the interpreter – a senior, experienced developer with domain knowledge. While proper requirements interpretation can be fun, it usually is not fun for a developer. Only requirements people will enjoy those nuances, generally.

You risk eliminating a career path and growth opportunities for your onshore resources with this model.

Division of Labor

One approach that keeps the growth opportunities open for your senior onshore technical resources is to have them play multi-product, or architectural roles. This presents an opportunity for these individuals to apply organizational insights across products, finding synergies between applications and driving consistency and consolidation among applications. You essentially split the responsibilities “broad and deep” where the onshore designer is looking across the portfolio and the offshore designer has ownership responsibilities for a single application or scope. This is similar to the division of responsibilities that works well for tackling enterprise product management.

Instead of preventing communication of requirements across the high-latency channel, it minimizes it. The onshore designer acts as the liaison between multiple offshore designers, fielding interpretation questions, and more effectively, preventing them. Developers have a language all their own. Through a shared perspective on common experiences, they can communicate very efficiently – by analogy, symbolically, and via design patterns. These communication opportunities (between like-minded individuals) can have very high information density. For example, one developer can say “MVC pattern” to another, and that serves more effectively to communicate an approach to designing a solution (and a context / interpretation of requirements) far more efficiently than a product manager describing requirements that multiple tools and platforms should expose the same set of capabilities or behaviors [and that is overly short, because I don't feel like typing a comprehensive explanation of the MVC pattern].

Another approach is to silo the developers vertically – having some applications (or modules) following a low-level outsourcing model, and others practicing complete technical outsourcing. Any given team is operating discretely with a clearly defined set of responsibilities. The teams may just be operating differently. Essentially, you’re saying that your “average” is high level outsourcing, even though it is really a mix of two models. This doesn’t really count, since none of your teams are addressing the communication challenges of this model – your company is leapfrogging over it, but choosing to mitigate risk by doing it a little bit at a time.

You can also take the approach of having the onshore designer be responsible for over-arching and high-complexity design issues, with offshore designers owning more straightforward and lower risk design activities.

The best approach for maximizing your team’s effectiveness (and your HR goals) will be overwhelmingly determined by the individuals involved. If you’re in a large company, you probably don’t get to maximize team effectiveness – you are likely to be forced into a particular model. If you’re doing the forcing, recognize that one size does not fit all.

High Latency Communication And Designing

When circling back to the main challenge of offshoring – communication – you need to look at the nature of the communication that is subject to the high-latency of temporal displacement within the team. The key to making this model work is to leverage the efficiency of cross-talk between developers. That means having experienced people on both the offshore and onshore teams who share common design backgrounds (patterns, analogies, examples) and deep domain understanding (reducing the provision and clarification of context across the communication channel).

Design reviews are also very effective at eliminating the impact of this latency on communication. Design reviews typically happen as follows:

1. Designer A spends time creating design based on an interpretation of requirements.
2. Designers A & B get together and review the design in a relatively short meeting (or Designer B reviews a design document). Feedback is delivered in a burst.
3. Designer A spends time updating the design based on feedback from step 2.
4. Return to step 2 as many times as is needed.

There is a big chunk of work involved in incorporating feedback from a design review. This can be folded into the “white space” between communications. In other words, while designer B is sleeping for the night, designer A is making updates. This looks like, but is not the follow-the-sun pattern.

The follow-the-sun pattern is one where someone is always working. I’ve made this work on a project where there were two teams working 12 hour shifts with a 4 hour overlap (and a 4 hour “blackout”). Note – that isn’t sustainable, just anecdotal. I think the linked article is right, commonly, follow-the-sun implementation does not work, for the reasons cited in that article.

What makes this very different is that we are synchronizing the naturally-occurring time lag between design reviews with the geographically-induced time lag in communication.

Conclusion

The strategy for successfully utilizing offshore resources for both implementation and design starts and ends with communication. It also requires attention to your specific people and their career aspirations.

• Utilize design reviews to take advantage of serendipitous time-lags in the communication cycle.
• Assure that your role definitions are clear, and aligned with the aspirations of the people on your team.
• Follow the tips for effective offshore development – this strategy builds on that one, it does not replace it.

Economic pressures are driving most companies in high-developer-salary markets to explore using offshore development teams as part of their approach to developing software. Developing software with a global team presents new challenges as well as new benefits. If you do it right, you can have a more cost-effective team. If you do it wrong, you can have a disaster.

Different Models for Offshore Development

There are essentially four different models for managing a software development team, with respect to onshore and offshore roles.

1. No offshore development, also known as insourcing.
2. Low-level outsourcing, having the implementation (but not design) team members offshore.
3. High-level outsourcing, having the implementation and design team members offshore.
4. Complete technical outsourcing, having all technical implementation team members offshore.

Each different model has the same set of people involved in the process, with the same channels of communication. The differences are in which communications happen across geographic and temporal boundaries, and which communications happen in the same time zone.

For this article, we are focusing specifically on low-level outsourcing, where the communication channel most affected by different timezones is the one between design and implementation.

Low-Level Outsourcing

Consider the following software development process diagram (from Four Application Development Outsourcing Models):

Each area surrounded by a dashed line represents a different type of work, or a different required dominant skill set. Every stage in the development process requires a different set of dominant skills, and all areas share a set of common skills. When exploring different offshoring models, teams are most effective when they identify the distinctions in dominant skill set requirements, and divide responsibilities along those boundaries. When you create an artificial (or arbitrary) boundary within one of the regions in the diagram, you create opportunities for misunderstanding. With those misunderstandings, you can have people redundantly working on the same thing, or even worse, you can have tasks that don’t get accomplished. You also introduce the possibility of discord within your team as people either proclaim “that’s my job” or “that’s not my job.” You can split a team within the areas shown above (and we’ll talk about that in a future article), but it is harder to do successfully.

Low level outsourcing is an approach where the implementation team members who write the code and tests are offshore. The requirements work is done onshore (where salaries are expensive). Interpretation of the requirements is also done onshore. The design of the testing plan is done onshore, and the design of the solution is also done onshore. The creation of tests and the implementation of the code is done offshore.

In the diagram above, testing of requirements happens on the left, and testing of the implementation happens implicitly on the right. If you haven’t been reading Tyner Blain for a couple years, that may not make much sense. Testing of requirements and implementation are different, and you need to do both.

Testing: Isolation of Variables Reduces Costs

Testing is something that can be approached from a few different perspectives, and the word “testing” means different things to different people. In this process flow, we are focusing on two main areas of testing – testing the requirements and testing the implementation. When you test the requirements, you are asking the question “does this solution do what the product manager intended?” When you test the implementation, you are asking the question “does my solution behave as designer intended?” This is a nuanced difference, and non-technical people may say “what is the difference?” The designer intends exactly the same thing that the product manager intends.

If you think back to your high school science class, you’ll remember the concept of “control of experiments.” This is the field of practical application of logic to scientific experimentation. By taking a logically rigorous approach to designing a science experiment, you can isolate variables, and test each of them independently. This prevents you from drawing false conclusions from your data. The same process leads you to test both the requirements and the implementation. If you’ve ever submitted a bug and had the implementation team close it out with the statement “working as designed”, you already know the benefit of testing both. Just because something is designed to do X does not mean it is not supposed to do Y. By introducing a designer between the product manager and the testing, you introduce the possibility that the designer is the source of the bug – by misinterpreting the requirements.

It is possible to “test” the design and then test the implementation – this will isolate the design from the implementation. Unfortunately, the only way to “test” a design (without also testing the implementation) is conceptually, with a thought experiment. And that’s exactly what the designer does as part of designing. No one else is really going to understand the design well enough to do that. And if the designer is doing the thought-testing, there is no way to isolate if the designer misinterpreted the requirements in the first place. That same misinterpretation will influence his testing in the same way that it influenced his design (See error source E3 in Where Bugs Come From for more details). This is critical to designing, but it does not work for testing a design.

It is possible to test just the requirements. You create tests based solely on the documented requirements. You run those tests against the implemented solution. When the test passes, every step in the process worked. These are known as black-box tests, because you can run the tests without any insight into how the software is written (it is a “black box”). The problem comes when a test fails – you know something is wrong, but you have to do (expensive) research to figure out the source of the problem. It could be that the implementation failed to do what was designed. Or it could be that the software design failed to meet the objectives of the requirements. There is a way to reduce the cost of this analysis – by testing both the requirements and the implementation.

You can easily test the implementation. An implementation test, commonly known as a white-box test, and usually implemented as a unit-test, inspects the effectiveness of a particular implementation at doing what the designer intended. When you combine implementation testing with requirements testing, you isolate the designer variable. If a requirements test fails but the implementation tests pass, the problem is with the design (or with the design of the test). When both requirements and implementation tests fail, you know that at least the implementation is wrong.

In the diagram above, the testing on the left side represents testing of requirements. The right side of the diagram implicitly includes testing of the implementation as part of implementing. You need to ingrain your implementation testing into your development philosophy. Would you deliver code without compiling it? Why, as a developer, would you consider delivering it without testing it? Compilation is not just a build step, it is also an implicit test of compilability. You should also include the implicit test of “does what I intended it to do.”

Combining the discipline of continuous integration with test driven development is the most effective way to accomplish this. Note: remember this part, as it is a critical component to making low-level offshore development work. Without this, you may as well give up – you certainly aren’t going to be more cost-effective.

Communication Across Time And Space

The key to making offshoring effective, as with any development process, is to make the communication work. For communication that happens between people in the same location (or at least roughly the same time zone), the problems and solutions are no different than when you have an insourcing model. What’s different is the communication that happens between members of the onshore team and the offshore team. This communication is not just remote (technology helps us solve these problems with instant messaging, phone calls, and other real-time (or near-real-time) techniques), but also phase-shifted in time. When you have team members working while other team members are sleeping, you slow down the collaborative process. You introduce a near-crippling latency into the communication channel.

Imagine the following expensive question and answer session:

• Person A asks person B a question.
• 12 hours later, person B responds with a request for a clarification.
• 12 hours later, Person A clarifies the question.
• 12 hours later, Person B responds with an answer.
• 12 hours later, Person A acknowledges the answer.

When this communication channel happens between an onshore person and an offshore person, it takes 48 hours instead of 48 minutes. The more this happens, the more expensive it is to outsource. The key to making low-level outsourcing work cost effectively is to minimize the impact of this communication latency, while realizing the benefits of lower salaries in the offshore location.

Communications On Which To Focus

The “happy path” communication channels (shown with blue arrows in the diagram) are the transfer from “test design” to test, and from “implementation design” to implementation. You have to communicate the designs in such a way as to minimize misinterpretation of the design. Never prevent needed communication. Your goal is NOT to stop communication, but to preempt it by eliminating the need. The only thing worse than taking too long because you have a lot of communication is failing to communicate enough. Your goal is NOT to prevent communication, but to minimize the need for it.

The “trust but verify” communication involves making sure that the implementation meets the design. In (requirements) testing, it means reviewing that the tests exhaustively cover everything identified in the test design. It also means reviewing that each test (implementation) actually (effectively) tests what it is designed to test. As team members demonstrate their capabilities, they require less oversight – which is true of any mentoring relationship. In implementation, it means verifying that the code does what the design requires. You could read the code and make a determination, but that is a manual inspection of the code, and manual inspections have been shown to be at best 80% effective as a testing method. What you need to do is create a unit test suite, run it continuously, and only allow developers to check-in their code (to the trunk) when the entire suite passes. Then all you have to do is review the test suite to assure that it will test the design effectively. It wouldn’t hurt to also run the test suite locally (onshore) as a verification, but fundamentally, you are trusting that your developers will follow this continuous integration process.

You are using testing and test design as a mechanism to validate effective communication. You can think of it as a form of active listening. When you (or more appropriately, your design document) says “X”, you can review the test design to confirm that your listener designed tests that assure “X” will happen. Do not just rely on informal communication and acknowledgement. Cross-cultural communication introduces a lot of complexity and misinterpretation. People who do not share a common language or culture also tend to interpret symbols very differently, and will rarely have the same connotations for given interpretable words.

Definition of tests and validation of testing is important beyond the immediate communication of design. There are also the feedback loops that come from the “unhappy path” when something goes wrong (and a bug is introduced). Each of these “where was the bug introduced, and how do we fix it?” cycles is also affected by the latency of cross-shore communication. Good testing reduces the number of these otherwise unwarranted communication cycles.

Developing good design docs is also critical to the success of this communication. The definition of “good” for a design doc is so dependent on the exact circumstances that it is impractical to try and define what that is (at least within this article). A design doc needs to be written with the reader in mind, not the author. Beyond that, we won’t try and make any statements of truth.

Conclusion

The strategy to successful utilization of offshore resources for development and test implementation work starts with communication. The strategy also ends with communication.

• Create artifacts (good design docs) that minimize the clarification cycle across the onshore-offshore time boundary.
• Review implementation tests as an active-listening mechanism to confirm that communication of design intent was effective.
• Practice continuous integration (both as an offshore developer and as an onshore designer or development manager) to assure that your solution stays true to the design and the requirements.

And, as always, have great people – because people trump process.

No matter how good your quality process is, you are introducing bugs. This article reviews the places where bugs are introduced in the software development process (from stakeholders to users), and reviews ways to address those bugs.

One way to identify the sources of bugs is by listening to how people tell you about the bugs.

1. “That is not what I want”
2. “That is not what I said”
3. “That is not what I meant”
4. “That is not how it is supposed to work”
5. “That is not what they meant”
6. “That is not what I want it to do”

Visually, you can think of the development process (regardless of methodology) like the following flow

Each of the six sources of bugs is labeled in the above flow. The article, Where Bugs Come From, goes into a detailed explanation of each source of bugs. This article looks at what you can do to improve the ability of your process to reduce these bugs. I have been a stakeholder, a product manager, a business analyst, a developer and a tester. And I’ve been guilty of all of these offenses at one time or another. When I say you, I also mean me. So chill.

“That is not what I want”

Stakeholders introduce bugs into the process by being unable to define what they want. Some agile methodology proponents argue that as a stakeholder, you can’t know what you want until you have something you don’t want in front of you. In other words, those agilists are giving up on trying to prevent these errors – they claim it is futile.

You can change your mind. The team could be dealing with “That is not what I want now” – and change is something that can not be prevented – although the impact of change can be mitigated. Iterative development and course-correction through getting immediate feedback is one way to do that. Prototyping is another method of course correction. Prototypes can also be used to elicit implicit requirements.

But that still avoids trying to directly address the issue of you not effectively describing what you actually want. There are ways to improve your team’s engagement with you, so that you actually say what you want. First, an emphasis on valuable requirements drives critical thinking about why you need a particular capability or feature. One approach to achieve valuable requirements is to question their justification.

The key to making this work is to make sure you are approving your requirements. And watch out for the anti-patterns in the article.

“That is not what I said”

Business analysts, product managers, and requirements analysts introduce bugs into the process through misunderstanding the stakeholders. Using the techniques above to stay on top of changes in stakeholder needs is not enough. Even when you effectively engage your stakeholders so that they are self-aware and can articulate exactly what they need and why, you still have a problem.

You can improve how you listen, and you can improve how you document what you heard. Make sure that you write complete, consistent and unambiguous requirements. While you’re at it – make sure you are writing your requirements correctly.

“That is not what I meant”

Developers can misinterpret requirements. While there are situations where a requirement is adequately expressed, but the developer is not capable of understanding, that is not the norm. Developers are smart. The onus is on the person writing requirements to make sure that they are written for the audience. So, even though the source of these bugs appears to be the developers, it is actually your documentation. If your documents need to span cultural chasms, then that’s your reality.

Make sure you write your requirements unambiguously, so that they are not misinterpreted. Make sure that they are feasible requirements, and that they do not specify design. The requirements also need to be concise without being overly terse.

“That is not how it is supposed to work”"

Developers should test what they create. Given an understanding of what you believe the code is supposed to do, you should test that it actually does it. This is the stereotypical bug – we asked for X, and it doesn’t work. Continuous integration is the most effective technique for addressing bugs that are introduced at this point in the process.

In more complex situations, you can apply more advanced techniques to assure that your testing is complete.

“That is not what they meant”

The testing team can also misinterpret requirements. This is a source of false positive bugs – even when the developers properly interpret the requirements, a bug may be lodged against their code when there is no bug. That’s why we have the ability to mark issues as “not a bug” when closing them. Apply the same rules for well-written requirements and you will also address this issue. Further, make sure that your requirements can be measurable, or verifiable. We recently wrote more about why you should make requirements testable.

“That is not what I want it to do”

User education is the key here. Users should be encouraged to provide feedback about your product. You don’t want to discourage feedback, you want to minimize the number of false positives. False positives come from users misinterpreting how the application is behaving – a sign of poor design. False positives also come from users not understanding what the application can (or should) do.

You can address design issues by focusing on design with the user in mind. And you can educate the users through a combination of training and with a novel approach to documentation. Document the product in terms of what users are trying to accomplish, in the context of how they are trying to do it.

Conclusion

There are a lot of people involved in the software development lifecycle. As people, we introduce bugs. As informed software professionals, we also know how to address many of them.



]]> http://tynerblain.com/blog/2008/01/14/you-are-creating-bugs/feed/ 8 http://tynerblain.com/blog/2008/01/09/testing-your-requirements/ http://tynerblain.com/blog/2008/01/09/testing-your-requirements/#comments Thu, 10 Jan 2008 03:46:53 +0000 Scott Sehlhorst http://tynerblain.com/blog/2008/01/09/testing-your-requirements/



We’ve written before about several characteristics of well written requirements, and one of those characteristics is testability. Ahamad has written an list of 10 tests of requirements, with an emphasis on assessing the testability of the requirements. The testability of the requirement determines if the resultant product can be tested to determine if it meets the requirement.


Measurable Requirements

Before you can write a test of the software, based on the requirements, you have to make the requirement testable. Why?

Writing requirements without acceptance criteria introduces risk into our project. The last thing we want is to enter into a nitpicking discussion about what we have and have not delivered. We would much rather spend that time discussing what we can deliver next.

Writing Verifiable Requirements

Ahamad points out another benefit of (and tactic for) engaging with stakeholders to define acceptance criteria.

To find the quality measure we ask: “under what circumstances would the system fail to meet this requirement?” The stakeholders review the context of the system and decide that they would consider it a failure if…An attempt to define the quality measure for a requirement helps to rationalise fuzzy requirements.

Requirements Testing, Ahamad

By defining the test, you force the critical thinking required to define good requirements – requirements that are both verifiable. You also explore the justification of the requirement, creating an opportunity to apply the critical thinking needed to map the requirement to the goals that it supports.

Requirements Tests

Ahamad starts his list of requirements tests as a method of inspecting the requirements to assure that they are verifiable. His list goes on to address some other issues commonly faced with writing requirements well. It is a good approach to thinking about requirements – and it is a good idea to think about how you would inspect requirements.

While we don’t think his list represents a comprehensive checklist for assessing the quality of the requirements, it is a great way to think about how you would inspect a requirements document. It will inspire a future article about how to inspect a requirements document.



]]> http://tynerblain.com/blog/2008/01/09/testing-your-requirements/feed/ 0 http://tynerblain.com/blog/2008/01/02/test-smarter-not-harder-3/ http://tynerblain.com/blog/2008/01/02/test-smarter-not-harder-3/#comments Thu, 03 Jan 2008 04:00:06 +0000 Scott Sehlhorst http://tynerblain.com/blog/2008/01/02/test-smarter-not-harder-3/



This series is a reprint of an article by Scott Sehlhorst, written for developer.* in March 2006. A recent article on dailytech about “new” methods for software testing points to some very interesting research by the NIST (the National Institute of Standards and Technology) Information Technology Lab. We’ve split the original article into three articles to be more consistent in length with other Tyner Blain articles.

This is part 3 of a 3 part series.

Article Overview

• Part 1 of this article explores the problem of getting good testing coverage of complex software.
• Part 2 of this article discusses solution approaches (including those identified in the NIST research).
• Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

How to make it even better

When we don’t know anything, or don’t apply any knowledge about our application to our testing strategy, we end up with far too many tests. By applying knowledge of the application to our test design we can greatly reduce the size of our test suite. Tests that incorporate knowledge of the application being tested are known as whitebox tests.

Map out the control dependencies

In our previous examples, we applied no knowledge of the interactions of controls (or the interactions within the program of having made selections in the controls. If we consider a visual map of the controls and their possible relationships, it would look like the following diagram.



There is a possibly-relevant connection between the selections in every pair of controls. We have designed our testing around the lack of knowledge that is clearly visible in the diagram.

It is likely that we can rule out some of the dependencies, but possibly not all of them. Our approach should be conservative – only remove those dependencies that we know don’t exist. This knowledge comes from an understanding of the underlying application. Once we remove these links the diagram will look like this:



This clarified mapping allows us to reduce the size of our test suite dramatically, because we’ve identified the independence of many controls. In an ideal case, the result will be two or more completely disconnected graphs, and we can build a set of tests for our suite around each separate graph. As the diagram above shows, we do not have two completely independent graphs. We can take a testing approach as shown in the following diagram:



We’ve grouped all of the controls on the left in a blue box – these controls will be used with the N-wise generation tool to create a set of tests. The grouping of controls on the right will also be used to generate a set of tests.

In this example, we reduce the number of tests required by a significant amount when order matters.



Also note that we increase the number of tests required when order doesn’t matter, if we have any overlapping controls (if the graphs can’t be separated). When the graphs can be separated, this reduces the amount of testing even if order is irrelevant.

The key to separating the graphs is to make sure that all controls only connect to other controls within their region (including the overlapping region).

Eliminate equivalent values from the inputs

When we know how the code is implemented, or have insights into the requirements, we can further reduce the scope of testing by eliminating equivalent values.

Consider the following example requirements for an application:



And two variables that we are evaluating in our testing – imagine that they are controls in a user interface (or values imported from an external system).



Which we can collapse for testing purposes into:



This consolidation of equivalent values reduces the number of tests we need to run. For our simple pairwise test, we reduce the number from 18 to 12. When there are more controls involved, and when we are doing N-wise testing with N=3, the impact is much more significant.

Conclusion

We can test very complex software without doing exhaustive testing.

Random sampling is a common technique, but falls short of high quality goals – very good quality requires very high quantities of tests.

Pairwise testing allows us to test very complex software with a small number of tests, and reasonable (on the order of 90%) code coverage. This also falls short of high-quality goals, but is very effective for lower expectations.

N-wise testing with N=3 provides high quality capable test suites, but at the expense of larger suites. When the order of inputs into the software matters, N-wise approaches become limited in the number of variables they can support (fewer than 10), due to limitations of test-generation tools available today.

We can apply knowledge of the underlying software and requirements to improve our testing strategy. None of the previous techniques require knowledge of the application, and thus rely on brute force to assure coverage. This approach results in conceptually redundant tests in the suite. By mapping out the grid of interdependency between inputs and subdividing the testing into multiple areas we reduce the number of tests in our suite. By removing redundant or equivalent values from the test suite we also reduce the number of tests required to achieve high quality.

Article Overview

• Part 1 of this article explores the problem of getting good testing coverage of complex software.
• Part 2 of this article discusses solution approaches (including those identified in the NIST research).
• Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

One of the interesting points, not addressed by this article, and not



]]> http://tynerblain.com/blog/2008/01/02/test-smarter-not-harder-3/feed/ 0 http://tynerblain.com/blog/2007/12/27/test-smarter-not-harder-2/ http://tynerblain.com/blog/2007/12/27/test-smarter-not-harder-2/#comments Fri, 28 Dec 2007 04:00:03 +0000 Scott Sehlhorst http://tynerblain.com/blog/2007/12/27/test-smarter-not-harder-2/



This series is a reprint of an article by Scott Sehlhorst, written for developer.* in March 2006. A recent article on dailytech about “new” methods for software testing points to some very interesting research by the NIST (the National Institute of Standards and Technology) Information Technology Lab. We’ve split the original article into three articles to be more consistent in length with other Tyner Blain articles.

This is part 2 of a 3 part series.


Article Overview

• Part 1 of this article explores the problem of getting good testing coverage of complex software.
• Part 2 of this article discusses solution approaches (including those identified in the NIST research).
• Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

Solving the problem

There are a lot of applications that have millions or billons of combinations of inputs. They have automated testing. They have solutions to this problem. We just finished discussing how impractical it is to test exhaustively, so how do companies test their complex software?

Random sampling

Early on in the software testing world, someone realized that by randomly checking different combinations of inputs, they would eventually find the bugs. Imagine software that has one million possible combinations of inputs (half as complex as our previous example). Each random sample would give us 0.000001% coverage of all possible user sessions. If we run 1000 tests, we would still only have 0.001% coverage of the application.

Thankfully, statistics can help us make statements about our quality levels. But we can’t use “coverage” as our key measurement of quality. We have to think about things a little bit differently. What we want to do is express a level of confidence about a level of quality. We need to determine the sample size, or number of tests, that we need to run to make a statistical statement about the quality of the application.

First we define a quality goal: we want to assure that our software is 99% bug free. That means that up to 1% of the user sessions would exhibit a bug. To be 100% confident that this statement is true, we would need to test at least 99% of the possible user sessions, or over 990,000 tests.

By adding a level of confidence to our analysis, we can use sampling (selecting a subset of the whole, and extrapolating those results as being characteristic of the whole) to describe the quality of our software. We will leverage the mathematical work that has been developed to determine how to run polls.

We define our goal to be that we have 99% confidence that the software is 99% bug free. The 99% level of confidence means that if we ran our sample repeatedly, 99% of the time, the results would be within the margin of error. Since our goal is 99% bug free code, we will test for 100% passing of tests, with a 1% margin of error.

How many samples do we need, if there are one million combinations, to identify the level of quality with a 99% confidence, and a 1% margin of error? The math for this is readily available, and calculators for determining sample size are online and free. Using this polling approach, we find that the number of samples we require to determine the quality level with a 1% error and 99% confidence is 16,369.

If we test 16,369 user sessions and find 100% success, we have established a 99% confidence that our quality is at least at a 99% level. We only have 99% quality, because we have found 100% quality in our tests, with a 1% margin of error.

This approach scales for very large numbers of combinations. Consider the following table, where our goal is to establish 99% confidence in a 99% quality level. Each row in the following table represents an increasingly complex software application. Complexity is defined as the number of unique combinations of possible inputs).



We can see that the very few additional tests have to be run to achieve the same level of quality for increasingly complex software. When we have a modest quality goal, such as 99/99 (99% confidence in 99% quality), this approach is very effective.

Where this approach doesn’t scale well is with increasing levels of quality. Consider the quest for “five nines” (99.999% bug free code). With each increase in the desired level of quality, the number of tests we have to run grows. It quickly becomes an almost exhaustive test suite.

Each row in the following table represents an increasingly stringent quality requirement, with the complexity of the software staying constant at one million possible input combinations.



The random sampling approach does not provide a benefit over exhaustive testing when our quality goals are high.

Pairwise testing of input variables

Studies have shown that bugs in software tend to be the results of the combination of variables, not individual variables.

Consider a very simple laptop configuration page, having three selectable controls: CPU, Memory, and Storage. Each control has three possible values as shown in the table below.



We successfully pass tests of each of the different values available in the CPU control. However, we discover that the software test fails if the user selects a CPU value of “Consumer” and selects a Storage value of “Huge.” This highlights an unknown dependency between the CPI and Storage controls.

Pair-wise testing is designed to get coverage of every possible combination of two variables, without testing every possible combination of all the variables. For this example, there are 27 unique combinations of all of the selections. The following table shows the first 9 combinations. An additional 9 combinations are needed for each of the other CPU selections.



Exhaustive pair-wise testing will make sure that every unique combination of any two variables will be covered. The next table shows the combinations for this example.



With just 9 tests, we are able to exhaustively cover every unique pair of CPU and Memory, CPU and Storage, and Memory and Storage. Pair-wise testing allows us to get full coverage of the combinations of every two variables, with a minimal number of tests.

Pair-wise testing not only gives us full coverage of every pair of values, it also gives us (redundant) coverage of every single value for each control.

If we look back at our previous examples of laptop-configuration, we can calculate the numbers of tests required to get full pair-wise coverage. For the entry level laptop configurator, there are 32,256 possible unique combinations of inputs. We can test every unique combination of two variables with 31 tests. For the high-end laptop configurator, there are 2,322,432 unique combinations of inputs. We can test every unique combination of two variables with 36 tests.

N-wise testing

The concept of pair-wise testing can be extended to N-wise testing – looking at every combination of N possible inputs.

The following table shows how many tests are required to get full coverage of each N-wise combination of inputs for both the low-end and high-end laptops configurators.



This is a much more manageable situation. Using N=3, we get 179 tests versus 2.3 million tests for exhaustive coverage! Existing studies have consistently shown that N=3 creates on the order of 90% code coverage with test suites, although the number will vary from application to application. We will use N=3, based on practical experience that N=4 tests rarely uncover bugs that were missed with N=3.

This approach only works when users are forced to enter values in a proscribed sequence, and in cases where the sequence of entry is irrelevant. This set of tests won’t give us representative coverage of what the users will do when they are allowed to make selections in arbitrary but relevant order.

Order relevance and statistical testing

There’s been an assumption implicit in all of our calculations so far – that the order of selection in the controls is irrelevant. The available N-wise test calculation tools do not incorporate order of selection in their permutations – explicitly, they assume a fixed order of operations. When we test an API we have control over the order of processing – there are a fixed number of arguments, in a fixed order. N-wise testing People, however, do not always interact with the controls in a fixed order. And web service architectures may not be able to depend upon a predetermined sequence of events.

With 5 controls in an interface, we have 5! (factorial – see definition) or 120 possible sequences in which selections can be made by a person. Although the user interface may incorporate dynamic filtering that prevents some subsets of out-of-sequence selection, N-wise testing is blackbox testing, and will not have access to that information.

For an interface with M possible controls, each script created by an N-wise test generator will have to be tested in M! sequences to get exhaustive coverage. If the controls are split across multiple screens, then we can reduce the number of sequences. For example, if there are 5 controls on the first screen, and five controls on the next screen, instead of considering 10! (3.6 million sequences), we can consider all first-screen-sequences in combination with all second-screen sequences (5! * 5! = 120 * 120 = 14,400 script sequences).

In our example laptop configurators, there are 11 and 13 controls (all on the same page) for the low-end and high-end laptops respectively. This would imply 11! and 13! possible sequences (40 million and 6 billion).

We do not need to do exhaustive coverage of the sequencing permutations. An N-wise test is specifically analyzing the interdependence of any combination of N controls. As a lower-bound, we would only need N! sequences for each generated script. So our 179-script suite for the high-end laptop (with N=3) would need 3! (6) * 179 = 1,074 scripts to cover the product.

Here’s the table for the lower-bound of scripts required to account for different values of N for both laptop-configurators.



This is a lower bound, because it assumes a perfect efficiency in combining unique sequences of each group of N controls. Existing N-wise testing tools do not (to the author’s knowledge) take order of operations into account. For N=2, this is trivial – just duplicate the set of tests, in the exact reverse order.

We can take order of operations into account by treating the sequence as an additional input. We use the mathematical formula “X choose Y” which tells us the number of different combinations of Y values from a set of X values. The formula for calculating “X choose Y” is X!/(Y!*(X-Y)!) where X is the number of inputs and Y is the dimension of the desired N-wise test.

Here’s the table of the number of combinations for each N, for both the low-end and high-end laptop configuration screens we’ve been discussing.



Here are the values, generally, for varying numbers of inputs.

(click for larger version)

We would then calculate the N-wise testing using a value of N+1 as an input to the test-generation tool, and include the number of unique sequences as if it were a control input.

Unfortunately, we don’t have a solver capable of handling single dimensions larger than 52. This limits our ability to create a test suite for N=3 to a maximum of 7 controls.

To show the impact of sequencing on the test suite, consider an interface with 7 controls, each having 5 possible values. N=3 would require 236 tests if order is irrelevant. We then include sequence of selection as a parameter (by adding an 8^th control with 35 possible values, and testing for N=4), In this case, N=3 (with sequencing) requires 8,442 scripts. Our theoretical lower bound would be 236 * 35 = 8260.

Article Overview

• Part 1 of this article explores the problem of getting good testing coverage of complex software.
• Part 2 of this article discusses solution approaches (including those identified in the NIST research).
• Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.



]]> http://tynerblain.com/blog/2007/12/27/test-smarter-not-harder-2/feed/ 0 http://tynerblain.com/blog/2007/12/25/test-smarter-not-harder-1/ http://tynerblain.com/blog/2007/12/25/test-smarter-not-harder-1/#comments Wed, 26 Dec 2007 04:00:05 +0000 Scott Sehlhorst http://tynerblain.com/blog/2007/12/25/test-smarter-not-harder-1/



This series is a reprint of an article by Scott Sehlhorst, written for developer.* in March 2006. A recent article on dailytech about “new” methods for software testing points to some very interesting research by the NIST (the National Institute of Standards and Technology) Information Technology Lab. We’ve split the original article into three articles to be more consistent in length with other Tyner Blain articles.

This is part 1 of a 3 part series.

Article Overview

• Part 1 of this article explores the problem of getting good testing coverage of complex software.
• Part 2 of this article discusses solution approaches (including those identified in the NIST research).
• Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.

Introduction

In this article we focus on a method to minimize the cost of quality. Regardless of the method you use for calculating the cost of poor quality, minimizing the cost of assuring good quality is relevant to you. We will discuss techniques for minimizing the cost of creating and maintaining an automated regression testing suite for a software application.

Automated testing changed the software development process as much as the assembly line changed the manufacturing industry. We are assuming in this article that we already have automated testing as a part of our development process.

We open with a discussion of the problem. We will then discuss approaches to automated testing that are progressively better at minimizing costs. We hope you enjoy the article and would love to hear back from you with feedback and contributions.

The problems

Here is the sound bite version of the problems this article is designed to address

• We aren’t meeting our quality goals.
• We have a development team, regularly releasing new versions of our software.
• We have a suite of automated tests we run on our software.
• Our automated test suite may be large, but out product is complex. There’s no way to test every possible scenario. Bugs are still getting released to the field.

Not solving the problems

Two solutions that we have to consider are to test nothing, and to test everything. We would consider testing nothing if we can’t afford to test the software. When people don’t appreciate the complexities of testing or the limitations of automated testing, they are inclined to want to test everything. Testing everything is much easier to said than done.

We can’t afford to test it

I was able to attend a training session with Kent Beck a few years ago. I was also honored to be able to enjoy a large steak and some cold beer with him that night after the training. When asked how he responds to people who complain about the cost of quality, Kent told us he has a very simple answer, “If testing costs more than not testing then don’t do it.” I agree.

There are few situations where the cost of quality exceeded the cost of poor quality. These are situations where the needed infrastructure, test-development time, and maintenance costs outweighed the expected cost of having a bug. The expected cost is the likelihood (as a percentage) of the bug manifesting in the field, multiplied by the cost of dealing with the bug.

The techniques described in this article are designed to reduce the cost of quality, to make it even less likely that not testing is the best answer.

Just test everything – it’s automated

We’ve all been on at least one project or team where our manager has said

“I demand full testing coverage of the software. Our policy is zero tolerance. We won’t have bad quality on my watch.”

What we struggle with here is the lack of appreciation for what it means to have full coverage or any other guarantee of a particular defect rate.

There are no absolutes in a sufficiently complex system – but that’s ok. There are statistics, confidence levels, and risk-management plans. As engineers and software developers, our brains are wired to deal with the expected, likely, and probable futures. We have to help our less-technical brethren understand these concepts – or at least put them in perspective.

We may get asked “Why can’t we just test every combination of inputs to make sure we get the right outputs? We have an automated test suite – just fill it up and run it!” And we need to resist the urge to respond by saying “Monkeys with typewriters will have completed the works of Shakespeare before we finish a single run of our test suite!”

Complexity leads to futility

Consider a web page for customizing a laptop purchase. Take a look at Dell’s customize it page for an entry level laptop, if you’ve never configured a laptop online before.

The web page presents eleven questions to the user that have from two to seven responses each. Specifically, each decision the user has to make presents (2,2,2,2,2,3,2,2,3,4,7) choices. This is a simple configuration problem. The number of possible laptop configurations that could be requested by the user is the product of all of the choices. In this very simple page, there are 32,256 possibilities. The page for customizing Dell’s high end laptop at the time of this writing has a not-dissimilar set of controls with more choices in each control – (3,3,3,2,4,2,4,2,2,3,7,4,4). The user of this page can request any of 2,322,432 different laptop configurations! If Dell were to add one more control presenting five different choices, there would be over ten million possible combinations!

Creating a test suite that tries all two million combinations for their high end laptop could be automated, but even if every test took one tenth of second to run, the suite would take over 64 hours to run! And Dell changes their product offering in less time than that.

If we use a server farm to distribute the test suite across ten machines we could run it in about 6 hours. Ignoring the fact that we would be running this type of test for each customization page Dell has, 6 hours is not unreasonable.

Validating the two million results is where the really big problem is waiting for us. We can’t rely on people to manually validate all of the outputs – it is just too expensive. We could write another program, which inspects those outputs and evaluates them using a rules-based system (“If the user selects 1GB of RAM, then the configuration must include 1GB of RAM” and “The price for the final system must be adjusted by the price-impact of 1GB of RAM relative to the base system price for this model.”)

There are some good rules-based validation tools out there, but they are either custom software, or so general as to require a large investment to make them applicable to a particular customer. With a rules-based inspection system, we have the cost of maintaining the rules.

The validation rules are going to have to be updated regularly, as Dell changes the way they position, configure, and price their laptops regularly.

Since we aren’t Dell, we don’t have the scale (tens of billions of dollars of revenue) to justify this level of investment. The bottom line for us is that we can’t afford to exhaustively test every combination.

Article Overview

• Part 1 of this article explores the problem of getting good testing coverage of complex software.
• Part 2 of this article discusses solution approaches (including those identified in the NIST research).
• Part 3 of this article explores an approach to improving on the “best” solution identified in part 2.



]]> http://tynerblain.com/blog/2007/12/25/test-smarter-not-harder-1/feed/ 0 http://tynerblain.com/blog/2007/05/02/apr-prototype-update-2/ http://tynerblain.com/blog/2007/05/02/apr-prototype-update-2/#comments Thu, 03 May 2007 01:53:24 +0000 Scott Sehlhorst http://tynerblain.com/blog/2007/05/02/apr-prototype-update-2/



Just a quick update on the prototype for our agile project. We’ve dramatically improved test coverage, implemented authentication-restriction for some functionality, and refactored a litte of the code. Read on for the latest stats on coverage and testing.

Code and Test Metrics

While code-coverage metrics might be more useful, we don’t know how to calculate them yet. Here’s the same data we used in our previous update.

+----------------------+-------+-------+---------+---------+-----+-------+
| Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Controllers | 140 | 86 | 3 | 16 | 5 | 3 |
| Helpers | 7 | 6 | 0 | 0 | 0 | 0 |
| Models | 77 | 56 | 2 | 9 | 4 | 4 |
| Libraries | 233 | 145 | 3 | 30 | 10 | 2 |
| Components | 0 | 0 | 0 | 0 | 0 | 0 |
| Model specs | 169 | 134 | 0 | 0 | 0 | 0 |
| View specs | 86 | 62 | 0 | 0 | 0 | 0 |
| Controller specs | 74 | 54 | 0 | 0 | 0 | 0 |
| Helper specs | 86 | 62 | 0 | 0 | 0 | 0 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total | 872 | 605 | 8 | 55 | 6 | 9 |
+----------------------+-------+-------+---------+---------+-----+-------+
Code LOC: 293 Test LOC: 312 Code to Test Ratio: 1:1.1

Two interesting things to note:

• Our test-to-code ratio is now greater than 1. It should be even higher.
• We now have independent testing of the model, controllers, and views. What this means is that we can isolate specific behaviors and test them independently of other parts of the code.

More RSpec Tests

Here is the list of tests that we have written and that have been implemented.


Testing the controllers
When anonymous users interact with articles
- listing the articles should be successful
- showing an article should be successful
- editing an article should redirect to login
- creating a new article should redirect to login

When authenticated users interact with articles
- listing the articles should be successful
- showing an article should be successful
- editing an article should be successful
- creating an article should be successful

Testing the User Model

A user with articles
- must be able to find all her articles TBD
Multiple users
- must not have the same login ID
- must not have the same email address

A user, in general,
- must have a login ID
- must have an email address
- must have an email address with a valid format
- must have a password
- must have a long password

Testing the Article Model
An article
- must have a title
- must have a URL
- must not duplicate an existing article TBD
- must be associated with the user that created it TBD

Testing the Rendering (presentation) of Articles

/article/list
- should include the list of articles
- should include the individual article titles

Refactoring

One of the mantras of Rails developers is “DRY” – Don’t Repeat Yourself. And it applies to not writing the same code over and over again. With testing of the rendering (the user interface) of articles in place, I was able to refactor the implementation code to get improved re-use. The process went as follows:

1. Write code the first time.
2. Write tests, confirm their accuracy.
3. Rewrite the rendering code with re-use in mind (because we will list articles in many different ways – most recent, highest rated, etc).
4. Notice that the tests are failing.
5. Fix the new code until the tests pass.

Step 6 was a quick visual inspection / sanity-check that the UI was behaving as the tests assured – and it was.

Summary

At this stage, the prototype can

• Create/Edit users
• Create, Edit, Update, “Delete” articles
• Enforce the need to login to create an article

The next things to work on (where work = code + test) are

• Associate the article with the user who created it
• Create/Edit ratings (of articles, by users)
• Associate the rating with the user who created it
• Associate the rating with the article to which it applies
• Calculate the article “score” based on ratings
• Add a couple presentations (latest articles, highest scoring articles)
• Add a little styling/layout effort
• Deploy the prototype.

Still planning on getting a prototype version deployed before our second anniversary as a company (Cinco de Mayo). Most of today was spent learning how to test the different elements of the code with RSpec. Tomorrow will have more implementation accomplishments.
Anything missing?



]]> http://tynerblain.com/blog/2007/05/02/apr-prototype-update-2/feed/ 5 http://tynerblain.com/blog/2007/04/30/apr-infrastructure/ http://tynerblain.com/blog/2007/04/30/apr-infrastructure/#comments Tue, 01 May 2007 02:38:27 +0000 Scott Sehlhorst http://tynerblain.com/blog/2007/04/30/apr-infrastructure/



This morning I finished up the infrastructure setup for our project. A bunch of under the hood work. Definitely required some propeller-head skills. The goal of this work is to get us to a working prototype as soon as possible. There are a couple links to some good agile testing articles, and the rest of this one is a quick list of what we did on Friday and today.

Pre-Project Setup Stuff

Most of what we have to do to kick off this project really doesn’t fall under the project time line (like setting up source control), because it is a one-time setup. But we still had to do it, so it is still here. And it could easily apply to your dev team if you aren’t using subversion, or rails, etc.

Here’s what we have working now:

• Subversion (svn) source code repository set up on our server
• RSA-encrypted bash shell access for simpler scc operations
• svn and TortoiseSVN set up for local development, project started, some simple code checked in. Dev-process operational
• Aptana/RadRails IDE set up and working (not integrated to svn, but workable)
• Apache, MySQL, Mongrel, Ruby, Rails, RSpec set up for local dev

Most of it is standard developer stuff, but RSpec is pretty interesting as an approach for supporting test driven development. RSpec proponents describe it as behavior driven development (BDD), but coming at this from a product management perspective, it is a form of requirements verification that differs semantically from stereotypical developer unit testing. Will share some examples later, when there’s something working. For now rest assured that test development is concurrent with (or preceding) code development, as part of our continuous integration approach – another key component of our agile project.

To set expectations, I expect to have at least one working version of the first release prototyped and available this week for feedback from a select group of alpha-users. The only barriers are learning ruby and rails and rspec, learning capistrano (rails deployment tech), resolving any unanticipated issues with the server setup. Writing the code and tests is the easy part.



]]> http://tynerblain.com/blog/2007/04/30/apr-infrastructure/feed/ 2 http://tynerblain.com/blog/2007/04/16/is-agile-bad-for-software/ http://tynerblain.com/blog/2007/04/16/is-agile-bad-for-software/#comments Tue, 17 Apr 2007 03:00:26 +0000 Scott Sehlhorst http://tynerblain.com/blog/2007/04/16/is-agile-bad-for-software/



Last week, Ivan Chalif, a product manager / blogger, tapped into a thread criticising product managers for not adopting and espousing agile, or at least rapid-release techniques. In this article we look at Ivan’s comments and one of the articles that he referenced. We also share our own perspective and an alternative analysis of what may have happened.

Short Release Cycles are Bad

At the end of his article, Ivan references a well presented position on why short release cycles are bad that was published by Justin James on TechRepublic last June.

Ivan’s article basically says that agile approaches are fine for non-mission critical software. Justin’s article includes criticism of software he has used, and implies that the problems are due to the development methodologies used.

In a nutshell, they are both saying that short release cycles are bad because they result in bad software being released. Ivan qualifies his perspective by saying that for some applications, like “consumer web applications” that’s ok – because being bad doesn’t hurt as much.

Our perspective is a little bit different – short release cycles are the best thing ever. Immature or incomplete releases are bad. We define incomplete as meaning “not all of the must be requirements have been satisfied.” We define immature as meaning “quality, usability, and performance are not where they need to be for the audience.”

Ignorance is No Excuse

On the point of requirements, we’ll add that failure to implement an undiscovered / undocumented must-be requirement still qualifies as incompleteness. A product manager’s failure to document a critical requirement may absolve the implementation team from addressing and testing it – but the “entire team” is still on the hook for it.

In short, we don’t think agile approaches need to be forced to live in the “software that isn’t important to users” box. Agile techniques and approaches can be used on any project. Any project can succeed or fail, and any process can succeed or fail on a particular project. What is more important is the team. And unfortunately, the environment – people trump process, but politics trumps people. If your organization can not or will not support an iterative development cycle, your project will fail if you use an agile process.

Mission critical and non-mission critical software have mandatory requirements that must be satisfied. Must-be requirements, by definition, must be satisfied. A build of the software that does not satisfy those requirements is one that is incomplete, and should not be released for production use.

Note that we specified “production use.”

Anti-Agile Argument

We’ll take a point-counterpoint approach to one of the arguments raised by Justin. Our position is that his criticism, while anecdotally valid, should not be a criticism of the process, but of its poor execution.

Here is an example of what I mean: today, I made hotel reservations with a Hyatt hotel on their Web site.[...]Except there is one huge flaw: it does not use HTTPS! So not only is my credit card number being sent in plain text (along with my first and last name), but it is going to be stored in my browser’s auto-fill system in plain text!

Justin James

Is that a problem? Absolutely. Is secure transmission of user financial data a must-be requirement? Unquestionably. I explained to my mom that her credit card number is safer on the Amazon form than it is on the receipt at a restaurant. The whole ecommerce world is dependant on that assumption. The team that implemented the website Justin was using screwed up big time. But how did they screw up?

The team that built this Hyatt website screwed up in one of four ways:

1. They did not identify the security requirement.
2. They did not write code to satisfy the security requirement.
3. The code they wrote did not satisfy the security requirement.
4. They wrote code that broke a previously working solution to the security requirement.

This clearly can be blamed on the “entire team” but can not be clearly blamed on agile software development.

Is Agile To Blame?

Let’s look at each of the possible explanations from above in more detail…

They did not identify the security requirement

The first possibility – failure to identify the requirement, is a risk with every development process. Identification of the critical requirements is a matter of product manager competence (or product champion, or whoever on the team is gathering the requirements). More rigorous requirements gathering will identify more requirements. You don’t neccessarily have to have a big up-front requirements (BUFR) exercise before writing the first line of code. You do have to have “enough” requirements gathering. A BUFR process does not assure a complete set of requirements any more than an incremental process prevents it.

All teams will reach a point when they decide they have “enough” to risk developing against an incomplete spec. A product manager has to develop a sufficient breadth of understanding to make an informed decision about when “enough” is enough. That’s why we encourage defining the space with use case names and brief descriptions of use cases. Understanding the domain is the goal of this exercise, and it is not the same as defining all of the requirements up front.

This team failed to get an understanding of the domain, and overlooked something as obvious as the need for customers to have secure transmission of their credit card numbers. That is a failing of the product manager, not an agile approach. Note: we dismiss any process that says “don’t develop an understanding of the domain” as a bad one, agile or otherwise.

They did not write code to satisfy the security requirement.

Was the requirement mis-prioritized as being less than critically important? If so, blame the product manager again.

What if the requirement was defined, identified as a must-be requirement, and still wasn’t implemented? Who’s fault is it then? Did the team have poor execution, or did they follow a process that says “define the requirements, but don’t address them if you don’t feel like it?”

I’m not aware of an agile process that says you should release the software to production without implementing a solution to a known critical requirement. Definitely can’t blame this on agile. You could potentially blame it on an executive or project manager who said “release on date X, I don’t care if it’s done!” In which case, Justin just had bad timing, because any agile process would put this at the top of the list for the next release. If not, it isn’t agile.

The code they wrote did not satisfy the security requirement.

OK, now we’re looking at a team that 1) identified the requirement, and 2) attempted to implement it in their release, but 3) failed to test it (or released with a known but uncommunicated bug). I say uncommunicated because Justin didn’t get the message, even if someone on the team tried to send it.

Lack of testing and failure to communicate are not characteristics of agile processes. Just because you’re delivering code rapidly, managing the amount of work to be delivered in a timebox, does not mean that quality is ignored. That is certainly a choice that any foolish team (or irresponsible team) can make. But it is not unique to agile development, or even characteristic of agile development.

The agile projects that we’ve seen tend to have better quality than waterfall projects. Not because they are agile, but because the developers who are following an agile process have tended to (this is anecdotal data) have a greater appreciation for the benefits of continuous integration and quality in general. They live by the test cases. If it isn’t tested, it hasn’t been completed. And if it fails the tests, it hasn’t been completed.

There is a greater possibility that a waterfall project will allow developers to write code for months on end without creating tests – with the expectation that it will get tested in the end. We’ve worked with a client who’s timeline for a release of a key set of features included 3 months for development and one week for testing. When we asked “Where’s the time allocated to fix the code when you discover that it is broken?” we got the strangest looks. As an aside, in the year after we helped that team implement a continuous integration process, they’ve reduced their outstanding bug list by almost 80% and the team lead expects it to be cut in half again in the next month. They’re making great progress just by adding one agile technique (continuous integration). The team has to create, review, and pass tests before they check code into the trunk for the current release cycle.

Absolutely not a failing of agile software development – this would be a failing of the team to deliver quality product.

They wrote code that broke a previously working solution to the security requirement.

This is another variation of the previous situation. 1) They identified, scheduled, implemented and tested the functionality. 2) The later broke it, but didn’t retest it. If they discovered that they broke it, and didn’t fix it, that falls in the previous bucket. More likely, they didn’t discover that they broke it.

Regression testing is an important element of continuous integration, and from a cost perspective, the primary reason to automate testing as much as possible. Technically, as much as practical – if it is cheaper to test manually, test manually. As the code base builds, so does the test suite. And a simple addition to the check-in policy is all that is needed.

Without regression testing: If it doesn’t work, don’t check it in (and therefore don’t release it).
With regression testing: Also, if it breaks something else, don’t check it in without fixing the other thing (and therefore don’t release it).

The team we worked with before was working on a mammoth code base with over 100 developer-years and hundreds of thousands of lines of code. They had such a huge backlog of outstanding bugs because 1) there were another 5 to 10 developer-years of new functionality being added to the code every year, and 2) they had no practical way to discover or prevent regression bugs. The developers rarely checked in code that didn’t work, but regularly introduced errors or unearthed latent errors in the existing code. And effort spent fixing those errors (after they were discovered) introduced more errors. It was a vicious and never-ending cycle. [Ed: Here's the case study we published on integrating testing automation.]

When they adopted the continuous integration philosophy, they included regression testing as part of their required process for release – no existing tests could be broken. And they’ve been able to start dedicating time to addressing old bugs – some of them years old with confidence. They’ve also been able to refactor the existing code so that it reduced the cost of ongoing work, allowing them to accelerate the improvement of their quality levels. It is a great story.

Failure to regression test is not a characteristic of an agile process.

Conclusion

There’s no disputing Justin’s bad experience. We believe that a poorly delivered product is a result of poor team execution, not poor process decisions.



]]> http://tynerblain.com/blog/2007/04/16/is-agile-bad-for-software/feed/ 27 http://tynerblain.com/blog/2007/04/12/use-case-vs-test-case/ http://tynerblain.com/blog/2007/04/12/use-case-vs-test-case/#comments Fri, 13 Apr 2007 03:00:05 +0000 Scott Sehlhorst http://tynerblain.com/blog/2007/04/12/use-case-vs-test-case/



People who are new to software, requirements, or testing often ask “What’s the difference between a use case and a test case?” This article answers that question, by building on earlier articles about use cases and use case scenarios. At the soundbite level, each use case has one or more scenarios, and each use case scenario would lead to the creation of one or more test cases.

Definition of a Use Case



Use cases tell the story of how someone interacts with a software system to achieve a goal. A good use case will describe the interactions that lead to either achieving or abandoning the goal. The use case will describe multiple paths that the user can follow within the use case.

Definition of a Use Case Scenario



A use case is made up of one or more use case scenarios. Each path that can be followed within the use case is a use case scenario. Any given example of following a use case also follows a single scenario. Multiple executions of the use case can use the same or different scenarios.

Definition of a Test Case

There are many different kinds of testing, and they can be described in different ways. They can be done by different people to achieve different objectives. They can be manual or automated. Testing is a very large field, and this article is not trying to define all of the ways that people can test software.

When someone asks the question “What’s the difference between a use case and a test case?” they are generally referring to system tests, “end to end” tests, or blackbox tests. They are probably not thinking about unit tests or integration tests.

Check out this explanation of the difference between unit tests and system tests. Or this article for an introduction to Continuous Integration – an approach to test automation and software development, or this article on the essential practices of continuous integration.
System Test Cases

Many system tests are designed to simulate how a user interacts with the system, to make sure that the system responds appropriately. If you’ve defined your requirements by using goal driven use cases, you can use the use cases as a framework for defining these test cases.

These system tests should be created to test a single situation. When using the approach of use cases and use case scenarios to describe requirements, a system test should test a single use case scenario.

In an example use case we recently wrote, one alternate flow “3A1″ involves the user entering shipping and billing information that is unique to the order they are placing. You would define at least one use case scenario that involves these steps. Assume that the scenario you have defined follows alternate flow “3A1″ but otherwise follows the normal flow.



You could create two system tests that are designed to validate that the requirements of the use case are met. The first system test would involve a user that has an account but elects to use different information for this order. This is a realistic scenario – perhaps someone is ordering a gift to be delivered to their cousin who lives at a different address.

You could also define a system test that involves a user without a pre-existing account.

Both test cases follow the use case, and both test cases follow the use case scenario. But the test cases test different things (from a business / requirements perspective). They may be testing exactly the same code, but from a system test perspective, you neither know nor care because a system test is a blackbox test.

Summary

A use case represents the interactions (or observed behaviors) associated with achieving or abandoning a goal. A use case scenario represents one of the possible paths through a use case. A test case represents one set of inputs that exercises a single use case scenario.



]]> http://tynerblain.com/blog/2007/04/12/use-case-vs-test-case/feed/ 0 http://tynerblain.com/blog/2007/01/04/crossing-the-desert/ http://tynerblain.com/blog/2007/01/04/crossing-the-desert/#comments Fri, 05 Jan 2007 04:00:23 +0000 Scott Sehlhorst http://tynerblain.com/blog/2007/01/04/crossing-the-desert/



Johanna Rothman recently wrote an article with a poignant introduction: “A project team focuses on an interim milestone, works like the devil to meet that milestone. They meet the milestone, look up, and realize they’re not at the end of the project–they still have to finish the darn thing. They’re living the Crossing the Desert syndrome.” Fixing it isn’t enough – how do we prevent it from happening?

Unrealistic Scheduling

Johanna points out that when a team has to put in overtime (what we might call heroic efforts) to achieve an interim milestone in a project, the remaining project schedule is suspect. Johanna reccommends revisiting the remaining schedule, and we agree.

Recovery



Johanna also highlights the way to recover – give the people on the project a break. Have them move back to 40 hour weeks if they were working overtime. Force them to take the weekends off if they weren’t already. In short, restore a rational schedule. If people worked through vacations or holidays, require them to take the time off.

Its been said that it takes two weeks of down time to recover from work-burnout. Extending the desert analogy that Johanna credits to Jack Nevison of Oak Associates, we’ve reached the oasis in the middle of the desert, and we need to stay there long enough to rest and rehydrate.



Unfortunately, we’re still in the middle of the desert, and the rush to reach the oasis presumably had merit. If a project schedule is so intense that we have created the need for recovery in the middle of the project, the likelihood of having two weeks to recover is low. Whatever drove us to push to reach the oasis is still driving us, so resting for too long will only make it worse. And we’re still in the middle of the desert. We need to focus on prevention.

Prevention



We can’t always prevent the impetus to work long hours on a project. We can manage our execution in a way that reduces the chances of feeling like we’re crossing a desert.

• Improved estimation of tasks. Entire books are written about this topic, we won’t try and summarize ways to do this in a single blog post.
• Realistic effort allocation. When scheduling how many hours a day a developer can be “on task”, our experience has been that 5 to 6 hours is the maximum (when working full time or a little more). For requirements work, this might even be a little bit aggressive.
• Writing verifiable requirements. We need requirements that specify (or at least allow us to identify) when we’re done. Scope creep isn’t always implementing more features, it can be over-implementing features. With test-driven development (TDD) processes, the tests are written first (they fail), and the developer writes code until the tests pass. Once the tests pass, the code is done. Doing this requires us to write testable requirements. Practically speaking, this may only be realistic when using a continuous integration approach to code development.
• Managing smaller work chunks. Timeboxing (more details) is very effective for controlling the size of iterations. Iterations are important not only for the feedback cycle, but because they reduce the size of the “desert patches.”
• Feedback into the estimation cycle. Timeboxes become even more effective when we keep track of our estimation accuracy, and refine those estimates on the fly to help in replanning future releases. This is a key step in the maturation of a process or company from a CMMI perspective.
• Better communication of release content. Planning releases based on use cases / use case versions is a great way to target communication for external (to the team) stakeholders. It can also be really effective for helping people avoid the desert-crossing syndrome. Essentially you’re providing a map (“The next watering hole is just over that sand dune”) that helps keep efforts in context. One reason the desert image is so powerful is that we imagine being lost in a sea of sand – the hopelessness is a function of both the reality and the perception. Better communication helps address perceptions, while other elements help with the reality.

Conclusion

Overworking the team is bad. Burning them out in the middle of the project is worse. Prevention is the solution, through iterative development, better communication, and improved estimation/planning skills.



]]> http://tynerblain.com/blog/2007/01/04/crossing-the-desert/feed/ 1 http://tynerblain.com/blog/2006/10/02/ten-reasons-to-repeat-tests/ http://tynerblain.com/blog/2006/10/02/ten-reasons-to-repeat-tests/#comments Tue, 03 Oct 2006 04:38:32 +0000 Scott Sehlhorst http://tynerblain.com/blog/2006/10/02/ten-reasons-to-repeat-tests/



Why run a test more than once? If it passed the first time, we don’t need to run it again – or do we? James Bach provides ten good reasons to run the same test more than once.

Common Sense

Common sense dictates that we don’t need to run the same test twice – once we’ve run a test, we know the results – and we therefore don’t need to run it again.

The definition of insanity is doing the same thing over and over again and inspecting different results.

Benjamin Franklin

James Bach writes a great article that points out that we can never do exactly the same thng every time, so running the same test again is not insane. He presents the following ten reasons to run the same test more than once. The commentary here is ours – check out his original post for more details, and other good writing on the topic.

1. Recharge - By changing the code that is being tested, we recharge a test making it valuable again.
2. Intermittence - Some bugs repeat as a function of otherwise uncontrolled variables.
3. Retry - Sometimes we don’t run a test correctly, or can’t be certain that we ran it correctly.
4. Mutation - By creating a test that is a variation on an existing test, we are repeating that subset of the test that is unchanged, but in a different context, established by the mutated version of the test.
5. Benchmark - Performance measurement is most easily understood by establishing a baseline performance with a set of tests – then repeating those tests as the code is modified (or the platform it runs on is modified).
6. Inexpensive - As long as there is some value to a test, and the cost of running it is low – what does it hurt?
7. Importance - “This will never happen again” – best prevented with a regression test.
8. Enough – James uses virus scanning as an example – honestly, this one doesn’t click for us. If you understand it, please add a comment.
9. Mandated – The man always puttin’ me down, makin’ me run the same test over and over again.
10. Indifference / Avoidance – What if your goal isn’t to find bugs, but to avoid them? What better way than to run the same test over again?

Sightings in the Real World
Continuous integration is an approach to testing/development that explicitly calls for the repeated running of tests as part of software development. In James’ framework, continuous integrations uses recharge, intermittence, and mutation as the drivers for repeating tests.

Combining unit tests and functional tests is an effective way to approach system testing. This takes advantage of mutation to provide effective coverage.

By contrast, pairwise testing is designed to achieve enough without repeating any tests.



]]> http://tynerblain.com/blog/2006/10/02/ten-reasons-to-repeat-tests/feed/ 0 http://tynerblain.com/blog/2006/09/12/insight-into-tdd/ http://tynerblain.com/blog/2006/09/12/insight-into-tdd/#comments Wed, 13 Sep 2006 04:55:20 +0000 Scott Sehlhorst http://tynerblain.com/blog/2006/09/12/insight-into-tdd/



James Kovacs shares a great insight on software testing and the software testing process. His epiphany about test driven development makes it obvious to all of us why this technique is so powerful.

TDD

Test driven development is an approach to software development where you write the tests first, and the code second. This sounds ludicrous at first, but it really does work. Here’s James’ epiphany on the subject (emphasis added).

I didn’t understand why TDD practitioner were so zealous about writing the tests first. Why did it matter? I thought it was to ensure that some project manager doesn’t try to shave some time off the project by cutting the unit tests. Then I started doing some reading about TDD, poking around, asking questions, and trying it out myself. I discovered the real reason for writing tests first is that TDD isn’t about testing code, it’s about designing code.

A Tale of Two Epiphanies: TDD and Mocking, James Kovacs

These tests are unit tests, and they are black-box tests (or at least should be black-box tests – read James’ article).

Works Great With Structured Requirements

Not only is this a great way to approach design, but it is a great way to coordinate software development with structured requirements. A well-written requirement describes a need without specifying design. This description then works like a black-box contract. The implementer then must do something that meets the specification given. As James points out, TDD is very effective at creating black-box tests.

This makes sense, when you consider the outside-in approach that comes from starting with requirements. This outside in approach leads to API decisions (for a program or module or method). The API has to support all of the required possible inputs and outputs. Once an API is written, tests can be written.

Best Applied As Part Of Continuous Integration For Larger Teams

All of the tests will fail (because the code hasn’t been written). Then the implementer will write code until the tests pass. Using a continuous integration approach is a very effective way for entire teams to do this simultaneously on larger projects.

Faster Processes

We can also validate the design for completeness prior to creating the implementation. We can compare the generated APIs (and their rationalizations) with the software specification (or application requirements, or FRS… see alphabet soup article). This allows us to address some refactoring needs before the code has been implemented – saving time and money.

Conclusion

Use TDD as an outside-in approach that assures that the delivered software will meet the objectives of the requirements specification. It also saves time and money by supporting validation in advance of implementation.



]]> http://tynerblain.com/blog/2006/09/12/insight-into-tdd/feed/ 0 http://tynerblain.com/blog/2006/08/17/manual-tester/ http://tynerblain.com/blog/2006/08/17/manual-tester/#comments Fri, 18 Aug 2006 02:13:44 +0000 Scott Sehlhorst http://tynerblain.com/blog/2006/08/17/manual-tester/



There’s a piece of North American folklore about John Henry, who was a manual laborer during the expansion of the railroads in our country. His job was being replaced by steam-driven heavy equipment, as the railroad industry applied technology to become more efficient. The same dynamics are happening today with manual testers. We need to make sure that manual testers avoid John Henry’s fate – read on to see why.

Manual and Automated Testing

Software can not be tested efficiently with only manual testing. And it can not be tested completely with only automated testing. James Bach writes an insightful article about how he worked his way to enlightenment over several years, and how he has incorporated this perspective into his training classes.

My understanding of cognitive skills of testing and my understanding of test automation are linked, so it was some years before I came to understand what I now propose as the first rule of test automation:

Test Automation Rule #1: A good manual test cannot be automated.

James goes on to explain that if a test can be automated, it is not a good manual test.

We’ve discussed repeatedly the benefits of automated testing, one example is The Code Freeze is Killing the Dinosaurs. We’ve also stressed the importance of doing functional testing where we discuss the dynamics and tradeoffs of using manual and automated approaches to test software.

Generally, we’ve approached the problem from the perspective of blackbox versus whitebox testing (and more details), and thought about our development process from the perspective of continuous integration as a means to deliver more efficiently.

We’ve not thought about it from James’ perspective before. Even with our improved approaches to automated testing, we are still no different than the inventor of the steam-hammer in Henry’s fable.

Enlightenment

James puts things in perspective:

Rather than banishing human qualities, another approach to process improvement is to harness them. I train testers to take control of their mental models and devise powerful questions to probe the technology in front of them. This is a process of self-programming. In this way of working, test automation is seen as an extension of the human mind, not a substitute.

He highlights a very interesting point – we are missing a key benefit of having manual testers – the ability to gather feedback on intangibles, heuristics, usability, affordances, and other elements that we might classify as design bugs.

We talk about the importance of usable software, and different approaches to creating great software. But we’ve never really talked about how to test for software greatness.

A Broken Model

In the 1970’s, when the American automotive manufacturers were getting their clocks cleaned by the newly exporting Japanese companies like Honda, Datsun (now Nissan), and Toyota, a lot of people thought they understood the problem. Many more people pretended there wasn’t a problem, until the US government bailed out Chrysler.

The experts decided that the main problem was that quality was not “Job One”, that American manufacturers ignored the advice of W. Edwards Demming, and the Japanese did not. Toyota’s lean manufacturing philosophy is credited with much of the success.

It is true that the oil crisis of the time gave the Japanese companies an opportunity to penetrate the US market with smaller, more efficient cars. But that crisis ended (about the time that the American companies killed the muscle car and began building more efficient cars). But the Japanese cars didn’t go away. Once gas prices dropped, they lost a differentiating element – but they maintained two others. Cost and Quality.

Target of Opportunity Enables Strategic Advantage

Cost matters. But it matters tactically, as in “I can afford $X right now – what are my choices?” Quality creates Loyalty. And loyalty is strategic. The Japanese manufacturers gain and continue to gain success after success and award after award in the automotive industry. Why? Because they have good quality.

Automated and Manual Testing

As an engineer, I know that we can specify tolerances, inspect components, and test assemblies to make sure that products are “within specification.” And all of this testing can be automated – just like automated software testing. But passing these tests doesn’t make a product good, it merely indicates that the product is “within specification.” Did the Japanese manufacturers have tighter tolerances? Yes. But did they have better designs? Yes. And those better designs were about more than miles-per-gallon and horsepower and torque.

They were about qualitative items that many engineers struggle to absorb. “Feels Good” and “The controls are where I expect them to be” and “Makes sense” and “I like it.” Software developers often struggle with the same issues.

And this is where manual testing matters. Getting qualitative feedback on your product designs is the only way to improve qualitative elements of those designs. The American manufacturers showed their disdain and hubris in allowing their customers to provide that feedback. The Japanese companies got feedback before they sold their products.

We can use manual testing to provide us with the kinds of feedback that can’t be automated. Things like “This UI is confusing” and “Why won’t it let me…?”

Summary

Don’t give up on manual testing as we strive to achieve software product success. Automate everything we can, and use people to test what we can’t.  We need to make sure that we don’t lose the ability (or make sure that we find the ability) to test manually, for qualitative insight into our products.  We can’t let the testers burst their hearts like poor John Henry, trying to manually perform an automatable test.
Thanks again, James, for starting us down this path!



]]> http://tynerblain.com/blog/2006/08/17/manual-tester/feed/ 4 http://tynerblain.com/blog/2006/08/07/quality-and-requirements/ http://tynerblain.com/blog/2006/08/07/quality-and-requirements/#comments Tue, 08 Aug 2006 05:21:36 +0000 Scott Sehlhorst http://tynerblain.com/blog/2006/08/07/quality-and-requirements/



Quality writers are writing about requirements. Requirements writers are writing about quality. Just like the Reese’s Peanut Butter Cup – Two Great Tastes that Taste Great Together. You can’t have one without the other.

Quality

The Business Analyst Blog writes about speeding up testing:

Everyone has suggestions about how to improve your testing—implement a testing process or methodology, utilize IEEE standards, work towards CMMI compliance, etc. No one mentions that improving requirements will improve testing! [...] Why does it take so long? I would argue that one of the main reasons is that poor (or missing) requirements slow down the process as much as any other issue.

Well said. If we’re testing for the wrong things, we’re wasting time and resources.

Requirements

The Software Quality Blog writes about requirements best practices:

No process is more fundamental than the process of defining and managing business and technical requirements. It’s no surprise that studies cite inaccurate, incomplete, and mismanaged requirements as the primary reason for project failure.

Well said again. The Standish Group estimated that 40% to 60% of defects were due to errors in requirements, not code.

Convergence

In Where Bugs Come From, we showed that having the requirements errors happen throughout the process, not just at the beginning of the process. Numbers 3 and 4 of the following list are the painful and visible impacts of the problems both of our authors above are addressing. And those errors are introduced in numbers 1 and 2:

Requirements errors can happen

1. When stakeholders incorrectly define their goals. A focus on ROI is the best way to assure that our stakeholders are focusing on the right part of the business. Elicitation techniques help us to drive this process in a way that tends to focus on the right requirements. The centerpiece of elicitation is interviewing stakeholders – both users and beneficiaries of the system.
2. When product managers / business analysts incorrectly document stakeholder goals. Writing good requirements is an absolutely masterable skill. Check out these top ten tips for writing good requirements.
3. When developers write unit tests of the wrong implementation. Even with a great continuous integration strategy, all we end up doing is efficiently testing the wrong stuff.
4. When test teams assure that the wrong requirements have been implemented. We waste time testing requirements that aren’t really requirements. And we waste time tracking down false positive defects.

Conclusion

When we’re selling the value of requirements engineering, use this argument to explain the benefits (or impacts) of good (or bad) requirements. When we’re responsible for implementation (dev and test) on a project, remember why we care about requirements. It isn’t enough to verify that requirements are implementable. We have a vested interest in validating that the requirements are also complete and valuable.

Sidebar

[disclaimer: I am a bit of a dork.] I cracked myself up today when I typed “errors of omision”.



]]> http://tynerblain.com/blog/2006/08/07/quality-and-requirements/feed/ 0 http://tynerblain.com/blog/2006/06/13/writing-verifiable-requirements/ http://tynerblain.com/blog/2006/06/13/writing-verifiable-requirements/#comments Wed, 14 Jun 2006 03:32:58 +0000 Scott Sehlhorst http://tynerblain.com/blog/2006/06/13/writing-verifiable-requirements/



One of the ten big rules of writing a good MRD is writing verifiable requirements. Verification is both a function of having a precise goal, and having the ability to affordably measure the requirement. A precise goal is a verifiable requirement if we can clearly answer “yes” or “no” when asked if the requirement has been implemented. We also face the practical realities of being able to measure the results profitably.

The Big Rule of Writing Verifiable Requirements




From our previous article, Writing Good Requirements – Big Ten Rules:

We use a process that starts with market requirements, and then decomposes them into software requirement specifications. the market requirements must be written in a way that we can verify that the associated requirements specification will meet the market need.

Verifiable

We wrote previously that if a requirement can’t be tested, it should be rewritten. Roger Cauvin has recently reposted on testable requirements. Perhaps in anticipation of this article? :).

In the world of test driven design (TDD), the philosophy is to write the test first, and then write the code. We use a continuous integration approach to check in code regularly and run all of the tests as we go. When the test for a particular requirement passes, the code is done for that requirement.

We also get a big benefit in validating that we have delivered the right functionality to the customer. When we agree on the language of the requirement, we are also agreeing on the language of its verifiability.

Affordably Verifiable

With this big rule, we are assuming that we have already eliminated impractical or unattainable requirements.

Some tests are very expensive, or nigh on impossible. Roger alludes to this as testable in principle. Requirements should be testable in practice whenever possible. If a requirement can not be tested (“Must last 10 years”) in the delivery timeframe, we need to at least define a specific acceptance criteria – (“95% confidence in a cycle life of 10,000 cycles per procedure XYZ.doc”). This is the approach that Underwriter Labs uses for awarding UL certification to electrical appliances for safety.

Semantically, we can argue that this proxy criteria is in fact the requirement. It is the criteria by which acceptance is determined.

If the test is possible, but impossibly expensive, we shouldn’t run the test. If we can not agree on a practical proxy criteria, we are left with two choices. We can choose to not implement the requirement. Or we can choose to agree that the requirement is delivered when we say it is. Neither is a particularly good option, but the former is better than the latter, when it comes to maintaining a positive relationship with our customer.

Writing requirements without acceptance criteria introduces risk into our project. The last thing we want is to enter into a nitpicking discussion about what we have and have not delivered. We would much rather spend that time discussing what we can deliver next.

Conclusion

Write requirements that can be verified given the constraints of the project. Common constraints are time, money, equipment and expertise. Use language that makes the verification process explicit. The process of determining verifiability also dramatically helps eliminate ambiguity in our requirements.



]]> http://tynerblain.com/blog/2006/06/13/writing-verifiable-requirements/feed/ 0 http://tynerblain.com/blog/2006/05/29/test-smarter-not-harder-a-detailed-article/ http://tynerblain.com/blog/2006/05/29/test-smarter-not-harder-a-detailed-article/#comments Tue, 30 May 2006 03:18:32 +0000 Scott Sehlhorst http://tynerblain.com/blog/2006/05/29/test-smarter-not-harder-a-detailed-article/



developer.* has just published a 15-page article, Test Smarter, Not Harder by Scott Sehlhorst on test automation. We present the background for that article here (why you should read it) as well as a summary of what is covered in the article. Check it out at developer.*, and thanks to Dan Read, both for starting developer.* (which is a GREAT site) and for publishing the article. Skip to the end of this post to see how you can help.

Automated Testing Must Be a Core Competence

Automated testing has become a critical component of software product success. Processes like continuous integration require test automation to be effective. Unit testing requires automated testing to be effective as a rapid-development tool. Whitebox testing is almost always done with automation. More and more blackbox testing is being done with automation. A team without a good approach to automated testing is working at a severe disadvantage, and is arguably doomed to extinction.

Functional Testing (or system testing) is where a team can achieve process differentiation with automation. 80% of teams rely on manual functional testing today. These teams argue that automating functional tests is too expensive. The reasons most cited are complexity and rapid change in the underlying product.

Testing Complex Software

Complex software presents very challenging problems for test automation. First – how do we deal with the extreme complexity of many of today’s software applications? Billions or trillions of combinations (scripts) of user actions can be made. And software that works with one combination may be buggy in hundreds of others.

We can’t realistically test all of the combinations. Even if we did have the hardware needed to run them we would not be able to validate the results of all of those tests. So we can’t achieve exhaustive test coverage.

Without exhaustive test coverage, how do we know when we’ve tested enough?

Test Smarter Not Harder

The article starts with an analysis of the complexity of modern software and a brief discussion of the organizational realities and typical responses. Then the article explores common approaches and techniques in place today:

• Random Sampling
• Pairwise Testing
• N-wise Testing

The article explores the math, benefits, and limitations of each approach. While N-wise testing is the most effective approach in common use today, it has a limitation – it assumes that the user’s actions (input variables) happen in a prescribed sequence or are irrelevant. In most complex software this assumption is invalid. The article presents a technique for incorporating order-dependence into the statistical approaches for developing test coverage.

The article also demonstrates techniques for simplifying the testing representation using whitebox testing techniques to redefine the problem space, and then applying blackbox testing (statistical) approaches to execution of the resultant test parameters.

By combining the statistical techniques explained in the article with a communication plan for sharing the appropriate quality metrics with stakeholders, we can deliver higher quality software, at a lower cost, and with improved organizational support and visibility.

Why developer.*?

While many Tyner Blain readers are interested in test automation, most of our readers would not want to go into this much depth. The audience at developer.* has a more technical focus, and their site is a more natural place for this article. Personally, I am honored to be included as an article author there – and all of you who want more technical depth than we go into at Tyner Blain should really check out the stuff Dan Read has put together. They’ve also just published their first indie-publisher ‘real book‘, which you can get via their site. My copy is on the way as I type.

How can you help?

This is a great opportunity for PR for Tyner Blain – please share the article with your associates because we want our community to grow. Also, it would be great if you ‘digg’ the article at digg (just follow the link, and ‘digg it’), del.icio.us, blink, or any other networking site. In addition to being good content, I hope to make this an experiment in viral marketing. Lets see if we can collectively reach the tipping point that brings this article, and the Tyner Blain community to the next group of people.

[update: the article just made the front page at digg and has 374 diggs and 32 brutal comments :). Welcome to all diggers, mobile, agile, or hostile!]



]]> http://tynerblain.com/blog/2006/05/29/test-smarter-not-harder-a-detailed-article/feed/ 0