When companies first start off-shoring, they usually send the “low level” implementation work overseas first, to work out the process kinks and manage risk. Over time, your valued, domain-aware developers will perceive a lack of career opportunities with this limited role. Naturally, you will want to consider sending design work offshore too. You can make it work. If you do it wrong, you’re toast.

Different Models for Offshore Development

There are many ways to organize software-creation teams (where creation includes product management, design, development and testing). When developing an organizational design that incorporates elements of off-shoring, there are four primary models for outsourced software creation.

1. No offshore development at all. Occasionally referred to as insourcing, this is the traditional “everyone in one place” model – or at least “everyone in similar time zones.”
2. Low-level outsourcing. The implementation teams (both coding and testing) are located offshore, with design and product management staying local.
3. High-level outsourcing. The focus of this article. Keeping product management “local” but moving design and development responsibilities offshore.
4. Complete technical outsourcing. Everyone except the product manager is offshore.

The models can be most easily compared when the same process is compared in each – just with different locations. Co-location of team members has an impact when comparing face-to-face communication with online / phone / remote communication. But this factor is not a primary one in influencing team-decisions – it is no different than having someone who works from home. The key element in team dynamics is a dramatic shift in timezones.

This timezone shift causes a latency in the communication process, illustrated by the following example:

Imagine the following expensive question and answer session:

• Person A asks person B a question.
• 12 hours later, person B responds with a request for a clarification.
• 12 hours later, Person A clarifies the question.
• 12 hours later, Person B responds with an answer.
• 12 hours later, Person A acknowledges the answer.

When this communication channel happens between an onshore person and an offshore person, it takes 48 hours instead of 48 minutes. The more this happens, the more expensive it is to outsource. The key to making low-level outsourcing work cost effectively is to minimize the impact of this communication latency, while realizing the benefits of lower salaries in the offshore location.

Making Offshore Development Work

This is why proper communication design is the make-or-break element of making collaborative teams work when using an outsourcing model that involves anyone being offshore.

High-Level Outsourcing

Consider the following software development process diagram (from Four Application Development Outsourcing Models):

In this off-shoring process flow diagram, the shaded areas represent the activities that happen offshore. Note that this is the exact same process flow that is used to describe the other outsourcing models. The difference from other models is primarily where resources are located, but also the relevant scope of responsibility. In comparison with low-level outsourcing, the only difference in the process flow is that responsibility for test design and solution design is transitioned to the offshore development team.

Artificial Boundary

This model creates a bit of an artificial boundary between the “interpret requirement” step and the “design solution” and “design tests” steps. This artificial boundary creates a potentially odd dynamic within the team.

To make reading easier, we’ll talk about the development side of the process flow only, but the same ideas apply on the testing side.

This model has one developer interpreting requirements, and a different developer doing the design work. In an insourcing model, those two developers are usually the same person. In large development teams, a common breakout here is architect versus senior designer. The architect (the figure on top) would be responsible for those design considerations that span the enterprise and the senior designer would be responsible for those design considerations that affect a single application. Here’s a good background article by Scott Ambler on approaching enterprise architecture.

Why have an artificial boundary? Because this model is an emergent design.

1. A company starts with low-level outsourcing.
2. The offshore developers complain about a lack of growth opportunities (both career and skill).
3. Executives are not prepared to completely outsource all technical work (risk aversion), or the team is not ready to succeed with that approach.

Splitting the “interpret requirements” task from the “design a solution” task is a compromise. It minimizes the risk associated with providing growth opportunities to offshore team members. That risk is minimized by avoiding the high-latency communication channel (from onshore to offshore) when communicating requirements. Instead, the interpretation of the requirements is done onshore, and that interpretation is communicated.

You can argue that this model is the offspring of a trust issue within the organization. A company can absolutely say “We want growth opportunities for our offshore team members” and at the same time say “We are not ready to relinquish complete technical control.” This is definitely a trust issue, and therefore a perceived risk mitigation strategy. The risk may be very real, or non-existent. In either case, it is emotionally present.

Vague Scope and Role Conflict

If, while reading this, the notion of splitting interpretation from design feels uncomfortable, that’s because it is. For many teams that are evolving their outsourcing model, it feels uncomfortable, but it feels less uncomfortable than releasing complete technical control.

One problem, which I completely grok as a former developer, is that it is almost impossible to interpret requirements without imagining designs. But this model has different people performing the different tasks. So should the interpreter just discard those designs? Presumably the interpreter is more experienced, certainly more knowledgeable about the domain. It would be a shame to discard those design insights.

Robbing Peter to Pay Paul

This approach has clear benefits for the offshore developers who are now presented with growth opportunities. Unfortunately, you run the risk of sucking the fun out of the role of the interpreter – a senior, experienced developer with domain knowledge. While proper requirements interpretation can be fun, it usually is not fun for a developer. Only requirements people will enjoy those nuances, generally.

You risk eliminating a career path and growth opportunities for your onshore resources with this model.

Division of Labor

One approach that keeps the growth opportunities open for your senior onshore technical resources is to have them play multi-product, or architectural roles. This presents an opportunity for these individuals to apply organizational insights across products, finding synergies between applications and driving consistency and consolidation among applications. You essentially split the responsibilities “broad and deep” where the onshore designer is looking across the portfolio and the offshore designer has ownership responsibilities for a single application or scope. This is similar to the division of responsibilities that works well for tackling enterprise product management.

Instead of preventing communication of requirements across the high-latency channel, it minimizes it. The onshore designer acts as the liaison between multiple offshore designers, fielding interpretation questions, and more effectively, preventing them. Developers have a language all their own. Through a shared perspective on common experiences, they can communicate very efficiently – by analogy, symbolically, and via design patterns. These communication opportunities (between like-minded individuals) can have very high information density. For example, one developer can say “MVC pattern” to another, and that serves more effectively to communicate an approach to designing a solution (and a context / interpretation of requirements) far more efficiently than a product manager describing requirements that multiple tools and platforms should expose the same set of capabilities or behaviors [and that is overly short, because I don't feel like typing a comprehensive explanation of the MVC pattern].

Another approach is to silo the developers vertically – having some applications (or modules) following a low-level outsourcing model, and others practicing complete technical outsourcing. Any given team is operating discretely with a clearly defined set of responsibilities. The teams may just be operating differently. Essentially, you’re saying that your “average” is high level outsourcing, even though it is really a mix of two models. This doesn’t really count, since none of your teams are addressing the communication challenges of this model – your company is leapfrogging over it, but choosing to mitigate risk by doing it a little bit at a time.

You can also take the approach of having the onshore designer be responsible for over-arching and high-complexity design issues, with offshore designers owning more straightforward and lower risk design activities.

The best approach for maximizing your team’s effectiveness (and your HR goals) will be overwhelmingly determined by the individuals involved. If you’re in a large company, you probably don’t get to maximize team effectiveness – you are likely to be forced into a particular model. If you’re doing the forcing, recognize that one size does not fit all.

High Latency Communication And Designing

When circling back to the main challenge of offshoring – communication – you need to look at the nature of the communication that is subject to the high-latency of temporal displacement within the team. The key to making this model work is to leverage the efficiency of cross-talk between developers. That means having experienced people on both the offshore and onshore teams who share common design backgrounds (patterns, analogies, examples) and deep domain understanding (reducing the provision and clarification of context across the communication channel).

Design reviews are also very effective at eliminating the impact of this latency on communication. Design reviews typically happen as follows:

1. Designer A spends time creating design based on an interpretation of requirements.
2. Designers A & B get together and review the design in a relatively short meeting (or Designer B reviews a design document). Feedback is delivered in a burst.
3. Designer A spends time updating the design based on feedback from step 2.
4. Return to step 2 as many times as is needed.

There is a big chunk of work involved in incorporating feedback from a design review. This can be folded into the “white space” between communications. In other words, while designer B is sleeping for the night, designer A is making updates. This looks like, but is not the follow-the-sun pattern.

The follow-the-sun pattern is one where someone is always working. I’ve made this work on a project where there were two teams working 12 hour shifts with a 4 hour overlap (and a 4 hour “blackout”). Note – that isn’t sustainable, just anecdotal. I think the linked article is right, commonly, follow-the-sun implementation does not work, for the reasons cited in that article.

What makes this very different is that we are synchronizing the naturally-occurring time lag between design reviews with the geographically-induced time lag in communication.

Conclusion

The strategy for successfully utilizing offshore resources for both implementation and design starts and ends with communication. It also requires attention to your specific people and their career aspirations.

• Utilize design reviews to take advantage of serendipitous time-lags in the communication cycle.
• Assure that your role definitions are clear, and aligned with the aspirations of the people on your team.
• Follow the tips for effective offshore development – this strategy builds on that one, it does not replace it.

Economic pressures are driving most companies in high-developer-salary markets to explore using offshore development teams as part of their approach to developing software. Developing software with a global team presents new challenges as well as new benefits. If you do it right, you can have a more cost-effective team. If you do it wrong, you can have a disaster.

Different Models for Offshore Development

There are essentially four different models for managing a software development team, with respect to onshore and offshore roles.

1. No offshore development, also known as insourcing.
2. Low-level outsourcing, having the implementation (but not design) team members offshore.
3. High-level outsourcing, having the implementation and design team members offshore.
4. Complete technical outsourcing, having all technical implementation team members offshore.

Each different model has the same set of people involved in the process, with the same channels of communication. The differences are in which communications happen across geographic and temporal boundaries, and which communications happen in the same time zone.

For this article, we are focusing specifically on low-level outsourcing, where the communication channel most affected by different timezones is the one between design and implementation.

Low-Level Outsourcing

Consider the following software development process diagram (from Four Application Development Outsourcing Models):

Each area surrounded by a dashed line represents a different type of work, or a different required dominant skill set. Every stage in the development process requires a different set of dominant skills, and all areas share a set of common skills. When exploring different offshoring models, teams are most effective when they identify the distinctions in dominant skill set requirements, and divide responsibilities along those boundaries. When you create an artificial (or arbitrary) boundary within one of the regions in the diagram, you create opportunities for misunderstanding. With those misunderstandings, you can have people redundantly working on the same thing, or even worse, you can have tasks that don’t get accomplished. You also introduce the possibility of discord within your team as people either proclaim “that’s my job” or “that’s not my job.” You can split a team within the areas shown above (and we’ll talk about that in a future article), but it is harder to do successfully.

Low level outsourcing is an approach where the implementation team members who write the code and tests are offshore. The requirements work is done onshore (where salaries are expensive). Interpretation of the requirements is also done onshore. The design of the testing plan is done onshore, and the design of the solution is also done onshore. The creation of tests and the implementation of the code is done offshore.

In the diagram above, testing of requirements happens on the left, and testing of the implementation happens implicitly on the right. If you haven’t been reading Tyner Blain for a couple years, that may not make much sense. Testing of requirements and implementation are different, and you need to do both.

Testing: Isolation of Variables Reduces Costs

Testing is something that can be approached from a few different perspectives, and the word “testing” means different things to different people. In this process flow, we are focusing on two main areas of testing – testing the requirements and testing the implementation. When you test the requirements, you are asking the question “does this solution do what the product manager intended?” When you test the implementation, you are asking the question “does my solution behave as designer intended?” This is a nuanced difference, and non-technical people may say “what is the difference?” The designer intends exactly the same thing that the product manager intends.

If you think back to your high school science class, you’ll remember the concept of “control of experiments.” This is the field of practical application of logic to scientific experimentation. By taking a logically rigorous approach to designing a science experiment, you can isolate variables, and test each of them independently. This prevents you from drawing false conclusions from your data. The same process leads you to test both the requirements and the implementation. If you’ve ever submitted a bug and had the implementation team close it out with the statement “working as designed”, you already know the benefit of testing both. Just because something is designed to do X does not mean it is not supposed to do Y. By introducing a designer between the product manager and the testing, you introduce the possibility that the designer is the source of the bug – by misinterpreting the requirements.

It is possible to “test” the design and then test the implementation – this will isolate the design from the implementation. Unfortunately, the only way to “test” a design (without also testing the implementation) is conceptually, with a thought experiment. And that’s exactly what the designer does as part of designing. No one else is really going to understand the design well enough to do that. And if the designer is doing the thought-testing, there is no way to isolate if the designer misinterpreted the requirements in the first place. That same misinterpretation will influence his testing in the same way that it influenced his design (See error source E3 in Where Bugs Come From for more details). This is critical to designing, but it does not work for testing a design.

It is possible to test just the requirements. You create tests based solely on the documented requirements. You run those tests against the implemented solution. When the test passes, every step in the process worked. These are known as black-box tests, because you can run the tests without any insight into how the software is written (it is a “black box”). The problem comes when a test fails – you know something is wrong, but you have to do (expensive) research to figure out the source of the problem. It could be that the implementation failed to do what was designed. Or it could be that the software design failed to meet the objectives of the requirements. There is a way to reduce the cost of this analysis – by testing both the requirements and the implementation.

You can easily test the implementation. An implementation test, commonly known as a white-box test, and usually implemented as a unit-test, inspects the effectiveness of a particular implementation at doing what the designer intended. When you combine implementation testing with requirements testing, you isolate the designer variable. If a requirements test fails but the implementation tests pass, the problem is with the design (or with the design of the test). When both requirements and implementation tests fail, you know that at least the implementation is wrong.

In the diagram above, the testing on the left side represents testing of requirements. The right side of the diagram implicitly includes testing of the implementation as part of implementing. You need to ingrain your implementation testing into your development philosophy. Would you deliver code without compiling it? Why, as a developer, would you consider delivering it without testing it? Compilation is not just a build step, it is also an implicit test of compilability. You should also include the implicit test of “does what I intended it to do.”

Combining the discipline of continuous integration with test driven development is the most effective way to accomplish this. Note: remember this part, as it is a critical component to making low-level offshore development work. Without this, you may as well give up – you certainly aren’t going to be more cost-effective.

Communication Across Time And Space

The key to making offshoring effective, as with any development process, is to make the communication work. For communication that happens between people in the same location (or at least roughly the same time zone), the problems and solutions are no different than when you have an insourcing model. What’s different is the communication that happens between members of the onshore team and the offshore team. This communication is not just remote (technology helps us solve these problems with instant messaging, phone calls, and other real-time (or near-real-time) techniques), but also phase-shifted in time. When you have team members working while other team members are sleeping, you slow down the collaborative process. You introduce a near-crippling latency into the communication channel.

Imagine the following expensive question and answer session:

• Person A asks person B a question.
• 12 hours later, person B responds with a request for a clarification.
• 12 hours later, Person A clarifies the question.
• 12 hours later, Person B responds with an answer.
• 12 hours later, Person A acknowledges the answer.

When this communication channel happens between an onshore person and an offshore person, it takes 48 hours instead of 48 minutes. The more this happens, the more expensive it is to outsource. The key to making low-level outsourcing work cost effectively is to minimize the impact of this communication latency, while realizing the benefits of lower salaries in the offshore location.

Communications On Which To Focus

The “happy path” communication channels (shown with blue arrows in the diagram) are the transfer from “test design” to test, and from “implementation design” to implementation. You have to communicate the designs in such a way as to minimize misinterpretation of the design. Never prevent needed communication. Your goal is NOT to stop communication, but to preempt it by eliminating the need. The only thing worse than taking too long because you have a lot of communication is failing to communicate enough. Your goal is NOT to prevent communication, but to minimize the need for it.

The “trust but verify” communication involves making sure that the implementation meets the design. In (requirements) testing, it means reviewing that the tests exhaustively cover everything identified in the test design. It also means reviewing that each test (implementation) actually (effectively) tests what it is designed to test. As team members demonstrate their capabilities, they require less oversight – which is true of any mentoring relationship. In implementation, it means verifying that the code does what the design requires. You could read the code and make a determination, but that is a manual inspection of the code, and manual inspections have been shown to be at best 80% effective as a testing method. What you need to do is create a unit test suite, run it continuously, and only allow developers to check-in their code (to the trunk) when the entire suite passes. Then all you have to do is review the test suite to assure that it will test the design effectively. It wouldn’t hurt to also run the test suite locally (onshore) as a verification, but fundamentally, you are trusting that your developers will follow this continuous integration process.

You are using testing and test design as a mechanism to validate effective communication. You can think of it as a form of active listening. When you (or more appropriately, your design document) says “X”, you can review the test design to confirm that your listener designed tests that assure “X” will happen. Do not just rely on informal communication and acknowledgement. Cross-cultural communication introduces a lot of complexity and misinterpretation. People who do not share a common language or culture also tend to interpret symbols very differently, and will rarely have the same connotations for given interpretable words.

Definition of tests and validation of testing is important beyond the immediate communication of design. There are also the feedback loops that come from the “unhappy path” when something goes wrong (and a bug is introduced). Each of these “where was the bug introduced, and how do we fix it?” cycles is also affected by the latency of cross-shore communication. Good testing reduces the number of these otherwise unwarranted communication cycles.

Developing good design docs is also critical to the success of this communication. The definition of “good” for a design doc is so dependent on the exact circumstances that it is impractical to try and define what that is (at least within this article). A design doc needs to be written with the reader in mind, not the author. Beyond that, we won’t try and make any statements of truth.

Conclusion

The strategy to successful utilization of offshore resources for development and test implementation work starts with communication. The strategy also ends with communication.

• Create artifacts (good design docs) that minimize the clarification cycle across the onshore-offshore time boundary.
• Review implementation tests as an active-listening mechanism to confirm that communication of design intent was effective.
• Practice continuous integration (both as an offshore developer and as an onshore designer or development manager) to assure that your solution stays true to the design and the requirements.

And, as always, have great people – because people trump process.

One of the ten big rules of writing a good MRD is writing unambiguous requirements. Ambiguity is a function of communication. The writing can be generically ambiguous, or ambiguous to the writer. A requirement could be precise in intent, but ambiguous in interpretation by the reader. Understanding our audience is as important as precision in language. We write unambiguous requirements because misinterpretation of requirements is the source of 40% of all bugs in delivered software.

The Big Rule of Writing Unambiguous Requirements

From our previous article, Writing Good Requirements – Big Ten Rules:

A great requirement has a single interpretation. A good requirement has a single reasonable interpretation. As part of our development process, we will use listening skills like active listening to make sure that our engineering team understands the requirement we intended to write. The better the requirements, the less expensive and risky this communication process will be. Writing unambiguously is critically important when using outsourcing models that limit our interactions with other team members.

Ambiguous to the Writer

We introduce ambiguity with imprecise language. Using shall instead of should is one of the first things people suggest (or require!) when writing requirements. [Added 2010.08.17 - shall is ambiguous too - use must!]  Red flags are also raised with words like can and might. Marcus wrote a good post about vague requirements language

What do the terms: user-friendly, flexible, easy-to-use, fast, and intuitive mean to you? Do you think these terms mean the same thing to someone else? Generally, no!

from Speculative and Vague Terms

These are examples where the language is ambiguous, both to the writer and the reader. Any uninformed third party could read these requirements and identify the amiguity in the language. This makes these mistakes easy to catch – all that is required is a good working knowledge of the language.

Ambiguous to the Reader

Even precisely written, grammatically correct prose can be ambiguous to the reader. This ambiguity can come either from lack of expertice with the language, or from incompleteness of the requirement.

Language Ambiguity

With the ever-increasing outsourcing of teams, we have to think about writing requirements for outsourced team members. When we use a complete technical outsourcing model, we have to consider the possibility (or certainty in some cases) that the primary language of the readers of the MRD is not the language it is written in. Making a document easy to read (short sentences, common words) can be at odds with making the language of the requirements precise.

Some good research on vocabulary size data for comprehension of english can be found here. The average native english speaker knows 20,000 word families. With a vocabulary of 5,000 english words, only 98.5% of the words in a given text will be understood. 5,000 words is a lot for a speaker of a second language (3,000 words is considered a working knowledge).

An analysis of the reading-level at which a document is written can be helpfull in identifying if the language is likely to be challenging for readers with limited vocabularies. The Gunning-Fog Index provides a measure of the education level at which a text is written.

Incompleteness Ambiguity

When the language is both precise and understood, we still face challenges in ambiguity by failing to provide all the information. When we find ourselves wanting to say “That was implied, you should have known that!”, we are still being ambiguous. Clarity exists not only in language but in intent. Should the reader assume that when we specified user-authentication and role-based functionality that we intended users to have roles? If we specify that the best-matching search results must be presented within 1 second, is it ok if the rest of the results are presented later? And how many of the best-matches must be found?

Writing requirements like this is definitely a risk, and probably ambiguous. If we have a history, a rapport, and synchronous feedback cycles with the readers of the document, this may not be vague. We may be able to rely on them to assume the same things we assume. The language in the document may be serving effectively as shorthand for this communication. If we are working as a team with a shorter history of working together, this almost certainly will not communicate what we intended. There is also a risk of going too far.

Conclusion

Writing unambiguous requirements requires us to write complete requirements. It also requires us to use precise language that communicates information across domains to our readers. To determine the right level of effort, we need to monitor the effectiveness of our communication, and balance that with the amount of time we can afford to dedicate to word-smithing instead of other product management activities.

Allan Holmes, for CIO Magazine just posted a scathing and detailed autopsy of the disastrous Medicaid Claims System project run by CSNI and launched in January of 2005. Requirements elicitation failures combined with incompetent vendor selection and project mismanagement lead to a $30,000,000 oops for the state of Maine, jeopardizing its credit rating. The system failed to process 300,000 claims in the first 3 months of operations, causing many health care providers to close their doors – and presumably causing citizens of Maine to go without needed services. Maine is the only state in the union (as of April 2005) not complying with federal HIPAA regulations.

Autopsy results

There were crucial failures in essentially every step of the project. We’ll look at each of the following areas:

1. Defining requirements and creating an RFP (request for proposal)
2. Vendor selection
3. Requirements validation
4. Risk management
5. Execution (Project management and development)
6. Testing
7. Deployment / Change Management / Training

Defining requirements

April 2001. Maine issued an RFP for the new HIPPA compliant system. By the end of the year, only two bids were placed – one for $15 million and one for $30 million. Holmes tells us that this is a sign of a bad RFP:

…says J. Davidson Frame, dean of the University of Management and Technology in Arlington, Va. “Only two bidders is a dangerous sign,” he says, adding that the low response rate indicated that potential bidders knew the requirements of the RFP were unreasonable.

Requirements elicitation done poorly is the major source of defects in any project.

Taking a step back, we see from Holmes that Maine decided to use a new (to them) technology and develop the software themselves instead of outsourcing. The justification being that it would be easier to adapt to changing requirements (this becomes ironic later – read on).

The development of the new system was assigned to the IT staff in the DHS, which decided it wanted a system built on a rules-based engine so that as Medicaid rules changed, the changes could be programmed easily into the system.

Vendor selection

Quoting Holmes:

In this case, the low bidder, CNSI, had no experience in building Medicaid claims processing systems. In contrast, Keane had some experience in developing Medicaid systems, and the company had worked on the Maine system for Medicaid eligibility.

OK, maybe not so bad, but wait – more irony. The final costs (to the State) of going with the low-cost vendor exceed the bid from the high cost vendor.

Requirements validation

To begin with, the 65-person team composed of DHS IT staffers and CNSI representatives assigned to the project had difficulty securing time with the dozen Medicaid experts in the Bureau of Medical Services to get detailed information about how to code for Medicaid rules. As a result, the contractors had to make their own decisions on how to meet Medicaid requirements. And then they had to reprogram the system after consulting with a Medicaid expert, further slowing development. [emphasis ours]

We wouldn’t use the same language as Holmes, we would say “… the contractors decided to make their own interpretations of how to meet Medicaid requirements.” They never had to do it – they chose to do it. In Where bugs come from we show the impact of having or not having a feedback loop for validating requirements. Not having that feedback loop was either a decision of incompetence or hubris.

No one is blameless for this mistake. Maine’s IT department is responsible for making sure the contractors are doing what they really want. The contractors are responsible for doing what Maine wants. At a minimum, the SMEs should have been interviewed, and the contractors should have at least used active listening techniques to validate their interpretations of the statutes. All the way down to the developers, who should have required that they understand the context in which they are coding. They should have said “why?” until they got answers.

Risk Management

New vendor. New technology. Maine knew that the requirements were not good.

Thompson decided that the six months that would have been needed to redo the RFP was too much. “We had a requirement to get something in place soon,” Thompson says.

No access to SMEs (subject matter experts). No system tests (more on that later). No backup system. No contingency plans if the system didn’t work.
If there was a risk management plan in place, it certainly didn’t change the course of events.

Execution

Starting with project management:

• Oct 2001. CNSI is selected as vendor – project length: 12 months.
• Fall 2002. Project timeline doubled to an Oct 2003 delivery.
• Fall 2003. No delivery.
• Fall 2004. No delivery.
• Jan 2005. System goes live.
• Apr 2006. System now (claimed to be) operating at same level as legacy system.

And with development (here’s the aforementioned irony):

The development of the new system was assigned to the IT staff in the DHS, which decided it wanted a system built on a rules-based engine so that as Medicaid rules changed, the changes could be programmed easily into the system.

Errors kept cropping up as programmers had to reprogram the system to accept Medicaid rule changes at the federal and state levels.

Wow.

Testing

Hey, testing is optional.

testing the system from end to end was dismissed as an option. The state did conduct a pilot with about 10 providers and claims clearinghouses, processing a small set of claims. But the claims were not run through much of the system because it was not ready for testing.

Conclusion

Holmes presents excellent conclusions about the HIPAA project. Our conclusion – we need more people in Maine to read the blog. If you know someone in Maine, send them a link. In some seriousness, there’s a T-shirt that says “If you can’t be a good example, be a horrible warning.”

Thanks for the warning, Maine!

Frederick Boulanger picked up on our earlier article about different application development outsourcing models, and extended it with his own good ideas- making it easier for teams to decide which outsourcing model is right for them. Frederick identifies the three key factors that determine which model is most likely to succeed for a given team. They are control, coordination, and communication. Anyone else want to join in? Blog away, and trackback or comment here.

Control

Are we outsourcing turn-the-crank type activities? What about tasks that require only entry-level skills? Frederick points out that there is a continuum of desired control for any company or project. When we maintain as much control as possible, we only outsource very specifically defined work-elements. When we have less need for control, we can outsource more and more of the process. If we keep design in-house but outsource implementation, we still have a lot of control over the final results.

Coordination

Coordination of activity is central to the success of any team. When everyone on the team is in the same room, coordination almost happens automatically. Having everyone in the same building helps, and having everyone in nearby timezones is the norm these days for non-outsourced projects. A lot of people treat outsourcing as synonymous with offshoring – the term for outsourcing to teams on different continents.

Technology has eased the pain of a geographically distributed team – instant messaging, email, collaboration applications, and video conferencing have reduced the need to travel, and made coordination easier for dispersed teams. Offshoring also creates temporally distributed teams, as team members are working in very different time zones. When team members are eleven and a half hours out-of-phase with each other, coordination becomes much more important.

Asynchronous teams also face an efficiency challenge. The typically iterative tactical communication among team members may only take minutes when folks are simultaneously working. It can take a day per iteration when the communication happens only through email (with a multi-hour delay between each exchange of information).

Defining the communication process our team will use is important. Documenting this process helps to set expectations for the outsourcing team, and is required for most CMMI levels.

Communication

Precisely defined tasks (the turn-the-crank variety) require far less iterative communication than more strategic activities like validating requirements or architectural design. Offshoring projects will be more effective initially when the majority of the outsourced work is partitioned into narrowly scoped deliverables.

After developing a good relationship with our outsourcers, and working the kinks out of our communication process, we can begin to relinquish control by outsourcing work with greater scope of impact. I’ve had excellent success in doing this by leveraging documentation to communicate with overseas team members. When giving them greater responsibility, I would require that they document their design and their test design for review prior to any implementation work. This dramatically reduced the effect of ambiguity in the requirements, while implicitly providing an active-listening feedback loop. Most often, misinterpretation of requirements resulted in errors in the test designs. It also had the side benefit of helping my teammates grow their skills more quickly as it forced more critical thinking to occur early in their process.

Summary

Communication is important to any outsourcing effort, and increasingly critical when relinquishing increased levels of control. Process execution and coordination determine how repeatably we can communicate at any level of control.

On March 30th CIO magazine published an article titled Do’s and Don’ts of Outsourcing Benchmarks. The article spurred us to write about outsourcing models for product development – it is otherwise unrelated, but interesting.

Explaining Four Application Development Outsourcing Models

Outsourcing software development is different than outsourcing services like call-center operations. There are people who interact in several different roles in software development.

There are two distinct types of outsourcing. Bringing experts into our company, or sending work out of our company. Experts who join our team are “temporary team members” not “outsiders.” Sending work outside requires that we address several issues in communication, expectations, and accountability.

We will focus on the models for sending some of the work outside our company, to a team that operates in a very distant location and timezone.

Process overview

We’ve described the software development process (from market opportunities to delivered software) in several posts in detail including Software development process example and Where bugs come from. We’ll provide a brief overview of the flow only in this post, with some simplified versions of roles in the process. This post is focused on the communication elements of handing off responsibility.

In the following diagrams, each area (dashed-line border) is the responsibility of the person within the area (developer, QA, etc). All of the maroon arrows represent transfers of information or responsibility within the organization. Any blue arrows represent communication with outsourcers, who are accountable for the next step in the process. Outsourcing rectangles have a light blue shading. Note that feedback loops are not drawn, but are important to success. Details on feedback can be found in the previously linked posts.

In each model, we list a key to success which targets the largest additional challenge of each model. Each model still otherwise faces all of the challenges of developing great software. As such, we don’t list a key to success for model 1 – that’s our baseline.

Model 1: Insourcing

Keeping all of the work in house, and in nearby timezones (geographic split of the team is ok, as long as they have >75% overlap in working hours). Communication between members of the team is important, and clear lines of responsibility are drawn.

Model 2: Low-level outsourcing

The first step most companies take to outsourcing is to send the “boring” or “unchallenging” or “low risk” work to an outside company. This model is shown in the following diagram, with QA and development outsourcing of the implementation and testing steps.

The key to success with this model is in execution management. Code reads and validation that everything designed has been implemented.

Model 3: High level outsourcing

Additional responsibility (design) and more abstract tasks are transitioned to the outsourcing firm – usually after they have demonstrated the ability to execute on lower-level tasks.

The key to success with this model is the technical communication and interaction between the architect-level developers and testers who are responsible for interpreting the PRD, and their outsourced counterparts with design responsibility. These in-house technical people will approve the code-design and test-designs proposed by the outsourcing team.

Model 4: Complete technical outsourcing

Companies sometimes establish a mantra like ‘All technical work will be done in India’. Their process model looks like the following, where documented requirements are delivered to the outsourcing team, who interpret them, design and implement.

The key to success with this model is having a trusted partner, a great PRD, and extremely good communication in the validation of the interpretation of the specifications.

Conclusion

The best process will vary depending on the makeup of the organization, the goals of the business, and the nature of the project. There is no single right answer.

[Update] Follow-Up Articles

We’ve written some additional, in-depth articles specifically about how to make these different models work.  These articles focus on actionable strategies for making the models from this article work for your team.

• Making Offshore Development Work
• Making Offshore Design Work

Alan Cooper presents the analogy that software development is like making movies in his book, The Inmates are Running the Asylum. [This is a fantastic book for getting an understanding of exactly how Cooper's perspective evolved over the last decade.] Cooper is presenting the analogy in the context of validating the business case for investing in interaction design.

Cooper points out that they’ve been making movies for a lot longer than we’ve been making software, and he’s exactly right that there is something to learn from the film industry.

How the movie industry works

The movie industry manages movies in three phases:

• Pre-production. Determining what the movie will be about, raising funds, storyboarding and designing the movie, getting actors signed, writing the script, etc.
• Production. Shooting the film. Directors, actors, and crew all working very expensively to get the film shot.
• Post-production. Tweaking and finalizing the film.

How software development parallels movie making

Software development involves three phases as well: Decide what to do, do it, and deliver it.

The interesting thing to note is that the film industry universally invests time upfront in pre-production, to minimize the costs of production. They recognize that production is more expensive than pre or post-production. Many software teams take the same approach, although Agile development explicitly does not. We gleaned some insight into Cooper’s perspective from our coverage of a debate between Cooper and Kent Beck.

If we accept Cooper’s premise that production is more expensive than pre-production, then software should follow the same model.

It’s worth noting that an agile process results in more design, not less. Beck might argue that redesigning as we go is less expensive, because we improve our ability to understand what we actually want to create during the process of creating it. Cooper disagrees.

As much as we like Cooper’s insights, the movie cost structure is not paralleled in the software development structure. When we hire developers, it is analogous to the old movie studios keeping actors on retainer – the cost is “fixed.” And the infrastructure costs of production (set creation, for example) are not affected by the time spent in production – they too are fixed. If we have a project with contractor developers, then we have a variable cost, and we lose money while those developers are “sitting around.” However, today’s projects leverage outsourced overseas contractors more and more – and these actors are a lot cheaper than script writers.

What we know in spite of the analogy’s flaws

We absolutely save time and money by defining requirements before we write the software. We also know that it is important to design before we code.

Neither of these statements conflicts with agile philosophies, if we take the approach of treating “design everything” with “design this one thing” similarly. An agile approach will simply have multiple design/implement cycles, each focused on a subset of the software (and allowing for a redesign phase prior to delivery).

Stephen Larrison posted an article in his blog, Survive Outsourcing, about how to compete with offshore-outsourced projects by fixing the requirements mess up front. His background in manufacturing, as well as software development has lead him to a similar perspective as the one I touched briefly on in my post about where bugs come from.

Stephen points out that “You have to eliminate scrap and rework” in order to compete with teams that have lower cost implementation resources. He points out that this is the key to improving productivity in software development, just as it is in the manufacturing world.

I read a book a little over 10 years ago, The Machine that Changed the World, by Womack, Jones, and Roos. They first introduced me to the concept of the value of investments made upstream in the process. They specifically showed the payback of a dollar invested during the design of a new product (their focus was automotive) yielding the same results as ten dollars spent after a product enters manufacturing. It had the same effect as $100 spent fixing problems after they’ve been released to the field. IIRC, their point, in 1991, was that Japanese manufacturers were investing more at the design stage than U.S. Automakers. Seems almost prescient now with Toyota set to overtake GM as the largest car manufacturer in 2006.

There’s an analogous rule of thumb in software development that requirements bugs cost ten times as much to fix after they reach development, and 100 times as much to fix once they’ve been released to the field. A requirements bug caught in development seems cheap (you only have some wasted effort and minor delays associated with re-working the software to meet the corrected requirement), when you compare it to cost of releasing buggy software – which can result in lost sales and added costs (if the cost of “repair” is accounted for). But truly, the savings comes from fixing the bug before anyone starts building part of the solution based upon the incorrect requirement.

In Stephen’s article, he references CIO magazine, and an eye-opening statistic: “71 percent of software projects that fail do so because of poor requirements management”. It is seriously worth a read. Here are some more statistics on software project failures.
Scott