When companies first start off-shoring, they usually send the “low level” implementation work overseas first, to work out the process kinks and manage risk. Over time, your valued, domain-aware developers will perceive a lack of career opportunities with this limited role. Naturally, you will want to consider sending design work offshore too. You can make it work. If you do it wrong, you’re toast.

Different Models for Offshore Development

There are many ways to organize software-creation teams (where creation includes product management, design, development and testing). When developing an organizational design that incorporates elements of off-shoring, there are four primary models for outsourced software creation.

1. No offshore development at all. Occasionally referred to as insourcing, this is the traditional “everyone in one place” model – or at least “everyone in similar time zones.”
2. Low-level outsourcing. The implementation teams (both coding and testing) are located offshore, with design and product management staying local.
3. High-level outsourcing. The focus of this article. Keeping product management “local” but moving design and development responsibilities offshore.
4. Complete technical outsourcing. Everyone except the product manager is offshore.

The models can be most easily compared when the same process is compared in each – just with different locations. Co-location of team members has an impact when comparing face-to-face communication with online / phone / remote communication. But this factor is not a primary one in influencing team-decisions – it is no different than having someone who works from home. The key element in team dynamics is a dramatic shift in timezones.

This timezone shift causes a latency in the communication process, illustrated by the following example:

Imagine the following expensive question and answer session:

• Person A asks person B a question.
• 12 hours later, person B responds with a request for a clarification.
• 12 hours later, Person A clarifies the question.
• 12 hours later, Person B responds with an answer.
• 12 hours later, Person A acknowledges the answer.

When this communication channel happens between an onshore person and an offshore person, it takes 48 hours instead of 48 minutes. The more this happens, the more expensive it is to outsource. The key to making low-level outsourcing work cost effectively is to minimize the impact of this communication latency, while realizing the benefits of lower salaries in the offshore location.

Making Offshore Development Work

This is why proper communication design is the make-or-break element of making collaborative teams work when using an outsourcing model that involves anyone being offshore.

High-Level Outsourcing

Consider the following software development process diagram (from Four Application Development Outsourcing Models):

In this off-shoring process flow diagram, the shaded areas represent the activities that happen offshore. Note that this is the exact same process flow that is used to describe the other outsourcing models. The difference from other models is primarily where resources are located, but also the relevant scope of responsibility. In comparison with low-level outsourcing, the only difference in the process flow is that responsibility for test design and solution design is transitioned to the offshore development team.

Artificial Boundary

This model creates a bit of an artificial boundary between the “interpret requirement” step and the “design solution” and “design tests” steps. This artificial boundary creates a potentially odd dynamic within the team.

To make reading easier, we’ll talk about the development side of the process flow only, but the same ideas apply on the testing side.

This model has one developer interpreting requirements, and a different developer doing the design work. In an insourcing model, those two developers are usually the same person. In large development teams, a common breakout here is architect versus senior designer. The architect (the figure on top) would be responsible for those design considerations that span the enterprise and the senior designer would be responsible for those design considerations that affect a single application. Here’s a good background article by Scott Ambler on approaching enterprise architecture.

Why have an artificial boundary? Because this model is an emergent design.

1. A company starts with low-level outsourcing.
2. The offshore developers complain about a lack of growth opportunities (both career and skill).
3. Executives are not prepared to completely outsource all technical work (risk aversion), or the team is not ready to succeed with that approach.

Splitting the “interpret requirements” task from the “design a solution” task is a compromise. It minimizes the risk associated with providing growth opportunities to offshore team members. That risk is minimized by avoiding the high-latency communication channel (from onshore to offshore) when communicating requirements. Instead, the interpretation of the requirements is done onshore, and that interpretation is communicated.

You can argue that this model is the offspring of a trust issue within the organization. A company can absolutely say “We want growth opportunities for our offshore team members” and at the same time say “We are not ready to relinquish complete technical control.” This is definitely a trust issue, and therefore a perceived risk mitigation strategy. The risk may be very real, or non-existent. In either case, it is emotionally present.

Vague Scope and Role Conflict

If, while reading this, the notion of splitting interpretation from design feels uncomfortable, that’s because it is. For many teams that are evolving their outsourcing model, it feels uncomfortable, but it feels less uncomfortable than releasing complete technical control.

One problem, which I completely grok as a former developer, is that it is almost impossible to interpret requirements without imagining designs. But this model has different people performing the different tasks. So should the interpreter just discard those designs? Presumably the interpreter is more experienced, certainly more knowledgeable about the domain. It would be a shame to discard those design insights.

Robbing Peter to Pay Paul

This approach has clear benefits for the offshore developers who are now presented with growth opportunities. Unfortunately, you run the risk of sucking the fun out of the role of the interpreter – a senior, experienced developer with domain knowledge. While proper requirements interpretation can be fun, it usually is not fun for a developer. Only requirements people will enjoy those nuances, generally.

You risk eliminating a career path and growth opportunities for your onshore resources with this model.

Division of Labor

One approach that keeps the growth opportunities open for your senior onshore technical resources is to have them play multi-product, or architectural roles. This presents an opportunity for these individuals to apply organizational insights across products, finding synergies between applications and driving consistency and consolidation among applications. You essentially split the responsibilities “broad and deep” where the onshore designer is looking across the portfolio and the offshore designer has ownership responsibilities for a single application or scope. This is similar to the division of responsibilities that works well for tackling enterprise product management.

Instead of preventing communication of requirements across the high-latency channel, it minimizes it. The onshore designer acts as the liaison between multiple offshore designers, fielding interpretation questions, and more effectively, preventing them. Developers have a language all their own. Through a shared perspective on common experiences, they can communicate very efficiently – by analogy, symbolically, and via design patterns. These communication opportunities (between like-minded individuals) can have very high information density. For example, one developer can say “MVC pattern” to another, and that serves more effectively to communicate an approach to designing a solution (and a context / interpretation of requirements) far more efficiently than a product manager describing requirements that multiple tools and platforms should expose the same set of capabilities or behaviors [and that is overly short, because I don't feel like typing a comprehensive explanation of the MVC pattern].

Another approach is to silo the developers vertically – having some applications (or modules) following a low-level outsourcing model, and others practicing complete technical outsourcing. Any given team is operating discretely with a clearly defined set of responsibilities. The teams may just be operating differently. Essentially, you’re saying that your “average” is high level outsourcing, even though it is really a mix of two models. This doesn’t really count, since none of your teams are addressing the communication challenges of this model – your company is leapfrogging over it, but choosing to mitigate risk by doing it a little bit at a time.

You can also take the approach of having the onshore designer be responsible for over-arching and high-complexity design issues, with offshore designers owning more straightforward and lower risk design activities.

The best approach for maximizing your team’s effectiveness (and your HR goals) will be overwhelmingly determined by the individuals involved. If you’re in a large company, you probably don’t get to maximize team effectiveness – you are likely to be forced into a particular model. If you’re doing the forcing, recognize that one size does not fit all.

High Latency Communication And Designing

When circling back to the main challenge of offshoring – communication – you need to look at the nature of the communication that is subject to the high-latency of temporal displacement within the team. The key to making this model work is to leverage the efficiency of cross-talk between developers. That means having experienced people on both the offshore and onshore teams who share common design backgrounds (patterns, analogies, examples) and deep domain understanding (reducing the provision and clarification of context across the communication channel).

Design reviews are also very effective at eliminating the impact of this latency on communication. Design reviews typically happen as follows:

1. Designer A spends time creating design based on an interpretation of requirements.
2. Designers A & B get together and review the design in a relatively short meeting (or Designer B reviews a design document). Feedback is delivered in a burst.
3. Designer A spends time updating the design based on feedback from step 2.
4. Return to step 2 as many times as is needed.

There is a big chunk of work involved in incorporating feedback from a design review. This can be folded into the “white space” between communications. In other words, while designer B is sleeping for the night, designer A is making updates. This looks like, but is not the follow-the-sun pattern.

The follow-the-sun pattern is one where someone is always working. I’ve made this work on a project where there were two teams working 12 hour shifts with a 4 hour overlap (and a 4 hour “blackout”). Note – that isn’t sustainable, just anecdotal. I think the linked article is right, commonly, follow-the-sun implementation does not work, for the reasons cited in that article.

What makes this very different is that we are synchronizing the naturally-occurring time lag between design reviews with the geographically-induced time lag in communication.

Conclusion

The strategy for successfully utilizing offshore resources for both implementation and design starts and ends with communication. It also requires attention to your specific people and their career aspirations.

• Utilize design reviews to take advantage of serendipitous time-lags in the communication cycle.
• Assure that your role definitions are clear, and aligned with the aspirations of the people on your team.
• Follow the tips for effective offshore development – this strategy builds on that one, it does not replace it.

Measuring the return on investments in design may be the hardest ROI calculation you can do. It certainly is one of the rarest. To measure ROI, you have to be able to determine what would happen without the investment, and what happens with the investment. The difference between them is what happened because of the investment.

Fast Company Interview

Bill Breen posted an interview with Rob Wallace on the Fast Company blog about proving the value of design. The start of Bill’s article lets us know that measuring the ROI of design (ROD) is so rare that it may be impossible or impractical to do. He points to a Whirlpool survey of 15 design-focused companies – most of whom were “clueless” about the return on their design investments.

Rob explains that the reason for measuring the ROI of design investments is that ROI is the language of executives. “If you can’t measure it you can’t manage it. Businesspeople operate in a world of numbers.” Simply put, if you want to make someone invest in design, you have to show them why they should invest in design. One way to get a company to invest in design is to convince the benevolent dictator in charge of the product (or company) of the subjective merits of having good design. Apple is a great example of a company with a leader who is passionate about design. And you can argue that the Zune (from Microsoft) has had meager success against Apple’s iPod due to shortcomings in design. But can you objectively argue the point?

Argument by Extension

After Rob mentions that there is a dearth of measurement of design ROI, Bill challenges Rob’s premise – why should we expect that there is an ROI for design? Rob cites a series of studies done on Fortune 500 companies:

On average, based on two dozen case studies with Fortune 500 companies, for every dollar invested in advertising, packaging and promotion, and visual communication at the point of sale, companies realized a $7.21 ROI. But when the advertising didn’t change (or there was no advertising)—and packaging design was the only thing that did change—there was a $15.17 average ROI on every dollar invested.

Based on this data, Rob argues, it is easy to extend that if packaging (design) changes can yield ROI, then so to will product design changes.

Insights from Adaptive Path

Peter Merholz, President and founding partner of Adaptive Path wrote on communicating the value of design in Aug. 2002. In his article, based on a breakout session, the AIGA’s 5th Advance for Design Summit, he shares a series of internal benefits that they found to be more significant (or at least more measurable) than external benefits.

As we continued listing the value of our user experience design work, a more intriguing realization emerged — the bulk of our value comes from the efficiency that we can create in a company’s operations.

Some of the areas they identified are:

• Smarter Product Development Processes
• Lowered Maintenance Costs
• Less Internal Documentation
• Maximized IT Investments
• Scalability

Our Concerns

One challenge for each of these sources of ROI is that you can’t truly isolate the benefits – you would never have two teams design products for the same market, with differing levels of design investment, with a resultant analysis of project costs. Even if you did, you wouldn’t be able to isolate the impact that the team members had. You have to look at historical data, which introduces enough variables to question the validity of any quantitative conclusion.

Further, we will show that each of these is either a weak proposition, or potentially very false one.

Smarter Product Development Processes. No one has concretely identified a design-centric, purely agile, or structured requirements approach as being the best one. At least where best is defined as most profitable. Even the debate between Alan Cooper and Kent Beck highlighted more “stylistic differences” than quantified differences. There are elements of genius in each approach. Incorporating the needs of users is key to an effective design. Incremental delivery and incorporation of feedback is incredibly efficient. Tracing decisions, designs and requirements to the ultimate goals for the software is key to building the right software. If there is a “perfect process”, it is one that incorporates elements from all three disciplines. And as such – there is value to incorporating design into the process. We can’t quantify it, but we are convinced that it is there. And based on the economics of when software problems are fixed, we believe that investments in design early in the cycle will yield positive returns.

Lowered Maintenance Costs. There are a couple ways this benefit might be inferred. First, better design yields easier to maintain code. This is primarily a premise of agile development (where the design decisions are what Cooper calls program design), but perhaps the benefits also apply to interface design. Often, good interfaces simplify the complexity of, or at least obscure the means of implementation. Those interfaces are not necessarily easier to maintain. One reason complexity leaks through to users in a lot of software is that it is easier to code it that way. This value-prop probably doesn’t hold water.

Less Internal Documentation. The better the design of the interface, the less you need to teach someone how to use it. There is definitely potential for reduced external documentation, or producing the same amount of documentation may cost less, as it is easier to understand the product. It isn’t clear how internal documentation needs can be reduced by good design. Except perhaps that a picture is worth a thousand words. But a great picture may take as long to draw as it does to write a thousand words.

Maximized IT Investments. This is simply self-referential. Maximizing investments yields higher ROI. But in what way?

Scalability. Better design, from a user perspective does not imply that software will be more scalable. And often in an implementation, when user-tasks are simplified (by automating steps in the process), those decisions don’t disappear. By removing the burden from the users, you often add it to the software. However, Peter may have been talking about scalability of the team – the ability to leverage design investments across products (sharing branding, metaphors and workflow paradigms). In other words, design re-use. In that case, there is definitely a benefit, but re-usable design, while cheaper than green field design, is more expensive than no design.

Support Costs. One external benefit that Peter does identify is the reduced burden for training and support. Many software companies actually have profitable businesses built on support and training. Unfortunately, reducing the demand for those would reduce the profitability along that dimension.

Where We See Value

Fundamentally, we don’t perceive design investments as being significant cost reductions – we see them as significant revenue enhancers. Better design yields increased usage, more efficient usage, increased satisfaction, and increased word of mouth marketing.

Increased Usage. ROI forecasts are often built upon estimations (or suppositions) of levels of user adoption. As an example, consider the “Buy it now” button on eBay auctions. When people click on the “Buy it now” button, the auction is closed, and the transaction occurs at a fixed price. People are more likely to sell an auctioned item (generating a commission) by encouraging an impulse shopper. The seller risks getting less than they otherwise would, in exchange for the buyer knowing that they can get the item they want. Fewer auctions will expire unexecuted. More commissions are generated for eBay. Comparisons of final selling prices (and commissions to eBay) of equivalent items with and without the button can be done to measure the return.

More Efficient Usage. Better designs make it easier for users to do what they want to do. Easier often equates to faster. A call-center application that simplifies data access for the operator will allow them to process calls more quickly. The employer ends up with higher throughput from the call center. This throughput can be used to reduce costs, or increase customer satisfaction (with reduced “time per call” stats, or better experiences for the callers), or some of each. Comparing the results of operators using the new system versus the old system will yield measurable data.

Peter mentioned reduced training costs – and we see that as an external benefit that comes from accelerating a user’s growth from new user to proficient or even expert user. Bridging the canyon of pain to get to higher usage rates and more efficient usage faster. This also leads to increased satisfaction by creating more satisfied users more quickly.

Increased Satisfaction. Customer or user satisfaction surveys may be the only way to approximate (not really measuring) this benefit. And it isn’t clear that X% increase in satisfaction leads to Y% increase in sales. But it is likely to lead to increased sales, and may also lead to increases in sales of other products by the same company. Apple leverages this affect across its family of synergistic products. Perhaps a Bayesian analysis of survey results and sales stats could identify the strength of a correlation between satisfaction and sales. That would be expensive to do, but would lead to quantified estimates of the return.

Ultimately, you raise the utility of the software for the user. By increasing the value to the user, you increase their satisfaction, and thereby increase the likelihood that they will encourage other people to buy and use your products.

Increased Word of Mouth Marketing. When your customers become your sales people, you get more sales than you would have without those recommendations. Perhaps comparisons of growth curves for products with varying degrees of customer satisfaction (or varying degrees of “free publicity” – like reviews, blog articles, etc) would yield some predictive insight. Web 2.0 successes have been predominantly built on word-of-mouth. And their growth curves are exponential (as would be the propogation of good words from satisfied mouths). Does anyone know of analyses that validate this?

Conclusion

Design investments yield benefits. The way to measure those benefits is still up for debate. Some folks believe the benefits are primarily internal – we believe that the benefits are external

In our previous article in the series on the development of nexus, we discussed navigation and information architecture. We identified the challenge of filtering articles by category and by level of experience (beginner / expert), while also viewing the articles along a characteristic (most-viewed, highest-rated, etc). Between both url-creation and visible site-navigation, the challenge we explored was how to present one facet or dimension as primary and others as secondary.

One of our readers presented a third alternative – faceted navigation.

Scott Weisbrod commented:

Have you considered using faceted navigation? Faceted navigation means that level of expertise, category of article, and chosen approach can all live at the same level.

Therefore, as a user, I can pick the facet by which I want to begin navigating the site. For some, level of expertise is the most sensible place to start. Others will want to begin their search by category or even chosen approach.

Why not surface all of them and let your users pick the most appropriate starting place?

He also suggested looking at The Guardian’s search page. You get the idea by interacting with the controls in the sidebar – we’ll try and explain here with a static image, but go check it out.

The left sidebar shows a list of “active filters” (initially empty), and as you click to “add search filters” you narrow down the choices. Amazon.com also uses this approach, for narrowing by product category, price range, supplier, etc. While it may encourage an element of precedence, it allows users to filter choices in whatever sequence they desire. This is very cool presentation (in the two-dimensional sense).

When we started exploring our approaches, we were also concerned about how the URL looks, both for providing context to users and for sharing with others. Unlike the Guardian, this is the main navigation for our site – not just an admittedly cool way to do browse/search. If we look at a URL from their site, with facets selected, we see http://browse.guardian.co.uk/search/all/Politics/Labour+party? month=07&lDim=N%3D4294962719%2B4294962540%2B4294963588%2B3098 &year=2002&N=4294962719+4294962540. Not a very pretty picture [ed: spaces added to line-wrap the URL].  It exposes the implementation (looks like an OLAP cube database to me).  More importantly, it would confuse anyone trying to read it, and might even dissuade them from clicking on the link if it were shared in an email.

Nexus Implementation

Luckily for us, we did three things today. First, we decided that we agreed with Bob, at least enough to include the experience level dimension in our initial implementation (it added less than an hour of work). Second, we took an implementation approach that would actually support this faceted navigation presentation. Third, while we implemented it with a hierarchical presentation, we stuck to the Ruby on Rails convention of separating the presentation from the implementation.

This affords us the opportunity to refactor the presentation to look more “faceted” and less hierarchical.

Here’s what the navigation elements in the latest version in source control look like:

The top bar is clearly a hierarchical parent. It establishes context (articles or users, or your own page, etc). It also shows a “categories first” navigation bar. However, that navigation doesn’t “reset” the other choices.

If you’re looking at beginner product management articles, organized by their ratings, and you click on “Business Analysis” – you will see business analysis articles, for beginners, organized by highest rating. In effect, we have a faceted navigation implementation under the hood. We just presented it hierarchically. Note that we also agreed with Bob, and put experience level filtering “above” the organizational sorting (most read, etc) of the articles.

We critiqued The Guardian’s URL structure. We should look at ours. The URL for an example above would be http://tynerblain.com/nexus/business_analysis/beginner/highest_rated/. Incredibly easy to know exactly what to expect before clicking on that link. We can also extend this approach in a couple ways – we can add more skill levels or categories, we can add more sorting options (highest rated reviews, etc), and we can add subcategories (like testing:test-automation). We won’t add any subcategories until we have enough articles to warrant it. I think we can even add sub-categories selectively (have them for testing, but not business analysis, for example).

Design Conclusion

I am happy with the under-the-hood design, for both being extensible enough for likely future needs, and being adaptable enough to work with either a hierarchical or faceted presentation.

In order to get the alpha release launched as soon as possible, we will keep the “hierarchical-looking, faceted navigation” in place for the first limited release.

How You Can Help

In order to really get a feel for the effectiveness of the navigation, we have to have some critical mass of articles on the site – I would think at least 100. Please spend a few minutes to queue up some great articles to be reviewed – and when we release the alpha, submit those articles. Every page will include an “article count” in the sidebar – and when we get to 100, start evaluating the navigation and let us know how it works for you.

Thanks in advance!

Our agile project is to create a site that lets you rate articles. In our corporate goals, we defined the goal to make it easier for people to find and read great content. Last night I was doing some research on social networks and thinking about the nature of our ratings approach. In this article I share some of those thoughts, and the reason for changing the ratings approach relative to previous designs.

An interesting article on the history of social software (going back to the 1940’s) got me thinking about the timelessness of content, and the fact that there are different approaches to scoring or rating things.

Our Content Is Almost Timeless

Generally speaking, the topics we need to read about are “timeless” – at least their relevance can be measured in multiple years, not multiple minutes. An article can become obsolete, someone can build a “better” mousetrap, etc. But short of that, great articles will be great for a long time. The Mythical Man-Month is decades old, and still relevant. We need to make sure that our rating approach will keep “great” articles at the top of the heap, rather than bury them under the stack.

I was also looking at one of the real-time pages at digg. What an extremely cool thing to watch. However – one thing jumped out at me – the articles with the most immediacy of content were getting all the attention. A fifteen minutes of fame thing. If you look at the “most digged [Kevin and Alex say digged on their podcast, not dugg] of all time” – none of those articles were getting noticeable attention. They had become old news.

When there is a great resource, like any of Scott Ambler’s UML 2.0 pages, it needs to be easy to find for people using the site. Perhaps using a “rate it 1 to 5″ approach would be more effective.

Comparing Approaches

I created a couple bulletted lists of pros and cons of two general approaches – aggregating ratings and averaging ratings. Here’s what I came up with – feel free to augment or dispute in the discussion on this article.

Sum Of All Scores

An approach that sums all of the scores, so that the number of “votes” affects the score. For example, if 10 people voted “yes” for an article, it would have a score of 10. If 20 people voted, the score would rise to 20. This is the general approach used at digg.com.

• The score reflects the number of people who liked the article enough to vote on it.
• As the number of people using the site grows, newer articles will tend to get more votes and get more of the attention.
• The “runaway” article dynamic will happen – and the crowd will drive attention towards the most attention-getting articles.

Average Of All Scores

An approach that averages all of the scores for an article. For example, if 10 people rated an article an average of “3″, the article’s score would be 3. If 20 people rated it a 3, the score would be a 3. This is the general approach used at Amazon (and Tyner Blain) for rating books (and articles).

• The score obscures the number of people who liked the article enough to vote on it.
• Older “best” articles will be at the top and all articles will have to play “king of the hill” to see which become the top scoring articles. Tie-breakers can be based on the number of ratings for an article.
• Would need some minimum number of ratings for the score to have credibility. The community would probably self-police this by quickly voting on articles that have “biased” initial votes.
• As the community grows, obsolete articles will gain new (lower) ratings pushing them off the top of the heap naturally.

Conclusion

Based on this analysis, the “scoring” approach should be the same general approach used to rate articles at Tyner Blain and books at Amazon. I’ll update the domain model to reflect this change. I think a simple 1-5 scale (where 1 is bad and 5 is good) would work effectively.

Chloe had some great comments on the Use Case Briefs article discussion about people being able to learn from articles with negative reviews or low scores – as ideas to stay away from. That “unintented” use would also be better served with an averaging-score, if it were easy to display “low score, high number of ratings” articles.

Joshua Ledwell wrote a short article expressing his perspective on designing software that is neither too simple nor too complex. He also links to some excellent other articles on the topic.

Designing For Competent Users

We’ve written in the past about how to design for competent users. In fact, it was Joshua’s link to that article that helped us find his article – thanks! Users start out as novices, and most of them become competent users. Very few of them reach a level of expertise with your product. You need to make sure you support both their growth and their likely “end state” – competence. The feature set needs to support the fact that most users end up in a state of competence.

Maximizing ROI

One of several good links from Joshua’s article took us to Luke’s article on the sweet spot for selling software. Luke references an article from the Harvard Business Review and includes an interesting graph that shows the relationships between the number of features and profitability. In short, more features drive higher initial sales by satisfying the buying persona. Fewer features drive higher eventual sales through word-of-mouth marketing based on improved usability.

The HBR suggests a profit maximizing approach of seeking the middle ground in terms of feature quantity. Trying to please everyone tends to result in pleasing no one, so we don’t have a lot of confidence in this approach. But we still like the graph.

Some great thinking out there, what do y’all think in here?

The wisdom of crowds helps us avoid stupid decisions. Unfortunately, it also prevents innovative, passionate, fantastic decisions. Collective Intelligence is collective insipidness. We need to keep the inputs of individuals in the mix.

Collective Intelligence

Collective intelligence is the notion that a group of individuals will be smarter than a single individual. The old adage, two heads are better than one, is extended to mean several heads. Kathy Sierra goes on a bit of a rant about how this (narrowly applicable) idea has been twisted into the idea that mobs are better at designing stuff than individuals.

Design By Committee

Since we’re breaking out the old saws, we should acknowledge that people also have known for a long time that design by committee is a bad idea.

Eliciting and managing requirements requires communication, and prioritization. Both of these involve collaboration. When we collaborate, we risk introducing the negative effects of collective insipidity [If I were collaborating on this article with a group of other people, they wouldn't let me use the "word" insipidity].

Brainstorming involves collaboration in the requirements gathering phase of a project.

Prioritization also involves collaboration. We can prioritize the ideas from a brainstorming session, or we can prioritize requirements, or the sequence of use case delivery. Basically any process that involves filtering or sorting is a prioritization exercise.
Kathy concludes her passionate article with a great quote:

No matter what, I believe that in our quest to exploit the “We” in Web, we must not sacrifice the “I” in Internet.

So how do we avoid the dillutive effects of the crowd’s concensus?

Dictators Dodge Dillution

One way is to follow apple’s model – have a dictator. As long as the person calling the shots has good instincts, she’ll make good decisions. If she doesn’t, we’re toast. With this approach, we get both the downside (possible bad decisions) and the upside (possible great decisions).Another approach is to allow passion to stand out, when interpreting the inputs of the masses.

Weighted Prioritization

We wrote an article last September with tips to maximize the benefits of brainstorming.

The idea starts with having people vote on the ideas that were generated in a brainstorming session. This alone is a mathematical embodiment of the commonness of crowds. We end up with the lowest common denominator.

We then added the notion of “voting again with a 5X impact” on those items people feel passionate about. This allows the strongly held notions to compete with the universally not-unappealing ones. Read the article for more details.

Bypass Brainstorming

The guys at OK/Cancel are super effective at creating great content. They do it by using idea seeding instead of brainstorming.

In a nutshell, the two of them collaborate with an iterative process that involves creating ideas independently, transferring “ownership” of those ideas to each other, and then improving upon them. Really a novel concept – check it out.

Conclusion

Crowds can be leveraged to prevent mistakes. They can’t be leveraged to create greatness. Crowds serve to filter out the extremes on both ends. There is value in avoiding the mistakes, but the cost of avoiding greatness is too high. Whenever we bring in the crowds, we have to keep the individuals in the loop too.

[Updated Below] There are 15 ways for someone to shutdown a laptop running Windows Vista. This adds unwarranted complexity to our software. How can we avoid the same problem in our software?

Hat Tip

Joel (on Software) has an article where he lambastes the Vista team for providing 9 different ways to stop using the computer. Add in the hardware options on the laptop (like Fn F3), and closing the screen down or hitting the physical power button, and you get to 15 different ways to shut down.

Inevitably, you are going to think of a long list of intelligent, defensible reasons why each of these options is absolutely, positively essential. Don’t bother. I know. Each additional choice makes complete sense until you find yourself explaining to your uncle that he has to choose between 15 different ways to turn off a laptop.

Joel Spolsky

Make Everybody Nobody Happy

Joel posits that the unreasonable number has grown out of the desire to make everybody happy. We know that too many features makes everyone unhappy. This bloated list of choices is just one manifestation of too-many-features, and it doesn’t make anyone happy.

Inadvertantly Causing The Problem

It is pretty easy to imagine how we might inadvertantly cause this problem.

A structured requirements approach allows us to track and manage different requirements, as well as the implementation of those requirements. In Joel’s example, imagine requirements like the following:

• User secure the computer in order to walk away from it without other people getting access.
• User can shut down the computer such that it will stop consuming power.
• The computer will support rapid switching from one user to another.

Each requirement could drive an independent solution approach. As Joel points out, each requirement, evaluated in isolation, may lead to a different solution. The problem is that we are finding local optima, and not looking at the problems introduced by the complexity of supporting all of these solutions.

Avoiding This Problem

How can we avoid this sort of problem in our own product development? A holistic view of the proposed solution is required to assess if there are too many features. Prioritization of those features will help us reduce the number to a reasonable level.

Incorporating elements of interaction design into our solution approach may be the best way to do this (short of abandoning a structured requirements approach).

Each of the shutdown scenarios leads to a functional requirement. Each element of program design leads to one of the many ways to shut down or lock or restart the computer. Addressing the personal goals of each persona provides us with a cross-feature, holistic perspective. If our target audience includes Joel’s uncle, then we can make sure that our design approach accounts for the appropriate level of complexity and choice.

[Update]

Moishe Lettvin, one of the developers on the shutdown team for Vista, formerly at Microsoft, has written an article about the organizational and situational challenges faced by the team. He titled the article The Windows Shutdown Crapfest. That should give you a feel for how the now-Googler feels about the experience. There were 43 people involved, and they spent over a year. And the source code control system was frightening. Check out the comment thread on Moishe’s article too – it includes comments from other Microsoft and knowledgable anonymous people. Hat Tip to Scoble. Also, a followup from Moishe.
Summary

Multiple requirements can lead to multiple, locally-optimized solutions or features. A holistic view needs to be taken to assure that we aren’t introducing too much complexity with these variations of similar features. Interaction design gives us a big-picture perspective to make sure we aren’t making our software too hard for our target users to use.

Kathy Sierra writes (another) great article at Creating Passionate Users. This time, she talks about why users don’t upgrade and presents ways to get users to install the latest version. We focus in this article on one way in particular – using goal-driven documentation to encourage upgrading.

Background On Documentation

Over the last couple of days, we’ve written about how to structure documentation to define what users can do with the software, instead of capturing what the software can do for users. This reverse approach uses user goals to guide our documentation. We also presented, more concretely, how to trace and manage our goal-driven documentation with use cases.

We can leverage this documentation approach to achieve two additional benefits:

• Encourage users to install new upgrades
• Encourage users to migrate from other applications to ours

The same factor is at play in both situations – users already know how to use their existing software, either a previous version of our software or a competitors software. The fundamental question for users is “How will I do what I need to do?” This is a major barrier to both upgrades and migration from other applications.

Kathy On Upgrades

Kathy’s article on why users don’t upgrade is awesome. She creates a great visual graph (shared via her CC license) of the user’s perspective on upgrades:

Why they don’t upgrade (and what to do about it)

Kathy goes on to list several approaches to address the problem, all of them fall into either “make them want it” or “make it easier.” We will focus specifically on how writing goal-driven documentation can help.

Make Them Want It

We must apply the same prioritization approach to defining upgrades as we do to defining the initial release. This assures that the most valuable not-yet-implemented goals are addressed in the new version of the software. If we are using Kano as a framework for prioritization, we have the added benefit of effectively lowering the suck-threshold.

Make It Easier

We can make design decisions that make software easier to use. We can also follow Kathy’s other good suggestions. And we can write our documentation to make it easier for users to make the change.

Goal-driven documentation has two effects on the curve Kathy drew above. Here’s the same curve, representing a typical software upgrade:

With goal-driven documentation, the curve is shifted up and to the left:

This shift can be analyzed as two different factors:

The goal-driven documentation allows users to climb become more competent more quickly – effectively lowering the suck-threshold. When applied to upgrades, it reduces the size of the drop – preventing users from re-entering the suck-zone.

Faster Climb

Competence is not a measure of how quickly users can do everything the software can do – it is a measure of how quickly users can do everything they need to do with the software. When we define our software to prioritize solving the most valuable problems for our customers, we are defining the set of activities that the users need to do. And we identify this as a set of use cases.

We then document how users will follow those use cases, with the implementation we’ve given them.

This goal-driven approach to documentation helps users climb the competence curve much faster, immediately reaching the “Finally I can actually DO something” point (above the suck threshold). As long as we document the right somethings, we’re all set.

Shorter Drop

We apply this same approach to our upgrades. Those use cases that we’ve added in a new release yield new documentation. Those implementation areas we’ve modified as part of the upgrade yield updates to previously released docs.

Users will have an easy answer for “how do I do it now?” as well as “What can I do now?”

Collateral Benefit

We also get the benefit of making it much easier for users to migrate from other software packages (not just our earlier releases). There are books out there where authors address exactly this problem – Microsoft Word for Wordperfect Users and Java For Cobol Programmers come to mind. By showing users how to do what they need to do, we make it easy for them to switch.

Conclusion

• Goal driven documentation makes it easier for users to learn how to use our software.
• Goal driven documentation makes it easier for users to switch from other software programs to ours.
• Goal driven documentation makes it easier for users to upgrade to new versions of our software.

Yesterday we wrote about focusing our documentation on what our users are trying to accomplish. With a structured requirements approach, or with an interaction-design driven approach, we’ve already solved half the problem – determining what to document.

Our proposed documentation approach is to write about what users are doing with the software, not what the software can do for the users. Using a power drill as an example, we proposed the following representative approach:

1. How to drill a hole in a flat surface (wood or plastic, metal, masonry).
2. How to select the right screw for fastening items.
3. How to drive screws (phillips, flat-head) with your drill.
4. How to stir paint with your drill.
5. etc…

Suddenly, the documentation is helping us to achieve our goals.

Extending Structured Requirements

When we use a structured requirements approach, we have established a framework for defining the problems that our software will solve. We articulate how people solve those problems with use cases. These use cases define exactly what we need to document.

When we view structured requirements, they look like the following diagram:

From Non-Functional Requirements Equal Rights Amendment

We extend this structure by leveraging the fact that the use cases identify the highest priority tasks that users will perform to achieve goals. These are therefore the highest priority tasks to document. The documentation represents how a user would achieve a use case, with the implementation that we’ve created.

We can even trace each portion of our documentation work to the use case that it supports.

Extending Interaction Design

With interaction design, we are again focusing on what people intend to achieve while using our software. Even if we are writing game software (which is designed to be played without a “goal” in the traditional sense), the user still has a goal of “being entertained.”

With interaction design, we would write documentation that demonstrates the way that scenarios play out with the given implementation that we created.

Even when combining interaction design and structured requirements into a hybrid approach, we still have a source from which to define our documentation needs – the scenario.

Summary

We’ve previously defined the benefits of documenting what users do with our software instead of documenting what our software can do for users. In this article we present a framework for defining and tracing that documentation – building upon use cases or scenarios. This documentation approach makes for software that is easier to use, and therefore better.

(John Richards)

Stephen Turbek, at Boxes and Arrows, tells us how to get better results from our wireframes. Wireframe prototyping can provide feedback early in the design cycle, reducing costs and improving the quality of the final software. By putting a little flesh on the bone, we can get even better results.

Hooked From the Start

Stephen starts his article with the following quote…

How many times have you been asked, “So, is the new website going to be black and white too?” after presenting your wireframes to a client or a usability test subject?

A very solid opening to his position that when you only use wireframes, you introduce problems. Wireframes are designed to eliminate problems and “clutter” from the feedback session. Feedback sessions provide us with a lot of information, and the challenge is to separate the noise from the signal. The goal of wireframes is to eliminate sources of noise, to make it easier to focus on the signal. But using wireframes also introduces noise into the data.
He goes on to provide a real world example of a website under development for Verizon – showing wireframes and “low fidelity prototypes” that include more information than just the wireframes.

Why Wireframes?

Stephen makes an interesting point – wireframes (named after 3D modeling techniques) were initially designed to provide quick feedback and insight into 3D models, without the expense of complete rendering. As technology reduced the cost of of rendering, rendering cycles began to replace wireframes as early prototyping tools.

A wireframe, in the user interface world, is a minimalist visualization of a website or application. It shows where information will reside on a page, and what information will be shown. When using a wireframe to get feedback, it allows a designer to (attempt to) isolate the feedback about content and layout from other data (like feedback on color schemes and graphics). It also allows for more rapid prototyping because the prototype can be built as soon as a layout is done, without waiting for colors and graphics and branding to be incorporated into the design.

Why Not Wireframes?

Expanding on Stephen’s point, the lack of information detracts from an understanding of the usability of a design. Colors (such as blue hyperlinks) do provide visual cues for users. Branding and navigation elements provide a sense of context and comfort. The absence of these things can distract the people we were trying so hard to not distract.

What About Cost?

Another goal of using wireframes is to reduce the cost (and time equals money) of prototyping. Stephen shows how in less than 15 minutes, a wireframe prototype is converted into a prototype that leverages existing branding, navigation, colors, and images. 15 minutes is not a long time to spend to achieve this striking difference (check out the images in Stephen’s article). When showing multiple screens in a site with structured navigation, most of those investments will be re-used across multiple screens.

Why It Will Work

The ability to re-use the “extra bits” is the key to why it will work and not just the key to why it won’t cost more.

People who think about website advertising worry a lot about ad-blindness, or the ability of people to ignore ads over time. We actually depend on it here, to allow our regular readers to tune-out ads that appear in consistent locations and formats and instead focus on the content of the articles. This same phenomenon applies to these augmented wireframes.

The lack of context that a wireframe creates can be disconcerting or even confusing. The (valid) concern that drives the use of wireframes is that providing all the other stuff will distract the users and prevent them from providing feedback on content and layout.

The ad-blindness effect will quickly allow people to ignore the “other stuff” and focus on providing feedback on the content and layout of a page. People are good at scanning pages – they have an autopilot that takes them directly to the “meat” of the page. And the presence of the extra content allows them to do that – where the absence of that richness will immediately trigger a “what’s wrong?” or “what’s different?” analysis.

Tips to Making It Work

Stephen provides eight tips, which we really like. We are concerned about one of the tips – to use “real data” instead of fake data. I had always learned that real data risked distracting the users, who might fixate on the fact that you chose incorrect data to display.

Today, after a prototype review session, we got feedback from our reviewers that it would help them if we used real data.  The reviewers of our (fleshed out) wireframes were unable to visualize how the interface might behave with some real-world data examples. The data we were presenting and manipulating is fairly complex, and the contrived examples didn’t do a very good job of showing how the interface would handle the complexity it would need to support.

Using representative data is definitely a good idea – and as good as Stephen’s other ideas are, we’ll take his word for it that real data is even better.

Conclusion

Spend the incremental effort to make wireframes “feel real” by fleshing out some of the context that wraps the prototyped pages. That context will provide more comfort than distraction for users.

One of the ten big rules of writing a good MRD is writing requirements that do not specify design. How do we specify enough detail to be actionable without constraining the engineering team? How do we trust our developers to do the right thing?

The Big Rule of Avoiding Design-Agnosticism

From our previous article, Writing Good Requirements – Big Ten Rules:

Generally, a requirement should not specifiy any of the implementation choices. From a product manager’s perspective the requirement is the ‘what’ and the spec is the ‘how’. To a system designer or architect or lead developer, the requirement serves as a ‘why’.

Reviewing the Flow of Requirements from the Market to the PRD

Product managers begin by identifying a need in the market. The need can either be a problem that needs solving, or an opportunity that awaits a solution. Pragmatic Marketing teaches (and we agree) that people will pay more to solve problems than they will to address opportunities. Our previous article on innovation touches on the demands to reduce costs (address pain) that overshadow the attempts to seize new markets (opportunities).

Once a problem has been identified in the market, the product manager will capture it in an MRD. The MRD provides context for the entire project, and may include a vision statement and a product roadmap as well. This vision is built upon the identified market needs.

A PRD identifies those market problems that are to be solved within the scope of a product. The transition from MRD to PRD is an ideation process. The product manager must determine which problems are worth solving (high enough ROI) and which problems should be solved earlier or later. The PRD captures a definition of the problems to be solved with this product.

The software requirements specification, or SRS, is created from the PRD. This article is focused on the MRD, so covering the flow through the PRD is where we will stop.

To Act or Not To Act

Alas poor Product Manager. [Ed: silly idea]
Actionable Requirements

The MRD identifies the market problems with enough information for someone to ask “How should we solve this?” The creation of the PRD is the step where we say “We solve this one with our software, but not that one. And do this one first.”

For an MRD to be actionable it needs to provide insight into the problem, without assuming a particular solution approach. Software solutions are specified in the PRD. Here are a couple examples.

Bad

• Our premium skateboards are losing market share.

Good

• Our premium skateboards are losing market share to Tony Hawk’s new high-end board. We advertise in the same places, and have similar prices. Tony introduces a new limited edition board every month, which quickly sells out. His boards stay on the cutting edge of truck and bearing design, but his boards have half the life in tests that ours do.

Why is Good Good?

With the comparative information, we know that our competitor has differentiated technology, and we have better quality. This market research data (not opinion) allows us to make decisions about how we want to characterize and address the solution. We have not specified a solution.

Bad

• We need to update our social-networking website. Furl allows people to give ratings not just of linked pages, but also of other people. Furl is doubling in traffic every quarter while we have no growth. We need to add people-rating to meet our goal of doubling traffic every six months.

Good

• Our social-networking site is not growing as fast as we want. Furl is doubling in traffic every quarter. They allow people to ratings to other people, not just to linked pages like our site. We need to at least double our traffic every six months.

Why is Bad Bad?

There is an implicit assumption in the bad example – that adding people-rating capabilities will improve our traffic. The MRD should not include solutions, only identification of the problems. We might decide that we think people-rating is irrelevant, and that we will solve this problem with a marketing program. The research data is important, but the conclusions should not be drawn in the MRD.

Conclusion

Don’t draw conclusions in the MRD. Provide the data only. Conclusions represent design.

Interaction design creates a focus on (some of) the users

Interaction design, as described by Alan Cooper in The Inmates are Running the Asylum, is a process for designing software by focusing on the most important users. Unlike traditional requirements gathering and solution design processes, interaction design focuses on the goals of a specific class of users, represented as a persona. Those goals are considered when defining scenarios that represent how the primary persona will use the software. The combination of goals and scenarios leads to design artifacts and a functional specification. We will explore these steps in more detail in this post.

Persona selection

There are typically many users for an application. A persona is an imaginary user, who is representative of a class of users with common goals and roles. This is completely different from the user proxy (or surrogate) who we know to avoid when gathering requirements in Wiegers-esque fashion. The first thing an interaction designer does is create a detailed persona for each class of user. Users who share goals and roles will be represented by a common persona. The interaction designer will often include a picture that represents the persona – our picture for this post might represent Steve Marshall, 20-year veteran enterprise software salesman.

Goals for a persona are defined in two distinct areas – practical goals and personal goals. Practical goals are goals related to the role of the individual (meet sales quota, close at least one multi-million dollar deal each quarter). Personal goals are the tricky ones. Tricky for programmers anyway, because they appear to be completely irrelevant. Personal goals include things like don’t feel stupid, play 18 holes of golf every week, travel less, etc. However irrelevant they may seem, they are key to gaining insight to how the particular persona would interact with the software – expectations, prejudgements, tolerance level for bugs or confusing interfaces.

Once all the persona are defined, one or two primary persona are identified as the key users of the application. All requirements and feature development are done specifically for the primary persona. Other user’s needs are ignored. Programmers (homo-logicus, as Cooper calls them) will argue that “someone” will want feature X, and work to include it in the scope. If the persona doesn’t need it, the software doesn’t get it. Capabilities that optimize on selling the software, versus using the software, are also ignored.
Selection of the primary persona is a prioritization exercise. The persona with the combination of most-valuable corporate goals and most-frequent use of the system is the logical choice. There can be two primary personas – for example, the main person who enters information into a system and the main person who consumes the information from the system. Including any secondary or tertiary personas will result in dillution of the power of the software. The theory is that trying to be all things to all people results in sucky, bloated software that is difficult for all of the users to master.

Scenario definition

Scenarios describe what our primary persona needs to accomplish while using the software. The interaction designer first identifies all of the things that the persona needs to accomplish with the software. Each of those activities represents a scenario. The scenarios are categorized as daily use, necessary use, and edge case scenarios.

• The edge cases are immediately discarded by the interaction designer – these happen so infrequently that the penalty of enabling them outweighs the benefit. The penalties are not just measured in implementation costs, but costs to the user of having to understand, ignore, or otherwise work around the added capabilities needed to support the edge cases.
• Necessary use scenarios are scenarios that must happen, but happen very infrequently. The requirements specification will include the functional capabilities needed to support these scenarios. Interaction design effort will not be spent on optimizing these scenarios, however. The theory is that since they are so infrequent, users will tolerate whatever implementation exists to make them happen.
• Daily use scenarios are the ones that the primary persona performs most often. Most users will have one or two daily-use scenarios. More than three is rare according to Cooper. This is where new users must get up to speed the fastest, benefits from good design are reaped the most, and the lion’s share of interaction design effort is applied.

Designing the solution
Cooper doesn’t talk much about creating the requirements to support the daily use scenarios – he proposes moving directly into design of the solution. This differs from the more traditional technique of writing functional requirements to support use cases. Cooper also breaks down design into two components – program design and interaction design. Program design is everything you don’t see. Interaction design is everything you do see.

The interactions are designed to support the scenarios (which are built in the context of the practical goals). The design decisions are made to best support the persona in the context of his personal goals. This is the point in the post where design professionals are cheering and programmers are jeering. Programmers want to know how in the world knowledge of a persona’s hypothetical desires (play more golf) influence individual decisions about how to render a user interface or design an interaction. Draftsmen want to know what’s so special about Picaso’s paintings – the perspective lines are all messed up. This is simply a left-brain versus right-brain issue.

Usability helps to bridge the chasm between these two camps, and it definitely plays a role. Programmers can appreciate that testing, statistics, analysis and experience can drive decisions about how people should use a computer to achieve a (practical) goal. But the usability specialists tend to fall into one of two camps – data-driven, or insight-driven. [Ed: I apologize, I saw this idea presented on one of the blogs I read, but can't find it anywhere] This leaves the gap unfilled.
Cooper argues that designing the interaction should happen before any code is written. He uses a construction analogy to drive home his perspective – you can’t pour the concrete before you build the forms. Kent Beck, founder of the XP programming philosophy disagrees with the premise. Beck believes that the cost of changing software is low, and the imagery Cooper uses is hyperbole. We touch on, and link to that debate in this post.

Conclusion

There are some fantastic ideas in this approach. The prioritization of the most important users, and their associated daily use scenarios is very much like the requirements prioritization techniques we talk about here and here. The focus on minimizing features to maximize the usefulness of the software resonates as well. The lack of discussion of creating a software requirements specification or functional specification is concerning.

The jury is still out on this issue for us:

Should this process supplant or augment more traditional requirements gathering processes? Is it better or worse? What do you think?

Alan Cooper presents the analogy that software development is like making movies in his book, The Inmates are Running the Asylum. [This is a fantastic book for getting an understanding of exactly how Cooper's perspective evolved over the last decade.] Cooper is presenting the analogy in the context of validating the business case for investing in interaction design.

Cooper points out that they’ve been making movies for a lot longer than we’ve been making software, and he’s exactly right that there is something to learn from the film industry.

How the movie industry works

The movie industry manages movies in three phases:

• Pre-production. Determining what the movie will be about, raising funds, storyboarding and designing the movie, getting actors signed, writing the script, etc.
• Production. Shooting the film. Directors, actors, and crew all working very expensively to get the film shot.
• Post-production. Tweaking and finalizing the film.

How software development parallels movie making

Software development involves three phases as well: Decide what to do, do it, and deliver it.

The interesting thing to note is that the film industry universally invests time upfront in pre-production, to minimize the costs of production. They recognize that production is more expensive than pre or post-production. Many software teams take the same approach, although Agile development explicitly does not. We gleaned some insight into Cooper’s perspective from our coverage of a debate between Cooper and Kent Beck.

If we accept Cooper’s premise that production is more expensive than pre-production, then software should follow the same model.

It’s worth noting that an agile process results in more design, not less. Beck might argue that redesigning as we go is less expensive, because we improve our ability to understand what we actually want to create during the process of creating it. Cooper disagrees.

As much as we like Cooper’s insights, the movie cost structure is not paralleled in the software development structure. When we hire developers, it is analogous to the old movie studios keeping actors on retainer – the cost is “fixed.” And the infrastructure costs of production (set creation, for example) are not affected by the time spent in production – they too are fixed. If we have a project with contractor developers, then we have a variable cost, and we lose money while those developers are “sitting around.” However, today’s projects leverage outsourced overseas contractors more and more – and these actors are a lot cheaper than script writers.

What we know in spite of the analogy’s flaws

We absolutely save time and money by defining requirements before we write the software. We also know that it is important to design before we code.

Neither of these statements conflicts with agile philosophies, if we take the approach of treating “design everything” with “design this one thing” similarly. An agile approach will simply have multiple design/implement cycles, each focused on a subset of the software (and allowing for a redesign phase prior to delivery).

Organizing a test suite with tags (part 3)

This is the third in a three-part post about using tags as a means to organize an automated unit test suite.

Part 3 of this post can be read as a standalone article. If it were, it would be titled Design elements of an automated unit test framework using tags. If you’re only reading this post and not parts 1 and 2, pretend that this is the title.

• In part one of this post we developed an understanding of tagging as a technology, including its pros and cons.
• In part two of this post we defined the top five opportunities and problems inherent in the organization of automated test suites.
• In part three of this post we will consider the key design associated with using tags as a mechanism for organization.

Setting expectations

We designed a custom unit test automation system based on the use of tags to organize automated tests for a client. That system isn’t the subject of this post, but it did provide us with context and insight into the problem we’ve addressed. In this post we won’t be presenting a completed design – there are many good tools out there already for automating unit tests. We will be talking about a subset of the design decisions that are associated with the use of tags as a mechanism to organize the unit testing within the suite. In Marc Clifton’s advanced unit testing articles, he walks readers through the creation of a test automation tool for C#. The concepts presented here can be incorporated into a design relatively easily, but the details of doing that are a little too off-topic for most of our readers.

We are writing about designing software that tests other software. To keep our language consistent and easy to follow in this post, we will use two terms. Tool represents the test automation software that we are designing. Application represents software being tested with the tool, or more specifically, with tests maintained within the tool.

Design approach

After identifying the use cases and functional requirements for the tool, we began iterating on screen designs, business (requirement) object models and architectural (implementation) object models. We created an object oriented analysis (OOA) diagram to represent the concept concisely.
OOA diagram

An object oriented analysis diagram of the key relationships between scripts, inspections and tags.
A script is the embodiment of a user session in the application – it represents a set of actions that a user of the application would take. The user of the tool will create a script (as a separate file), and will create a reference to that script in the tool.

An inspection is an unit test of the application. The inspection evaluates a particular condition or makes a specific assertion about the state of the application (a properly filled out order is submitted when the user clicks “submit”) . The code that executes that inspection is represented outside of the tool. The user of the tool will create a reference to the inspection within the tool.

Inspections can be associated explicitly with any number of scripts (including none). An association between inspection 1 and script A is an instruction to the tool to run script A within the application, and evaluate inspection 1 against the script.

The processing of scripts and inspections is outside the scope of this document, but is covered in many other references, including Marc Clifton’s.

Any number of tags can be associated with each script. Tags could be used to represent different user actions (like deleting items from a shopping cart), different specific selections (user adds 1000 of an item to the shopping cart), different situations (shipping address does not match billing address), or any other relevant descriptor of the user session. A single script could have multiple tags.

Each inspection can be transitively associated with a set of scripts by explicitly associating it with one or more tags. By associating an inspection with a tag, we are instructing the tool to dynamically associate the inspection with all scripts that are associated with all of the identified tags. There are two benefits to this approach. First, this approach reduces the amount of time that a user of the tool must spend to associate an inspection with a set of relevant existing scripts. Second, this indirect mapping is utilized to assure that an inspection that is mapped to a tag will also automatically become associated with any future scripts that are added – as long as they have associations with the same tag or tags as the script. This reduces the cost of creating future mappings when scripts are added to the suite in the future.

We expect this design approach to provide significant labor savings in maintaining a test suite. We built our business case for this project upon that assumption. We also expect that this design approach will result in better testing coverage of the application by users of the tool. We did not incorporate that expectation into our cost benefit analysis when calculating the ROI of this project.

Followup

We will follow-up some months from now when we can evaluate data and draw conclusions from use of a tool built along similar lines for a client. Until then, we have confidence that it will work very well, but no tangible data.

Summary

• In part one of this post we developed an understanding of tagging as a technology, including its pros and cons.
• In part two of this post we defined the top five opportunities and problems inherent in the organization of automated test suites.
• In part three of this post we considered the key design elements associated with using tags as a mechanism for organization.

- – -Check out the index of software testing series posts for more articles.

Describing the software development process

Software development involves determining what to develop, documenting this decision, determining how to develop it, and actually developing it.

• How do we separate these very different processes so that we can talk about them?
• How do we staff a team to accomplish them?

We present a framework for describing this process in terms of layers of activity. Many people use pyramid analogies, which show the magnitude of effort in each layer (lines of code versus lines of requirements, for example). Many other people use inverted pyramids to reflect the importance (or impact) of work done at different layers (a sentance defining a strategy has more impact than a line of code). Some people show PERT diagrams of waterfalls or pretty circular arrows charts showing iterative lifecycles, or any of many good analogies.

We think of it as an onion.

When describing any software, we can discuss the software at any of multiple levels of abstraction. The heart of the onion is a secret to those who never cut onions – they only see the outside. The center is strong, but if you aren’t used to it it can make you cry.
Different people in an organization will discuss a software solution at the different layers. It isn’t because they are unable to discuss at a different level, it is because the level they discuss it at is most germaine to the decisions they are making. A developer will make decisions that are much more focused than a CIO. And a product manager will apply policy decisions at a different level from either of the others.

We will describe out framework in terms of different layers of abstraction. We’ll start at the inner level, or smelliest part of the onion and work our way out to the skin. We’ll move from the concrete to the abstract. From the implementation details to the goals.
The layers of abstraction

1. Implementation. The actual code that is written and running. The user interface that is presented. The tests that are included in a regression suite. We do what we’re asked, in the way that we’re asked.
2. Design. The architectural description of the implementation, UI and test suite. Design decisions define much of the reality for developers – good design decisions make implementation and maintenance easy. Bad design decisions can make for disaster.
3. Requirements. Different processes will use different names for these. In a process that uses structured requirements, this is the functional requirements, user requirements and business requirements. These can be captured in an FRS, SRS, or PRD (but please no, not all three!).
4. Market analysis. The problems or opportunities that express potential ROI opportunities. These can be captured in an MRD.

How do organizations handle this?

Most software is driven from the top down (4,3,2,1) and it can be like peeling an onion. First identify a value prop. Then figure out how to achieve that value in software. Create a design that will work, and implement it. Each step is peeling another layer from the onion.

There are exceptions to this rule – many startups have a “this is cool, what can we use it for?” mindset. Also, more mature businesses will often task someone with leveraging existing assets – which means “find out how to make more money from what we already have.” There are absolutely good examples of this both in software and in hardware. But that’s really not what software processes target.

Different people in the organizations have different responsibilities – they tend to operate exclusively at one layer of our framework or at most two layers.

Wearing one hat or two?
Looking at common roles within companies or on projects, we often find people doing two of the jobs in our four-layer framework.

• Design/Implement is a common pairing (combining 2 & 1). Architect/developer is a practical and common senior position. The skills required to design and implement are complimentary – one is expressly a superset of the other.
• Market/Spec is another common one (combining 4 & 3). A product manager often plays both roles (outbound and inbound are the industry jargon terms for “focus on the market” and “focus on the product”). There are synergies between understanding what problems need to be solved, and identifying what problems need to be solved with software.

A bad idea is combining the roles of “decide what to do” and “decide how to do it”.

• Spec/Design is rarely done intentionally. We’ve done this once, and succeeded only because we rigorously separated the two roles. Technically, I played a Spec/Design/Implementation role, while managing other developers and building a relationship with the clients. We do not encourage staffing people with this combination of responsibilities. It is no more natural as juggling while swimming.

Final thoughts

There are four roles in our framework, from identifying valuable opportunities to writing great code. It is a good idea to combine design and implementation responsibilities in a senior person on the development team. We would have avoided our requirement writing mistake if we had done that on our previous project. Combining inbound and outbound product management responsibilities makes sense for smaller teams and exceptional individuals.

Asking the same person (or the same document) to define both what software should do and how the software should do it is a bad combination for any but the most nimble switch-hitter. Shakespeare would qualify as a nimble switch-hitter.

[Update: We've created a few followup posts that talk about these topics in more detail

• Foundation series: User experience disciplines
• Software development process example
• Requirements versus design - which is which and why
• Software requirements - process and roles

]

Five easy steps to alienating your users with bad usability

1. Fail to simplify a comprehensive interface so that new users can quickly climb past the suck threshold.
2. Build an inconsistent UI layout or interaction design that varies throughout the application, creating a sense of dissonance for the users.
3. Interrupt the user’s workflow with pop-ups and other modal interruptions.
4. Limit expert users to following “new user” workflow, one tedius, repetitive step at a time when shortcuts would work.
5. Don’t suggest solutions when an error message is displayed.

Five difficult steps to making your users fall in love with your application

1. Plan for new users. Incorporate training, documentation, guides (or wizards), and feedback (breadcrumbs, confirmations) into your design. All of these “new user” solutions can be ignored once the users become experts.
2. Plan a consistent UI. Share a common stylesheet across web pages. Keep screenshots of the application taped to the wall where UI designers and developers can quickly assure commonality. Reuse interace code – derive from (or reference) common search widgets, sortable grids, etc. Make sure part of design reviews explicitly ask “is this the same as what’s already there” or “are we updating the existing stuff to match this?”
3. Provide passive warnings. Provide feedback in the status bar or page header when something “might not be right”. Add “Don’t ask me again” checkboxes to the “Are you sure you want to do this?” interuptions – let each user decide when they know what they’re doing and don’t want the interruption. Add undo capabilities for when users mess up.
4. See #1’s solution approach.
5. Add a “Fix this for me” button whenever a modal message must be displayed. At a minimum, offer targeted advice on how to fix the problem manually – right in the dialog. Add logging of all modal interruptions, so that developers can design new automated fixes (or prevent errors entirely) in future releases.