The first step to comparing products is understanding your customers.  This may seem counter-intuitive, but your product’s capabilities are meaningless unless you are comparing them from your customer’s point of view.  This article is part 2 in a series on comparing products.  Check out part 1, then continue with this article on the first steps of comparing products.

Overall Product Comparison Process

This is may be a long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers. (This article)
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

From that point of view, you can determine where to invest in your product.  That’s the whole point of doing the comparison – not to be self-congratulatory, but to guide forward thinking.  I’ve stopped being excited when I find existing “product comparisons” and game plans when starting to work with a product team.  To date, they have always been “positioning papers” designed to emphasize the product’s strength – usually prepared for internal sales teams, but unfortunately, sometimes prepared for executives (‘Look!  We’ve done a great job!”) Those documents are a useful source of information to get you started down the product comparison path, but if you let them guide you, they will take you down the primrose path.

Identify Your Customers

You want to have a model that represents how you are approaching your market. As a product manager focused on solving customer problems, you should have a customer-centric market model.

Your goal is to identify groups of customers that share sets of problems, and share a point of view about how important it is to solve those problems.  Effectively, this is a definition of a persona (yes, I know there are better ways to define personas).

The important thing is that you identify groups of people that will reach the same conclusion about your product, by answering the following questions similarly -

• Which problems that are important to me can I solve with this product?
• How well does this product solve those problems?
• How does this product compare to other products?

Personas, Goals, and Contexts

The image above is adapted from Robert W. Bailey’s diagram from Human Performance Engineering, (although I first saw it in the Handbook of Usability Testing).  One use of Bailey’s diagram is to understand that to assess the usability of a product, you have to take into account the human being who is using the product, the environment in which they are using the product, and the actions the person is trying to perform.

My extensions of the idea are that

• Groups of people with similar sensibilities (personas) will form similar conclusions about how much they like a product when,
• They are using the product in similar contexts,
• To achieve similar goals

One benefit of abstracting this way is that you don’t get caught up in the mechanics or procedures of how someone is solving a problem (the actions).  Focus on the action is appropriate when evaluating the usability of a particular product, but not when comparing how different products solve the same problem.  I also like something that reminds me that people use products in different environments and situations.

Example Personas

Usually, when I see personas they are focused purely on marketing and demographic data, or they are based solely on contextual inquiry intended to inform a design aesthetic.  The former may tell you where to market a product to groups of people (magazines, tv ads, luxury travel websites, etc), and the latter informs how to implement solutions.  What we need is something that helps us assess the relative importance of solutions to problems shared by groups of people.

This series of articles is looking at the Kindle Fire as a proxy product by which I will explain the techniques (without actually performing a data-driven analysis – I’ll invent data for the purpose of this article series).

I found a presentation on Slideshare:

BlackBerry Consumer Preferences and Segmentation

Where some MBA students documented the research and analysis of the market of prosumers – “professional” consumers.

What I particularly like is the way they presented a summary of part of their analysis on slide 27.  That’s the part we’re going to pretend applies to the Kindle Fire (because it might just).  Note that their presentation includes a “RIM Confidential” clause in the footer, so it might have been removed by the time you read this article.  I’ll briefly describe what they did.  They did an analysis of survey results, identified groups of people that placed similar importance on different activities and aspects of smartphone usage, and identified three “clusters” of people with statistically significant shared sensibilities.

1. Hi-Tech Prosumers – people who placed high value on most features, anticipated high usage of most features, were price-sensitive, had lower incomes (less than $75K USD per year) and were the younger members of the target demographic.
2. Typical Blackberry Prosumers - people who placed high value on network features, moderate value on other features, made heavy use of “standard” capabilities, but did not plan to use apps, or multimedia.
3. Basic Prosumer - people who placed high value on phone features, were heavy users of text and voice features;  had low use of other capabilities, but anticipated becoming heavy users of “smartphone capabilities” (email, calendar, browsing, apps, camera, etc) when they purchased their next phone.

What’s notable is that they did the right type of analysis – finding clusters of people that can serve to quantify market value of different personas – this will inform the determination of the relative importance of different groups of customers (step 3 in the process of comparing products).  By interviewing individual survey respondents that are in each of the clusters, they can get the additional information they need to develop personas.

For this series of articles, I will pretend that three personas were developed, in order to demonstrate this approach to comparing products:

1. Hi-Tech Prosumers – people who get excited about convergence, highly value additional multimedia capabilities, live a multi-device existence, and are acutely aware of competitive products.
2. Typical Amazon Kindle Users – people who placed high value on the reading experience, the convenience of being able to easily get new content, and the absence of a subscription fee.
3. Basic Consumers - people who read primarily on paper media, have recently started consuming multimedia on their computers, and anticipate becoming part of the “post pc era.”

I will also invent additional data as we explore the rest of the product comparison process.

Developing Personas

Please don’t finish this article believing that persona creation is trivial – it isn’t.  I’ve heard the saying before that a bad persona is better than no persona at all, and I believe that.  Without a target persona, you will almost certainly create the elastic user anti-pattern.

Persona creation is more of a precondition of this process than a part of this process – you should already have personas developed for your product.  This article is not intended to go into the details of how to create a persona, but rather a review of the aspects of that persona development that are needed for the product comparison process to be effective:

• Identification of people in your market that share perspectives on problems – e.g. knowing that for one group of potential customers solving problem X is important and problem Y is not, even when problem Y’s solution is important to another group of customers.
• Being able to estimate the relative number of people that are represented by each persona – e.g. knowing how much revenue potential exists for sales to each group of people.

Summary

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers. (This article)
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Attributions

* Thanks Alaska Dude for the Lovely Lass photo.

Recently, the gadget-reviewer crowd has caught on to something we’ve known for a long time.  Comparing products is not about comparing specs, it is about comparing how well the products solve problems that customers will pay to solve.  That begs the question – how should you compare products?  Read on to see the product comparison technique I recommend.

Inspiration

Writing about how to compare products has been on my backlog for about two years, as a key component of how to perform competitive analysis.  This topic is easily a chapter-length discussion, perhaps that’s what delayed writing an article-length discussion.  A flurry of recent articles, The Death of the Spec, Do Device Specs Really Matter Anymore, and Device Specs, from Techcrunch, iDownloadBlog, and Daring Fireball, respectively, all discussed how the Amazon Kindle Fire will likely succeed, in spite of not having particularly good specs (specifications).  The Techcrunch article also opines that Consumer Reports, because of its fixation on specs, has made itself irrelevant as a company providing advice to consumers.

John Gruber hits closest to the mark in the Daring Fireball article where he says

Specs are something the device makers worry about insofar as how they affect the experience of using the device. Just like how focal length and lens aperture are something the cinematographer worries about insofar as how they affect what the viewer will see on screen.
John Gruber

Combine this with a conversation I had last night after recording the latest Start With the Customer Podcast, about writing more frequent articles on Tyner Blain.

Product Management and Specs

The discussions are mostly centered on the utility of specifications, speeds and feeds, in informing a customer’s buying decision.  As Pragmatic Marketing espouses, we should be market-driven, with a focus on solving the problems that customers will pay to solve.  A screen resolution does not solve a problem, but an easy-to-read screen does – it prevents eye-strain and makes a long-session reading experience (like you would have when reading a book, versus reading this article) better.  Avoiding eye-strain is a problem people are willing to pay to solve – we’ve seen that with the success of products that use e-Ink technology.

Let’s look at how this example impacts what you do as a product manager.

You want your team to create a product that avoids eye-strain.  But you also know that you need to write requirements that are unambiguous and measurable.  We know from research that higher resolution displays (to a point) delay eye fatigue.  We also know, from Apple’s successful marketing, that promotion of a retina display (326 dpi, for a phone) is effective, at least with buyer personas, at addressing the perceived problem as well.

A “normal” consumer is not going to be able to make an informed comparison of the likely difference in eye strain over time – the problem the consumer actually cares about – when looking at the specs for a 225 dpi device and a 250 dpi device.  The authors of the articles are exactly right about that – but we, as product managers, already know this.

The problem comes when writing the requirements for your product team – do you just say “create a low eye-strain screen” and trust the team to pick a resolution that is effective? In an ideal world, yes, you would.  Someone (that someone may have to be you) on your team would do the research and come back and tell you that a 225 dpi interface will cause moderate eye-strain in 80% of people after 12 hours of continuous reading (but only 20% of people after 4 hours), and that that number drops to 20% of people experiencing eye strain after 12 hours when using a 250 dpi screen**.  You have an understanding of the value of a 250 dpi resolution over a 225 dpi resolution – an additional 8 hours of reading time for the majority of your customers.  Your customers will not know this (unless you tell them).  But you know it, and that’s enough.

Now you have to understand the incremental cost of creating a product with a 250 dpi resolution versus one with a 225 dpi resolution.  In this example, the incremental device cost is $25 per device for the next 12 months (given projected manufacturing levels).  At your target margins, this would reflect in an increase of $40 in device pricing to your customers.

This higher-capability, higher-priced product will simultaneously appeal to more customers (people who read for more than 4 hours at a time), and fewer customers (people who are price sensitive).  Your hypothesis (backed with market-research) indicates that you will generate 50% more profit by offering the lower-capability resolution (225 dpi), because most of the people in your target market do not regularly read for 12 hours at a time – and those that do will not “blame” your product for their eyestrain – they will “blame” their own behavior.

Now you’re ready to to specify that your product will be built with a 225 dpi resolution interface.  Not because a 225 dpi interface is in any way intrinsically valuable, but because it is the measurable and unambiguous requirement needed to satisfy your goal (achieving profitability) based on a solution to a more is better (see more articles on Kano analysis) market problem (eye strain that occurs in long reading sessions).

Specifications are useful, because they help you characterize capabilities.  The diagram above depicts a more is better characteristic, as described in Kano analysis.  The resolution measurements (dpi) give you a measurable criterion for what you are building, that translates into a measurable criterion (hours of continuous use without eye strain) that your customers will realize.  That criterion reflects the horizontal axis of the diagram.  The longer your customer can read without straining her eyes, the more capable your product is.

The vertical axis reflects how much your customer cares about solving the eye-strain problem.  Note that there are diminishing returns.  Enabling 4 hours of use (without eye strain) versus 2 hours of use is more highly valued than enabling 6 hours of use versus 4 hours (or 8 versus 6).

The above analysis compared the selection of discrete components, but it also applies when looking at incremental investment.  The speed at which the screen refreshes when turning pages is a good example.  You can have an e-Ink screen that refreshes in 1 second, or one that refreshes in 0.5 seconds, each time your user turns the page.  There is no difference in incremental cost, and your team is operating with a fixed budget of time and resources – so there’s no impact on the allocated fixed costs (or margins).  You are faced with a different set of compromises – what are you willing to give up (by reducing investment in other aspects of your product) to achieve incremental improvement in page-turning responsiveness?  It may be that page-turning snappiness is the one idea of your product (it was a differentiator for the Barnes & Noble Nook, but now the e-Ink Kindle has achieved parity).  Or you may be dealing with one of the “host of others” capabilities, in which case you simply want to satisfice.

Bigger Picture

The above example shows how specs matter (indirectly) in the everyday lives of product managers.  Specs are the measurements by which we evaluate the effectiveness of our products at solving the problems our customers care about.

If you are designing a product that has no competition – for customers that have no alternatives, this would be enough.  It would be more than enough, actually, because you could create “any” product, and it would sell.  Your customers always have alternatives – so you always have competition. Sometimes, your competition is “build your own” or even “tolerate the problem.”  But for any interesting market problem, you have at least one competitor trying to solve it too (or you will.  Very soon).

Comparing Products Matters

As a product manager, you need to know what your product needs to be (or do) to be competitive.  That’s where comparing products matters.

The above analysis looks at a single problem (eye strain) for a single product (yours), to try and determine what specs to give to your team.  In a competitive environment (that means you), you need to put the effectiveness of your product in context from your customer’s point of view.  That involves identifying

• Who are your customers and what problems do they care about solving?
• How important, relative to each other, are the solutions to those problems, to your customers?
• How important, relative to each other, is each group of customers?
• What solutions (products) do your customers consider as alternatives (competition) to your product?
• How effective is each product at solving each problem?

From that information, you can synthesize a point of view on how competitive your product is (or will be).

That point of view helps you identify

• Which problems you need to invest in solving, or solving more, or solving more effectively.

Summary & Series

Specification is not a good tool for helping non-expert customers compare products.  It is, however, a good tool – an unambiguous measurement tool – that product managers can use when specifying how a product should be built, or how one product compares to another.

Products must be compared based on their effectiveness (and perceived effectiveness) at solving problems that customers will pay to solve.  Those comparisons should also take into account the relative importance of those problems, as well as the relative importance of different groups of customers, to the success of the product.

Even at 1600 words, this article barely introduces the topic of comparing products.

Recapping the overall flow of this series of articles on product comparison (I’ll update this article with links to future articles in a series on comparing products.):

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers) (This article)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Attributions & Notes

* Thanks Roger Wollstadt for the original Volkswagon Assembly photo

** The dpi-related eyestrain statistics are fictional, and written to demonstrate the importance of measurement only

Your boss wants a commitment.  You want to offer a prediction.  Agile, you say, only allows you to estimate and predict – not to commit.  ”Horse-hockey!” your boss exclaims, “I want one throat to choke, and it will be yours if you don’t make a commitment and meet it.”  There’s a way to keep yourself off the corporate gallows – estimate, predict, and commit – using agile principles.

This is an article about agile product management and release planning.

Change and Uncertainty

In the dark ages before your team became agile, you would make estimates and commitments.  You never exactly met your commitments, and no one really noticed.  That was how the game was played.  You made a commitment, everyone knew it would be wrong, but they expected it anyway.  Maybe your boss handicapped your commitment, removing scope, lowering expectations, padding the schedule.  Heck, that’s been the recipe for success since they planned the pyramids.

It makes sense.

1. Your early estimates are wrong. When you add them up, the total will be wrong.  If you do PERT estimation, the law of large numbers will help you in aggregate.  But you’ll still be wrong.
2. The outside demands on, and availability of, your people will change. Unplanned sick time, attrition, levels of commitment over time, lots of “people stuff” is really unknown.
3. The needs of your customers will change. Markets evolve over time.  You get smarter, your competitors get better, your customer’s expectations change.

Agile processes are designed to help you deliver what your customer actually needs, not what was originally asked for. Contrast the two worlds.

In the old world, you would commit to delivering a couple pyramids. After spending double your budget, with double the project duration, you would have delivered one pyramid. When you deliver it, you find out that sphinxes are all the rage. Oops.

Your team changed to agile, so that you could deliver the sphinx. But your Pharaoh still wants a commitment to deliver a couple pyramids (the smart ones will be expecting to get just one).  You can stay true to agile, and still mollify your boss’ need to have a commitment, if you take advantage of the first-principles of why agile estimation works.

Estimation

A commitment is a factual prediction of the future.  ”This will take two weeks.”  Nobody is prescient.

A factual prediction has to be nuanced.  ”I expect* this will take no more than two weeks.”

*in reality, this is shorthand for a mathematical prediction, such as “I expect, with 95% confidence, that this will take no more than two weeks.”

Few non-scientist, non-engineers, non-mathematicians understand that 95% confidence has a precise meaning.  People usually interpret it to mean “a 5% chance that it will take more than two weeks.”  What it really means is that if this exact same task were performed twenty thousand times (in a hypothetical world, of course), then nineteen thousand of those times, it would be completed in under two weeks – do you feel lucky?

To make a statement like this, you actually have to create a PERT estimate – identifying the best-case, worst-case, and most-likely case for how long a task will take.

Unfortunately, we’re rarely asked to make a commitment about a single task – but rather a large collection of tasks – well-defined, ill-defined, and undefined.

You can combine PERT estimates for the individual tasks, resulting in an overall estimate of the collection of tasks.

The beauty of this approach is that the central limit theorem, and the law of large numbers, work to help you estimate a collection of tasks – you can actually provide better estimates of a group of tasks than a single task.  This obviously helps with the well-defined tasks that you know about at the start of the project.  This even helps with the ill-defined tasks.  Rationalists will argue that the key, then, is to do more up-front research to discover the undefined tasks – and then we’re set.  As Frederick Brooks (Mythical Man-Month) points out in The Design of Design, this debate has been going on since Descartes and Locke.  It is not a new idea.

Big Up-Front Design and Requirements (BUFD & BUFR) hasn’t worked particularly well, so far.

Don’t throw out the baby with the bath-water, however.  The math of estimation is still important and useful.  Even if empiricism is not the silver bullet.

Prediction

Estimation is a form of prediction.  Even agile teams do it.  In Scrum, you estimate a collection of user stories – in story points that represent complexity, and you predict how many points the team can complete in this sprint.  Note the time factor.  If you’re working a two-week sprint, there is very little risk of changes in staffing during a two-week period.  There’s also very little risk that your market will change significantly in two weeks – and if it does, what are the odds that you will notice and materially change your requirements in two weeks?

Visually, let’s take that PERT estimate and turn it sideways – so we can introduce the dimension of time.  Imagine you estimated all of the tasks (well-defined, ill-defined, and a guess about the undefined), as if they were all to happen in the first sprint.  Ignore inter-task dependencies, and pretend you had unlimited resources and the ability to perform all tasks in parallel.

The graph above shows the aggregate estimate – the circle is your best prediction, with error bars representing your confidence interval in the estimate.  If you were using PERT estimates, these could represent that 5% and 95% confidence lines.  Subjectively pick something based on your team’s experience in the domain and your confidence in your guesses (about the undefined tasks).

We need a segue into the “best of waterfall” approach to estimating projects, to steal and invert a good idea.

The Cone of Uncertainty

The folks at Construx have published a nice explanation of the cone of uncertainty – an adaptation of an idea from Steven McConnell’s Software Estimation: Demystifying The Black Art (2006).  That article uses his imagery with permission – so please go look at it there.  The idea is that as the project becomes better defined (e.g. during the project), the amount of uncertainty is reduced.

The findings show that initial estimates are off by 400% (either low by a factor of 4 or high by a factor of 4)!  Even after “nailing down” requirements, estimates are still off by 30% to 50%!

As bad as that sounds, it is actually worse.  This is a prediction for the original project (delivering pyramids).  Not only are your estimates wrong – but they are bad estimates for delivering the wrong product.

But – the core idea is sound – the further into the future you have to execute, the greater the mistakes in your estimate.

Taking that concept, and applying it to our diagram, we get the following:

The further into the future you are trying to predict, the less accuracy you have in your prediction.  This reduction in accuracy is reflected as a widening of the confidence bands for your estimate.

• A couple sprints’ worth of work is not much different than one sprint – so your estimation range is not much changed.
• An entire release of sprints (say 6 to 10 sprints) has much more opportunity for the unknown to rear its head.

Now, your prediction is (probably) unusably vague and imprecise.  ”This set of tasks will take X plus or minus a factor of two.”

That’s the reality.

Note: This has always been the reality.  People have historically reduced this “risk to timing” by hiding the “risk of change” aspects – and waterfall processes encourage you to deliver the wrong thing, as close to on-time as possible.

That’s not what we want to do, however.

We still want to deliver the (not-yet-defined) right product, as efficiently as possible.  That’s the goal of agile.  (For folks who haven’t been here at Tyner Blain for long – “right” includes both value and quality).

Refinement

Because we’re agile, and we’re willing to “get smarter” about our product over time, we have an opportunity to improve.  Because of the nature of compounding estimates and the cone of uncertainty, our uncertainty gets smaller over time.

Let’s remove our artificial simplification that we could do everything “right now” and look at what we think we know right now, about the end of the release.

Our ability to predict the amount of effort (for today’s definition of the product) at the end of the release is not very good.

Our ability to predict (today’s definition of the product) one sprint into the future is much better.

After completing the first sprint, we are a little bit smarter – the ill-defined tasks are better defined.  Maybe some of the undefined tasks are now ill-defined.  The same cone of uncertainty is now a little bit smaller – we are a little bit smarter, and the time horizon of the release date is a little bit closer.

The trend continues – each sprint gets us closer to the release date, and with each sprint (assuming we get feedback from our customers, and continue to study our markets) we get a little bit smarter.  We also get better at predicting the team’s velocity (how much “product” they can deliver during each sprint).

Commitment

Your boss still wants a commitment, however.  And that’s where we get to change the way we look at this (again).

The above diagrams all display how we converge on an estimate for a stable body of work.  However, we know that the body of work is constantly changing.

Backlog! [you say]

Yes!  The backlog.  The backlog is an ordered, prioritized list of user stories and bugs.  I was talking with Luke Hohmann of Innovation Games last month, and one of the most popular online Innovation Games is now the one they created based on prioritizing by bang for the buck.  Play it today online (for free!).  How cool is that?

The backlog represents the work the team is going to do – in the order in which the team is going to do it.  Over time, as we get smarter, we will add and remove items from the backlog – because we discover new capabilities that are important, and because we learn that some things aren’t worth doing.  We will even re-order the backlog as we recognize shifting priorities in the markets (or in our changing strategy).

As this happens, it turns out that the items at the top of the list are least likely to get displaced, and therefore most likely to still be part of the product by the time we get to the release.

Instead of thinking about uncertainty in terms of how long it takes, think about uncertainty in terms of how much we complete in a fixed amount of time. In agile, generally, we apply a timebox approach to determining what gets built.

Now, uncertainty, instead of manifesting as “when do we finish?” becomes “what will we finish?”

Your boss is rational.  She appreciates the constraints, she just wants to know what you can commit.  Every boss I’ve worked with has been willing (sometimes only after much discussion) to treat this uncertainty in terms of what instead of when.  They acknowledge that they need to translate (usually for their boss) into a “fixed” commitment.

The solution: commit to a subset of what you predict you can complete.

At the start of the release, you may have 500 points worth of stories.  Based on your team’s expected velocity, and the number of sprints in the release, you predict that you can complete 320 points worth of stories (5 people on the team, a team velocity of 40 points per sprint, and 8 sprints in the release).  Starting at the top of the backlog and working down, draw a cut-line at the last story you can complete (when you reach 320 points).  This is your prediction.

Now the commitment part.  You’ll have to figure out what you’re comfortable with.  Maybe for 8 sprints (say, 16 weeks into the future), you may only be comfortable committing to half that amount – 160 points.  Go back to the top of the backlog, and count down until you reach 160 points.  Everything above the line is what you commit to delivering.

Maybe you are comfortable committing to 240 points, maybe only 80.  This is like playing spades.  The more you can commit to, without missing, the better off you are.  Your tolerance for risk is different than mine.

You can also negotiate with your boss.  Commit to 160 points now, and provide an update after every other sprint.  More likely than not, you will be increasing the scope of your commitment with every update.

Mid-project updates of “we can do more” are always better than “we can do less.”  And both are better than end-of-project surprises.  This also allows you to have updates that look like this:

We didn’t know this at the start of the release, but X is really important to our customers – and we will be able to deliver X in addition to what we already committed.  Without slipping the release date.

Conclusion

Making commitments with an agile process is not impossible.  It just needs to be approached differently (if you want to stay true to agile).  The end result: better predictions, more realistic commitments, and the likelihood that each update will be good news instead of bad.

[Update: Changed initial image.  Thanks Dennis for the great photo!]

Requirements Management – I’m embarking on a journey to help several teams manage their requirements with their existing systems and tools.  This is the first in a series of articles, where the rubber meets the road.  I’ll look at both the theory and the realities of what works (and doesn’t) in practice.  I hope you’ll come along for the ride.

The Idea That Makes it Tricky To Manage Requirements

OK, so I’m working with a product management team that is working with multiple software development teams.  The development teams already have tools and systems in place for tracking work within projects.  The development teams are mostly using Atlassian’s JIRA for tracking activity.  The teams also have Atlassian’s Confluence wiki for capturing “other stuff.”

The development team has a good working process for managing their delivery activities within the “JIRA world.”  From the top down view, you find a project (in JIRA) that represents what will be delivered in a particular release.  Within that project, you see the issues (or user stories or requirements) and associated tasks that were previously scheduled for that release.

This is pretty effective for getting a handle on what’s happening.  Where it falls short is in getting a comprehensive understanding of why it is happening.  And that’s the view product managers need.

Why something needs to happen is completely different from when something needs to happen.  That’s what makes this tricky.  Consider the following analogy.

You’re driving across the country from Los Angeles (LA) to Manhattan (NYC).  Your goal is to get to Manhattan.  Your product manager creates a map, breaking down, roughly, how you will get from LA to NYC.  Your product manager is riding shotgun in the car, and he’s responsible for telling the driver where to go.  Your developer is driving the car, and is responsible for getting the car wherever the product manager tells him.  They share the responsibility for getting to NYC – neither of them can do it alone.

They climb in the car, and start driving.  When the developer has a question about a particular way to go (which highway to take, etc), they communicate.  Together, they make great progress.  After a while, the gas tank is empty, and they stop to fill up.  This is a logical break in the drive, and they stretch their legs, get some dinner, whatever.  With a full tank of gas, they get back in the car and start driving again.

Managing requirements inside the context of a release (in JIRA) is like saying every time you stop for gas (finish a release), you should create a new map.  The goal (destination) is independent of the release cycles (gas tanks).  The requirements should live outside the release.  However, the turn-by-turn information is useful inside the release.  The product manager needs to provide the next level of detail (first, get to Las Vegas, then stay at the Westin,…) for each “tank of gas” – after which, that information is not very valuable, and can be discarded.  But the goal (reach NYC) stays alive.

A process and system for managing requirements should not force product managers to recreate documentation for each release.  Your stakeholder (perhaps you’re delivering a donated kidney to a hospital) needs to know when you’re going to deliver the kidney.  The last place they should have to look for that information is within the context of the current project.

This is what makes it tricky.

JIRA and Confluence

This is not a series of articles around picking the best requirements management solution available today.  These articles will cover the exploration of using JIRA and Confluence to make (1) the product management teams more effective, and (2) the combined product delivery teams (management, marketing, development, quality) more effective.  We’re faced with a real-world constraint that we will use these tools that are already in place.

Product Management Goals

There are a series of goals that we have, that will inspire our implementation choices (of how we use Confluence and JIRA), and ultimately provide the measure of our success.

• We will not remove or regress things that are working today.  The implementation teams (development and test) are delivering products, and using JIRA to track issues and tasks.  Whatever we do must not break this current world.
• As product managers, we are continually investing in, and updating, our understanding of our markets.  We need a solution that allows us to record that information in a way that we can (a) share with each other effectively, (b) remember what we need to remember, (c) make changes as needed so that the view we have documented is always reflected of the understanding we have “in our heads.”
• As a delivery team, we need to connect the two worlds (“what am I doing” and “why are you doing it”), so that the product managers can let the implementation team know what to do next (and what they will eventually be working on).
• As a company, we need to set and manage expectations – internally to stakeholders and externally to customers and partners.

These are the problems we’ll be tackling.

This series of articles will capture elements of implementation choices, why we made them, and the rationale “behind the whys.”

I hope you’ll come along for the ride.

Nokia, the Finnish mobile phone manufacturer, is getting clobbered as their market rapidly moves away from them.  Recent analyst reports show that Android and iOS (Apple’s platform) based phones are rapidly gaining market share.  Nokia sells neither.  Nokia has a major press event in a few hours, where they will announce their smartphone strategy.  I think a maximin strategy is both likely and correct.

Nokia’s Press Event on Friday 11 Feb, 2011

Reuters is reporting a press event from Nokia in London, at 0730 GMT (just over 4 hours from now) – where new president and CEO, Stephen Elop will announce Nokia’s smart phone strategy.  Previously, Elop was the head of the Microsoft business division.

Nokia’s Current Situation

As a very quick background – Nokia already has a smart phone platform, called Symbian, which was dominating the market until manufacturers began shipping iOS and Android based devices.  They “owned” the market, but don’t any more.  This is a great example of the Innovator’s Dilemma.

Keeping things “as is” is not a viable option for Nokia – they have to change.  Significantly.

The Wall Street Journal is reporting that they received a copy of a leaked memo where Mr. Elop telegraphed that things will be changing.  From Jennifer LeClaire’s reporting of The Journal’s reporting:

According to the Journal, the memo describes Nokia as cornered by competitors and in need of a major transformation. Elop compared Nokia to a man standing on a burning oil platform who jumps into icy waters to escape the flames, the Journal reported.

Jennifer LeClaire, Newsfactor

Several great journalists and technology reporters have articulated Nokia’s current situation and the state and direction of the mobile phone industry – including the “inevitable” cannibalization of feature phones by smart phones.  Instead of repeating their synopses – I’ll link to a particularly good analysis from Peter Rojas at GDGT.com.

Mr. Rojas sums up that Nokia is presented with the following choice of two options, and the possible outcomes of both (bold is mine):

Nokia now has to either dig in and find a way to create its own mobile ecosystem, complete with amazing handsets and a world-class OS that developers want to make apps for (something which is becoming harder with each passing day as iOS and Android further entrench themselves), or it needs to suck it up and work with an existing platform and try and use its massive resources to become the dominant player on that platform.

[...]

If Nokia bets big on being able to create a game-changing ecosystem and it fails to catch on in the market, there will be no salvaging the company at that point.

[...]

I have no doubt that the prospect of a Nokia handset running [Android or Windows Phone 7] worries Samsung, HTC, Motorola, Sony Ericsson, and LG, all of which are trying furiously to compete with the iPhone right now. The last thing they want is for Nokia to have a serious option in the high-end of the market. [...] The market is overcrowded with Android phones right now, and Nokia isn’t necessarily going to dominate it or even stand out, at least not without a ferocious fight.

[...]

So it may be that Nokia hedges its bets and introduces either Android or WP7 on a smattering of handsets in order to buy itself some time while it works on creating its own ecosystem. [...] Buying some time like this isn’t an elegant strategy, [...] but it’s probably the least bad of Nokia’s bad options.

Peter Rojas, GDGT

Summarizing Nokia’s Strategic Choices

There are many factors and variables that make this situation look complex – but they roll up into a choice for Nokia between two strategies (excluding the uninteresting strategy of business as usual):

1. Aggressively invest in their own software platform so that it becomes competitive with alternatives that are available to their customers.
2. Adopt one of the competitive software platforms, and focus on differentiating their hardware.

The first strategy has the largest possible downside, and the largest possible upside.  If the first strategy fails, Nokia becomes irrelevant in the smart phone market.  The definition of success for this strategy leads to massive relevance (and profits).

The second strategy has a downside that keeps Nokia in the smart phone business, but as just another one of many manufacturers.  Not great, but not as bad as exiting the market.  The upside of the second strategy is also not as appealing as winning with the first strategy.  The best Nokia can do is be the “hardware winner” but without control over the software environment.  Even if there were as much profit opportunity, the risk that comes with lack of control would discount the value of winning with this strategy in Nokia’s eyes.

The Maximin Strategy – Game Theory

When you focus on the two choices – one with large up and downsides, and one with moderate up and downsides, you see that Nokia is facing a choice, in game theory, of adopting either the minimax or maximin strategy.

Nokia can either pick the strategy that has the highest possible best case, or pick the strategy that has the highest possible worst case outcome.

A maximin strategy maximizes the “minimum guaranteed payout” in game-theory-jargon.  Translated, it is the conservative approach – minimize the downside of failing in the execution of the strategy.  Mr. Rojas describes Nokia s a company that has “internalized a culture of mediocrity” (see his previous analysis for more details).

If Mr. Rojas’ analysis is even close to accurate, it seems like any strategy other than maximin would be foolhardy – almost as bad as changing nothing.

Nokia should definitely take the conservative strategy, using maximin to attempt to ensure that “the worst” is not that bad.  And that means embracing [at least*] one of the other already successful smart phone software platforms for their future devices.

Nokia’s Next Move

In a few hours we will learn what Mr. Elop has decided to do.  Or, if you heard about his decision first and are reading this second, you’ll now have either a favorable opinion of my analysis or not.

Last week, on This Week in Tech, John C. Dvorak made an eloquent argument as follows:

• Nokia can’t survive with business as usual.
• An “invest in Symbian” strategy is actually a red herring – Nokia is already doing that – but failing – so it would lead to assured destruction, and Nokia therefore must embrace another platform.
• Mr. Elop, as a former Microsoft person, would lose credibility if Nokia were to choose Windows Phone 7 as their platform.
• Mr. Elop, as a former Microsoft person, would not choose “only Android” because of the fallout it would cause in injuring Microsoft (as a perceived gesture of “lack of faith in Windows Phone 7″).
• Mr. Elop, therefore, will choose to introduce Nokia smart phones based on both Android and Windows Phone 7.

While I happen to agree with Mr. Dvorak’s analysis of the futility of doubling down on Symbian – primarily because of the challenges Mr. Rojas articulated in his article, that almost doesn’t matter.

Even if Nokia does have a viable strategic choice of investing in Symbian, they shouldn’t.  The risk of failing to overcome a culture of mediocrity is too great.

Nokia’s “best” choice is to adopt either (or both) Windows Phone 7 or Android as their official platform moving forward.  As Mr. Dvorak points out – doing both could be very smart.  The two platforms were built with different personas in mind, and Nokia should offer phones for each group of people, based on the platforms designed for each group.

A lot of teams that I’ve worked on and with get hung up when thinking about defining requirements for “migration projects” and “system upgrades.”  There’s some intangible barrier to being market focused when it comes to improving existing internal systems.  Every new product represents a solution to an existing problem.  Why do so many projects move forward with teams that are blind to the actual requirements?

New Products

Even “revolutionary” products like the iPod are just addressing existing market problems.  Part of what makes this statement true is looking at things from a market perspective – thinking about the valuable problems that people are willing to pay to solve.  Even if the iPod were the first mp3 player (it wasn’t), it would still only be an improvement.  The first mp3 player was not new, it provided an improved way to listen to your music on the go.  You could put your whole music library in your pocket.  Much better than a stack of cassettes melting in the glove box of your car, or a tape getting caught on your keys and unraveled when you pull it out of your backpack.

What’s important, and difficult (especially for people with technology backgrounds), is to think about it in terms of what people are trying to accomplish – not how they are trying to accomplish it.  In a business process view, it is the difference between process (why) and procedure (how).

There’s a continuum of migration projects, ranging from completely new to identical processes.  Neither extreme technically exists – think of it as a range from infinite change in the existing process to 1 / (infinite change) in the existing process.

Near the “completely new process” /infinite change end of the spectrum are processes that are completely new to you.  Remember, you’re solving a problem, perhaps in a very innovative way, that people are already solving some other way.  You’re just providing a better solution approach.

Near the identical process end of the spectrum are projects are “pin-compatible” platform migrations and near-sighted legacy system migrations.  Moving from an old gas-guzzling car to a new, more efficient model is a good example.  I mention “near sighted” because that old system was designed to meet an old set of market needs, so the new system will, by definition, not meet current market needs – it will only meet the old market needs.  Pragmatically, when considering organizational change, it may make sense to do your system upgrade in two stages: migrate the systems (“nearly identical process”) and deal with all the gotchas of migration first, then start re-engineering the processes and optimizing the procedures to address new market needs (major and minor process changes, respectively) second.

Set aside the possible, possibly rational rationale* to migrate the implementation first, and the functionality second.  Consider that a deployment and logistics detail and get back to the problem at hand.  Most of the times I’ve been involved in system migrations, they have been initiated as cost-savings platform projects.  Over the last few years, more often the migration project gets prioritized as a means to an end – “doing new thing X is prohibitively expensive on the old system.”  That opens the door to talking about goals.

* Couldn’t pass that one up.

Capturing Goals

On a recent project to migrate part of a company’s operations from one platform to another (system consolidation, after an acquisition), I created the following Ishikawa diagram to represent the goals for the migration project.

Even a “don’t change anything” project has real underlying goals.  Once you discover them, you open the door to having conversations about making things better.  Might be a “do it later” situation, but often, there is an opportunity to grab some of the low-hanging fruit during the system implementation.

The most important reason to capture the goals when “everyone already knows the goals” – aside from the fact that that is never true – is to make sure decisions are being made eyes-open.

Requirements live outside of the timeline of a particular project.  Once you’ve identified those requirements (aka market needs / company strategy), the next question is to determine which of those goals this project is intended to support or advance.  That question drives a lot of clarity into strategic thinking, and those answers drive a lot of clarity into project execution.

When you don’t share the context of the goals of the project, you are effectively blindfolding your team.  You prevent them from discovering opportunities to make things better.  You prevent them from making the right choice, when decisions are otherwise (without the context of goals) arbitrary.

For example, we talk about satisficing as a means to know when to ship – when it is “good enough.”  Without context, the opinion of “good enough” comes from someone on the team, without guidance.  In an agile environment, with self-directed teams, you’re making a trust-based decision to explicitly empower the team to make those decisions.  Don’t you want to give them some information, to help them make that decision?

Conclusion

This article really only explores part of the process of making real, large, projects happen in large environments.  That is a topic I’ll be talking about in a couple months.  You won’t get a sense of sated accomplishment from reading just this article – too much of the end-to-end, start-to-finish of real projects is unaddressed.

You will, however, know how to start the project.  Define the goals.  Someone will tell you the goal is to “Copy what the old system did.  We don’t have time to re-engineer.  Why are you bothering me?  Let me know if I need to find someone who can get the job done.”

Now you know how to respond.

**Attributions

Thanks foxtongue for the blindfolded typists photo

Estimating the “value” of features is a waste of time.  I was in a JAD session once where people argued about if the annoying beeping (audible on the conference line) was a smoke alarm or a fire alarm.  Yes, you can get to an answer, but so what?! The important thing is to solve the problem.

Solutions Versus Features

Everyone on that conference call had an immediate and visceral appreciation of the value of making the beeping stop.  That’s the power of solving a problem.  The methods of solving the problem – mute the offender, replace the battery, throw the alarm out the window – do not have implicit value.  They have an indirect value, in an “end justifies the means” kind of way.  But not direct value.

The same sort of thing applies when talking about prioritizing features.  Eric Krock (@voximate) just wrote a really good article, Per-Feature ROI Is (Usually) a Stupid Waste of Time, where he does two great things, and (barely) missed an opportunity for a hat trick.

The first great thing Eric did was look at the challenges of determining relative (ordinal or cardinal) value of “several things.”  He points out several real world challenges:

1. When you have a product with several things already and you want to determine the value of yet another thing – how do you allocate a portion of future revenue to the new thing versus the things you already have?
2. When thing A and thing B have to be delivered together, to realize value, how do you prioritize things A & B?  Relative to each other?
3. The opportunity cost of having your product manager do a valuation exercise on a bunch of things is high.  She could be doing more valuable things.
4. You won’t perform a retrospective on the accuracy of your valuation.  So you won’t know if it was a waste of time, and you won’t get better at future exercises.

The second great thing Eric did was reference a Tyner Blain article from early 2007 on measuring the costs of features.  I mean “great” on three levels.

1. As a joke (for folks who don’t know me, figured I’d mention that I’m kidding, just in case you get the wrong idea).
2. There is some good stuff in that earlier costing article about allocation of fixed and variable costs (with a handy reminder.
3. Eric’s article gives me an opportunity to shudder at the language I was using in 2007, see how much some of my thinking has evolved in four years, and improve a bit of it here and now.

What Eric slightly missed is the same thing I completely missed in 2007 – features don’t have inherent value.  Solutions to problems do have value.  He only slightly missed it because he got the problem manifestation right – it takes a lot of effort, for little reward, to spend time thinking about what features are worth.  I also missed the opportunity in an article looking at utility curves as an approach to estimating benefits, written two days after the one on cost allocation.  We were both so close!

People don’t buy features.  They buy solutions.

Valuing Solutions Instead of Features

Estimating the value of solutions addresses a lot of the real problems that Eric calls out.  It also has a side benefit of keeping your perspective outside-in versus inside-out.  Or as others often say, it keeps you “market driven.”

Anything that you’re doing, as a product manager, that has you focused on understanding your market and your customers and their problems is a good thing.  It may even be the most important thing.  I would contend that it eliminates objection 3 – the opportunity cost of estimating the value of solutions is minimal or zero.  There may be activities with more urgency, but off the top of my head, none that are more important, for a product manager.

Comment if I’m missing something (it’s late and I just got home from another week on the road).

The way I approach determining the value of a solution is by developing a point of view about how much incremental profit I will get when my product starts solving this additional problem.  Revenue can increase from additional sales, or from the ability to increase prices.  Cost can increase if new marketing and other operations (launches, PR campaigns, etc) are required to realize the incremental revenue.

I start with a customer-centric market model.

A given solution, or improved solution (as in “solves the problem better,” or “solves more of the problem”) – which only applies to some problems – is interesting to some customers, in some market segments.

A solution has value when it brings in incremental customers, in a targeted market segment.  It also has value when it reduces or prevents erosion of your current customer base (in a SaaS or maintenance-revenue model) to competitive solutions.

The time you spend thinking about buyer and user personas, the problems they care about, and the nature of those problems (which varies by persona) is not time wasted – or even spent “at the cost of doing something else.”

To make this useful, you have to have a forecast – without solution A, we will sell X; with solution A we will sell Y (and to whom).  A good product manager will be looking at sales, and will be able to reconcile the sales with the projections.  That helps with objection 4 (but doesn’t completely address it – you don’t know if your projections were accurate, so you can’t really know if your estimation is accurate).

This also helps you deal with challenge #1.  You’ve got a model that says “the current product works great for high school students, but not college students, because they also have problem A, which they solve today by…” Your intention is to create solution A, making your product viable to college students.  Allocate the incremental profits from college-student sales to solution A.

My approach to challenge #2 is a little more tactical.

Coupled Solutions

There are a couple ways that Eric’s “must deliver A and B” scenario are interesting, when looking at the value of solutions.

Scenario 1: Solution A solves part of problem X for persona M.  Solution B solves part of problem X for persona M.  Combined, they solve more of problem X for persona M.

This makes sense for “more is better” problems – where “more” solution yields “more” value.

In this case, I have a forecast (the more time I spend on it, the better it will be) that maps incremental sales to improved solutions.  The “first” solution to be released will have more value than the second.  If they are being released together, then I don’t care about the allocation – I combine them.

Scenario 2: If, however, the two solutions are valuable to different personas, then I treat them separately – even if they solve “the same problem,” it is not the same problem (for the same person).

Conclusion

Prioritization by “Bang For the Buck” is worth doing.  Just make sure you are prioritizing solutions, not features.

Also note: this article talked about valuation – what you do with that valuation, prioritizing by market, can be trickier.

Agile values working software over comprehensive documentation – it is 1/4th of the original manifesto.  That doesn’t mean don’t document!  It means don’t document more than you need to document.  Documentation does have value, but the practice of documenting got excessive – that’s why a reaction to the bad stuff earned a spot as one of the pillars of agile.  How do you avoid over-reacting when changing a culture of over-documentation?

The Need for Documentation

We need documentation to help us communicate.  You can define an agile team as “the people creating the product.”  You can interpret “creating” as the people we normally think of as executing to accomplish the goals, or you can be more inclusive, and think of the people who have the goals as being part of the team too.

Philosophically, agile stresses collaboration.  Usually it is couched in terms of (1) people who are executing collaborate to devise the best solutions and (2) people who are part of the team collaborate with the people for whom they are creating the product.  Personally, my experience has been that projects with committed sponsors succeed, and projects without them fail – so I always think of the sponsors as part of the team.

Regardless of your definition – collaboration requires communication, and communication benefits from documentation.

Temporal Communication

Communication among groups of people happens over time.

Some collaboration is transient – communication happens right now, and is only important right now.  Other communications are persistent – the collaboration happens right now, but we need to remember later what we agreed upon and why.  There are variations of “right now” and varying degrees of “later”, but slightly oversimplifying,

• There is communication for now.
• There is communication for later.

Both are important.

Documentation, in the vision from your nightmares – giant requirements documents, architectural analyses, and detailed psychographic profiles – is very inefficient when communicating for now.

These massive documents, while getting in the way of a conversation, do provide the opportunity to remember (in the future) why we make the decisions we make (today).

Documentation, as emphasized in most agile discussions – user stories and acceptance criteria, on an index card, taped on the wall – is not a robust solution for communication that happens later.

This low-overhead approach to capturing today’s ideas actually facilitates and improves today’s conversation – but does not give us a record we can use in the future for why we made our decisions.

As an agile team, there are ways we can approach both the persistent and transient documentation tactics to make them more valuable.  We can tweak our transient documents to make them more useful for persistence.  We can take an agile approach to development of persistent documents, so that we only do just enough documentation – trading off the minimum amount of short term efficiency to avoid long-term blunders.

Transient Documents for the Future

The image above is from a workshop defining the product backlog for an initial sprint for a new project.  The team identified the most important stories for the first sprint, their acceptance criteria, and the size, in story points, for each story.  As part of scrum, the stories are written in user-story format, with acceptance criteria for each story identified and documented on additional index cards (taped to the index cards for the associated stories).  Great transient way to capture the must do and makes it better stories that the product owner is thinking about.

As a team, we made a couple minor tweaks to make these story cards more useful.  First, all of the stories were organized into themes that help establish the context (for the team), and combine the stories into “meaningful areas of focus” for communication with stakeholders.

Each story got a blob of color in the top left hand corner, identifying which of the five themes were being supported.  You can see a few of them circled in the image above.

We also developed a set of 8 proto-personas (prototype, early draft, very rough personas – almost stereotypes) that the overall product / project was designed to support.  Each persona got a different color star.

Each user story got a corresponding colored-star sticker to indicate for which persona the story has been written.  Specifically, when multiple personas could use a story, we identified for which persona we were implementing the story in this sprint.

In the “right now” world, these tweaks helped us understand, question, and confirm the themes and personas being supported in the first story.  Those conversations allowed us to reach pragmatic compromises about which stakeholders were going to benefit and when.  They also allowed us to craft the outbound message that is step one of stakeholder expectation management.

The list of themes changed while we were doing this – moving from 4 to 5 themes, as we realized that there was value in splitting one of the themes.  The group of proto-personas grew from an initial set of 3 to a set of 8, as we quickly realized that the team was trying to “build something for everyone” and we wanted to avoid the elastic user problem, and instead, roll out capabilities sequentially that satisfied valuable groups of users with similar goals and perspectives.

Culture Change for Stakeholders

As you would expect, having a bunch of cards stuck on the wall provides a great, visceral, now communication for the team – both in implementing and communicating.  However, cards on the wall is very transient.  It also doesn’t provide a good way to communicate with old-school stakeholders (imagine the execs who have their admins print out their emails for them to read and annotate – those folks still exist!).

This can make it hard for the “not part of the implementation” part of the team to collaborate and communicate.

In User Stories Applied, Mike Cohn stresses that the brevity of user stories is intentionally designed to facilitate (and I would add “dependent upon”) conversation.  I find it to be both a strength and a weakness of user stories.  It is strong because you can cover a lot of ground quickly (breadth) and capture a number of stories, much like the list above.  It is weak because the requirements documentation does not stand on its own – it requires conversation to fill in the details.  I’ve had some success using the “Verify that…” user acceptance tests as the method of documenting those details (depth), in conjunction with the brief, easily consumable stories.

Stakeholders in a Barrel

Those conversations happened as part of getting the stories defined, getting them on the board in the right order, and defining the acceptance criteria.  Stakeholders often have a short memory, especially when they were the ones making a compromise.  You need some way to retain the context of the conversation that led to the particular organization and content of the stories on the wall.

Persistent Documents in the Present

Agile proponents complain about the massive get-in-your-way documents that slow down the start of projects, prevent the teams from adapting to changing market conditions, and otherwise make it harder to deliver great products.  I agree.  Agile proponents also rail against the “build it all, then release it,” waterfall, process for creating software – and propose an alternative – iterative development of the software.  I agree again.

Why not apply the same principle of iterative development to persistent documents?

Consider the example above of using colored-star stickers to identify the people for whom each story is being implemented in any given sprint.  The majority of the time spent on this analysis is around understanding which people are important (to the success of the product), and for which people the ability to perform this user story is important (to the person).  The combination of these two importances is an input to prioritization.  A minimal amount of time is spent putting stickers on cards, and creating multiple cards for the same story (each with a different sticker for a different persona and potentially different acceptance criteria).

Add a tiny bit of overhead, and capture the information in a spreadsheet (or whatever).

The image above shows a mapping of use cases to actors.  It could just as easily show a mapping of user stories to personas.  The transition from thinking about actors to thinking about personas is a small one.  This chart comes from an earlier article on communicating a delivery schedule with use cases – it helps stakeholders get a big picture view of who benefits when – a user-centric, yet still top-down perspective.

Note: Another nuance to incremental delivery is that you can introduce the “bare bones” version of a story now, and the “better” version of it later.  You can also communicate that stories get better over time, for some users, with the same type of chart.

A very similar, and also simple to create table can show how each theme is getting “representation” within each sprint.  That view facilitates great discussions about trying to emphasize one theme and then another sequentially, versus doing the most important items in each theme first – gradually making progress in each theme.  Creating that view, and using it to communicate, has helped me address concerns that “external” stakeholders have shared about how a team appeared to be “chaotic and random” in their approach to prioritization and implementation.

Many Models

There are many types of documentation – and many types of models for emphasizing and discussing particular aspects of a complex system (like users and goals and solutions).  Creating a simple diagram like the following whiteboard sketch is almost required for effective transient communication in some projects.

[from Simple Agile Model Example]

When most folks are complaining about documentation, they are not complaining about the sketch above.  They would consider that sketch to be augmenting a conversation.  Fine.  That’s what I did, at the time.  Then I took a picture of it.  Suddenly, it was documentation.  A stakeholder came by later, confirmed that the system needed to work this way, and signed the whiteboard – and the photo was updated.

Every heavyweight document can start out this way.  And it can evolve.  It should evolve.  If you’re going to succeed, it must evolve as your team gets smarter, your competitors react, and your market changes.

The trick is that you don’t have to write the final version before you get started.  Capture at a high level what you know right now – just like a roadmap.  Capture in just enough detail what you need right now – just like a story backlog.  And change it as you go.

Conclusion

Documentation is not bad.  Documenting stuff you will never use is bad.  Documenting stuff you don’t need for a long time is risky, because it will probably change before you refer back to your document.

Documenting why you made decisions right now, as part of transient collaboration and communication is important, so that you collectively remember.  That documentation persists and your team can build upon knowledge, incrementally – just as your team builds upon the evolving code base, incrementally.

Ask someone what they want, and they’ll tell you they want a faster horse.  Provoke them, and they’ll tell you they have a ‘get there faster’ problem, an ‘equine waste disposal’ problem, and issues with total cost of ownership.

Thought Provoking

If your requirements elicitation session looks like the photo above, you’re doing it wrong.  However, just asking people what they want and confirming that you heard what they said is also not enough.  Active listening is important, but to capture great requirements, you also have to provoke thought about why someone is expressing a “requirement.”

Adrian Reed wrote a great article this week (hat tip to Kevin Brennan) on asking provoking questions that leverage lateral thinking techniques to get better insight into the true requirements.  Adrian presents eight questions, such as “Imagine if we fast-forward to 2 years after the implementation of this project, what will the organisation look like?”  Some of his questions remind me a lot of the ideas behind Enthiosys’ Innovation Games (and Luke Hohmann’s Innovation Games book).  The remember the future, and product box games immediately come to mind.

Unprovoked Thoughts

Most good subject matter experts I’ve met, when asked about the important problems to be solved, try and be really helpful and incorporate elements of solutions in their descriptions of problems.  They will say things like “the system must integrate with [other system] to do X.”  They may even ultimately be right, that this particular system integration is a constraint, and that “X” is the only acceptable (by policy) way to achieve “Y.”  But usually, neither constraint is a requirement – it is a solution approach.

Subject matter experts who are not as good at having and sharing insights about their domain often confuse problem manifestations with their underlying problems.  By analogy, it is requesting treatment for a runny nose, when the problem the you have is the flu.  You can dry up your nose and still feel horrible.

Provoking Questions Reveal Real Problems

Adrian’s questions are designed to help you understand that you’re treating the flu, and not a runny nose.  Requirements gathering is a lot like diagnosis of a medical malady.  You have to discover the real problems.  The problems that people are willing to pay to solve.  You have to uncover the latent problems that are “hidden” behind problem manifestations.

In a (rare for me) American football analogy – the problem manifestation is that your quarterback is completing 1 of 20 forward passes.  Replacing the quarterback and receivers will not solve the problem.  The problem is that your offensive line is not able to give the quarterback sufficient time to throw higher-probability-of-success passes.

Asking questions that force people to describe their objectives differently is a good way to bypass solution-design answers.  It also creates chinks in the armor of problem manifestations.  Completing more passes is not the future you’re looking for, winning more games is the goal.  When you’re treating your flu, your goal is not to be sick – but with a dry nose.  Your goal is to be well.  When you ask someone to remember the future, they will will describe being not sick, not being dry-nosed.  The product box will be a description of a winning team.

Check out Adrian’s list of questions, and ask yourself, how do you get to the root causes?  Ishikawa diagrams (also known as cause and effect or fishbone diagrams) provide a great visualization tool if you’re a spatial thinker or a whiteboard-talker.  In the example below, you can quickly see that spending too much on fuel is part of the real problem – that the cost of operation is too high.  You can likewise see that under-inflated tires are a source of poor fuel economy.  Check out the Ishikawa article for an explanation, or this article on providing context (with Ishikawa diagrams), and this article on buyer and user personas for more examples of problem decomposition.

If you’ve got any examples of problem-statement-turned-problem, chime in below…

A picture is worth a thousand words.  A prototype is worth a thousand lines of code.  Two key elements of product management – and of agile development are elicitation and feedback.  Low fidelity artifacts can significantly improve both.  Polished, codified prototypes can create problems that prevent you from getting the benefits of communication.

Prototyping Anti-Pattern

David Bernstein has three good quick-read articles about prototyping in the agile zone.  The first one depicts the primary anti-pattern of prototyping – mis-set expectations.

As I was walking him through the different screens I could see he was starting to get uneasy. As I talked I could see him becoming paler and tenser, as if he saw a ghost. After a while I asked him if he was ok. He finally said, “You mean to tell me that I just wrote you a check for over $100,000 for less than a week of work?”

Prototyping Caveat

The second article is quick reminder of one of the goals of a prototype.

The purpose of a prototype is to elicit feedback from others or verify a design approach will work. When doing this we generally only concern ourselves with the “happy path” through our code.

A Prototype is Not a Product

David’s third article explores a little more about the mis-setting of expectations, and provides a good parallel from the film industry.

Prototypes are rough sketches without the robustness of a final product and it is often confusing to users when we show them a half-baked version of their project for feedback, even though they know it is not yet finished.

Agile Prototyping on a Napkin

David’s advice is (good and) primarily about using low-fidelity prototypes to avoid disrupting the elicitation and feedback process.  Folks in the user experience community have known this for a long time, and adapted their processes accordingly.  Back in 2006, I wrote about prototype fidelity as part of exploring ways to use this insight in requirements gathering.  Prototyping, in particular, is a great way to elicit implicit requirements – those unspoken (“you should have asked!”) requirements that your customer or stakeholder assumed you knew, or didn’t think to tell you.

Multiple Levels of Interaction

David focused on the initial “will this approach work?” elements of getting feedback on a design – and by inference, some low-fidelity user acceptance testing that the proposed approach will meet your customer’s requirements.  Jan Miksovsky wrote an excellent article (also in 2006) that provides excellent guidance on how much fidelity to build into your prototype, depending on where you are in the design process.  This is important – the type of feedback you need at different stages of your design process varies.  Jan proposes that the first prototypes be rough sketches, just as David points out.  Jan goes on to show how and when to add additional fidelity to your prototypes (read Jan’s article to see his great visual examples).  Like I said, the user experience community has known about this for a long time.

Prototypes Are For Two-Way Communication

The key element, and the reason for creating prototypes, is to get two-way communication.  A prototype is not just a status update about your design.

A prototype is the start of a conversation.

[thanks uninen for the image]

Prototyping is not only useful for design conversations, it is critical to understanding your requirements.  Yes, a prototype will you get feedback about the design-execution of your proposed solution.  More importantly, a prototype can help you make sure you’re solving the right market problems.

Prototypes Are More Than Interface Mock-Ups

The first thing we think about, and everything you’ve read so far in this article, is about getting feedback on the user interface of the product.  Prototypes are also useful for gathering business rules*, understanding system complexity, and defining, clarifying, and validating requirements.

*In that article, I show prototypes as “not being useful” for gathering business rules.  At that time, I was thinking in terms of interface mock-ups only, not other prototypes.

Use interface mock-ups when you need feedback about user interaction.  Use other artifacts when you need feedback about other aspects of your product.  You can use prototypes as an active-listening technique when gathering market data too.  There are lots of ways to draw stuff to help with communication.

I’ve regularly used flow charts to prototype processes.  Within those flow-charts, which are initially prototypes of how the product will behave, are decision diamonds.  Those decision diamonds prototype the enforcement of business rules within the to-be-created product.  In the spirit of DRY (don’t repeat yourself), a quick polishing of your process diagram can serve as a persistent artifact of the requirements.  Just like a user interface mock-up.

In more complicated scenarios, I’ve also used UML statecharts and data flow diagrams to prototype behavior.

In the section on up-front planning in last week’s article on risk management and release cadence, I talked about the mental leap that people need to make to envision a product that doesn’t yet exist.  Prototypes are like big springboards that help people leap that chasm of imagination.

Important Safety Tip

Don’t use the wrong prototype (mock-up, flow chart, etc) for the wrong conversation or with the wrong stakeholder.  Showing when and where business rules are being enforced (in the process being designed) is great for getting feedback about your interpretation and potential embodiment of those rules.  It will be a disaster if you try and have this conversation with someone who is not the owner of those policies – like a representative user.  Sometimes, policies are “owned” by non-technical people who struggle to read diagrams.  You may have to walk them through how the diagram works.  They’re smart – they’ll get it.

Just remember – use the right prototype to emphasize the right ideas, and elicit the right feedback.

I’ve been thinking about the software development process.  Big, upfront, design and requirements.  User research and analysis.  Market insights, gained on exploration or over time.  Release cadence – how quickly you get, and incorporate, feedback from your customers about your product.  How quickly you react to your competitors’ reactions to your actions.

Big and Up-Front

Leander Kahney published the full transcript of an interview with John Sculley, former CEO of Apple, last week.  Of particular interest are the discussions about how Steve Jobs is a designer first, CEO second.

He always looked at things from the perspective of what was the user’s experience going to be? But unlike a lot of people in product marketing in those days, who would go out and do consumer testing, asking people, “What did they want?” Steve didn’t believe in that.

John Sculley interview

The theme of understanding what users want and need comes up repeatedly in the article.  A few years ago, Alan Cooper and Kent Beck got in a debate about the merits of interaction-design-driven versus emergent-design software development.  Four years later, I don’t accept the premise that you have to choose between the two.  You can treat both decisions independently and make the right choice for your team and market.

Thanks Jim Holland (also PM Tribe author), for bringing this article from Wayne Mulligan (guesting on On PM) to our attention today.  It’s a great read, and the money-quote from Wayne about going back to first-principles is:

And that’s the biggest lesson I learned here – you can reap all of the benefits of an agile product development approach without following the rules to the letter.  It’s the spirit of the manifesto that counts and not the precise implementation.

4 Ways that Agile Methods Brought Sanity to My Company

In the spirit of that reality – I believe decoupling how do you plan? from how do you deploy? is fair game.

The Risk of Being (Big Picture) Wrong

[larger image]

When you do no upfront analysis at all, you have complete risk of creating the wrong product.  The more analysis you do, the more likely you are to create a product that solves the right problems – problems that your target customers, in your target market, are willing to pay to solve.  The units in the graphs in this article can be mostly ignored – all you can really capture, when talking generally about this is the trending.  The more time you spend understanding your market, the less understanding you gain.  There is definitely a law of diminishing returns.  This effect is one of the reasons analysis paralysis is so bad.  With analysis paralysis, you delay the launch of your product (at some cost), for no appreciable gain.

The two curves are labeled complex market, and simple market.  The idea here is that the easier it is to understand your market, the less benefit you get from analyzing it.  Complexity can be in the form of competitive concentration, nuances of problems being solved, specifics of the customers for whom you are solving the problem, and even the notion of just how innovative your solution is (think cars, when horses were common, or smart phones in a world of land-lines).

As a product manager, you have to act with imperfect information.  Even if you try and uncover every relevant piece of information (which you can’t do), you will misunderstand some things.  You will develop a mental model of your customers’ problems – and that model will be wrong.  You will also predict competitive responses incorrectly.  To top it off, you’ll make mistakes about “what’s next?” for your customers.

That does not mean you should do no analysis – it means you should not do too much analysis.

Not all product managers are equally good at distilling insights from the vapor of nuanced market data.  Great product managers will have (correct) epiphanies about the important problems to solve, and the impacts of disruption on the market.  Others will not.  The better you are at having these insights, the faster you reduce risk – and the faster you reach the point of diminishing returns of additional analysis.  Some product managers and markets may require weeks of analysis, others may require days.

The Risk of Being (Execution) Wrong

Another great quote from the Sculley interview:

[Jobs] said, “How can I possibly ask somebody what a graphics-based computer ought to be when they have no idea what a graphic based computer is? No one has ever seen one before.” He believed that showing someone a calculator, for example, would not give them any indication as to where the computer was going to go because it was just too big a leap.

John Sculley

In The Design of Design, Frederick Brooks discusses the debate between empiricists and rationalists.  Empiricists, in product creation, emphasize feedback as the means of gaining insight about what your product should be.  Rationalists use intuition and deduction to presuppose what a product should be.  In the hyperbole of many agile-waterfall debates, empiricists are burdened with the impossibility of designing the right product in advance, while rationalists struggle under the yoke of being wrong from the start and wasting their efforts up until the first failed product release.

Obviously, neither of these extreme perspectives is wholly true – while both have significant elements of truth.

True insights about your market grow stale with time.  The damage from erroneous conclusions about your market grows over time.  The risk of creating the wrong product grows over time.

[larger image]

If you released continuously, including getting feedback continuously, you would face the minimal possible risk that the market has changed since last you checked.  You would also maximize the opportunities to correct mistaken conclusions – thus minimizing the risk of being wrong.

You reduce the size of this risk by getting feedback.  You get feedback by “releasing” your product to customers, either as a prototype or as a product.  Jobs (per Mr. Sculley) realizes the catch-22 of not being able to get inputs from others, until they can realize a version of your idea. Sculley attributes the following to Mr. Jobs: “There was no way to do consumer research on it so I had to go and create it and then show it to people and say now what do you think?“

The longer you wait to get feedback, the greater the cost of effort invested in the wrong ideas.  The longer you wait to get feedback, the greater the risk that your market has changed since you formed your ideas.

In stagnant markets, where problems and their solutions are well understood, and change happens slowly and competitors don’t innovate, you can release less frequently with less risk.  Until someone decides to innovate, and make your market more dynamic.  Markets with active competitors, customers who’s expectations change, and problems that evolve are more dynamic.  The risk of delays is greater.

Rationalism and Empiricism Can Co-Exist

My experience has been that both concepts – “think before you act” and “fail fast, get feedback” can happily co-exist.

• You can develop an understanding of your market, your customers, and the very real problems they face, before creating a product that solves those problems.
• Introducing a new product / solution into a market changes that market – so you have to revisit what you know at least as frequently as your market changes.
• Even when you understand the problem, you need feedback to know if the product is actually the solution.  The greater the “leap,” the more representative your prototype must be for people to give you feedback about the (eventual) product.

What have your experiences been?

Almost everything I’ve read about use cases focuses on describing what needs to be added to your product.  Agile development says “get it working first, make it better second.”  That means changing the way the software enables a user to do something they can already do.  How do you manage requirements for incremental improvement?

Iterative Development and Incremental Improvement

Iterative development is the process by which you release software in iterations – small, incrementally better versions of your software.  People usually think about this solely in terms of enable the most valuable capabilities first, then the next most valuable capability next.

That’s fine, when your product is new.  Eventually (or quickly!) you will reach the point when the next most valuable improvement is not to add a new capability, but rather, to make an existing capability more valuable.

Everything is an Upgrade

In a couple earlier articles about organizing migration projects and requirements for migration projects, I wrote about how migration is a bit of a misnomer.  Everything is a migration.

My argument in those earlier articles is that the problems you’re solving with your “new” solution are not new.  Your customers are currently solving them in different ways.  The relevant question for migration projects is around how much your users are changing the way they solve their problems.

[As a slight segue, failing to appreciate this distinction can lead you to a project that is based on a faulty problem statement.]

When you are delivering incremental improvements to your product, through iterative development, you are upgrading your software.  You are migrating your users from their old solution to their new solution.  It just so happens that you are replacing yourself, as you migrate them from the old version to the new one.

A viable strategy, that appeals to me personally, is to continuously innovate, disrupting your market, before someone else does.  With this approach, you are intentionally reinventing your solution, making it difficult for someone else to out-innovate you.  This is a great way to approach developing your roadmap strategically.

Your market will change.  Rapidly. Do you want to react to your competitors, or keep them on their heels while you stay on the balls of your feet?

The reverberations of the “new” Twitter still haven’t settled, and many of Twitter’s competitors are on their heels – some of them only now realizing that Twitter has always been a competitor and not just “a platform.”  The new Twitter doesn’t really change anything about how people use Twitter, except make it (markedly) better.

Avoiding Featuritis

When is it more valuable to improve what your software already enables, versus enabling users to accomplish more with your software?  Kathy Sierra introduced us to the concept of featuritis, the idea that too much is too much.

In an article on viral product management, I proposed that by making your software better, you can tap into an altruistic mechanism by which people will promote your product.  As an example, I proposed improving the usability of your software as a means to cross this viral tipping point.

One of the key ideas is that making it better for users makes your product better for your company.

Managing “Improvement” Requirements

Now that we all agree that there is value in making your software better [comment below if you disagree] the question becomes “How?!”

Use Cases can be used as the keystone to your requirements arch.  Sometimes, user stories are better than use cases, but use cases are perfectly agile.  When you’re starting a project, you want to apply an outside-in approach to defining the scope of that project – or more particularly, to define the scope of problems you are solving in your roadmap. This article will talk in the language of use cases, but identical concepts and approaches apply when using user stories as well.

When you have prioritized a use case as “the next most valuable capability to introduce” you add it to the backlog for your sprint.  The implementation team then provides feedback that use case is “too big” and you have to slice it up.  It is important that you don’t split the use case  in such a way that you only solve half the problem.

You also shouldn’t deliver a “bad” product.  In fact, the ideal way to introduce a new capability is to satisfice, and not introduce the “perfect” solution in your first iteration.  Make your first solution “good enough.”  Part of the art is defining “good enough,” but rest assured, it is not the same as “the best you can possibly do.”

Summarizing some key elements from above demonstrates the eventual reality you will face:

1. Every problem* your users face is already being solved, you are just providing a better solution.
2. There is diminishing, and eventually negative return on adding more capabilities to your product.
3. Your first release of a given capability will only be “good enough.”  That leaves room for improvement.

*Yes, you can pedantically define “the problem” as “the existing solution is not good enough” – but you still end up in the same place, so why bother?

Eventually, the value of improving something you’ve already released will be greater than the value of releasing something new.

What do you do?  You implement the same use case again.

If your product is Software as a Service (SaaS), you absolutely should be thinking about things this way.  Becoming better is a continuous improvement objective that is implicit in the economics of SaaS products.

Revisiting Use Cases

Not to be confused with re-use of use cases, revisiting a use case is specifically revisiting – with the goal of improving – the implementation that is already in place within your product.  This is a special case of a migration project – you’re migrating your users from your old solution to your new and improved solution.

You may be migrating to an identical process, perhaps with faster performance than your previous release was capable of.  Or you may be improving the procedure (improving usability, interaction design, visual design, etc) without affecting the process. Generally, a “make it better” use case improvement effort will have no more than a minor process impact.  Your users are still doing the same thing, they are just enjoying it more, or doing it more effectively.

When slicing use cases to make them fit within a single sprint, you may design them for a single user persona in the first release, knowing that you won’t meet the expectations of a different user persona until the next release.  In this case, your non-functional requirements, constraints, or acceptance criteria will be different.  Your use case may also be different (in the details) while appearing to be the same.  This also common scenario can be addressed with the same approach.

Find that old use case (from several iterations ago) and put it back in the backlog.  Remember, agile is about conversation, not artifacts.  If you need to add an explanation to the use case, because your team fixates on the fact that it is “already working,” add a qualifier that it needs to be “better.”  Then have a conversation, and explain how it needs to be better.

Is Design Part of Implementation?

Some teams organize such that designers (both architects designing code and designers designing interfaces are being addressed here) are part of the implementation team.  Other companies treat designers as stakeholders – providing guidance and input to the product managers and product owners.

When a “new design” is being requested from outside the team, include that design guidance as part of the input to the implementation team – through artifact and / or clarification.

When a “new design” is being requested of the team (the designer is part of the team), express the new acceptance criteria to the team.  Note that these need to be measurable requirements if they are to be considered good requirements.

Summary

There will come a time when the most valuable thing you can do is improve the user experience for a capability your product already embodies.  When that time comes, use the same use case again, updated with the new constraints, non-functional requirements, and acceptance criteria.

Each requirement you write represents a single market need, that you either satisfy or fail to satisfy.  A well written requirement is independently deliverable and represents an incremental increase in the value of your software.  That is the definition of an atomic requirement.  Read on to see why atomic requirements are important.

Atomic Requirements – Revisiting

As part of the ongoing series, Writing Good Requirements – The Big Ten Rules, I wrote about the importance of atomic requirements first in 2006.  That article touched on only one aspect of atomic requirements – being able to ask “is it done?”

Writing atomic requirements is important from two perspectives – delivering value to your customers, and operating efficiently.  You get benefits both operationally, and in how you are delivering value when you write atomic requirements.

Atomic Requirements Accelerate Value Delivery

In a recent article, I proposed methods for splitting user stories when they are too large to be delivered within a single Scrum sprint.  That article looked at ways to break up complete user stories (an agile requirement artifact, similar to a use case) that were already atomic, making them even smaller.  Think of that article as Atomic Requirements 301 – sort of an advanced class on splitting the atom.  There are opportunities to subdivide molecular requirements – those made up of multiple atomic goals – the topic of this article, which would be Atomic Requirements 101.

If you’re using an agile development process like Scrum, consider a user story like the following:

• As an online shopper, I need to find and compare similar products, a few times a year, so that I can make an informed purchasing decision.

The key element, from an atomicity perspective, is find and compare.  These are actually discrete user goals, although it may not be immediately obvious.  Rewriting as follows, will highlight this:

• As an online shopper, I need to find products and compare similar products, a few times a year, so that I can make an informed purchase.

Rewritten like this, find products and compare similar products are clearly different activities supporting different user goals.  They should be split into separate, atomic user stories.

1. As an online shopper, I need to find products, a few times a year, so that I can purchase desired products.
2. As an online shopper, I need to compare similar products when shopping online, so that I can make an informed purchase.

These two user stories can be delivered separately.  If this example were for B2B (business-to-business) eCommerce, the example would be different – because the online shopper is (often) not the right persona.  A typical situation in B2B is that one person will do the research to determine what to buy, and another person will make the decision of from whom to buy it.

This rule applies for non-agile requirements development as well – consider the following “BRD-style” requirement example:

• The system must record all purchases and submit them to the fulfillment system for processing.

Recording purchases and submitting them for processing are two different activities.  While they likely support the same goal, they may also support different goals and could be implemented separately.  Recording purchases is important not only for fulfillment, but potentially also for analytics.  Pushing data to a fulfillment system can be done in either a manual or automated fashion – providing a distinct cost-reduction opportunity.  To write these as atomic requirements, they must be separated.

1. The system must record all purchases.
2. The system must submit all recorded purchases to the fulfillment system for processing.

This example shows only the bare bones – there is more involved in writing either requirement, particularly around defining the non-functional requirements and business rules (see also: why you benefit when you separate rules from requirements, and the difference between business rules and business requirements) that affect and constrain each requirement.  This is easier to see when the requirements are atomic. These two atomic requirements may be rewritten as follows:

1. The system must record all purchases on the same day that they are created, recording information per [BR117: Purchase Data Rules].
2. The system must submit all recorded purchases to the fulfillment system for processing per [BR231: Fulfillment System Data Interchange Specification].

These additional constraints are not non-atomic additional requirements, they are constraints that specify how the system must perform the atomic actions.

The previous examples allude to another key benefit of atomicity – clean traceability.  Each requirement (or user story) exists to support one or more goals.  Requirements traceability can get complex, when many goals are dependent upon multiple requirements to be realized and many requirements enable multiple goals.  This approach to representing goal-decomposition (or user story decomposition) is critical to assuring that you are writing valuable requirements.

The simple version of traceability can be visualized with this view of structured requirements, presenting (and slightly extending) some of Karl Wiegers’ work on requirements.  If you’re doing any work in requirements of any kind, you need to read Karl’s stuff – his Software Requirements is a must-have.

This diagram makes it seem pretty straightforward.  Where it gets messy is when the multi-goal-per-requirement and multi-requirement-per-goal dependencies (that always exist) turn it from a simple tree into a complex graph.  Non-atomic requirements make that graph messier (a hassle, but not a problem)as each “requirement that is really multiple requirements” ads extra dependency relationships to your traceability model.  This becomes a problem, however, when you need to change.

Every project has implementation tasks that take longer than originally expected.  Using a time-box approach to managing the content of each release gives you a straightforward framework for determining how much stuff will be delivered in each release.  When you have to slip something, you have to revisit prioritization.  With well-defined requirements, each valuable requirement that could be delivered provides value by enabling users (or systems as users) to achieve a goal.  When you have to delay the implementation of a requirement, you delay the goals that it supports.

As soon as you realize you are considering delaying part of a requirement, that is a red flag that your requirement is not atomic.

Atomic Requirements Simplify Operations

Atomic requirements allow you to make these on-the-fly prioritization decisions with much better insight into the resultant delay in value.  This allows you to better manage communication with your stakeholders – validating that you are delaying the realization of the right goals.

The same benefits apply when originally planning your software iterations and releases.  Conceptually, you can imagine “scheduling” everything for the first release, immediately discovering that most things need to slip to future releases.  That’s how most* software projects really play out.

*There are times when you intentionally delay particular capabilities, based on external factors – market positioning, customer-adoption cadences, sales-team capacity (to absorb change), etc.

Conclusion

Writing atomic requirements helps you

• Organize and describe why you are asking the team to build what they are building, and how stakeholders will benefit from what will be built.
• Understand the assignable value of each requirement by maximizing clarity in the dependency tree – each requirement is in place for specific reasons.
• Communicate the relevance of each requirement to the implementation team – informing cost-benefit discussions around the inevitable schedule-change discussions.
• Test the deliverables – atomic requirements get graded as pass-fail, not “X% working.”

Attributions:

• Thanks clix for both “atomic” photos!

Every team that transitions to agile faces this problem – some stories are too big to fit in a single sprint.  Most of the teams that I have worked with have the wrong instinct – to solve half of the problem for all users.

The right approach is to first solve all of the problem for a subset of the users.

The Goal of the Story

User stories were created as lower-overhead use cases that served to better describe chunks of valuable software that could be rapidly developed, delivered, reviewed, used, and improved. Writing user stories that undermine that intent – rapid, valuable, incremental delivery – is a bad way to write user stories.

Imagine you are part of the team building GoToAssist, a product designed to do this. Your product manager and product owner put the following story into the top of your backlog – it is the most valuable capability you could add.

As a provider of IT support, I need to remotely resolve issues on multiple computers simultaneously, every day, so that those computers will run well.

The acceptance criteria include requiring that the solution working when accessing Apple computers, and PCs running various versions of windows and Linux. Other acceptance criteria include allowing the solution to work well through network connections having minimum characteristics, not requiring the remote users to take action, and other elements that characterize what would be considered acceptable solutions by the stakeholders.

Your team immediately raises a red flag that this user story is “too big” for a single sprint.

Since the goal of the sprint is to deliver valuable, shippable software, you have a problem with any user story that cannot be completed within a single sprint.

No problem, just decompose the user story into smaller user stories…

Inside-Out User Stories (Bad)

Note: Don’t confuse making smaller user stories with defining the tasks required to implement a user story. Task definition is important and valuable for helping the development team organize their activities.  Making smaller user stories is not a good approach for organizing the team’s activities – use tasks instead.  The size and structure of user stories should align with the user’s goals.

An inside out perspective or approach is one that starts with an awareness of what is involved in creating the software, and either never, or only later considers the perspective of what it is like use the software.

When approaching the problem inside-out, you will break the story down as follows:

1. As a provider of IT support, I need to remotely gain access to multiple computers simultaneously…
2. As a provider of IT support, I need to remotely authenticate as an administrator to each computer I access…
3. As a provider of IT support, I need to remotely control each computer I access…
4. As a provider of IT support, I need to remotely restore control of each computer…

This approach looks very appealing, because it is very clear what has to be done to support each new user story, and each new user story is appreciably smaller. Assume that each new user story is “small enough” for the team and does not need to be further reduced in size. It is very easy to see how the scope of development work has been reduced for each user story.

The problem shows up when you approach this schedule from a user’s perspective. Imagine that this product shipped with only stories 1 & 2, and not 3 & 4. How useful would the user find the product to be if she could remotely access a computer and authenticate, but not actually control the computer (to do her remote support work) or return control to the computer’s owner?

The reason you should not break the original user story up into these pieces is that the new stories do not independently represent delivery of valuable software. These proposed smaller user stories are atomic (each step is discrete), but each step has no independent value. The value is implicit in the sequence of steps, and is only realized when all four steps can be completed. The original story is already the smallest set of activities that allow the user to achieve his goal – a complete user story.

The problem with decomposing the original user story from the inside-out is that the new user stories don’t provide independent value to any of the users.

What about trying an approach that provides all of the value to some of the users?

Outside-In Story Decomposition (Good)

You are still faced with the challenge of making the story smaller, but now with the constraint of only delivering pieces that provide independent value. You need an approach that will work to split the story up per-user. “Provider of IT support” is an overly broad category of users – it is heterogeneous, within which different groups of users will use the product differently. I am a “provider of IT support” to my family, but I never have a need to access multiple computers simultaneously. I also don’t need to do this more than once a month on average. A corporate IT specialist responsible for 100 users would really benefit from simultaneous remote access to multiple machines – allowing her to multi-task, starting updates on a second machine while the first machine is working, etc.

There are at least two different personas that care about the “remote administration” user story, within the “provider of IT support” user category. One persona who needs to access only one machine at a time, and one persona who needs to access multiple machines simultaneously. This split allows you to decouple all of the work associated with managing simultaneous connections from the work associated with the sequence of steps (as applied through a single connection). This may reduce the size of the original user story enough to complete it in one sprint, but probably not, since it only “shaves off a little work.”

Another way to discover different sets of people who could benefit from delivering the user story differently is to look at the acceptance criteria.

Right off the bat, we’ve called out that the solution needs to work when the remotely accessed computers are running several different operating systems. You can make the user story smaller by constraining to one supported platform at a time. Whichever platform you pick first – say Linux – has some users who will value your solution. Start with them, then add another platform. Work with the product manager to determine how many customers in the market will require each platform or combination of platforms – that will guide the sequence in which you build support for each platform.

Another acceptance criteria required that the remote users not need to be involved – enabling “unattended” remote support – a key market differentiator. A key differentiator for the corporate IT specialist persona, who gets value from not having to coordinate support activities with multiple remote users. In fact, it is probably a must-have feature for those customers. As a family-support specialist, I don’t really have this need. My mom is going to call me, and I can fix her pc while we’re on the phone. I’m not trying to remotely administer her machine when a corporate policy has changed (on my schedule) – I’m doing it when a problem arises (on her schedule). This criteria can be deferred for a later iteration of the story

This approach works because, while it may be not be intuitive (from the inside) that this is a viable way to decompose the story, it makes send from an outside-in, user perspective.  Think about the different sources of value / problems, and how different people value solutions to the same problems differently.  The important characteristics of the problem will vary from one persona to another.

What About Development Efficiency?

As a developer, you may very correctly complain that this is a really inefficient development process. Some portion of the work done to enable controlling a single machine may need to be discarded and recreated in order to control multiple machines in parallel.

While these extra development costs are very visible, especially to developers, they are much smaller than the benefits that come from delivering the working software earlier. The cost of resolving bugs increases by orders of magnitude as you move through the product creation life cycle (from development to testing to deployment). The value of solutions also increases by orders of magnitude as they move through the process. Getting working product deployed earlier allows you to start realizing value (and therefore revenue) earlier.

That increase in revenue far outweighs the cost of a less-efficient development approach. Ignoring revenue side and focusing only on the cost side is being penny-wise but pound-foolish.

Dogma and Pragmatism

None of this should be taken as dogma. My experience has been that this approach has almost always worked, sometimes requiring more radical thinking than others. When it does not work (e.g. does not provide additional value, or does not sufficiently reduce the size of the stories), don’t do it. I’ll always remember Kent Beck saying to me – “If it costs more to test (in advance) than it costs to fix the bugs (after the fact), then don’t test.”

The same applies here, be pragmatic about it.

If you’ve been biting your tongue since the middle of the “Inside-Out” section, because you can always just wait until all four user stories are completed in separate sprints, before releasing the software, then my apologies. Of course you can do that. But you are undermining a precept of Scrum – that every sprint could be released, if you needed to release it.

That may not apply to you – I work with some teams that develop enterprise software, and for them, releasing every sprint makes no sense – their customers can’t absorb the rapid improvements, their sales teams aren’t willing to sell differently, etc. So, please forgive me, and go back and read the Outside-In section again, since you were probably ignoring that section as you started planning a great comment about this.

Who knows – a “too big” user story may appear in the backlog for the last sprint within your current release – wouldn’t it be great if you had the choice to release something (valuable) in the current release?

Think about this problem from an outside-in perspective. The smallest story is one that allows a single customer to perform a single task, for which he will pay you.

People Over Process – Conclusion

The objective that every sprint deliver valuable, working software, is not a requirement. It is a design principle of the Scrum process.

Agile is not about mandating processes, it is about getting better and faster. When you can’t reasonably decompose a story into something that can be delivered in a single sprint, don’t. You can have a sprint where you don’t deliver anything, because you only complete part of one thing. Yes, it will look bad on the burn-down chart.

Will it mess with your velocity? A bit. But who cares? Velocity is a measurement that helps inform projected delivery schedules. You may have one sprint with zero velocity, but in the next sprint, when you do deliver the “two sprint story”, you’ll have double the velocity. Magically, your running average is back to normal.

Don’t let the rulers you use to measure the team’s output cause you to act irrationally.

Attributions:

• Thanks to John Loo for the half-a-coin image.
• Thanks to Handige Harry for the puzzle pieces image.

Writing Verifiable Requirements should be a rule that does not need to be written.  Everyone reading this has seen or created requirements that can not be verified.  The primary reason for writing requirements is to communicate to the team what they need to accomplish.  If you can’t verify that what the team delivered is acceptable, neither can the team.  This may be the most obvious of the rules of writing requirements – but it is ignored every day.

Verifiable Requirements – Revisiting

In 2006, I first looked at how and why writing verifiable requirements is important - as part of the ongoing series on the rules of writing good requirements.  The focus of that article was on testability, as is this one.  When visiting verifiability again, however, I will add that not only must specifications be objectively measurable, but so must goals be written so that you can identify when a solution has met the objectives that justified its creation.

Directly Verifiable Requirements

Most of the requirements we come across can be directly verified.  The only problem with these requirements is that they lack specificity.  The following example is almost verifiable, and only needs a little help:

• The web site must make it easier for users who use site search to find the products they want to buy.

This requirement presents the challenge of resolving the ambiguity of the requirement. This lack of specificity prevents the requirement from being verifiable.  You have to identify how much easier the site search process must become for users.  Findability is a more is better characteristic in the Kano Analysis model for defining requirements.

You have to acknowledge that making it easier to find what you’re looking for is better for users than making it harder – there is an upward slope to the function.  However, there are also diminishing returns.

[larger image]

• A search that includes “the perfect item” is massively better than a search that does not include what you are looking for.
• A search that includes the perfect item on the first page of results is significantly better than one that returns the perfect item on the second page of results.
• A search that includes the perfect item as the first result is only marginally better than a search that returns that perfect item as the second or third result.

Improving findability as articulated above, from a user perspective, manifests in value to your company in two ways: immediate revenue impact and indirect revenue impact.

The immediate impact can be measured in terms of increases in commerce.  The indirect impact results from improving the user experience.  These improvements in experience result in increased word of mouth, as users altruistically encourage others to visit your website, and also result in an increase in satisfaction causing users to return to your site more frequently and make more purchases from you.

You can measure these impacts in terms of

• Conversion percentage (percentage of people who search and then purchase the products found in the results)
• Revenue attributed to users who search (an absolute measurement of purchases of products found via search)
• Site traffic levels (the number of people that visit your site over time)
• Visitor-recency statistics (the amount of time that elapses between return visits for returning visitors)

Conversion percentage, for users who search, normalized against users who do not search, is the most isolated (from other variables and noise) and fastest responding (as a delayed measurement of impact) measurement of the value of making it “easier” for users to search for the products they want to buy.

Your team will design different approaches to achieving these improvements.  You have to estimate both the cost and the potential benefit of each approach.  Balancing cost estimates with potential benefits will yield the ideal requirement – perhaps a 10% improvement.  [Note: I've collapsed at least another article's worth of balancing cost versus benefit, and multiple articles of "understanding and measuring site search" into one paragraph here, in hopes of staying on task with writing verifiable requirements.]

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The web site must make it easier for users who use site search to find the products they want to buy.”
• After: “Users who search on the site will be at least 10% more successful at finding the products they want to buy when using site search.”

Impossible to Verify Requirements

Often, stakeholders will present us with requirements that are impossible to verify (as requested).

• The home page needs to load fast.

At first blush, this looks just like the previous requirement (easier search is similar to faster page load).  You can use the same techniques to determine a measurable “requirement” like “the home page needs to load in under 1 second.”  However, that’s not a good requirement, because it is not an implicitly valuable requirement.  This statement provides no insight into why a fast-loading home page might be valuable to a particular user or why it would be valuable to the company.

To address this issue, you have to meet with the stakeholder(s) and understand the actual underlying requirements.  You will ask why, determine motivations and underlying problems, and apply active listening skills to discover the underlying requirement.

The problem is that the statement, “the home page needs to load fast” is a specification.  In Requirements Documents – One Man’s Trash…, I first used the following diagram to show how different people in the process of designing software view their piece of providing a solution.

A developer may receive a spec – “home page must load in under 5 seconds” and feel like the why component is perfectly well defined – it is a matter of context and perspective.  Another way to think about this is that the problem is with the problem statement.

A product manager, however, needs to do research that follows a path like the following:

Given that we are building an eCommerce website, we know that people have expectations around page load times (per Forrester / Akamai report (requires registration)).

[larger image]

Further, those expectations manifest as people abandoning slow-loading websites (from the same report).

[larger image].

We know from this market research that page-load times (for websites) represent another more is better Kano attribute, but one that has must be characteristics at the extremes.

[larger image]

While more speed is better, what the market research reveals is that for any given person, there is a minimum-speed threshold, at which speed is a must-be characteristic – too slow, and you lose customers (immediately).

A  combination of market research and elicitation will reveal that there is a true underlying goal of being fast enough to satisfy as many users as possible.  Combining this with practicalities of scaling your solution, and the associated costs; in the context of a particular solution design (an eCommerce website), you will arrive at a goal that allows you to rework your requirement so that it is verifiable.

Note: The requirement in this example is a subordinate requirement to a goal focused on maximizing conversion percentage (the percentage of customers who visit the site making purchases) – with a recognition that a major source of non-conversion is abandonment of the site, and that a large contributor to visitor abandonment is page load times.  An Ishikawa (or other model) will help you articulate this complexity and formulate requirements at a level of abstraction that is both valuable and actionable.

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The home page needs to load fast.”
• After: “No more than 10% of potential site visitors will abandon our site before viewing the page.”

Indirectly Verifiable Requirements

Some requirements are impossible to literally verify, and must be inferentially verified.

• The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.

Models are Models, Not Reality.

Imagine trying to create a test by organizing a million visitors to hit your SaaS web solution on the same day, with at least 10,000 of them hitting the site simultaneously.  That is completely impractical.  That doesn’t however, invalidate the requirement or make it untestable.  The combination of modeling, hypothesis formulation, and extrapolation gives you a reasonable way to verify that your solution probably meets this requirement.

You’re already used to the concept that models are representations of reality – think about the maps you use – a map may have a scale like one inch equals one mile.  The map is a model of reality.  A map where one inch equals one inch would be impractical.

An effective approach to measuring this type of scalability requirement is to simulate users (with load testing and realistic user-behavior scripts) with automation.  That takes care of most of the coordination problem.  However, the cost of simulating a million users and 10,000 simultaneous users is high.  You can, more realistically, measure the site’s performance when 10, 100, and 1,000 simulated users simultaneously access a server.  You can extrapolate from those results to estimate how the system is likely to perform under 10,000 user loads.

Note: Extrapolation is dangerous (follow the Elvis link below)- you have to justify why extrapolation holds true, and identify when it does not, to minimize the risk of invalidating your hypothesis.  Make sure you have confidence in the engineering prowess of whoever is doing your extrapolation and modeling.

[see tip #5 of these ROI calculation tips for the Infinite Elvis anti-pattern]

While this requirement does not to be rewritten in order to be verifiable, it does need to be augmented:

• Original: “The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.”
• Additional: “Testing of the site must show that 500 simultaneous users following <user script X> will yield no more than 1% page load times over 200 ms.”

Conclusion

Every requirement you write must be verifiable.  When you can’t verify something, and can’t rewrite it into a verifiable form, that should be a sign that either it is a vision statement or a red herring.  Vision statements guide how we approach creating products and engage markets – they are valuable, but they are not requirements.  Red herrings are well-meaning but ill-advised inputs into the process that need to be culled – they are neither valuable nor requirements.

Make sure all your requirements are verifiable.

Writing unambiguous requirements is about understanding what is written, and what is read.  Without a clear understanding of your market, you can’t write unambiguously.  Even when you understand your market, you risk writing something that is ambiguous to your readers.  Documenting requirements is about communication.  Don’t break this rule, or you’ve wasted all the energy you spent understanding your requirements.

Unambiguous Requirements – Revisiting

Fifty months and five days ago (another grin), I wrote about the importance of writing requirements so that they can be clearly understood.  Writing requirements so that there is one, unambiguous, interpretation of what you wrote.  This article is the seventh in a series on the rules of writing good requirements.  In that version of the article, I focused primarily on language ambiguity, and framed the discussion in terms of ambiguity to the writer and to the reader.

As part of improving that series, I am revisiting each of the articles now that I have fewer hairs and the remaining ones are more gray.  The same framework still applies, but will be expanded in scope, using the following modified definitions of the framework.

• Ambiguity for the Writer – not having a clear interpretation of the requirements.
• Ambiguity for the Reader – requirements that, as written, can be interpreted in more than one way.

It takes two to tango.  To lead, you have to know where you’re going.  To follow, you have to know where you’re being led.  The same is true with requirements – you have to know what you’re writing, and how your writing will be read.  Misinterpretation of requirements is the source of over 40% of all software defects.

Ambiguity for the Writer

When you don’t know why you’re developing a particular set of requirements, it is because you either don’t have sufficient understanding of your market, or of your corporate objectives.  While it is always a lack of knowledge, sometimes, that lack of understanding is insidiously hidden.  You, as the writer of requirements, are the reader of the market – and what you’re reading is likely to be ambiguous.

For Whom You Are Designing

The most overlooked element I’ve personally seen in product creation is failing to understand who the users and buyers of your product will be.  I wanted to type “forgetting to understand,” but I struggle to imagine someone who understands the value of defining user personas and buyer personas, who then forgets to define them when gaining an understanding of their market.

The reason that this is an ambiguity issue is that when you fail to define personas explicitly, you define them implicitly.  An implicit persona is an ambiguous persona.  Alan Cooper named this anti-pattern the elastic user, and the problem manifestation as failing to provide insight to guide design decisions.  When you have an ambiguous user, you lose what Frederick Brookes calls the clarity of design. This results in at best a mediocre product without a core idea or aesthetic.

Different users value the same solutions differently.  Users vary in competency, and therefore have varied personal goals as they learn how to better use your product over time.  When you don’t identify who the user is, you prioritize requirements without insight into what is important to the undefined, elastic, and ambiguous user.

The readers of your requirements are trusting that you will prioritize requirements correctly – but you can’t, because you lack an understanding of who the users will be.  You therefore lack an understanding of which problems are valuable, and which problems are more or less valuable than other problems.  This ambiguity makes it particularly difficult to write valuable requirements.

Precision in Language

The other aspect of writing unambiguously is in using language that has meaning.  Using words like “can” and “may,” is imprecise in requirements.  ”The system can respond in under three seconds.”  Does that mean that it must respond in under three seconds?  Does that mean it is acceptable if the system responds in twenty seconds?  This is an area of overlap with the rule of writing measurable, verifiable requirements, so I won’t go into more details here.

Another source of imprecision in language, not only for the reader, but also for the writer, is jargon.  You should never use jargon, as you have no way to know if your reader will actually interpret the jargon terms correctly – as readers often vary in the depth of their domain knowledge.  Jargon is also a high risk of ambiguity source, precisely because it is jargon.  Jargon is short-hand for communicating efficiently among domain experts.  Are you enough of a domain expert that you are confident that you absolutely, unambiguously understand the accepted meaning of the jargon term?

Hopefully, you’ve achieved a perfect understanding of your market.   Long-time readers will know that I’m joking.  Market analysis suffers from it’s own Heisenberg Uncertainty Principle – if you actually did manage a perfect understanding, the market will have changed – the best you can hope for is a statistical understanding of the behavior of your market.  You can document your interpretations, assumptions, and decisions – achieving clarity in your requirements – at least from your point of view.

Ambiguity for the Readers

As I mentioned earlier – using requirements is a dance – and it takes two, both the writer and the reader.  You need to write requirements so that they can only be interpreted in a single way, or you will stumble.

“You keep using that word.  I do not think it means what you think it means.” – Inigo Montoya (IMDB)

Shared Language

The word “shall” as in “the system shall…” can be misinterpreted when your readers don’t share the same native language as you – which is why you must always use “must.“

Some good research on vocabulary size data for comprehension of english can be found here. The average native english speaker knows 20,000 word families. With a vocabulary of 5,000 english words, only 98.5% of the words in a given text will be understood. 5,000 words is a lot for a speaker of a second language (3,000 words is considered aworking knowledge).

An analysis of the reading-level at which a document is written can be helpfull in identifying if the language is likely to be challenging for readers with limited vocabularies. The Gunning-Fog Index provides a measure of the education level at which a text is written.

Writing Unambiguous Requirements (2006)

Can’t write that section any better :).

Ambiguous Use of Language

I had the privilege to attend a presentation by professor Daniel M. Berry in 2007, on the ambiguous use of language in business rules and requirements.  In that presentation he referenced The Ambiguity Handbook that he authored (80 page pdf), which I just re-read last week.  There are many good examples of ambiguity in the use of English in business documents.

The following are some of the sources of ambiguity, for which I entirely credit professor Berry for pointing out to me.  I’ve only added some illustrative examples:

• Plurality Causes Ambiguity.  Consider “Emails must include sender addresses” and “emails must include recipient addresses.”  Must each email include one sender address and one recipient address?  Must each email include multiple sender addresses?  Solution – don’t use plural subjects.  ”Each email must include a sender address” and “each email must include recipient addresses.”
• Associative Ambiguity.  Professor Berry calls this the parenthesis problem.   What does “A and B or C” mean to you?  Does it mean “A, and either of B or C” or does it mean “either C, or both A and B” when you read it?  In math and programming, we learn very specific rules for how to interpret these types of structures.  We are also given parentheses, that we can use when we want to be specific and unambiguous.  The reader of your requirements is not a compiler, so don’t assume that she will interpret this the same way that you do.  Solution – use parentheses and explicit language to eliminate ambiguity that would arise from ignorance of the associative property.
• Anaphor Ambiguity.  For non-linguists: “don’t use pronouns.”  I love the example that wikipedia uses – “We gave the bananas to the monkeys because they were hungry.  We gave the bananas to the monkeys because they were ripe.
  
  We gave the bananas to the monkeys because they were here.”  The pronoun, “they,” has an ambiguous binding.  Should “they” be a reference to the bananas or the monkeys?  In each of the first two sentences, you could reasonably assume that the reader will properly bind “they” to the monkeys and bananas respectively.  In the third sentence, an assumption of how the reader will interpret “they” is not reasonable.  Reasonable assumptions are still ambiguous – and the context is less likely to be obvious – so don’t use pronouns and rely on readers to bind them to the appropriate nouns.  Solution – repeat the noun for every reference.

Incompleteness Ambiguity

Unwritten requirements are by definition ambiguous.  You must not assume that unstated requirements will be implemented.

A security exploit was recently discovered (and eliminated) in Facebook.  Previously, if you attempted to login to Facebook with an email address and a password that did not match the email address, Facebook would present you with a typical “failed login” screen and a link to reset your password.  However, in an effort to be friendly, Facebook would also include the first name and last name that were associated with that email account in the friendly warning message.

That is nice, except it is a privacy problem, and anyone who knew about this before could use the login page to discover the first and last name to associate with any email address.  ”Bad people” with email address lists could discover the name to associate with many of those email addresses – allowing more sophisticated phishing or social engineering attacks.

The requirement to not provide any personal information when a user fails a login attempt was almost certainly an unwritten requirement.

Conclusion

Precision in your writing will reduce ambiguity in your requirements.  Active listening will help you discover when ambiguity has slipped through your editing net.  You can never completely eliminate this problem, but you can improve.  You must.

vs.

What happens when billionaire media magnate, Rupert Murdoch, pits his idea against a Nobel-prize winning idea from the beautiful mind of economist and mathematician John Nash?

When you act on what you hope your market will do, instead of what you predict your market will do – you’re in trouble.

This is a story about understanding your market, and an example of using game theory – specifically, the Nash Equilibrium in “non-cooperative game theory” to predict market responses to your products.

Murdoch’s Idea

To sum up the particular idea I’m talking about – Mr. Murdoch has put the Times of London behind a pay-wall (you have to subscribe to access the content online), and removed the articles from Google’s search index.  He did this because he believes that people accessing the content for free are “stealing it” and that the only way to make money in the news business online is by forcing people to pay for content.  For more nuanced and deeper discussions of what Mr. Murdoch is doing, check out these articles from The Atlantic explaining what he’s attempting, the Editor & Times report of a 66% drop in traffic, Newser‘s investigative journalism of what’s happening inside the company, or Business Insider’s take on the matter.

My opinion is that Mr. Murdoch expects his market will respond the way he wants them to – by subscribing to the online version of the Times of London – in stark contrast both to pundit predictions and what professor Nash’s Nash Equilibrium model of non-cooperative gaming predicts will happen.

As results have finally started surfacing, they appear to completely support what game-theory would predict and utterly refute what Mr. Murdoch would have hoped for.

Game Theory for Predicting Market Behavior

I was enthralled by Scott Steven’s lecture series, Games People Play: Game Theory in Life, Business, and Beyond – a DVD set of a series of lectures from The Teaching Company [link goes to Amazon].

Understanding your market involves not only knowing what problems your customers face, but also predicting how your competitors will behave.  Competitive analysis is not just capturing a snap-shot of their products and positioning today, but also forming predictions of how they will respond to the disruptions you will create in your market by innovating.  Markets are not static – you have to understand both how your customers’ needs will evolve and how your competitor’s offerings will change, in order to understand how your product will perform.

One of the “games” in game theory is called The Coordination Game, and it is the one that captures Mr. Murdoch’s conundrum.  The coordination piece of this game is in acknowledging that what you do (to pay-wall or not to pay-wall), and what your competitors do (pay-wall or not) are interdependent decisions, of which you only control your own decision.

You can’t (successfully) unilaterally decide what you are going to do, without taking into account what your competition is going to do.  And vice-versa.  That’s the part that makes it hard – there is a circular reference that has to be resolved.  I suspect that’s the source of elegance in the Nash Equilibrium – he realized that there are “stable” and instable combinations of the decisions – yours, and your competitors.

The Pay-wall Coordination Game

Mr. Murdoch and “everyone else” (his competition for consumers of news) are both faced with a choice – should they allow free access to the articles, or not.  Free access to articles can generate revenue by two mechanisms.

The first is one of the freemium business models – billing Peter to pay for Paul – charging money to advertisers for access to viewer’s attention.

The second mechanic is by leveraging word of mouth – people reading your free article, then encouraging others to read your other articles.  This approach can either be used to get a percentage of people as paying customers (with a “leaky pay-wall” model that allows some free views, but encourages paid-subscriptions), or to get additional traffic that increases revenue from advertising.  One way to maximize this approach is by making your product implicitly viral so that it promotes itself.

Alternately, Mr. Murdoch, or any of his competitors can create a pay-wall, where only paying subscribers will have access to the content.  Presumably, certainly historically, a subscription model with the same level of readership – if that could be achieved – would generate more revenue for the publisher than either freemium model.

The crux is that we have tons of market data to show that “the same level of readership” simply will not happen.

There are four possible scenarios that can happen in this market -

1. Both Mr. Murdoch and “everyone else” institute pay-walls, making all online content available only to paying subscribers.
2. Mr. Murdoch erects a pay-wall, but everyone else continues to allow free access to content.
3. Mr. Murdoch allows free access to content while everyone else erects pay-walls to prohibit access to content.
4. No one erects a pay-wall, and free access is allowed to all content.

[Update 2010.07.21: Why "everyone?"  See comment #2 below]

In scenario 1, assuming people don’t just stop reading online, everyone wins – everyone charges subscription prices, achieving the maximum amount of revenue.  So why didn’t this happen?  Because the decisions are interdependent, and because the market is starting out in scenario 4.

Mr. Murdoch has moved the market (from his perspective) into scenario 2, which is clearly bad for him, in hopes that everyone else will move as well, to reach scenario 1.  Once Mr. Murdoch put the Times of London into scenario 2, then everyone else’s best move is to stay there.  No one else will join him, because everyone else never will.  The market, once stabilized in scenario 4 (the starting point) can never move to scenario 1 – without collusion, which is conveniently illegal.

Someone will always be there to refuse to move to scenario 1.  ”Everyone else” is destined to allow free access, or exit the market.

Understanding Your Customers

It is possible that Mr. Murdoch looked at the Wall Street Journal, which has a pay-wall, and said “I want to do that too.”  And perhaps he discounted all of this game-theory “nonsense” because if it worked for them, it can work for him – the model is BS.

However, I believe the Journal’s market is different, and it works for them for different reasons.  The market for business news pays a premium for immediacy and quality of information – because getting the best information first is worth a lot of money to them.  There are “free” immediate sources of business news information, but the market indicates that it perceives that information to be of lower quality.  And there are free high-quality information sources, without the immediacy.

The reality, which you can only get from understanding your customer’s goals – is that the business-news market is stable in scenario 1. It could be destabilized, by someone offering free, timely, high quality information.  If that happened, as professor Nash showed, the market would either re-stabilize in scenario 1 (the company that changes says “oops” and reverts their policy), or the market would transition to a stable state in scenario 4 (everyone offers free content).

The more concentrated the market [see: measuring market concentration], the more likely a reversion to scenario 1.  The more competitive the market, the more likely that it will stabilize in scenario 3.

Non-business news on the internet?  Very competitive.  And stable in scenario 4, no matter what Mr. Murdoch might hope.  Sadly, there are profits to be made in scenario 4, but they aren’t as appealing as the double-rainbow of scenario 1 – however impossible it is to achieve.

Attributions

The two photos at the start of this article of Mr.s Murdoch and Nash are licensed through the Creative Commons Attribution Share-Alike 2.0 Generic and the Attribution Share-Alike 3.0 Unported licenses, respectively.

All other content in this article is licensed per the Creative Commons link at the bottom of this page.

As product managers, we talk about creating the right solutions with our products. Understanding the very real problems our customers face, understanding the very real opportunities our markets present, and manifesting that understanding in a product roadmap.

Other than being “not as good,” how expensive is it to build the wrong product?

The Cost of Poor Quality

There’s an analog to the market dynamics of making poor product decisions – executing with poor quality. Many research studies and articles have identified the market impacts of poor quality.  This has become so well accepted that people today cite it like a law of physics (one example here based on this 1988 IEEE research by Barry Boehm and Philip Papaccio) as the “1-10-100 rule.”  The primary conclusion of that research is that ten dollars spent on fixing bugs:

• Costs and saves $10 when you catch (and fix) the bug during implementation.
• Avoids $100 in costs when you catch the bug during QA and send the product back to development (then test again).
• Avoids $1,000 in costs versus waiting until your customers catch the bug in the field, causing the team to remedy the problems, rush out a patch release, and/or go to heroic lengths to manage a PR problem.

This is an opportunity in front of your product team – a 100x payback from investing in quality during the development process.  Of course, be pragmatic about it - if the cost of testing exceeds the cost of bugs, don’t test.

This is not a solved problem, by any stretch, but the solutions and methods to solve this problem are well understood now.  In fact, a 2001 article by Barry Boehm and Victor Basili shows that in some cases, the labor costs to resolve bugs can be as low as 5:1 – when considering a subset of smaller systems, when using more “agile” processes.  That lowered ratio does not take into account the lost market opportunities and the costs of cleaning up collateral damage to your product – just the immediately realizable (and measurable) costs of resolution.

One very real problem, when talking about “bugs” is in defining what a “bug” is. And the definition of a bug is a matter of perspective. A developer can reasonably assert that “if it meets the spec it is not a bug, it is working as designed.” What if the spec is wrong? The developer may not be guilty, but collectively, your team screwed up. There’s a “bug” in the requirements.

What Is A Requirements Bug?

Now things are getting interesting.

If you wrote a requirement that you interpret as “A” and your developers interpret as “B” – you definitely have a bug – the team won’t build the right product. For each $1 you could spend making sure you have bug-free requirements, you could:

• Make sure you have a shared understanding of the documented requirements through active listening before development begins ($1). Following the Rules of Writing Requirements will help prevent this miscommunication.
• Wait until the engineering team is ready to demo their progress ($10). They will have to build it again, because they built the wrong stuff.
• Wait until development is complete and QA is validating that the code meets the spec ($100). This gets tricky if you are thinking “A”, the developers are thinking “B”, and QA is thinking “C.”
• In classic throw-it-over-the-wall mode, wait until the product is launched, and it is the wrong product ($1000). Assuming “A” was the right problem to solve, the cost of entering the market with a solution to “B”, leaving “A” unaddressed, is impressively high.

This gets interesting because the above assumes that “A” was the right problem to solve. What if “G” was the right problem to solve, and “A” was the wrong market problem? Even if everything (else) is working perfectly – you document requirements for “A”, the engineering team creates a marvelous “A” and it launches without implementation errors – you still fail, and incur the 1,000x cost of a failed product launch.

There is an even larger opportunity in front of your product team – a 1,000x payback on discovering and choosing to solve the right problems for your customers and markets.

• Would Palm still be independent if the Pre had solved a compelling problem?
• Why did Intuit have to buy Mint.com – could they have embraced the same customers with Quicken?
• What is Garmin going to do now that “free” GPS mapping and turn-by-turn directions are becoming ubiquitous? If it is “more of the same,” how much are they wasting?

Wrapping Up

I’m not aware of any studies that show that “requirements bugs” fit the same 1/10/100/1000 cost explosion model that “implementation bugs” exhibit. Emotionally, it “feels about right” to me – it passes my “sniff test.”

There are times when days of research would have been required to avoid or redirect a few hours of implementation effort on projects I’ve worked on. And I’ve seen man-years invested solving problems that didn’t involve much more research.

My intuition from products and teams I’ve worked with is that it probably averages out somewhere around 10x.

What does your gut (or your data – if you have some, post a link below!) tell you?

Consistency in writing requirements is important on two levels – strategic and tactical.  Tactically, you need to write your requirements with grammatical consistency, so that potentially ambiguous statements will be interpreted similarly.  You also need to write requirements that are logically consistent, so that you avoid “impossible” requirements and gaps of unspecified meaning.   Strategically, your requirements need to reflect a focus on markets and problems that are consistent with your business objectives and the vision your company is manifesting

Consistent Requirements – Revisiting

Thirteen hundred ninety-six days ago (grin), I wrote about the importance of grammatical and logical consistency in writing requirements – an emphasis on the tactical aspects of consistency in writing requirements.  This article adds a brief discussion on the consistency of the requirements you write.  This article is the sixth update of a series of articles on the rules of writing requirements.

Grammatical Consistency

[source]

Writing requirements is not like other forms of writing in one key way – you don’t want to vary the terms and language you use in your requirements documents.  Remember the critiques you received when you studied writing in school for a moment.  Your teacher told you not to repeatedly use the same word, but to use synonyms for that word – walked, ambled, stepped, capered, etc.  Those synonyms allowed you to create imagery and helped your readers avoid boredom.  Your teacher encouraged you to use metaphors that created even stronger imagery – flew, slogged, muddled, danced, etc.  Those metaphors allowed you to infuse emotion into your narrative and helped your writing come to life for your readers.

As Yoda said, “Unlearn, you must!”

Requirements are not a narrative.  Your primary goal is not entertainment, it is enlightenment.  Don’t use varying terms to describe the same action or object – use the same terms over and over again.  Don’t vary the modifiers of terms except when you are explicitly varying the context and referenced items – and then make certain that you use the same modifiers consistently for the same context and references.  When dealing with anything that has some complexity, provide a figure or diagram (possibly in an appendix or glossary) that explains the explicit meanings of each term and term-modifier.

For example, if you are defining requirements for a classification system for authored works, be specific about the definitions of author, editor, publisher, and reviewer.  You may also need to define a term that is used collectively for all of them, like creators.

You must consistently use the same voice  - ideally the active voice.  And of course, you must not use ‘the system shall.’ [There's a great debate in the comments on that article - one thing to note - all of the people on both sides of the debate emphatically expressed their consistent use of one form or the other.]

Logical Consistency

[source]

Logical inconsistency most commonly comes from expressing the same requirement twice.  If you use the exact same words to express a requirement twice, you’re being redundant and repetitive (grin #2).  Most likely, the second expression of the same requirement is both unintended, and worded differently – running the risk that it will be interpreted differently, and therefore inconsistently.

You will also create logically inconsistent requirements when you express mutually exclusive requirements.  These can be directly incompatible, like writing “the system must infer the value…” and “the user must provide the value…”   Or they can be indirectly inconsistent like “the animal must bark” and “the animal must be a cat” – which is logically inconsistent because cats cannot bark (thereby making at least one of the requirements infeasible).

Strategic Consistency

One of the elements that defines valuable requirements is that they “support your business strategy.”  In the context of value, that means you have created a decomposition model for how you intend to realize value – manifesting a strategy or a component of a strategy.

A strategy, however is not always defined by a single measure.  Your strategy could include financial metrics and targets, as well as a stated mission regarding customer service levels, and a long-term investment that is hoped to eventually return long-term rewards.  Your requirements, while not only completely defining the mechanisms for achieving value, must also be consistent with the other elements of your strategy.

For example, you should not take advantage of uninformed costumers when you have a core value of integrity – regardless of how effective that may be at meeting financial goals.  The requirements that articulated the (subordinate) goal of grifting your customers (to maximize revenue) would be inconsistent with the goal of operating with integrity.

Conclusion

Consistent Requirements are requirements that

• Are not in conflict with any of your strategic objectives, vision, or corporate goals
• Avoid logical inconsistencies
• Utilize consistent structure, terms, and grammar

One of my colleagues got into a debate with one of his colleagues about the differences between goals and requirements.  His opponent fired the following salvo: “[That] is not a business requirement in any company of the world…”

What you call your requirements is less important than how you communicate them.

Labels

I’ve worked with a lot of companies that use different labels to describe their descriptions of the problems that need to be solved. This is not another article about requirements versus specification (specification = design constraint, btw).

If you’re not a long time reader who’s followed the debates and discussions on this topic in the past, check out these articles and comments, where we’ve had some great discussions on that topic – with particular thanks to Roger Cauvin for keeping the debate front and center.

• Requirements Versus Design: Which is Which and Why
• Requirements Docs – One Man’s Trash is…

The central theme is that the debate about “what versus why” centers around differences in people’s perspective on the problem.

This challenge is not really about that (but it makes for good background).  The complaint my colleague received was really a critique of the way that the why of a set of requests was communicated and labeled.

Actionable Goals

Companies have strategic goals (increase shareholder value, gain market share, etc) that serve to guide the investments and activities of the organization.  However, for the teams that are creating products, those goals are too vague to be actionable.  I guess this is where my set of labels comes in.  I don’t think the labeling of a goal as a goal (versus labeling it as a requirement) is abstractly important, however, when people value making a distinction, here’s how I do it.

A goal is something your business (or user) is trying to achieve.  It is too abstract to have a single valid solution approach.  Once your business stakeholder(s) decide on a strategy by which they will achieve that goal, the business will define requirements for the actionable project(s) that execute this strategy.  This is an ideation step that lives inside the “why” box from a business analyst’s perspective.  Here’s an example:

Warehouse Manager

You’re in the IT department of a large company.  You’ve been assigned to support the warehouse manager (or COO, or vice president of order fulfillment or whatever).  Your warehouse manager is responsible for procuring the products that your company sells, keeping inventory on hand, and delivering those products to customers when they have been purchased.

Your warehouse manager is measured (or promoted) based on her performance relative to the following:

• The time that a customer has to wait between placing an order and receiving the product.
• The cost impact of warehouse operations on the products.

Your warehouse manager focuses on these two metrics because of your company’s goals – provide good customer service, be profitable, and grow revenue.  Those abstract goals influence product pricing – it must be low enough to grow revenue in the target markets, and be high enough to be profitable.  Since market pressures drive pricing decisions, typically a profitability target will then create cost-reduction pressures (profitability also influences pricing decisions, selection of markets, positioning within that market, etc).  A customer service strategy can also include a “you get your product quickly” component – as it does in this example.

Your warehouse manager now comes to you and says “I have an initiative to lower costs.”

Is this a goal or is it a business requirement?

Ultimately, it doesn’t matter what you call it, but for you to actually do something you need to know more.  That’s why my approach is to call it a goal.  Your warehouse manager’s problem is to lower costs.  You need to decompose that problem to understand how you (and your team) can do something to help her achieve that goal.  The reason you have to do that is that you need a problem statement at the right level of abstraction.

In elicitation conversations with your warehouse manager, you learn that all of the costs of operating the warehouse are allocated to the different business units, who operate as profit centers.  The warehouse operates as a cost center.  The warehouse manager treats each business unit as a customer, and provides a service of “fulfilling product orders” in exchange for funds that cover the costs of operation.  The allocation of costs is done per product that appears in the warehouse.  Since allocation is per product, it could be something like $1.30 per product in “warehouse costs.”  There are multiple sources of those costs (that are then allocated per product), and there are multiple ways to approach reducing the cost per product.

Your warehouse manager develops a strategy (probably a “mini-strategy” from the company’s perspective) to lowering costs.  She rules out “sell more products” as an approach, because she can’t control that.  She also can not affect her fixed costs (rent, depreciation, labor).

Eventually you discover that there is another cost hidden in the provisioning model – products are purchased, they sit in the warehouse for a while, then they are sold and delivered, and eventually money is collected from customers to pay for the products.  Your company has to pay in advance for the products, and doesn’t get their money back until funds are collected from the customers.  That money is “tied up in inventory” when it could have been invested otherwise – so there is a cost associated with high inventory levels, that directly correlates with the amount of inventory in stock.  When a new version / revision of a product is introduced, your inventory may become obsolete (it is definitely devalued).  And since markets are moving targets, your prospects of selling a product change over time – a risk that you will not be able to sell all of your inventory.

At the end of the day, the amount of inventory you have on hand has a cost.  The more you have, the more it costs.  A rule of thumb I heard 20 years ago was that inventory costs could be as high as 1/3 of the cost of the product for each year you keep it around.  If that is true, that is something like 0.5% added cost for every week the product is in inventory.  Here’s something the warehouse manager can potentially control.

Currently, your warehouse manager has 8 weeks of inventory on hand (on average) for products, which allows her to ship quickly.  She realizes she is adding 4% to the cost of the product.  She decides that if she can cut those costs in half, she will be a heroine.

Her business requirement is to reduce inventory levels from 8 weeks (on average) to 4 weeks (on average).  She also has a constraint that no more than 1% of orders be delayed by more than a day (due to running out of inventory).

Now you have requirements that are unambiguous and measurable.

Conclusion

Don’t trap yourself in worrying about the labeling of goals / requirements, but if you have to work with someone who does find that really important, before you call it a requirement, make sure it is at the right level of abstraction.  And of course, make sure you follow the rules of writing requirements.