Why do products fail?  Trying to organize all of the reasons that your product might fail is a Herculean effort.  Understanding how your product did, will, or might fail will help you focus on what you need to do next.

An Exploration of Product Management

A personal goal for me is to become better at product management, so that I can help create better products.

As a product manager, the most important thing you should be doing is understanding the problems that your customers face.  If you treat “improving at product management” as the product, you should start with an exploration of the problem space.  I’m framing that problem space as products that fail.  I think it is also fair to also think about products that under-perform.  They “succeed” given the goals by which they are being measured, but they never realize their full potential.  I’ll keep the “not as good as it could be” notion in the back of my head, and talk to “failed” as the larger issue.

There are conversations, blogs, books, processes, and frameworks for “how to be a (good) product manager.”  I suspect looking at this from the outside-in (problem first, solution second) may yield some interesting insights.  Thanks to Leisa Reichelt for her article on why most UX is bad, which inspired me to have the conversation here, approaching the problem as a root cause analysis.

As an agile product manager, I’m going to approach this iteratively.  I hope that you will provide insights and corrections, helping to adapt this as we go.  Your contributions will make this better, faster.

Root Cause Analysis of Failed Products

Ishikawa diagrams were originally created to allow engineers to figure out why something broke – a visual tool for organizing root cause analyses.  I’ve been using this powerful decomposition tool for several years to solve problems and organize my thoughts.  It may provide the perfect framework for gaining understanding about why products fail.

Let’s dive right in.  Here’s my first crack at the very top level of an Ishikawa for product failure.

[larger version]

Looks pretty sparse (for now).  I will fill it in, as I drill down into each area.

If you’re like me, you’ve got some “reasons for failure” in your head right now – maybe from past experience, maybe from watching products in the market today.  Do any of those reasons not map into the categories above? Tell us about it in the comments (or tell me privately, if you must) – that’s the first vector for informing an improved version of this diagram.

Here are some prose descriptions of what I’m thinking about (for now) for each of those branches on the diagram – I expect them to fill out with smaller branches attached to each of the main branches.

Product Fails in the Market

• Business Case is Flawed – This is where we would capture things like a product strategy that is not profitable, for example, your model was dependent on exponential growth – so even though you had consistently growing market share, it wasn’t enough for the product to be considered “a success” by you.
• Picked the Wrong Market – Maybe this market is about to go away, like buggy whips or audio cassettes.  Maybe the competitors in this market are just too good.  Maybe entering this market is too divergent from your corporate strategy and dilutes focus and investment in your company’s other products.  Another example would be if your team does not have the skills and resources needed to win in a particular market.
• Takes Too Long to Enter Market – Whatever it is you’re doing to enter the market, it took you too long.  Competitors have “out-gunned” you, your customer’s needs have changed, etc.  This is to capture causes where even if everything else was good, you simply didn’t move quickly enough.  At first blush, I expect organizational problems (lack of alignment, bureaucracy, insufficient resources) will all land here.
• Doesn’t Solve the Right Problems – This is where most product managers focus most of their time – making sure we’re actually solving the right problems (the ones customers are willing to pay to solve).  Problems of design – where we intended to solve the right problem, but the proposed solution doesn’t cut it – would not be in this branch – they would be in the “not good enough” branch.
• Positioning & Sales Approach is Wrong – For the first iteration, this is my catch-all for marketing and sales.  All of the problems that are “your potential customers don’t think of your product as a solution to their problems (even though it is).”  Also the problems of “your potential customers decided not to purchase (when they should have)” and “your potential customers have never heard of your product.”  This is definitely an area where you can contribute more than I can to the framework.  How would you (product marketing managers, I’m looking at you) frame this?
• Product is Not Good Enough – Execution is key here.  Not solving problems completely (although that might move to the “right problems” branch) is an example.  Bad design is an example – both bad user-experience and bad architecture.  Poor execution also lands here – broken windows, sloppy implementations, poor quality.  For this branch to work, “product” is not just the product that your development team builds, but also your customer relationships, distribution, services, etc.

Open Questions in This Model

There are definitely some design decisions I made in the approach above that are worth questioning.  Here are some that come to mind for me – add your own in the comments.

• Designs that fail to solve the target problems – I put this in the the “not good enough” bucket, because I felt like there was value in grouping the “how” separate from the “why.”  Many times, an inadequate design is a design that works great for inadequate requirements – which means the problem is in the “why” column.  How would you organize “bad requirements execution” and “bad design execution” in this model?
• Pricing and Cost – together, they reflect profitability.  ”Not profitable” implies a business case problem.  Incorrect pricing implies a positioning problem or is a red herring that is masking a sales problem.  High cost is reflective of bad design decisions and/or execution problems (both of which are in the “not good enough”) bucket.  Perhaps if you can’t create the product at the cost you need to, for your business model to work, when selling at a given price in order to compete, you have picked the wrong market.  Where would you put pricing and cost issues?
• Team Capabilities and Support. Not having (enough of) the right skills on a team limits what solutions you can create.  Not having enough support to gain needed skills constrains the solution space as well.  When your team does not have what it takes, or have what they need, to create solutions that will succeed in a given market, where would you put this?  For now, it is in the “picked the wrong market” bucket, because trying to compete in that market is infeasible.  There’s probably a better way to frame this – how would you do it?

A Litmus Test

One experiment to validate this model is to look at the main causes of project failure and see if they map well into this space.  In the Chaos Report findings from the Standish Group, the following are the top ten responses from the companies they surveyed, along with my thoughts about how they map into this framework.

1. Lack of User Input – In my experience, this manifests both as not solving the right problems and as delivering a product that is not good enough.  The mechanisms are different flavors of “not listening.”
2. Incomplete Requirements & Specifications – Ends up causing delays, and possibly not solving the right problems, or having designs that are not good enough (because the requirements that drove the designs were not good enough).
3. Changing Requirements & Specifications – I put this squarely in the “your mindset is broken” bucket.  Requirements change, even if your requirements document does not.  Depending on how this plays out (for your team, for your process), you either slog on and deliver solutions to the wrong problems, or you take too long to deliver.
4. Lack of Executive Support – This can be a takes too long problem.  Or it could be that lack of support causes a product to be abandoned (even if it were succeeding), or constrains options for your team to the point that you aren’t able to succeed within the parameters.  This one definitely doesn’t fit the model.  Should we update the model, or treat this one as “out of scope?”
5. Technology Incompetence – insufficient skill results in not good enough and usually delivery takes too long.  If really bad, it results in “impossible (for this team) to deliver.”  For the really bad cases, does it land in picked the wrong market, because that market is infeasible?
6. Lack of Resources – Just a subset of lack of executive support.
7. Unrealistic Expectations – in the worst cases, it is a problem with the business model.
8. Unclear Objectives – lands in not solving the right problem.
9. Unrealistic Time Frames – same as unrealistic expectations.
10. New Technology – yes, change is hard.

Not surprisingly, the list from the Chaos report uses a project-centric language.  Other than the (very valid) “failure profile” that lack of executive support causes projects to fail, the model seems to hold up reasonably well.

Your turn – what would you add or change?

At this point in the product comparison series, you know who your customers are, which problems are important to them, and which products compete to solve those problems.  It’s time to score the competing products and see how the solutions your product provides (or will provide) will stack up.  This is the latest in a series on comparing products, jump back to the start of the series if you came here first, but hurry up :).

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem. (This article)
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Summarizing Effectiveness

Earlier in the series, we identified (and refined) the list of important, relevant problems that our target customers have.

[larger image]

This is the “ruler” by which each competitive product is going to be measured.  We also identified several competitors.

[larger image]

The next step is to assess how effectively each competitive product (including your own) solves each important problem.  Then you need to assign a “score” for how effectively each product solves each problem.  To do that, for each problem you have to articulate an opinion about what it means to solve the problem poorly or completely, or anywhere in-between.

Read Anywhere – Previously clarified as “Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.”

Environments can be indoors/outdoors, extremely cold to moderate to extremely hot temperatures, with variable lighting from a dark room to direct sunlight.  It might also capture environmental context – sitting, walking, riding on a bus, driving, etc.; quiet to noisy; physically serene, or getting bumped a lot (like in a crowded coffee shop).

Start by defining the endpoints.  I’ve been using a 9-point scale in this type of analysis, to provide enough granularity to make relative comparisons.  For read anywhere, a score of 1 would mean “can be used to read in a single, idealized environment / location.”  A score of 9 would mean “can be used to read in any realistic situation.”  Mapping out the scores in-between 1 and 9 requires you to think about the nature of the problem being solved – and here’s where Kano analysis is useful (again).

This capability is a good example of an extreme more-is-better capability.  Increasing the range of environments where the product can be used (to read) provides a perceivable benefit to customers, but with diminishing returns.  Also, there is some minimum bar, or table stakes, of environments where the user needs to be able to read, or the product is not considered a viable solution.  On the high end, being able to read literally anywhere, would truly distinguish one product – making that capability a delighter and a strong differentiator.

[larger image]

How do you decide “What is a 3 score?” You inform these relative scores based on user research (ideally), and your subjective opinion (when you don’t have research).  For folks who haven’t been reading Tyner Blain articles for the last few years – a product manager is market driven, which means you need to use market data to do this as a product manager; however, using your own opinion as a product designer is better than having no data at all.

For this example, my [manufactured, invented, made up for this series of articles,] data defines scores for the Read Anywhere capability as follows:

1. Below The Bar - User must be seated, and have power and internet connectivity (at the time of reading) in order to use the device.
2. na
3. Usable but Very Annoying – User can read while standing without being connected to power or the internet.
4. Not Quite Happy, but Whatever - User must be indoors, with reasonable lighting and temperature.
5. Meh - User can read while riding on the bus or in a car.
6. OK, but Nothing Special- User can read in outdoor lighting.
7. Good - User can read pretty much anywhere except really noisy, jarring, and / or low-light environments.
8. na
9. Wow! - User can read anywhere that the user would want to read.

Note that there are no entries for 2 or 8, to reflect that there’s a step-function decrease (or increase) in perceived value at this point in the curve.  Plotted on the Kano analysis extreme more is better curve, this rating scale looks like the following:

[larger image]

One of the things you can see when looking at the scoring system visually is that you may have to make significant investments in order to make incremental improvements, depending on where you are on the curve.  For example, moving from (4) to (6) looks hard – you are objectively increasing the capability measurably – and increasing the subjectively perceived value by a comparable amount.

A small shift in scoring – for example, from (7) to (9) – can have a dramatic impact on perceived value (specifically, satisfaction received from improving the capability).  Conversely, improving from (6) to (7) may be really hard (expensive) to do, but realistically only improves the way your market perceives your product by a marginal amount.

This view can help you answer questions like

• Why does the iPod shuffle outperform the Sansa Clip so dramatically? Because the iTunes-centric ecosystem turns the dial from (7) to (9).
• Why doesn’t our detergent outsell theirs, when ours gets clothes cleaner in soft water?  Because moving from (6) to (7) doesn’t make much of a difference.

Zooming in on the low-end of the curve:

[larger image]

And at the high end of this capability, we see:

[larger image]

Applying this rating scale to the competitive products [more made-up data here] we see

[larger image]

It may be that different personas would use markedly different criteria for scoring relative capability.  So far, when I’ve done this, I’ve found that different persona care different amounts about the scores for a particular capability, but generally use the same approach to scoring.  When I do come across different personas that would use markedly different scoring criteria for the same capability, I will create use the appropriate scoring criteria for each persona, and add more complexity to this process.  To date, I haven’t had to do that.

Scoring All of the Capabilities for All of the Products

Applying the same process (determine the nature of each capability, determine the criteria for each capability, assign a score to each product for each capability) will result in something that looks like the following [manufactured to illustrate the concepts] data:

[larger image]

Alternate Scoring Method

You could also approach determining the score for a single capability like this by giving points for each characteristic, versus trying to define a continuum like the example above.  For example, you could give

• 2 points for being able to use the device without power.
• 1 point for being able to read when you are not connected to the internet.
• 1 point for being able to read in bright sunlight
• 1 point for being able to read in a dark room
• etcetera

Then tally up the score for each product, to describe how well they meet the need that users have to read anywhere.

Interpreting the Comparison

If you were to stop here, you would just conclude that the iPad2 is the best, and that the two Kindle products are “close,” and that the Nook and using a PC are in last place by a wide margin.  However, you aren’t going to stop here.

Stopping here would ignore completely that different customers care different amounts about solving different problems.

In the next article in this series, we’ll see how to incorporate that info, and answer the two questions you need to be able to answer:

1. Which product is best for a specific persona?
2. Which product is best overall.

Summary

To create a competitive product, you need to know how your product stacks up against the competition – and that means you need to know how effective your solutions are (or are perceived to be) at solving the problems your customers will pay to solve.  You can compare today’s product to assess your current position, and you can inform the decisions about what your product needs to be.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem. (This article)
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Taking it to the next level, as a product manager, your decisions about tomorrow’s product should be in the context of where you expect tomorrow’s competition (and tomorrow’s customers) to be.  There is a danger – especially after investing in the “state of the industry” analysis above – that you will continue to compete in yesterday’s race, when you should probably be innovating to redefine the game.  The comparison you just did is not a waste (because it looks at today’s problems – they become the table stakes for tomorrow).

Remember, this exercise informs future product decisions, it does not define them.

Attributions

Thanks Rick Cox for the height comparison photo.

You start with a point of view about what makes a minimum viable product.  When your product launches, it is your customer’s point of view that matters.  You must understand which problems your customers care about solving, and what solutions are available to your customers today.  You need to understand your competition to make informed decisions about your product.  This is the latest in a series on comparing products – jump back to the beginning of the series to catch up, we’ll wait.

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition. (This Article)
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

No Competition

If you think your product has no competition, you’re wrong.

The photo above shows an El Camino, a combination of a car and a truck.  The front of the vehicle, the cabin interior, and the overall height / profile of the vehicle is that of a car; but the vehicle has a tailgate and a truck bed for hauling stuff.  A pretty novel idea at the time.  There were no other car-truck combination vehicles, but that doesn’t mean the El Camino had no competition.

You should not define competitive products as the products that exactly match the features of your product.  You should not define competitive products as those that solve (exactly) the same set of problems that your product solves.  Every interesting problem already has a solution – and therefore, you have competition.  It may be that there are not any particularly good solutions in the market.  It may be that the competitive products are not well known, and would not be likely to compete effectively.

It may be that the solution your customers choose is “deal with it.”  The “deal with it” solution, at first glance, from a B2C (business – to – consumer) perspective feels like “there is no competition,” but I believe that embracing that point of view puts you at risk of being myopic about your market.  In the B2B (business – to – business) area, “deal with it” is one version of the “in house solution” competitor – a very real scenario.  Stacking ridiculous amounts of stuff on top of your car is analogous to managing your accounts receivable in a spreadsheet.  You can do it, but it is a solution that is fraught with other risks.

Your customer generally has four options from which to choose when deciding if they should buy your product.

1. Buy your product.  Hooray.
2. Buy someone else’s product.
3. Build their own solution.  Your customer believes that rolling their own is a better alternative to purchasing a solution from someone else.
4. Deal with the problem.  The cost of the solution is believed to be higher than the cost of the problem.

How do you identify which products are compelling competitors?

Problem Abstraction

One thing you learn as a product manager, when eliciting requirements, is to keep asking “why?” (and more and more on asking “why?” – check out the comments on these articles).  The goal in elicitation is to find the actual underlying problem, and not just the obvious manifestation of the problem.  One technique you can use is to create an Ishikawa Diagram that helps you define the problem(s) your product is attempting to solve.

Your goal is to find the right level of abstraction at which to determine who your competition is.  If we consider Tim, the hi-tech prosumer who consumes niche content, we would see that the decomposition of his high level goals looks like the following:

[larger image]

There are several benefits to using this approach of describing the decomposition of Tim’s goals:

1. You get traceability of your requirements, allowing you to see why each goal matters, from Tim’s perspective.
2. The graphical layout forces you to be concise in how you describe Tim’s goals.
3. You have a framework for validating that you have defined the complete set of requirements needed to satisfy Tim.
4. Looking at “everything” in one view helps you assure that your requirements are consistent.
5. You know “when to stop” in decomposition when your requirements are atomic.
6. You have a framework for assuring that you focus on those goals that Tim is passionate about.
7. You have tractable goals that you can use to think outside of the box when identifying competitors.
8. You have a clear and visceral tool for communicating with your team, to inspire innovative designs and clever solutions.

Finding Competitors

Some of the competitive solutions will be very straightforward, and obvious based on your understanding of your product domain.  Others you will discover through research.

When we look at the Kindle Fire as a product, we know from Amazon’s positioning that Tim will be able to read books & magazines (in color), consume multimedia content, browse the internet, stream movies, and borrow books.  He’ll have access to “millions of books,” be able to “read anywhere,” and connect with people over email from the device.  The Barnes & Noble Nook Tablet is positioned directly as a competitor to the Kindle Fire (and can even be purchased through Amazon).

It is occasionally annoying to read reviews of the Kindle Fire that basically say “not an iPad killer,” where the reviewer compares the features of the Kindle Fire, a single purpose device with multiple capabilities, with the iPad, a multi-purpose device that can be used for (many) single purposes.  The iPad is a viable competitor for the Kindle Fire, but a comparison in the other direction does not make sense.  The products are asymmetric substitutes – they are only substitutes for each other in a particular context.  If you are Tim, trying to enjoy niche content, then the iPad is a viable competitive product.  If you are a 15 year old boy who wants to play a first person shooter when you’re being shuttled to soccer practice, the Kindle Fire is a non-starter.

Finding a needle in a haystack is easier than identifying competitive products.  You know there is only one needle – once you find it, you stop looking.

Identifying competitive products is more akin to finding out how many needles are in the haystack

You can do research on the internet – start with the positioning from already identified competitors, look for editorials, rumor articles, reviews, research papers, customer reviews, discussion forums.  Search social media streams – find conversations (and broadcasts) about the product and the domain.  This gives you either minimal or overwhelming information, depending on how many competitors are already positioning themselves as products in this market segment.

Interview people who share goals with your personas – they don’t have to match your personas (but they can) – you are trying to discover solutions that you don’t yet know about.  Find out how they address those goals.  Maybe they use a web browser on their desktop computer, cobbling together a cabal of single-purpose solutions.  If they already own a competitive product, find out what they did before they bought that product.  As an example, someone like our Tim persona, before buying a purpose-built product, did the following:

• Used The Sword and Laser podcast to discover new content and get recommendations.
• Used Facebook to get and share suggestions with his friends, and have conversations about his favorite works.
• Used wikipedia and artist individual websites to find other works by his favorite artists.
• Subscribed by email to artists that published serials, blogs, other recurring content.
• Purchased tangible copies of works from Amazon.com.
• Borrowed tangible copies of works from IRL (in-real-life) friends.
• Purchased ebooks in pdf, epub or other formats that could be read on his computer and phone.
• Used Dropbox (or drag-and-drop by USB) to synchronize content across devices.

The problems that people are solving did not come into existence when the solutions were invented. The problems predate the solutions. [Note:if you have a Forrester subscription, check out Mary Gerush's Exploit the Requirements Lifecycle for a more detailed discussion].

The roll your own solution is a viable “competitor” for a B2C product.  In the B2B world, this is usually called best of breed. [Note: it is usually called this by one of the vendors of one of the pieces of the cobbled-together solution].  Using these amalgam-of-products solution strategies can make sense – and for some customers it will be the best solution.

We know that Christina and Time (two of our personas, defined earlier in the series) are different personas – they value different goals differently.  They will also use different approaches to create their own mix and match solutions to their problems.  Previously, I said that you didn’t have to interview people who represented your personas in order to identify competitors.  You do, however, need to figure out which product bundles represent the roll your own solution for each persona.  That is best done by interviewing representative customer prospects.

Our Example Competitors List

In this series, we are showing the mechanics of performing a product comparison, not sharing an actual analysis of the Kindle Fire.  As such, this is (again) where I get to manufacture illustrative data.  Here’s the list of competitive products that could have resulted from the research and analysis described above:

1. The Kindle Fire
2. The Nook Tablet
3. The Kindle Touch
4. The iPad 2
5. A laptop running windows, with a pdf reader, the free kindle app, and an HTML 5-compliant web browser.

The list above is not exhaustive.  Like most analysis activities, you can go on forever, but with diminishing returns.  Where you draw the line is a matter of risk mitigation.  Add as many competitors as you feel you need to accurately assess how well your product (or a future version of your product) will compete in the market.  There is a risk associated with both too much analysis and too little analysis.

Too much analysis slows you down and is expensive.  You will get a comprehensive view, and minimize the risk of missing something, but you run the risk of taking too long.  You also run the risk of never stopping – you can never find all of the needles in the haystack – and therefore you run the risk of never shipping or shipping too late.

Too little analysis saves money and time, but you run the risk of delivering the wrong product.  If you happen to identify all of the competitors that have (or will have) success in the market, you’ll be sufficiently informed (because your view will be a super-set of your customers’ views).  If you overlook a product that is driving the market, you run the risk of watching the world go by through rose colored glasses – you’ll watch people buy that other product, and not understand why they aren’t buying yours.

Here’s where agile provides a perspective that lets you escape the catch-22.  If you do “too much” you can never recover.  The money you spent is a sunk cost.  The time you lost is gone.

If you do too little, you can always do more later, when you realize you need to, as long as your team is capable of embracing change.

Summary

To create a competitive product, you need to know how your product stacks up against the competition – and that means you need to know who the competition is.  A product marketing manager may take this information to emphasize strengths and “turn weaknesses into strengths.”  A product manager can use the insights from this analysis to paint an informed picture of what the product needs to be (and why).

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition. (This article)
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Attributions

Thanks Matthiew Yglesias for the El Camino photo.

Your boss wants a commitment.  You want to offer a prediction.  Agile, you say, only allows you to estimate and predict – not to commit.  ”Horse-hockey!” your boss exclaims, “I want one throat to choke, and it will be yours if you don’t make a commitment and meet it.”  There’s a way to keep yourself off the corporate gallows – estimate, predict, and commit – using agile principles.

This is an article about agile product management and release planning.

Change and Uncertainty

In the dark ages before your team became agile, you would make estimates and commitments.  You never exactly met your commitments, and no one really noticed.  That was how the game was played.  You made a commitment, everyone knew it would be wrong, but they expected it anyway.  Maybe your boss handicapped your commitment, removing scope, lowering expectations, padding the schedule.  Heck, that’s been the recipe for success since they planned the pyramids.

It makes sense.

1. Your early estimates are wrong. When you add them up, the total will be wrong.  If you do PERT estimation, the law of large numbers will help you in aggregate.  But you’ll still be wrong.
2. The outside demands on, and availability of, your people will change. Unplanned sick time, attrition, levels of commitment over time, lots of “people stuff” is really unknown.
3. The needs of your customers will change. Markets evolve over time.  You get smarter, your competitors get better, your customer’s expectations change.

Agile processes are designed to help you deliver what your customer actually needs, not what was originally asked for. Contrast the two worlds.

In the old world, you would commit to delivering a couple pyramids. After spending double your budget, with double the project duration, you would have delivered one pyramid. When you deliver it, you find out that sphinxes are all the rage. Oops.

Your team changed to agile, so that you could deliver the sphinx. But your Pharaoh still wants a commitment to deliver a couple pyramids (the smart ones will be expecting to get just one).  You can stay true to agile, and still mollify your boss’ need to have a commitment, if you take advantage of the first-principles of why agile estimation works.

Estimation

A commitment is a factual prediction of the future.  ”This will take two weeks.”  Nobody is prescient.

A factual prediction has to be nuanced.  ”I expect* this will take no more than two weeks.”

*in reality, this is shorthand for a mathematical prediction, such as “I expect, with 95% confidence, that this will take no more than two weeks.”

Few non-scientist, non-engineers, non-mathematicians understand that 95% confidence has a precise meaning.  People usually interpret it to mean “a 5% chance that it will take more than two weeks.”  What it really means is that if this exact same task were performed twenty thousand times (in a hypothetical world, of course), then nineteen thousand of those times, it would be completed in under two weeks – do you feel lucky?

To make a statement like this, you actually have to create a PERT estimate – identifying the best-case, worst-case, and most-likely case for how long a task will take.

Unfortunately, we’re rarely asked to make a commitment about a single task – but rather a large collection of tasks – well-defined, ill-defined, and undefined.

You can combine PERT estimates for the individual tasks, resulting in an overall estimate of the collection of tasks.

The beauty of this approach is that the central limit theorem, and the law of large numbers, work to help you estimate a collection of tasks – you can actually provide better estimates of a group of tasks than a single task.  This obviously helps with the well-defined tasks that you know about at the start of the project.  This even helps with the ill-defined tasks.  Rationalists will argue that the key, then, is to do more up-front research to discover the undefined tasks – and then we’re set.  As Frederick Brooks (Mythical Man-Month) points out in The Design of Design, this debate has been going on since Descartes and Locke.  It is not a new idea.

Big Up-Front Design and Requirements (BUFD & BUFR) hasn’t worked particularly well, so far.

Don’t throw out the baby with the bath-water, however.  The math of estimation is still important and useful.  Even if empiricism is not the silver bullet.

Prediction

Estimation is a form of prediction.  Even agile teams do it.  In Scrum, you estimate a collection of user stories – in story points that represent complexity, and you predict how many points the team can complete in this sprint.  Note the time factor.  If you’re working a two-week sprint, there is very little risk of changes in staffing during a two-week period.  There’s also very little risk that your market will change significantly in two weeks – and if it does, what are the odds that you will notice and materially change your requirements in two weeks?

Visually, let’s take that PERT estimate and turn it sideways – so we can introduce the dimension of time.  Imagine you estimated all of the tasks (well-defined, ill-defined, and a guess about the undefined), as if they were all to happen in the first sprint.  Ignore inter-task dependencies, and pretend you had unlimited resources and the ability to perform all tasks in parallel.

The graph above shows the aggregate estimate – the circle is your best prediction, with error bars representing your confidence interval in the estimate.  If you were using PERT estimates, these could represent that 5% and 95% confidence lines.  Subjectively pick something based on your team’s experience in the domain and your confidence in your guesses (about the undefined tasks).

We need a segue into the “best of waterfall” approach to estimating projects, to steal and invert a good idea.

The Cone of Uncertainty

The folks at Construx have published a nice explanation of the cone of uncertainty – an adaptation of an idea from Steven McConnell’s Software Estimation: Demystifying The Black Art (2006).  That article uses his imagery with permission – so please go look at it there.  The idea is that as the project becomes better defined (e.g. during the project), the amount of uncertainty is reduced.

The findings show that initial estimates are off by 400% (either low by a factor of 4 or high by a factor of 4)!  Even after “nailing down” requirements, estimates are still off by 30% to 50%!

As bad as that sounds, it is actually worse.  This is a prediction for the original project (delivering pyramids).  Not only are your estimates wrong – but they are bad estimates for delivering the wrong product.

But – the core idea is sound – the further into the future you have to execute, the greater the mistakes in your estimate.

Taking that concept, and applying it to our diagram, we get the following:

The further into the future you are trying to predict, the less accuracy you have in your prediction.  This reduction in accuracy is reflected as a widening of the confidence bands for your estimate.

• A couple sprints’ worth of work is not much different than one sprint – so your estimation range is not much changed.
• An entire release of sprints (say 6 to 10 sprints) has much more opportunity for the unknown to rear its head.

Now, your prediction is (probably) unusably vague and imprecise.  ”This set of tasks will take X plus or minus a factor of two.”

That’s the reality.

Note: This has always been the reality.  People have historically reduced this “risk to timing” by hiding the “risk of change” aspects – and waterfall processes encourage you to deliver the wrong thing, as close to on-time as possible.

That’s not what we want to do, however.

We still want to deliver the (not-yet-defined) right product, as efficiently as possible.  That’s the goal of agile.  (For folks who haven’t been here at Tyner Blain for long – “right” includes both value and quality).

Refinement

Because we’re agile, and we’re willing to “get smarter” about our product over time, we have an opportunity to improve.  Because of the nature of compounding estimates and the cone of uncertainty, our uncertainty gets smaller over time.

Let’s remove our artificial simplification that we could do everything “right now” and look at what we think we know right now, about the end of the release.

Our ability to predict the amount of effort (for today’s definition of the product) at the end of the release is not very good.

Our ability to predict (today’s definition of the product) one sprint into the future is much better.

After completing the first sprint, we are a little bit smarter – the ill-defined tasks are better defined.  Maybe some of the undefined tasks are now ill-defined.  The same cone of uncertainty is now a little bit smaller – we are a little bit smarter, and the time horizon of the release date is a little bit closer.

The trend continues – each sprint gets us closer to the release date, and with each sprint (assuming we get feedback from our customers, and continue to study our markets) we get a little bit smarter.  We also get better at predicting the team’s velocity (how much “product” they can deliver during each sprint).

Commitment

Your boss still wants a commitment, however.  And that’s where we get to change the way we look at this (again).

The above diagrams all display how we converge on an estimate for a stable body of work.  However, we know that the body of work is constantly changing.

Backlog! [you say]

Yes!  The backlog.  The backlog is an ordered, prioritized list of user stories and bugs.  I was talking with Luke Hohmann of Innovation Games last month, and one of the most popular online Innovation Games is now the one they created based on prioritizing by bang for the buck.  Play it today online (for free!).  How cool is that?

The backlog represents the work the team is going to do – in the order in which the team is going to do it.  Over time, as we get smarter, we will add and remove items from the backlog – because we discover new capabilities that are important, and because we learn that some things aren’t worth doing.  We will even re-order the backlog as we recognize shifting priorities in the markets (or in our changing strategy).

As this happens, it turns out that the items at the top of the list are least likely to get displaced, and therefore most likely to still be part of the product by the time we get to the release.

Instead of thinking about uncertainty in terms of how long it takes, think about uncertainty in terms of how much we complete in a fixed amount of time. In agile, generally, we apply a timebox approach to determining what gets built.

Now, uncertainty, instead of manifesting as “when do we finish?” becomes “what will we finish?”

Your boss is rational.  She appreciates the constraints, she just wants to know what you can commit.  Every boss I’ve worked with has been willing (sometimes only after much discussion) to treat this uncertainty in terms of what instead of when.  They acknowledge that they need to translate (usually for their boss) into a “fixed” commitment.

The solution: commit to a subset of what you predict you can complete.

At the start of the release, you may have 500 points worth of stories.  Based on your team’s expected velocity, and the number of sprints in the release, you predict that you can complete 320 points worth of stories (5 people on the team, a team velocity of 40 points per sprint, and 8 sprints in the release).  Starting at the top of the backlog and working down, draw a cut-line at the last story you can complete (when you reach 320 points).  This is your prediction.

Now the commitment part.  You’ll have to figure out what you’re comfortable with.  Maybe for 8 sprints (say, 16 weeks into the future), you may only be comfortable committing to half that amount – 160 points.  Go back to the top of the backlog, and count down until you reach 160 points.  Everything above the line is what you commit to delivering.

Maybe you are comfortable committing to 240 points, maybe only 80.  This is like playing spades.  The more you can commit to, without missing, the better off you are.  Your tolerance for risk is different than mine.

You can also negotiate with your boss.  Commit to 160 points now, and provide an update after every other sprint.  More likely than not, you will be increasing the scope of your commitment with every update.

Mid-project updates of “we can do more” are always better than “we can do less.”  And both are better than end-of-project surprises.  This also allows you to have updates that look like this:

We didn’t know this at the start of the release, but X is really important to our customers – and we will be able to deliver X in addition to what we already committed.  Without slipping the release date.

Conclusion

Making commitments with an agile process is not impossible.  It just needs to be approached differently (if you want to stay true to agile).  The end result: better predictions, more realistic commitments, and the likelihood that each update will be good news instead of bad.

[Update: Changed initial image.  Thanks Dennis for the great photo!]

Intellectual Property.  The legal jargon definition of this term has come to effectively mean “something I’ve patented, copyrighted, or hold as a trade secret.”  A more general interpretation is “an idea.”  For product managers, the most valuable ideas are insights.

The Value of Insight as Intellectual Property

Last week, the folks at Ryma Technologies Solutions invited me back to present another webinar as part of their GrandView Product Management View webinar series.  [My previous PMV webinar was on Kano Analysis] Thanks to Val and Bradley and Johnny for the invitation, and for making it a great experience!

You can watch the video version, listen to the audio, or get the slides in pdf format from the GrandView site.  If you want to get a 5-minute sample to try before you buy (with your attention, that is – this is 100% free), check out the sampler on youtube.com.

Or – if you want to flip through the slides (also on slideshare) now, and then decide – go right ahead:

Or go directly to the page on slideshare.net.

The main idea in the presentation is that your understanding of your market is the information that is most valuable.  There are several examples of this type of information in the presentation – you can see the visuals in the slides, but listening is probably required to explain most of them.

The Contentious Idea

I got some feedback that my central theorem – that insights are more valuable than patents – would be controversial.  I addressed this a little bit in the webinar.  Expanding on those thoughts here might serve to get a debate and conversation going.

I’m a former engineer (electro-mechanical controls design), and I hold 4 patents.  Each of those patents represents a body of “protected information” describing a solution to a problem.  A very specific solution.  Those patents have value.  But those patented solutions are not the only solutions to those problems.

When I was taking economics classes in college, I learned about the notions of substitute and complementary goods.  Once I became a design engineer, and learned how patents work, I also learned how to get around patents.  By designing perfect substitutes.  The process is pretty simple:

1. Pick a patented device.
2. Understand what it does – what function does it perform (and precisely how – this is the patented part)?
3. Understand what problem it was designed to solve – note: this part is not protected information.
4. Determine the important performance characteristics for solving the problem in (3).
5. Invent another way to solve the problem from (3), that meets the criteria from (4), without copying (2).

That’s it.

Every problem can be solved in many ways.  And when those alternatives all solve the problem equivalently, those solutions become perfect substitutes.  That makes any single solution effectively a commodity.  You can’t intentionally create a valuable solution to the problem without understanding the problem.

The value is in the understanding of the problem, and your market’s willingness to pay to solve it.  Not in having protection of one of the infinite number of possible solutions.  That’s the point I make in the webinar, before quickly moving to examples that show ways of developing insights about your customers [note: the examples are influenced by my approach using a customer-centric market model].

Disagree?

Chime in below or on the GrandView site, and let’s have a great discussion.  And as always, thanks for listening/watching/reading!

A lot of teams that I’ve worked on and with get hung up when thinking about defining requirements for “migration projects” and “system upgrades.”  There’s some intangible barrier to being market focused when it comes to improving existing internal systems.  Every new product represents a solution to an existing problem.  Why do so many projects move forward with teams that are blind to the actual requirements?

New Products

Even “revolutionary” products like the iPod are just addressing existing market problems.  Part of what makes this statement true is looking at things from a market perspective – thinking about the valuable problems that people are willing to pay to solve.  Even if the iPod were the first mp3 player (it wasn’t), it would still only be an improvement.  The first mp3 player was not new, it provided an improved way to listen to your music on the go.  You could put your whole music library in your pocket.  Much better than a stack of cassettes melting in the glove box of your car, or a tape getting caught on your keys and unraveled when you pull it out of your backpack.

What’s important, and difficult (especially for people with technology backgrounds), is to think about it in terms of what people are trying to accomplish – not how they are trying to accomplish it.  In a business process view, it is the difference between process (why) and procedure (how).

There’s a continuum of migration projects, ranging from completely new to identical processes.  Neither extreme technically exists – think of it as a range from infinite change in the existing process to 1 / (infinite change) in the existing process.

Near the “completely new process” /infinite change end of the spectrum are processes that are completely new to you.  Remember, you’re solving a problem, perhaps in a very innovative way, that people are already solving some other way.  You’re just providing a better solution approach.

Near the identical process end of the spectrum are projects are “pin-compatible” platform migrations and near-sighted legacy system migrations.  Moving from an old gas-guzzling car to a new, more efficient model is a good example.  I mention “near sighted” because that old system was designed to meet an old set of market needs, so the new system will, by definition, not meet current market needs – it will only meet the old market needs.  Pragmatically, when considering organizational change, it may make sense to do your system upgrade in two stages: migrate the systems (“nearly identical process”) and deal with all the gotchas of migration first, then start re-engineering the processes and optimizing the procedures to address new market needs (major and minor process changes, respectively) second.

Set aside the possible, possibly rational rationale* to migrate the implementation first, and the functionality second.  Consider that a deployment and logistics detail and get back to the problem at hand.  Most of the times I’ve been involved in system migrations, they have been initiated as cost-savings platform projects.  Over the last few years, more often the migration project gets prioritized as a means to an end – “doing new thing X is prohibitively expensive on the old system.”  That opens the door to talking about goals.

* Couldn’t pass that one up.

Capturing Goals

On a recent project to migrate part of a company’s operations from one platform to another (system consolidation, after an acquisition), I created the following Ishikawa diagram to represent the goals for the migration project.

Even a “don’t change anything” project has real underlying goals.  Once you discover them, you open the door to having conversations about making things better.  Might be a “do it later” situation, but often, there is an opportunity to grab some of the low-hanging fruit during the system implementation.

The most important reason to capture the goals when “everyone already knows the goals” – aside from the fact that that is never true – is to make sure decisions are being made eyes-open.

Requirements live outside of the timeline of a particular project.  Once you’ve identified those requirements (aka market needs / company strategy), the next question is to determine which of those goals this project is intended to support or advance.  That question drives a lot of clarity into strategic thinking, and those answers drive a lot of clarity into project execution.

When you don’t share the context of the goals of the project, you are effectively blindfolding your team.  You prevent them from discovering opportunities to make things better.  You prevent them from making the right choice, when decisions are otherwise (without the context of goals) arbitrary.

For example, we talk about satisficing as a means to know when to ship – when it is “good enough.”  Without context, the opinion of “good enough” comes from someone on the team, without guidance.  In an agile environment, with self-directed teams, you’re making a trust-based decision to explicitly empower the team to make those decisions.  Don’t you want to give them some information, to help them make that decision?

Conclusion

This article really only explores part of the process of making real, large, projects happen in large environments.  That is a topic I’ll be talking about in a couple months.  You won’t get a sense of sated accomplishment from reading just this article – too much of the end-to-end, start-to-finish of real projects is unaddressed.

You will, however, know how to start the project.  Define the goals.  Someone will tell you the goal is to “Copy what the old system did.  We don’t have time to re-engineer.  Why are you bothering me?  Let me know if I need to find someone who can get the job done.”

Now you know how to respond.

**Attributions

Thanks foxtongue for the blindfolded typists photo

Estimating the “value” of features is a waste of time.  I was in a JAD session once where people argued about if the annoying beeping (audible on the conference line) was a smoke alarm or a fire alarm.  Yes, you can get to an answer, but so what?! The important thing is to solve the problem.

Solutions Versus Features

Everyone on that conference call had an immediate and visceral appreciation of the value of making the beeping stop.  That’s the power of solving a problem.  The methods of solving the problem – mute the offender, replace the battery, throw the alarm out the window – do not have implicit value.  They have an indirect value, in an “end justifies the means” kind of way.  But not direct value.

The same sort of thing applies when talking about prioritizing features.  Eric Krock (@voximate) just wrote a really good article, Per-Feature ROI Is (Usually) a Stupid Waste of Time, where he does two great things, and (barely) missed an opportunity for a hat trick.

The first great thing Eric did was look at the challenges of determining relative (ordinal or cardinal) value of “several things.”  He points out several real world challenges:

1. When you have a product with several things already and you want to determine the value of yet another thing – how do you allocate a portion of future revenue to the new thing versus the things you already have?
2. When thing A and thing B have to be delivered together, to realize value, how do you prioritize things A & B?  Relative to each other?
3. The opportunity cost of having your product manager do a valuation exercise on a bunch of things is high.  She could be doing more valuable things.
4. You won’t perform a retrospective on the accuracy of your valuation.  So you won’t know if it was a waste of time, and you won’t get better at future exercises.

The second great thing Eric did was reference a Tyner Blain article from early 2007 on measuring the costs of features.  I mean “great” on three levels.

1. As a joke (for folks who don’t know me, figured I’d mention that I’m kidding, just in case you get the wrong idea).
2. There is some good stuff in that earlier costing article about allocation of fixed and variable costs (with a handy reminder.
3. Eric’s article gives me an opportunity to shudder at the language I was using in 2007, see how much some of my thinking has evolved in four years, and improve a bit of it here and now.

What Eric slightly missed is the same thing I completely missed in 2007 – features don’t have inherent value.  Solutions to problems do have value.  He only slightly missed it because he got the problem manifestation right – it takes a lot of effort, for little reward, to spend time thinking about what features are worth.  I also missed the opportunity in an article looking at utility curves as an approach to estimating benefits, written two days after the one on cost allocation.  We were both so close!

People don’t buy features.  They buy solutions.

Valuing Solutions Instead of Features

Estimating the value of solutions addresses a lot of the real problems that Eric calls out.  It also has a side benefit of keeping your perspective outside-in versus inside-out.  Or as others often say, it keeps you “market driven.”

Anything that you’re doing, as a product manager, that has you focused on understanding your market and your customers and their problems is a good thing.  It may even be the most important thing.  I would contend that it eliminates objection 3 – the opportunity cost of estimating the value of solutions is minimal or zero.  There may be activities with more urgency, but off the top of my head, none that are more important, for a product manager.

Comment if I’m missing something (it’s late and I just got home from another week on the road).

The way I approach determining the value of a solution is by developing a point of view about how much incremental profit I will get when my product starts solving this additional problem.  Revenue can increase from additional sales, or from the ability to increase prices.  Cost can increase if new marketing and other operations (launches, PR campaigns, etc) are required to realize the incremental revenue.

I start with a customer-centric market model.

A given solution, or improved solution (as in “solves the problem better,” or “solves more of the problem”) – which only applies to some problems – is interesting to some customers, in some market segments.

A solution has value when it brings in incremental customers, in a targeted market segment.  It also has value when it reduces or prevents erosion of your current customer base (in a SaaS or maintenance-revenue model) to competitive solutions.

The time you spend thinking about buyer and user personas, the problems they care about, and the nature of those problems (which varies by persona) is not time wasted – or even spent “at the cost of doing something else.”

To make this useful, you have to have a forecast – without solution A, we will sell X; with solution A we will sell Y (and to whom).  A good product manager will be looking at sales, and will be able to reconcile the sales with the projections.  That helps with objection 4 (but doesn’t completely address it – you don’t know if your projections were accurate, so you can’t really know if your estimation is accurate).

This also helps you deal with challenge #1.  You’ve got a model that says “the current product works great for high school students, but not college students, because they also have problem A, which they solve today by…” Your intention is to create solution A, making your product viable to college students.  Allocate the incremental profits from college-student sales to solution A.

My approach to challenge #2 is a little more tactical.

Coupled Solutions

There are a couple ways that Eric’s “must deliver A and B” scenario are interesting, when looking at the value of solutions.

Scenario 1: Solution A solves part of problem X for persona M.  Solution B solves part of problem X for persona M.  Combined, they solve more of problem X for persona M.

This makes sense for “more is better” problems – where “more” solution yields “more” value.

In this case, I have a forecast (the more time I spend on it, the better it will be) that maps incremental sales to improved solutions.  The “first” solution to be released will have more value than the second.  If they are being released together, then I don’t care about the allocation – I combine them.

Scenario 2: If, however, the two solutions are valuable to different personas, then I treat them separately – even if they solve “the same problem,” it is not the same problem (for the same person).

Conclusion

Prioritization by “Bang For the Buck” is worth doing.  Just make sure you are prioritizing solutions, not features.

Also note: this article talked about valuation – what you do with that valuation, prioritizing by market, can be trickier.

Agile values working software over comprehensive documentation – it is 1/4th of the original manifesto.  That doesn’t mean don’t document!  It means don’t document more than you need to document.  Documentation does have value, but the practice of documenting got excessive – that’s why a reaction to the bad stuff earned a spot as one of the pillars of agile.  How do you avoid over-reacting when changing a culture of over-documentation?

The Need for Documentation

We need documentation to help us communicate.  You can define an agile team as “the people creating the product.”  You can interpret “creating” as the people we normally think of as executing to accomplish the goals, or you can be more inclusive, and think of the people who have the goals as being part of the team too.

Philosophically, agile stresses collaboration.  Usually it is couched in terms of (1) people who are executing collaborate to devise the best solutions and (2) people who are part of the team collaborate with the people for whom they are creating the product.  Personally, my experience has been that projects with committed sponsors succeed, and projects without them fail – so I always think of the sponsors as part of the team.

Regardless of your definition – collaboration requires communication, and communication benefits from documentation.

Temporal Communication

Communication among groups of people happens over time.

Some collaboration is transient – communication happens right now, and is only important right now.  Other communications are persistent – the collaboration happens right now, but we need to remember later what we agreed upon and why.  There are variations of “right now” and varying degrees of “later”, but slightly oversimplifying,

• There is communication for now.
• There is communication for later.

Both are important.

Documentation, in the vision from your nightmares – giant requirements documents, architectural analyses, and detailed psychographic profiles – is very inefficient when communicating for now.

These massive documents, while getting in the way of a conversation, do provide the opportunity to remember (in the future) why we make the decisions we make (today).

Documentation, as emphasized in most agile discussions – user stories and acceptance criteria, on an index card, taped on the wall – is not a robust solution for communication that happens later.

This low-overhead approach to capturing today’s ideas actually facilitates and improves today’s conversation – but does not give us a record we can use in the future for why we made our decisions.

As an agile team, there are ways we can approach both the persistent and transient documentation tactics to make them more valuable.  We can tweak our transient documents to make them more useful for persistence.  We can take an agile approach to development of persistent documents, so that we only do just enough documentation – trading off the minimum amount of short term efficiency to avoid long-term blunders.

Transient Documents for the Future

The image above is from a workshop defining the product backlog for an initial sprint for a new project.  The team identified the most important stories for the first sprint, their acceptance criteria, and the size, in story points, for each story.  As part of scrum, the stories are written in user-story format, with acceptance criteria for each story identified and documented on additional index cards (taped to the index cards for the associated stories).  Great transient way to capture the must do and makes it better stories that the product owner is thinking about.

As a team, we made a couple minor tweaks to make these story cards more useful.  First, all of the stories were organized into themes that help establish the context (for the team), and combine the stories into “meaningful areas of focus” for communication with stakeholders.

Each story got a blob of color in the top left hand corner, identifying which of the five themes were being supported.  You can see a few of them circled in the image above.

We also developed a set of 8 proto-personas (prototype, early draft, very rough personas – almost stereotypes) that the overall product / project was designed to support.  Each persona got a different color star.

Each user story got a corresponding colored-star sticker to indicate for which persona the story has been written.  Specifically, when multiple personas could use a story, we identified for which persona we were implementing the story in this sprint.

In the “right now” world, these tweaks helped us understand, question, and confirm the themes and personas being supported in the first story.  Those conversations allowed us to reach pragmatic compromises about which stakeholders were going to benefit and when.  They also allowed us to craft the outbound message that is step one of stakeholder expectation management.

The list of themes changed while we were doing this – moving from 4 to 5 themes, as we realized that there was value in splitting one of the themes.  The group of proto-personas grew from an initial set of 3 to a set of 8, as we quickly realized that the team was trying to “build something for everyone” and we wanted to avoid the elastic user problem, and instead, roll out capabilities sequentially that satisfied valuable groups of users with similar goals and perspectives.

Culture Change for Stakeholders

As you would expect, having a bunch of cards stuck on the wall provides a great, visceral, now communication for the team – both in implementing and communicating.  However, cards on the wall is very transient.  It also doesn’t provide a good way to communicate with old-school stakeholders (imagine the execs who have their admins print out their emails for them to read and annotate – those folks still exist!).

This can make it hard for the “not part of the implementation” part of the team to collaborate and communicate.

In User Stories Applied, Mike Cohn stresses that the brevity of user stories is intentionally designed to facilitate (and I would add “dependent upon”) conversation.  I find it to be both a strength and a weakness of user stories.  It is strong because you can cover a lot of ground quickly (breadth) and capture a number of stories, much like the list above.  It is weak because the requirements documentation does not stand on its own – it requires conversation to fill in the details.  I’ve had some success using the “Verify that…” user acceptance tests as the method of documenting those details (depth), in conjunction with the brief, easily consumable stories.

Stakeholders in a Barrel

Those conversations happened as part of getting the stories defined, getting them on the board in the right order, and defining the acceptance criteria.  Stakeholders often have a short memory, especially when they were the ones making a compromise.  You need some way to retain the context of the conversation that led to the particular organization and content of the stories on the wall.

Persistent Documents in the Present

Agile proponents complain about the massive get-in-your-way documents that slow down the start of projects, prevent the teams from adapting to changing market conditions, and otherwise make it harder to deliver great products.  I agree.  Agile proponents also rail against the “build it all, then release it,” waterfall, process for creating software – and propose an alternative – iterative development of the software.  I agree again.

Why not apply the same principle of iterative development to persistent documents?

Consider the example above of using colored-star stickers to identify the people for whom each story is being implemented in any given sprint.  The majority of the time spent on this analysis is around understanding which people are important (to the success of the product), and for which people the ability to perform this user story is important (to the person).  The combination of these two importances is an input to prioritization.  A minimal amount of time is spent putting stickers on cards, and creating multiple cards for the same story (each with a different sticker for a different persona and potentially different acceptance criteria).

Add a tiny bit of overhead, and capture the information in a spreadsheet (or whatever).

The image above shows a mapping of use cases to actors.  It could just as easily show a mapping of user stories to personas.  The transition from thinking about actors to thinking about personas is a small one.  This chart comes from an earlier article on communicating a delivery schedule with use cases – it helps stakeholders get a big picture view of who benefits when – a user-centric, yet still top-down perspective.

Note: Another nuance to incremental delivery is that you can introduce the “bare bones” version of a story now, and the “better” version of it later.  You can also communicate that stories get better over time, for some users, with the same type of chart.

A very similar, and also simple to create table can show how each theme is getting “representation” within each sprint.  That view facilitates great discussions about trying to emphasize one theme and then another sequentially, versus doing the most important items in each theme first – gradually making progress in each theme.  Creating that view, and using it to communicate, has helped me address concerns that “external” stakeholders have shared about how a team appeared to be “chaotic and random” in their approach to prioritization and implementation.

Many Models

There are many types of documentation – and many types of models for emphasizing and discussing particular aspects of a complex system (like users and goals and solutions).  Creating a simple diagram like the following whiteboard sketch is almost required for effective transient communication in some projects.

[from Simple Agile Model Example]

When most folks are complaining about documentation, they are not complaining about the sketch above.  They would consider that sketch to be augmenting a conversation.  Fine.  That’s what I did, at the time.  Then I took a picture of it.  Suddenly, it was documentation.  A stakeholder came by later, confirmed that the system needed to work this way, and signed the whiteboard – and the photo was updated.

Every heavyweight document can start out this way.  And it can evolve.  It should evolve.  If you’re going to succeed, it must evolve as your team gets smarter, your competitors react, and your market changes.

The trick is that you don’t have to write the final version before you get started.  Capture at a high level what you know right now – just like a roadmap.  Capture in just enough detail what you need right now – just like a story backlog.  And change it as you go.

Conclusion

Documentation is not bad.  Documenting stuff you will never use is bad.  Documenting stuff you don’t need for a long time is risky, because it will probably change before you refer back to your document.

Documenting why you made decisions right now, as part of transient collaboration and communication is important, so that you collectively remember.  That documentation persists and your team can build upon knowledge, incrementally – just as your team builds upon the evolving code base, incrementally.

Ask someone what they want, and they’ll tell you they want a faster horse.  Provoke them, and they’ll tell you they have a ‘get there faster’ problem, an ‘equine waste disposal’ problem, and issues with total cost of ownership.

Thought Provoking

If your requirements elicitation session looks like the photo above, you’re doing it wrong.  However, just asking people what they want and confirming that you heard what they said is also not enough.  Active listening is important, but to capture great requirements, you also have to provoke thought about why someone is expressing a “requirement.”

Adrian Reed wrote a great article this week (hat tip to Kevin Brennan) on asking provoking questions that leverage lateral thinking techniques to get better insight into the true requirements.  Adrian presents eight questions, such as “Imagine if we fast-forward to 2 years after the implementation of this project, what will the organisation look like?”  Some of his questions remind me a lot of the ideas behind Enthiosys’ Innovation Games (and Luke Hohmann’s Innovation Games book).  The remember the future, and product box games immediately come to mind.

Unprovoked Thoughts

Most good subject matter experts I’ve met, when asked about the important problems to be solved, try and be really helpful and incorporate elements of solutions in their descriptions of problems.  They will say things like “the system must integrate with [other system] to do X.”  They may even ultimately be right, that this particular system integration is a constraint, and that “X” is the only acceptable (by policy) way to achieve “Y.”  But usually, neither constraint is a requirement – it is a solution approach.

Subject matter experts who are not as good at having and sharing insights about their domain often confuse problem manifestations with their underlying problems.  By analogy, it is requesting treatment for a runny nose, when the problem the you have is the flu.  You can dry up your nose and still feel horrible.

Provoking Questions Reveal Real Problems

Adrian’s questions are designed to help you understand that you’re treating the flu, and not a runny nose.  Requirements gathering is a lot like diagnosis of a medical malady.  You have to discover the real problems.  The problems that people are willing to pay to solve.  You have to uncover the latent problems that are “hidden” behind problem manifestations.

In a (rare for me) American football analogy – the problem manifestation is that your quarterback is completing 1 of 20 forward passes.  Replacing the quarterback and receivers will not solve the problem.  The problem is that your offensive line is not able to give the quarterback sufficient time to throw higher-probability-of-success passes.

Asking questions that force people to describe their objectives differently is a good way to bypass solution-design answers.  It also creates chinks in the armor of problem manifestations.  Completing more passes is not the future you’re looking for, winning more games is the goal.  When you’re treating your flu, your goal is not to be sick – but with a dry nose.  Your goal is to be well.  When you ask someone to remember the future, they will will describe being not sick, not being dry-nosed.  The product box will be a description of a winning team.

Check out Adrian’s list of questions, and ask yourself, how do you get to the root causes?  Ishikawa diagrams (also known as cause and effect or fishbone diagrams) provide a great visualization tool if you’re a spatial thinker or a whiteboard-talker.  In the example below, you can quickly see that spending too much on fuel is part of the real problem – that the cost of operation is too high.  You can likewise see that under-inflated tires are a source of poor fuel economy.  Check out the Ishikawa article for an explanation, or this article on providing context (with Ishikawa diagrams), and this article on buyer and user personas for more examples of problem decomposition.

If you’ve got any examples of problem-statement-turned-problem, chime in below…

Almost everything I’ve read about use cases focuses on describing what needs to be added to your product.  Agile development says “get it working first, make it better second.”  That means changing the way the software enables a user to do something they can already do.  How do you manage requirements for incremental improvement?

Iterative Development and Incremental Improvement

Iterative development is the process by which you release software in iterations – small, incrementally better versions of your software.  People usually think about this solely in terms of enable the most valuable capabilities first, then the next most valuable capability next.

That’s fine, when your product is new.  Eventually (or quickly!) you will reach the point when the next most valuable improvement is not to add a new capability, but rather, to make an existing capability more valuable.

Everything is an Upgrade

In a couple earlier articles about organizing migration projects and requirements for migration projects, I wrote about how migration is a bit of a misnomer.  Everything is a migration.

My argument in those earlier articles is that the problems you’re solving with your “new” solution are not new.  Your customers are currently solving them in different ways.  The relevant question for migration projects is around how much your users are changing the way they solve their problems.

[As a slight segue, failing to appreciate this distinction can lead you to a project that is based on a faulty problem statement.]

When you are delivering incremental improvements to your product, through iterative development, you are upgrading your software.  You are migrating your users from their old solution to their new solution.  It just so happens that you are replacing yourself, as you migrate them from the old version to the new one.

A viable strategy, that appeals to me personally, is to continuously innovate, disrupting your market, before someone else does.  With this approach, you are intentionally reinventing your solution, making it difficult for someone else to out-innovate you.  This is a great way to approach developing your roadmap strategically.

Your market will change.  Rapidly. Do you want to react to your competitors, or keep them on their heels while you stay on the balls of your feet?

The reverberations of the “new” Twitter still haven’t settled, and many of Twitter’s competitors are on their heels – some of them only now realizing that Twitter has always been a competitor and not just “a platform.”  The new Twitter doesn’t really change anything about how people use Twitter, except make it (markedly) better.

Avoiding Featuritis

When is it more valuable to improve what your software already enables, versus enabling users to accomplish more with your software?  Kathy Sierra introduced us to the concept of featuritis, the idea that too much is too much.

In an article on viral product management, I proposed that by making your software better, you can tap into an altruistic mechanism by which people will promote your product.  As an example, I proposed improving the usability of your software as a means to cross this viral tipping point.

One of the key ideas is that making it better for users makes your product better for your company.

Managing “Improvement” Requirements

Now that we all agree that there is value in making your software better [comment below if you disagree] the question becomes “How?!”

Use Cases can be used as the keystone to your requirements arch.  Sometimes, user stories are better than use cases, but use cases are perfectly agile.  When you’re starting a project, you want to apply an outside-in approach to defining the scope of that project – or more particularly, to define the scope of problems you are solving in your roadmap. This article will talk in the language of use cases, but identical concepts and approaches apply when using user stories as well.

When you have prioritized a use case as “the next most valuable capability to introduce” you add it to the backlog for your sprint.  The implementation team then provides feedback that use case is “too big” and you have to slice it up.  It is important that you don’t split the use case  in such a way that you only solve half the problem.

You also shouldn’t deliver a “bad” product.  In fact, the ideal way to introduce a new capability is to satisfice, and not introduce the “perfect” solution in your first iteration.  Make your first solution “good enough.”  Part of the art is defining “good enough,” but rest assured, it is not the same as “the best you can possibly do.”

Summarizing some key elements from above demonstrates the eventual reality you will face:

1. Every problem* your users face is already being solved, you are just providing a better solution.
2. There is diminishing, and eventually negative return on adding more capabilities to your product.
3. Your first release of a given capability will only be “good enough.”  That leaves room for improvement.

*Yes, you can pedantically define “the problem” as “the existing solution is not good enough” – but you still end up in the same place, so why bother?

Eventually, the value of improving something you’ve already released will be greater than the value of releasing something new.

What do you do?  You implement the same use case again.

If your product is Software as a Service (SaaS), you absolutely should be thinking about things this way.  Becoming better is a continuous improvement objective that is implicit in the economics of SaaS products.

Revisiting Use Cases

Not to be confused with re-use of use cases, revisiting a use case is specifically revisiting – with the goal of improving – the implementation that is already in place within your product.  This is a special case of a migration project – you’re migrating your users from your old solution to your new and improved solution.

You may be migrating to an identical process, perhaps with faster performance than your previous release was capable of.  Or you may be improving the procedure (improving usability, interaction design, visual design, etc) without affecting the process. Generally, a “make it better” use case improvement effort will have no more than a minor process impact.  Your users are still doing the same thing, they are just enjoying it more, or doing it more effectively.

When slicing use cases to make them fit within a single sprint, you may design them for a single user persona in the first release, knowing that you won’t meet the expectations of a different user persona until the next release.  In this case, your non-functional requirements, constraints, or acceptance criteria will be different.  Your use case may also be different (in the details) while appearing to be the same.  This also common scenario can be addressed with the same approach.

Find that old use case (from several iterations ago) and put it back in the backlog.  Remember, agile is about conversation, not artifacts.  If you need to add an explanation to the use case, because your team fixates on the fact that it is “already working,” add a qualifier that it needs to be “better.”  Then have a conversation, and explain how it needs to be better.

Is Design Part of Implementation?

Some teams organize such that designers (both architects designing code and designers designing interfaces are being addressed here) are part of the implementation team.  Other companies treat designers as stakeholders – providing guidance and input to the product managers and product owners.

When a “new design” is being requested from outside the team, include that design guidance as part of the input to the implementation team – through artifact and / or clarification.

When a “new design” is being requested of the team (the designer is part of the team), express the new acceptance criteria to the team.  Note that these need to be measurable requirements if they are to be considered good requirements.

Summary

There will come a time when the most valuable thing you can do is improve the user experience for a capability your product already embodies.  When that time comes, use the same use case again, updated with the new constraints, non-functional requirements, and acceptance criteria.

A market can be thought of as the collection of contexts in which you might sell your product. You can split your market into a set of market segments. Each of those segments represents a group of customers, each of whom shares a set of problems for which they would pay for solutions.

Customer-Centric Market Model

I’ve been developing a model for describing markets that emphasizes the critical customer-centric perspective that product managers need to have.  This article is a first draft of that model – I’m sharing it here in hopes of getting feedback to improve the model and my approach for communicating it.

Key goals of the model:

• Provide a framework that helps product managers make decisions about products.
• Establish an outside-in bias that encourages product managers to think first about customers and second about implementation.
• Encourage teams to design products that solve real problems that people will pay to solve.
• Support both Agile and Waterfall development processes.

Please comment below (or privately) with any feedback about the model or presentation thereof (both prose and graphics).  Thanks in advance!

A Model for Markets, Segments, Customers, and Problems

Markets and Market Segments

Market segments are traditionally defined based on aggregate data that makes it easy to operationally address groups of customers.  Business-to-business (B2B) and business-to-consumer (B2C) markets are typically divided using the same approach, with a few data sources that are uniquely relevant for each type of company-to-customer relationship.

[larger image]

Business-To-Business (B2B)

When defining market segments for B2B products, your customers are companies.  Groups of companies are defined, and organizational structures are built around those segmentation models.  Different factors and combinations of factors are used to define the different market segments. Common B2B segmentation factors include:

• NAICS Industry Definitions – A standard taxonomy  for defining the different businesses in which your customers engage.
• Demographic Data – For companies, this can be number of employees, annual revenue, corporate structure, or other characterizing metrics that are relevant to your product.
• Transaction History – The history of purchases by the company.  Absolute magnitude, frequency, and recency of purchases can all be used.
• Geography – Where a particular company is located.  While not always correlating with variation in companies, it may serve as an effective way to “split the work” for internal organization – such as geographic sales teams.  This type of segmentation can also help when business rules or policies vary by geographic region – such as tax, policy, and other legal variations that apply either to your company or your customer.

Customers

The term customer is ambiguously applied to both buyers and users, and is only accurate when it represents someone who is both the purchaser and the end-user of your product.  You should explicitly think about your customers as buyer personas and user personas.

Buyer personas compare products based on their suitability to solve the problems that the buyer perceives as valuable.  User personas value products based on how effectively the products solve their real problems.  This is the most important distinction between buyers and users, while both are focused on value, only the end users will realize the value directly.

Market research should be focused on developing personas that represent homogeneous groups of customers, so that you can focus on the problems that each group distinctly faces.

Problems

Problems to be solved for the groups of customers that make up your market segments are what define a market.  Kano analysis provides you with the tools to classify these problems, allowing you to make informed decisions about which problems to focus on in your product roadmap.

[larger image]

Kano analysis, as originally defined, classifies products in terms of the features or capabilities they provide.  The naming originally used by professor Kano  (as translated) is slightly different than the terms used above.  The change in terminology here provides more clarity, particularly when applying Kano analysis from a market-perspective (outside-in) versus a feature-perspective (inside-out).

As a market-driven product manager, you should tweak the language of Kano into classification of problems and solutions, because customers value solutions, not features.  All of the techniques of Kano analysis apply, the nuance of this approach is to refocus on problems (outside-in) not features (inside-out).

Use Kano analysis to identify each problem faced by customers in your market as being in one of the following four classifications:

• Delighter – A solution to this problem will delight your customers.  As markets change over time, customers lose their sense of delight, and come to expect solutions to previously solved problems.
• Must Be – This problem must be solved by your product, or your prospective customer will refuse to buy or recommend it.
• More Is Better – This is the classic shades of grey problem, in contrast to the black and white nature of a must be problem.  Solving part of this problem is good, solving more of it is better.  The easiest way to think of this problem is that solutions are partial solutions, and the greater the portion of this problem your product solves, the more satisfied your prospective customers will be with your product.
• Indifferent – This is a problem for which your customers are indifferent to the solution.  Buyer decisions about purchasing your product (versus the competition) are not affected by how capably this problem is solved.  Users will not judge your product’s effectiveness based on how well it solves this problem.

Conclusion / Request for Feedback

This view of the market as an organization of problems that are shared by groups of customers allows you to

• Gain more insights from competitive analysis.
• Prioritize the content of each release to grow (revenue or profits or market share) faster.
• Design solutions that particular groups of customers will love.

That pretty much describes the model.  Please share feedback below (in the comments section) or offline.

Thanks again!

Each requirement you write represents a single market need, that you either satisfy or fail to satisfy.  A well written requirement is independently deliverable and represents an incremental increase in the value of your software.  That is the definition of an atomic requirement.  Read on to see why atomic requirements are important.

Atomic Requirements – Revisiting

As part of the ongoing series, Writing Good Requirements – The Big Ten Rules, I wrote about the importance of atomic requirements first in 2006.  That article touched on only one aspect of atomic requirements – being able to ask “is it done?”

Writing atomic requirements is important from two perspectives – delivering value to your customers, and operating efficiently.  You get benefits both operationally, and in how you are delivering value when you write atomic requirements.

Atomic Requirements Accelerate Value Delivery

In a recent article, I proposed methods for splitting user stories when they are too large to be delivered within a single Scrum sprint.  That article looked at ways to break up complete user stories (an agile requirement artifact, similar to a use case) that were already atomic, making them even smaller.  Think of that article as Atomic Requirements 301 – sort of an advanced class on splitting the atom.  There are opportunities to subdivide molecular requirements – those made up of multiple atomic goals – the topic of this article, which would be Atomic Requirements 101.

If you’re using an agile development process like Scrum, consider a user story like the following:

• As an online shopper, I need to find and compare similar products, a few times a year, so that I can make an informed purchasing decision.

The key element, from an atomicity perspective, is find and compare.  These are actually discrete user goals, although it may not be immediately obvious.  Rewriting as follows, will highlight this:

• As an online shopper, I need to find products and compare similar products, a few times a year, so that I can make an informed purchase.

Rewritten like this, find products and compare similar products are clearly different activities supporting different user goals.  They should be split into separate, atomic user stories.

1. As an online shopper, I need to find products, a few times a year, so that I can purchase desired products.
2. As an online shopper, I need to compare similar products when shopping online, so that I can make an informed purchase.

These two user stories can be delivered separately.  If this example were for B2B (business-to-business) eCommerce, the example would be different – because the online shopper is (often) not the right persona.  A typical situation in B2B is that one person will do the research to determine what to buy, and another person will make the decision of from whom to buy it.

This rule applies for non-agile requirements development as well – consider the following “BRD-style” requirement example:

• The system must record all purchases and submit them to the fulfillment system for processing.

Recording purchases and submitting them for processing are two different activities.  While they likely support the same goal, they may also support different goals and could be implemented separately.  Recording purchases is important not only for fulfillment, but potentially also for analytics.  Pushing data to a fulfillment system can be done in either a manual or automated fashion – providing a distinct cost-reduction opportunity.  To write these as atomic requirements, they must be separated.

1. The system must record all purchases.
2. The system must submit all recorded purchases to the fulfillment system for processing.

This example shows only the bare bones – there is more involved in writing either requirement, particularly around defining the non-functional requirements and business rules (see also: why you benefit when you separate rules from requirements, and the difference between business rules and business requirements) that affect and constrain each requirement.  This is easier to see when the requirements are atomic. These two atomic requirements may be rewritten as follows:

1. The system must record all purchases on the same day that they are created, recording information per [BR117: Purchase Data Rules].
2. The system must submit all recorded purchases to the fulfillment system for processing per [BR231: Fulfillment System Data Interchange Specification].

These additional constraints are not non-atomic additional requirements, they are constraints that specify how the system must perform the atomic actions.

The previous examples allude to another key benefit of atomicity – clean traceability.  Each requirement (or user story) exists to support one or more goals.  Requirements traceability can get complex, when many goals are dependent upon multiple requirements to be realized and many requirements enable multiple goals.  This approach to representing goal-decomposition (or user story decomposition) is critical to assuring that you are writing valuable requirements.

The simple version of traceability can be visualized with this view of structured requirements, presenting (and slightly extending) some of Karl Wiegers’ work on requirements.  If you’re doing any work in requirements of any kind, you need to read Karl’s stuff – his Software Requirements is a must-have.

This diagram makes it seem pretty straightforward.  Where it gets messy is when the multi-goal-per-requirement and multi-requirement-per-goal dependencies (that always exist) turn it from a simple tree into a complex graph.  Non-atomic requirements make that graph messier (a hassle, but not a problem)as each “requirement that is really multiple requirements” ads extra dependency relationships to your traceability model.  This becomes a problem, however, when you need to change.

Every project has implementation tasks that take longer than originally expected.  Using a time-box approach to managing the content of each release gives you a straightforward framework for determining how much stuff will be delivered in each release.  When you have to slip something, you have to revisit prioritization.  With well-defined requirements, each valuable requirement that could be delivered provides value by enabling users (or systems as users) to achieve a goal.  When you have to delay the implementation of a requirement, you delay the goals that it supports.

As soon as you realize you are considering delaying part of a requirement, that is a red flag that your requirement is not atomic.

Atomic Requirements Simplify Operations

Atomic requirements allow you to make these on-the-fly prioritization decisions with much better insight into the resultant delay in value.  This allows you to better manage communication with your stakeholders – validating that you are delaying the realization of the right goals.

The same benefits apply when originally planning your software iterations and releases.  Conceptually, you can imagine “scheduling” everything for the first release, immediately discovering that most things need to slip to future releases.  That’s how most* software projects really play out.

*There are times when you intentionally delay particular capabilities, based on external factors – market positioning, customer-adoption cadences, sales-team capacity (to absorb change), etc.

Conclusion

Writing atomic requirements helps you

• Organize and describe why you are asking the team to build what they are building, and how stakeholders will benefit from what will be built.
• Understand the assignable value of each requirement by maximizing clarity in the dependency tree – each requirement is in place for specific reasons.
• Communicate the relevance of each requirement to the implementation team – informing cost-benefit discussions around the inevitable schedule-change discussions.
• Test the deliverables – atomic requirements get graded as pass-fail, not “X% working.”

Attributions:

• Thanks clix for both “atomic” photos!

Every team that transitions to agile faces this problem – some stories are too big to fit in a single sprint.  Most of the teams that I have worked with have the wrong instinct – to solve half of the problem for all users.

The right approach is to first solve all of the problem for a subset of the users.

The Goal of the Story

User stories were created as lower-overhead use cases that served to better describe chunks of valuable software that could be rapidly developed, delivered, reviewed, used, and improved. Writing user stories that undermine that intent – rapid, valuable, incremental delivery – is a bad way to write user stories.

Imagine you are part of the team building GoToAssist, a product designed to do this. Your product manager and product owner put the following story into the top of your backlog – it is the most valuable capability you could add.

As a provider of IT support, I need to remotely resolve issues on multiple computers simultaneously, every day, so that those computers will run well.

The acceptance criteria include requiring that the solution working when accessing Apple computers, and PCs running various versions of windows and Linux. Other acceptance criteria include allowing the solution to work well through network connections having minimum characteristics, not requiring the remote users to take action, and other elements that characterize what would be considered acceptable solutions by the stakeholders.

Your team immediately raises a red flag that this user story is “too big” for a single sprint.

Since the goal of the sprint is to deliver valuable, shippable software, you have a problem with any user story that cannot be completed within a single sprint.

No problem, just decompose the user story into smaller user stories…

Inside-Out User Stories (Bad)

Note: Don’t confuse making smaller user stories with defining the tasks required to implement a user story. Task definition is important and valuable for helping the development team organize their activities.  Making smaller user stories is not a good approach for organizing the team’s activities – use tasks instead.  The size and structure of user stories should align with the user’s goals.

An inside out perspective or approach is one that starts with an awareness of what is involved in creating the software, and either never, or only later considers the perspective of what it is like use the software.

When approaching the problem inside-out, you will break the story down as follows:

1. As a provider of IT support, I need to remotely gain access to multiple computers simultaneously…
2. As a provider of IT support, I need to remotely authenticate as an administrator to each computer I access…
3. As a provider of IT support, I need to remotely control each computer I access…
4. As a provider of IT support, I need to remotely restore control of each computer…

This approach looks very appealing, because it is very clear what has to be done to support each new user story, and each new user story is appreciably smaller. Assume that each new user story is “small enough” for the team and does not need to be further reduced in size. It is very easy to see how the scope of development work has been reduced for each user story.

The problem shows up when you approach this schedule from a user’s perspective. Imagine that this product shipped with only stories 1 & 2, and not 3 & 4. How useful would the user find the product to be if she could remotely access a computer and authenticate, but not actually control the computer (to do her remote support work) or return control to the computer’s owner?

The reason you should not break the original user story up into these pieces is that the new stories do not independently represent delivery of valuable software. These proposed smaller user stories are atomic (each step is discrete), but each step has no independent value. The value is implicit in the sequence of steps, and is only realized when all four steps can be completed. The original story is already the smallest set of activities that allow the user to achieve his goal – a complete user story.

The problem with decomposing the original user story from the inside-out is that the new user stories don’t provide independent value to any of the users.

What about trying an approach that provides all of the value to some of the users?

Outside-In Story Decomposition (Good)

You are still faced with the challenge of making the story smaller, but now with the constraint of only delivering pieces that provide independent value. You need an approach that will work to split the story up per-user. “Provider of IT support” is an overly broad category of users – it is heterogeneous, within which different groups of users will use the product differently. I am a “provider of IT support” to my family, but I never have a need to access multiple computers simultaneously. I also don’t need to do this more than once a month on average. A corporate IT specialist responsible for 100 users would really benefit from simultaneous remote access to multiple machines – allowing her to multi-task, starting updates on a second machine while the first machine is working, etc.

There are at least two different personas that care about the “remote administration” user story, within the “provider of IT support” user category. One persona who needs to access only one machine at a time, and one persona who needs to access multiple machines simultaneously. This split allows you to decouple all of the work associated with managing simultaneous connections from the work associated with the sequence of steps (as applied through a single connection). This may reduce the size of the original user story enough to complete it in one sprint, but probably not, since it only “shaves off a little work.”

Another way to discover different sets of people who could benefit from delivering the user story differently is to look at the acceptance criteria.

Right off the bat, we’ve called out that the solution needs to work when the remotely accessed computers are running several different operating systems. You can make the user story smaller by constraining to one supported platform at a time. Whichever platform you pick first – say Linux – has some users who will value your solution. Start with them, then add another platform. Work with the product manager to determine how many customers in the market will require each platform or combination of platforms – that will guide the sequence in which you build support for each platform.

Another acceptance criteria required that the remote users not need to be involved – enabling “unattended” remote support – a key market differentiator. A key differentiator for the corporate IT specialist persona, who gets value from not having to coordinate support activities with multiple remote users. In fact, it is probably a must-have feature for those customers. As a family-support specialist, I don’t really have this need. My mom is going to call me, and I can fix her pc while we’re on the phone. I’m not trying to remotely administer her machine when a corporate policy has changed (on my schedule) – I’m doing it when a problem arises (on her schedule). This criteria can be deferred for a later iteration of the story

This approach works because, while it may be not be intuitive (from the inside) that this is a viable way to decompose the story, it makes send from an outside-in, user perspective.  Think about the different sources of value / problems, and how different people value solutions to the same problems differently.  The important characteristics of the problem will vary from one persona to another.

What About Development Efficiency?

As a developer, you may very correctly complain that this is a really inefficient development process. Some portion of the work done to enable controlling a single machine may need to be discarded and recreated in order to control multiple machines in parallel.

While these extra development costs are very visible, especially to developers, they are much smaller than the benefits that come from delivering the working software earlier. The cost of resolving bugs increases by orders of magnitude as you move through the product creation life cycle (from development to testing to deployment). The value of solutions also increases by orders of magnitude as they move through the process. Getting working product deployed earlier allows you to start realizing value (and therefore revenue) earlier.

That increase in revenue far outweighs the cost of a less-efficient development approach. Ignoring revenue side and focusing only on the cost side is being penny-wise but pound-foolish.

Dogma and Pragmatism

None of this should be taken as dogma. My experience has been that this approach has almost always worked, sometimes requiring more radical thinking than others. When it does not work (e.g. does not provide additional value, or does not sufficiently reduce the size of the stories), don’t do it. I’ll always remember Kent Beck saying to me – “If it costs more to test (in advance) than it costs to fix the bugs (after the fact), then don’t test.”

The same applies here, be pragmatic about it.

If you’ve been biting your tongue since the middle of the “Inside-Out” section, because you can always just wait until all four user stories are completed in separate sprints, before releasing the software, then my apologies. Of course you can do that. But you are undermining a precept of Scrum – that every sprint could be released, if you needed to release it.

That may not apply to you – I work with some teams that develop enterprise software, and for them, releasing every sprint makes no sense – their customers can’t absorb the rapid improvements, their sales teams aren’t willing to sell differently, etc. So, please forgive me, and go back and read the Outside-In section again, since you were probably ignoring that section as you started planning a great comment about this.

Who knows – a “too big” user story may appear in the backlog for the last sprint within your current release – wouldn’t it be great if you had the choice to release something (valuable) in the current release?

Think about this problem from an outside-in perspective. The smallest story is one that allows a single customer to perform a single task, for which he will pay you.

People Over Process – Conclusion

The objective that every sprint deliver valuable, working software, is not a requirement. It is a design principle of the Scrum process.

Agile is not about mandating processes, it is about getting better and faster. When you can’t reasonably decompose a story into something that can be delivered in a single sprint, don’t. You can have a sprint where you don’t deliver anything, because you only complete part of one thing. Yes, it will look bad on the burn-down chart.

Will it mess with your velocity? A bit. But who cares? Velocity is a measurement that helps inform projected delivery schedules. You may have one sprint with zero velocity, but in the next sprint, when you do deliver the “two sprint story”, you’ll have double the velocity. Magically, your running average is back to normal.

Don’t let the rulers you use to measure the team’s output cause you to act irrationally.

Attributions:

• Thanks to John Loo for the half-a-coin image.
• Thanks to Handige Harry for the puzzle pieces image.

Writing Verifiable Requirements should be a rule that does not need to be written.  Everyone reading this has seen or created requirements that can not be verified.  The primary reason for writing requirements is to communicate to the team what they need to accomplish.  If you can’t verify that what the team delivered is acceptable, neither can the team.  This may be the most obvious of the rules of writing requirements – but it is ignored every day.

Verifiable Requirements – Revisiting

In 2006, I first looked at how and why writing verifiable requirements is important - as part of the ongoing series on the rules of writing good requirements.  The focus of that article was on testability, as is this one.  When visiting verifiability again, however, I will add that not only must specifications be objectively measurable, but so must goals be written so that you can identify when a solution has met the objectives that justified its creation.

Directly Verifiable Requirements

Most of the requirements we come across can be directly verified.  The only problem with these requirements is that they lack specificity.  The following example is almost verifiable, and only needs a little help:

• The web site must make it easier for users who use site search to find the products they want to buy.

This requirement presents the challenge of resolving the ambiguity of the requirement. This lack of specificity prevents the requirement from being verifiable.  You have to identify how much easier the site search process must become for users.  Findability is a more is better characteristic in the Kano Analysis model for defining requirements.

You have to acknowledge that making it easier to find what you’re looking for is better for users than making it harder – there is an upward slope to the function.  However, there are also diminishing returns.

[larger image]

• A search that includes “the perfect item” is massively better than a search that does not include what you are looking for.
• A search that includes the perfect item on the first page of results is significantly better than one that returns the perfect item on the second page of results.
• A search that includes the perfect item as the first result is only marginally better than a search that returns that perfect item as the second or third result.

Improving findability as articulated above, from a user perspective, manifests in value to your company in two ways: immediate revenue impact and indirect revenue impact.

The immediate impact can be measured in terms of increases in commerce.  The indirect impact results from improving the user experience.  These improvements in experience result in increased word of mouth, as users altruistically encourage others to visit your website, and also result in an increase in satisfaction causing users to return to your site more frequently and make more purchases from you.

You can measure these impacts in terms of

• Conversion percentage (percentage of people who search and then purchase the products found in the results)
• Revenue attributed to users who search (an absolute measurement of purchases of products found via search)
• Site traffic levels (the number of people that visit your site over time)
• Visitor-recency statistics (the amount of time that elapses between return visits for returning visitors)

Conversion percentage, for users who search, normalized against users who do not search, is the most isolated (from other variables and noise) and fastest responding (as a delayed measurement of impact) measurement of the value of making it “easier” for users to search for the products they want to buy.

Your team will design different approaches to achieving these improvements.  You have to estimate both the cost and the potential benefit of each approach.  Balancing cost estimates with potential benefits will yield the ideal requirement – perhaps a 10% improvement.  [Note: I've collapsed at least another article's worth of balancing cost versus benefit, and multiple articles of "understanding and measuring site search" into one paragraph here, in hopes of staying on task with writing verifiable requirements.]

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The web site must make it easier for users who use site search to find the products they want to buy.”
• After: “Users who search on the site will be at least 10% more successful at finding the products they want to buy when using site search.”

Impossible to Verify Requirements

Often, stakeholders will present us with requirements that are impossible to verify (as requested).

• The home page needs to load fast.

At first blush, this looks just like the previous requirement (easier search is similar to faster page load).  You can use the same techniques to determine a measurable “requirement” like “the home page needs to load in under 1 second.”  However, that’s not a good requirement, because it is not an implicitly valuable requirement.  This statement provides no insight into why a fast-loading home page might be valuable to a particular user or why it would be valuable to the company.

To address this issue, you have to meet with the stakeholder(s) and understand the actual underlying requirements.  You will ask why, determine motivations and underlying problems, and apply active listening skills to discover the underlying requirement.

The problem is that the statement, “the home page needs to load fast” is a specification.  In Requirements Documents – One Man’s Trash…, I first used the following diagram to show how different people in the process of designing software view their piece of providing a solution.

A developer may receive a spec – “home page must load in under 5 seconds” and feel like the why component is perfectly well defined – it is a matter of context and perspective.  Another way to think about this is that the problem is with the problem statement.

A product manager, however, needs to do research that follows a path like the following:

Given that we are building an eCommerce website, we know that people have expectations around page load times (per Forrester / Akamai report (requires registration)).

[larger image]

Further, those expectations manifest as people abandoning slow-loading websites (from the same report).

[larger image].

We know from this market research that page-load times (for websites) represent another more is better Kano attribute, but one that has must be characteristics at the extremes.

[larger image]

While more speed is better, what the market research reveals is that for any given person, there is a minimum-speed threshold, at which speed is a must-be characteristic – too slow, and you lose customers (immediately).

A  combination of market research and elicitation will reveal that there is a true underlying goal of being fast enough to satisfy as many users as possible.  Combining this with practicalities of scaling your solution, and the associated costs; in the context of a particular solution design (an eCommerce website), you will arrive at a goal that allows you to rework your requirement so that it is verifiable.

Note: The requirement in this example is a subordinate requirement to a goal focused on maximizing conversion percentage (the percentage of customers who visit the site making purchases) – with a recognition that a major source of non-conversion is abandonment of the site, and that a large contributor to visitor abandonment is page load times.  An Ishikawa (or other model) will help you articulate this complexity and formulate requirements at a level of abstraction that is both valuable and actionable.

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The home page needs to load fast.”
• After: “No more than 10% of potential site visitors will abandon our site before viewing the page.”

Indirectly Verifiable Requirements

Some requirements are impossible to literally verify, and must be inferentially verified.

• The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.

Models are Models, Not Reality.

Imagine trying to create a test by organizing a million visitors to hit your SaaS web solution on the same day, with at least 10,000 of them hitting the site simultaneously.  That is completely impractical.  That doesn’t however, invalidate the requirement or make it untestable.  The combination of modeling, hypothesis formulation, and extrapolation gives you a reasonable way to verify that your solution probably meets this requirement.

You’re already used to the concept that models are representations of reality – think about the maps you use – a map may have a scale like one inch equals one mile.  The map is a model of reality.  A map where one inch equals one inch would be impractical.

An effective approach to measuring this type of scalability requirement is to simulate users (with load testing and realistic user-behavior scripts) with automation.  That takes care of most of the coordination problem.  However, the cost of simulating a million users and 10,000 simultaneous users is high.  You can, more realistically, measure the site’s performance when 10, 100, and 1,000 simulated users simultaneously access a server.  You can extrapolate from those results to estimate how the system is likely to perform under 10,000 user loads.

Note: Extrapolation is dangerous (follow the Elvis link below)- you have to justify why extrapolation holds true, and identify when it does not, to minimize the risk of invalidating your hypothesis.  Make sure you have confidence in the engineering prowess of whoever is doing your extrapolation and modeling.

[see tip #5 of these ROI calculation tips for the Infinite Elvis anti-pattern]

While this requirement does not to be rewritten in order to be verifiable, it does need to be augmented:

• Original: “The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.”
• Additional: “Testing of the site must show that 500 simultaneous users following <user script X> will yield no more than 1% page load times over 200 ms.”

Conclusion

Every requirement you write must be verifiable.  When you can’t verify something, and can’t rewrite it into a verifiable form, that should be a sign that either it is a vision statement or a red herring.  Vision statements guide how we approach creating products and engage markets – they are valuable, but they are not requirements.  Red herrings are well-meaning but ill-advised inputs into the process that need to be culled – they are neither valuable nor requirements.

Make sure all your requirements are verifiable.

People who already use Scrum will only find one new thing in this article – a way to communicate what happens inside a sprint that has proven effective for me.  People who are new to Scrum who wonder “how do things work inside a sprint?” will see how things work in a way that avoids hyperbole and is easy to map to what they already understand from traditional software development processes.

Two Teams Separated by a Common Process

George Bernard Shaw was an Irish author in London who memorably said “England and America are two countries separated by a common language.”  The same can be said about teams developing in agile and waterfall processes.

The story of agile adoption is one that has many colorful examples of adoption and of spreading the gospel.  Some practitioners traveled across the ocean from the agile continent to the lands of agile and opportunity to reap the rewards.  A few of those, after realizing the benefits of agile, invade our consciousness with a hellfire and brimstone tale of doom for anyone who doesn’t convert to the new religion.

Others seem hell-bent on kidnapping entire development teams, smuggling them across the ocean in the belly of steamer ships and unceremoniously dumping them in the land of agile, ready or not.  Scrum is but one branch of the agile movement.

Stalwarts of the old way, which has worked fine for me, thank you very much, refuse to leave their comfortable lives to step into the unknown wilds of agile.  Open-minded but responsible potential converts ask questions to gain an understanding of what life in the new world might be like.  What will life be like if I join this Scrum congregation?

How Does Scrum Really Work?

The best explanations of how scrum works that I’ve read come from Mike Cohn and Mountain Goat Software.  The training, videos, and explanations they share provide fantastic top-down introductions (as well as guidance after adopting Scrum).  His book, User Stories Applied, is the ultimate reference when gaining an understanding of the mechanics of using user stories as the central artifacts for developing software with scrum.

[larger image]

The process diagram above provides a good outside-in, top-down view of how “this newfangled process” results in shippable product, incrementally.  It is a great way to present the concept of Scrum, and incremental development in general.

When people I’ve worked with first gain this understanding, they often ask, what are the artifacts that live in these backlogs?  User stories live there.  Sometimes use cases make more sense than user stories for Scrum- it depends (read the article to see which works best in your circumstance).  The common element is an outside-in, user-centric approach to describing which problems the software is intended to solve.  This is also known as goal-driven development.

Once people understand the top-down view of the process, and have an idea of what user stories are, the next question they tend to ask is “how do stories move through the Scrum process?”

The first attempt at answering this question leans on the process diagram above:

1. The product owner takes the most important stories from the product backlog and puts them into the sprint backlog – collaborating with the team to agree on the scope of work for this sprint (based on the amount of work, capacity of the team, real-world constraints, etc).
2. The team “works the stories“, meeting every day to communicate effectively and provide progress updates – best visualized with a burndown chart.  This chart says “we started with X work to do, and every day, we track how much of X remains to be done.”
3. At the end of the sprint, the work is done, and the software (or an update to the software) is ready to deliver.
4. Sometimes, multiple sprints happen between releases – launches of the updated software, because customers (and the rest of your organization) incur a cost when changing to the latest version.  But the output of each sprint is deliverable – that’s a key concept – even if you choose not to deliver it just yet.

What Does it Mean to Work The Stories?

I’ve repeatedly had people ask, after absorbing the above explanation, “what does it mean to work the stories?”

I’ve had consistent success (in bridging the language divide) using the following diagrams (drawn on a whiteboard) to explain how stories flow through the sprint process in Scrum.

The first diagram shows that user stories have a structure – the story itself, and the acceptance criteria for the story. I also establish that the story is going to go through a life cycle within the sprint.

[larger image]

Next I remind folks that the user stories make it into the sprint backlog because they are the most important (highest priority) not-yet-implemented stories in the product backlog.

[larger image]

Developers on the team pull stories from the sprint backlog and begin working on them.  These stories are work in progress, and are owned by individuals.  Some teams maintain that priority order within the sprint backlog (e.g. implement the most important story in the sprint first, etc.), while other teams use a more coarse-grained approach, relying on the product backlog prioritization to assure that everything in this sprint is the next most important stuff, and the order that the team implements, within the sprint, is up to the team.  Some folks like to strenuously debate this decision (don’t prioritize within sprints vs. do prioritize within sprints).  Both approaches are valid, and you should choose the right one for your team.

[larger image]

As each user story owner believes that her work is complete (all unit tests pass, and the developer believes that the acceptance criteria have been met), the story is ready to be tested by QA.  This step, in particular, helps people who are new to scrum and who think of things as “code and test”, to make some sense of the inner workings of a sprint.  When working with a team that was under-staffed in the QA department, I discovered that calling out this queue of user stories waiting to be tested helped convince management to increase QA investment in the team.  When too many stories pile up here, you know you have a problem.

[larger image]

The scrum team member responsible for testing the user story has the responsibility of assuring that it meets all of the acceptance criteria.  He pulls the story from the queue of “done” stories, and tests.

[larger image]

If the story fails to meet any of the acceptance criteria, it moves back into either the sprint backlog or the being developed column, where a member of the team resumes ownership of the user story and works to resolve the issue.  Once the issue is resolved, the user story is re-queued for testing.  When the user story meets the acceptance criteria, it is moved into the done done column.

[larger image]

Conclusion

If you are new to Scrum, and wondered how the sausage is made inside a sprint, this gives you a framework for understanding what is going on, and how the team delivers.  If you already know or use Scrum, you may have learned a couple things.

• Using this framework to describe how a sprint works is effective when explaining to people who do not have any experience with agile.
• Laying things out this way visually, will give you an easy-to-see signal if your team is unbalanced between developers and testers.

I’ve been involved with agile teams and processes for just over 11 years now, and incremental development is burned into my brain.  If you are new to scrum, please let me know if this seemed like a good way for you to consume this material.  If you’re an old hat, and have other suggestions or presentations, please add them in the comments below, because future readers will benefit from the additional ideas.

“For what one idea do you want your product to stand in the mind of your customer?”  I heard Roger Cauvin ask that question at the most recent ProductCamp Austin [correction - he said it here - thanks Roger], and the quote has been jumping to the front of my mind almost daily ever since.  Maybe by writing about it I can exorcise the demon and get back to using the idea instead of being haunted by it.

The One Idea

You’ve just been given a new assignment – your company needs a new software thingamajig so that you can play in the whatchamacallit space, and you’re going to drive product management for it.  You’ve also been asked to deliver the first version in six weeks.  Of course you’ll be able to do follow-on releases, after all, you are agile.  That’s how “agile” works, isn’t it?

Cool.  Exciting.  Challenging.

You sit down with the folks who will be building the product, and find out that they have already spoken with several stakeholders.  Excellent. You circle back with the stakeholders, and start to gather data about your market and audience.

Your schedule is aggressive, so you think about what is realistic to get done in a very small timebox.  With the current schedule, you can’t afford to get caught in analysis paralysis, and you can’t realistically do in depth market analysis, persona development, hyper-accurate requirements prioritization, etc.  You do, however, have time to do the most important thing – define the one idea that will define your product.

If you cannot come up with this idea, you need to push back on your management team, and delay the launch of your product until you have that one idea.

Don’t Abuse Agile

Agile is designed to help you rapidly create iteratively better products, with iterative development and continuous infusion of market feedback and data.  Agile is not a process by which you start typing without any idea of what you intend, releasing it and then getting feedback in an iterative process.  If that’s how you’re approaching agile, your process is broken.

[image from Market Driven Competitive Advantage]

Unless you’re very lucky, your first iteration will be a waste of time and money.  Wouldn’t you rather be intentional than lucky? Make sure you have that one idea before you start developing.

Strategic Alignment

You need to come up with a one idea that is aligned with and supports your corporate strategy.  At a minimum, you have to make sure that you have an idea that is aligned with your stakeholder’s goals, based on the assumption that those goals are aligned with corporate strategy.

Wow

Your one idea really needs to make your users say “Wow!”

I was explaining the importance of this to a colleague (who is not a product manager), and his response was “Oh, you mean the hook.”

I was reminded that you not only have to provide a wow experience for your users, but you have to present a hook that will capture the imagination of your buyers.  Remember that buyer personas make decisions based on their perceptions of user problems [more on convincing buyers].

Satisfice

Remember that you don’t want to try and do everything perfectly in the first release – you want to satisfice.  You may only be able to target a single persona with a solution to a single problem.  You just need to make sure you are solving the right problem – which is where Kano analysis is very handy for identifying the Minimum Market Acceptance criteria.

[From Minimum Market Acceptance]

Conclusion: A Balancing Act

Defining the one idea for your product is a balancing act.

• You have to align the one idea with your corporate strategy and stakeholder goals
• You have to solve a valuable problem for your users, presented in a way that captivates your buyers
• You have to create a product that is compelling right away – good enough in execution to enter your market
• And you have to get the first release out in six weeks!

Good luck!  Or better yet – Good Intent!

April Dunford just presented Startup Marketing 101 at DemoCamp Toronto.  Great ideas from the ‘marketing and your startup’ point of view.  I’ve often said that product managers and product marketers care about much of the same market data, they just do different things with it.  The idea of minimal feature set came up in April’s presentation – this article talks about product management, agile, and initial market acceptance.

Initial Market Acceptance and Minimum Market Acceptance

April mentions (in slide 4) that an important event to the timing of marketing activities is achieving the “_minimal feature set (for certain markets)_.”  April organizes those marketing investments in terms of three stages of “product marketing lifecycle”:

1. What to do when your startup has a “pre-product” product.
2. Where to focus when you’re focused on early adopters.
3. How to invest in your marketing programs when your focus is on scale / growth.

Coming from an agile background, I think about two distinct notions of “minimum” or “initial” when it comes products and markets.  The following definitions are in a B2B context, as I’ve been focusing on this recently with a client in the B2B space.  The same ideas are relevant to B2C and B2B2C companies, but the language would be slightly different.

I’m assuming that when April used the phrase “minimal feature set”, she was really talking about “minimum market acceptance” in the way that I’m describing here.  The distinction is that features describe what a product does, which is an inside-out view of the product.  To be market focused, you have to think about which market problems are being solved, for whom they are solved, and how well they are being solved – an outside-in product view.

If you focus on, and organize around your features, you are likely to miss the mark.  You must focus on the problems that your customers need to solve.  In your product, you will prioritize the development of capabilities (embodied through features) that are designed to help your customers (buyers and users) solve their problems.  Your market analysis should be geared around understanding how many customers in each segment face each problem, and how much they are willing to pay to solve those problems.

Agile or Waterfall or Waterfragile?

As an agile product manager, and former developer, I know that when you do “everything we need to be successful” in the first release, you’re not being agile – we’re being waterfall.  A slight extension of this is “everything we need to be minimally successful” – and that is still waterfall.

An agile team should focus on the first release addressing the initial market acceptance criteria – the minimal set for a single customer.  The goal is to get the product in front of customers as soon as possible – this starts your revenue stream earlier, gives you valuable market feedback, and gives you the opportunity to establish momentum through repeated incremental releases of your product.  Each release will solve addressed problems “better” and / or address additional problems, until you reach minimum market acceptance.

Kano Analysis for Understanding Problems

Last year, I was thrilled to share my approach for applying Kano Analysis as a product manager with the PMV webinar audience as well as the Austin IIBA chapter.  One of the key ideas in the application of Kano Analysis to understanding your market is developing the personas within that market, and understanding how each of them treats each problem / capability.

[slideshare presentation & PMV webinar]

This analysis, in addition to being useful for understanding the market (or a segment) as a whole, also helps you identify your first customer.  Once you know who your first customer is, you can determine the initial market acceptance criteria for that customer, and that determines the must have capabilities.

Note that finding the initial solution (that the first customer will buy) does not mean releasing a poor quality product – it just means releasing a product that solves enough problems (well enough) for one customer.  With the feedback from that customer, you can drive the prioritization for your next iteration – making the next iteration better for that customer (which can really help your word of mouth), and gaining more customers.  Repeat this process until you get to minimum market acceptance.

Timing Marketing Investments for Minimum Market Acceptance

[slide 9 from April's presentation on slideshare]

Since I wasn’t at April’s presentation, I don’t know exactly what she would think of the following, but I believe it makes sense:

• Initial Market Scceptance (one customer would buy your product) marks the transition from pre-product to early adopters.
• Minimum Market Acceptance (enough to succeed in the market) happens before the transition from early adopters to scale.  Note that it may not (probably doesn’t) mark the transition, but I suspect happens mid-stream.

Would love to see what other folks think about mapping this product management centric view to the marketing timeline.  Please chime in below.

I came across a really interesting article LukeW.com, showing how making changes to the way an input form on a website increased interaction by 25 to 40%. The changes reflect the value of thinking outside-in, investing in user experience, and performance measurement.

Bonus: the idea is cool.

Mad Libs Input Form

[full size image available in Luke's article]

The idea being presented is to replace the old boring web input form designed for a computer. The new, fun form is a fill-in-the-blank (aka Mad Libs) layout. The article is on Luke Wroblewski’s site.  The team at Vast.com created and tested a version of this form for the Kelly Blue Book site. [Luke cites Huffduffer.com as the first place where he saw this technique.] Thanks, Luke for sharing this with all of us!

Outside-In Development

Product managers are acutely aware of the need to solve market problems. We are regularly reminded that we’re creating solutions for people in our markets who use our products to solve their problems. When we write requirements, we avoid writing design specifications, and thereby lose the ability to enforce that the proposed solutions also take an outside-in approach. We maintain an outside-in perspective, but we lost the ability to influence an outside-in aesthetic.

Note: Outside-In Software Development is a great book, you should read it.

That’s one area where user experience works well in concert with product management – assuring that the same drivers of what to build are informing the design of how it is built.

The genius (or at least elegance) of this Mad Libs form from Vast / Kelly Blue Book is that it humanizes the experience of requesting that a nearby dealer contact you to discuss a possible purchase of a car from them. The forms we’ve all had to fill out on countless web sites are very inside-out. They expose the inner workings of the program – “gather this info, and that info, and this other info.” Those forms force users to do what the program wants. Blech. The form that Ron Kurti designed, however, forces the program to do what the users want. Huzzah!

Bad Requirements Prevent Elegance

I’ve written about the importance of writing design-free requirements several times.  Avoiding design constraints in your requirements is even one of the pillars of the big 10 rules of writing requirements.

If you had written the requirements for this contact the dealer page, would you have specified the design of the form?

Most of the requirements analysts I’ve worked with would have. I’ve even worked with companies that have developed templates / stencils defining how these interface specifications must be documented – requiring that all the fields be aligned, with specific pixel gaps, etc.

If you had specified the design (read: limited the choices of the designer), you would have prevented this solution.

Mr. Kurti’s elegant solution is clearly better.

One way to write the requirements that would not have prevented this solution would be with a user story and acceptance criteria:

As a car-shopper, I want to engage a car dealer, once every few years, so that I can purchase the car I selected.

• The car-shopper can provide his contact info (name, email, phone) for the car dealer.
• The system will display the information identifying the selected vehicle.
• The system will display the information identifying the dealer being contacted.
• The system will include an opt-in newsletter-subscription option.
• The system will notify the specified dealer when the car-shopper indicates.

The user story nudges the developers into an outside-in perspective by emphasizing the user’s goal. The acceptance criteria make it clear that any solution that meets those criteria is acceptable to the product manager. It allows the interaction designer to create a compelling solution, rather than forcing him to recreate a boring experience.

Measurement Rocks

The Vast team measured the impact of making this change to their form, and saw between a 25% and 40% increase in conversion (people contacting dealers versus people abandoning the process when they see the form). That’s a lot more leads for the dealers, and presumably a lot more money for Kelly Blue Book and Vast. This is a great example of how you can measure the impact of investing in user experience, because it should spark ideas for you.

What can you measure and test and improve?

Now that’s ROI for you.

You give your requirements to the engineering team, and they look complete.  The team builds your product, you launch it and the market soundly rejects it.  Why?  Because your requirements weren’t complete – they didn’t actually solve the problem that needed to be solved.

Complete Requirements – Revisiting

Going back almost four years, I wrote Writing Complete Requirements, as part of the Big Rules of Writing Requirements series.  That article centered on two key ideas of requirements completeness.

• Data, not opinion, is the most important input into determining completeness.
• There is no absolute way to determine the completeness of your requirements in advance.

Completeness is best measured by market acceptance, so technically you can’t know if your requirements were complete until your customers buy (or fail to buy) your product.  This doesn’t mean you should give up on this rule of requirements – you can still focus on, and improve, the completeness of your requirements.

Objective and Heuristic Assessment

Given a market problem and the associated requirements (which you can argue are actually the same thing), you can review those requirements to determine if they objectively appear to be complete.  This assessment is based on what you objectively know about your market (needs, opportunity costs, competitive alternatives, etc).  You can also apply heuristic, or logical analysis to identify gaps in the language of your requirements.

Objective assessment is the application of market knowledge (data) to determine if the nature of the problem is being addressed completely by your requirements.  A bird-feeding maven would know that the requirements for your bird feeder would be incomplete if they didn’t address the problem of squirrels stealing food from the birds.

Heuristic analysis is the identifications of logical omissions in your requirements, without the need to apply market insights.  A logician would know that the requirements for your bird feeder were incomplete if they didn’t address the need to refill an empty feeder.

When you’re writing requirements, you need to do both.  Heuristic analysis of your requirements will identify where your requirements do not fully solve the problems you already know about – an eCommerce checkout process without the ability to receive payment, for example.  Objective assessment will help you identify the problems that you overlooked but need to solve – the need to allow customers to have different billing and shipping addresses when placing an order online, for example.

Complete Requirements for Incomplete Solutions

Don’t confuse having complete requirements (a good thing) with completely solving a problem (not always a good thing).  In agile development, the idea of satisficing is key to scoping individual iterations.  The idea, simply put, is to solve enough of the problem, and not delay delivery (and value) until you can build a solution to the entire problem.  Because of the law of diminishing returns, for many market problems, there is little or no incremental value in solving the entire problem.  Those resources can be better applied to solving additional problems.

Kano analysis provides a framework for identifying which market problems must be completely solved, and which ones have a more is better characteristic.  In practice, more is better problems exhibit the law of diminishing returns, and should not be “completely” solved.

[larger image, or view the entire Kano analysis presentation]

Conclusion

Complete Requirements are requirements that

• Identify all of the market problems that need to be solved and their nature (absolute vs. diminishing returns).
• Are logically complete in their coverage / articulation of those market needs.

To objectively improve the completeness of your requirements, apply market data in an assessment of your requirements.  To heuristically improve the completeness of your requirements, look for logical inconsistencies and gaps in reasoning or coverage.

Engagement – that’s what this whole product management blogging thing is about.  Check out what Tyner Blain readers found to be the most engaging articles in 2009.

Deep Dives

If you’re new to Tyner Blain, you may be surprised by the length of the articles here.  I joke that they are long because I don’t have time to edit.  Stewart Rogers jokes that they are long because I’m incapable of writing a short article.  If you’ve been here a while, you know what you’re in for.  If you’ve been here a long while, then you’re glad (like me) that I don’t write one per day any more.

Product Management is simultaneously a broad and deep discipline, requiring us to have a breadth of perspective combined with a depth of insight.  We then have to apply that in a market context, effectively navigating the political waters of our organizations.  Most of the articles here either try to skim the breadth of a range of related topics, or plumb the depths of a single topic.  Doing that in under a thousand words is pretty hard.  Most of the articles here also link to other articles, to try and provide even more depth and context, and encourage additional critical thinking.

One measure of the quality of the articles here is how often they stimulate readers to read further, or dive deeper into the topics.  While web analytics won’t allow us to measure how thought-provoking an article is, we can look to see how many people dive into the linked articles, versus how many people move on to something else.  We can measure the bounce rate of an article to see how often people leave a page without following any of the “tell me more” links.

Looking at ~170,000 page views at Tyner Blain in 2009, narrowed down to only those articles with at least 100 page views, here is the top-ten list of most engaging (or “least abandoned” if you’re a “cup is half empty” person) articles:

1. Use Case Series: Introduction: A collection of articles on the “traditional” forms of use cases – informal, formal, UML.
2. Agile Development of Use Cases: A dive into the dynamics and cadence of an agile process for developing use cases.
3. How Do You Manage Market Data?: A collection of articles exploring different market-immersion techniques in depth.
4. Scheduling Requirements Changes – Part 2: A focus on practical techniques for managing change with an agile development process.
5. Yesterday’s BPMN Post Was A Big Fat Lie: A mea culpa and clarification of interest to folks following the BPMN Series.
6. Everything I Needed To Know I Forgot in Kindergarten: Why asking why? is important.
7. Plan Your Next Sprint by ROI – Part 1: Prioritizing by Bang for the Buck, not just Bang.
8. Ten Common Mistakes of Going Agile: A collection of articles about common pitfalls encountered when adopting agile practices.
9. Perpetually Almost Finished Projects: Organizing a project with discrete deliverables and rolling-wave planning to avoid the “90% done, 90% remaining” problem.
10. How To Use Timeboxes for Scheduling Software Delivery: One of my personal favorites.  A rational approach to making trade-offs when your “perfect” plan has to change.

OK Stewart, that’s only 519 words.