Your boss wants a commitment.  You want to offer a prediction.  Agile, you say, only allows you to estimate and predict – not to commit.  ”Horse-hockey!” your boss exclaims, “I want one throat to choke, and it will be yours if you don’t make a commitment and meet it.”  There’s a way to keep yourself off the corporate gallows – estimate, predict, and commit – using agile principles.

This is an article about agile product management and release planning.

Change and Uncertainty

In the dark ages before your team became agile, you would make estimates and commitments.  You never exactly met your commitments, and no one really noticed.  That was how the game was played.  You made a commitment, everyone knew it would be wrong, but they expected it anyway.  Maybe your boss handicapped your commitment, removing scope, lowering expectations, padding the schedule.  Heck, that’s been the recipe for success since they planned the pyramids.

It makes sense.

1. Your early estimates are wrong. When you add them up, the total will be wrong.  If you do PERT estimation, the law of large numbers will help you in aggregate.  But you’ll still be wrong.
2. The outside demands on, and availability of, your people will change. Unplanned sick time, attrition, levels of commitment over time, lots of “people stuff” is really unknown.
3. The needs of your customers will change. Markets evolve over time.  You get smarter, your competitors get better, your customer’s expectations change.

Agile processes are designed to help you deliver what your customer actually needs, not what was originally asked for. Contrast the two worlds.

In the old world, you would commit to delivering a couple pyramids. After spending double your budget, with double the project duration, you would have delivered one pyramid. When you deliver it, you find out that sphinxes are all the rage. Oops.

Your team changed to agile, so that you could deliver the sphinx. But your Pharaoh still wants a commitment to deliver a couple pyramids (the smart ones will be expecting to get just one).  You can stay true to agile, and still mollify your boss’ need to have a commitment, if you take advantage of the first-principles of why agile estimation works.

Estimation

A commitment is a factual prediction of the future.  ”This will take two weeks.”  Nobody is prescient.

A factual prediction has to be nuanced.  ”I expect* this will take no more than two weeks.”

*in reality, this is shorthand for a mathematical prediction, such as “I expect, with 95% confidence, that this will take no more than two weeks.”

Few non-scientist, non-engineers, non-mathematicians understand that 95% confidence has a precise meaning.  People usually interpret it to mean “a 5% chance that it will take more than two weeks.”  What it really means is that if this exact same task were performed twenty thousand times (in a hypothetical world, of course), then nineteen thousand of those times, it would be completed in under two weeks – do you feel lucky?

To make a statement like this, you actually have to create a PERT estimate – identifying the best-case, worst-case, and most-likely case for how long a task will take.

Unfortunately, we’re rarely asked to make a commitment about a single task – but rather a large collection of tasks – well-defined, ill-defined, and undefined.

You can combine PERT estimates for the individual tasks, resulting in an overall estimate of the collection of tasks.

The beauty of this approach is that the central limit theorem, and the law of large numbers, work to help you estimate a collection of tasks – you can actually provide better estimates of a group of tasks than a single task.  This obviously helps with the well-defined tasks that you know about at the start of the project.  This even helps with the ill-defined tasks.  Rationalists will argue that the key, then, is to do more up-front research to discover the undefined tasks – and then we’re set.  As Frederick Brooks (Mythical Man-Month) points out in The Design of Design, this debate has been going on since Descartes and Locke.  It is not a new idea.

Big Up-Front Design and Requirements (BUFD & BUFR) hasn’t worked particularly well, so far.

Don’t throw out the baby with the bath-water, however.  The math of estimation is still important and useful.  Even if empiricism is not the silver bullet.

Prediction

Estimation is a form of prediction.  Even agile teams do it.  In Scrum, you estimate a collection of user stories – in story points that represent complexity, and you predict how many points the team can complete in this sprint.  Note the time factor.  If you’re working a two-week sprint, there is very little risk of changes in staffing during a two-week period.  There’s also very little risk that your market will change significantly in two weeks – and if it does, what are the odds that you will notice and materially change your requirements in two weeks?

Visually, let’s take that PERT estimate and turn it sideways – so we can introduce the dimension of time.  Imagine you estimated all of the tasks (well-defined, ill-defined, and a guess about the undefined), as if they were all to happen in the first sprint.  Ignore inter-task dependencies, and pretend you had unlimited resources and the ability to perform all tasks in parallel.

The graph above shows the aggregate estimate – the circle is your best prediction, with error bars representing your confidence interval in the estimate.  If you were using PERT estimates, these could represent that 5% and 95% confidence lines.  Subjectively pick something based on your team’s experience in the domain and your confidence in your guesses (about the undefined tasks).

We need a segue into the “best of waterfall” approach to estimating projects, to steal and invert a good idea.

The Cone of Uncertainty

The folks at Construx have published a nice explanation of the cone of uncertainty – an adaptation of an idea from Steven McConnell’s Software Estimation: Demystifying The Black Art (2006).  That article uses his imagery with permission – so please go look at it there.  The idea is that as the project becomes better defined (e.g. during the project), the amount of uncertainty is reduced.

The findings show that initial estimates are off by 400% (either low by a factor of 4 or high by a factor of 4)!  Even after “nailing down” requirements, estimates are still off by 30% to 50%!

As bad as that sounds, it is actually worse.  This is a prediction for the original project (delivering pyramids).  Not only are your estimates wrong – but they are bad estimates for delivering the wrong product.

But – the core idea is sound – the further into the future you have to execute, the greater the mistakes in your estimate.

Taking that concept, and applying it to our diagram, we get the following:

The further into the future you are trying to predict, the less accuracy you have in your prediction.  This reduction in accuracy is reflected as a widening of the confidence bands for your estimate.

• A couple sprints’ worth of work is not much different than one sprint – so your estimation range is not much changed.
• An entire release of sprints (say 6 to 10 sprints) has much more opportunity for the unknown to rear its head.

Now, your prediction is (probably) unusably vague and imprecise.  ”This set of tasks will take X plus or minus a factor of two.”

That’s the reality.

Note: This has always been the reality.  People have historically reduced this “risk to timing” by hiding the “risk of change” aspects – and waterfall processes encourage you to deliver the wrong thing, as close to on-time as possible.

That’s not what we want to do, however.

We still want to deliver the (not-yet-defined) right product, as efficiently as possible.  That’s the goal of agile.  (For folks who haven’t been here at Tyner Blain for long – “right” includes both value and quality).

Refinement

Because we’re agile, and we’re willing to “get smarter” about our product over time, we have an opportunity to improve.  Because of the nature of compounding estimates and the cone of uncertainty, our uncertainty gets smaller over time.

Let’s remove our artificial simplification that we could do everything “right now” and look at what we think we know right now, about the end of the release.

Our ability to predict the amount of effort (for today’s definition of the product) at the end of the release is not very good.

Our ability to predict (today’s definition of the product) one sprint into the future is much better.

After completing the first sprint, we are a little bit smarter – the ill-defined tasks are better defined.  Maybe some of the undefined tasks are now ill-defined.  The same cone of uncertainty is now a little bit smaller – we are a little bit smarter, and the time horizon of the release date is a little bit closer.

The trend continues – each sprint gets us closer to the release date, and with each sprint (assuming we get feedback from our customers, and continue to study our markets) we get a little bit smarter.  We also get better at predicting the team’s velocity (how much “product” they can deliver during each sprint).

Commitment

Your boss still wants a commitment, however.  And that’s where we get to change the way we look at this (again).

The above diagrams all display how we converge on an estimate for a stable body of work.  However, we know that the body of work is constantly changing.

Backlog! [you say]

Yes!  The backlog.  The backlog is an ordered, prioritized list of user stories and bugs.  I was talking with Luke Hohmann of Innovation Games last month, and one of the most popular online Innovation Games is now the one they created based on prioritizing by bang for the buck.  Play it today online (for free!).  How cool is that?

The backlog represents the work the team is going to do – in the order in which the team is going to do it.  Over time, as we get smarter, we will add and remove items from the backlog – because we discover new capabilities that are important, and because we learn that some things aren’t worth doing.  We will even re-order the backlog as we recognize shifting priorities in the markets (or in our changing strategy).

As this happens, it turns out that the items at the top of the list are least likely to get displaced, and therefore most likely to still be part of the product by the time we get to the release.

Instead of thinking about uncertainty in terms of how long it takes, think about uncertainty in terms of how much we complete in a fixed amount of time. In agile, generally, we apply a timebox approach to determining what gets built.

Now, uncertainty, instead of manifesting as “when do we finish?” becomes “what will we finish?”

Your boss is rational.  She appreciates the constraints, she just wants to know what you can commit.  Every boss I’ve worked with has been willing (sometimes only after much discussion) to treat this uncertainty in terms of what instead of when.  They acknowledge that they need to translate (usually for their boss) into a “fixed” commitment.

The solution: commit to a subset of what you predict you can complete.

At the start of the release, you may have 500 points worth of stories.  Based on your team’s expected velocity, and the number of sprints in the release, you predict that you can complete 320 points worth of stories (5 people on the team, a team velocity of 40 points per sprint, and 8 sprints in the release).  Starting at the top of the backlog and working down, draw a cut-line at the last story you can complete (when you reach 320 points).  This is your prediction.

Now the commitment part.  You’ll have to figure out what you’re comfortable with.  Maybe for 8 sprints (say, 16 weeks into the future), you may only be comfortable committing to half that amount – 160 points.  Go back to the top of the backlog, and count down until you reach 160 points.  Everything above the line is what you commit to delivering.

Maybe you are comfortable committing to 240 points, maybe only 80.  This is like playing spades.  The more you can commit to, without missing, the better off you are.  Your tolerance for risk is different than mine.

You can also negotiate with your boss.  Commit to 160 points now, and provide an update after every other sprint.  More likely than not, you will be increasing the scope of your commitment with every update.

Mid-project updates of “we can do more” are always better than “we can do less.”  And both are better than end-of-project surprises.  This also allows you to have updates that look like this:

We didn’t know this at the start of the release, but X is really important to our customers – and we will be able to deliver X in addition to what we already committed.  Without slipping the release date.

Conclusion

Making commitments with an agile process is not impossible.  It just needs to be approached differently (if you want to stay true to agile).  The end result: better predictions, more realistic commitments, and the likelihood that each update will be good news instead of bad.

[Update: Changed initial image.  Thanks Dennis for the great photo!]

Almost everything I’ve read about use cases focuses on describing what needs to be added to your product.  Agile development says “get it working first, make it better second.”  That means changing the way the software enables a user to do something they can already do.  How do you manage requirements for incremental improvement?

Iterative Development and Incremental Improvement

Iterative development is the process by which you release software in iterations – small, incrementally better versions of your software.  People usually think about this solely in terms of enable the most valuable capabilities first, then the next most valuable capability next.

That’s fine, when your product is new.  Eventually (or quickly!) you will reach the point when the next most valuable improvement is not to add a new capability, but rather, to make an existing capability more valuable.

Everything is an Upgrade

In a couple earlier articles about organizing migration projects and requirements for migration projects, I wrote about how migration is a bit of a misnomer.  Everything is a migration.

My argument in those earlier articles is that the problems you’re solving with your “new” solution are not new.  Your customers are currently solving them in different ways.  The relevant question for migration projects is around how much your users are changing the way they solve their problems.

[As a slight segue, failing to appreciate this distinction can lead you to a project that is based on a faulty problem statement.]

When you are delivering incremental improvements to your product, through iterative development, you are upgrading your software.  You are migrating your users from their old solution to their new solution.  It just so happens that you are replacing yourself, as you migrate them from the old version to the new one.

A viable strategy, that appeals to me personally, is to continuously innovate, disrupting your market, before someone else does.  With this approach, you are intentionally reinventing your solution, making it difficult for someone else to out-innovate you.  This is a great way to approach developing your roadmap strategically.

Your market will change.  Rapidly. Do you want to react to your competitors, or keep them on their heels while you stay on the balls of your feet?

The reverberations of the “new” Twitter still haven’t settled, and many of Twitter’s competitors are on their heels – some of them only now realizing that Twitter has always been a competitor and not just “a platform.”  The new Twitter doesn’t really change anything about how people use Twitter, except make it (markedly) better.

Avoiding Featuritis

When is it more valuable to improve what your software already enables, versus enabling users to accomplish more with your software?  Kathy Sierra introduced us to the concept of featuritis, the idea that too much is too much.

In an article on viral product management, I proposed that by making your software better, you can tap into an altruistic mechanism by which people will promote your product.  As an example, I proposed improving the usability of your software as a means to cross this viral tipping point.

One of the key ideas is that making it better for users makes your product better for your company.

Managing “Improvement” Requirements

Now that we all agree that there is value in making your software better [comment below if you disagree] the question becomes “How?!”

Use Cases can be used as the keystone to your requirements arch.  Sometimes, user stories are better than use cases, but use cases are perfectly agile.  When you’re starting a project, you want to apply an outside-in approach to defining the scope of that project – or more particularly, to define the scope of problems you are solving in your roadmap. This article will talk in the language of use cases, but identical concepts and approaches apply when using user stories as well.

When you have prioritized a use case as “the next most valuable capability to introduce” you add it to the backlog for your sprint.  The implementation team then provides feedback that use case is “too big” and you have to slice it up.  It is important that you don’t split the use case  in such a way that you only solve half the problem.

You also shouldn’t deliver a “bad” product.  In fact, the ideal way to introduce a new capability is to satisfice, and not introduce the “perfect” solution in your first iteration.  Make your first solution “good enough.”  Part of the art is defining “good enough,” but rest assured, it is not the same as “the best you can possibly do.”

Summarizing some key elements from above demonstrates the eventual reality you will face:

1. Every problem* your users face is already being solved, you are just providing a better solution.
2. There is diminishing, and eventually negative return on adding more capabilities to your product.
3. Your first release of a given capability will only be “good enough.”  That leaves room for improvement.

*Yes, you can pedantically define “the problem” as “the existing solution is not good enough” – but you still end up in the same place, so why bother?

Eventually, the value of improving something you’ve already released will be greater than the value of releasing something new.

What do you do?  You implement the same use case again.

If your product is Software as a Service (SaaS), you absolutely should be thinking about things this way.  Becoming better is a continuous improvement objective that is implicit in the economics of SaaS products.

Revisiting Use Cases

Not to be confused with re-use of use cases, revisiting a use case is specifically revisiting – with the goal of improving – the implementation that is already in place within your product.  This is a special case of a migration project – you’re migrating your users from your old solution to your new and improved solution.

You may be migrating to an identical process, perhaps with faster performance than your previous release was capable of.  Or you may be improving the procedure (improving usability, interaction design, visual design, etc) without affecting the process. Generally, a “make it better” use case improvement effort will have no more than a minor process impact.  Your users are still doing the same thing, they are just enjoying it more, or doing it more effectively.

When slicing use cases to make them fit within a single sprint, you may design them for a single user persona in the first release, knowing that you won’t meet the expectations of a different user persona until the next release.  In this case, your non-functional requirements, constraints, or acceptance criteria will be different.  Your use case may also be different (in the details) while appearing to be the same.  This also common scenario can be addressed with the same approach.

Find that old use case (from several iterations ago) and put it back in the backlog.  Remember, agile is about conversation, not artifacts.  If you need to add an explanation to the use case, because your team fixates on the fact that it is “already working,” add a qualifier that it needs to be “better.”  Then have a conversation, and explain how it needs to be better.

Is Design Part of Implementation?

Some teams organize such that designers (both architects designing code and designers designing interfaces are being addressed here) are part of the implementation team.  Other companies treat designers as stakeholders – providing guidance and input to the product managers and product owners.

When a “new design” is being requested from outside the team, include that design guidance as part of the input to the implementation team – through artifact and / or clarification.

When a “new design” is being requested of the team (the designer is part of the team), express the new acceptance criteria to the team.  Note that these need to be measurable requirements if they are to be considered good requirements.

Summary

There will come a time when the most valuable thing you can do is improve the user experience for a capability your product already embodies.  When that time comes, use the same use case again, updated with the new constraints, non-functional requirements, and acceptance criteria.

I came across a really interesting article LukeW.com, showing how making changes to the way an input form on a website increased interaction by 25 to 40%. The changes reflect the value of thinking outside-in, investing in user experience, and performance measurement.

Bonus: the idea is cool.

Mad Libs Input Form

[full size image available in Luke's article]

The idea being presented is to replace the old boring web input form designed for a computer. The new, fun form is a fill-in-the-blank (aka Mad Libs) layout. The article is on Luke Wroblewski’s site.  The team at Vast.com created and tested a version of this form for the Kelly Blue Book site. [Luke cites Huffduffer.com as the first place where he saw this technique.] Thanks, Luke for sharing this with all of us!

Outside-In Development

Product managers are acutely aware of the need to solve market problems. We are regularly reminded that we’re creating solutions for people in our markets who use our products to solve their problems. When we write requirements, we avoid writing design specifications, and thereby lose the ability to enforce that the proposed solutions also take an outside-in approach. We maintain an outside-in perspective, but we lost the ability to influence an outside-in aesthetic.

Note: Outside-In Software Development is a great book, you should read it.

That’s one area where user experience works well in concert with product management – assuring that the same drivers of what to build are informing the design of how it is built.

The genius (or at least elegance) of this Mad Libs form from Vast / Kelly Blue Book is that it humanizes the experience of requesting that a nearby dealer contact you to discuss a possible purchase of a car from them. The forms we’ve all had to fill out on countless web sites are very inside-out. They expose the inner workings of the program – “gather this info, and that info, and this other info.” Those forms force users to do what the program wants. Blech. The form that Ron Kurti designed, however, forces the program to do what the users want. Huzzah!

Bad Requirements Prevent Elegance

I’ve written about the importance of writing design-free requirements several times.  Avoiding design constraints in your requirements is even one of the pillars of the big 10 rules of writing requirements.

If you had written the requirements for this contact the dealer page, would you have specified the design of the form?

Most of the requirements analysts I’ve worked with would have. I’ve even worked with companies that have developed templates / stencils defining how these interface specifications must be documented – requiring that all the fields be aligned, with specific pixel gaps, etc.

If you had specified the design (read: limited the choices of the designer), you would have prevented this solution.

Mr. Kurti’s elegant solution is clearly better.

One way to write the requirements that would not have prevented this solution would be with a user story and acceptance criteria:

As a car-shopper, I want to engage a car dealer, once every few years, so that I can purchase the car I selected.

• The car-shopper can provide his contact info (name, email, phone) for the car dealer.
• The system will display the information identifying the selected vehicle.
• The system will display the information identifying the dealer being contacted.
• The system will include an opt-in newsletter-subscription option.
• The system will notify the specified dealer when the car-shopper indicates.

The user story nudges the developers into an outside-in perspective by emphasizing the user’s goal. The acceptance criteria make it clear that any solution that meets those criteria is acceptable to the product manager. It allows the interaction designer to create a compelling solution, rather than forcing him to recreate a boring experience.

Measurement Rocks

The Vast team measured the impact of making this change to their form, and saw between a 25% and 40% increase in conversion (people contacting dealers versus people abandoning the process when they see the form). That’s a lot more leads for the dealers, and presumably a lot more money for Kelly Blue Book and Vast. This is a great example of how you can measure the impact of investing in user experience, because it should spark ideas for you.

What can you measure and test and improve?

Now that’s ROI for you.

Engagement – that’s what this whole product management blogging thing is about.  Check out what Tyner Blain readers found to be the most engaging articles in 2009.

Deep Dives

If you’re new to Tyner Blain, you may be surprised by the length of the articles here.  I joke that they are long because I don’t have time to edit.  Stewart Rogers jokes that they are long because I’m incapable of writing a short article.  If you’ve been here a while, you know what you’re in for.  If you’ve been here a long while, then you’re glad (like me) that I don’t write one per day any more.

Product Management is simultaneously a broad and deep discipline, requiring us to have a breadth of perspective combined with a depth of insight.  We then have to apply that in a market context, effectively navigating the political waters of our organizations.  Most of the articles here either try to skim the breadth of a range of related topics, or plumb the depths of a single topic.  Doing that in under a thousand words is pretty hard.  Most of the articles here also link to other articles, to try and provide even more depth and context, and encourage additional critical thinking.

One measure of the quality of the articles here is how often they stimulate readers to read further, or dive deeper into the topics.  While web analytics won’t allow us to measure how thought-provoking an article is, we can look to see how many people dive into the linked articles, versus how many people move on to something else.  We can measure the bounce rate of an article to see how often people leave a page without following any of the “tell me more” links.

Looking at ~170,000 page views at Tyner Blain in 2009, narrowed down to only those articles with at least 100 page views, here is the top-ten list of most engaging (or “least abandoned” if you’re a “cup is half empty” person) articles:

1. Use Case Series: Introduction: A collection of articles on the “traditional” forms of use cases – informal, formal, UML.
2. Agile Development of Use Cases: A dive into the dynamics and cadence of an agile process for developing use cases.
3. How Do You Manage Market Data?: A collection of articles exploring different market-immersion techniques in depth.
4. Scheduling Requirements Changes – Part 2: A focus on practical techniques for managing change with an agile development process.
5. Yesterday’s BPMN Post Was A Big Fat Lie: A mea culpa and clarification of interest to folks following the BPMN Series.
6. Everything I Needed To Know I Forgot in Kindergarten: Why asking why? is important.
7. Plan Your Next Sprint by ROI – Part 1: Prioritizing by Bang for the Buck, not just Bang.
8. Ten Common Mistakes of Going Agile: A collection of articles about common pitfalls encountered when adopting agile practices.
9. Perpetually Almost Finished Projects: Organizing a project with discrete deliverables and rolling-wave planning to avoid the “90% done, 90% remaining” problem.
10. How To Use Timeboxes for Scheduling Software Delivery: One of my personal favorites.  A rational approach to making trade-offs when your “perfect” plan has to change.

OK Stewart, that’s only 519 words.

Unless you live in a world filled with unicorns and rainbows, writing realistic requirements is critical.  When you set unattainable goals, the best result you can hope for is a frustrated engineering team.  Write requirements that are attainable, and your team will surprise you with what they can achieve.

Attainable Requirements – Revisiting

A little over three years ago (ten unicorn years), I wrote Writing Attainable Requirements, as part of the Big Rules of Writing Requirements series.  In that article, we focused on the unique situations that every team faces – politics and people, and the shared challenge of implicit practicality.  Those factors are as important today as they were then.

Defining Attainable

Attainability, for requirements, is simply the answer to the question – can this be reasonably done?  Most, but not all, of the answer to that question comes from understanding if your team can build and test a solution to the problem you identify with your requirement.  Being able to create a solution requires that the problem you’re solving is tractable, and that the team creating the solution has the skills to solve it.

Being able articulate a problem and it’s solution is necessary, but insufficient – solving the problem requires that there be a feasible solution.

As the founders of the agile development movement explained in their pitch for “low overhead” process – people trump process.  I think it was Alistair Cockburn who added, “but politics trumps people.”

Finally, process is the often-overlooked element of determining attainability.  Realizing the benefits of a novel solution to an existing problem usually involves process changes – sometimes significant ones.

The next few sections go into more detail on each of these – tractability, feasibility, people, and process.

Tractability

Goldilocks ruled out the large bed because it was too large.  Eating the elephant, drinking from the fire-hose, boiling the ocean – these are all common phrases in the American vernacular, because so many people try to do too much (at one time).  There are many ways to decompose a large, intractable problem into smaller, addressable components.  Practicality requires that the problem you’re trying to solve is not too large.

If everything you do is small, however, you won’t ever accomplish anything big.  That’s where strategic product management shines.  As part of understanding your market, you identify large, valuable problems to be solved.  To solve those large problems, you have to break them up into smaller problems, and then address them individually.  You also have to make sure your team understands the big picture and the larger problem that ultimately needs to be solved.  A great way to share context while decomposing problems is with the use of Ishikawa diagrams.

And zooming in…

Maintaining context is a particularly critical component of working with agile teams, where each deliverable’s possible size is constrained further to what a team can accomplish in an iteration.

[click for larger version]

For some problems (and some teams!) this may be enough information to provide the context to develop a product backlog.  And your agile team will move forward into managing their own execution from the product backlog stage.  For larger or more complex problems (such as this example), you will need more detailed communication before diving into user stories / use cases.

The same Ishikawa diagram, with more detail, looks like the following:

[click for larger version]

Agile Product Management: Providing Context

This reduced granularity represents the smallest go-to-market atomic requirement.  Further decomposition results in smaller requirements, but they become too small, because they are no longer independently valuable.  I’m not talking about “the sum of the whole is greater than the sum of the parts” stuff here – I’m talking about “if you only get X, without Y and Z, it has no value” stuff.

Feasibility

Even after decomposing market problems into addressable chunks, you still have to make sure that there are feasible solutions.  One of the brighter software developers I worked with back in the day used to say “We’re writing software – it’s all ones and zeros – we can do anything.”  He also followed that up with “…but should we be doing that?”  The team I was working with at the time probably could do just about anything.  But not everything can be done with a given budget and a specific deadline.  Some things are simply infeasible.

Scrum teams measure velocity – a reflection of how much software they deliver in a given iteration, as a function of how much work they believe is involved in delivering.  This is a great “protected” measure of efficiency.  The team is protected from requests to do work that has little or no value – by measuring throughput in terms of effort, a team can get more efficient at doing more.  A requirement is infeasible, and therefore unattainable, when it exceeds what the team can accomplish given a fixed amount of capacity.  Project managers refer to this as the Iron Triangle – given scope, cost, and quality, pick any two as inputs and the third will be a variable output (absorbing the impact of uncertainty that arises during the development process).  I prefer to use timeboxes rather than that rusty old triangle, to emphasize my perspective that quality is inherent in delivery, and should not be a “variable output of uncertainty.”  Because quality is what usually bears the brunt of uncertainty.

plus  yields

When the solution to a defined problem is prohibitively expensive, it is not attainable.  The notion of cost should also be considered in the context of value.  Requirements with higher value can justify implementing solutions with higher costs.

Prohibitively expensive solutions make requirements unattainable.  Relatively expensive (relative to their value) solutions also reflect unattainable requirements.  You need to collaborate with your implementation team to understand the costs associated with any particular requirement to know if it is attainable.  This is one of the strongest arguments against throw-it-over-the-wall interactions between product managers and implementation teams.  Without that feedback about feasibility and cost of a solution, you can’t assure that the requirements you are proposing are attainable.

Regular readers of Tyner Blain will realize that this is just an alternate way of stressing the importance of considering ROI when defining requirements.

Politics

There are political realities that every organization faces.  It could be a benevolent dictator CEO, or an executive with an agenda or pet project.  It may simply be that there is value in the problems you’ve identified, but solving them is not aligned with your corporate strategy.  You may be focusing on dominating an existing market, where expansion into new markets is the company’s focus – or vice versa.  You may be focusing on top-line growth opportunities, while the company is fixated on cost-reduction in a time of economic contraction.

Each of these scenarios causes you to be fighting an uphill battle internally.  When that hill is too steep, you’re writing unattainable requirements.

Process

When deciding what to build, you also have to think about how you launch.  Launching, as Dave Daniels regularly points out on his Launch Clinic blog, is not just putting your product up on your website for sale.  You have to think organizationally about all of the other moving parts:

• Do your customers have the ability to absorb an update to your software?
• Can your customer service reps be trained in time to promote and support the new capabilities?
• Will your sales team be able to communicate a message that helps them close deals based on what you’ve just built?

Process is more than just the process of typing, compiling, and testing code.  Process includes everything needed to successfully go to market and sustain your product.  If you’re defining requirements that are too far ahead of your market they are not attainable.  If your requirements embody process changes that are too difficult or expensive for your customers or your organization to absorb, your requirements are not attainable.

Conclusion

Attainable requirements are requirements that

• Have a reasonable cost to implement,
• Are aligned with your company’s strategy and stakeholder’s agendas, and
• Result in solutions that can be consumed by your company and your customers.

Make sure you’re writing attainable requirements.

Perpetually intermediate (competent) users.  Users who briefly exist as novice users and never become experts. Most of your users are competent, and you should design for them.  Competent users have different needs and different expectations than novice or expert users.  How do you know your user’s competency levels, so you can design for them?

User Competency

User competency is a concept I first read about in Alan Cooper’s The Inmates Are Running The Asylum.  Cooper’s contention is that the level of expertise of your users follows a bell curve, or normal distribution.

When we’re designing software we need to keep in mind that most of our users will be competent – neither experts nor beginners. Alan Cooper’s studies tell us that user skill levels follow a bell curve. He talks about competent users as perpetual intermediates. Some users drop out of the bell curve when they stop using our software. The rare user becomes an expert. Most users only learn enough to get their real job done.
Competent Users and Software Requirements

Cooper contends that the combination of learning curves and natural user tendencies to stop learning or abandon a software application are the sources of this distribution of user competency.

Experience Curves

Experience curves represent the diminishing costs over time of manufacturing something repeatedly.  The process of manufacturing something gets more efficient as you get smarter about the process.  The process of using software is at least analogous to, if not a specific example of a manufacturing process.  If you’re using an email application, you are manufacturing email messages.  In a CRM system, you are manufacturing contacts, or contact reports, etc.

By treating any “using the software to do something” interactions as a process, you can measure the cost (how long it takes) of the user’s interactions.  Applying the math behind experience curves, you can predict the reduction in cost (to your users) over time, for any set of interactions.  Experience curves take into account that some processes are inherently more learnable than others.  This property of learnability is reflected as an efficiency coefficient – how efficiently someone can learn ways to reduce the cost (time) needed to perform the interactions.

This gives us an approach to quantitatively model user competency.  Having a definition allows us to model competence.  And measuring competence allows us to manage product design in the context of user competency - designing for competent users.

Defining Competence

The first step to measuring competency is to define the model.  I am proposing a definition in this article that I suspect will yield insights (to help us manage our products). I was unable to find any quantified definitions of competence, when researching it as part of a client engagement.  If you have, or know of a model, please share it in the discussion below this article.

• A competent user is someone who learns to perform a task in half the time it initially took them.
• An expert user is someone who can complete a task in 10% of the initial time.

This definition is guided by an expectation that Alan Cooper’s premise about perpetually intermediate users is true.  Being a novice user is a very transient state, and becoming an expert is very infrequent.  The goal of the definition is to be able to segment your users and make well-informed design decisions.

A Proposal, Not a Doctrine

This is a proposal that doubling performance reflects competence, and achieving a ten-times improvement represents expertise.  It may be that some different measure of performance improvement more accurately reflects competence and expertise.  We have to test it to know.

The experience curve is defined mathematically by Henderson’s Law.  It states that the time to complete a task is a function of the number of times you have previously done that task, adjusted by the “elasticity” of the cost of that task.  In other words, some tasks are easier to improve than others.  If you populate a table with the results of applying Henderson’s Law, you get the following:

[larger image]

• Each row in the table represents the Nth repetition of a task.
• Each column represents how easy that task is to learn – progressing from “hard to improve” on the left, to “easy to improve” on the right.
• Each cell in the table represents how long it would take to perform the Nth repetition of the task, as a function of how easy it is to improve your performance at the task.

The above definitions represent what experience curves predict mathematically, when the task initially takes 60 seconds.  The following definitions reflect the proposed user competency model:

• A user is a novice user until she has learned enough to cut the time-on-task in half.  These cells have a white background (and are in the upper left area of the table).
• A user is a competent user when the time needed to complete the task is between one half and one tenth of the initial time.  These cells are shown with a yellow background (and are in the central area of the table).
• A user is an expert user when she can complete the task in less than one tenth of the time required to initially complete the task.  Theses cells are shown with a red background (and are in the lower right area of the table).

In the absence of empirical data, I used my intuition to suggest that a representative experience curve for a typical task performed in software would be one with an “elasticity” of 50%.  For a task with those learning characteristics:

• A novice user would cut the needed time in half on the 4th repetition of the task, and would be considered to be a competent user.
• A competent user would further reduce the time needed to one tenth of the initial time on the 100th repetition of the task, and would be considered an expert user.
• The 50% elasticity column is surrounded by a black border, and the number of repetitions required to advance to competent or expert status is also highlighted with a black border.

Inferring Competence

Since different processes have different learning characteristics, you have to figure out how easy it is for your users to improve at your processes.  To do that, you have to study (or at least measure) your users’ interactions with your software.  In the 50% curve highlighted above, a user is capable of cutting their time-on-task in half by the fourth time they perform the task.

If data from your initial testing (or measurement) reveals this to be true, then you have selected the correct curve (the correct column in the table).  If it takes more or less time to reach this level of improvement, shift to the left or right to find the appropriate curve.  If software-interactions are reasonable analogs to manufacturing processes, then the experience curve projects an expected rate of improvement on task.

The following graph isolates the time-on-task data for a user who is learning to improve when repeating a task (process) that matches the 50% elasticity curve.

[larger image]

Note that the number of repetitions of the task (the X axis) is represented as a logarithmic scale.  The data points along the curve correspond to the cell values in the table above (for the 50% column).

Shades of Gray

One nice thing about this quantitative approach to inferring competency by measuring usage is that your measurements are per-process.  Users are not “purely novice” or “purely expert” – they can be experts at some processes, while remaining neophytes at other.  There is also awareness, for any particular process, of “how much competency” a user has.  This allows you to refine your assumptions of the steepness of the learning curve, and of the thresholds (doubled performance and ten-times performance improvements).

Improvement Over Time

Any particular learning curve can be considered relative to calendar time, to see how quickly a user will progress along the curve (as a function of frequency of use).  This can be useful for determining the ROI of improvement in a particular process.

The following graph shows how an 80% learning curve overlays a calendar for tasks that happen daily, weekly, and hourly.

[larger image]

The graph shows that for a weekly frequency, after 16 weeks, the task time has reduced from 300 seconds to 100 seconds. With a daily frequency, the task time is even lower – about 60 seconds. This graph shows nothing other than converting the academic learning curve graph into one that incorporates calendar time and frequency of occurrence.

Software Usability and Learning Curves

One approach to inferring user competency would be to measure how long a user has been using your software.  The variation in how frequently different users perform the same task will introduce an error into that inference.  You can avoid introducing that error into your modeling by counting the number of times a user has performed a task.

Applying the User Competency Model

The advice in previous articles, and from Cooper’s book, and from this great article on the  Coding Horror site, encourages us to focus on the competent users.

I’m working with a client who needs to prioritize a set of capabilities and establish design principles for a product.  We will incorporate this user competency model as part of our analysis.

Hopefully we’ll have an opportunity to collect data to validate and / or refine the model.  I’m proposing that we first gain some insight into the which users (novice, competent, expert) drive the most revenue and profit from use of the product – to establish the importance of each category of user.

For this product, I suspect that we will find many more novice users than a normal distribution would predict.  If that is true, the next question will be to understand if we are dealing with a normal behavioral dynamic, or if characteristics of the current product “force” novice users to abandon it before they achieve competence.

Either way, we will have a framework for prioritizing the goals of the novice, competent, and expert users.

How would you apply a model like this to improving your product?

Growth is a make or break measurement for products and companies.  Investment is often determined by expected value, which is based (in part) on expectations of growth.  When you create a product, there are aspects of growth – how many people can use your product, and how many people do use your product.  When dealing with a freemium business model, there are two elements of use - paid use and free use.

Three Goals of Growth

You can think about growth in terms of three goals – increasing your servable market, increasing your market share, and increasing your paid market share.

• Servable Market – The number of customers that could potentially use your product.
• Market Share – The percentage of the market (or of the servable market) that does use your product.
• Paid Market Share – The percentage of the market (or the servable market, or your users) that use and pay for your product.

As a product manager, when you consider capabilities to add to your product, you need to understand which goal your capability is designed to address.  These are all inter-dependent metrics, so it can be challenging to stay focused on each.  For example, if you have a 1% conversion rate from free customers to paying customers with your freemium business model, you may assume that by increasing the number of free users, you will automatically increase the number of paying customers.  That might be true, it might not.  Ultimately, you want to increase the number of paying customers – so that is part of all of these decisions.  When making each decision about prioritizing the next capability to add to your product, make sure you understand the primary growth goal you are affecting.

To keep the language from being too contorted, in the rest of the article, I’ll talk about users and customers as people, and not try and differentiate companies and people.  Users are people (or companies) that use your product or service.  Customers are people (or companies) that pay to use your product or service.  Your product could be a physical product, a software product (or license), software provided as a service, or a service.  I’ll refer to it as a product.

Growing Your Servable Market

The pedantic definition of your servable market is all users who could use your product.  A better way to think about your servable market is all people who experience the problems you’re trying to solve, who have reasonable access to your product.  People who don’t have those problems are not part of your market.  People who have those problems, but can’t reasonably use your product are part of your un-servable market.

Imagine you build a product that only runs on Windows Vista.  Windows XP users are not part of your servable market – they could upgrade to Vista and use your product, but they haven’t yet.  There’s a barrier to entry that they have to overcome.  Perhaps you have a pizza shop that delivers within a 10 mile radius and allows customers to place an order for pick-up and drive to your store and pick up the pizza.  If your shop is in Mountain View, technically people could drive from downtown San Francisco to pick up a pizza – but there’s a barrier to entry (the time and expense of driving to Mountain View) that makes those downtown San Francisco people part of your un-servable market.  

The size of your servable market reflects the potential number of users you could serve.  Your focus on growing your business can include changes designed to increase the size of your servable market.  You can write (non-functional) requirements that express this goal.  One type of non-functional requirement is platform support - supporting different operating systmes; supporting netbooks (with less-powerful processors) as well as laptop computers; supporting devices with slower network connections; and supporting visually impaired users through your user interface.  Sometimes, the requirement to support a particular platform is represented as a constraint (e.g. you must provide support for linux users).

Many products are developed today as software as a service (SaaS) applications - and they are usually implemented as websites, where people can use the product through a web browser user interface.  The approach of ubiquity of web browsers and “always connected” devices leads to platform selection for web-based products being a less-constraining  factor in definition of your servable market.  You may still be constraining yourself based on the specific browsers (Internet Explorer, Safari, Opera, Chrome, Firefox, etc) that you support.  You can gain a lot of insight through an analysis of your traffic, or you can rely on general trends.

Your business model may only support customers in the United States of America (like the soon to be launched Google Voice).  Your user interface may be text only (and not support screen readers or other devices).  You may be building a Facebook application.  You may be selling headphones with controls for the new Apple iPod Shuffle (that has no integrated controls).

What’s important is that you identify the things you need to do (added capabilities, platform support, country availability) to increase the size of your servable market – and associate them with the theme (or context) of increasing the size of your servable market.

Growing Your Market Share

Your market share is the percentage of those people you could be serving who are your users.  You can also think of this as the size of your user base, but thinking of it as a proportion of possible users provides more insight (because it provides visibility into how many people aren’t in your user base too).  As part of developing a business model (or financial justification for investment in a new product), you will have developed a model with some assumptions about market share.  Venture Capitalists used to joke about pitches that projected capturing “1% of a trillion dollar market, and therefore a $10 billion business” without providing justification.

You have to develop a plan for how and why you will achieve a particular market share.  One approach is to develop a viral marketing message, for products that “sell themseleves.”  Another approach is to develop a product that is inherently viral.  You can, of course, apply multiple tactics.  

The point here is to identify those product capabilities or features that will encourage users who can use your product to use your product.  Adding support for a platform does not encourage users to use your product – it just makes it possible.  Adding a solution to a valuable problem encourages people to start using your product.  Your product may be one that is designed to inspire love, or one that is designed to reduce annoyances.  An effective (and right-minded, in my opinion) approach is to identify personas within your servable market, and focus on their problems.  As a product manager, you choose which of those problems to address – and your team invents a solution approach.  You then validate that this approach is an effective one for the particular problems of the target persona.  

You may also want to define market segments to subdivide your servable market, so that you can develop unique strategies for presenting your solutions to potential users.  For example, you may find yourself with a surplus of cheap black pearls.  You can create two market segments – people who buy cheap jewelry and people who buy very expensive jewelry.  For the first group, you may position your product as cost-effective elegance.  For the big-spenders, you may position your product as something rare, unique, and not like those “everyday” pearls their friends all wear.  [I am pretty sure this is a reference to an anecdote from Predictably Irrational, but I'm not sure.]

What’s important is that you identify the things you choose to do (develop solutions to specific problems) in order to increase your market share – getting viable potential users to become active, actual users – and associate them with the theme (or context) of increasing your market share.

Growing Your Paid Market Share

People normally talk about this as conversion - conversion of free users into paying customers.  We’ll do the same.  I wanted consistent section headings. so I came up with “Paid Market Share” as a title.

Conversion is a key element to a freemium business model – one that offers two or more versions of a product.  One version is free to use and the other is a premium version that users pay to use (thus becoming customers).  Your conversion rate is the percentage of your users who are paying customers.  As we mentioned in the previous section on growing market share, you will have developed a business model for how much market share you would get and when and why.  If your business model is (or involves) a freemium product offering, your plan will also include projections around the number of paying customers you will have over time.

This is where focus becomes valuable.  As you identify problems to be solved (as part of developing a market-share growth strategy), you need to also decide which problems should only be solved (or should be solved better) for premium customers.  For example, the free version of MediaMonky (audio software) solves the problem of being able to listen to your audio files on your computer.  The catch is that when you add new music, you have to click a button that updates your “library” (so that MediaMonkey will play the new music).  The premium ($20US) version will automatically detect those new files.

You may also decide that a product is free for some users (companies with fewer than 10 users), and charge a fee to larger companies (10 or more users).  I don’t consider this to be a freemium model  In a freemium model, any user can choose either version of the product.  A related model is one where one set of users pay while another set doesn’t.  The distinction is probably best made by thinking about market segments.  If all (potential) users in one market segment only consider the premium version, and all (potential) users in another market segment would get no additional benefit from the premium version, then you would approach your growth strategy differently.  You wouldn’t be focused on converting customers, you would be focused on attracting paying customers – which is the same approach you use for growing market share.  In this case, you’re trying to grow the “pay for our product” market segment.

When you do have a freemium model – one where any given user can choose to pay or not – your focus is on identifying problems to solve for premium customers.  Not only do you have the normal “is there value in solving this problem?” question, you also have a positioning question.  Will you alienate the users of your free product (or inhibit market share growth) by offering a capability only to paying customers?  Another interesting strategy is to introduce new capabilities to premium customers first, and then have them “trickle down” to the free version as new capabilities are introduced for premium customers.

This strategy seems to be a powerful way to leverage market insights to continuously make your product more valuable than your competition’s products.  This could be a great way to “set the pace” for your market – forcing your competition to be reactive.  Slower competitors will lose not only to your premium product, but to your free version, which will out-value their paid products.  You get to define the rate at which your market changes.

What’s important is that you identify the things you choose to do (develop solutions to specific problems, and position them appropriately) in order to increase your rate of conversion – getting current free-product users to become premium-product (paying) customers.

One Way to Bring it all Together

I’ve done work with a client who has a distributed leadership team, so whenever I wanted to write on a whiteboard or stick stuff to the walls, I had to do it electronically.  We created a “strategy” Google-spreadsheet.  We have three columns one each for – grow the servable market, grow our market share, and grow our paid market share.  We added items into the appropriate columns, and developed relative priorities within each column.  The client strategy at any point in time, clearly drove the balance of emphasis for a given release across those columns.  Some releases, for example, were entirely around growing the servable market.

Conclusion

It isn’t enough to just consider the value of a particular capability, you have to put it in the context of an overall growth strategy.  Even if you’re solving problems from all three columns (servable market, market share, and paid market share growth), your top down strategy should drive how much effort you dedicate into each column, for any given release.

Planning by ROI.  Hmmm.  Isn’t that impractical?  In an econometric way, yes.  But you can still estimate the relative value of the capabilities / stories you’re planning for your scrum sprints.  The point is – don’t look only at value – also look at costs.  While “ROI” may be a poor choice of terms, “bang for the buck” is not.

Mea Culpa

In the previous article, Plan Your Next Spring By ROI: Part 1, I made a bad choice of sloppily using ROI (about a dozen times) to describe “bang for the buck.”

Just over a year ago, we showed how prioritizing by ROI delivers value faster.  The reason this works is that each sprint (or timebox) has a fixed capacity for work, so you want to cram as much value into that box as possible – not just do the most valuable thing first.  Here are a couple images from the previous article that show the impact of using this approach.

[...]

So far, we’ve argued that prioritization should be by ROI, and while accounting for other people, should be user focused.  Actually, nothing above is sprint-specific, the same concepts apply to any product planning and development approach.

Plan Your Next Sprint By ROI: Part 1

[Editor: Note - the author has also messed up again, this time by burying the lead at the end of the article.]

Giants.  And Their Shoulders.

I had a great conversation over the weekend with Luke Hohmann, founder and CEO of Enthiosys, and realized my mistake.  Consider the titles of three (excellent) articles Luke wrote earlier this summer:

1. Why Prioritizing Your Product Backlog for ROI Doesn’t Work (Part 1 of 3)
2. Developing Attributes for Backlog Prioritization (Part 2 of 3)
3. Prioritizing for Profit (Part 3 of 3)

Not to mention giving a presentation at Agile’08.

You’d think I would have noticed, and been more careful in my use of language.  Sorry about that.

Luke, in his first article, provides several arguments against econometric assessment of ROI, and I agree with all of his points.  In his second article, Luke goes into details about defining attributes for reflecting the goals of internal stakeholders, external stakeholders (buyers and users), and the system.  I promised to do the same in this article, but frankly, why reinvent the wheel?  Our differences from Luke’s article would at best be nuanced, so the best use of your time (and mine) is to just go read his second article.

Our experience is that successful Agile Product Managers have an almost natural, intuitive feel for prioritizing backlog items to drive profitable growth. Fortunately, this natural ability is greatly enhanced when they make the link between the backlog and their ability to drive profit explicit. As an aside, it is this set of attributes that most clearly distinguish the Scrum-centric role of a Product Owner with the full responsibilities of an Agile Product Manager.

Developing Attributes for Backlog Prioritization (Part 2 of 3)

Including Costs in Sprint Planning

There are two dimensions by which to keep costs in mind when planning a sprint.  The first is in determining how much work to schedule for any given sprint.  When you use timeboxing to plan a sprint, you’re saying “we have the capacity to do X work per sprint” and “we should only schedule X work per sprint.”  Your plans are based on estimates, so you need to have a feedback mechanism to make sure you’re doing a better job in planning each sprint than you did with the previous sprint.  Burndown graphs are a great way to do this.  The burndown provides feedback within the sprint too, not just at the end.

Mike Cohn, of Mountain Goat Software, gave a great presentation on how to estimate costs for an agile project.  Without giving it away, he proposed (starting on slide 14) using planing poker to get quick estimates of user stories [more details] at the product backlog level.  Then, more detailed estimates are created for the tasks within the sprint.  If your team uses use cases, you can choose to use use case points as a low-overhead estimation method (but not nearly as low as planning poker) to determine the initial estimates of costs.

A Planning Example

You have the following set of items being evaluated in your product backlog.  Note that there are a mixture of strategic, user-centric, and code-refactoring items under consideration.

• Profile Editing – As an insurance agent in the community, I want to be able to edit my profile to personalize my identity within the site.
• Data Abstraction Refactoring – The system’s data model is inelegant and needs to be refactored, if ongoing development is to be easier.
• Action Notification – As a manager of agents, I want to be notified whenever my agents submit quotes to potential customers.
• Grouping Items – As an insurance agent, I want to be able to group contacts, proposals, communications and other items, so that I can organize my work.
• Reporting – As a manager of agents, I want to be able to run reports to track financial metrics and activities by agent, product, geography, and time period, so that I can manage my team effectively.
• Serializable Objects – The system’s implementations for data backup and data export are redundant and brittle and should be refactored to clean up the code.
• Customer Centric UI – The user interface is currently product-centric in approach.  Our strategy to offer multi-vendor product lines will require agents to focus on customers first.

Using the value-estimation and cost estimation techniques outlined by Luke and Mike you determine the following value and cost estimates for each “story.”

If you were to ignore the cost estimates you would do the data abstraction refactoring and customer-centric UI stories first, followed by the profile editing, action-notification, and other tasks in order of descending value.  As it turns out, that is not the way to maximize value over time.  At first glance, the profile editing story looks like the biggest bang for the buck.

Create a scatter plot of value versus cost.

[larger version]

You can clearly see the differences in value and differences in cost of the different stories.  If, however, you add the element of value/cost – bang for the buck – and prioritize by it, you will schedule stories in a different order.  Value / cost is also the slope of a line from the origin of the graph to each story.  If you draw those lines, you see the following:

[larger version]

To maximize value delivery over time, work across the chart in a clock-wise direction.  Profile editing is the first task, but grouping items and data abstraction refactoring are tied for second place.  This is different because you’re prioritizing by bang-for-the-buck, not just bang.

Collaboration And Agile

When you consider the agile manifesto

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas

© 2001, the above authors. this declaration may be freely copied in any form, but only in its entirety through this notice.

you see that there is an opportunity to take this even further.

Have Developers Improve ‘The Plan’

I proposed an idea to the development lead on an agile team a couple weeks ago.  A couple hours later, he pinged me to let me know he had just applied it and was very excited.  A quick whiteboard session (started with a version of the diagrams above), and he was able to immediately make his sprint better.

I told him that when I was still slinging code, and later, when working with teams of engaged and talented developers, I often ran into the problem of having a “cool capability” and wanting to get that capability into the current release.  I wasn’t pushing for the capability because it was high bang-for-the-buck, or even high-bang – I just thought it was cool.  So I would implement it.  Bad Scott, no biscuit.

What I proposed was that when a developer has a favorite feature, instead of just trying to squeeze it in, the developer should figure out how to make it cost less, until it had the highest bang for the buck.  Each initial cost estimate is a planning-poker estimate.  By spending cycles on design, a developer can figure out a cheaper way to implement it.  And that moves the story to the left on the chart.

Your challenge, visually, if you want to bump reporting up in the planning backlog, looks like the following:

[larger version]

You, as a member of the implementation team, have to come up with a design for reporting that is roughly one-third the cost that was originally estimated.  Do that, and it gets bumped up in the queue.  [And yes, I realize that very few developers get excited about reporting.]

The product manager‘s job is to make sure you get all the dots in the right place vertically.

The implementation team puts the dots in the right place horizontally.

Everyone embraces the ability to change the graph.  The product manager moves the dots up and down as she learns more about the needs of the market.  The implementation team moves them to the left (and sometimes to the right) as they design better solutions.

Quoting Luke again – “Our experience is that successful Agile Product Managers have an almost natural, intuitive feel for prioritizing backlog items to drive profitable growth. Fortunately, this natural ability is greatly enhanced when they make the link between the backlog and their ability to drive profit explicit.”

This feels intuitive to me.  What about you?

You’ve got a giant backlog of user stories and product capabilities.  How do you determine which stories to implement right now?  By the estimated value of each story?  Pick the ones the developers want to build next?  How about picking the stories that maximize the ROI of the sprint?  To do that, you need to estimate both value and cost.  While remaining agile.

ROI Refresher

In our earlier Definition of ROI: Return on Investment article, we provide an explanation and example of calculating ROI.  In short, ROI is the acronym for return on investment.  How much return do you get, as a percentage of what you have to invest.  If you invest $100 to get $120, your return is $20 (the profit).  Your investment is $100.  Your ROI is 20% (20/100).  Another way to think about it is bang for the buck.

Prioritizing by ROI versus Prioritizing by Value

You can prioritize based on value – “Do the most valuable thing first.”  And that is a good, but suboptimal approach.  We even suggested it as “the best” prioritization technique in our prioritization article from almost three years ago.  Since then, by continuing to focus on software product success, I’ve learned to believe in a better way.

Just over a year ago, we showed how prioritizing by ROI delivers value faster.  The reason this works is that each sprint (or timebox) has a fixed capacity for work, so you want to cram as much value into that box as possible – not just do the most valuable thing first.  Here are a couple images from the previous article that show the impact of using this approach.

Identified stories, from highest value to lowest value (V = Value, W = Work (or cost)).

If you can schedule 30 units of work in a sprint, your first sprint will deliver 19 units of value, and your second sprint will deliver an additional 9 units of value.  The thirs sprint delivers the remaining 15 units of value.  Why more value in the third sprint than the second?  Because you didn’t sort by ROI, you sorted by value – but were constrained by cost.  When you sort by ROI, you get the same tasks, in the following order:

With this sorting, your first sprint delivers 25 units of value, your second sprint delivers 9 units, and your final sprint delivers an additional 9 units of value.  There are more details in the original value-maximization article.

This is critical to executing a market-driven strategy that will dominate your competitors.  You not only have to stay tapped in to the market needs, but you have to deliver faster than the other guys, or your distinctive insights will be wasted.

Political Prioritization

Really?  Yes.  The first risk to the success of a project is that it gets canceled.  Only the projects that don’t get killed even have a chance to be right, and therefore maximize value for your customer (or company).  That means you need to prioritize in a way that satisfies your stakeholders.  After attending last year’s IBRF, I wrote the following article as an interpretation (and slight extension) of Roger Burlton’s very good presentation about process improvement.  When you’re tasked with improving a company’s processes, the first question you have to ask is “which ones should we improve first?”

From that article:

This analysis focuses on principal stakeholders. The goal of this analysis is to encourage your principal stakeholders, or sponsors, to prioritize the most valuable processes first. Not all stakeholders are created equal, so you have to weight the relative importance of each stakeholder. You can weight them politically, by influence, or by whatever factor is most likely to get buy-in from the executives.

Stakeholder Value Matrix

The end result is a prioritized list of processes to improve, based upon which ones cause the combination of most pain (current state) and most gain (when improved) across your stakeholders, based on maximizing the utility of the group of stakeholders.

Visually, the preceding diagram shows the end results of aggregating the stakeholder inputs in terms of pain and gain.  A couple utility curves are overlaid (arrow shows increasing utility) to show what to do first.  The 1,2,5 processes are clustered together in the “highest utility” or “most value” range.  This approach would focus on those processes first.

Persona Prioritization

When developing software, you should primarily be user-focused.  [Note: this is somewhat of a religious debate, I guess we worship at the chapel of UX.]  Luckily, most of the time, stakeholder goals and user goals are not in conflict.  However, to sell your solution, you need to make sure you are addressing both buyer goals and user goals.  Sometimes they are the same, like with consumer products, and sometimes they are not, like many enterprise software products.

As a product manager, you have to know when to listen to, and when to ignore the squeeky wheels.  If you’re in a start-up doing its second product, solving a new problem for a new market segment, you’re dealing with founder’s dilemma.  That founder has a bunch of stakeholder goals for you, and the juice to make sure you address them.  If you’re trying to get analyst’s attention [good article by April Dunford at Rocket Watcher], you may feel pressured to prioritize “check the box” features that may be important only to buyers and analysts, without actually adding value for the users.

Of course, you can’t ignore analysts and buyers and internal stakeholders.  If your product gets cancelled before you get to market, you’re done.  If analysts criticize or ignore your product (for some markets), you lose a lot of sales.  And if buyers don’t “think they want it”, they won’t buy.  You never get a chance to demonstrate value to the users.

But focusing solely on those folks means you won’t have a product that people love, and you won’t generate word-of-mouth, customer loyalty, or value for your customers.

So any solution needs to account for the needs of the users.  Organizing users into personas, and making strategic decisions about which ones are important will help.  Include the other people (buyers, etc) but use a persona-centric model for describing what you’re doing – just to make sure the users are at the center of your decisions.

Intermission

So far, we’ve argued that prioritization should be by ROI, and while accounting for other people, should be user focused.  Actually, nothing above is sprint-specific, the same concepts apply to any product planning and development approach.

In part 2, we’ll look at defining user stories and user persona.  We’ll look at the value of each story relative to cost, and show a couple low-overhead ways to easily visualize this info – both driving scheduling and motivating the team to improve.

Business rules are often hidden in processes as hidden decisions.  Once you discover that hidden decision, how do you communicate the impact of exposing and managing the decision?

Hidden Decision

In our previous article on hidden business rules and enterprise decision management, we looked at a process that had a hidden decision.

First, take a look at the process with the decision still hidden:

Here’s the process (with the decision exposed) from that article:

The decision that was hidden was an implicit decision – “Pick One?” – for which the answer was always “No.”  After making this discovery, and communicating with your team, you then have to decide if you are going to make the decision explicit.  Making the decision explicit will involve defining the business rules for how to make the decision, and (if done in software) implementing the decision, or the ability for a person to make the decision.

Impact of a Decision

The diagrams above allude to the fact that there is money to be made in the process, in the “Continue Automated Process” step.  We need to provide a better understanding of how / when money is made.  A slight expansion of the process flow above (replacing the “Continue Automated Process” with a couple steps, looks like the following:

[click for larger version]

The last step introduces a real-world decision: “Sell products?” and only when the answer is yes does the process “Earn Revenue.”

If you’re analytically minded, you can apply percentages to each branch of each decision, and add everything up to figure out an answer.  That might be reasonable for a simple example like this one, but it can become too complex with a more complicated process.

There’s an easier way.  And since the goal is to communicate the impact to someone, you want an easier way.

Truth Tables and Decisions

A truth tables is a tried and true artifact for describing combined sets of decisions.  You can use it effectively to extract information from a decision-heavy process diagram.  Consider the following application of a truth table to the process above:

[larger version]

Each of the decisions in the process flow has a column.  Each row represents a unique path through the flow.  Each cell contains the result of the decision, for that path.  Note that when a path does not involve a decision, there is no value in the cell.

That provides a comprehensive view of the process, which you could build upon.  However, in this case, you are focusing on the impact of the hidden decision (“Pick One”).  The following paths, from the truth table, involve that decision:

[larger version]

Ignoring the other paths, and re-organizing, you find the following:

[larger version]

50% of the time, when “Many” results are found in the first decision in the process, if you pick one of the results, you will earn revenue.

[larger version]

When you do not pick one, 20% of the time, the customer (or user) abandons the process.  For the users that do not abandon the process, you earn revenue 50% of the time.  Taking both into account, you earn revenue only 40% of the time.

Adding Up the Impacts

When there are “Many” results from search, you can improve the performance of this part of your process by 20% – specifically, you increase revenue by 20%.  But only if you always “Pick One” of the many results. You can extend this analysis to determine the percentage of None/One/Many outputs from the search at the start of the process, and determine an overall impact on the process.  If search returned many results 10% of the time, you can increase revenue (overall) by 1% by always picking one of those results.  That gives you the business justification to calculate ROI on exposing the hidden decision.

The end result is that you have a simple way to communicate the impacts (truth table) in the language of stakeholders (ROI).

Your strategy should be driven by the needs of the market.  Becoming market-driven is critical to intentional product success.  But it is not enough to understand your market.  You have to sustain your understanding, and take advantage of it, competitively.

Markets Evolve Over Time

We all acknowledge that markets are not completely static.  If they were, there would be no impetus for innovation.  Customers change, their environment changes, and the competitive landscape changes.  All of these changes cause a market to change.  Even if a given market were static, you don’t have perfect information about it.  Your understanding of that market increases over time.

The market is an abstract concept, and you have to make decisions about something tangible.  What is tangible is your understanding of the market.  And you make decisions based on your understanding of the market. You can visualize the market in a way that may cause you to rethink (or solidify) your approach to solving market problems.

In the chart above, we are showing that there is some amount of information (think of information as an understanding that we haven’t developed yet) about a market.  It represents the potential of what could be understood about the market.  Over time, the amount that can be understood increases.  Also note that there is already a non-zero amount of “understanding” at the start of the chart.  The market did not spontaneously spring forth from an oyster the moment you decided to look at it.

You Affect Your Markets

There is also an important dynamic to consider – you affect your markets by introducing solutions.

Whenever someone introduces an innovative solution to a market problem, the market changes.  Solving that problem exposes another problem.  Or it may introduce another problem.

• When cars replaced horses, they introduced atmospheric polution problems.  Before that, we only had to worry about horse waste.
• When airplanes solved the problem of traveling long distances in a short amount of time, more people began traveling further.  And those people discovered a need (desire) to stay connected (phone, internet, etc) while en-route.
• When it became convenient to carry music around with you on a portable player, the problem of carrying all of your music around with you was discovered / created.

You can argue that these problems were introduced because of innovative solutions, or that the problems were latent, and were only discovered because of the innovations.  It doesn’t matter.  What matters is that these problems were previously unaddressable, and now solutions to these problems have value.

Your Knowledge of Your Markets

Using the diagram above as a baseline of what can be known, and overlaying your knowledge looks like the following:

At the moment when you decide to understand a market, you don’t know anything about it.  Your knowledge rapidly approaches what “can be known”, and assuming you have perfect research, elicitation, and synthesis skills, you eventually understand all that can be understood about the needs of the market.

So much advice has been written about being market focused, that it is easy to think that this is all you need.  As soon as you understand “everything” you define requirements and start creating products.  Being market focused is extremely important.  But you have to sustain that focus.

Competition in Your Market

In the diagram above, you see the dynamics of your market.  You develop an understanding of your market, a competitor delivers an innovative solution to a problem, and you have to play catch-up.  Note – this is only talking about playing catch-up in your understanding of the market.  You also need to continue to develop new solutions for the market in order to capitalize on your understanding.

The next diagram shows what this will look like when there is active competition in the market.

The diagram above shows three disruptive events.  You innovate, someone else innovates, and you innovate again.  Each time someone else innovates, they have an edge on understanding the market.  They just solved a problem, uncovering new problems that can be solved, and you have to gain that knowledge too – it wasn’t available (or didn’t exist) previously.

Competitive Advantage

Each time you innovate, you establish a temporary competitive advantage.

When you introduce a novel solution to a problem, the market uncovers new problems.  Presumably, you have already thought about the next steps.  You have already engaged the market (through prototyping and other elicitation activities) to get feedback on your solution.  Hopefully, as part of that exercise, you’ve also explored “what’s next” and incorporated that into your plans.  This is the advantage you have created.  If you don’t explore “what’s next,” you waste that opportunity.

The opportunity won’t last forever.  Your competition will see your solution, talk to your customers, and otherwise explore “what’s next.”  It will take time, but they will catch up.  Over the long term, you benefit from exploiting a series of these opportunities.  Look at the iPod and the Zune.  Apple has delivered a series of disruptive changes, continually building on what they have learned.  Microsoft is playing catch-up with the Zune.  Each version of the Zune shares a lot of design cues and features with earlier iPods.  To Microsoft’s credit, they are trying to innovate too – like with the “squirt” technology that lets you share a song temporarily with another Zune user.  Either this is not a valuable problem, or the solution is not effective – because it has not taken off (in buzz, market share, or general clamoring for an iPod-based equivalent).

Fast Follower Competition

Some competitors intentionally act as second movers.  Instead of focusing their efforts on discovering problems or developing innovative solutions, they focus on playing catch-up very very well.

A fast-follower reduces (but does not eliminate) your temporary advantage.  At the same time, the fast follower creates their own advantage relative to every other competitor in the market.  This can be a very effective strategy, especially when there are multiple innovators in a single market.  The fast follower can follow the best ideas of each innovator, possibly even ending up with a better overall product than any of the innovators (who are slow followers of other companies).

Microsoft is clearly a follower of Mozilla when it comes to browsers.  The tabbed navigation concept introduced in Firefox was a game changer, at least for the tech-savvy who need to juggle multiple web pages at the same time.  Personally, I was thrilled to move from multiple Internet Explorer windows to a single Firefox window with multiple tabs.  IE7 has that capability now too.  I don’t know the browser market well enough to know if I would characterize Microsoft as a fast follower, but my guess is no.

After the post-World War 2 reconstruction in Japan, the Japanese automakers definitely acted as fast followers.  They did it with assistance from the US, and even moved past the US automakers by choosing to listen to Deming (a guru on efficiency and quality) instead of ignoring him like the US big 3 (automakers) did.  And it did indeed pay off for them – Toyota is poised to unseat GM as the number one automaker in the US for 2008.  There are certainly other factors at play, but the Japanese companies would never have had a chance if they had not been competitive in the first place.

Business Agility

Business agility is one of the hotter buzz-phrases these days.  The business-rules-automation market is thriving on the desire of businesses to become more agile.  Why does agility matter?  Because it can make you dramatically more competitive.

Business Agility is not the same as agile development!

Business Agility is a measure of the ability of a business to adapt to the changing needs of the market.  To adapt quickly, your company has to accomplish two things:

• Quickly identify and understand the changes in your market.
• Quickly deliver new and improved solutions that address the changing needs of your markets.

[How to quickly execute is covered in other articles in the business rules (including their impact on agility) and agile development categories on Tyner Blain.]

In the diagram above, you (the red curve on the top) are innovating repeatedly.  You are also adapting to the changes in your market from other innovations (and other sources of change, not shown).  You are practicing agile product management.  Your competition (the green curve on the bottom) is not.  As this trend continues, you see that your exploitable advantage becomes a sustained advantage.  You will have an ever-increasing advantage as you stay tapped into your market.  If you are executing (to deliver product) and communicating (to educate), you will be able to leverage your insights into a position of thought leadership.  [Note: "thought leader" is a title that can not be self-annointed, it has to be earned and perceived within your market / community.]

The end result is that you will be consistently aware of unserved market needs that you can serve before your competitors can.  That translates into increased demand for your products, which you can use to grow share, grow profits, etc.

That’s why business agility matters.  It isn’t cost reduction that is important, its top-line growth.

Agile Product Management

Staying tapped into your market like this is a strategic component of agile product management.  While continuously staying tapped-in (or Tuned-In), you regularly interact with your internal stakeholders to make agile development processes work.  Enthiosys has a great introduction to agile product management (free download with registration), if you’re getting frustrated with the many “empty presentations” about agile product management, then definitely check that one out – it has meat.  Relative to their onion, the topics in this article map to the portfolio and product layers.

If you aren’t already convinced of the value of a tight market focus, consider the following diagram, contrasting the impact of agile product management and traditional, waterfall product management.

• The red curve on the top (same as previous slides) represents an agile product manager staying tapped into her market.
• The grey curve in the middle represents a product manager staying tapped into the market, but not as tightly.
• The green curve on the bottom shows what happens when your organization is trapped in a waterfall delivery model.

The green curve, instead of reflecting understanding, reflects the understanding against which the team executes.  When you lock in or “freeze” the requirements, then begin a sequential development process, you stop taking advantage of market insights.  You essentially decouple your implementation team from the market – “we know enough already.”  So even though your product manager may be learning more (the grey curve), you aren’t taking advantage of that knowledge, so it is adds no value.

As an agile product manager, working with an agile implementation team, you will run circles around your waterfall competition.  Not only will you establish thought leadership, but you will be creating a distinctive competence for your company.  Your ability to understand the market, and quickly respond to changes in the market with valuable solutions will differentiate your company.

You also effectively change the dynamics of your sales over time.  Your accelerated insights allow you to release products (or capabilities) earlier.  Here’s a diagram from an earlier article on the ROI of agile that highlights the impact of this:

If we take the most conservative approach to modeling sales with an agile process, the graph would look like the following:

The sales volume curve is shifted to the left without changing its shape. The exact same growth curve applies to the product (this is the conservative assumption). The decline in sales continues to decline. The difference is that our previous time horizon (which remains fixed) now includes additional sales.

Product Life Cycle and the ROI of Agile Development

Agile Without Product Management

There are people who suggest that product management has no place in an agile environment.  Those people believe that markets can’t be understood until they are served.  They will argue that customers “don’t know what they want” based upon the feedback they get from prototypes.  Customers change their minds.  Yup.  But it isn’t that the customers don’t know what they want, nor is it that markets don’t have shared needs/problems/goals/requirements – they do.  Providing solutions to those problems drives next-level thinking, and the identification of new and different problems.

If you don’t have product management, you can only succeed through luck.  Throw stuff against the wall and see what sticks.  If something sticks, you win.  This is better than a waterfall model that ignores the market (for long periods of time where release plans are “set in stone”).  But it falls well short of driving a very efficient, adaptable implementation team directly towards identifiable market needs.

Is your product successful because you were lucky, or because you were methodical and intentional?

Do you want to build a plan where you are dependent on good fortune, or do you want to make your own “luck?” Both approaches work, but only one makes sense as an intention. Slide 3 of your presentation to a venture capitalist should not say “And then we get lucky!”

Product Success Is Not Easy

Saeed Khan wrote a critique recently, in On Product Management, of an article by Phil Meyers on the Tuned In blog. Phil’s article is an analysis of the pending re-organization at Starbucks, and the one quote that Saeed keyed in on was:

At the end of the day, its simple. Create a product or service that your buyers want to buy and the rest takes care of itself.

Saeed’s point is that it is not that easy. A lot of hard work goes into creating a successful product. And Saeed’s right.

Maybe Phil’s point is that the executive should not worry about the details, and trust in his team to do all the hard work. But he doesn’t really come out and say that, so we can’t really back him up on that front. Let’s give him the benefit of the doubt anyway. There’s another point that Phil makes that is potentially disturbing:

Looking at metrics like average same day sales and products per square foot lead you down some strange paths. Schultz even admitted as much in a letter from the board about a year ago in which he worried about the company ‘losing its core’.

Yes, abandoning your goals to pursue your metrics is bad. But don’t abandon your metrics to pursue your goals – unless all you want to do is get lucky.

You might argue that a company like Animoto got lucky when their product spread virally within Facebook and their user base jumped from 25,000 to 700,000 users. But if you listen to the interview that Amber MacArthur did with co-founder Brad Jefferson, you will realize that it was only when they tweaked their product offering – a response to empirical analysis of product adoption metrics – did their success explode. I would argue that they made their luck by being intentional.

Empirical Processes

When you can perfectly model your business analytically, you can measure the inputs and know (with certainty) the outputs. As a college engineering student, I learned this. Real world processes can never be perfectly modeled analytically. As a professional engineer, I learned this too. Real world processes are empirical. The secret to great engineering is to apply analytics to those empirical processes to create disruptive innovation, and combine it with empirical controls that help you statistically predict the likely outputs. The answer is simply that neither analytics nor empiricism alone can help you achieve greatness.

Business processes are also empirical in nature. Even when you can devise an analytical model for the behavior of an isolated system, you have to acknowledge that no real world system is isolated. You have to expect unexpected inputs into the system. As Saeed points out, you have to apply analyses to make smart decisions when developing your process (or business model, or product). And what Phil appears to discount is that you also need to apply empirical measurements to your process (or business or product) to control the expected results.

Creating Successful Products Intentionally

This article proposes that there are two paths to product success. The first path is simple. Cross your fingers, then get lucky. The second path is harder. Be intentional.

1. Identify your market (and therefore your customers and competition).
2. Identify their problems, and select the ones you will solve.
3. Create a product roadmap (aka a “plan”) to solve those problems.
4. Design and implement solutions to the most valuable problems.
5. Get feedback on your solutions.
6. Incorporate your feedback into your plan (step 3) and repeat.
7. Revisit your market (step 1) and the problems you choose to solve (step 2) and repeat.

Note: Step 7 should occasionally replace step 6, so that you stay focused on your market, and not just an out-of-date snapshot of what used to be important to your customers.

1. Identify Your Market

There are a lot of ways to pick a market to focus on. You can chase demographics – there sure will be a lot of retired people in the US thanks to the baby-boomers. You can go with what you know – years of paying attention to an industry presents ample opportunities to understand the market. You can create an entirely new – blue ocean – market by solving problems people don’t even realize they have until you offer a solution. Choosing a market is the subject of many books, not just an item in a list.

Once you choose a market, you need to segment the market into groups of people who share the same problems and who value solutions to those problems similarly. You can apply market research to improve your market segmentation.

2. Select The Problems You Will Solve

You use elicitation skills to identify the problems that your customers face. And when you have to address multiple market segments, you can prioritize the problems across those market segments.

3. Create a Product Roadmap

Once you’ve prioritized the problems you are going to solve, create a product roadmap. Your product roadmap should show the problems you are solving, and the order in which you will solve them. When you define sequencing, you also must define your approach to scheduling product releases.

4. Design and Implement Solutions

There are two keys to successful execution of the plan you built with your product roadmap. First – use rolling-wave planning to define near-term details and long term vagaries. Second – make sure you have continuous integration as a key component to managing quality throughout the process, instead of checking quality at the end (or even worse – ignoring quality).

5. Get Feedback on Your Solutions

This is half of why agile (when done right) works [Can you believe the discussion over the last year on this article is up to 27 comments?!]. Feedback is not just something you get when sharing a prototype with stakeholders. Feedback is also something you must get as part an incremental release methodology.

You can even measure the ROI of your designs, and incorporate feedback at the design level.

6. Incorporate Feedback Into Your Plan

There’s no point in gathering feedback if your organization and your process are organized to resist changes to the plan. Contrary to what the Standish Group’s CHAOS study has always implied [and we've made this implicit mistake too], release schedules are not the primary measure of project success. In a fantastic article at Dr. Dobb’s Journal, Scott Ambler points out that in his survey results, almost two thirds of professionals find that doing the right thing is more important than meeting a project schedule.

Schedule: 61.3 percent of respondents said that it is more important to deliver a system when it is ready to be shipped than to deliver it on time.

Do the right thing. If the right thing involves changing the schedule, that doesn’t make it wrong. What makes this work is the fact that you are getting feedback early and often. It is a risk mitigation strategy, designed to reduce the possibility that you will create an unsuccessful product. It is not a strategy designed to keep your project on schedule no matter how mis-aligned you are to your market.

7. Revisit What You Are Doing And Why

If step 6 is small-scale course correction, this is large-scale course correction. You may discover that solving a big problem for your customers exposes a new “biggest” problem that wasn’t there before. By revisiting step 2, you can choose to tackle or ignore those newly discovered problems.

You may also discover that by leveraging your investments to date, you dramatically improve the ROI of solving problems in another valuable market segment. This is because even if solutions to those problems have the same value, solutions to those problems now have much lower costs (for you). By revisiting step 1, you give yourself the opportunity to best manage your strategy, your resources, and your plans.

Conclusion

Product Success may have an element of luck, but you should never plan to hit the lottery. Be intentional in what you will do, and a good plan, executed well and adapting to change, will get you there.

Learning how to write use cases can be tough, but it is simple compared to the balancing act of determining which use cases to write and how to manage the expectations of all the stakeholders that are involved. It can be a difficult balancing act to prioritize use cases to assure that you meet the goals of the business while satisfying the needs of the users.

Goal Driven Use Cases

In the world of structured requirements, we apply an approach of top-down decomposition to determine what we need to build.

In an article stressing the importance of non-functional requirements, we summarized the relationships from the above diagram as follows:

• Goals are achieved by enabling use cases.
• Goals drive non-functional requirements.
• Use cases are enabled by implementing functional requirements.
• Use cases influence non-functional requirements.
• Non-Functional Requirements define the characteristics of functional requirements.
• Functional requirements drive design decisions
• Design choices are restricted by constraints
• Design choices guide implementation
• Implementation is product

Non-Functional Requirements Equal Rights Amendment

The key element, with respect to this article, is that goals are achieved by enabling use cases. In other words, without the use cases that people* perform, the goals of the business can not be achieved.

[*Sometimes the use case is performed by a system - not all actors are people]

Principal Stakeholders Own Business Goals

We learned a lot from Outside-In Software Development, by Carl Kessler and John Sweitzer. One of the important things we learned is how to understand the goals of principal stakeholders. Principal stakeholders are the “owners” of the ROI that we are attempting to achieve with our product or solution. The business has the goals, but the business is an abstract entity. If the goal of the business is to increase sales by 10%, we can’t call up the business and ask “did we meet your goal?” We have to call up the vice president of sales. The business is organized so that there are people who are responsible for the initiatives and goals of the business – they are responsible for growth, profitability, costs, strategy. The insight we got from Kessler and Sweitzer is an appreciation that this ownership lies with one or more individuals.

When a goal has an owner, we can validate that we have met the goal. We can validate that we have satisfied the owner of the goal. You can’t do that validation with an abstract entity, a “business”, but you can with a person. Certainly, we can do the math ourselves, but if we want someone to pay the bills – and more importantly – invite us back to do more projects, we have to satisfy an individual. And that individual is our principal stakeholder.

Principals Own Use Cases

Our principal stakeholder owns the business goal – and therefore owns the use cases that enable that goal to be achieved. Our stakeholder will validate the completeness of our use cases by answering the question, “If we enable all of these use cases, will we completely meet your goal?”

End Users Own Use Cases

User-centered design is also one of our core principles, and an acknowledged best-practice. For many companies, it is more than mandatory – it is a way of life. The entire software development process is sometimes built around a focus on the end users. This approach puts the user in the center.

You can identify end user goals with the following approach:

1. Identify the actors that interact with the software.
2. Identify the personas that represent those actors – thus avoiding the elastic user syndrome.
3. Develop an understanding of the personal goals of the personas – and by extension, the actors through ethnographic research.

Stakeholder Goals: Principal vs. End User

This creates what appears to be a conflict – do principals own the use cases, or do end users own the use cases?

Mixing Interaction Design With Structured Requirements

One approach to resolving this conflict is to eliminate it by mixing interaction design with structured requirements. In this approach, the interaction design component applies end user ownership as an input to the design, but not the content of the use cases, as the following diagram shows:

In the diagram above, the changes (relative to the Wiegers process) are marked in blue.

The interaction design process starts with two parallel activities. On the left is the identification of corporate goals, which is analogous to the creation of an MRD. An MRD can be a reasonable mechanism for documenting the corporate goals. It provides more detail than “Increase profits” with statements like “Increase profits with improved pricing.” On the right side is the creation of personas to represent the most important users.

Interaction Design and Structured Requirements

This approach side-steps the issue by relegating the end users to “design input” and not allowing them to contribute to prioritization and management. For many applications, this is a great way to approach things. Specifically, it works very well when the user’s goals are tightly coupled or highly aligned with the principal stakeholder’s goals.

When Goals Conflict

User goals are not always aligned with stakeholder goals. In our earlier article on identifying these conflicts, we proposed a simple grid view for identifying these conflicts.

In that article, we looked at situations where the goals of the actor were primarily aligned with those of the principal stakeholder, and therefore the business. We discussed using the grid to inspire solutions that removed the conflicts. Two birds with one stone would be the easiest way to summarize.

What about when goals are implicitly in conflict – or even diametrically opposed? We’ve worked on systems where that is exactly the case. Consider a two-tier sales model, and products / websites designed for the people in the second tier.

Two-Tier Sales Models

Here are some common business models where the users have goals that are directly in opposition to the goals of the principal stakeholders.

• Automotive Manufacturer and Dealer. A website, designed by a car manufacturer, to allow dealers to improve their ability to sell cars. The automotive manufacturer makes money by selling cars to the end customer, and the dealer makes money by causing the sale to happen. The manufacturer is compensated by profits on the sale, whereas the dealer is compensated simply by making the sale. In other words, the dealer is motivated to lower the price because a sale at any price generates the same profit for the dealer. The manufacturer is motivated to raise the price because fewer sales at a higher margin generate the same profit, but higher ROA (return on assets) than more sales at a lower margin. This is an implicit conflict.
• Book Publisher and Book Retailer. A book publisher sets a price for sale to the retailer, and a suggested (list) price to the end customer. The retailer will typically pay a negotiated discount, in the form of a percentage off list, for the book. Any books not sold by the retailer are returned to the publisher or (usually) destroyed. The retailer does not have to pay for books that are not sold. The retailer has limited shelf-space, and makes money based upon a combination of the size of the discount and the number of books that are sold, regardless of publisher. The retailer is therefore incented to increase the discount, and thus increase profits for a particular title – and the publisher is incented to reduce the discount, and thus increase the profits for a particular title. Since the list prices are fixed, the retailer negotiates the discount, and as leverage over the publisher, gets to influence the placement of the book within the store. Each publisher is forced to compete with other publishers. These goals are also in conflict.
• High-Tech Equipment Manufacturer and Value-Added Reseller. A value-added reseller (VAR) provides solutions to end customers. They do this by combining products from manufacturers, and adding value – in the form of planning, custom engineering, installation services, etc. The VAR will negotiate a price with the end customer for a solution. The typical VAR sells products from multiple high-tech companies. The VAR will negotiate with the manufacturers to get the lowest possible prices from the manufacturers. The manufacturer wants the end-customer price (when it includes their equipment) to be as low as possible – to increase the number of sales that are made. The manufacturer gets the same profit (based on the price to the VAR) regardless of the end-customer price. Every dollar that the end-customer saves, the VAR loses in profit. The VAR makes profit on the difference between price-to-the-VAR and the final selling price. The VAR will also want to drive down the price it pays to the manufacturer. Reducing the price-to-the-VAR transfers profits directly from the manufacturer to the VAR. These goals are diametrically opposed.

When goals are in conflict like this, we can’t just relegate the “end user” goals to design inputs – we have to incorporate them into the prioritization process. In these examples, the “end users” are the VAR, the book retailer, and the automotive dealer.

Imagine that the high-tech equipment manufacturer sells networking hardware. The manufacturer has an internal (direct) sales force, as well as the VAR channel (indirect) sales force. The manufacturer identifies tools that help sales reps to close deals. Large purchases are commonly made at as much as 80% off list price. The list prices are irrelevant – they are not even guidelines in many cases. So the manufacturer develops tools to help sales reps know what prices to set for end customers.

Internal sales reps have generally well-aligned goals with principal stakeholders. The sales rep will be compensated based on a combination of revenue (sales) and margin (profits). This compensation model creates implicit alignment between the user goals (get a bonus) and the company goals (make profitable sales).

One approach to helping the sales rep close profitable deals is to share an indication of the profitability of the deal with the rep. The rep can (will!) set a price that is designed to maximize the sales rep’s bonus. If the compensation system is well designed – this is also optimal for the business, and therefore the principal stakeholder. The sales rep will get increased compensation for raising the price (and still closing the deal) by getting a percentage of the increase in margin. The sales rep is motivated to increase the manufacturer’s profits.

The VAR, however, should not know the profitability of a particular transaction. The VAR is motivated to lower the price (the price to the VAR) as much as possible – even at the expense of losing a “profitability bonus.” If the VAR is given a percentage of the increased margin (to the manufacturer) it comes at the cost of losing 100% of the change in price to the VAR. Remember, the VAR makes money on the difference between the price from the manufacturer, and the price to the customer.

Depending on the manufacturer’s business model, this could lower the priority of a margin-inspection capability, or dramatically raise the priority of features that restrict margin-viewing to only internal sales reps.

Also remember that the manufacturer and the VAR also have aligned goals – like closing the deal. And the VAR can close the deal with the manufacturer, or with products from a competitor. So the manufacturer is incented to provide valuable functionality to the VAR. The manufacturer cannot ignore the VAR’s needs, or the VAR will work with a competitor and the manufacturer will lose all the business.

You have to determine the value to the principal stakeholder of the value of the feature to the end user. In other words – even if a feature “hurts” the manufacturer, is the pain “worth it” because of the value of the relationship and expected future business that the company will do with the VAR?

Once the value of all of the functionality is identified, you can prioritize it. An enlightening way to visualize this value is with a value-delivery matrix. Thanks again to Roger Burlton for inspiring our approach to prioritizing based upon a combination of pain and gain.

You want your software to be used, not to sit on the shelf. You can’t achieve the ROI of your software if people don’t use it. And you can’t achieve the ROI of your software by forcing people to use it either.  Some will fail to achieve the benefits, and others will delay using it or refuse to use it entirely.  You have to make them want to use it, and you have to design the software for the users who must use it.  Otherwise, you won’t achieve the ROI.

The ROI of User Adoption

The image above is great – it shows one person choosing to use the software, and one person choosing to do the work by hand instead. It is a classic user adoption problem. The good thing about it is that you can talk in concrete terms about the goals of your software, and how user adoption plays a role.

Let’s start with a definition of return on investment (and read the full article for more details and examples)

ROI is the acronym for return on investment. Another way to think of it is “How much profit will we make if we invest in this project?” Profit is revenue minus costs. Technically, the question should be “How much profit will we make, relative to our investment, if we invest in this project?”

Definition of ROI – Return on Investment

OK, with a definition of ROI, you have to develop a model for return, and one for investment. The model for investment is built by looking at the cost of developing the software. That defines how much it costs. Assume it costs $1,000,000 to develop and deploy the solution.

The model for return is a little more complicated.

For our user adoption example, assume that the software saves the company $1 every time it is used, relative to whatever the company does in the absence of the software. There are several ways to calculate the ROI of design, we’re picking one Assume we have 1000 users, each of whom would use (or not use) the software 10 times per day. That is a potential savings of $10,000 per day. Call it $2,000,000 per year once all the users are using it all the time.

Unfortunately, many people stop there. They see a 6-month payback period, they fund the project, and they stop thinking about it – the money is already in the bank. But you’re not going to do that.

Imagine that this is your project. You write perfect requirements, you have an excellent development team, and the software works perfectly – saving a dollar every time it is used. The problem is, only 100 people are willing to use the software. Now, instead of a 6-month payback period, it takes five years to recover the cost of developing the software.

The False Promise of the Mandate

“Not a problem for us,” you say. “Everyone must use the software. Or they’re fired!”

Many corporate IT departments work this way. The business tells them what they need (“Save us a dollar!”), and the IT guys gather requirements, develop a perfect application and confirm that it saves a dollar every time someone uses it. And then they mandate that everyone use it. Simple. 6-month payback.

There are a couple problems with this approach.

1. Not everyone uses it right away, even if everyone is required to use it.
2. Not everyone uses it effectively, even when required to use it.

The explanation is a little less obvious, but the brutal reality of it is just as true.

Delayed User Adoption

You issue a mandate – all 1000 users will use the software! The users are conveniently spread out into 10 business units, each with 100 users. One of those business units will go first. And your software is a disaster (we’ll get to #2 in a minute). The plan was to pilot the software with the first 100 users, then roll it out a month later to the other 900. No problem – 7 month payback, and less risk to the business.

The problem is that the people running the business units – and thereby managing the users – are smart too. They see the train wreck that Tom’s department became, and they refuse to use the software. They have their own financial targets, and they don’t want to jeopardize them. These business unit directors, if they lack the power, will escalate, until their demands are heard.

What’s the first place in the hierarchy where there’s a common manager? Is it the CEO? Maybe you’re lucky, and it is only a couple VP’s arguing about it. What conclusion do you think they will reach when presented with the following arguments:

• If we use the software, we will miss our numbers – it slows people down. And that will cost you $X.
• But we want them to use the software – they have to. If they were smarter, it would save money

Even if the argument isn’t accurate – who will win? Best case, your roll-out is delayed. Worst case, it is killed before it ever gets a chance to work.

Ineffective User Adoption

It turns out, after watching the pilot group, that expert users can in fact save a dollar every time. Competent users break even – there’s no savings for them. And beginners were better off with the old solution. It was less efficient, but the new way is so hard that they can’t do it well.

You’ve mandated that people use the software. Since they didn’t otherwise want to use it, you had to spend a bunch of energy (and money, in opportunity cost) to make it happen. The rollout was delayed, and at the end of the day, you break even. Most users have no savings, and the few that do are offset by those that caused an increase in cost.

Mandating user adoption creates a false sense of confidence, and does not assure ROI – it only assures adoption. We’ll say that again, to let it sink in.

Mandating user adoption does not assure ROI – it only assures adoption.

So what can you do?

Measuring User Adoption

As product managers who care about user adoption, our first thought is “measure it!” The biggest challenge is in measuring the right thing.

We can’t just measure how many clicks, because while that might correlate with ease of use – it does not necessarily cause increased user adoption.

To stay aligned with the ROI model, we have to measure, directly, that user adoption is meeting the forecast used to create the ROI model. The only measurement that is assured to be an accurate reflection of user adoption is measurement of user adoption directly.

Prevent Mistakes, Then Monitor Them

If we measure poor user adoption after it has proven to be inadequate, that doesn’t help very much. It is generally accepted that a good design leads to higher user adoption rates. In the case of the “false mandate,” good design yields faster roll-outs and allows more users to be more effective with the software.

There are also other reasons to invest in good design.

The right solution is to invest in good design – targeting competent users (not novice users) to achieve the desired ROI. A good design is easier to learn too. You can track the rate of improvement among users.

Here’s a chart from an article that goes into learning curves in more depth:

[larger image]

The graph shows that for a weekly frequency, after 16 weeks, the task time has reduced from 300 seconds to 100 seconds. With a daily frequency, the task time is even lower – about 60 seconds. This graph shows nothing other than converting the academic learning curve graph into one that incorporates calendar time and frequency of occurance.

Note the odd, vertical drops in task time during week 1. This is just a manifestation of looking at a weekly time scale. For a task that happens once per hour, the user will rack up 40 repetitions during the first week. From a decision-making standpoint, you don’t have time to react to that rapid rate of learning, so it is ok that it is collapsed on this graph.

Software Usability and Learning Curves

Thanks, now go hire some designers and achieve your ROI!

I am a big fan of boxes and arrows, but this time, Jeffrey Davidson found a great article by Dan Willis before I did, and told me about it. Thanks Jeffrey! The article is about how to deal with the what and how of requirements and design – and it provides some really sage advice. But what got my attention was the lack of prioritization of requirements in his example.

One Man’s Trash

Dan’s article is about the difference between what and how. He provides good advice on changing the conversation to be what-centric, not how-centric. Dan is writing from the perspective of an information architect, and he defines what and how as follows:

As I started to work in jobs where some people communicated with one another using requirements, it seemed reasonable to keep separating “the what” from “the how.” These two definitions are the result:

• Business requirements: What the project, system, or solution needs to accomplish.
• Functional requirements: From a high level, how the project, system, or solution needs to accomplish what it needs to accomplish.

Are Useful Requirements Just A Fairy Tale? (and why an IA should care)

Although we used slightly different language, we wrote an article about how different people in the software development life cycle think about different artifacts created in the process. The following diagram sums it up:

Requirements Documents – One Man’s Trash…

Dan describes the product management view of the world in his article. It is a good article, and you should read it – his pragmatic solutions to common frustrations are good ones. They center around turning the conversation (with product managers) into one of What do you need? instead of How do you think I should build something?.

Prioritization Hell

As good as his article is about the frustration of mis-matched roles, here’s the paragraph that really got my attention:

First, you get a group of stakeholders in a room so they can “blue-sky brainstorm” to figure out how they want to improve their product. The group comes up with a list that has everything from “Make the little blue buttons bigger” to “Bring world peace.” The list goes to an engineering team who organizes the list into three categories: Things we can do now, things we can do by the end of the year, and things that will take a long time. They email the categorized list to a project manager who changes the categories to Phase 1, Phase 2, and Phase 3 and sets up a project schedule for the first 10 things in the Phase 1 list.

Dan Willis

Aargh!

This is why prioritization matters.

You should always prioritize the capabilities that yield the highest returns on your investments. Those returns may be based upon a measurement of risk, a strategy for gaining market share, or straightforward cost reductions. You can also refine this approach by applying the principles of Kano analysis to maintain a user-focus. You may need to balance the goals of users with those of stakeholders. But definitely don’t let someone prioritize features based on how long they take to implement.

Dan, we feel your pain!

We all know the story about the emperor’s new clothes. I’ve been thinking about prioritization and scheduling, and as far as I know, no one is promoting that we maximize value – they (and we) have been promoting that we do the most valuable stuff first. Doing the most valuable things first does not result in getting value the fastest. In this article, we show why not.

Genesis

About a month ago, I read an article by Kelly Waters on how to prioritize intuitively. He presents a magic square diagram, showing both the “how valuable is it” and the “how hard is it to do” axes. I’m oversimplifying, read his article for more details – he incorporates elements of risk, complexity, etc. I really liked that he was addressing the “missing element” of how much work is involved. However, his diagrammatic approach, while presenting this information very well, does not really yield insights into what to do first. Kelly and I had a great discussion over the next couple weeks, exploring the interplay of work and value in prioritization, trying to find a way to encourage value-maximizing decisions.

Prioritize By Value

We have talked about prioritizing the most valuable requirements first repeatedly. And in the last of those links, we accidentally hinted at, but didn’t grasp the real goal:

We will only consider those steps where the profitability of change exceeds our hurdle rate for investment.

We’ve also talked in the past about using use cases as the basis for scheduling, as each use case represents realizable value. For the rest of this article, we’ll talk in the context of scheduling use cases across releases.

Consider a very simple example – you have five use cases, with values of 10, 9, 9, 8, and 7 respectively (the units don’t matter). If you sort those use cases in order by value, from left to right, they would look like the following:

The size of each box shows the relative value of having the use case implemented.

Based on our previous guidance (and everyone else’s), you would implement them in order, from left to right. Makes sense. Do the most valuable thing first. Do the next most valuable thing next. Repeat until the value is not high enough to continue.

What About The Amount of Work?

OK, the amount of work required should play a role too. In our time-boxing article, we describe each release as having a given capacity, which you can visualize in terms of cost (resource) and time (duration of applying resources):

We fill up that capacity with use cases, based upon how much work is involved.

We can estimate the work involved with each use case by any of a number of methods – but the earliest estimates can be developed using use case points.

Consider the following “work” measurements, identified for each of our use cases from above:

We have the same sequence of use case implementation (based on value), but now we can visually see that there are different amounts of work associated with each use case. The area of each box represents the relative level of effort required to implement the use case.

Prioritize By Value Results

The best way to explain the flaw with the classical “prioritize by value” approach is to show what happens after the first release. Consider that you can accomplish 30 units of work in the first release.

We can schedule the first two use cases for this release. The size of the time-box above represents the amount of work that can be accomplished. With the first two use cases scheduled, the time-box will look like the following:

We have completely used up the available capacity of the team (Work = W = 30 = 10 + 20) by delivering the two most valuable use cases. We have delivered 19 units of value (Value = V = 10 + 9 = 19).

Value Maximization

When we consider both the value (V) and the cost (in terms of work, W) of each use case, we see that some use cases generate more value per unit of work than other use cases. If we consider the ratios of value to work (V/W), and sort the use cases based on this approach, we would see the following:

And with the previous specific value and work values:

If we were to organize our delivery of use cases based on this ratio, we would be saying “prioritize the most effective use cases in terms of value per unit of work.” This may seem counter intuitive, but it makes sense – get the most bang for the buck earlier, and you will get more value faster.

Consider what our first release would look like:

We would complete three use cases (using 25 units of available work), and we would deliver 25 units of value. We would also be able to start working on one of the use cases that would be delivered in the next release.

True Maximization

To find the mathematically proven maximal value, we have to do a bunch more work. This prioritization exercise is actually an example of the bin-packing problem, an np-complete computer science puzzle. To make a long story short, we can’t use a simple heuristic and guarantee that in all cases it will be optimal. But we can do better than “most valuable first.”

If we use the scheduling rule as follows:

Schedule the use cases in order based on the highest value-to-work ratio, skipping use cases that are “too big” for the current release.

Then we will get value out of the system as fast as possible. There are a couple problems with this approach:

1. It does not take into account that you can apply work from one release to a use case that is scheduled in a future release. Intuitively, any “remaining time” after scheduling complete use cases should be spent on the next highest-ratio use case. I haven’t proven that mathematically, but it makes sense.
2. Use cases, and their underlying requirements, and the implementation tasks to support those requirements are not actually independent. You may need to introduce one use case before another – the second use case may not be possible without the first one. Implementing a requirement that is shared across use cases will reduce the “remaining work” for those other use cases – causing a need to recalculate ratios. Implementation tasks are often dependent upon one another. The discrete tasks to support a valuable use case may require implementation work that is also leveraged in lower value use cases. Further, some implementation tasks must be performed sequentially. You can’t optimize a query before defining a database schema, for example.

The second problem actually applies to any prioritization approach that incorporates value. The nature of software development introduces constraints (X must be completed before Y can be started). Those constraints narrow the possible scheduling choices. And they make it impractical to determine the “optimal” solution. At least with commercially available tools, excluding expert systems, which can be used to solve this type of problem.

Conclusion

• Sequencing use cases based solely on value does not maximize the delivery of value over time.
• Sequencing those use cases based upon the ratio of value to effort will increase the rate at which value is delivered to customers.
• It is impractical (and possibly marginally valuable) to determine the optimal sequence for scheduling use cases to maximize value.

You should use the “highest ratio first” approach, and when a use case can’t be delivered yet because of interdependencies, skip it. Also – apply judgement to sanity check if you are doing something that seems odd – like delaying a high ratio, high value use case. Explore with the development team if there are ways to adjust their dependencies to allow for a “more valuable” delivery sequence

Measuring the return on investments in design may be the hardest ROI calculation you can do. It certainly is one of the rarest. To measure ROI, you have to be able to determine what would happen without the investment, and what happens with the investment. The difference between them is what happened because of the investment.

Fast Company Interview

Bill Breen posted an interview with Rob Wallace on the Fast Company blog about proving the value of design. The start of Bill’s article lets us know that measuring the ROI of design (ROD) is so rare that it may be impossible or impractical to do. He points to a Whirlpool survey of 15 design-focused companies – most of whom were “clueless” about the return on their design investments.

Rob explains that the reason for measuring the ROI of design investments is that ROI is the language of executives. “If you can’t measure it you can’t manage it. Businesspeople operate in a world of numbers.” Simply put, if you want to make someone invest in design, you have to show them why they should invest in design. One way to get a company to invest in design is to convince the benevolent dictator in charge of the product (or company) of the subjective merits of having good design. Apple is a great example of a company with a leader who is passionate about design. And you can argue that the Zune (from Microsoft) has had meager success against Apple’s iPod due to shortcomings in design. But can you objectively argue the point?

Argument by Extension

After Rob mentions that there is a dearth of measurement of design ROI, Bill challenges Rob’s premise – why should we expect that there is an ROI for design? Rob cites a series of studies done on Fortune 500 companies:

On average, based on two dozen case studies with Fortune 500 companies, for every dollar invested in advertising, packaging and promotion, and visual communication at the point of sale, companies realized a $7.21 ROI. But when the advertising didn’t change (or there was no advertising)—and packaging design was the only thing that did change—there was a $15.17 average ROI on every dollar invested.

Based on this data, Rob argues, it is easy to extend that if packaging (design) changes can yield ROI, then so to will product design changes.

Insights from Adaptive Path

Peter Merholz, President and founding partner of Adaptive Path wrote on communicating the value of design in Aug. 2002. In his article, based on a breakout session, the AIGA’s 5th Advance for Design Summit, he shares a series of internal benefits that they found to be more significant (or at least more measurable) than external benefits.

As we continued listing the value of our user experience design work, a more intriguing realization emerged — the bulk of our value comes from the efficiency that we can create in a company’s operations.

Some of the areas they identified are:

• Smarter Product Development Processes
• Lowered Maintenance Costs
• Less Internal Documentation
• Maximized IT Investments
• Scalability

Our Concerns

One challenge for each of these sources of ROI is that you can’t truly isolate the benefits – you would never have two teams design products for the same market, with differing levels of design investment, with a resultant analysis of project costs. Even if you did, you wouldn’t be able to isolate the impact that the team members had. You have to look at historical data, which introduces enough variables to question the validity of any quantitative conclusion.

Further, we will show that each of these is either a weak proposition, or potentially very false one.

Smarter Product Development Processes. No one has concretely identified a design-centric, purely agile, or structured requirements approach as being the best one. At least where best is defined as most profitable. Even the debate between Alan Cooper and Kent Beck highlighted more “stylistic differences” than quantified differences. There are elements of genius in each approach. Incorporating the needs of users is key to an effective design. Incremental delivery and incorporation of feedback is incredibly efficient. Tracing decisions, designs and requirements to the ultimate goals for the software is key to building the right software. If there is a “perfect process”, it is one that incorporates elements from all three disciplines. And as such – there is value to incorporating design into the process. We can’t quantify it, but we are convinced that it is there. And based on the economics of when software problems are fixed, we believe that investments in design early in the cycle will yield positive returns.

Lowered Maintenance Costs. There are a couple ways this benefit might be inferred. First, better design yields easier to maintain code. This is primarily a premise of agile development (where the design decisions are what Cooper calls program design), but perhaps the benefits also apply to interface design. Often, good interfaces simplify the complexity of, or at least obscure the means of implementation. Those interfaces are not necessarily easier to maintain. One reason complexity leaks through to users in a lot of software is that it is easier to code it that way. This value-prop probably doesn’t hold water.

Less Internal Documentation. The better the design of the interface, the less you need to teach someone how to use it. There is definitely potential for reduced external documentation, or producing the same amount of documentation may cost less, as it is easier to understand the product. It isn’t clear how internal documentation needs can be reduced by good design. Except perhaps that a picture is worth a thousand words. But a great picture may take as long to draw as it does to write a thousand words.

Maximized IT Investments. This is simply self-referential. Maximizing investments yields higher ROI. But in what way?

Scalability. Better design, from a user perspective does not imply that software will be more scalable. And often in an implementation, when user-tasks are simplified (by automating steps in the process), those decisions don’t disappear. By removing the burden from the users, you often add it to the software. However, Peter may have been talking about scalability of the team – the ability to leverage design investments across products (sharing branding, metaphors and workflow paradigms). In other words, design re-use. In that case, there is definitely a benefit, but re-usable design, while cheaper than green field design, is more expensive than no design.

Support Costs. One external benefit that Peter does identify is the reduced burden for training and support. Many software companies actually have profitable businesses built on support and training. Unfortunately, reducing the demand for those would reduce the profitability along that dimension.

Where We See Value

Fundamentally, we don’t perceive design investments as being significant cost reductions – we see them as significant revenue enhancers. Better design yields increased usage, more efficient usage, increased satisfaction, and increased word of mouth marketing.

Increased Usage. ROI forecasts are often built upon estimations (or suppositions) of levels of user adoption. As an example, consider the “Buy it now” button on eBay auctions. When people click on the “Buy it now” button, the auction is closed, and the transaction occurs at a fixed price. People are more likely to sell an auctioned item (generating a commission) by encouraging an impulse shopper. The seller risks getting less than they otherwise would, in exchange for the buyer knowing that they can get the item they want. Fewer auctions will expire unexecuted. More commissions are generated for eBay. Comparisons of final selling prices (and commissions to eBay) of equivalent items with and without the button can be done to measure the return.

More Efficient Usage. Better designs make it easier for users to do what they want to do. Easier often equates to faster. A call-center application that simplifies data access for the operator will allow them to process calls more quickly. The employer ends up with higher throughput from the call center. This throughput can be used to reduce costs, or increase customer satisfaction (with reduced “time per call” stats, or better experiences for the callers), or some of each. Comparing the results of operators using the new system versus the old system will yield measurable data.

Peter mentioned reduced training costs – and we see that as an external benefit that comes from accelerating a user’s growth from new user to proficient or even expert user. Bridging the canyon of pain to get to higher usage rates and more efficient usage faster. This also leads to increased satisfaction by creating more satisfied users more quickly.

Increased Satisfaction. Customer or user satisfaction surveys may be the only way to approximate (not really measuring) this benefit. And it isn’t clear that X% increase in satisfaction leads to Y% increase in sales. But it is likely to lead to increased sales, and may also lead to increases in sales of other products by the same company. Apple leverages this affect across its family of synergistic products. Perhaps a Bayesian analysis of survey results and sales stats could identify the strength of a correlation between satisfaction and sales. That would be expensive to do, but would lead to quantified estimates of the return.

Ultimately, you raise the utility of the software for the user. By increasing the value to the user, you increase their satisfaction, and thereby increase the likelihood that they will encourage other people to buy and use your products.

Increased Word of Mouth Marketing. When your customers become your sales people, you get more sales than you would have without those recommendations. Perhaps comparisons of growth curves for products with varying degrees of customer satisfaction (or varying degrees of “free publicity” – like reviews, blog articles, etc) would yield some predictive insight. Web 2.0 successes have been predominantly built on word-of-mouth. And their growth curves are exponential (as would be the propogation of good words from satisfied mouths). Does anyone know of analyses that validate this?

Conclusion

Design investments yield benefits. The way to measure those benefits is still up for debate. Some folks believe the benefits are primarily internal – we believe that the benefits are external

Learning curves have been studied for decades when evaluating manufacturing systems and proposing cost reductions. The Boston Consulting Group did an oft-cited analysis in the 1960′s that describes how people get faster at tasks through repetition. Peter Abilla looked at the application of learning curves to writing software. We look at how they apply to using software.

Learning Curves

Wikipedia tells us that the learning curve phenomenon was discovered in the 1800′s by a psychologist, but industrial applications of the concept reference the Boston Consulting Group study on experience curves.

Learning curves describe how people become more efficient through repetition of a task. The experience curve concept extends this idea to organizations, suggesting that overall costs for an operation will drop over time in the same way. People argue about the validity of this – suggesting that economies of scale take over with high volume activities. Regardless, the element of learning still applies.

Learning Curve Explanation

The easiest way to explain the concept is with an example. Consider a task that takes five minutes (300 seconds). Over time, a person will get faster at performing that task. The graph shows the amount of time required to complete the task decreasing with repetition of the task.

To make it more concrete, consider the task to be “Enter an issue into the bug-tracking system.”

[larger image]

There are three curves in the graph above, labeled 90%, 85%, and 80%. Each curve shows a progressively faster rate of learning. Note that the scale of repetitions is logarithmic. The first time the user enters an issue in the bug tracking system, it takes five minutes. By the thousandth time, the user will spend under two minutes on the “90% curve.”

The percentage-labels of the curve represent the quickness of learning. Each time the number of repetitions doubles, the amount of time a person spends on the task is reduced by the listed percentage.

On the 80% curve, the user will spend 300 seconds on the first issue, 240 seconds on the second issue, 192 seconds on the fourth issue, etc.

Usability Is Important

Usability is an important factor in software development. It may be a challenge to define the usability requirements, but usability affects user satisfaction, and usability sells software.

When investing in usability, you have to take into account how valuable each investment is. Any product will have a number of business use cases, and each use case will have a frequency with which it happens. Usability investments affect two factors of that use case:

• The initial amount of time it takes.
• The speed with which someone improves at the task.

ROI of Usability

Once your company reaches stage 5 in corporate usability awareness, you will be prioritizing usability initiatives, and prioritization should be driven by return on investment. That analysis is typically done based on the initial amount of time that a task takes.

Incorporating the element of learning into that analysis will yield a better (and higher) indication of the value of usability investments.

If you’re math-oriented, you’ll notice that the learning curve never really ends (theoretically). You have to “stop” the learning at some point when performing a financial analysis, based on the net present value of future savings. Three years is a realistic time horizon for software (and might even be a little long).

However, you won’t continue to see benefits for three years.

Competent Users

There’s a phenomenon with software – people stop learning when they become competent. Most people get “good enough” to satisfy their own needs, and don’t move on. They don’t become experts, so don’t design for experts, and don’t propose an ROI that assumes that everyone will be an expert.

Grasping Time Horizons

The theoretical learning curve is handy, but you have to do a little more work to make it useful. Consider first the frequency of repetition of the task. Is it a weekly, daily, or more frequent activity? This frequency will determine how quickly (on the calendar) someone will achieve the high levels of repetition that result in improved efficiency.

The following graph shows how an 80% learning curve overlays a calendar for tasks that happen daily, weekly, and hourly.

[larger image]

The graph shows that for a weekly frequency, after 16 weeks, the task time has reduced from 300 seconds to 100 seconds. With a daily frequency, the task time is even lower – about 60 seconds. This graph shows nothing other than converting the academic learning curve graph into one that incorporates calendar time and frequency of occurance.

Note the odd, vertical drops in task time during week 1. This is just a manifestation of looking at a weekly time scale. For a task that happens once per hour, the user will rack up 40 repetitions during the first week. From a decision-making standpoint, you don’t have time to react to that rapid rate of learning, so it is ok that it is collapsed on this graph.

Further Limiting The Time Horizon For ROI

There are three vertical, dashed red lines on the graph. The original learning curve theory stays pretty, well, theoretical – using log graphs for a time horizon. This is fine for academics that want to understand the underlying trends and drivers. An ROI analysis brings in a set of relevance constraints driven primarily by the time-value of money.

However, software’s time cycle should limit the time horizon even further in order to be practical.

• First Impression. Software is often evaluated for usability or suitability based on first impressions. The reduced barrier to entry for many software products makes it extremely easy for a user to say “I’ll download something else – this isn’t good enough” after a couple minutes with the software.
• End of Training. Other software sales cycles involve training as part of the evaluation period. This is more common with enterprise software, and may be the way your target users (or decision makers) approach the evaluation. Everyone is different. When this is the case, the user’s ability to be effective with the software will be evaluated after some nominal period – perhaps two weeks.
• End of Attention Span. The competent user phenomenon limits the attention span of most users – they simply stop learning. A realistic way to represent this may be to draw an (admittedly arbitrary) line in the sand at the three-month mark. At this point, users either like the software or they don’t. They’re as good as they’re going to get.

Conclusion

You will have to determine which time horizon is most relevant for your product, customer, and users. That decision will determine the relevance and impact that learning should play in prioritizing usability investments for your product. If your product will thrive or die on first impressions, then the learning curve should be all-but-irrelevant to your decisions. Only the word-of-mouth-marketing factor will come into play. You need to stress investments that lower the initial task times over those that accelerate learning.

Over 90% of the cost of software development is software maintenance (cite). This alarming trend was predicted as early as 1972. McKinsey suggests that CIOs should spend no more than 40-60% on maintenance. Gartner’s IT Spending and Demand Survey (2002) reports that CIOs are spending 80% of their budgets on maintenance (p12 of presentation). Agile development can help reverse this trend.

The Cost Trends of Software Maintenance

Jacoozi published an analysis of the impact of continuous refactoring on software maintenance costs. Continuous refactoring is an element of agile software development, where the developers continuously make minor improvements to the architecture and design as they maintain the code.

(larger image)

Thanks to Levent Gurses for providing this modified version of the chart from his article, Continuous Refactoring and ROI. In his article, he discusses both recurring “big bang” refactoring (the pink curve) and continuous refactoring (the green curve).

What Levent’s chart shows with the black line is that the cost of maintenance grows at a significant rate over time when you don’t refactor the code.

We can re-use the diagram from our earlier analysis of agile development and ROI, and overlay this cost structure. The green curve represents sales volume, contrasted with the shaded curve representing development costs.

In the product lifecycle diagram above, there is an initial “hump” of development cost. Note that when you are using incremental development, the hump extends past the end of the development stage of the product life cycle.

The largest part of the shaded area represents the ongoing costs – 90% of which are maintenance costs. Note – we are assuming that companies use a rational investment strategy – they continue to maintain the software until the costs equal or exceed the revenue. The investment should stop when the opportunity cost of continued investment exceeds the benefits of continued investment.

Broken Windows

Continuous refactoring is making small investments in improving the code over time. The absence of those investments allows the code to grow more expensive to maintain over time. Gartner estimated that 50% of the cost of ongoing maintenance labor is spent trying to understand the existing code base. This is very inefficient.
In The Tipping Point, Malcolm Gladwell described this phenomenon by analogy to the broken windows in East New York City. This is just one gripping example from his book of the same title.

As the code gets more convoluted over time, these two factors serve to increase costs dramatically – the increased difficulty of doing the job, combined with increased apathy about doing it right. Costs would go up simply because the code gets larger (as the software is expected to do more and more). These factors accelerate the rate at which the phenomenon occurs – consistent with Levent’s faster than linear cost growth function.

Fixing The Windows

When faced with the challenge of reducing the ongoing maintenance costs, you have a few choices:

• Eliminate Ongoing Maintenance and Development. This was the Autodesk strategy (fire the engineers, milk the product for revenue). It worked great for a very short time. Profit growth was tremendous. This of course accelerated the decline in sales, eliminating revenue (and therefore profit).
• Reduce Spending On Maintenance and Development. Simply reducing the budget has all of the negative impact of cutting the budget, but with fewer gains. A reduced budget (with no other changes) increases the frustration both of unstatisfied customers and of overwhelmed developers. This is a bad idea.
• Refactor The Code To Improve Efficiency. You can make ongoing maintenance more efficient by making the code easier to understand and modify. This generates cost savings – the “big money” in ROI calculations.

Improving efficiency reduces the costs of ongoing development, as the following diagram shows:

Other Benefits

By reducing the cost of ongoing maintenance, you can improve the profitability of the product. You also free up resources for investment in new product development. This helps move your organization to McKinsey’s recommended 40% to 60% maintenance budget.

You will also get intangible benefits and improved efficiency by reducing the number of broken windows in the product’s code base. This results in increased motivation of the team members and greater job satisfaction. The increase in motivation will decrease the cost of development of other functionality.

You may also extend the useful life of the product – by extending the amount of time when ongoing maintenance is still profitable. This additional development work could result in increased sales – extending the product life cycle.

The product life cycle is a description of the presence or behavior of a product in the marketplace over time. The framework for description is a function of the sales volume of the product versus time. Over time, products are created and introduced, and sales grow, peak and decline. The product life cycle uses phases to describe these different periods in the life of a product. Understanding the product life cycle is also key to calculating the ROI of agile development.

The Five Stages of A Product Life Cycle

There are five product life cycle stages. The stages are defined as having different sales-growth characteristics

1. Development Stage. In the development stage, there are no sales – the product is being created and is not available.
2. Introduction Stage. The introduction stage starts with the first sale and then sales begin to grow. Sales growth is slow at first but accelerating.
3. Growth Stage. This is the period of rapid growth in sales for the product.
4. Maturity Stage. This phase is characterized by growth as well, but the sales growth is decelerating. Sales volume reaches its maximum at the end of the maturity stage.
5. Decline Stage. This phase represents the period of declining sales – they may still be very high, but they are declining.

Here is the product life cycle diagram [credit to wikipedia and NetMBA for both using this approach to drawing the graph]:

The area under the curve is total sales. This model was developed with traditional product development in mind.

What About Agile Development?

Agile development uses an approach of incremental delivery to deliver a partially-completed product to market earlier. As long as the product is “complete enough”, it will shift the curve to the left. By shortening the period of time with no product revenue (aka Development), we spend more of our time in the last four stages of the product life cycle.

Think of beta releases of software. GMail, for example, was released as a beta a few years ago. The product was just finally released to the general public in the last month. Think of all the advertising revenue that Google earned during the beta period – while GMail was being completed. If they had waited until the product (and infrastructure) were complete, and not released GMail until last month, Google would have lost all that revenue.

If we take the most conservative approach to modeling sales with an agile process, the graph would look like the following:

The sales volume curve is shifted to the left without changing its shape. The exact same growth curve applies to the product (this is the conservative assumption). The decline in sales continues to decline. The difference is that our previous time horizon (which remains fixed) now includes additional sales.

The time horizon stays fixed because releasing the software earlier neither reduces or increases our ability to predict the future. Our net present value calculations will be based on time, not “time with a product in the market.” In a concrete example: Assume we are willing to base our financial decisions on a five year forecast. Having an incremental release six months before the product is complete does not have any effect on our decision to look out five years.

It may be that sales are increased by getting to market earlier, but that is a less-conservative assumption. By getting to market earlier, when we are in a race to capture market share, we may ultimately capture more of the market. An example might be Intel’s recent multi-core processor launch. By getting to market faster than AMD (with a comparable product), Intel will probably sell more CPUs than they would had their launch been delayed until AMD was ready with a competitive product.

It may also be reasonable to assume that sales will peak at the same (fixed) point in time in the future. This would further increase the benefits of incremental delivery. One way this might make sense is if our product is filling a temporary hole in the market, and we know that the hole is going to be filled on a given date. An example might be an emulator that allows people to run Windows Vista programs on Windows XP. The time-table for transitioning to Vista is unchanged by our release schedule, and is the dominant factor in our sales forecast. Therefore our forecasted peak would be unchanged.

Conclusion

This is a handy visualization for the conservative calculation of ROI of agile development. Note that this only shows sales, it does not show costs. The costs of agile (versus non-agile) development will be the topic of another article. This is only half of the picture.