At this point in the product comparison series, you know who your customers are, which problems are important to them, and which products compete to solve those problems.  It’s time to score the competing products and see how the solutions your product provides (or will provide) will stack up.  This is the latest in a series on comparing products, jump back to the start of the series if you came here first, but hurry up :).

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem. (This article)
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Summarizing Effectiveness

Earlier in the series, we identified (and refined) the list of important, relevant problems that our target customers have.

[larger image]

This is the “ruler” by which each competitive product is going to be measured.  We also identified several competitors.

[larger image]

The next step is to assess how effectively each competitive product (including your own) solves each important problem.  Then you need to assign a “score” for how effectively each product solves each problem.  To do that, for each problem you have to articulate an opinion about what it means to solve the problem poorly or completely, or anywhere in-between.

Read Anywhere – Previously clarified as “Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.”

Environments can be indoors/outdoors, extremely cold to moderate to extremely hot temperatures, with variable lighting from a dark room to direct sunlight.  It might also capture environmental context – sitting, walking, riding on a bus, driving, etc.; quiet to noisy; physically serene, or getting bumped a lot (like in a crowded coffee shop).

Start by defining the endpoints.  I’ve been using a 9-point scale in this type of analysis, to provide enough granularity to make relative comparisons.  For read anywhere, a score of 1 would mean “can be used to read in a single, idealized environment / location.”  A score of 9 would mean “can be used to read in any realistic situation.”  Mapping out the scores in-between 1 and 9 requires you to think about the nature of the problem being solved – and here’s where Kano analysis is useful (again).

This capability is a good example of an extreme more-is-better capability.  Increasing the range of environments where the product can be used (to read) provides a perceivable benefit to customers, but with diminishing returns.  Also, there is some minimum bar, or table stakes, of environments where the user needs to be able to read, or the product is not considered a viable solution.  On the high end, being able to read literally anywhere, would truly distinguish one product – making that capability a delighter and a strong differentiator.

[larger image]

How do you decide “What is a 3 score?” You inform these relative scores based on user research (ideally), and your subjective opinion (when you don’t have research).  For folks who haven’t been reading Tyner Blain articles for the last few years – a product manager is market driven, which means you need to use market data to do this as a product manager; however, using your own opinion as a product designer is better than having no data at all.

For this example, my [manufactured, invented, made up for this series of articles,] data defines scores for the Read Anywhere capability as follows:

1. Below The Bar - User must be seated, and have power and internet connectivity (at the time of reading) in order to use the device.
2. na
3. Usable but Very Annoying – User can read while standing without being connected to power or the internet.
4. Not Quite Happy, but Whatever - User must be indoors, with reasonable lighting and temperature.
5. Meh - User can read while riding on the bus or in a car.
6. OK, but Nothing Special- User can read in outdoor lighting.
7. Good - User can read pretty much anywhere except really noisy, jarring, and / or low-light environments.
8. na
9. Wow! - User can read anywhere that the user would want to read.

Note that there are no entries for 2 or 8, to reflect that there’s a step-function decrease (or increase) in perceived value at this point in the curve.  Plotted on the Kano analysis extreme more is better curve, this rating scale looks like the following:

[larger image]

One of the things you can see when looking at the scoring system visually is that you may have to make significant investments in order to make incremental improvements, depending on where you are on the curve.  For example, moving from (4) to (6) looks hard – you are objectively increasing the capability measurably – and increasing the subjectively perceived value by a comparable amount.

A small shift in scoring – for example, from (7) to (9) – can have a dramatic impact on perceived value (specifically, satisfaction received from improving the capability).  Conversely, improving from (6) to (7) may be really hard (expensive) to do, but realistically only improves the way your market perceives your product by a marginal amount.

This view can help you answer questions like

• Why does the iPod shuffle outperform the Sansa Clip so dramatically? Because the iTunes-centric ecosystem turns the dial from (7) to (9).
• Why doesn’t our detergent outsell theirs, when ours gets clothes cleaner in soft water?  Because moving from (6) to (7) doesn’t make much of a difference.

Zooming in on the low-end of the curve:

[larger image]

And at the high end of this capability, we see:

[larger image]

Applying this rating scale to the competitive products [more made-up data here] we see

[larger image]

It may be that different personas would use markedly different criteria for scoring relative capability.  So far, when I’ve done this, I’ve found that different persona care different amounts about the scores for a particular capability, but generally use the same approach to scoring.  When I do come across different personas that would use markedly different scoring criteria for the same capability, I will create use the appropriate scoring criteria for each persona, and add more complexity to this process.  To date, I haven’t had to do that.

Scoring All of the Capabilities for All of the Products

Applying the same process (determine the nature of each capability, determine the criteria for each capability, assign a score to each product for each capability) will result in something that looks like the following [manufactured to illustrate the concepts] data:

[larger image]

Alternate Scoring Method

You could also approach determining the score for a single capability like this by giving points for each characteristic, versus trying to define a continuum like the example above.  For example, you could give

• 2 points for being able to use the device without power.
• 1 point for being able to read when you are not connected to the internet.
• 1 point for being able to read in bright sunlight
• 1 point for being able to read in a dark room
• etcetera

Then tally up the score for each product, to describe how well they meet the need that users have to read anywhere.

Interpreting the Comparison

If you were to stop here, you would just conclude that the iPad2 is the best, and that the two Kindle products are “close,” and that the Nook and using a PC are in last place by a wide margin.  However, you aren’t going to stop here.

Stopping here would ignore completely that different customers care different amounts about solving different problems.

In the next article in this series, we’ll see how to incorporate that info, and answer the two questions you need to be able to answer:

1. Which product is best for a specific persona?
2. Which product is best overall.

Summary

To create a competitive product, you need to know how your product stacks up against the competition – and that means you need to know how effective your solutions are (or are perceived to be) at solving the problems your customers will pay to solve.  You can compare today’s product to assess your current position, and you can inform the decisions about what your product needs to be.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem. (This article)
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Taking it to the next level, as a product manager, your decisions about tomorrow’s product should be in the context of where you expect tomorrow’s competition (and tomorrow’s customers) to be.  There is a danger – especially after investing in the “state of the industry” analysis above – that you will continue to compete in yesterday’s race, when you should probably be innovating to redefine the game.  The comparison you just did is not a waste (because it looks at today’s problems – they become the table stakes for tomorrow).

Remember, this exercise informs future product decisions, it does not define them.

Attributions

Thanks Rick Cox for the height comparison photo.

Intellectual Property.  The legal jargon definition of this term has come to effectively mean “something I’ve patented, copyrighted, or hold as a trade secret.”  A more general interpretation is “an idea.”  For product managers, the most valuable ideas are insights.

The Value of Insight as Intellectual Property

Last week, the folks at Ryma Technologies Solutions invited me back to present another webinar as part of their GrandView Product Management View webinar series.  [My previous PMV webinar was on Kano Analysis] Thanks to Val and Bradley and Johnny for the invitation, and for making it a great experience!

You can watch the video version, listen to the audio, or get the slides in pdf format from the GrandView site.  If you want to get a 5-minute sample to try before you buy (with your attention, that is – this is 100% free), check out the sampler on youtube.com.

Or – if you want to flip through the slides (also on slideshare) now, and then decide – go right ahead:

Or go directly to the page on slideshare.net.

The main idea in the presentation is that your understanding of your market is the information that is most valuable.  There are several examples of this type of information in the presentation – you can see the visuals in the slides, but listening is probably required to explain most of them.

The Contentious Idea

I got some feedback that my central theorem – that insights are more valuable than patents – would be controversial.  I addressed this a little bit in the webinar.  Expanding on those thoughts here might serve to get a debate and conversation going.

I’m a former engineer (electro-mechanical controls design), and I hold 4 patents.  Each of those patents represents a body of “protected information” describing a solution to a problem.  A very specific solution.  Those patents have value.  But those patented solutions are not the only solutions to those problems.

When I was taking economics classes in college, I learned about the notions of substitute and complementary goods.  Once I became a design engineer, and learned how patents work, I also learned how to get around patents.  By designing perfect substitutes.  The process is pretty simple:

1. Pick a patented device.
2. Understand what it does – what function does it perform (and precisely how – this is the patented part)?
3. Understand what problem it was designed to solve – note: this part is not protected information.
4. Determine the important performance characteristics for solving the problem in (3).
5. Invent another way to solve the problem from (3), that meets the criteria from (4), without copying (2).

That’s it.

Every problem can be solved in many ways.  And when those alternatives all solve the problem equivalently, those solutions become perfect substitutes.  That makes any single solution effectively a commodity.  You can’t intentionally create a valuable solution to the problem without understanding the problem.

The value is in the understanding of the problem, and your market’s willingness to pay to solve it.  Not in having protection of one of the infinite number of possible solutions.  That’s the point I make in the webinar, before quickly moving to examples that show ways of developing insights about your customers [note: the examples are influenced by my approach using a customer-centric market model].

Disagree?

Chime in below or on the GrandView site, and let’s have a great discussion.  And as always, thanks for listening/watching/reading!

Estimating the “value” of features is a waste of time.  I was in a JAD session once where people argued about if the annoying beeping (audible on the conference line) was a smoke alarm or a fire alarm.  Yes, you can get to an answer, but so what?! The important thing is to solve the problem.

Solutions Versus Features

Everyone on that conference call had an immediate and visceral appreciation of the value of making the beeping stop.  That’s the power of solving a problem.  The methods of solving the problem – mute the offender, replace the battery, throw the alarm out the window – do not have implicit value.  They have an indirect value, in an “end justifies the means” kind of way.  But not direct value.

The same sort of thing applies when talking about prioritizing features.  Eric Krock (@voximate) just wrote a really good article, Per-Feature ROI Is (Usually) a Stupid Waste of Time, where he does two great things, and (barely) missed an opportunity for a hat trick.

The first great thing Eric did was look at the challenges of determining relative (ordinal or cardinal) value of “several things.”  He points out several real world challenges:

1. When you have a product with several things already and you want to determine the value of yet another thing – how do you allocate a portion of future revenue to the new thing versus the things you already have?
2. When thing A and thing B have to be delivered together, to realize value, how do you prioritize things A & B?  Relative to each other?
3. The opportunity cost of having your product manager do a valuation exercise on a bunch of things is high.  She could be doing more valuable things.
4. You won’t perform a retrospective on the accuracy of your valuation.  So you won’t know if it was a waste of time, and you won’t get better at future exercises.

The second great thing Eric did was reference a Tyner Blain article from early 2007 on measuring the costs of features.  I mean “great” on three levels.

1. As a joke (for folks who don’t know me, figured I’d mention that I’m kidding, just in case you get the wrong idea).
2. There is some good stuff in that earlier costing article about allocation of fixed and variable costs (with a handy reminder.
3. Eric’s article gives me an opportunity to shudder at the language I was using in 2007, see how much some of my thinking has evolved in four years, and improve a bit of it here and now.

What Eric slightly missed is the same thing I completely missed in 2007 – features don’t have inherent value.  Solutions to problems do have value.  He only slightly missed it because he got the problem manifestation right – it takes a lot of effort, for little reward, to spend time thinking about what features are worth.  I also missed the opportunity in an article looking at utility curves as an approach to estimating benefits, written two days after the one on cost allocation.  We were both so close!

People don’t buy features.  They buy solutions.

Valuing Solutions Instead of Features

Estimating the value of solutions addresses a lot of the real problems that Eric calls out.  It also has a side benefit of keeping your perspective outside-in versus inside-out.  Or as others often say, it keeps you “market driven.”

Anything that you’re doing, as a product manager, that has you focused on understanding your market and your customers and their problems is a good thing.  It may even be the most important thing.  I would contend that it eliminates objection 3 – the opportunity cost of estimating the value of solutions is minimal or zero.  There may be activities with more urgency, but off the top of my head, none that are more important, for a product manager.

Comment if I’m missing something (it’s late and I just got home from another week on the road).

The way I approach determining the value of a solution is by developing a point of view about how much incremental profit I will get when my product starts solving this additional problem.  Revenue can increase from additional sales, or from the ability to increase prices.  Cost can increase if new marketing and other operations (launches, PR campaigns, etc) are required to realize the incremental revenue.

I start with a customer-centric market model.

A given solution, or improved solution (as in “solves the problem better,” or “solves more of the problem”) – which only applies to some problems – is interesting to some customers, in some market segments.

A solution has value when it brings in incremental customers, in a targeted market segment.  It also has value when it reduces or prevents erosion of your current customer base (in a SaaS or maintenance-revenue model) to competitive solutions.

The time you spend thinking about buyer and user personas, the problems they care about, and the nature of those problems (which varies by persona) is not time wasted – or even spent “at the cost of doing something else.”

To make this useful, you have to have a forecast – without solution A, we will sell X; with solution A we will sell Y (and to whom).  A good product manager will be looking at sales, and will be able to reconcile the sales with the projections.  That helps with objection 4 (but doesn’t completely address it – you don’t know if your projections were accurate, so you can’t really know if your estimation is accurate).

This also helps you deal with challenge #1.  You’ve got a model that says “the current product works great for high school students, but not college students, because they also have problem A, which they solve today by…” Your intention is to create solution A, making your product viable to college students.  Allocate the incremental profits from college-student sales to solution A.

My approach to challenge #2 is a little more tactical.

Coupled Solutions

There are a couple ways that Eric’s “must deliver A and B” scenario are interesting, when looking at the value of solutions.

Scenario 1: Solution A solves part of problem X for persona M.  Solution B solves part of problem X for persona M.  Combined, they solve more of problem X for persona M.

This makes sense for “more is better” problems – where “more” solution yields “more” value.

In this case, I have a forecast (the more time I spend on it, the better it will be) that maps incremental sales to improved solutions.  The “first” solution to be released will have more value than the second.  If they are being released together, then I don’t care about the allocation – I combine them.

Scenario 2: If, however, the two solutions are valuable to different personas, then I treat them separately – even if they solve “the same problem,” it is not the same problem (for the same person).

Conclusion

Prioritization by “Bang For the Buck” is worth doing.  Just make sure you are prioritizing solutions, not features.

Also note: this article talked about valuation – what you do with that valuation, prioritizing by market, can be trickier.

A market can be thought of as the collection of contexts in which you might sell your product. You can split your market into a set of market segments. Each of those segments represents a group of customers, each of whom shares a set of problems for which they would pay for solutions.

Customer-Centric Market Model

I’ve been developing a model for describing markets that emphasizes the critical customer-centric perspective that product managers need to have.  This article is a first draft of that model – I’m sharing it here in hopes of getting feedback to improve the model and my approach for communicating it.

Key goals of the model:

• Provide a framework that helps product managers make decisions about products.
• Establish an outside-in bias that encourages product managers to think first about customers and second about implementation.
• Encourage teams to design products that solve real problems that people will pay to solve.
• Support both Agile and Waterfall development processes.

Please comment below (or privately) with any feedback about the model or presentation thereof (both prose and graphics).  Thanks in advance!

A Model for Markets, Segments, Customers, and Problems

Markets and Market Segments

Market segments are traditionally defined based on aggregate data that makes it easy to operationally address groups of customers.  Business-to-business (B2B) and business-to-consumer (B2C) markets are typically divided using the same approach, with a few data sources that are uniquely relevant for each type of company-to-customer relationship.

[larger image]

Business-To-Business (B2B)

When defining market segments for B2B products, your customers are companies.  Groups of companies are defined, and organizational structures are built around those segmentation models.  Different factors and combinations of factors are used to define the different market segments. Common B2B segmentation factors include:

• NAICS Industry Definitions – A standard taxonomy  for defining the different businesses in which your customers engage.
• Demographic Data – For companies, this can be number of employees, annual revenue, corporate structure, or other characterizing metrics that are relevant to your product.
• Transaction History – The history of purchases by the company.  Absolute magnitude, frequency, and recency of purchases can all be used.
• Geography – Where a particular company is located.  While not always correlating with variation in companies, it may serve as an effective way to “split the work” for internal organization – such as geographic sales teams.  This type of segmentation can also help when business rules or policies vary by geographic region – such as tax, policy, and other legal variations that apply either to your company or your customer.

Customers

The term customer is ambiguously applied to both buyers and users, and is only accurate when it represents someone who is both the purchaser and the end-user of your product.  You should explicitly think about your customers as buyer personas and user personas.

Buyer personas compare products based on their suitability to solve the problems that the buyer perceives as valuable.  User personas value products based on how effectively the products solve their real problems.  This is the most important distinction between buyers and users, while both are focused on value, only the end users will realize the value directly.

Market research should be focused on developing personas that represent homogeneous groups of customers, so that you can focus on the problems that each group distinctly faces.

Problems

Problems to be solved for the groups of customers that make up your market segments are what define a market.  Kano analysis provides you with the tools to classify these problems, allowing you to make informed decisions about which problems to focus on in your product roadmap.

[larger image]

Kano analysis, as originally defined, classifies products in terms of the features or capabilities they provide.  The naming originally used by professor Kano  (as translated) is slightly different than the terms used above.  The change in terminology here provides more clarity, particularly when applying Kano analysis from a market-perspective (outside-in) versus a feature-perspective (inside-out).

As a market-driven product manager, you should tweak the language of Kano into classification of problems and solutions, because customers value solutions, not features.  All of the techniques of Kano analysis apply, the nuance of this approach is to refocus on problems (outside-in) not features (inside-out).

Use Kano analysis to identify each problem faced by customers in your market as being in one of the following four classifications:

• Delighter – A solution to this problem will delight your customers.  As markets change over time, customers lose their sense of delight, and come to expect solutions to previously solved problems.
• Must Be – This problem must be solved by your product, or your prospective customer will refuse to buy or recommend it.
• More Is Better – This is the classic shades of grey problem, in contrast to the black and white nature of a must be problem.  Solving part of this problem is good, solving more of it is better.  The easiest way to think of this problem is that solutions are partial solutions, and the greater the portion of this problem your product solves, the more satisfied your prospective customers will be with your product.
• Indifferent – This is a problem for which your customers are indifferent to the solution.  Buyer decisions about purchasing your product (versus the competition) are not affected by how capably this problem is solved.  Users will not judge your product’s effectiveness based on how well it solves this problem.

Conclusion / Request for Feedback

This view of the market as an organization of problems that are shared by groups of customers allows you to

• Gain more insights from competitive analysis.
• Prioritize the content of each release to grow (revenue or profits or market share) faster.
• Design solutions that particular groups of customers will love.

That pretty much describes the model.  Please share feedback below (in the comments section) or offline.

Thanks again!

Writing Verifiable Requirements should be a rule that does not need to be written.  Everyone reading this has seen or created requirements that can not be verified.  The primary reason for writing requirements is to communicate to the team what they need to accomplish.  If you can’t verify that what the team delivered is acceptable, neither can the team.  This may be the most obvious of the rules of writing requirements – but it is ignored every day.

Verifiable Requirements – Revisiting

In 2006, I first looked at how and why writing verifiable requirements is important - as part of the ongoing series on the rules of writing good requirements.  The focus of that article was on testability, as is this one.  When visiting verifiability again, however, I will add that not only must specifications be objectively measurable, but so must goals be written so that you can identify when a solution has met the objectives that justified its creation.

Directly Verifiable Requirements

Most of the requirements we come across can be directly verified.  The only problem with these requirements is that they lack specificity.  The following example is almost verifiable, and only needs a little help:

• The web site must make it easier for users who use site search to find the products they want to buy.

This requirement presents the challenge of resolving the ambiguity of the requirement. This lack of specificity prevents the requirement from being verifiable.  You have to identify how much easier the site search process must become for users.  Findability is a more is better characteristic in the Kano Analysis model for defining requirements.

You have to acknowledge that making it easier to find what you’re looking for is better for users than making it harder – there is an upward slope to the function.  However, there are also diminishing returns.

[larger image]

• A search that includes “the perfect item” is massively better than a search that does not include what you are looking for.
• A search that includes the perfect item on the first page of results is significantly better than one that returns the perfect item on the second page of results.
• A search that includes the perfect item as the first result is only marginally better than a search that returns that perfect item as the second or third result.

Improving findability as articulated above, from a user perspective, manifests in value to your company in two ways: immediate revenue impact and indirect revenue impact.

The immediate impact can be measured in terms of increases in commerce.  The indirect impact results from improving the user experience.  These improvements in experience result in increased word of mouth, as users altruistically encourage others to visit your website, and also result in an increase in satisfaction causing users to return to your site more frequently and make more purchases from you.

You can measure these impacts in terms of

• Conversion percentage (percentage of people who search and then purchase the products found in the results)
• Revenue attributed to users who search (an absolute measurement of purchases of products found via search)
• Site traffic levels (the number of people that visit your site over time)
• Visitor-recency statistics (the amount of time that elapses between return visits for returning visitors)

Conversion percentage, for users who search, normalized against users who do not search, is the most isolated (from other variables and noise) and fastest responding (as a delayed measurement of impact) measurement of the value of making it “easier” for users to search for the products they want to buy.

Your team will design different approaches to achieving these improvements.  You have to estimate both the cost and the potential benefit of each approach.  Balancing cost estimates with potential benefits will yield the ideal requirement – perhaps a 10% improvement.  [Note: I've collapsed at least another article's worth of balancing cost versus benefit, and multiple articles of "understanding and measuring site search" into one paragraph here, in hopes of staying on task with writing verifiable requirements.]

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The web site must make it easier for users who use site search to find the products they want to buy.”
• After: “Users who search on the site will be at least 10% more successful at finding the products they want to buy when using site search.”

Impossible to Verify Requirements

Often, stakeholders will present us with requirements that are impossible to verify (as requested).

• The home page needs to load fast.

At first blush, this looks just like the previous requirement (easier search is similar to faster page load).  You can use the same techniques to determine a measurable “requirement” like “the home page needs to load in under 1 second.”  However, that’s not a good requirement, because it is not an implicitly valuable requirement.  This statement provides no insight into why a fast-loading home page might be valuable to a particular user or why it would be valuable to the company.

To address this issue, you have to meet with the stakeholder(s) and understand the actual underlying requirements.  You will ask why, determine motivations and underlying problems, and apply active listening skills to discover the underlying requirement.

The problem is that the statement, “the home page needs to load fast” is a specification.  In Requirements Documents – One Man’s Trash…, I first used the following diagram to show how different people in the process of designing software view their piece of providing a solution.

A developer may receive a spec – “home page must load in under 5 seconds” and feel like the why component is perfectly well defined – it is a matter of context and perspective.  Another way to think about this is that the problem is with the problem statement.

A product manager, however, needs to do research that follows a path like the following:

Given that we are building an eCommerce website, we know that people have expectations around page load times (per Forrester / Akamai report (requires registration)).

[larger image]

Further, those expectations manifest as people abandoning slow-loading websites (from the same report).

[larger image].

We know from this market research that page-load times (for websites) represent another more is better Kano attribute, but one that has must be characteristics at the extremes.

[larger image]

While more speed is better, what the market research reveals is that for any given person, there is a minimum-speed threshold, at which speed is a must-be characteristic – too slow, and you lose customers (immediately).

A  combination of market research and elicitation will reveal that there is a true underlying goal of being fast enough to satisfy as many users as possible.  Combining this with practicalities of scaling your solution, and the associated costs; in the context of a particular solution design (an eCommerce website), you will arrive at a goal that allows you to rework your requirement so that it is verifiable.

Note: The requirement in this example is a subordinate requirement to a goal focused on maximizing conversion percentage (the percentage of customers who visit the site making purchases) – with a recognition that a major source of non-conversion is abandonment of the site, and that a large contributor to visitor abandonment is page load times.  An Ishikawa (or other model) will help you articulate this complexity and formulate requirements at a level of abstraction that is both valuable and actionable.

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The home page needs to load fast.”
• After: “No more than 10% of potential site visitors will abandon our site before viewing the page.”

Indirectly Verifiable Requirements

Some requirements are impossible to literally verify, and must be inferentially verified.

• The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.

Models are Models, Not Reality.

Imagine trying to create a test by organizing a million visitors to hit your SaaS web solution on the same day, with at least 10,000 of them hitting the site simultaneously.  That is completely impractical.  That doesn’t however, invalidate the requirement or make it untestable.  The combination of modeling, hypothesis formulation, and extrapolation gives you a reasonable way to verify that your solution probably meets this requirement.

You’re already used to the concept that models are representations of reality – think about the maps you use – a map may have a scale like one inch equals one mile.  The map is a model of reality.  A map where one inch equals one inch would be impractical.

An effective approach to measuring this type of scalability requirement is to simulate users (with load testing and realistic user-behavior scripts) with automation.  That takes care of most of the coordination problem.  However, the cost of simulating a million users and 10,000 simultaneous users is high.  You can, more realistically, measure the site’s performance when 10, 100, and 1,000 simulated users simultaneously access a server.  You can extrapolate from those results to estimate how the system is likely to perform under 10,000 user loads.

Note: Extrapolation is dangerous (follow the Elvis link below)- you have to justify why extrapolation holds true, and identify when it does not, to minimize the risk of invalidating your hypothesis.  Make sure you have confidence in the engineering prowess of whoever is doing your extrapolation and modeling.

[see tip #5 of these ROI calculation tips for the Infinite Elvis anti-pattern]

While this requirement does not to be rewritten in order to be verifiable, it does need to be augmented:

• Original: “The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.”
• Additional: “Testing of the site must show that 500 simultaneous users following <user script X> will yield no more than 1% page load times over 200 ms.”

Conclusion

Every requirement you write must be verifiable.  When you can’t verify something, and can’t rewrite it into a verifiable form, that should be a sign that either it is a vision statement or a red herring.  Vision statements guide how we approach creating products and engage markets – they are valuable, but they are not requirements.  Red herrings are well-meaning but ill-advised inputs into the process that need to be culled – they are neither valuable nor requirements.

Make sure all your requirements are verifiable.

“For what one idea do you want your product to stand in the mind of your customer?”  I heard Roger Cauvin ask that question at the most recent ProductCamp Austin [correction - he said it here - thanks Roger], and the quote has been jumping to the front of my mind almost daily ever since.  Maybe by writing about it I can exorcise the demon and get back to using the idea instead of being haunted by it.

The One Idea

You’ve just been given a new assignment – your company needs a new software thingamajig so that you can play in the whatchamacallit space, and you’re going to drive product management for it.  You’ve also been asked to deliver the first version in six weeks.  Of course you’ll be able to do follow-on releases, after all, you are agile.  That’s how “agile” works, isn’t it?

Cool.  Exciting.  Challenging.

You sit down with the folks who will be building the product, and find out that they have already spoken with several stakeholders.  Excellent. You circle back with the stakeholders, and start to gather data about your market and audience.

Your schedule is aggressive, so you think about what is realistic to get done in a very small timebox.  With the current schedule, you can’t afford to get caught in analysis paralysis, and you can’t realistically do in depth market analysis, persona development, hyper-accurate requirements prioritization, etc.  You do, however, have time to do the most important thing – define the one idea that will define your product.

If you cannot come up with this idea, you need to push back on your management team, and delay the launch of your product until you have that one idea.

Don’t Abuse Agile

Agile is designed to help you rapidly create iteratively better products, with iterative development and continuous infusion of market feedback and data.  Agile is not a process by which you start typing without any idea of what you intend, releasing it and then getting feedback in an iterative process.  If that’s how you’re approaching agile, your process is broken.

[image from Market Driven Competitive Advantage]

Unless you’re very lucky, your first iteration will be a waste of time and money.  Wouldn’t you rather be intentional than lucky? Make sure you have that one idea before you start developing.

Strategic Alignment

You need to come up with a one idea that is aligned with and supports your corporate strategy.  At a minimum, you have to make sure that you have an idea that is aligned with your stakeholder’s goals, based on the assumption that those goals are aligned with corporate strategy.

Wow

Your one idea really needs to make your users say “Wow!”

I was explaining the importance of this to a colleague (who is not a product manager), and his response was “Oh, you mean the hook.”

I was reminded that you not only have to provide a wow experience for your users, but you have to present a hook that will capture the imagination of your buyers.  Remember that buyer personas make decisions based on their perceptions of user problems [more on convincing buyers].

Satisfice

Remember that you don’t want to try and do everything perfectly in the first release – you want to satisfice.  You may only be able to target a single persona with a solution to a single problem.  You just need to make sure you are solving the right problem – which is where Kano analysis is very handy for identifying the Minimum Market Acceptance criteria.

[From Minimum Market Acceptance]

Conclusion: A Balancing Act

Defining the one idea for your product is a balancing act.

• You have to align the one idea with your corporate strategy and stakeholder goals
• You have to solve a valuable problem for your users, presented in a way that captivates your buyers
• You have to create a product that is compelling right away – good enough in execution to enter your market
• And you have to get the first release out in six weeks!

Good luck!  Or better yet – Good Intent!

April Dunford just presented Startup Marketing 101 at DemoCamp Toronto.  Great ideas from the ‘marketing and your startup’ point of view.  I’ve often said that product managers and product marketers care about much of the same market data, they just do different things with it.  The idea of minimal feature set came up in April’s presentation – this article talks about product management, agile, and initial market acceptance.

Initial Market Acceptance and Minimum Market Acceptance

April mentions (in slide 4) that an important event to the timing of marketing activities is achieving the “_minimal feature set (for certain markets)_.”  April organizes those marketing investments in terms of three stages of “product marketing lifecycle”:

1. What to do when your startup has a “pre-product” product.
2. Where to focus when you’re focused on early adopters.
3. How to invest in your marketing programs when your focus is on scale / growth.

Coming from an agile background, I think about two distinct notions of “minimum” or “initial” when it comes products and markets.  The following definitions are in a B2B context, as I’ve been focusing on this recently with a client in the B2B space.  The same ideas are relevant to B2C and B2B2C companies, but the language would be slightly different.

I’m assuming that when April used the phrase “minimal feature set”, she was really talking about “minimum market acceptance” in the way that I’m describing here.  The distinction is that features describe what a product does, which is an inside-out view of the product.  To be market focused, you have to think about which market problems are being solved, for whom they are solved, and how well they are being solved – an outside-in product view.

If you focus on, and organize around your features, you are likely to miss the mark.  You must focus on the problems that your customers need to solve.  In your product, you will prioritize the development of capabilities (embodied through features) that are designed to help your customers (buyers and users) solve their problems.  Your market analysis should be geared around understanding how many customers in each segment face each problem, and how much they are willing to pay to solve those problems.

Agile or Waterfall or Waterfragile?

As an agile product manager, and former developer, I know that when you do “everything we need to be successful” in the first release, you’re not being agile – we’re being waterfall.  A slight extension of this is “everything we need to be minimally successful” – and that is still waterfall.

An agile team should focus on the first release addressing the initial market acceptance criteria – the minimal set for a single customer.  The goal is to get the product in front of customers as soon as possible – this starts your revenue stream earlier, gives you valuable market feedback, and gives you the opportunity to establish momentum through repeated incremental releases of your product.  Each release will solve addressed problems “better” and / or address additional problems, until you reach minimum market acceptance.

Kano Analysis for Understanding Problems

Last year, I was thrilled to share my approach for applying Kano Analysis as a product manager with the PMV webinar audience as well as the Austin IIBA chapter.  One of the key ideas in the application of Kano Analysis to understanding your market is developing the personas within that market, and understanding how each of them treats each problem / capability.

[slideshare presentation & PMV webinar]

This analysis, in addition to being useful for understanding the market (or a segment) as a whole, also helps you identify your first customer.  Once you know who your first customer is, you can determine the initial market acceptance criteria for that customer, and that determines the must have capabilities.

Note that finding the initial solution (that the first customer will buy) does not mean releasing a poor quality product – it just means releasing a product that solves enough problems (well enough) for one customer.  With the feedback from that customer, you can drive the prioritization for your next iteration – making the next iteration better for that customer (which can really help your word of mouth), and gaining more customers.  Repeat this process until you get to minimum market acceptance.

Timing Marketing Investments for Minimum Market Acceptance

[slide 9 from April's presentation on slideshare]

Since I wasn’t at April’s presentation, I don’t know exactly what she would think of the following, but I believe it makes sense:

• Initial Market Scceptance (one customer would buy your product) marks the transition from pre-product to early adopters.
• Minimum Market Acceptance (enough to succeed in the market) happens before the transition from early adopters to scale.  Note that it may not (probably doesn’t) mark the transition, but I suspect happens mid-stream.

Would love to see what other folks think about mapping this product management centric view to the marketing timeline.  Please chime in below.

You give your requirements to the engineering team, and they look complete.  The team builds your product, you launch it and the market soundly rejects it.  Why?  Because your requirements weren’t complete – they didn’t actually solve the problem that needed to be solved.

Complete Requirements – Revisiting

Going back almost four years, I wrote Writing Complete Requirements, as part of the Big Rules of Writing Requirements series.  That article centered on two key ideas of requirements completeness.

• Data, not opinion, is the most important input into determining completeness.
• There is no absolute way to determine the completeness of your requirements in advance.

Completeness is best measured by market acceptance, so technically you can’t know if your requirements were complete until your customers buy (or fail to buy) your product.  This doesn’t mean you should give up on this rule of requirements – you can still focus on, and improve, the completeness of your requirements.

Objective and Heuristic Assessment

Given a market problem and the associated requirements (which you can argue are actually the same thing), you can review those requirements to determine if they objectively appear to be complete.  This assessment is based on what you objectively know about your market (needs, opportunity costs, competitive alternatives, etc).  You can also apply heuristic, or logical analysis to identify gaps in the language of your requirements.

Objective assessment is the application of market knowledge (data) to determine if the nature of the problem is being addressed completely by your requirements.  A bird-feeding maven would know that the requirements for your bird feeder would be incomplete if they didn’t address the problem of squirrels stealing food from the birds.

Heuristic analysis is the identifications of logical omissions in your requirements, without the need to apply market insights.  A logician would know that the requirements for your bird feeder were incomplete if they didn’t address the need to refill an empty feeder.

When you’re writing requirements, you need to do both.  Heuristic analysis of your requirements will identify where your requirements do not fully solve the problems you already know about – an eCommerce checkout process without the ability to receive payment, for example.  Objective assessment will help you identify the problems that you overlooked but need to solve – the need to allow customers to have different billing and shipping addresses when placing an order online, for example.

Complete Requirements for Incomplete Solutions

Don’t confuse having complete requirements (a good thing) with completely solving a problem (not always a good thing).  In agile development, the idea of satisficing is key to scoping individual iterations.  The idea, simply put, is to solve enough of the problem, and not delay delivery (and value) until you can build a solution to the entire problem.  Because of the law of diminishing returns, for many market problems, there is little or no incremental value in solving the entire problem.  Those resources can be better applied to solving additional problems.

Kano analysis provides a framework for identifying which market problems must be completely solved, and which ones have a more is better characteristic.  In practice, more is better problems exhibit the law of diminishing returns, and should not be “completely” solved.

[larger image, or view the entire Kano analysis presentation]

Conclusion

Complete Requirements are requirements that

• Identify all of the market problems that need to be solved and their nature (absolute vs. diminishing returns).
• Are logically complete in their coverage / articulation of those market needs.

To objectively improve the completeness of your requirements, apply market data in an assessment of your requirements.  To heuristically improve the completeness of your requirements, look for logical inconsistencies and gaps in reasoning or coverage.

Kano Analysis, while initially created to understand customer satisfaction with features, can be used by product managers to better understand customer problems.  I gave a presentation last week for the Product Management View webinar series on Kano Analysis for product managers.

Kano Analysis – The Webinar

You can watch (flash) or listen to (mp3) the webinar (just under an hour) at the Product Management View webinar page for the Kano Analysis presentation.  I delivered the webinar on Sep 23rd, 2009.  Thanks Val Workman for inviting me to join the group and share an hour with a bunch of great folks.  Also, thanks to the people who asked questions, tweeted, and commented on the presentation.  I really appreciate it.

Kano Analysis – The Slides

If you want to flip through the slides to get a feel for the treatment I gave to Kano Analysis before committing an hour to the webinar, you can do so right here (or view the Kano Analysis slides on slideshare.net).

Thanks again to everyone, and if you have any feedback, include it below or on the PMV site.  Several links to other articles about Kano Analysis are in the comments on the PMV page, if you want to do further research.