Writing Verifiable Requirements should be a rule that does not need to be written.  Everyone reading this has seen or created requirements that can not be verified.  The primary reason for writing requirements is to communicate to the team what they need to accomplish.  If you can’t verify that what the team delivered is acceptable, neither can the team.  This may be the most obvious of the rules of writing requirements – but it is ignored every day.

Verifiable Requirements – Revisiting

In 2006, I first looked at how and why writing verifiable requirements is important - as part of the ongoing series on the rules of writing good requirements.  The focus of that article was on testability, as is this one.  When visiting verifiability again, however, I will add that not only must specifications be objectively measurable, but so must goals be written so that you can identify when a solution has met the objectives that justified its creation.

Directly Verifiable Requirements

Most of the requirements we come across can be directly verified.  The only problem with these requirements is that they lack specificity.  The following example is almost verifiable, and only needs a little help:

• The web site must make it easier for users who use site search to find the products they want to buy.

This requirement presents the challenge of resolving the ambiguity of the requirement. This lack of specificity prevents the requirement from being verifiable.  You have to identify how much easier the site search process must become for users.  Findability is a more is better characteristic in the Kano Analysis model for defining requirements.

You have to acknowledge that making it easier to find what you’re looking for is better for users than making it harder – there is an upward slope to the function.  However, there are also diminishing returns.

[larger image]

• A search that includes “the perfect item” is massively better than a search that does not include what you are looking for.
• A search that includes the perfect item on the first page of results is significantly better than one that returns the perfect item on the second page of results.
• A search that includes the perfect item as the first result is only marginally better than a search that returns that perfect item as the second or third result.

Improving findability as articulated above, from a user perspective, manifests in value to your company in two ways: immediate revenue impact and indirect revenue impact.

The immediate impact can be measured in terms of increases in commerce.  The indirect impact results from improving the user experience.  These improvements in experience result in increased word of mouth, as users altruistically encourage others to visit your website, and also result in an increase in satisfaction causing users to return to your site more frequently and make more purchases from you.

You can measure these impacts in terms of

• Conversion percentage (percentage of people who search and then purchase the products found in the results)
• Revenue attributed to users who search (an absolute measurement of purchases of products found via search)
• Site traffic levels (the number of people that visit your site over time)
• Visitor-recency statistics (the amount of time that elapses between return visits for returning visitors)

Conversion percentage, for users who search, normalized against users who do not search, is the most isolated (from other variables and noise) and fastest responding (as a delayed measurement of impact) measurement of the value of making it “easier” for users to search for the products they want to buy.

Your team will design different approaches to achieving these improvements.  You have to estimate both the cost and the potential benefit of each approach.  Balancing cost estimates with potential benefits will yield the ideal requirement – perhaps a 10% improvement.  [Note: I've collapsed at least another article's worth of balancing cost versus benefit, and multiple articles of "understanding and measuring site search" into one paragraph here, in hopes of staying on task with writing verifiable requirements.]

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The web site must make it easier for users who use site search to find the products they want to buy.”
• After: “Users who search on the site will be at least 10% more successful at finding the products they want to buy when using site search.”

Impossible to Verify Requirements

Often, stakeholders will present us with requirements that are impossible to verify (as requested).

• The home page needs to load fast.

At first blush, this looks just like the previous requirement (easier search is similar to faster page load).  You can use the same techniques to determine a measurable “requirement” like “the home page needs to load in under 1 second.”  However, that’s not a good requirement, because it is not an implicitly valuable requirement.  This statement provides no insight into why a fast-loading home page might be valuable to a particular user or why it would be valuable to the company.

To address this issue, you have to meet with the stakeholder(s) and understand the actual underlying requirements.  You will ask why, determine motivations and underlying problems, and apply active listening skills to discover the underlying requirement.

The problem is that the statement, “the home page needs to load fast” is a specification.  In Requirements Documents – One Man’s Trash…, I first used the following diagram to show how different people in the process of designing software view their piece of providing a solution.

A developer may receive a spec – “home page must load in under 5 seconds” and feel like the why component is perfectly well defined – it is a matter of context and perspective.  Another way to think about this is that the problem is with the problem statement.

A product manager, however, needs to do research that follows a path like the following:

Given that we are building an eCommerce website, we know that people have expectations around page load times (per Forrester / Akamai report (requires registration)).

[larger image]

Further, those expectations manifest as people abandoning slow-loading websites (from the same report).

[larger image].

We know from this market research that page-load times (for websites) represent another more is better Kano attribute, but one that has must be characteristics at the extremes.

[larger image]

While more speed is better, what the market research reveals is that for any given person, there is a minimum-speed threshold, at which speed is a must-be characteristic – too slow, and you lose customers (immediately).

A  combination of market research and elicitation will reveal that there is a true underlying goal of being fast enough to satisfy as many users as possible.  Combining this with practicalities of scaling your solution, and the associated costs; in the context of a particular solution design (an eCommerce website), you will arrive at a goal that allows you to rework your requirement so that it is verifiable.

Note: The requirement in this example is a subordinate requirement to a goal focused on maximizing conversion percentage (the percentage of customers who visit the site making purchases) – with a recognition that a major source of non-conversion is abandonment of the site, and that a large contributor to visitor abandonment is page load times.  An Ishikawa (or other model) will help you articulate this complexity and formulate requirements at a level of abstraction that is both valuable and actionable.

Rewriting the requirement as follows makes the requirement verifiable:

• Before: “The home page needs to load fast.”
• After: “No more than 10% of potential site visitors will abandon our site before viewing the page.”

Indirectly Verifiable Requirements

Some requirements are impossible to literally verify, and must be inferentially verified.

• The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.

Models are Models, Not Reality.

Imagine trying to create a test by organizing a million visitors to hit your SaaS web solution on the same day, with at least 10,000 of them hitting the site simultaneously.  That is completely impractical.  That doesn’t however, invalidate the requirement or make it untestable.  The combination of modeling, hypothesis formulation, and extrapolation gives you a reasonable way to verify that your solution probably meets this requirement.

You’re already used to the concept that models are representations of reality – think about the maps you use – a map may have a scale like one inch equals one mile.  The map is a model of reality.  A map where one inch equals one inch would be impractical.

An effective approach to measuring this type of scalability requirement is to simulate users (with load testing and realistic user-behavior scripts) with automation.  That takes care of most of the coordination problem.  However, the cost of simulating a million users and 10,000 simultaneous users is high.  You can, more realistically, measure the site’s performance when 10, 100, and 1,000 simulated users simultaneously access a server.  You can extrapolate from those results to estimate how the system is likely to perform under 10,000 user loads.

Note: Extrapolation is dangerous (follow the Elvis link below)- you have to justify why extrapolation holds true, and identify when it does not, to minimize the risk of invalidating your hypothesis.  Make sure you have confidence in the engineering prowess of whoever is doing your extrapolation and modeling.

[see tip #5 of these ROI calculation tips for the Infinite Elvis anti-pattern]

While this requirement does not to be rewritten in order to be verifiable, it does need to be augmented:

• Original: “The site must support 1 million visitors per day, with a peak of 10,000 simultaneous with page-load times under 5 seconds.”
• Additional: “Testing of the site must show that 500 simultaneous users following <user script X> will yield no more than 1% page load times over 200 ms.”

Conclusion

Every requirement you write must be verifiable.  When you can’t verify something, and can’t rewrite it into a verifiable form, that should be a sign that either it is a vision statement or a red herring.  Vision statements guide how we approach creating products and engage markets – they are valuable, but they are not requirements.  Red herrings are well-meaning but ill-advised inputs into the process that need to be culled – they are neither valuable nor requirements.

Make sure all your requirements are verifiable.

Writing unambiguous requirements is about understanding what is written, and what is read.  Without a clear understanding of your market, you can’t write unambiguously.  Even when you understand your market, you risk writing something that is ambiguous to your readers.  Documenting requirements is about communication.  Don’t break this rule, or you’ve wasted all the energy you spent understanding your requirements.

Unambiguous Requirements – Revisiting

Fifty months and five days ago (another grin), I wrote about the importance of writing requirements so that they can be clearly understood.  Writing requirements so that there is one, unambiguous, interpretation of what you wrote.  This article is the seventh in a series on the rules of writing good requirements.  In that version of the article, I focused primarily on language ambiguity, and framed the discussion in terms of ambiguity to the writer and to the reader.

As part of improving that series, I am revisiting each of the articles now that I have fewer hairs and the remaining ones are more gray.  The same framework still applies, but will be expanded in scope, using the following modified definitions of the framework.

• Ambiguity for the Writer – not having a clear interpretation of the requirements.
• Ambiguity for the Reader – requirements that, as written, can be interpreted in more than one way.

It takes two to tango.  To lead, you have to know where you’re going.  To follow, you have to know where you’re being led.  The same is true with requirements – you have to know what you’re writing, and how your writing will be read.  Misinterpretation of requirements is the source of over 40% of all software defects.

Ambiguity for the Writer

When you don’t know why you’re developing a particular set of requirements, it is because you either don’t have sufficient understanding of your market, or of your corporate objectives.  While it is always a lack of knowledge, sometimes, that lack of understanding is insidiously hidden.  You, as the writer of requirements, are the reader of the market – and what you’re reading is likely to be ambiguous.

For Whom You Are Designing

The most overlooked element I’ve personally seen in product creation is failing to understand who the users and buyers of your product will be.  I wanted to type “forgetting to understand,” but I struggle to imagine someone who understands the value of defining user personas and buyer personas, who then forgets to define them when gaining an understanding of their market.

The reason that this is an ambiguity issue is that when you fail to define personas explicitly, you define them implicitly.  An implicit persona is an ambiguous persona.  Alan Cooper named this anti-pattern the elastic user, and the problem manifestation as failing to provide insight to guide design decisions.  When you have an ambiguous user, you lose what Frederick Brookes calls the clarity of design. This results in at best a mediocre product without a core idea or aesthetic.

Different users value the same solutions differently.  Users vary in competency, and therefore have varied personal goals as they learn how to better use your product over time.  When you don’t identify who the user is, you prioritize requirements without insight into what is important to the undefined, elastic, and ambiguous user.

The readers of your requirements are trusting that you will prioritize requirements correctly – but you can’t, because you lack an understanding of who the users will be.  You therefore lack an understanding of which problems are valuable, and which problems are more or less valuable than other problems.  This ambiguity makes it particularly difficult to write valuable requirements.

Precision in Language

The other aspect of writing unambiguously is in using language that has meaning.  Using words like “can” and “may,” is imprecise in requirements.  ”The system can respond in under three seconds.”  Does that mean that it must respond in under three seconds?  Does that mean it is acceptable if the system responds in twenty seconds?  This is an area of overlap with the rule of writing measurable, verifiable requirements, so I won’t go into more details here.

Another source of imprecision in language, not only for the reader, but also for the writer, is jargon.  You should never use jargon, as you have no way to know if your reader will actually interpret the jargon terms correctly – as readers often vary in the depth of their domain knowledge.  Jargon is also a high risk of ambiguity source, precisely because it is jargon.  Jargon is short-hand for communicating efficiently among domain experts.  Are you enough of a domain expert that you are confident that you absolutely, unambiguously understand the accepted meaning of the jargon term?

Hopefully, you’ve achieved a perfect understanding of your market.   Long-time readers will know that I’m joking.  Market analysis suffers from it’s own Heisenberg Uncertainty Principle – if you actually did manage a perfect understanding, the market will have changed – the best you can hope for is a statistical understanding of the behavior of your market.  You can document your interpretations, assumptions, and decisions – achieving clarity in your requirements – at least from your point of view.

Ambiguity for the Readers

As I mentioned earlier – using requirements is a dance – and it takes two, both the writer and the reader.  You need to write requirements so that they can only be interpreted in a single way, or you will stumble.

“You keep using that word.  I do not think it means what you think it means.” – Inigo Montoya (IMDB)

Shared Language

The word “shall” as in “the system shall…” can be misinterpreted when your readers don’t share the same native language as you – which is why you must always use “must.“

Some good research on vocabulary size data for comprehension of english can be found here. The average native english speaker knows 20,000 word families. With a vocabulary of 5,000 english words, only 98.5% of the words in a given text will be understood. 5,000 words is a lot for a speaker of a second language (3,000 words is considered aworking knowledge).

An analysis of the reading-level at which a document is written can be helpfull in identifying if the language is likely to be challenging for readers with limited vocabularies. The Gunning-Fog Index provides a measure of the education level at which a text is written.

Writing Unambiguous Requirements (2006)

Can’t write that section any better :).

Ambiguous Use of Language

I had the privilege to attend a presentation by professor Daniel M. Berry in 2007, on the ambiguous use of language in business rules and requirements.  In that presentation he referenced The Ambiguity Handbook that he authored (80 page pdf), which I just re-read last week.  There are many good examples of ambiguity in the use of English in business documents.

The following are some of the sources of ambiguity, for which I entirely credit professor Berry for pointing out to me.  I’ve only added some illustrative examples:

• Plurality Causes Ambiguity.  Consider “Emails must include sender addresses” and “emails must include recipient addresses.”  Must each email include one sender address and one recipient address?  Must each email include multiple sender addresses?  Solution – don’t use plural subjects.  ”Each email must include a sender address” and “each email must include recipient addresses.”
• Associative Ambiguity.  Professor Berry calls this the parenthesis problem.   What does “A and B or C” mean to you?  Does it mean “A, and either of B or C” or does it mean “either C, or both A and B” when you read it?  In math and programming, we learn very specific rules for how to interpret these types of structures.  We are also given parentheses, that we can use when we want to be specific and unambiguous.  The reader of your requirements is not a compiler, so don’t assume that she will interpret this the same way that you do.  Solution – use parentheses and explicit language to eliminate ambiguity that would arise from ignorance of the associative property.
• Anaphor Ambiguity.  For non-linguists: “don’t use pronouns.”  I love the example that wikipedia uses – “We gave the bananas to the monkeys because they were hungry.  We gave the bananas to the monkeys because they were ripe.
  
  We gave the bananas to the monkeys because they were here.”  The pronoun, “they,” has an ambiguous binding.  Should “they” be a reference to the bananas or the monkeys?  In each of the first two sentences, you could reasonably assume that the reader will properly bind “they” to the monkeys and bananas respectively.  In the third sentence, an assumption of how the reader will interpret “they” is not reasonable.  Reasonable assumptions are still ambiguous – and the context is less likely to be obvious – so don’t use pronouns and rely on readers to bind them to the appropriate nouns.  Solution – repeat the noun for every reference.

Incompleteness Ambiguity

Unwritten requirements are by definition ambiguous.  You must not assume that unstated requirements will be implemented.

A security exploit was recently discovered (and eliminated) in Facebook.  Previously, if you attempted to login to Facebook with an email address and a password that did not match the email address, Facebook would present you with a typical “failed login” screen and a link to reset your password.  However, in an effort to be friendly, Facebook would also include the first name and last name that were associated with that email account in the friendly warning message.

That is nice, except it is a privacy problem, and anyone who knew about this before could use the login page to discover the first and last name to associate with any email address.  ”Bad people” with email address lists could discover the name to associate with many of those email addresses – allowing more sophisticated phishing or social engineering attacks.

The requirement to not provide any personal information when a user fails a login attempt was almost certainly an unwritten requirement.

Conclusion

Precision in your writing will reduce ambiguity in your requirements.  Active listening will help you discover when ambiguity has slipped through your editing net.  You can never completely eliminate this problem, but you can improve.  You must.

Accept has invited me to participate in their webinar series on Transparency and Innovation – this Wednesday, July 28, 2010 (10AM Pacific, 1PM Eastern).

Join us and join in!

The Importance of Innovation and Transparency

Cool, huh? :)

Accept is hosting a free 5-part series on transparency, and has invited me to be one of their speakers.

The other speakers are rock stars – Nils Davis & Alex Lobba [replay available], Tom Grant, Roy Wildeman, and Brian Lawley.

Each of us will be looking at the importance of innovation and transparency from a different perspective.  Combine that with encouraged audience participation, and we should really be able to tease out some powerful ideas!

Join Us at 10AM Pacific on Wednesday, July 28th, 2010

I’ll be talking about how transparency (of the needs of your customers) affects the entire product creation team – not just product managers.  Register today for free, and I hope to get some great questions from you.

Thanks!

vs.

What happens when billionaire media magnate, Rupert Murdoch, pits his idea against a Nobel-prize winning idea from the beautiful mind of economist and mathematician John Nash?

When you act on what you hope your market will do, instead of what you predict your market will do – you’re in trouble.

This is a story about understanding your market, and an example of using game theory – specifically, the Nash Equilibrium in “non-cooperative game theory” to predict market responses to your products.

Murdoch’s Idea

To sum up the particular idea I’m talking about – Mr. Murdoch has put the Times of London behind a pay-wall (you have to subscribe to access the content online), and removed the articles from Google’s search index.  He did this because he believes that people accessing the content for free are “stealing it” and that the only way to make money in the news business online is by forcing people to pay for content.  For more nuanced and deeper discussions of what Mr. Murdoch is doing, check out these articles from The Atlantic explaining what he’s attempting, the Editor & Times report of a 66% drop in traffic, Newser’s investigative journalism of what’s happening inside the company, or Business Insider’s take on the matter.

My opinion is that Mr. Murdoch expects his market will respond the way he wants them to – by subscribing to the online version of the Times of London – in stark contrast both to pundit predictions and what professor Nash’s Nash Equilibrium model of non-cooperative gaming predicts will happen.

As results have finally started surfacing, they appear to completely support what game-theory would predict and utterly refute what Mr. Murdoch would have hoped for.

Game Theory for Predicting Market Behavior

I was enthralled by Scott Steven’s lecture series, Games People Play: Game Theory in Life, Business, and Beyond – a DVD set of a series of lectures from The Teaching Company [link goes to Amazon].

Understanding your market involves not only knowing what problems your customers face, but also predicting how your competitors will behave.  Competitive analysis is not just capturing a snap-shot of their products and positioning today, but also forming predictions of how they will respond to the disruptions you will create in your market by innovating.  Markets are not static – you have to understand both how your customers’ needs will evolve and how your competitor’s offerings will change, in order to understand how your product will perform.

One of the “games” in game theory is called The Coordination Game, and it is the one that captures Mr. Murdoch’s conundrum.  The coordination piece of this game is in acknowledging that what you do (to pay-wall or not to pay-wall), and what your competitors do (pay-wall or not) are interdependent decisions, of which you only control your own decision.

You can’t (successfully) unilaterally decide what you are going to do, without taking into account what your competition is going to do.  And vice-versa.  That’s the part that makes it hard – there is a circular reference that has to be resolved.  I suspect that’s the source of elegance in the Nash Equilibrium – he realized that there are “stable” and instable combinations of the decisions – yours, and your competitors.

The Pay-wall Coordination Game

Mr. Murdoch and “everyone else” (his competition for consumers of news) are both faced with a choice – should they allow free access to the articles, or not.  Free access to articles can generate revenue by two mechanisms.

The first is one of the freemium business models – billing Peter to pay for Paul – charging money to advertisers for access to viewer’s attention.

The second mechanic is by leveraging word of mouth – people reading your free article, then encouraging others to read your other articles.  This approach can either be used to get a percentage of people as paying customers (with a “leaky pay-wall” model that allows some free views, but encourages paid-subscriptions), or to get additional traffic that increases revenue from advertising.  One way to maximize this approach is by making your product implicitly viral so that it promotes itself.

Alternately, Mr. Murdoch, or any of his competitors can create a pay-wall, where only paying subscribers will have access to the content.  Presumably, certainly historically, a subscription model with the same level of readership – if that could be achieved – would generate more revenue for the publisher than either freemium model.

The crux is that we have tons of market data to show that “the same level of readership” simply will not happen.

There are four possible scenarios that can happen in this market -

1. Both Mr. Murdoch and “everyone else” institute pay-walls, making all online content available only to paying subscribers.
2. Mr. Murdoch erects a pay-wall, but everyone else continues to allow free access to content.
3. Mr. Murdoch allows free access to content while everyone else erects pay-walls to prohibit access to content.
4. No one erects a pay-wall, and free access is allowed to all content.

[Update 2010.07.21: Why "everyone?"  See comment #2 below]

In scenario 1, assuming people don’t just stop reading online, everyone wins – everyone charges subscription prices, achieving the maximum amount of revenue.  So why didn’t this happen?  Because the decisions are interdependent, and because the market is starting out in scenario 4.

Mr. Murdoch has moved the market (from his perspective) into scenario 2, which is clearly bad for him, in hopes that everyone else will move as well, to reach scenario 1.  Once Mr. Murdoch put the Times of London into scenario 2, then everyone else’s best move is to stay there.  No one else will join him, because everyone else never will.  The market, once stabilized in scenario 4 (the starting point) can never move to scenario 1 – without collusion, which is conveniently illegal.

Someone will always be there to refuse to move to scenario 1.  ”Everyone else” is destined to allow free access, or exit the market.

Understanding Your Customers

It is possible that Mr. Murdoch looked at the Wall Street Journal, which has a pay-wall, and said “I want to do that too.”  And perhaps he discounted all of this game-theory “nonsense” because if it worked for them, it can work for him – the model is BS.

However, I believe the Journal’s market is different, and it works for them for different reasons.  The market for business news pays a premium for immediacy and quality of information – because getting the best information first is worth a lot of money to them.  There are “free” immediate sources of business news information, but the market indicates that it perceives that information to be of lower quality.  And there are free high-quality information sources, without the immediacy.

The reality, which you can only get from understanding your customer’s goals – is that the business-news market is stable in scenario 1. It could be destabilized, by someone offering free, timely, high quality information.  If that happened, as professor Nash showed, the market would either re-stabilize in scenario 1 (the company that changes says “oops” and reverts their policy), or the market would transition to a stable state in scenario 4 (everyone offers free content).

The more concentrated the market [see: measuring market concentration], the more likely a reversion to scenario 1.  The more competitive the market, the more likely that it will stabilize in scenario 3.

Non-business news on the internet?  Very competitive.  And stable in scenario 4, no matter what Mr. Murdoch might hope.  Sadly, there are profits to be made in scenario 4, but they aren’t as appealing as the double-rainbow of scenario 1 – however impossible it is to achieve.

Attributions

The two photos at the start of this article of Mr.s Murdoch and Nash are licensed through the Creative Commons Attribution Share-Alike 2.0 Generic and the Attribution Share-Alike 3.0 Unported licenses, respectively.

All other content in this article is licensed per the Creative Commons link at the bottom of this page.

As product managers, we talk about creating the right solutions with our products. Understanding the very real problems our customers face, understanding the very real opportunities our markets present, and manifesting that understanding in a product roadmap.

Other than being “not as good,” how expensive is it to build the wrong product?

The Cost of Poor Quality

There’s an analog to the market dynamics of making poor product decisions – executing with poor quality. Many research studies and articles have identified the market impacts of poor quality.  This has become so well accepted that people today cite it like a law of physics (one example here based on this 1988 IEEE research by Barry Boehm and Philip Papaccio) as the “1-10-100 rule.”  The primary conclusion of that research is that ten dollars spent on fixing bugs:

• Costs and saves $10 when you catch (and fix) the bug during implementation.
• Avoids $100 in costs when you catch the bug during QA and send the product back to development (then test again).
• Avoids $1,000 in costs versus waiting until your customers catch the bug in the field, causing the team to remedy the problems, rush out a patch release, and/or go to heroic lengths to manage a PR problem.

This is an opportunity in front of your product team – a 100x payback from investing in quality during the development process.  Of course, be pragmatic about it - if the cost of testing exceeds the cost of bugs, don’t test.

This is not a solved problem, by any stretch, but the solutions and methods to solve this problem are well understood now.  In fact, a 2001 article by Barry Boehm and Victor Basili shows that in some cases, the labor costs to resolve bugs can be as low as 5:1 – when considering a subset of smaller systems, when using more “agile” processes.  That lowered ratio does not take into account the lost market opportunities and the costs of cleaning up collateral damage to your product – just the immediately realizable (and measurable) costs of resolution.

One very real problem, when talking about “bugs” is in defining what a “bug” is. And the definition of a bug is a matter of perspective. A developer can reasonably assert that “if it meets the spec it is not a bug, it is working as designed.” What if the spec is wrong? The developer may not be guilty, but collectively, your team screwed up. There’s a “bug” in the requirements.

What Is A Requirements Bug?

Now things are getting interesting.

If you wrote a requirement that you interpret as “A” and your developers interpret as “B” – you definitely have a bug – the team won’t build the right product. For each $1 you could spend making sure you have bug-free requirements, you could:

• Make sure you have a shared understanding of the documented requirements through active listening before development begins ($1). Following the Rules of Writing Requirements will help prevent this miscommunication.
• Wait until the engineering team is ready to demo their progress ($10). They will have to build it again, because they built the wrong stuff.
• Wait until development is complete and QA is validating that the code meets the spec ($100). This gets tricky if you are thinking “A”, the developers are thinking “B”, and QA is thinking “C.”
• In classic throw-it-over-the-wall mode, wait until the product is launched, and it is the wrong product ($1000). Assuming “A” was the right problem to solve, the cost of entering the market with a solution to “B”, leaving “A” unaddressed, is impressively high.

This gets interesting because the above assumes that “A” was the right problem to solve. What if “G” was the right problem to solve, and “A” was the wrong market problem? Even if everything (else) is working perfectly – you document requirements for “A”, the engineering team creates a marvelous “A” and it launches without implementation errors – you still fail, and incur the 1,000x cost of a failed product launch.

There is an even larger opportunity in front of your product team – a 1,000x payback on discovering and choosing to solve the right problems for your customers and markets.

• Would Palm still be independent if the Pre had solved a compelling problem?
• Why did Intuit have to buy Mint.com – could they have embraced the same customers with Quicken?
• What is Garmin going to do now that “free” GPS mapping and turn-by-turn directions are becoming ubiquitous? If it is “more of the same,” how much are they wasting?

Wrapping Up

I’m not aware of any studies that show that “requirements bugs” fit the same 1/10/100/1000 cost explosion model that “implementation bugs” exhibit. Emotionally, it “feels about right” to me – it passes my “sniff test.”

There are times when days of research would have been required to avoid or redirect a few hours of implementation effort on projects I’ve worked on. And I’ve seen man-years invested solving problems that didn’t involve much more research.

My intuition from products and teams I’ve worked with is that it probably averages out somewhere around 10x.

What does your gut (or your data – if you have some, post a link below!) tell you?

Most companies ignore their markets – and they will struggle to survive.  Some companies listen to their markets – and they have an opportunity to succeed.  You have the opportunity to understand your market, and transform it into your market – but you can’t get there just by listening.

Don’t listen to your market, understand your market.

Zappos All-Hands Meeting

Thanks Thomas Knoll (@thomasknoll) for pointing out that Zappos’ all-hands meeting was being live streamed today!  [Aside - how cool is that?!]

Chip Conley (@chipconley) spoke to the Zappos (and Bellagio) folks about several things, including being customer-driven.  He was able to pull several ideas from his book, Peak: How Great Companies Get Their Mojo from Maslow, but one of them stuck out in particular as being critically important to product managers.

Transformation Pyramid

There’s a copy of this on slide 28 of this presentation on Slideshare, essentially an adaptation of Maslow’s hierarchy of needs, applied to customer relationships.  It is not clear if this is an approved (by Mr. Conley) posting, so I’m not going to include a fair-rights copy of the image.  You can find out more by reading his book, visiting his website, or dive right into an application of this model to Wall Street from Mr. Conley’s blog.

The basic idea (for focusing on your customers) boils down to this:

1. At the base of the pyramid, you have companies that meet the expectations of their customers- they provide what customers expect (and no more).  In his presentation today, Mr. Conley pointed out that these companies will struggle for survival.  Good enough isn’t anymore.
2. At the next level of enlightenment are companies who meet the desires of their customers.  These are companies Mr. Conley points out as ones who will succeed.
3. At the top of the pyramid are companies who meet the unrecognized needs of their customers.  These companies, through understanding their customers, satisfy needs that people didn’t know they had.

In an earlier article on sustaining a market-driven competitive advantage, I wrote about listening to your customers (#2) and innovating in order to uncover the latent requirements.

You can argue that these problems were introduced because of innovative solutions, or that the problems were latent, and were only discovered because of the innovations.  It doesn’t matter.  What matters is that these problems were previously unaddressable, and now solutions to these problems have value.

That article was written with an inside-out perspective – essentially answering two questions -

• “Why should you innovate?”

• “Why Should You Innovate Repeatedly?”

As you repeatedly invest in understanding your customers, you gain insight into their latent requirements and needs – and get an opportunity to address them.  When you do this repeatedly, you become a thought leader for your markets, which you can leverage as a distinct competitive advantage.

Conclusion

Mr. Conley is approaching innovation, with respect to customers, from an outside-in perspective – and reaching a similar conclusion.

When you understand your customers, well beyond meeting their must-have minimum expectations, and beyond addressing their explicitly stated desires, you have the opportunity to solve the problems they don’t realize that they had in the first place.

From my perspective, his insights make it even more important that you continually focus on what your customers really need, and let your market guide you.  Not to be confused with “give your customers what they ask for” – they’ll only ask for a faster horse.

“For what one idea do you want your product to stand in the mind of your customer?”  I heard Roger Cauvin ask that question at the most recent ProductCamp Austin [correction - he said it here - thanks Roger], and the quote has been jumping to the front of my mind almost daily ever since.  Maybe by writing about it I can exorcise the demon and get back to using the idea instead of being haunted by it.

The One Idea

You’ve just been given a new assignment – your company needs a new software thingamajig so that you can play in the whatchamacallit space, and you’re going to drive product management for it.  You’ve also been asked to deliver the first version in six weeks.  Of course you’ll be able to do follow-on releases, after all, you are agile.  That’s how “agile” works, isn’t it?

Cool.  Exciting.  Challenging.

You sit down with the folks who will be building the product, and find out that they have already spoken with several stakeholders.  Excellent. You circle back with the stakeholders, and start to gather data about your market and audience.

Your schedule is aggressive, so you think about what is realistic to get done in a very small timebox.  With the current schedule, you can’t afford to get caught in analysis paralysis, and you can’t realistically do in depth market analysis, persona development, hyper-accurate requirements prioritization, etc.  You do, however, have time to do the most important thing – define the one idea that will define your product.

If you cannot come up with this idea, you need to push back on your management team, and delay the launch of your product until you have that one idea.

Don’t Abuse Agile

Agile is designed to help you rapidly create iteratively better products, with iterative development and continuous infusion of market feedback and data.  Agile is not a process by which you start typing without any idea of what you intend, releasing it and then getting feedback in an iterative process.  If that’s how you’re approaching agile, your process is broken.

[image from Market Driven Competitive Advantage]

Unless you’re very lucky, your first iteration will be a waste of time and money.  Wouldn’t you rather be intentional than lucky? Make sure you have that one idea before you start developing.

Strategic Alignment

You need to come up with a one idea that is aligned with and supports your corporate strategy.  At a minimum, you have to make sure that you have an idea that is aligned with your stakeholder’s goals, based on the assumption that those goals are aligned with corporate strategy.

Wow

Your one idea really needs to make your users say “Wow!”

I was explaining the importance of this to a colleague (who is not a product manager), and his response was “Oh, you mean the hook.”

I was reminded that you not only have to provide a wow experience for your users, but you have to present a hook that will capture the imagination of your buyers.  Remember that buyer personas make decisions based on their perceptions of user problems [more on convincing buyers].

Satisfice

Remember that you don’t want to try and do everything perfectly in the first release – you want to satisfice.  You may only be able to target a single persona with a solution to a single problem.  You just need to make sure you are solving the right problem – which is where Kano analysis is very handy for identifying the Minimum Market Acceptance criteria.

[From Minimum Market Acceptance]

Conclusion: A Balancing Act

Defining the one idea for your product is a balancing act.

• You have to align the one idea with your corporate strategy and stakeholder goals
• You have to solve a valuable problem for your users, presented in a way that captivates your buyers
• You have to create a product that is compelling right away – good enough in execution to enter your market
• And you have to get the first release out in six weeks!

Good luck!  Or better yet – Good Intent!

Consistency in writing requirements is important on two levels – strategic and tactical.  Tactically, you need to write your requirements with grammatical consistency, so that potentially ambiguous statements will be interpreted similarly.  You also need to write requirements that are logically consistent, so that you avoid “impossible” requirements and gaps of unspecified meaning.   Strategically, your requirements need to reflect a focus on markets and problems that are consistent with your business objectives and the vision your company is manifesting

Consistent Requirements – Revisiting

Thirteen hundred ninety-six days ago (grin), I wrote about the importance of grammatical and logical consistency in writing requirements – an emphasis on the tactical aspects of consistency in writing requirements.  This article adds a brief discussion on the consistency of the requirements you write.  This article is the sixth update of a series of articles on the rules of writing requirements.

Grammatical Consistency

[source]

Writing requirements is not like other forms of writing in one key way – you don’t want to vary the terms and language you use in your requirements documents.  Remember the critiques you received when you studied writing in school for a moment.  Your teacher told you not to repeatedly use the same word, but to use synonyms for that word – walked, ambled, stepped, capered, etc.  Those synonyms allowed you to create imagery and helped your readers avoid boredom.  Your teacher encouraged you to use metaphors that created even stronger imagery – flew, slogged, muddled, danced, etc.  Those metaphors allowed you to infuse emotion into your narrative and helped your writing come to life for your readers.

As Yoda said, “Unlearn, you must!”

Requirements are not a narrative.  Your primary goal is not entertainment, it is enlightenment.  Don’t use varying terms to describe the same action or object – use the same terms over and over again.  Don’t vary the modifiers of terms except when you are explicitly varying the context and referenced items – and then make certain that you use the same modifiers consistently for the same context and references.  When dealing with anything that has some complexity, provide a figure or diagram (possibly in an appendix or glossary) that explains the explicit meanings of each term and term-modifier.

For example, if you are defining requirements for a classification system for authored works, be specific about the definitions of author, editor, publisher, and reviewer.  You may also need to define a term that is used collectively for all of them, like creators.

You must consistently use the same voice  - ideally the active voice.  And of course, you must not use ‘the system shall.’ [There's a great debate in the comments on that article - one thing to note - all of the people on both sides of the debate emphatically expressed their consistent use of one form or the other.]

Logical Consistency

[source]

Logical inconsistency most commonly comes from expressing the same requirement twice.  If you use the exact same words to express a requirement twice, you’re being redundant and repetitive (grin #2).  Most likely, the second expression of the same requirement is both unintended, and worded differently – running the risk that it will be interpreted differently, and therefore inconsistently.

You will also create logically inconsistent requirements when you express mutually exclusive requirements.  These can be directly incompatible, like writing “the system must infer the value…” and “the user must provide the value…”   Or they can be indirectly inconsistent like “the animal must bark” and “the animal must be a cat” – which is logically inconsistent because cats cannot bark (thereby making at least one of the requirements infeasible).

Strategic Consistency

One of the elements that defines valuable requirements is that they “support your business strategy.”  In the context of value, that means you have created a decomposition model for how you intend to realize value – manifesting a strategy or a component of a strategy.

A strategy, however is not always defined by a single measure.  Your strategy could include financial metrics and targets, as well as a stated mission regarding customer service levels, and a long-term investment that is hoped to eventually return long-term rewards.  Your requirements, while not only completely defining the mechanisms for achieving value, must also be consistent with the other elements of your strategy.

For example, you should not take advantage of uninformed costumers when you have a core value of integrity – regardless of how effective that may be at meeting financial goals.  The requirements that articulated the (subordinate) goal of grifting your customers (to maximize revenue) would be inconsistent with the goal of operating with integrity.

Conclusion

Consistent Requirements are requirements that

• Are not in conflict with any of your strategic objectives, vision, or corporate goals
• Avoid logical inconsistencies
• Utilize consistent structure, terms, and grammar

April Dunford just presented Startup Marketing 101 at DemoCamp Toronto.  Great ideas from the ‘marketing and your startup’ point of view.  I’ve often said that product managers and product marketers care about much of the same market data, they just do different things with it.  The idea of minimal feature set came up in April’s presentation – this article talks about product management, agile, and initial market acceptance.

Initial Market Acceptance and Minimum Market Acceptance

April mentions (in slide 4) that an important event to the timing of marketing activities is achieving the “_minimal feature set (for certain markets)_.”  April organizes those marketing investments in terms of three stages of “product marketing lifecycle”:

1. What to do when your startup has a “pre-product” product.
2. Where to focus when you’re focused on early adopters.
3. How to invest in your marketing programs when your focus is on scale / growth.

Coming from an agile background, I think about two distinct notions of “minimum” or “initial” when it comes products and markets.  The following definitions are in a B2B context, as I’ve been focusing on this recently with a client in the B2B space.  The same ideas are relevant to B2C and B2B2C companies, but the language would be slightly different.

I’m assuming that when April used the phrase “minimal feature set”, she was really talking about “minimum market acceptance” in the way that I’m describing here.  The distinction is that features describe what a product does, which is an inside-out view of the product.  To be market focused, you have to think about which market problems are being solved, for whom they are solved, and how well they are being solved – an outside-in product view.

If you focus on, and organize around your features, you are likely to miss the mark.  You must focus on the problems that your customers need to solve.  In your product, you will prioritize the development of capabilities (embodied through features) that are designed to help your customers (buyers and users) solve their problems.  Your market analysis should be geared around understanding how many customers in each segment face each problem, and how much they are willing to pay to solve those problems.

Agile or Waterfall or Waterfragile?

As an agile product manager, and former developer, I know that when you do “everything we need to be successful” in the first release, you’re not being agile – we’re being waterfall.  A slight extension of this is “everything we need to be minimally successful” – and that is still waterfall.

An agile team should focus on the first release addressing the initial market acceptance criteria – the minimal set for a single customer.  The goal is to get the product in front of customers as soon as possible – this starts your revenue stream earlier, gives you valuable market feedback, and gives you the opportunity to establish momentum through repeated incremental releases of your product.  Each release will solve addressed problems “better” and / or address additional problems, until you reach minimum market acceptance.

Kano Analysis for Understanding Problems

Last year, I was thrilled to share my approach for applying Kano Analysis as a product manager with the PMV webinar audience as well as the Austin IIBA chapter.  One of the key ideas in the application of Kano Analysis to understanding your market is developing the personas within that market, and understanding how each of them treats each problem / capability.

[slideshare presentation & PMV webinar]

This analysis, in addition to being useful for understanding the market (or a segment) as a whole, also helps you identify your first customer.  Once you know who your first customer is, you can determine the initial market acceptance criteria for that customer, and that determines the must have capabilities.

Note that finding the initial solution (that the first customer will buy) does not mean releasing a poor quality product – it just means releasing a product that solves enough problems (well enough) for one customer.  With the feedback from that customer, you can drive the prioritization for your next iteration – making the next iteration better for that customer (which can really help your word of mouth), and gaining more customers.  Repeat this process until you get to minimum market acceptance.

Timing Marketing Investments for Minimum Market Acceptance

[slide 9 from April's presentation on slideshare]

Since I wasn’t at April’s presentation, I don’t know exactly what she would think of the following, but I believe it makes sense:

• Initial Market Scceptance (one customer would buy your product) marks the transition from pre-product to early adopters.
• Minimum Market Acceptance (enough to succeed in the market) happens before the transition from early adopters to scale.  Note that it may not (probably doesn’t) mark the transition, but I suspect happens mid-stream.

Would love to see what other folks think about mapping this product management centric view to the marketing timeline.  Please chime in below.

ProductCamp Austin is here again!  The Spring 2010 session is this Saturday, 27 March 2010 at the AT&T Conference Center on the UT campus in downtown Austin.  Make sure and say hi when you’re there!

ProductCamp

ProductCamp is an unconference for product managers and product marketing professionals.  This is the fourth time Austin has had one – we were fast followers of the wildly successful Silicon Valley PCamp in 2008, which Rich Mironov was instrumental in getting off the ground and growing.  Paul Young is the guy who brought together the people in Austin to get our events off the ground.  Paul graciously deflects the credit for getting things going, but he deserves it regardless.

Check out Paul’s great description of ProductCamp. Here’s the key paragraph, if you’re on the fence about attending:

The sessions and networking are the main reason that people keep coming back. Over three ProductCamps, we’ve achieved over 98% of participants answering “yes” to the questions “would you come back to ProductCamp” and “would you recommend ProductCamp to a peer?” This high customer satisfaction is why our local community loves to come out – in droves. Last time, we had over 500 sign up and over 300 participate. This time we are limited by our venue and won’t be able to hold more than 300 or so, so register today and reserve your space. It’s free of course.

Sign Up Now

Today (Thursday) is the last chance to sign up, to give folks a few minutes to finalize planning for space, make name tags, etc.

You can sign up at the official website for PCamp Austin.  The doors open at 8:30, things get started at 9:00, sessions are from 10-4 (with provided lunch at noon), then awards and a happy hour at 4:30.

View Larger Map

Sessions

41 Sessions have been proposed for this ProductCamp, so make sure and come, vote for your favorites, then enjoy them!

Leaf Launch

Very exciting news for this PCamp – Paul Young is launching his new product, Leaf!

From Paul: “Information overload is a problem that every product manager has.  Between email, phone, IM, tweets, status updates, geolocations, and keeping track of who knows what, just communicating can feel like a full-time job.  Social networking has amplified the problem – we now have access to more information than ever before – but how’s that working out for us?”

Does that sound like you?  You should attend his session, then.  I certainly will.

See you there!

The Laudi Group and Red Canary organized and shared a great set of questions for product managers and answers from a panel of product management leaders.  Steve Johnson, another leader in our space shared his answers to the same questions, and in this article, I share mine.

The Product Manager Questions

Hat tip to Steve Johnson at Pragmatic Marketing, for extending the discussion and providing his answers to the questions.  And thanks to the folks at Red Canary and the product managers who originally shared their answers.

I really enjoyed reading their answers – thanks to all of you!

Here are the questions that were put forth and answered:

1. Tell us about the best product you’ve ever encountered.  Why do you like it?
2. How do you know a great product manager when you meet one?
3. What’s your favorite interview question?
4. When is the best time for a start-up to hire a product manager?
5. What has been the defining moment in your career?
6. Mistakes.  What was your biggest?

I’ve personally enjoyed and grown from the great writing that many product managers have shared with us over the last few years.  I’d love it if they would share their answers.  So, either share your answers, or pester your favorite writers to share theirs.

Without further ado, here are my answers to the same questions.

Why Do You Like the Best Product You’ve Ever Encountered?

I love when products solve obvious (in hindsight) problems with elegant designs.  The products where, once they exist, you say “well, duh” or slap your head and ask “why didn’t I think of that?”  These innovative products tend to be disruptive, redefining markets.  Some of the products are rather mundane – like the ketchup bottle where the lid doubles as the base (reducing waste and preventing countless red-water-stained buns).  Other products are more technology-dependent – like Tesla’s radio, xerographic photocopying, solid-fuel rockets.

I think my favorite for elegant design might be Velcro – if you don’t know how it works, or just want to know how it was invented, check out the wikipedia article.  The time that parents save tying tiny shoes is value enough, but it is far from the only (or first) use of Velcro.  An accurate clock that could be used on board a ship was a biggie too, although I never personally encountered one, at least when it mattered.  Accurate time-telling on a ship was the key to dramatically improved navigation back in the days of sailing by the stars.

How Do You Know a Great Product Manager?

Great product managers are polymaths, with several areas of deep expertise and skill.  They are Renaissance women and men, with many areas of interest.  They are great communicators.

Most importantly, they are sooth-sayers.  By the last, I mean that they not only understand the big picture and context of their markets and their business, but they know what is likely to change their business, and where their markets are sensitive to or ripe for disruption.  One definition of sooth-sayer is “one who tells the truth.”  You can’t do that without data, and the ability to understand what the data is telling you.

Add to this a dash of humility and a full dose of open-mindedness, and you have a great product manager.

Of course there’s all of the esoteric skills that the role requires.  When they aren’t present yet, I feel like I’ve met a great product manager to be.

I know it when our conversation rambles all over, goes “depth charge deep” in areas, and then bounces back to the broad view, all with an eye for the relevance and insights that matter for the topic at hand.

What’s  Your Favorite Interview Question?

“Why?”

I almost don’t care what the context of that question is – reviewing a candidate’s previous experience, asking them to provide a “fresh view” on my current situation, or convincing them to dance through a hypothetical situation.  What I want to know is why they think there is value, or a problem , or an opportunity, or … whatever.  A collection of well-dispersed, and sometimes-immediately-sequential “Why?” questions can tell me more about how someone thinks, and more importantly, how they are likely to solve problems, create solutions, and dominate markets than any other question I’ve found.

When is the Best Time for a Start-Up to Hire a Product Manager?

Not counting the founder?

Three to six months before the first product peaks.  All successful start-ups have one good product – solving a single valuable problem, for a single market.  Most (my opinion / observation) start-ups don’t have a second good product or market.  A passionate and insightful founder can spend a long time understanding a market, a problem, and a solution.  That knowledge and passion can yield a successful product.  When is that founder, who is busy running the company, going to find a new problem to solve, a new market to dominate, or a new solution to replace his original idea?

Alternately, I guess the founder can hire someone else to run the company, and anoint herself “president of the product.”  That still counts as hiring a product manager.

What Has Been the Defining Moment in [My] Career?

Switching from electro-mechanical design engineering to software development.  The shift in innovation time scales, the different approach to problem solving, and the markedly different economics of product creation had a profound effect on me.  The evolution from “creating solutions to (defined) problems” to “identifying problems worth solving” was more gradual, as I evolved into a programmer-analyst and consultant and product manager.

“Going agile” half-way through my software development career was pretty eye opening too.  I’ll throw that in as my backup answer.

Mistakes.  What Was [My] Biggest?

From someone else’s perspective, it was leading a team down a six-figure software-development path that solved a problem no one really cared about.  That’s probably defining-moment number 3, when I started including validation of value as part of my scope of engagement as a programmer.

From my own perspective, as they say, it’s the train you don’t see that hits you.  I’ll guess that it was something I didn’t do, not something I did, that would turn out to be the key plot device in my Frank Capra movie.

I came across a really interesting article LukeW.com, showing how making changes to the way an input form on a website increased interaction by 25 to 40%. The changes reflect the value of thinking outside-in, investing in user experience, and performance measurement.

Bonus: the idea is cool.

Mad Libs Input Form

[full size image available in Luke's article]

The idea being presented is to replace the old boring web input form designed for a computer. The new, fun form is a fill-in-the-blank (aka Mad Libs) layout. The article is on Luke Wroblewski’s site.  The team at Vast.com created and tested a version of this form for the Kelly Blue Book site. [Luke cites Huffduffer.com as the first place where he saw this technique.] Thanks, Luke for sharing this with all of us!

Outside-In Development

Product managers are acutely aware of the need to solve market problems. We are regularly reminded that we’re creating solutions for people in our markets who use our products to solve their problems. When we write requirements, we avoid writing design specifications, and thereby lose the ability to enforce that the proposed solutions also take an outside-in approach. We maintain an outside-in perspective, but we lost the ability to influence an outside-in aesthetic.

Note: Outside-In Software Development is a great book, you should read it.

That’s one area where user experience works well in concert with product management – assuring that the same drivers of what to build are informing the design of how it is built.

The genius (or at least elegance) of this Mad Libs form from Vast / Kelly Blue Book is that it humanizes the experience of requesting that a nearby dealer contact you to discuss a possible purchase of a car from them. The forms we’ve all had to fill out on countless web sites are very inside-out. They expose the inner workings of the program – “gather this info, and that info, and this other info.” Those forms force users to do what the program wants. Blech. The form that Ron Kurti designed, however, forces the program to do what the users want. Huzzah!

Bad Requirements Prevent Elegance

I’ve written about the importance of writing design-free requirements several times.  Avoiding design constraints in your requirements is even one of the pillars of the big 10 rules of writing requirements.

If you had written the requirements for this contact the dealer page, would you have specified the design of the form?

Most of the requirements analysts I’ve worked with would have. I’ve even worked with companies that have developed templates / stencils defining how these interface specifications must be documented – requiring that all the fields be aligned, with specific pixel gaps, etc.

If you had specified the design (read: limited the choices of the designer), you would have prevented this solution.

Mr. Kurti’s elegant solution is clearly better.

One way to write the requirements that would not have prevented this solution would be with a user story and acceptance criteria:

As a car-shopper, I want to engage a car dealer, once every few years, so that I can purchase the car I selected.

• The car-shopper can provide his contact info (name, email, phone) for the car dealer.
• The system will display the information identifying the selected vehicle.
• The system will display the information identifying the dealer being contacted.
• The system will include an opt-in newsletter-subscription option.
• The system will notify the specified dealer when the car-shopper indicates.

The user story nudges the developers into an outside-in perspective by emphasizing the user’s goal. The acceptance criteria make it clear that any solution that meets those criteria is acceptable to the product manager. It allows the interaction designer to create a compelling solution, rather than forcing him to recreate a boring experience.

Measurement Rocks

The Vast team measured the impact of making this change to their form, and saw between a 25% and 40% increase in conversion (people contacting dealers versus people abandoning the process when they see the form). That’s a lot more leads for the dealers, and presumably a lot more money for Kelly Blue Book and Vast. This is a great example of how you can measure the impact of investing in user experience, because it should spark ideas for you.

What can you measure and test and improve?

Now that’s ROI for you.

You give your requirements to the engineering team, and they look complete.  The team builds your product, you launch it and the market soundly rejects it.  Why?  Because your requirements weren’t complete – they didn’t actually solve the problem that needed to be solved.

Complete Requirements – Revisiting

Going back almost four years, I wrote Writing Complete Requirements, as part of the Big Rules of Writing Requirements series.  That article centered on two key ideas of requirements completeness.

• Data, not opinion, is the most important input into determining completeness.
• There is no absolute way to determine the completeness of your requirements in advance.

Completeness is best measured by market acceptance, so technically you can’t know if your requirements were complete until your customers buy (or fail to buy) your product.  This doesn’t mean you should give up on this rule of requirements – you can still focus on, and improve, the completeness of your requirements.

Objective and Heuristic Assessment

Given a market problem and the associated requirements (which you can argue are actually the same thing), you can review those requirements to determine if they objectively appear to be complete.  This assessment is based on what you objectively know about your market (needs, opportunity costs, competitive alternatives, etc).  You can also apply heuristic, or logical analysis to identify gaps in the language of your requirements.

Objective assessment is the application of market knowledge (data) to determine if the nature of the problem is being addressed completely by your requirements.  A bird-feeding maven would know that the requirements for your bird feeder would be incomplete if they didn’t address the problem of squirrels stealing food from the birds.

Heuristic analysis is the identifications of logical omissions in your requirements, without the need to apply market insights.  A logician would know that the requirements for your bird feeder were incomplete if they didn’t address the need to refill an empty feeder.

When you’re writing requirements, you need to do both.  Heuristic analysis of your requirements will identify where your requirements do not fully solve the problems you already know about – an eCommerce checkout process without the ability to receive payment, for example.  Objective assessment will help you identify the problems that you overlooked but need to solve – the need to allow customers to have different billing and shipping addresses when placing an order online, for example.

Complete Requirements for Incomplete Solutions

Don’t confuse having complete requirements (a good thing) with completely solving a problem (not always a good thing).  In agile development, the idea of satisficing is key to scoping individual iterations.  The idea, simply put, is to solve enough of the problem, and not delay delivery (and value) until you can build a solution to the entire problem.  Because of the law of diminishing returns, for many market problems, there is little or no incremental value in solving the entire problem.  Those resources can be better applied to solving additional problems.

Kano analysis provides a framework for identifying which market problems must be completely solved, and which ones have a more is better characteristic.  In practice, more is better problems exhibit the law of diminishing returns, and should not be “completely” solved.

[larger image, or view the entire Kano analysis presentation]

Conclusion

Complete Requirements are requirements that

• Identify all of the market problems that need to be solved and their nature (absolute vs. diminishing returns).
• Are logically complete in their coverage / articulation of those market needs.

To objectively improve the completeness of your requirements, apply market data in an assessment of your requirements.  To heuristically improve the completeness of your requirements, look for logical inconsistencies and gaps in reasoning or coverage.

Engagement – that’s what this whole product management blogging thing is about.  Check out what Tyner Blain readers found to be the most engaging articles in 2009.

Deep Dives

If you’re new to Tyner Blain, you may be surprised by the length of the articles here.  I joke that they are long because I don’t have time to edit.  Stewart Rogers jokes that they are long because I’m incapable of writing a short article.  If you’ve been here a while, you know what you’re in for.  If you’ve been here a long while, then you’re glad (like me) that I don’t write one per day any more.

Product Management is simultaneously a broad and deep discipline, requiring us to have a breadth of perspective combined with a depth of insight.  We then have to apply that in a market context, effectively navigating the political waters of our organizations.  Most of the articles here either try to skim the breadth of a range of related topics, or plumb the depths of a single topic.  Doing that in under a thousand words is pretty hard.  Most of the articles here also link to other articles, to try and provide even more depth and context, and encourage additional critical thinking.

One measure of the quality of the articles here is how often they stimulate readers to read further, or dive deeper into the topics.  While web analytics won’t allow us to measure how thought-provoking an article is, we can look to see how many people dive into the linked articles, versus how many people move on to something else.  We can measure the bounce rate of an article to see how often people leave a page without following any of the “tell me more” links.

Looking at ~170,000 page views at Tyner Blain in 2009, narrowed down to only those articles with at least 100 page views, here is the top-ten list of most engaging (or “least abandoned” if you’re a “cup is half empty” person) articles:

1. Use Case Series: Introduction: A collection of articles on the “traditional” forms of use cases – informal, formal, UML.
2. Agile Development of Use Cases: A dive into the dynamics and cadence of an agile process for developing use cases.
3. How Do You Manage Market Data?: A collection of articles exploring different market-immersion techniques in depth.
4. Scheduling Requirements Changes – Part 2: A focus on practical techniques for managing change with an agile development process.
5. Yesterday’s BPMN Post Was A Big Fat Lie: A mea culpa and clarification of interest to folks following the BPMN Series.
6. Everything I Needed To Know I Forgot in Kindergarten: Why asking why? is important.
7. Plan Your Next Sprint by ROI – Part 1: Prioritizing by Bang for the Buck, not just Bang.
8. Ten Common Mistakes of Going Agile: A collection of articles about common pitfalls encountered when adopting agile practices.
9. Perpetually Almost Finished Projects: Organizing a project with discrete deliverables and rolling-wave planning to avoid the “90% done, 90% remaining” problem.
10. How To Use Timeboxes for Scheduling Software Delivery: One of my personal favorites.  A rational approach to making trade-offs when your “perfect” plan has to change.

OK Stewart, that’s only 519 words.

Why does cross-selling, the process of selling something additional to someone who is already making a purchase, work?  This article explores some of the theory and rationale behind cross-selling – from qualification to motivation and profitability.

Cross-Selling is Big Business

You have an eCommerce site where people purchase products from you.  Adding cross-sale capabilities to your site can have a significant impact on your bottom line.

The e-tailing group’s 2009 report shows (by survey of eCommerce executives) that 55% of online retailers will include cross-sell and upsell capabilities in their sites in 2009 (already there or being added).  For retailers that measure the data, cross-sale promotions result in a range of conversions (additional sales) from under 1% to over 10% – with a plurality of responses in the 1% to 4% range.  That represents additional revenue the retailers would not get without using cross-selling.

According to the Get Elastic blog, Amazon reported that cross-selling accounted for 35% of their 2006 revenue.

Understanding why cross-selling works requires understanding customer’s purchasing processes work.

Customer Purchase Process

The following is a model for how customers make purchases.

• Customers start by thinking about their problem – what problem are they trying to solve (with a purchase)?
• Some people will try and understand the problem space, while others are happy to jump to solutions.
• Understanding the solution space (what are my options?) is next – and may be a shallow or deep analysis.
• Within the solution space, customers will select a product and decide to buy it (from you) or not.
• Customers who reject their first product choice may select another product or may abandon your store.

You can take a higher-level view of these customer activities and decisions, and categorize them into areas:

• Learning - Your customer is learning about her problem and possible solutions.
• Choosing - Your customer is comparing possible solutions, with the intent to purchase one of them.
• Buying - Your customer has selected a solution, and is trying to buy it.

Customer Qualification

In direct sales, one of the first steps a sales rep will take is to qualify a prospective customer – how likely is this prospect to make a purchase?  A similar model can be applied, when treating your website as a sales rep, to understand how likely it is that a visitor to your website will make a purchase.  There is a ton of additional qualification (assessing a prospect’s ability to pay, for example) that we won’t talk about in this article.  In this article, we’ll focus just on this simplified purchasing model:

• Customers who are learning are more likely to buy than visitors who are browsing (window shopping).
• Customers who are choosing are more likely to buy than customers who are learning.
• Customers who are buying are more likely to buy than customers who are choosing.

That last bullet seems silly – of course customers who are buying are more likely to buy – like 100% likely, right?

Wrong.

Prospective customers abandon the buying process all of the time.  SeeWhy reported ~70% abandonment rates during 2009’s post-Thanksgiving online shopping spree.  They also used the phrase “a relatively healthy 63%” to describe abandonment rates in August 2009.  SeeWhy is specifically measuring abandonment of the shopping cart (inside the “buying” area), but there is abandonment (often called leakage) throughout the process above.

The further a customer is into the purchase process, the more likely they are to actually make that purchase.

Clearing Hurdles

As a customer moves through the purchase process, they are clearing hurdles that would prevent them from making the purchase.  Each time they clear a hurdle, the pending “mental cost” of making the purchase gets smaller.

One of the hurdles is making a decision to purchase now, another is making the decision to purchase from you.

Offering cross-sale promotions to customers who have already (probably) decided to purchase from you, now, means that you’re offering the promotions to customers who are qualified.

Relevance

A defining element of a cross-sale is relevance.  You’ve got a customer who is purchasing a product, to solve her problem.  You want to increase the value of the transaction – both for your customer and for yourself.  To do that, you have to offer a relevant cross-sale item.  Selling french fries with a sandwich makes sense.  Selling car wax with a new car makes sense.

Selling car wax with a sandwich?  You probably won’t have a lot of success with that.

How do you determine relevance?  You can start out with an “expert opinion” – car wax is relevant to cars, for example.  Or you can do some data mining of past orders – “7% of people who bought cars also bought car wax.”  That lets you start out with a hypothesis (that car wax is a relevant cross-sell item for purchasers of cars).  If you’re data mining past orders, however, you may not know the direction of the cross-sell.  Cross-selling works because the two products are complementary goods – usually asymmetric complements.

Complementary goods are rarely symmetrical.  Peanut butter and jelly is a good example of symmetric complements – they have comparable price points, and both generally can be improved by purchase of the other.  In the mixer-cookbook example above, the cookbook is a great complement product for the mixer.

However, if your customer had selected the book initially, the encouragement to “add a stand mixer” would fall on deaf ears.

Foundation Series: Substitutes and Complements

You can measure behavior on your site to determine the nature of the complementary goods relationship.  For the orders that include two particular products (that are offered as cross-sale promotions for each other), in what percentage of orders with both products was each product selected first?  Alternately, what is the conversion % of product A when added to a transaction for product B, versus the conversion % for product B when added to a transaction for product A.

Measurement of Cross Selling

The obvious metric that most people think of when evaluating cross-sale promotion effectiveness is conversion rate.  Conversion rate identifies the percentage of people who, when buying the “original” product, choose to also buy the “promoted” product.  Making changes that raise your conversion rate increase your sales of the promoted product.  However, this conversion ratio is more a reflection of the relevance of the promoted product to the original product than it is a measure of profitability.

Your goal is to maximize profitability, while providing additional value to customers (who are better off or more satisfied with the original purchase when they also purchase the promoted item).

Introducing a cross-sale promotion can increase, decrease, or have no effect on the rate of purchase (conversion percentage) of the original product.  You need to measure the original-product conversion rate for customers who were shown the cross-selling promotion versus those that were not.

Second, you’ll want to know what the ideal products to promote are.  Typically, more than one cross-sale promotion is presented to a customer at a time.  For this example, assume 3 promotions are displayed.  Also assume that your data-mining exercise has identified 5 products that could be promoted as cross-selling items for this “original” product.  Which three of the five do you select?

You should select the three that you expect to be the most profitable.

“Most profitable” can be calculated as (profit per promoted item) x (conversion % – of the promoted item in the context of the original item).  When you don’t have conversion percentage data, you can either gather it (through testing) or predict it (through modeling).  There are many aproaches for predicting the degree of affinity, or implied relevance, of one product to another – but all of them are too detailed to cover in a blog article.  When you don’t have a model, you can test the possibilities to see which ones perform the best.

Some companies also report average order value (AOV), but that’s not necessarily an indicator of profitability.  It may be a component of profitability, but not necessarily.

Product Managing Cross-Selling

If you’re product managing your website, and exploring adding cross-selling capabilities, or enhancing current capabilities, here are some things to keep in mind:

• Your customers are in the middle of a buying process, and will be interested only in complementary products that would give them a better purchase – more value, even if it means more money.  This drives both the importance of relevance and value as criteria for selection of products to promote.
• There may not be any one person who is responsible for (rewarded for) the profitability of your cross-selling capabilities, as the complementary products being promoted may be “owned” by different parts of your organization.
• You will want to test the effectiveness of the approach you use for promoting particular products – which original products, promoted products, and unique combinations of the two are the most profitable?
• You will want to be able to test the effectiveness of changes in the presentation of promotions (images, page location, text, etc) on profitability (or on conversion percentage as an isolated variable.
• You may want to offer discounts that apply only in the context of the cross-sale promotion (e.g. “Buy these together and save!”) and measure the impact on profitability – do the added sales offset the reduced profit per sale?
• You may find that a given complementary product has a different degree of relevance (and therefore effectiveness) depending on the market segment to which you are promoting it.  As an example, a game controller may be more relevant to consumers buying a large computer monitor than small business owners buying the same monitor.

Conclusion

Cross-selling works when you promote a relevant product that is complementary to the original product.  It works because the prospective customer is already in the process of purchasing the original product, and is therefore already qualified.  You should only promote products where the additional sale increases value to your customer and increases value to you.

Ultimately, cross-sale is about profitability.

Do you know about substitute goods and complementary goods?  If you’re doing any eCommerce, and are thinking about cross-sell and upsell, you should understand the basics about substitutes and complements.

Substitutes

You’re considering purchasing a specific product (e.g. a blank compact disc from Memorex).  You could consider purchasing an alternate product (e.g. a blank compact disc from TDK), substituting the alternative for the original.  At a high level, that’s the definition of a substitute good.  Any product that could be substituted for another product, and still satisfy the needs that the original product was intended to address.

Complements

You’re purchasing one product (e.g. a slice of bread with peanut butter).  The value you get from the original product would be increased by the purchase of a complementary product (e.g. a slice of bread spread with jelly).  The basic definition of complementary goods is “products that are purchased together.”

Economic Models: Substitutes and Complements

The definitions for substitute products and complementary products come from the world of micro-economics.  Substitutes and complements are used to model the interdependent nature of the changes of prices on the supply and demand of “related” products.

You can imagine a junior economist who tried a pricing experiment to determine the price elasticity of demand of Jif brand peanut butter.  He predicted that by raising prices on the peanut butter, that the store would sell less peanut butter.  Sure enough, demand (at the higher price) was lower, and sales dropped.

However, he also discovered that sales of Skippy brand peanut butter went up, almost by exactly the amount that Jif sales dropped.  This is because Jif and Skippy peanut butter are substitute products (economists call them “goods”).

Later on, this economist tried another experiment.  He raised the prices on both Jif and Skippy at the same time.  This time, the store sold less peanut butter in total (there were no other brands), and the economist thought his experiment was concluded.

Another surprise for the economist – sales of jelly dropped too.  Sales of peanut butter and of jelly are tied together – when you sell more of one, you sell more of the other.  Economists, trying to be difficult, would say that when you raise the price of one, the demand for the other falls.  Technically true, but a little confusing.

Thus entered substitute and complementary products into the world of economics and pricing.

Substitutes and Upselling

You know that you’re upselling – trying to replace one product that is about to be selected with an alternative product – when your customer is considering substitute products.  Notice in the screenshot from amazon.com (above) that ~9% of the people who were otherwise going to buy the original product instead were convinced to buy a more expensive substitute product.

The goal in upselling is to create a win-win situation.  Your customer gets more value from the substitute product, and you get more profit from the sale of the substitute than you would have received from the original.  Everyone wins.

Complements and Cross-selling

The number one mistake I see when people talk about cross-selling is they call it “upselling [sic].”  Its enough to make me Cranky. The example above, also from amazon.com, shows an encouragement, when purchasing the first book (in the series), to also purchase the second and third books. What’s especially clever is that there is no discount – the total price is the same as the sum of the prices if purchased individually.

You can take advantage of the complementary product model to create product bundles, identifying products that should be sold together.

Although that may not be the best idea.

Often, you will see bundles created by combining products that do not go well together – products that are not complements.  This is a sneaky way for companies to sell products that otherwise would not sell as well.

The recording industry made money in the 1950s and 1960s selling singles.  The recording industry then made a lot more money selling albums that “bundled” 8 or 9 mediocre songs with one or two hits when compact discs hit the market.  Digital downloads have re-introduced the popular purchasing of singles, and revenues are declining – not because of piracy, but because a “take advantage of our customers” bundling practice is now broken.

Complementary goods should be used to create bundles that increase value for your customers.

A baking cookbook is a good complementary product for a customer purchasing a stand-mixer (a key appliance for baking).

Symmetric Substitutes and Asymmetric Complements in Context

Substitute products are symmetric – either product works effectively as a substitute for the other – in a specific context.

If you need a new computer to use at your desk, then a desktop computer and a laptop are symmetric substitutes.  Regardless of which one is your originally selected product, the other is a valid alternative.  If however, you are looking for a computer that you can use while travelling, the desktop is not a valid alternative for the laptop (nor would it be your first choice).

When you’re considering the travelling scenario, the products are not substitutes.  Economists will say that they are substitutes as long as they share common uses.  But economists are looking at aggregated behavior.  You have to make decisions in context – and when the context implies that products are not substitutes, they are not substitutes.

Complementary goods are rarely symmetrical.  Peanut butter and jelly is a good example of symmetric complements – they have comparable price points, and both generally can be improved by purchase of the other.  In the mixer-cookbook example above, the cookbook is a great complement product for the mixer.

However, if your customer had selected the book initially, the encouragement to “add a stand mixer” would fall on deaf ears.  Surprisingly, amazon.com still recommended adding a mixer to my book purchase.  Technically rated a “toss up” by GetElastic in their great cross-selling do’s and don’ts article, but I put it squarely in the don’t bucket (until measured and proven otherwise).

Summary

• Complementary Products – consider cross-selling an additional product.  Complements are usually asymmetrical.
• Substitute Products – an upsell is a suggestion to replace the original product with an alternative product.  Substitutes are symmetrical, but only in a context where both products address the relevant needs of your customer.

Unless you live in a world filled with unicorns and rainbows, writing realistic requirements is critical.  When you set unattainable goals, the best result you can hope for is a frustrated engineering team.  Write requirements that are attainable, and your team will surprise you with what they can achieve.

Attainable Requirements – Revisiting

A little over three years ago (ten unicorn years), I wrote Writing Attainable Requirements, as part of the Big Rules of Writing Requirements series.  In that article, we focused on the unique situations that every team faces – politics and people, and the shared challenge of implicit practicality.  Those factors are as important today as they were then.

Defining Attainable

Attainability, for requirements, is simply the answer to the question – can this be reasonably done?  Most, but not all, of the answer to that question comes from understanding if your team can build and test a solution to the problem you identify with your requirement.  Being able to create a solution requires that the problem you’re solving is tractable, and that the team creating the solution has the skills to solve it.

Being able articulate a problem and it’s solution is necessary, but insufficient – solving the problem requires that there be a feasible solution.

As the founders of the agile development movement explained in their pitch for “low overhead” process – people trump process.  I think it was Alistair Cockburn who added, “but politics trumps people.”

Finally, process is the often-overlooked element of determining attainability.  Realizing the benefits of a novel solution to an existing problem usually involves process changes – sometimes significant ones.

The next few sections go into more detail on each of these – tractability, feasibility, people, and process.

Tractability

Goldilocks ruled out the large bed because it was too large.  Eating the elephant, drinking from the fire-hose, boiling the ocean – these are all common phrases in the American vernacular, because so many people try to do too much (at one time).  There are many ways to decompose a large, intractable problem into smaller, addressable components.  Practicality requires that the problem you’re trying to solve is not too large.

If everything you do is small, however, you won’t ever accomplish anything big.  That’s where strategic product management shines.  As part of understanding your market, you identify large, valuable problems to be solved.  To solve those large problems, you have to break them up into smaller problems, and then address them individually.  You also have to make sure your team understands the big picture and the larger problem that ultimately needs to be solved.  A great way to share context while decomposing problems is with the use of Ishikawa diagrams.

And zooming in…

Maintaining context is a particularly critical component of working with agile teams, where each deliverable’s possible size is constrained further to what a team can accomplish in an iteration.

[click for larger version]

For some problems (and some teams!) this may be enough information to provide the context to develop a product backlog.  And your agile team will move forward into managing their own execution from the product backlog stage.  For larger or more complex problems (such as this example), you will need more detailed communication before diving into user stories / use cases.

The same Ishikawa diagram, with more detail, looks like the following:

[click for larger version]

Agile Product Management: Providing Context

This reduced granularity represents the smallest go-to-market atomic requirement.  Further decomposition results in smaller requirements, but they become too small, because they are no longer independently valuable.  I’m not talking about “the sum of the whole is greater than the sum of the parts” stuff here – I’m talking about “if you only get X, without Y and Z, it has no value” stuff.

Feasibility

Even after decomposing market problems into addressable chunks, you still have to make sure that there are feasible solutions.  One of the brighter software developers I worked with back in the day used to say “We’re writing software – it’s all ones and zeros – we can do anything.”  He also followed that up with “…but should we be doing that?”  The team I was working with at the time probably could do just about anything.  But not everything can be done with a given budget and a specific deadline.  Some things are simply infeasible.

Scrum teams measure velocity – a reflection of how much software they deliver in a given iteration, as a function of how much work they believe is involved in delivering.  This is a great “protected” measure of efficiency.  The team is protected from requests to do work that has little or no value – by measuring throughput in terms of effort, a team can get more efficient at doing more.  A requirement is infeasible, and therefore unattainable, when it exceeds what the team can accomplish given a fixed amount of capacity.  Project managers refer to this as the Iron Triangle – given scope, cost, and quality, pick any two as inputs and the third will be a variable output (absorbing the impact of uncertainty that arises during the development process).  I prefer to use timeboxes rather than that rusty old triangle, to emphasize my perspective that quality is inherent in delivery, and should not be a “variable output of uncertainty.”  Because quality is what usually bears the brunt of uncertainty.

plus  yields

When the solution to a defined problem is prohibitively expensive, it is not attainable.  The notion of cost should also be considered in the context of value.  Requirements with higher value can justify implementing solutions with higher costs.

Prohibitively expensive solutions make requirements unattainable.  Relatively expensive (relative to their value) solutions also reflect unattainable requirements.  You need to collaborate with your implementation team to understand the costs associated with any particular requirement to know if it is attainable.  This is one of the strongest arguments against throw-it-over-the-wall interactions between product managers and implementation teams.  Without that feedback about feasibility and cost of a solution, you can’t assure that the requirements you are proposing are attainable.

Regular readers of Tyner Blain will realize that this is just an alternate way of stressing the importance of considering ROI when defining requirements.

Politics

There are political realities that every organization faces.  It could be a benevolent dictator CEO, or an executive with an agenda or pet project.  It may simply be that there is value in the problems you’ve identified, but solving them is not aligned with your corporate strategy.  You may be focusing on dominating an existing market, where expansion into new markets is the company’s focus – or vice versa.  You may be focusing on top-line growth opportunities, while the company is fixated on cost-reduction in a time of economic contraction.

Each of these scenarios causes you to be fighting an uphill battle internally.  When that hill is too steep, you’re writing unattainable requirements.

Process

When deciding what to build, you also have to think about how you launch.  Launching, as Dave Daniels regularly points out on his Launch Clinic blog, is not just putting your product up on your website for sale.  You have to think organizationally about all of the other moving parts:

• Do your customers have the ability to absorb an update to your software?
• Can your customer service reps be trained in time to promote and support the new capabilities?
• Will your sales team be able to communicate a message that helps them close deals based on what you’ve just built?

Process is more than just the process of typing, compiling, and testing code.  Process includes everything needed to successfully go to market and sustain your product.  If you’re defining requirements that are too far ahead of your market they are not attainable.  If your requirements embody process changes that are too difficult or expensive for your customers or your organization to absorb, your requirements are not attainable.

Conclusion

Attainable requirements are requirements that

• Have a reasonable cost to implement,
• Are aligned with your company’s strategy and stakeholder’s agendas, and
• Result in solutions that can be consumed by your company and your customers.

Make sure you’re writing attainable requirements.

A couple weeks ago, our article on writing design-free requirements triggered some great discussion around requirements and design (also known as “reqs and specs”).  What happens when you’re dealing with a website?  There are many stakeholders, who are clear about their own goals.  Who then turns them into requirements?

A Website Requirements Scenario

Consider the following situation.  Your company has a website, and through that website you sell hats- it is an eCommerce haberdashery.  You have at least three stakeholders that are important in this scenario:

• Customers (or prospective customers) who interact with the website and purchase products from your company.
• A Merchandiser who is responsible for the content (images and text) that is presented on your website, designed to sell products to customers.
• A product manager who is responsible for sales of a line of products, including through your website.
• A host of others, including SEO experts (responsible for getting traffic to the website), user experience teams responsible for how customers interact with the site, and implementation teams responsible for building different elements (such as the CMS or the customer-data master or the product catalog).

The Customer

The customer has a straightforward requirement: having a great shopping experience while getting great deals on the perfect products.  This is an infinitely deep area for discussion, but to keep this article short(er), the focus will be on the internal players at your haberdashery.

The Product Manager

The product manager is responsible for his product line.  In this scenario, he makes high-end, retro-styled hats.  The two most important products are the bowler hat and the top hat.  The product manager is responsible for the profitability of this retro-chic hat line.  The bowler hat sells for $175 with a $75 profit margin, and the top hat sells for $250 with a $125 profit margin.  The product manager has a strategic requirement.  We’ll use a user story to describe his requirement.

• As a product manager, I need to increase the profits for my retro-chic hats product line, which already dominates the market, so that our stock price will rise.

This requirement is too ambiguous and abstract to be actionable.  The product manager can’t really give this to anyone else.  What he can do is create an Ishikawa diagram that helps him identify the components of product line profitability, so that he can choose one to focus on.  Our haberdashed product manager decides (for this example) that the strategic lever he wants to pull is to change the product mix.  He feels that growing the total number of hats sold is not a good strategy – his market is saturated.

Instead, his goal is to convince his customers to buy more top hats – at the expense of buying fewer bowler hats.  Each customer who switches from the bowler to the top hat will generate an additional $50 in profit.  He further needs to make sure that he doesn’t lose sales to customers who really wouldn’t be happy with anything but a bowler hat.  Being measurable is also important.  To meet his performance objectives, the product manager realizes he needs to shift the mix from 50/50 to 60/40 (60 of every 100 retro-chic hats sold need to be top hats).  His requirement can be rewritten.

• As the retro-chic hats product manager, I need to increase the percentage of top hats sold to 60%, without reducing the total number of hat sales, so that I can increase the profitability of my product line.

OK, that’s measurable.  Whatever someone does, you’ll know if it succeeded.  But is it actionable?  No – it is still ambiguous.

The Merchandiser

The merchandiser is responsible for finding the best photos, writing the best ad-copy, and determining the marketing messages that are displayed to customers on the website.  The merchandiser is held accountable for conversion – the percentage of visitors who come to the site, view the content, and ultimately decide to purchase.  Since your company is not dis-functional, the merchandiser has aligned her goals with the product manager.

• As the retro-chic hats merchandiser, I need to convince online customers who would have bought the bowler hat to buy the top hat instead, so that the percentage of top-hats sold is at least 60% of all retro-chic hats.

There’s an implicit choice here – a specification.  Here’s another user story that would achieve the same goal.

• As the retro-chic hats merchandiser, I need to increase the conversion rate for customers that view the top hat page from 2% to 3%, so that the percentage of top-hats sold is at least 60% of all retro-chic hats.

Since they are both actionable, but both very different, that works as a good litmus test that the product manager’s requirement, while measurable, was still ambiguous.

There could also be an SEO expert, for whom the story would be “…increase traffic to the top-hats page without decreasing conversion…”  A member of the the site design (or user experience) team, could try and improve conversion by creating a more efficient flow (like “one click ordering”) that increases conversion rates – but only enable it for the top hat product.

There are just too many cooks in the kitchen.  Every one wants to feed the hungry people, but each specializes in a different dish.

You have at least four viable ways to approach solving the product manager’s goal.  You need to pick one before you get the development team involved.

A Matter of Perspective

The merchandiser, SEO expert, and site design teams all have the same goal – support the product manager and achieve a 60/40 mix in retro-chic hat sales.  Each of them can develop a reasonable requirement, within their own domain.

To a man with a hammer, every job looks like a nail.  The merchandiser is not going to focus on purchase-path redesigns, nor is the SEO expert going to suggest changing from a boring photo of a manikin to a photo of an attractive man laughing with friends over an expensive meal.

The product manager, however, should not be telling the cooks how to make the stew.  The product manager (of the product being sold) is not responsible for, and should not be driving the decisions about how to achieve his 60/40 mix goal.  The product manager could think of the people in each area as vendors, “selling” alternate solutions to his problem.  Each solution has a cost, so the product manager probably can’t just “do them all.”  Unfortunately, the product manager is not qualified to pick the right vendor – only to evaluate the promised benefits (from each) at the projected costs (from each) and hope he chose wisely.

A Website Product Manager

The critical need to product managing your website is more visible than ever.  Without it, you won’t make good decisions about how to address your real business goals.  You need someone who understands the risks, trade-offs, and possibilities inherent in each of the four solution approaches.  If your website is large, you may have a portfolio manager for your website, with individual product managers owning areas of the site.

Recognize that not only are your (company’s) customers your customers – but so are your other stakeholders.  As a website, you create commerce, you don’t just sell products.  You have to help (external) customers have great shopping experiences, while helping (internal) customers meet their goals.

The website product manager works with the retro-chic product manager to understand his goal of achieving a 60/40 mix in sales.  Taking into account the big picture from a website perspective (who are our customers, who is the competition, what is our vision), the website product manager has to make a “design decision” and choose one of the solution approaches.

The website product manager is constraining the solution space by selecting one of the hammers.  Is this bad?  The retro-chic product manager constrained the solution space too (when deciding that a shift in mix was the “right” answer).

Then the how would we solve it, if constrained to solving it this way, user stories have to be developed and given to the implementation team.

Conclusion

You need to have someone acting as a product manager for your website.  That person is in the ideal position to constrain the solution approaches to addressing any given stakeholder.  Those constraints do limit your options – so you better have a good product manager.  It really is no different, however than saying you better have a good general manager, so that the strategy you are supporting is a good one.

What are your interactions like with the folks that own your website?  Or are you already a website product manager?

SEO, Search Engine Optimization, is an area that every online website needs to think about.  The idea is that the more traffic you can get to your website, the more products you’ll sell.  Just because you can lead a horse to water doesn’t mean you can make him drink.  What a great opportunity to product manage your website and ask why about SEO.

SEO

When you’re building a website, you have four primary channels by which you get traffic (visitors) to your site:

1. Direct Traffic – people who type in the URL (address) of a page on your website directly into their browser.
2. Referral Traffic – people who are sent to a URL on your website from another website (usually by clicking on a link).
3. Paid Search Traffic – people who click on an advertisement that you run (on other websites) to reach a URL on your site.
4. Organic Search Traffic – people who use a search engine (like Bing or Google) to try and find an answer to a question, who then click on a link to a URL on your website within the search engine results.

[Note that the graph above combines #3 and #4 into one channel of traffic, as the source of about 1/3 of the traffic for this website.]

Search Engine Optimization, or SEO, is collectively the set of activities you do to increase (“optimize”) the amount of traffic you get from organic search (#4 above).  SEO has at best a second-order effect on the other three channels from which you may get traffic – they are effectively unaffected by your SEO activities.

There are entire companies, in fact an entire industry, built to help companies increase the traffic they get from “organic” search.  It is called “organic” search based on the premise that search engine algorithms will determine (through their own proprietary algorithms) the “best” results for any given search.  Like nature, this can be influenced by you, but is out of your control – hence “organic.”  This is in contrast to paid search, where you are directly controlling when someone comes to your site (by clicking on an advertisement that you paid for).

Since SEO is such a large topic, even though this is a longer article, it only scratches the surface.  This article focuses on the decision making you would do (and the measurements needed to support those decisions).  It does not attempt to be a one-stop-shop for all of your SEO needs.

SEO Has ROI (or Does It?)

People who promote their SEO services will tell you – build it, and they will come.  What you do with the traffic once it arrives is up to you – SEO just gets people to show up.  Makes for a good movie, but it doesn’t always work that way.  This is where product management can help.

You want the right people to show up, not just anyone.  In keeping with the baseball analogy, applied to the web: your website is the stadium.  The games are free.  Your SEO efforts get people to show up at the stadium.  But you’re not selling tickets.  The only way you make money is if people buy hot dogs or pennants or peanuts.  You have products for sale at the stadium and you want people who will buy them to show up.  You don’t get any value from just filling the seats.

If the people who show up don’t buy anything, it’s as worthless as if no one ever came in the first place.

I’ve worked with a client in the past who had a project that successfully increased the number of visitors to one of their websites by over 30% for several months, resulting in almost no change in the amount of products they sold during that period.  An SEO-only person would say “hey, the people showed up, I did my job,” but a product manager is acutely aware that this was an exercise in futility.

Perhaps the people who showed up had no intention of buying anything.  That would be consistent with the data my client collected.  Perhaps something intrinsic to the website prevented the new visitors from purchasing (while allowing a consistent percentage of the previous visitors to continue making purchases).  To the best of my knowledge, the data needed to perform that root cause analysis was not available.

When you’re product managing your website, you are responsible not only for getting people to your website, but also for getting them to convert their presence into purchases.

Measuring SEO – Easy Versus Important

The challenge in any measurement activity is determining what to measure.  There always seem to be a bunch of things that are easy to measure, and a handful of things that are important to measure.  When you’re lucky, there’s some overlap.  The challenge is to resist the urge to measure stuff just because it is easy – you’re making work for yourself – both in taking the measurements and in reviewing the measurements.

To select the right measurements for your website, you first have to understand your goals.  Assume that your goal is to sell products, profitably.  This is the driver for your “bottom line measurements” – at the end of the day, how is your product (website) performing?  If you aren’t measuring revenue and profits, you won’t know if you’ve filled your stadium with “empty chairs” who don’t buy any peanuts.

You also have to understand the buying processes that users (website visitors) use to make purchases from you.  One way to characterize the buying process is to look at the process from the perspective of the user, who goes through the following stages of activity:

1. Identify a need to solve a problem (possibly by purchasing a product).
2. Discover possible solutions to the identified problem (possibly products that solve the problem).
3. Compare solutions (products) and decide which to implement (purchase).
4. Solve the problem (purchase and use the product).
5. Re-evaluate the solution (product).

Step 2, discover possible solutions, is where SEO can help.  One way people can discover solutions (and there are many others) is by using a search engine to discover solutions.  Those people may search for phrases that describe their problem (“webcam not working with Skype”) or ask questions as if the search engine could solve their problem directly (“how do I make my webcam work with Skype?”).  Some people will determine (or assume) that the problem is with their webcam, and that they need a new one.  Those people will search for the solution they’ve already envisioned (“best webcam”), or combine steps 2 and 3, and search for comparison data (“webcam reviews”) or pricing data (“webcam deals”).

In addition to the goal-measurements (step 4), you’ll also want to include search-effectiveness measurements (step 2).  Keep in mind that there are other factors at play in your website – not every visitor who searches has a propensity to buy, visitors who arrive may abandon your site because they don’t like your pricing, etc.

Measuring SEO Effectiveness

Keep in mind that you’re trying to sell products (profitably), and this particular effort is focused on improving organic search as a mechanism by which you attract customers to which to sell products (profitably).  Your measurements should focus on these two aspects.  Here are some things that are interesting to measure [note: the screenshots below are of Tyner Blain traffic - I won't share any client data - so these values are low compared to a successful eCommerce site - but they are illustrative.]:

Organic Search Engine Traffic

• How many visitors come to your site from search engines (and from which search engines?)?
• 
• What are the keywords / phrases that visitors used to find your website?
• 
• To which pages did the search engines direct the most traffic?
• 
• What search terms sent traffic to which pages?
• 
• SERP Ranking (Search Engine Results Page Ranking) – How high up is your result for one of your targeted keywords?  Is it on the front page?
• 

All of those measures, while easy to do (the screenshots above are from the free Google Analytics program), only tell you about how many people came to your stadium, they don’t give you any insight into peanut sales.  You can use these measurements to provide feedback as you make changes in your SEO implementation – to determine if each change increases or reduces traffic (or fills seats).

You also need to know about the peanuts.

The specific financial measurements you make will depend on your strategy for how you engage your market.  Are you a discounter, trying to maximize sales in the short term, with a transactional focus?  Are you focused on the long term, building relationships, and maximizing the lifetime value of customers (better to sell more, later, than less, now)?  Are you trying to gain market share, or maximize profits in a mature market?  The specific ways you measure will vary, but some common measurements are:

• Conversion Percentage – how many of the people who come to your site end up purchasing (during that visit)?
• Number of Purchases – how many orders were placed?
• Average Order Value (AOV) – how much is each order “worth” in both revenue and profit?
• Lifetime Customer Value - how much is each customer “worth” in both revenue and profit?

The way you get real insights from these measurements is by combining them.  Utilize the “SEO mechanics” measurements (above the peanuts) to slice or segment the financial measurements.  Ask questions like “How many orders were placed by people who came to the site and landed on the webcam page?” or “What is the conversion percentage for people who came to the site by searching for best webcam?” or “How much revenue do we get from each keyword that sends organic traffic to our site?”

SEO Goals – Getting Actionable

Combining the results of the two forms of measurement (mechanics vs. revenue) will identify for you where you are getting the most value, and where you have the most opportunity to increase value.  You’ll have to do analyses of “do I make one thing (page, keyword, etc) better, or do I try and make all things better?”

One technique that can help you reach those decisions is by noticing that most of the metrics you see follow power law (long tail) curves.  Roughly speaking, 80% of your traffic will come from 20% of your keywords.  80% of your sales may be to 20% of your customers.  20% of your pages may generate 80% of your visits (or 80% of your revenue).  You’ll have to “do the math” for each project to estimate the impact of those projects – and determine if your best bet is to improve a single page (a lot), or make a change to your site that improves all of the pages (a little).

You may find that your website is not sufficiently instrumented to make the measurements you need.  My suggestion is to instrument your website first, and then try and improve it second.  If you can’t measure it, it didn’t happen.

The current “state of the industry” for SEO has a hint of product management influence in it already – there’s some awareness that these sorts of aggregate statistics are limited in their ability to give you insight about your customers.  Where SEO seems to be pushing today is in being able to segment the above data views against personas or market segments.  This looks to me like an industry that is maturing beyond the elastic user problem that software development and user experience teams have been addressing for years.  This approach will allow you to target your website development initiatives with an eye on the uniqueness of customers in different market segments.

SEO Product Management Conclusion

Search engine optimization is important.  Focusing your efforts to improve your business, as it relates to search engines, is what is really important – and search engine optimization is how you get there.  Discover the factors that have a bottom line impact, and then execute to address them.  Don’t just assume that more traffic is better traffic.

Design-Free requirements are important for two reasons, and hard for two other reasons.

Design-free requirements are hard because you “know what you want” when you should be documenting “why you want it.”  Writing design-free requirements can be hard when you don’t trust your development team to “do the right thing” even though it is not your job to design the solution.

Design-Free Requirements – Revisiting

It has been three years since I wrote Writing Design-Free Requirements as part of The Big Ten Rules of Writing Requirements.  In that time, agile development practices have moved from being an esoteric development methodology to being the topic on the tips of everyone’s tongues as executives and organizations try to either (1) get the benefits of the state of the art in software-development process or (2) do something shiny and fashionable.

The previous article centered on elements of designs within MRD, PRD, and SRS artifacts.  A regular alphabet soup of artifacts.  Honoring the values behind the agile manifesto encourages us to emphasize working software over comprehensive documentation.  In that light, we’ll take a “what is needed” approach to talking about how requirements, design, and implementation are all needed – rather than issue an edict about where that information should be captured.

Agile Development Inputs

When creating software, someone needs to know:

• What problems are being solved, and how important (valuable) they are to be solved.
• Who has the problems and who is using the software to (help) solve those problems.
• What constraints limit the space that defines the universe of possible viable solutions.
• What acceptance criteria define if the delivered solution will be acceptable.

Subject to those inputs, someone needs to make design decisions:

• User experience design - what interactions will a user of our solution love?
• Program design – how (in a nuts and bolts way) will our solution function?

When talking about agile and talking about design, we should take a look at how Kent Beck and Alan Cooper, as respective though leaders in each space, view this.

Cooper doesn’t talk much about creating the requirements to support the daily use scenarios – he proposes moving directly into design of the solution. This differs from the more traditional technique of writing functional requirements to support use cases. Cooper also breaks down design into two components – program design and interaction design. Program design is everything you don’t see. Interaction design is everything you do see.

[...]

Cooper argues that designing the interaction should happen before any code is written. He uses a construction analogy to drive home his perspective – you can’t pour the concrete before you build the forms. Kent Beck, founder of the XP programming philosophy disagrees with the premise. Beck believes that the cost of changing software is low, and the imagery Cooper uses is hyperbole. We touch on, and link to that debate in this post.

Interaction Design Process Overview

I don’t believe that Beck is arguing for “don’t do design” – I believe he is arguing for “don’t do big upfront design that would delay implementation.”  He’s championing the agile values that emphasize creating working software and responding to change.  I can imagine him saying “no one will buy a design, they want a solution.”  Cooper’s point is that create a solution without first understanding how someone will use it, you can’t create a great product.

Program design has many hidden impacts – cost of maintenance, cost to change, and the performance of the solution.  You can have a design (or architecture) that makes everything easier to do in the future – at the cost of delaying the delivery of anything.  Or you can have a design that minimizes the time to deliver the first thing, while increasing the cost to deliver any particular next thing.  Or you can design a solution that falls somewhere between those extremes.

If you say “we don’t do design” you’re lying.  Every solution includes design.  Your team’s design process may be “big and upfront” or it could be a couple sketches on a whiteboard, or some email exchanges.  You may create storyboards and wireframes.  Or design may be the thing that happens right before the programmer’s fingers strike the keys.  Design may be explicit or it may be implicit – but it never “doesn’t happen.”

In the movie Amadeus, Salieri is astounded that there are no rough drafts of Mozart’s compositions.  That’s because Mozart did the design in his head, not because he didn’t do design.  I’ve worked with programmers who’s “implicit designs” were great, but that is the exception, not the rule.

Who Owns Design?

Since design happens sometime before fingers strike the keyboard, the real question is – “who owns the design?”

• You have a product manager developing an understanding of market problems, and prioritizing the problems that should be solved with your product.
• You may have a product owner managing the backlog and clarifying those requirements (problems) for the development team.
• You may have an interface (or interaction) designer or team of designers who are broadly responsible for “how users interact with our products.”
• You may have an architect or lead engineer who is responsible for the “big picture design” of how your product works.
• You have developers and testers who are responsible for delivering your product. [Note: coding without testing is like typing code without compiling - maybe it works, but probably not.]

You may not have distinct individuals with each of these titles – every small team I’ve worked with has people wearing multiple hats.  This is especially true when you have an agile team full of specializing generalists – any given story (or task) may have a different architect and different implementer.  I’ve only worked with one company where the architects are NOT part of the development team.  If your team is set up that way, please comment below – I had never seen that before, and I’m not convinced that it is the best way to organize – what have your experiences been?

Generally, “program design” is clearly owned by the development team – and product managers (and product owners) know better than to specify program design in their requirements.  Neither the lead engineer nor the product manager believes that the product manager is a better programmer – so the product manager better not be specifying program design and mislabeling it as requirements.  Coincidentally, Steve Johnson, at Pragmatic Marketing has a post running right now with a bit of a quiz – is this a req(uirment) or a spec (design)?

Where the line is more blurred is around interface and/or interaction design.  Some development teams have interface designers as part of the team.  Some companies organize with interface design as a “shared service” within the company.  Either approach can be the “right one” – it depends on too many details to make a sweeping generalization.  When the designers are members of the development team, the solution (from a product management perspective) is the same – the “team” is responsible for all design.  When the designers are not part of the development team, the developers have to consume two sets of guidance – “solve this problem” from product management, and “the solution needs to look like / act like this” from the designers.

Collaboration between product management and user experience people is the ideal solution.  The “requirements” and “design” inputs to the development team are comprehensive and consistent.

Design-Free Requirements

There are benefits – especially when being agile and minimizing documentation – to delivering requirements and design at the same time.  Just don’t do it by embedding design constraints within the requirements.

• When people on your team misinterpret design as requirements, they are unnecessarily constrained.
• As a product manager – are you the best designer on your team?  If you’re busy designing, who’s product managing?

This is trickiest when writing use cases – sequencing a set of interactions is interaction design.  That is one benefit of writing a user story instead of a use case.  An approach that has worked well for me with multiple teams is to deliver user stories (requirements) combined with storyboards (interaction design) and wireframes (interface design).  When details are needed (usually when “changing” versus “creating” an interface), screenshots can replace wireframes.  When business processes are complicated, process flows can replace storyboards.

Just don’t embed the designs within the requirements.