The High Costs of Building the Wrong Product

A Praying Mantis

As product managers, we talk about creating the right solutions with our products. Understanding the very real problems our customers face, understanding the very real opportunities our markets present, and manifesting that understanding in a product roadmap.

Other than being “not as good,” how expensive is it to build the wrong product?

The Cost of Poor Quality

A frayed Steel cable

There’s an analog to the market dynamics of making poor product decisions – executing with poor quality. Many research studies and articles have identified the market impacts of poor quality.  This has become so well accepted that people today cite it like a law of physics (one example here based on this 1988 IEEE research by Barry Boehm and Philip Papaccio) as the “1-10-100 rule.”  The primary conclusion of that research is that ten dollars spent on fixing bugs:

  • Costs and saves $10 when you catch (and fix) the bug during implementation.
  • Avoids $100 in costs when you catch the bug during QA and send the product back to development (then test again).
  • Avoids $1,000 in costs versus waiting until your customers catch the bug in the field, causing the team to remedy the problems, rush out a patch release, and/or go to heroic lengths to manage a PR problem.

This is an opportunity in front of your product team – a 100x payback from investing in quality during the development process.  Of course, be pragmatic about it - if the cost of testing exceeds the cost of bugs, don’t test.

This is not a solved problem, by any stretch, but the solutions and methods to solve this problem are well understood now.  In fact, a 2001 article by Barry Boehm and Victor Basili shows that in some cases, the labor costs to resolve bugs can be as low as 5:1 – when considering a subset of smaller systems, when using more “agile” processes.  That lowered ratio does not take into account the lost market opportunities and the costs of cleaning up collateral damage to your product – just the immediately realizable (and measurable) costs of resolution.

One very real problem, when talking about “bugs” is in defining what a “bug” is. And the definition of a bug is a matter of perspective. A developer can reasonably assert that “if it meets the spec it is not a bug, it is working as designed.” What if the spec is wrong? The developer may not be guilty, but collectively, your team screwed up. There’s a “bug” in the requirements.

What Is A Requirements Bug?

A very unlikely hockey stick revenue forecast graph

Now things are getting interesting.

If you wrote a requirement that you interpret as “A” and your developers interpret as “B” – you definitely have a bug – the team won’t build the right product. For each $1 you could spend making sure you have bug-free requirements, you could:

  • Make sure you have a shared understanding of the documented requirements through active listening before development begins ($1). Following the Rules of Writing Requirements will help prevent this miscommunication.
  • Wait until the engineering team is ready to demo their progress ($10). They will have to build it again, because they built the wrong stuff.
  • Wait until development is complete and QA is validating that the code meets the spec ($100). This gets tricky if you are thinking “A”, the developers are thinking “B”, and QA is thinking “C.”
  • In classic throw-it-over-the-wall mode, wait until the product is launched, and it is the wrong product ($1000). Assuming “A” was the right problem to solve, the cost of entering the market with a solution to “B”, leaving “A” unaddressed, is impressively high.

This gets interesting because the above assumes that “A” was the right problem to solve. What if “G” was the right problem to solve, and “A” was the wrong market problem? Even if everything (else) is working perfectly – you document requirements for “A”, the engineering team creates a marvelous “A” and it launches without implementation errors – you still fail, and incur the 1,000x cost of a failed product launch.

There is an even larger opportunity in front of your product team – a 1,000x payback on discovering and choosing to solve the right problems for your customers and markets.

  • Would Palm still be independent if the Pre had solved a compelling problem?
  • Why did Intuit have to buy Mint.com – could they have embraced the same customers with Quicken?
  • What is Garmin going to do now that “free” GPS mapping and turn-by-turn directions are becoming ubiquitous? If it is “more of the same,” how much are they wasting?

Wrapping Up

Wrapping paper rolls

I’m not aware of any studies that show that “requirements bugs” fit the same 1/10/100/1000 cost explosion model that “implementation bugs” exhibit. Emotionally, it “feels about right” to me – it passes my “sniff test.”

There are times when days of research would have been required to avoid or redirect a few hours of implementation effort on projects I’ve worked on. And I’ve seen man-years invested solving problems that didn’t involve much more research.

My intuition from products and teams I’ve worked with is that it probably averages out somewhere around 10x.

What does your gut (or your data – if you have some, post a link below!) tell you?

Post to Twitter Post to Facebook

This article was published on Monday, June 21st, 2010 at 8:15 am and is filed under Business Analysis, Product Management, Requirements, Requirements gathering.
You can leave a comment on this post

5 Comments

  1. Garmin, well, they are building a GPS with a phone….versus a phone with a GPS.

    http://www8.garmin.com/nuvifone/

    Take that, Apple!

  2. @Alex – funny. I was going to say “doomed to fail!” but when I followed the link from your link to “more info from ATT” I got this: http://www.wireless.att.com/cell-phone-service/cell-phone-details/?device=Nuvifone(TM)+G60&q_sku=sku4000279

    which says “The Nuvifone (TM) G60 you’re searching for is no longer available.”

    So “doomed” may not be exactly the right word here :)

  3. You hit it on the head. (Ouch!) See my very old, related trope on post-course corrections, http://www.mironov.com/articles/post_course_correction/ .

  4. well done Scott! for more studies on cost of wrong requirements, i’ve got a pop up at: http://www.ebgconsulting.com/reqtproblems.php#l2 (click on the “Read More” link. Reifer’s 2007 data is most recent i’ve found.

    will be interesting to see if/how agile practices alter these data.

    ~ ellen

46 Trackbacks

  1. By Scott Sehlhorst on June 21, 2010 at 2:17 pm

    New on Tyner Blain: The High Costs of Building The Wrong Product http://bit.ly/bmtQmO @prodmgmt

  2. By Joshua Duncan on June 21, 2010 at 2:20 pm

    great read RT @sehlhorst: New on Tyner Blain: The High Costs of Building The Wrong Product http://bit.ly/bmtQmO @prodmgmt

  3. By Roger L. Cauvin on June 21, 2010 at 2:36 pm

    Tyner Blain's @sehlhorst quantifies the cost poor #prodmgmt and requirements. http://bit.ly/bUHJ2s

  4. By Mike Boudreaux on June 21, 2010 at 3:10 pm

    RT @sehlhorst New on Tyner Blain: The High Costs of Building The Wrong Product http://bit.ly/bmtQmO #prodmgmt

  5. By ElizabethQuintanilla on June 21, 2010 at 3:12 pm

    RT @MikeBoudreaux: RT @sehlhorst New on Tyner Blain: The High Costs of Building The Wrong Product http://bit.ly/bmtQmO #prodmgmt

  6. By waltboyes on June 21, 2010 at 4:06 pm

    RT @MikeBoudreaux: RT @sehlhorst New on Tyner Blain: The High Costs of Building The Wrong Product http://bit.ly/bmtQmO #prodmgmt #pauto #mfg

  7. By Vijay V on June 21, 2010 at 4:33 pm

    By @sehlhorst: The High Costs of Building the Wrong Product http://bit.ly/d0A65h

  8. By kinaze on June 21, 2010 at 4:51 pm

    The High Costs of Building the Wrong Product http://bit.ly/byfoCk

  9. By SolutionsIQ on June 21, 2010 at 5:05 pm

    Great read: The High Costs of Building the Wrong Product http://ow.ly/21gXm #agile

  10. By Jerome on June 21, 2010 at 5:11 pm

    RT @kinaze: The High Costs of Building the Wrong Product http://bit.ly/byfoCk

  11. By Produktmanager Blog on June 21, 2010 at 7:45 pm

    RT @MikeBoudreaux: RT @sehlhorst New on Tyner Blain: The High Costs of Building The Wrong Product http://bit.ly/bmtQmO #prodmgmt

  12. By Yama on June 21, 2010 at 7:48 pm

    The High Costs of Building the Wrong Product by @sehlhorst #prodmgmt http://ping.fm/CGoT4

  13. By Glenn on June 21, 2010 at 8:24 pm

    By @sehlhorst: The High Costs of Building the Wrong Product http://bit.ly/d0A65h

  14. By Paul W. Thomas on June 21, 2010 at 8:35 pm

    RT @waltboyes: RT @MikeBoudreaux: RT @sehlhorst New on Tyner Blain: The High Costs of Building The Wrong Product http://bit.ly/bmtQmO

  15. By Cliff Gerrish on June 21, 2010 at 10:13 pm

    The High Costs of Building the Wrong Product http://ff.im/-mtyZu

  16. By Scott Sehlhorst on June 22, 2010 at 1:12 am

    Thanks @joshua_d @paulthomaspharm @jyamasaki @vijay_vaddem @waltboyes @mikeboudreaux @rcauvin 4 http://bit.ly/bmtQmO RTs!

  17. By sumeet_moghe on June 22, 2010 at 9:27 am

    http://bit.ly/a9dbMD The high costs of building the wrong product #yam #twuos #agile

  18. By Seilevel on June 22, 2010 at 8:40 pm

    By @sehlhorst: The High Costs of Building the Wrong Product http://bit.ly/d0A65h

  19. By April Dunford on June 24, 2010 at 6:25 pm

    reading – The High Costs of Building the Wrong Product http://bit.ly/bmtQmO from @sehlhorst #prodmgmt

  20. By Foundora on June 25, 2010 at 2:10 pm

    The High Cost Building the Wrong Product – http://ow.ly/239Be

  21. By Antoine HULIN on June 28, 2010 at 12:16 pm

    The High Costs of Building the Wrong Product : http://xrl.us/bhp53v

  22. By Roby Van Damme on June 28, 2010 at 5:04 pm

    [Reading] The High Costs of Building the Wrong Product: http://bit.ly/94Lgwj

  23. By TomGrantForr on June 28, 2010 at 8:25 pm

    The high cost of building the wrong product http://bit.ly/94Lgwj #prodmgmt #requirements

  24. By Kurt Weisenberger on June 28, 2010 at 8:33 pm

    RT @TomGrantForr: The high cost of building the wrong product http://bit.ly/94Lgwj #prodmgmt #requirements

  25. By txaggie95 on June 28, 2010 at 9:03 pm

    RT @kurtmw: RT @TomGrantForr: The high cost of building the wrong product http://bit.ly/94Lgwj #prodmgmt #requirements

  26. By Cataphora on June 28, 2010 at 9:30 pm

    RT @TomGrantForr: The high cost of building the wrong product http://bit.ly/94Lgwj #prodmgmt #requirements

  27. By Rich Mironov on June 28, 2010 at 10:52 pm

    RT @TomGrantForr and @sehlhorst: The high cost of building the wrong product http://bit.ly/94Lgwj #prodmgmt #requirements

  28. By ellen gottesdiener on June 29, 2010 at 1:44 am

    RT @RichMironov: RT @TomGrantForr and @sehlhorst: The high cost of building the wrong product http://bit.ly/94Lgwj #prodmgmt #requirements

  29. By tomhumbarger on June 29, 2010 at 3:16 am

    Acknowledging the Cost of Bad Product Management | from @sehlhorst on the Tyner Blain blog http://ow.ly/24vCc #requirements

  30. By yess_padilla on June 30, 2010 at 9:33 am

    By @sehlhorst: The High Costs of Building the Wrong Product http://bit.ly/d0A65h

  31. By Deva Sidhai on July 1, 2010 at 12:13 pm

    As product managers, we talk about creating the right solutions with our products. Understanding the very real http://sorturl.net/?bd3adc

  32. By VasilyKomarov_RSS on July 1, 2010 at 12:13 pm

    As product managers, we talk about creating the right solutions with our products. Understanding the very real http://sorturl.net/?bd3adc

  33. By VasilyKomarov RSS on July 1, 2010 at 12:13 pm

    As product managers, we talk about creating the right solutions with our products. Understanding the very real http://sorturl.net/?bd3adc

  34. By ellen gottesdiener on July 1, 2010 at 3:55 pm

    @sehlhorst $ of Building Wrong Product http://bit.ly/bmtQmO #agileba #agilepm #baot #pmot #prodmgmt >more data at link http://bit.ly/b7OzP6

  35. By Fernando Garrido Vaz on July 6, 2010 at 12:15 pm

    The High Costs of Building the Wrong Product http://bit.ly/bkEDxj

  36. By eportelance on July 11, 2010 at 1:17 am

    Acknowledging the Cost of Bad Product Management http://instapaper.com/zHxe5shK

  37. By Why you should fix the bug | paul schreiber on July 11, 2010 at 11:57 am

    [...] “how I tried to save Apple $11,000,000”: Avoids $1,000 in costs versus waiting until your customers catch the bug in the field, causing [...]

  38. By cindyalvarez on July 13, 2010 at 4:59 pm

    The high costs of building the wrong product http://bit.ly/9YpjNr #prodmgmt

  39. By 500Friends on July 13, 2010 at 6:15 pm

    RT @cindyalvarez: The high costs of building the wrong product http://bit.ly/9YpjNr #prodmgmt

  40. By LeanBot by @SaintSal on July 13, 2010 at 6:17 pm

    @cindyalvarez: The high costs of building the wrong product http://bit.ly/9YpjNr #prodmgmt

  41. [...] bring some additional data and perspective to bear on this.  First, we have the Tyner Blaine post, The High Costs of Building the Wrong Product, which is a fantastic explanation of the concepts discussed between myself and Alain Breillatt.  [...]

  42. By Peter Müller on July 20, 2010 at 3:38 pm

    RT @ellengott: @sehlhorst $ of Building Wrong Product http://bit.ly/bmtQmO #agileba #agilepm #baot #pmot #prodmgmt >more data at lin …

  43. By Jane Powell on August 3, 2010 at 12:56 pm

    The High Costs of Building the Wrong Product http://bit.ly/c97uDu

  44. By Debbie Howell on August 21, 2010 at 3:55 pm

    RT @sehlhorst: The High Costs of Building the Wrong Product http://bit.ly/bmtQmO

  45. [...] @sehlhorst: http://www.tynerblain.com [...]

  46. By Cadence Versus Risk on October 31, 2011 at 5:18 am

    [...] True insights about your market grow stale with time.  The damage from erroneous conclusions about your market grows over time.  The risk of creating the wrong product grows over time. [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>