Market Problems – Comparing Products Part 3

Comparing products without an understanding of the important market problems by which to compare the products is a waste of time.  This is the third in a series on comparing products - jump back to the introduction if you haven’t already read the previous articles.  Go ahead, we’ll wait, then come back.

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

  1. Introduction & Overview (so that the step-numbers align with the article numbers)
  2. Identify your customers.
  3. Articulate the problems they care about solving. (This article)
  4. Determine how important solving each problem is, relative to the other problems, for your customers.
  5. Characterize how important it is for you to solve the problems of each group of customers.
  6. Discover which (competitive) products your customers consider to be your competition.
  7. Assess how effectively each competitive product solves each important problem.
  8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Too many product comparisons are actually mislabeled positioning papers, highlighting differences, and putting one product in the best possible light.  An honest product comparison must compare products in the context of problems that customers are willing to pay to solve.  Those are the important market problems.

Important Market Problems

Probably everyone reading this has heard the Henry Ford quip about how if he had listened to his customers, he would have built faster horses.  A great marketing sound-bite, but not actually true.  The market problem that cars initially addressed wasn’t “get there faster”, the market problem was “spent fuel” disposal (for horses) in major cities – 2.5 million pounds per day in New York City (hat tip to Super Freakonomics for the first reference I saw).  If cars had generated as much manure as horses, do you think they would have been as successful?

The first step in identifying the important market problems is to identify a set of market problems that might be important. Then you can form an opinion about which ones are important, based on which problems are important to the potential customers in the markets on which you are focusing.  Identifying problems is an unconstrained activity, like collecting seashells on a beach.  There will always be more shells.

You will find shells that aren’t very pretty or are broken.  You will find some shells that are “perfect”, and there will always be some shells that exist, that are “perfect” that you won’t find.  The tough part is knowing when to stop looking for shells and move forward with the ones that you’ve already found.  You have to decide when your shell collection is “good enough.”  You can’t have a collection that includes all of the seashells, you don’t want a meager collection with too few shells, and you want the best shells you can find to be the ones you display on your coffee table.

This is a good example of the debate between big up-front design (BUFD) & big up-front requirements (BUFR) and incremental product development.  In The Design of Design, Frederick Brooks provides a great discussion about this debate between rationalism and empiricism.  The best approach I’ve found to date is to use a combination of agile and waterfall techniques to strike a balance between good enough and right now.

The best approach that I’ve found to do this to combine incremental development techniques to the activities of product management, combined with timeboxing to impose practical, artificial constraints on the process.  An agile product management process looks roughly like the following:

[larger version]

The steps of market research, analysis, and synthesis are the activities you perform.  Data is what you collect, and insights are what you form.  Analysis of data also leads you to perform more research.  Synthesis of those insights into a cohesive vision for your product will lead you back to more analysis and more research.  Note that this process has no ending – it goes on forever – or for as long as you are trying to solve these particular market problems.

As an example – in the previous article we identified a set of personas for our Kindle Fire example:

For this series of articles, I will pretend that three personas were developed, in order to demonstrate this approach to comparing products:

  1. Hi-Tech Prosumers – people who get excited about convergence, highly value additional multimedia capabilities, live a multi-device existence, and are acutely aware of competitive products.
  2. Typical Amazon Kindle Users – people who placed high value on the reading experience, the convenience of being able to easily get new content, and the absence of a subscription fee.
  3. Basic Consumers – people who read primarily on paper media, have recently started consuming multimedia on their computers, and anticipate becoming part of the “post pc era.”

I will also invent additional data as we explore the rest of the product comparison process.
Comparing Products Part 2 – Who Are Your Customers?

However, defining requirements for those personas risks creating a product that only satisfies the basest of goals – the lowest common denominators.  These examples don’t take into account the context of use – e.g. why is someone reading on their device? – and thus we are at risk of not creating the right product for any of them.  Even introducing two simple contexts

  • Education - Reading for work or school (directed study) or to gain knowledge about a topic (undirected study)
  • Entertainment - Reading for pleasure, the literary equivalent of watching a movie

You’re probably not going to want to take notes on the uses of foreshadowing and allegory when reading for entertainment.  It may be critically important to annotate key concepts and ideas when reading for education.

This single distinction should be clanging a bell in your head – it is very easy for one product to be better than another in one context of usage than in the other.  Looking at the three example personas from the previous article, there is not a good way to say “educational use is more important” for one group of people than for another.  If we iterate and refine our list of personas it will lead to a better comparison of products.

  1. Tina - A hi-tech prosumer who is using the device to get smarter about the latest trends in her industry
  2. Tim - A hi-tech prosumer who is using the device to enjoy niche fiction content, particularly comics, e-zines and self-published works
  3. Kenny - A typical kindle user who is using the device for his work in the finance space, studying proposals and business plans, etc
  4. Karla - A typical kindle user and voracious reader who is using the device to eliminate the large pile of books on her nightstand
  5. Chris - A basic consumer who would is studying business in college
  6. Christina - A basic consumer who is in a book club, and who is always reading the latest best seller

Identifying Possibly Important Problems

Now that we have a (better) idea of the customers for whom we are solving market problems, we need to create a list of candidate problems that might be worth solving.  I am not an expert on user research – so there may very well be a better way to do this, and hopefully someone will share that with us in the comments section – but here’s how I’ve learned to do it.  It is at least “good enough.”  The approach I’ve used is a 3-step process.

  1. Seed the Cloud – Start with what you think you know already
  2. Refine the List – Validate and enhance that list through qualitative interviews
  3. Empirical Validation – Assess the relative importance of those problems through quantitative research

Seeding the Cloud

I like to start by capturing my initial ideas in a mindmap or concept map – an ad hoc, semi-structured view of ideas.

[larger image]

I will then drive some brainstorming (more on brainstorming) or idea-seeding or Innovation Games exercises to develop a refined list of market problems that might be worth solving.  These processes may lead to other artifacts and usually result in updates to my mindmap.  These exercises bring others into the exploration, so that it isn’t The World According to Scott – but it is still the world as seen “from the inside.”

Refine The List

What we’ve done at this point, while creating a set of questions worth asking, possibly, is still heavily biased as an inside-out view of the market.  It may be informed by previous research and synthesis of ideas, but it still reflects what we think about the market, not what customers in the market thinks about their problems.

The next step is to gather qualitative information from customers and potential customers that represent your target personas.

Anecdotally, this is the same process for driving outside-in innovation from a product management perspective.

Mostly-open-ended conversations with people that are reflective of your personas is how I transform from an inside-out view to an outside-in view.  I’m not asking people “what they want” (remember the faster horse), I’m getting validation that the problems they face are the ones I think I understand (waste disposal).  This usually uncovers additional problems I have not thought of, and discounts problems I may have thought were important.  Ethnographic research provides this type of information – and when I’m working with a team that does this work, I utilize it.  Usually, I’m not, so I will find representative users and ask them questions, or observe their behavior.  Essentially, I’m playing the 80/20 game – some insight is better than no insight.

Empirical Validation

At this stage, you still only have a list of possibly important market problems, but now you have much more confidence that the list is valid and mostly complete.  You also have some qualitative insights into the relative importance of problems to be solved (maybe you did a card-sorting exercise, or created interrelation digraphs, or played the 20-20 Vision game, to get those inputs).

Now you can gather empirical data, to gain some statistical significance to your analysis.  You can do surveys that help you characterize and profile user behavior.  You can even do analysis of social networks, to see which problems resonate and why particular products get good (or bad) word of mouth.

Amplified Analytics has a presentation on slideshare showing how they can mine statistically significant information about features in the context of products. [Hat tip to Gregory, for the pointer to this].

I would suggest using a similar approach to try and understand which problems people face (and solve) versus which features they prefer (as solutions).  But the same applies.  I’ve had good results creating surveys and using Google Docs to aggregate results that allow me to characterize problems, (reported) behavioral patterns – like frequency of use, and (reported) relative importance of solving particular problems (like performance times for specific actions), and quantifying data-characteristics (e.g. how many emails someone sends, and to how many different people, in a given day).

Example Problem List

Given that we’re not actually doing a rigorous (and free) analysis of the Kindle, I get to “pretend” that I did research following the examples above, resulting in the following list of market problems, that are of (at least some) interest to our target personas.  We’ll use this list moving forward in the rest of the series.

  1. Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.
  2. Be able to annotate / highlight what I’m reading for future review.
  3. Be able to have conversations with other people who are reading what I’m reading.
  4. Make it easier for me to find other content that I would like to read.
  5. Be able to subscribe to magazines / newspapers / blogs / serial publications.
  6. Be able to read what people I trust are reading.


It is important to be creating a product with an identity.  That means knowing for whom the product is designed, and for which uses.  You have to know who your users are, and which problems they want to solve.  If you’re going to compare products, you should compare them based on their suitability as solutions to these important problems.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

  1. Introduction and Overview (so that the step-numbers align with the article numbers)
  2. Identify your customers.
  3. Articulate the problems your customers care about solving. (This article)
  4. Determine how important solving each problem is, relative to the other problems, for your customers.
  5. Characterize how important it is for you to solve the problems of each group of customers.
  6. Discover which (competitive) products your customers consider to be your competition.
  7. Assess how effectively each competitive product solves each important problem.
  8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.


30 thoughts on “Market Problems – Comparing Products Part 3

  1. Pingback: Alan Kleber
  2. Pingback: Dan Trojacek
  3. Pingback: Joshua Duncan
  4. Pingback: ~Cindy+F+Solomon~
  5. Pingback: Alltop Agile
  6. Pingback: Larry McKeogh
  7. Pingback: Emmanuel Niclot
  8. Pingback: Giles Farrow
  9. Scott,

    I so agree with Stephan Haux! NPD must be linked to solving problems, and you’ve shown how this can be done. I liked your examples of integrating (Qual and Quant) market research techniques to get to a solid list of personas and a pretty good idea of what problems to solve.

    Glad you stressed getting out of the office to get reads from the market. Also your reminder that once you have collected MI you need to go back, re-think/mull over all the intel/info is spot on. With results as you describe and corresponding analytics, it’ll be easier to obtain budget for future road trips. I bet your success rate increases as well and the not so great ideas get killed sooner.

    Of course, there’s the always present “must manage upward” hurdle. I keep thinking mgt won’t be patient enough…will you be addressing this?

    Looking forward to the next article.

    1. Thanks, Karol!

      You raise a really interesting point about managing upwards, expectation management, and patience.

      Because it is an unconstrained problem, you could do market research indefinitely, with continuous, but diminishing returns for the time you invest, in understanding the information that is already out there. Further, the more time you spend with your head “in the space,” the more opportunity you have to develop a disruptive insight that could change your market and vault your product into a successful position.

      I start by doing a minimal analysis – enough to form a point of view (of questionable value). Then I iterate on that view, until either (1) I feel like the information is valid, or (2) external timing dependencies force me to move forward, even if I’m not comfortable.

      This is where using agile processes helps a lot. I have a list of market problems, in priority order, but I don’t know how accurate my list is. The team starts working on the first item on the identified list – even if that items isn’t actually first, it is likely that this item is somewhere near the top of the actual list. So the risk that the team’s effort is being wasted is pretty low. While they are doing that, I am continuing to get smarter about my market, and re-prioritizing the list.

      Whatever the team works on next will not be the second item from my original list, it will be the top item from my refined list. This works because I’m maintaining an evergreen view of the market – iterating to make it better with each sprint, just as the team is iterating to make the product better with each sprint.

      Given that context, the conversations with management tend to go well. It’s the difference between firing a guided “smart bomb” at the target and firing a shell from a cannon. With the cannon, if you’re a little bit wrong at first, you will miss – and you probably won’t know that you’ve missed until after the shell hits. With a smart-bomb, you have the opportunity to course-correct until the very last second. When your target is (a) moving, and (b) changing direction, you’ll never hit it with a cannon (unless you are very lucky) – your only hope is to use a guided missile – which only works when you continuously update the guidance information.

  10. Pingback: Roger L. Cauvin
  11. Pingback: Steve Johnson
  12. Pingback: Stephan Haux
  13. Pingback: Rich Mironov
  14. Pingback: product anonymous
  15. Pingback: Skip Cody
  16. Pingback: Tim Johnson
  17. Pingback: Karol M McCloskey
  18. Pingback: Dov Bigio
  19. Pingback: UX Feeder
  20. Pingback: UX Factory
  21. Pingback: Max Leisten
  22. Pingback: Markus Grundmann

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>