Measuring the return on investments in design may be the hardest ROI calculation you can do. It certainly is one of the rarest. To measure ROI, you have to be able to determine what would happen without the investment, and what happens with the investment. The difference between them is what happened because of the investment.
Fast Company Interview
Bill Breen posted an interview with Rob Wallace on the Fast Company blog about proving the value of design. The start of Bill’s article lets us know that measuring the ROI of design (ROD) is so rare that it may be impossible or impractical to do. He points to a Whirlpool survey of 15 design-focused companies – most of whom were “clueless” about the return on their design investments.
Rob explains that the reason for measuring the ROI of design investments is that ROI is the language of executives. “If you can’t measure it you can’t manage it. Businesspeople operate in a world of numbers.” Simply put, if you want to make someone invest in design, you have to show them why they should invest in design. One way to get a company to invest in design is to convince the benevolent dictator in charge of the product (or company) of the subjective merits of having good design. Apple is a great example of a company with a leader who is passionate about design. And you can argue that the Zune (from Microsoft) has had meager success against Apple’s iPod due to shortcomings in design. But can you objectively argue the point?
Argument by Extension
After Rob mentions that there is a dearth of measurement of design ROI, Bill challenges Rob’s premise – why should we expect that there is an ROI for design? Rob cites a series of studies done on Fortune 500 companies:
On average, based on two dozen case studies with Fortune 500 companies, for every dollar invested in advertising, packaging and promotion, and visual communication at the point of sale, companies realized a $7.21 ROI. But when the advertising didn’t change (or there was no advertising)—and packaging design was the only thing that did change—there was a $15.17 average ROI on every dollar invested.
Based on this data, Rob argues, it is easy to extend that if packaging (design) changes can yield ROI, then so to will product design changes.
Insights from Adaptive Path
Peter Merholz, President and founding partner of Adaptive Path wrote on communicating the value of design in Aug. 2002. In his article, based on a breakout session, the AIGA’s 5th Advance for Design Summit, he shares a series of internal benefits that they found to be more significant (or at least more measurable) than external benefits.
As we continued listing the value of our user experience design work, a more intriguing realization emerged — the bulk of our value comes from the efficiency that we can create in a company’s operations.
Some of the areas they identified are:
- Smarter Product Development Processes
- Lowered Maintenance Costs
- Less Internal Documentation
- Maximized IT Investments
- Scalability
Our Concerns
One challenge for each of these sources of ROI is that you can’t truly isolate the benefits – you would never have two teams design products for the same market, with differing levels of design investment, with a resultant analysis of project costs. Even if you did, you wouldn’t be able to isolate the impact that the team members had. You have to look at historical data, which introduces enough variables to question the validity of any quantitative conclusion.
Further, we will show that each of these is either a weak proposition, or potentially very false one.
Smarter Product Development Processes. No one has concretely identified a design-centric, purely agile, or structured requirements approach as being the best one. At least where best is defined as most profitable. Even the debate between Alan Cooper and Kent Beck highlighted more “stylistic differences” than quantified differences. There are elements of genius in each approach. Incorporating the needs of users is key to an effective design. Incremental delivery and incorporation of feedback is incredibly efficient. Tracing decisions, designs and requirements to the ultimate goals for the software is key to building the right software. If there is a “perfect process”, it is one that incorporates elements from all three disciplines. And as such – there is value to incorporating design into the process. We can’t quantify it, but we are convinced that it is there. And based on the economics of when software problems are fixed, we believe that investments in design early in the cycle will yield positive returns.
Lowered Maintenance Costs. There are a couple ways this benefit might be inferred. First, better design yields easier to maintain code. This is primarily a premise of agile development (where the design decisions are what Cooper calls program design), but perhaps the benefits also apply to interface design. Often, good interfaces simplify the complexity of, or at least obscure the means of implementation. Those interfaces are not necessarily easier to maintain. One reason complexity leaks through to users in a lot of software is that it is easier to code it that way. This value-prop probably doesn’t hold water.
Less Internal Documentation. The better the design of the interface, the less you need to teach someone how to use it. There is definitely potential for reduced external documentation, or producing the same amount of documentation may cost less, as it is easier to understand the product. It isn’t clear how internal documentation needs can be reduced by good design. Except perhaps that a picture is worth a thousand words. But a great picture may take as long to draw as it does to write a thousand words.
Maximized IT Investments. This is simply self-referential. Maximizing investments yields higher ROI. But in what way?
Scalability. Better design, from a user perspective does not imply that software will be more scalable. And often in an implementation, when user-tasks are simplified (by automating steps in the process), those decisions don’t disappear. By removing the burden from the users, you often add it to the software. However, Peter may have been talking about scalability of the team – the ability to leverage design investments across products (sharing branding, metaphors and workflow paradigms). In other words, design re-use. In that case, there is definitely a benefit, but re-usable design, while cheaper than green field design, is more expensive than no design.
Support Costs. One external benefit that Peter does identify is the reduced burden for training and support. Many software companies actually have profitable businesses built on support and training. Unfortunately, reducing the demand for those would reduce the profitability along that dimension.
Where We See Value
Fundamentally, we don’t perceive design investments as being significant cost reductions – we see them as significant revenue enhancers. Better design yields increased usage, more efficient usage, increased satisfaction, and increased word of mouth marketing.
Increased Usage. ROI forecasts are often built upon estimations (or suppositions) of levels of user adoption. As an example, consider the “Buy it now” button on eBay auctions. When people click on the “Buy it now” button, the auction is closed, and the transaction occurs at a fixed price. People are more likely to sell an auctioned item (generating a commission) by encouraging an impulse shopper. The seller risks getting less than they otherwise would, in exchange for the buyer knowing that they can get the item they want. Fewer auctions will expire unexecuted. More commissions are generated for eBay. Comparisons of final selling prices (and commissions to eBay) of equivalent items with and without the button can be done to measure the return.
More Efficient Usage. Better designs make it easier for users to do what they want to do. Easier often equates to faster. A call-center application that simplifies data access for the operator will allow them to process calls more quickly. The employer ends up with higher throughput from the call center. This throughput can be used to reduce costs, or increase customer satisfaction (with reduced “time per call” stats, or better experiences for the callers), or some of each. Comparing the results of operators using the new system versus the old system will yield measurable data.
Peter mentioned reduced training costs – and we see that as an external benefit that comes from accelerating a user’s growth from new user to proficient or even expert user. Bridging the canyon of pain to get to higher usage rates and more efficient usage faster. This also leads to increased satisfaction by creating more satisfied users more quickly.
Increased Satisfaction. Customer or user satisfaction surveys may be the only way to approximate (not really measuring) this benefit. And it isn’t clear that X% increase in satisfaction leads to Y% increase in sales. But it is likely to lead to increased sales, and may also lead to increases in sales of other products by the same company. Apple leverages this affect across its family of synergistic products. Perhaps a Bayesian analysis of survey results and sales stats could identify the strength of a correlation between satisfaction and sales. That would be expensive to do, but would lead to quantified estimates of the return.
Ultimately, you raise the utility of the software for the user. By increasing the value to the user, you increase their satisfaction, and thereby increase the likelihood that they will encourage other people to buy and use your products.
Increased Word of Mouth Marketing. When your customers become your sales people, you get more sales than you would have without those recommendations. Perhaps comparisons of growth curves for products with varying degrees of customer satisfaction (or varying degrees of “free publicity” – like reviews, blog articles, etc) would yield some predictive insight. Web 2.0 successes have been predominantly built on word-of-mouth. And their growth curves are exponential (as would be the propogation of good words from satisfied mouths). Does anyone know of analyses that validate this?
Conclusion
Design investments yield benefits. The way to measure those benefits is still up for debate. Some folks believe the benefits are primarily internal – we believe that the benefits are external
The problem is that website design is still in the “art” side of the house. Website design should be done just like ad copywriting. They’re both creative tasks, but one is held to a different standard.
When contracting for copywriting for an ad, a company may very well hire multiple copywriters. They then would place the ads wherever and see which one got more of the desired result.
That ad becomes the “control” and they constantly test new copywriters’ ads against the “control”, always seeking to get better response.
With modern technology, companies can do a simple A/B Split on web traffic and test out two (or more) different designs to see which one has the most of the desired effect (more usage, more traffic, more purchases, more requests, whatever).
Dan
Hey Daniel,
Thanks for reading and commenting. Great point about A/B testing. That was one thing I always felt like Amazon did correctly. When rolling out changes to their site, they would randomly route select customers to the new version of the software – allowing them to track specific statistics, like how many people complete transactions (measuring the funnel). Or if version X of the cross-sell design is more profitable than version Y.
Return on design is obvious. A company turns to art when it has no other differentiator. Brand is a matter of commoditization. It helps before commoditization, but it becomes essentail after commoditization. Without art, the company would die.
If you could depreciate products on the basis of the last time they differentiated themselves and the length of time that the differentiation held or earned a premium, you could easily see that other offer elements like design were earning their keep.
Return on design is obvious in the economically indifferent outcomes. If you are earning more since you went art, then blame it on art. It’s pretty hard for some founders, if they are still around, to let art be the winner, but it is a phase specific issue. It is not always the answer. Nor, does it necessarily hold for long.
The brand dependent company is also promo spend dependent. Eventually, even art is copied, so it ceases to differentiate.
Hey David, thanks for chiming in. One of the downsides of ascii is that it can be hard to pick up the tone of someone’s comments. I apologize, but I’m just not quite picking up what you’re throwing down.
Do you see this design-ROI discussion purely as “payback for art” aka “make it pretty and I’ll give you a dollar?” I ask because it seems like the argument you’re making is that “making it pretty has value, but that value is subordinate to everything else.”
If so, then totally my bad – I’m really trying to talk about all of the elements of design from aesthetics to affordances to information architecture to usability. I was trying to make a much bigger point, and I guess I didn’t.
One of the fun things about revisiting an article after 18 months is that you get to look at your examples and anecdotes from a fresh perspective. In the article above, I called out the “buy it now” button on eBay as a design element that positively affected eBay.
While that may have been true in the short term, there is a perception now is that this may have been when eBay jumped the shark. The “buy it now” button, while having that immediate effect on auctions, also had the unintended (?) consequence of allowing people to force a storefront square-peg through an auction-site round hole. Many auctions look like an online shopping mall, where the auction-dynamics are overwhelmed by the mall-perception.
Will eBay win versus craigslist for people who still want that “auction experience?”