You have an eCommerce site.  You sell products online.  Do you cross-sell additional products?  Do you upsell to better products?  This article explains the difference between cross-sell and upsell, and looks at some real-world data about the effectiveness of both.

Cross-Sell and Upsell – What Are They?

Cross-selling and upselling are marketing techniques that are applied during the sales process to increase the value of the transaction to both the buyer and the seller.  Technically, they only increase the value to the seller – but they should also be increasing the value to the buyer.

The key idea behind both cross-selling and upselling is that you are changing a transaction that is already in process.  You might think that marketing ends once buying begins.  Not so.  Marketing is about getting the right message (buy something from us) to the right person (someone who needs our products) at the right time (when they are ready to buy).

• What better time to market than when someone is in the process of buying already?
• Who better to market to than someone who is in the process of buying from us?

The trick then, is in sending the right marketing message.

Cross-selling and upselling only make sense in the context of an ongoing sales process.  For an eCommerce retailer (a company that sells a product online), that means that the customer (technically, the prospective customer, also known as a prospect) is in the process of making a purchase – either looking for the right product, evaluating a specific product, or having selected (but not yet purchased) a product.

• Cross-selling is defined as selling an additional product when the customer is purchasing the original product.
• Upselling is defined as selling a more expensive product instead of the product that the customer was originally purchasing.

As a retailer, you have to know when to attempt to cross-sell, and when to propose an upsell – and when to do both.  To decide when to try and modify (and risk losing) a sale, you have to look at the economic impact on your business of trying to change an ongoing sale.  Cross-selling does not help you make a sale that you wouldn’t already have made – although an upsell suggestion may help a customer discover a “better” product for their needs, and close a sale that would have been abandoned otherwise.

Note – to keep the language of this article easier to read, the word “product” is being used to represent (traditional) products and services – anything a business would sell.

Economics of Selling

To evaluate the economics of cross-selling, you have to first establish the economic measures of selling.  When you sell something you have the following:

• Price - the price the customer pays for the product being purchased.  This is also known as revenue.
• Cost - the cost to the merchant to acquire or create the product being purchased.  Also known as “COGS” – an acronym for cost of goods sold.
• Gross Profit – the price paid by the customer minus the cost of the product.
• Cost of Sale – the cost the merchant incurs to make the sale.  For an individual transaction, this is called “cost per quote” or “cost per order” or “cost per sale.”
• Net Profit – the gross profit minus the cost per sale.  This is also known as operating income.

As an online retailer, you will likely track all of the above.  Your Cost of Sale is potentially difficult to measure – you will probably have a mixture of variable costs and fixed costs that can be allocated to the cost of sales.

• If you pay $1,000 per month to host your eCommerce website (making sales possible) and you make 1,000 sales per month, you could allocate $1 per sale as a cost per sale.
• If you pay 2.5% of the price collected to a credit card processing service, and you sold a product for $100, you would incur a $2.50 cost per sale for that transaction.

A financial analysis of your business will involve aggregating all of the revenue and costs, and calculating the total operating income (all revenue minus all costs) for a period of time.  Since you sell different products (with different costs) at different prices, any given transaction will have a different net profit.  As part of managing your sales and pricing, you may also measure

• Average Revenue per Order - 100 orders for a total $1,500 in revenue would yield and average revenue per order of $15.  Calculated as Revenue / Number of Orders – in this example $1,500/100 = $15.
• Average Gross Profit per Order – 100 orders at $1,500 in revenue with $1,100 in COGS would yield an average gross profit of $4 per order.  Calculated as (Total Revenue – Total COGS)/Number of Orders – in this example ($1,500 – $1,100)/100 = $4.
• Gross Profit Margin (Percentage)- a product purchased for $100 for which the retailer paid $60 to acquire the product has a gross profit margin of 40%.  Calculated as (Revenue – COGS)/Revenue – in this example ($100 – $60)/$100 = 40%.

Economics of Cross-Selling

Cross-selling is when you convince a customer (who is in the process of buying something) to buy an additional product.  When you successfully cross-sell a product, you are increasing the revenue for the order.  This results in an increase in average revenue per order.  The sale of the additional product will also increase the average gross profit per order.  The cross-sell may increase the gross profit margin of the order, or it may not.  When the product originally being purchased is less profitable than the additional product being cross-sold, the margin is increased.  When the original product is more profitable than the additional product, the margin is decreased.

If your current operations strategy involves increasing your profit margins, you need to make sure your cross-sell activities only recommend additional products with higher margins than the products against which they are being cross-sold.  When your strategy is prioritizing growth over profitability, your cross-sell activities should focus on conversion – increasing the percentage of the time are you successfully cross-selling additional products.

Economics of UpSelling

Upselling is when you convince a customer (who is in the process of buying something) to buy something else – specifically, something more expensive.  This replacement of the original item with a new item is known in economics as product substitution.  Since the products are not identical (one is more expensive and presumably “better” than the other), the products are by definition imperfect substitutes.  A perfect substitute is one that would be identical to the product it replaced.

[Update: Check out Complementary and Substitute Products for more on product substitution.]

Successfully upselling a product results in an increase in revenue, and ideally an increase in profits.  It may also result in an increase in profit margin (but may not).  Consider a customer who intends to purchase a 200GB hard drive for $100 (at a cost of $45).  This purchase would yield $100 is revenue, and $55 dollars in profit at a 55% profit margin.  If you successfully upsold the customer to purchase a 500GB hard drive for $200 (at a cost of $100), the purchase would yield $200 in revenue, and $100 in profit at a 50% profit margin.

You have to understand if your strategy is prioritizing an increase in revenue, profits, or profit margins.  This will determine which upselling recommendations you want to make to customers.

Measuring Cross-Selling and Upselling

In addition to measuring your sales you want to specifically measure the impact that cross-selling and upselling has on your measurements.  Those measurements are described above.  You may be breaking those measurements down by product category, product price levels, market segments, or any other decomposition that helps guide future decisions.  You also want to measure the effectiveness of your cross-selling and upselling solutions.  You do that by measuring conversion – the percentage of customers that change their purchases in response to your cross-sell and upsell marketing.

Get Elastic, the eCommerce blog, shares some 2009 survey data provided by the e-tailing group on cross-sell and upsell conversion statistics.  Two-thirds of retailers that measure cross-sell and upsell conversion rates reported less than 5% conversion rates.  At the same time, Get Elastic reports that Amazon reported 35% of their 2006 revenue came from cross-sells.

If you’re adding cross-selling and upselling capabilities to your eCommerce website, you should set your initial expectations of effectiveness low, and your aspirations high.  You won’t start out with results like Amazon’s – no more than 98% of retailers that measure conversion see results far lower than Amazon’s.  In fact, most of them see results an order of magnitude smaller.

When prices go up, demand goes down.  But how much does it go down?  Price elasticity of demand is the term economists use for the math that describes this behavior.

Cause and Effect

Prices go up and sales go down.  Prices go down, and sales go up.  There is definitely a correlation – and economists will argue that it is cause and effect.  I think they’re right – and here’s the rationale.

Imagine an apple, a group of people who are interested in buying that apple.  Each person would be willing to pay a different amount for that apple, because each person assigns a different value, or utility, to having that apple.   Even two identical twins who normally would value the apple the same might value it differently right now.  Imagine that one of our twins has not eaten in a couple days, and the other just finished a very large meal.  They would realize different amounts of utility right now from that same apple.  Or imagine a millionaire and a pauper, who value apples equally, but place less importance on a small amount of money.  Or imagine someone with a bushel of apples, and someone with none – to whom is “one more apple” worth more?

Because different people are willing to pay different amounts (for different reasons) to buy an apple, we have the effect of demand (a proxy for eventual sales) that varies with price.

At price X, half of this group of people would be willing to buy the apple.  Some people might see it as a great bargain, while others are only just barely willing to pay the price of X.  There are also some people who are almost willing, but the price is just a bit too high.  And finally, some people for whom a price of X is just unreasonable.

If you lower the price, all of the people who were previously willing to buy the apple are still willing.  The people who were almost willing before will now buy the apple.  If, on the other hand, you had raised the price, the people who were just barely willing to buy will instead not buy.

The graph above shows price on the vertical axis, and quantity on the horizontal.  When you lower the price from P0 to P1, the quantity that will be sold increases from Q0 to Q1.

Price Elasticity of Demand

When you’re talking about pricing impacts on demand, you’ll usually just say “price elasticity” but it is worth remembering the “of demand” part.  For example – there’s also Price Elasticity of Supply.  Think of the apple example above.  Imagine that the group of people are all selling apples, and there’s a single buyer.  Depending on the price the buyer sets, some people would be willing to sell, and others unwilling.  Change that market price, and the number of available apples for sale will change.  For now, we’ll focus on the impact of changing prices on demand.

The reason price elasticity is interesting is because it gives us some insight into how much the quantity will change when prices change.

Economists have a couple formulas for price elasticity (of demand), including e = % change in quantity / % change in price.

When the relative quantity changes by exactly the same amount as the relative price, you have e=-1, referred to as unitary elasticity.  If you reduce your price by 10%, you increase the quantity sold by 10%.

As a numerical example, you price bags of apples at $10 each, and are able to sell 20 bags of apples for $200.  With e=-1, unitary elasticity, if you lowered the price to $9, you would then sell 22 bags for $198.  If you had raised the price to $11 per bag, you would have sold 18 bags for $198.  If you dropped the price to $5 per bag, you would have sold 40 bags for $200.  Ignoring rounding errors, you would always have the same revenue.

The areas of the two rectangles formed by matching price and quantity are identical.  This leads to the conclusion that you will get the same revenue (assuming you can meet demand) regardless of the price you set for your product.  However, don’t worry about this too much – as far as I know, no real-world products have unitary elasticity.

Changes, More or Less

Elasticity is still useful for understanding the magnitude of change in demand that results from changes in price.  Does demand change by more than your change in price, or by less?

An inelastic demand curve has an elasticity value between 0 and -1.

The resultant relative changes in quantity from a given change in price are smaller than the change in prices.  You can see from the steepness of the cuve in the inelastic demand curve above that a large drop in price (from P0 to P1) results in a small increase in quantity sold (from Q0 to Q1).  However, the converse is also true – a relatively large increase in price will cause a proportionally smaller decrease in demand.

This tends to be the case with products that people “must have.”  In other words, people don’t have good alternatives (substitute goods, or doing without).  When the price of gas doubled, did you reduce the amount of gas you purchase by half, or by less than half?  On the whole, demand for gas dropped by less than 50% when the price doubled.  Another example is anti-biotics.

An elastic demand curve is one where the quantity is very sensitive to changes in price, and has elasticity values below -1.

Small changes in price result in large changes in quantity with an elastic demand curve.  This tends to happen when there are reasonable alternatives to purchasing the product.  For example, when mass-scale production of chicken started and poultry prices dropped in the US, demand increased faster than prices dropped.  This happened at the expense of beef and pork purchases.

Conclusion

Understanding price elasticity of demand is important to making decisions about pricing your products.  There are many factors that determine price elasticity.  The value of your product is a key driver, as it determines the utility curve on which a customer places your product – how much is “what your product does” worth to your potential customer.  Understanding the alternatives your customer has are important – are you selling a commodity product like typing paper, or a unique product like a Tesla?  The more “perfect substitutes” there are for your product, the more elastic your demand curve will be.  If the problem you’re solving can also be solved in other ways, you are again facing substitution.  A pencil is an “imperfect substitute” for a pen, for customers that care only about recording messages on paper.  These also tend to make your demand curve more elastic.

In Predictably Irrational, the authors identify several pricing concepts, like anchoring where people are psychologically influenced to believe one product is more valuable than another.  They look at some very interesting experiments that appear to highlight things that influence people’s perception of utility – making them willing to pay different prices.  An interesting extension of their ideas is that you can influence the price elasticity of demand for your product through positioning and marketing actions.

From The Customer’s Point of View

If you’re selling software as a service, you hopefully don’t need a Foundation Series introduction to SaaS.  If you’re considering purchasing software as a service, then maybe you do.  This article is not an article about the challenges of product management for a SaaS product, nor is it about multi-tenant architectures and other SaaS implementation details.

This article looks to simply compare, from the customer’s point of view, software purchased as a service versus software purchased as a product.

Software as a Service?

Software as a service, is an interesting name.  It implies that instead of purchasing the software, you are purchasing a service – and that service is the ability to use the software.  You are also (usually) purchasing a hosting and infrastructure service.  The SaaS provider will maintain the hardware, perform upgrades, backup your data (sometimes), and otherwise perform all of the “keep the lights on” services and activities required to keep the software running.

Imagine a typical, 1990’s style software purchase.  You buy a source code control system.  You set up a server somewhere, and install the software.  You have ongoing support costs – providing power to the server, keeping the server cool, applying security and operating system updates to the server.  You have costs associating with administering the hardware.  You also, when you get updates to the software, and when you purchase upgrades to the software, have to spend money (labor costs) to update the software.

You also carry the risk of a botched upgrade.  And you carry the risk of a hardware failure causing downtime, or causing you to lose data.  You also have to bear the costs of security – do you allow your people to access the software (on the server) from other computers on your network?  Do you allow them to access the software when they are not on the network (travelling, working from home, etc)?  You have to invest in designing and maintaining a secure system – to prevent your competitors from stealing – or even worse – destroying your data.

Now imagine that you’re outsourcing all of the “keep the lights on” activities above.  You pay an IT services firm to manage the hardware and the software for you, including the security model – and you just use it.  That’s one of the benefits of purchasing software as a service.  To really grasp the economics of SaaS you have to contrast it with the economics of software license purchases.

Widespread Misunderstanding

There is a widespread misunderstanding about purchasing software.  In the last section, we used the word “purchase” but that isn’t actually correct.  You don’t purchase a copy of the software – you purchase a license to use the software (with restrictions).  You probably have heard the phrase “site license”, which means that you are purchasing the right for everyone in your building (or company) to use the software.  Sometimes software is sold in terms of “numbers of seats” – the number of people that are licensed to use the software at any one time.  You could have 100 engineers, who all share ten seats (single-seat licenses) of analysis software.  Since each engineer only spends about 5% of their time using the software, they can easily share licenses.  At any given time, five engineers (on average) will need to use the software.  With a license for ten simultaneous users, each engineer is likely to be able to use the software whenever they desire.

The point of these examples is to point out that even when you think (or say) you are purchasing software, you aren’t.

Now that we have that behind us, the idea of purchasing a service is not as different from purchasing a license.  In neither case are you purchasing software.  In both situations, you are purchasing the right to use the software.

Economics of Software Licensing

There are an infinite number of creative ways to purchase a software license.  The most common situation is that you purchase a license, and then later purchase upgrades.

An obvious example is Microsoft Office (productivity software).  Microsoft releases a new version of Office every couple of years.  If you own the previous version, you can purchase an upgrade for less than the cost of buying the software for the first time.  You are not required to purchase an upgrade, but you may want to.  If the people you work with all upgrade, you may want to upgrade too – so that you can use the documents that they create.  Microsoft does a good job of providing free utilities to read documents from the newer versions, and allowing people with newer versions to create documents that can be used by people with older versions.  Microsoft, therefore, gives you a choice.  They rely on market forces to create the pressure to upgrade, but you never have to upgrade.

Intuit, makers of Quickbooks (small business accounting software), is a little pushier.  They release a new version of the software every year.  And you can continue to use your old version.  Unless, however, you use one of the services that they also sell.  Intuit sells services that integrate with Quickbooks.  And at least with some releases, when a new version of Quickbooks comes out, they will stop supporting the integration of those services with older versions of Quickbooks.  If you need those services, you must upgrade.

When companies sell software (licenses), they usually sell a version of the software, and then make updates to that software with some frequency, from daily to annually.  Companies also manage those updates as two distinct types of updates – major and minor.  Minor updates are usually free, and major updates usually require you to purchase an upgrade.  Minor updates might be bug fixes, or features that were intended to be in the major release, but were delayed.  Or they might just be the introduction of capabilities with “small” value to their customers.  A lot of software will automatically notify you, download the update, and install it for you.  That’s great service.  Major updates are usually more significant – they introduce capabilities that have “large” value to their customers, or are intended to make the product appealing to additional markets.

To understand the economics of software license purchases, you have to look at both the value over time and the costs over time of purchasing a software license.

To keep this simple, we’ll assume the model described above – minor updates happen frequently and are free, and major updates require the purchase of an upgrade to the latest version of the software.  We’ll also assume that every new update introduces something valuable to you as a customer.

Starting with the value model for the software, from the software company’s perspective:

The axes represent increasing value (up) versus the passage of time (to the right).  Time starts when you purchase a license to use the software.  You’re immediately getting value.  As minor updates are made (and minor releases are made – sharing the updates with customers), the value gets marginally higher.  Whenever a major release (a major update) is made, the value curve jumps up dramatically.  This represents the introduction of new, valuable capabilities.

As each new customer purchases a license to the latest version of the software, the company gets more revenue.  As each existing customer purchases upgrades to their existing software, the company gets more revenue.  A company makes money from finding new customers and from keeping existing customers.  Companies make more (per purchase) from finding new customers than from getting existing customers to upgrade.  Until a product builds a large base of existing customers, the company’s financial focus will be on finding new customers.  Satisfying existing customers is at risk of becoming a secondary priority, purely based on economics.  And yes, this is an incomplete picture, there are indeed other factors.  But it is absolutely an influence.

Software License Benefits

We started this article with a promise to look at this from a customer’s perspective.  If the model above represents how a software company views its products, here’s how a customer would view the same thing.  Remember – in our example, we have a perfect product manager – every update has the same perceived value to customers as it does to the company.  This chart shows the same value-model, but overlays your purchasing behavior as a customer.

As a customer, you make an initial purchase (the left-most callout), and then get free incremental increases in value from each minor release.  You also purchase an upgrade to the latest version of the software as soon as it is available.  You then start getting incremental value from the minor updates to that version of the software.   The older version does not keep getting updates, so if you don’t purchase the upgrade, you don’t get the benefits of the latest minor releases.  A second major release happens, but you don’t purchase it for a short while.  Then a minor release is made, with a fix to an annoying bug that really bothers you.  So you purchase another upgrade.

The value to you looks like the following:

The green area represents the value you get*.  Notice that you did not get the value of the last major release until you actually purchased it.  You also did not get the incremental value of minor releases that happened after that release.  Once you did purchase the upgrade, you immediately got the benefits of all the minor releases.  You got “back in the game.”

*The value is often a function of how much you use the software (enabling the benefit), and as such, it is a function of time.  The more you use it, the more value you get.  So showing this as an area is informative.

Software License Costs

Your costs over time are also important.

The obvious costs are the checks you write to the software company to pay for the software and for the major upgrades.  The chart above can be a little misleading – we are depicting “one time costs” that add up over time.  Showing this as a stair-step area instead of a series of spikes will make sense in a moment.  The key thing to appreciate is that once you make a purchase, your license purchasing costs do not go up again until you make another purchase.

At the start of the article, we identified several “cost of ownership” costs – supporting the software and hardware, for example.  It is critical that you keep those costs in mind when evaluating software license purchases.  These costs are ongoing costs – the more you use the software, the more the costs add up.

There are also training costs – the cost of lost productivity as people learn to use the software and adapt to changes in the software.

When you combine these, you get a model for the total cost of purchasing a software license over time.  The jargon term for this is Total Cost of Ownership (TCO).

As you can see, “purchasing” software one time actually has a continuously increasing total cost of ownership.  Different types of software will have different relative costs for infrastructure support, training expense, and license fees.  But generally, training expenses are much lower than the other costs of ownership.

SaaS – A Simple Definition

Software as a service is usually provided as follows:

1. A company creates a software product and hosts that product on multiple servers.  The company manages the hardware and software – and realizes the cost of that management.
2. Customers subscribe to the service – getting the right to use the software, for as long as they continue to pay the recurring subscription fees.
3. The company makes both major and minor updates to the software, and the customers automatically get those updates as part of their subscription.

There are many examples, some obvious ones are Salesforce.com, Kadient’s inciteKnowledge, and 37signals’ Basecamp.

SaaS Economics

Software as a service is purchased with the same mechanics as subscribing to a magazine or cable television or satelite radio.  You pay a recurring fee for the right to use the software, just as you pay a recurring fee for the right to watch cable television.  You might even get a discount for purchasing a longer-term subscription and paying up front.  When you want to stop using the service, you stop paying the fee.

The model for creating value with SaaS products is the same as with licensed software.

The graph looks the same as the first one in this article, because it is the same graph.  Where things change slightly is in the value model from the perspective of the customer.

There is a single trigger to the realization of value – starting the subscription.  You automatically get the minor and major updates “for free” by continuing to pay the subscription fee.

SaaS Costs Are Different

Where software as a service differs from the purchase of a license to use software is on the cost side.

The training costs associated with using software have nothing to do with the mechanism for payment, so those costs are the same.  The cost of subscribing to the service (blue cross-hatched region) is new, and goes up over time.  It is also typically higher than the training costs.  Note that there are no “large purchase” spikes in the costs – because you never purchase a license.  And there are no infrastructure costs, because the company that provides the service realizes those costs.

The idea is that this approach is more cost-effective when it comes to infrastructure costs, so the company can pass on those savings to you.

Comparing the two cost models side by side is the way to really see the difference:

Both models have costs that increase over time.  For many technical reasons, the SaaS architecture is more efficient and has lower costs for the software company – which tends to result in lower costs for the customers.  This is not always the end result, but it is directionally correct.

Another interesting factor to consider is the financial pressure on the SaaS provider.  Where a software licensing model creates pressure to prioritize finding new customers, a SaaS model creates pressure to keep existing customers.  SaaS providers get the same revenue from a new customer as from an existing customer, instead of the “new vs. upgrade” dynamic seen with software licensing models.  It is always cheaper to keep an existing customer than to find a new one.  The net result – financial pressure to retain existing customers.  This can drive a different behavior, more like that of a retail sales model, where keeping your existing customers is critical.

Conclusion

The SaaS model ultimately provides the same type of products as a software licensing model.  But with a better economic model – one that is lower in cost to me, and one that is structurally inclined to keep getting better for me with every new release.

Personally, I really like the idea of purchasing from a company that is financially motivated to keep me happy, not one who’s pressured to find another customer as soon as I’ve written my check.

The best companies try to reinvent themselves and improve their products continuously.  Over time, the best companies will move to SaaS models, which align their financials with their objectives.

Check out the index of the Foundation Series posts for other introductory articles.

A Simple Experiment

Studies have shown that the temperature of soil has an effect on the germination rates of seeds. But does soil temperature affect the rate of growth of plants? You decide to find out, and you pot a bunch of grass in individual containers, stick thermometers in each pot, and measure the grass height and temperature every day. You spread the pots out in your back yard in areas that get different amounts of sunlight. The pots that get the most light are hottest, the ones that get the least light are the coolest. All plants get the same amount of water. Assume all of your data from measurement is accurate.
Imagine that you had data that showed the following combinations of soil temperature and plant growth.

• Soil temperature: 70 F. Grass growth: 1 inch
• Soil temperature: 72 F. Grass growth: 2 inches
• Soil temperature: 74 F. Grass growth: 3 inches
• Soil temperature: 76 F. Grass growth: 4 inches
• Soil temperature: 78 F. Grass growth: 5 inches
• Soil temperature: 80 F. Grass growth: 6 inches

Can you conclude that warmer soil temperatures between 70 and 80 degrees cause grass to grow more?

That is the common mistake that many people make when looking at data.

You could just as easily conclude that faster growing grass causes warmer soil temperatures, and you would be just as correct. Or rather, incorrect.

Correlation

Correlation between two sets of data means that they have related trends. The data in our example above are correlated. Rising temperatures coincide with increasing growth rates. Increasing growth rates coincide with warmer soil temperatures.

When you can not say that one factor caused the other, you have a correlation.

Consider another experiment – you plant the same grass in the same pots. This time, you install heaters under each pot, and you control the temperature of all of the pots. You create pots with each of the different temperatures you observed in the previous experiment. Then you put them all of the pots in a dark closet with no light.

None of the grass grows.

Causation

The grass didn’t grow in the closet because the warmth isn’t what caused the grass to grow.

In the first experiment, the pots were placed in areas of the yard that got varying amounts of sunlight. The sunlight caused the grass to grow and it caused the soil to get warmer. If you had measured the amount of sunlight, as well as the soil temperature and the grass growth you would have seen three values that were correlated.

To confirm that sunlight was the causal, or independent variable, you can inspect the pots that were warmed in the dark. You could also run an experiment that combined the variation in sunlight with constant soil temperatures – isolating the sunlight variable. You might even have to do that experiment to rule out the possibility that fresh air contributes to grass growth (because fresh air is present in the outside experiment, but not when the pots are put in the closet).

Jumping to Conclusions

One of the goals in writing software requirements is to make them measurable. Some requirements are easily measurable – the query returns results in 10 seconds, for example. You write a query, test it, and look at the results. If the code is too slow, you rewrite and test again, until you meet the requirement as defined.

Other requirements, while easily measurable, may be hard to control. Consider a usability requirement like “A novice user will be able to complete the task in 2 minutes.” This is easily measurable – but if you run the test, and the novice user takes 3 minutes – what do you change to reduce the amount of time? Do you add interactive help? This would certainly make it easier for the user once she becomes competent – but she will have to spend time reading the instructions when she is still a novice.

[As an aside - measuring the impact of usability investments is very hard - and one of the reasons is the challenge of separating causality and correlation.]
We run the risk of jumping to conclusions when defining these metrics. Image that you are doing an analysis of a shopping cart, as part of designing a new one. You find that 80% of the users who add an item to the shopping cart (from a product page) never purchase the product. You also find that there are 8 steps required to complete a purchase, starting with adding the item to the cart (and then viewing the cart, confirming your order, etc…). You may notice that almost all of these users stay on the site until the 7th step – where only a few ultimately purchase.

You could write a requirement that says that purchasing a product requires no more than 6 steps. If you did, you would be jumping to the conclusion that the number of steps causes the users to abandon the purchase. You would also be specifying design (which is bad). It may be that step 7 requires them to enter their social security number – and that this is the deal-breaker. If your developers implement a solution that only has 4 steps, but keep the social security number requirement, do you expect to get a massive improvement in your funnel?

Writing good measurable requirements is hard, not because measuring is hard (although sometimes it is), but because correlation is often mistaken for causality – and we therefore choose bad things to measure. Choosing what to measure is the hard part.

Utility is an abstract concept usually relegated to economics. What is it? How does it work?

Definition of Utility

The economic definition of utility is

An economic term referring to the total satisfaction received from consuming a good or service

The Free Dictionary

That doesn’t helps us very much – but visuals make this concept strikingly clear.

Consider money. Everyone likes money. People derive satisfaction, or utility from money. The more money they have, the more satisfaction they get from it.

Note that the law of diminishing returns is a factor here. As someone acquires more money, the same size of increase in money does not result in the same size increase in utility. $10,000 would provide a lot of satisfaction to someone working at minimum wage. It would provide much less (but not zero) satisfaction to a billionaire. That’s the law of diminishing returns.

Looking at another concept – leisure time. When someone increases their leisure time, it increases their satisfaction.

Again, we have the law of diminishing returns. An extra hour of leisure time in a week is very valuable to a single parent working two jobs and raising three kids. That same hour is less valuable if the person has no responsibilities or time commitments.

Tradeoffs

While both graphs make sense, what they can’t tell us is if someone would rather have more leisure time, or more money. When we present this as a tradeoff, we need a way to describe how people make those tradeoffs.

Our intuition is that people with a lot of free time would rather have more money, and people with a lot of money would rather have more free time. That’s good intuition. When we graph leisure time versus money, we have a utility curve.

Economists also refer to this as an indifference curve. For someone who is on any point on the curve, they are indifferent to being on any other point on the same curve. When a person has very little money, they will trade a lot of leisure time for more money. When they have a lot of money, they will trade a lot of it for a small increase in leisure time.

This is one of the reasons our economy works. People with a lot of time sell something cheap (their time) to people with a lot of money, who will pay dearly for something that is expensive to them (their time).

Not everyone has the same shape of utility curve.

Some people inherently like money more than time, or vice versa. Or a person’s situation is such that they temporarily shift their values. Each individual has a unique curve at a unique point in time. And for that person, all points on the curve provide equivalent satisfaction.

Improvements

OK, understanding tradeoffs is nice, but what we want to understand more is how to improve the human condition. Products are not very compelling when people are indifferent about buying them. How do we make people want to buy our products?

People want to increase their utility. If someone is indifferent to trading an hour for a dollar, they would be excited to trade half an hour for a dollar. Conversely, they are uninterested in trading two hours for a dollar. Graphically, in addition to being indifferent to all points on their curve, they are excited about all points above their curve, and uninterested in moving to a point below their curve.

Rational people (and economists make a point of excluding irrational people from most of their work) will at least be content, and usually prefer to increase their utility.

Why Indifference Curves Never Intersect

You’ll notice that the two curves in the figure above are never cross each other. Indifference curves never intersect, because by definition, all points on the same curve represent equivalent satisfaction. If two curves were to overlap, then that would create a graph (for a single individual) that looked like the previous graph (with red and green curves).

Pick a single point on the “Money” axis. There are two intersections along the “leisure Time” axis – one each for the red and green curves. The red data point must have the same utility as the place where the curves cross. The green data point will also have the same utility as the place where the indifference curves intersect. This would require both the red and green data points (with different values for “leisure Time” and identical values for “Money” would represent the same amount of utility.

That would violate the premise that “more leisure time is always better, when all else is equal.” This also hilights a limitation of using utility curves – they only apply to monotonically increasing functions. Only when “a little bit more” is always better, can a utility curve be drawn.
Practical Application

This is fine, if theoretical. How do we use this as part of developing great software?

Now that we understand how people treat tradeoffs, we can look at prioritization of requirements. The key is to recognize that there are tradeoffs. As creators of software, we are faced with internal tradeoffs. We can only accomplish so much. We can only release a certain number of features in a given timebox. And we make those tradeoffs based on ROI, or our perception of the balance between costs and return.

Sometimes return is measured in purely financial terms. Sometimes, that return is measured in utility. Consider the decision process around a set of usability requirements. Should we make the application easier to learn, or faster to use?

A utility curve helps us visualize this tradeoff.

The answer is some of both. But we have to balance the need to get past the suck-threshold with creating word-of-mouth marketing and driving user adoption by designing for the competent user. Understanding that each user has their personal set of tradeoffs helps us make these decisions.

Inbound product manager or outbound product manager – what is the difference? We’ll look at the overall role, and the breakdown of responsibilities.

Product Manager Role Definition

More often than not, people talk about product managers without the qualifier of inbound or outbound. We wrote an article last spring where we discussed the role of the product manager. Michael Shrivathsan defined the role of product management as being split into six areas:

1. Market Research
2. Product Definition and Design
3. Project Management
4. Evangelize the Product
5. Product Marketing
6. Product Life Cycle Management

For more details on those six areas, check out our earlier article, as well as Michael’s article.

Everywhere At Once

The product manager is responsible for interacting with customers, marketing, and sales teams. This communication helps the product manager develop an understanding of the market. The product manager also researches the market – understanding competitors and existing products. From the understanding he gains, the product manager can identify needs and opportunities.

The product manager then works to turn this understanding of market needs into a set of requirements that drive the creation of a product.

Product managers shepherd the product through its life cycle. Product managers identify how the product will evolve and adapt to changing market needs. The product manager may drive the evolution of the product to compete in additional markets.

Product managers also help prepare the company to successfully sell the product. They define strategies for attacking the target market. They help the sales team understand what the product does, and how to position it against competitors. And they support the sales team in the field.

How do we split this up into inbound and outbound?

Inbound Product Management

The inbound part of of the job focuses on understanding what the product needs to be.
Inbound activities include

• Understanding the market needs and our capability to meet them.
• Defining a product strategy to meet those needs.
• Planning the creation of the product, including documenting requirements.

Outbound Product Management

The outbound part of the job focuses on making sure that the product is a success in the market.
Outbound activities include

• Defining a go-to-market strategy.
• Preparing the sales team with an understanding of the product and market.
• Supporting the sales team in execution with a focus on multi-customer activities.

Soundbite

Inbound product management is more about listening, and outbound product management is more about talking.

- – -Check out the index of the Foundation Series posts for other introductory articles.

JAD is an acronym that stands for Joint Application Design. JAD sessions are collaborative meetings where the customers meet with developers to determine what the product needs to be or do.

JAD Beginnings

Chuck Morris and Tony Crawford of IBM created the idea of JAD sessions in the 1970s (Yatco). JAD sessions were designed to gather good requirements and specifications. JAD sessions are often referred to as JAD workshops. JAD has been used as a name to describe brainstorming sessions, formal requirements gathering sessions, and even interface design meetings.

Today, JAD is commonly used for strategic business planning, strategic IS plans, IS architecture definition, re-engineering business processes, detailed system design, process and data modeling, and project management.This might be the broadest application of JAD so far.

Mei C Yatco

Although different people use “JAD” in different ways to solve different problems, there is a common theme – JAD sessions are facilitated requirements gathering sessions.

JAD Benefit

JAD sessions are believed to reduce the time required to gather requirements and develop software. At the time of their creation, JAD sessions were certainly more cost effective than existing “build, feedback, fix” cycles. It isn’t clear if JAD sessions themselves are more effective than other requirements elicitation techniques. JAD sessions are probably best used as one of many tools for gathering requirements.

JAD Session Structure

A JAD session has a facilitator, a scribe, participants and observers.

• The Facilitator – Guides the participants through the process of gathering requirements.
• The Scribe – Diagrams models, takes notes, records information.
• Participants – Business users and subject matter experts (SMEs) who contribute the content in the session.
• Observers – Developers who observe the process.

There are 9 steps to a JAD session (wikipedia):

1. Identify project objectives and limitations
2. Identify critical success factors
3. Define project deliverables
4. Define the schedule of workshop activities
5. Select the participants
6. Prepare the workshop material
7. Organize workshop activities and exercises
8. Prepare, inform, educate the workshop participants
9. Coordinate workshop logistics

More Reading

A top rated JAD book at Amazon is Joint Application Development by Jane Wood and Denise Silver.

[Update 2007/02/08.  I have really been enjoying the book that Kevin reccommended in the comments below – Requirements by Collaboration: Workshops for Defining Needs by Ellen Gottesdiener.  Thanks Kevin for the great tip!

What is a data dictionary and how is it used when communicating and managing requirements?

Definition

A data dictionary is a collection of the definitions of the structure of information that is relevant to a set of requirements. That’s a lot of words for a simple concept. We need to know (and constrain) a set of information about some business element when managing our requirements. We use a data dictionary to define what that information is, and any constraints on how it must be used.

Viewing The System

When using object oriented analysis (OOA) as part of defining requirements, we represent business concepts as objects and processes. For example, an order management system might define orders as having line items and customers. We can represent that information graphically with a UML diagram like the following:

In prose, we could also capture the same information as follows:

1. The system shall include a representation of customer orders.
2. Each order will have a single associated customer, and each customer can have multiple orders. Note that a customer is not required to have any orders.
3. Each order will have at least one, and possibly multiple line items. Each line item is uniquely associated with a single order.
4. Each line item represents a single product. Note that a product is not required to be represented in a line item. A product can be represented in multiple line items (even within the same quote).

While this diagram tells us about the structure at a high level, it doesn’t tell us enough information to go implement the solution. What exactly is a line item? What information does it contain? And what format must that information be in?

A Dictionary Entry

We could create a data dictionary entry for the line item object as follows:

Line Item

A line item represents a portion of a customer order that describes a product being ordered, as well as the quantity of that product being ordered. Each line item must include the following information:

• A reference to the product being ordered, using the product ID per constraint X1. [Note, the constraint is imposed by the existing product data management system, with which our software is required to integrate.]
• The quantity of the product being ordered, where the quantity is a positive integer. [Note, we would include a maximum value, if there were a constraint imposed by some other part of the system.]

Note that we have not specified that a line item includes a price. It is very likely that a line item would have a price, but we would be specifying implementation details if we did. Pricing may be done per product, or may be unique for each product for any given customer. Discounts may be applied based upon quantity of products in a line item, or dollar amount for an order. Discounts may be applied based upon all products ordered by a customer over a period of time. These different possibilities are a function of the requirements of the system.

When those business requirements are defined, they will dictate the ownership of properties by business objects. With that information, we can include the data as appropriate. For example, a list price property may be defined for the product object, or a customer-price may be defined for a line-item as a function of (product, quantity, customer). We would add that data as part of the business modeling. Note that this is a description of the problem domain, not a description of the implementation.
Another Data Dictionary Example

Here’s an example of a “Customer”

A customer represents the business or person for whom an order has been placed. Note that all character fields are to be represented in unicode 4.1.0 or later per corporate policy ABC. A customer has the following information:

• Name. 50 characters representing the name of the customer.
• Shipping Address 1. 100 characters representing the first line of the address to which all customer shipments are made.
• Shipping Address 2. 100 characters representing the second line of the address to which all customer shipments are made.
• Shipping Country. 50 characters…
• Billing Address.
• Customer Contact.
• etcetera.

This list is intended to show all of the elements of information that must be present in the “customer object” to support the requirements of the system.

Further Reading

Joe, at Seilevel wrote a post back in March with a good explanation of data dictionary entries. As Joe points out, requirements can drive the need for specific information.

For example, my business users have told me that the number of decimal places of each weight value tracked by the system is very important for monitoring and reporting. It stands to reason that other objects and attributes might require the same level of specification. If you figure it out once, you can use it in many places.

Barbara, at B2T Training points out the importance of understanding the details of the data for a system. She also touches on the value of having that information in a separate document.

Many BAs document data as part of the business process or part of the Use Case. Our recommendation is that you document data in a separate part of the requirements package because it is often used in multiple places.

Usage

A data dictionary should be defined as a repository of all data definitions like the examples above. Those examples should be referenced in all requirements documents that rely on the defined objects. Requirements documents should not specify the content of the objects, they should defer to the referenced dictionary entries.

Some projects, especially migration projects, have many constraints tied to data formats and structure. These projects will have extensive data dictionaries, and multiple references to entries throughout the requirements document. Other projects will have far fewer constraints on data formats, but will still have explicit structural definitions for business objects (like our line item example).

- – -

Check out the index of the Foundation Series posts which will be updated whenever new posts are added.

Use cases represent the activities that people do when interacting with a system to achieve their goals. Use cases are a very effective tool for communicating and documenting what a system is intended to accomplish. Formal use cases are use cases with a specific structure to represent the information. Knowing how to read a formal use case is important.

Formal Use Case

We wrote previously about formal use cases, covering the pros and cons with a brief overview, as part of our use case series. In this article, we focus on the elements of a formal use case, and how to interpret them.

When using a structured requirements approach to creating software, use cases represent the activities of classes of users, as they relate to the software being developed. Each goal, or market requirement, will have one or more use cases that support it. The easy way to think about it is that for a company to achieve a goal, someone has to do something (Make the Sale, Answer the Support Call, Update Account Information, etc). The use cases capture those somethings.
Here are the elements of a formal use case, with explanations:

Meta data

This is the information that identifies when the use case was first written and by whom. It may also include revision information, who made the changes, what was changed, and when and why it was changed. Revision history is a common meta data element in most business documents – we just normally don’t call it “meta data.” That’s an example of software jargon.

Title

The title of a use case is more than just a heading like “My Summer Vacation.” The title is intended to provide enough information to identify what the use case represents without reading the use case. This can be useful when getting an overview of a planned project, validating requirements, or when scanning use cases as part of reviewing materials. The title should provide enough information for someone conversant in the domain, while being short. The ideal length is between 4 and 8 words and is a sentance fragment. Anything longer than a sentance is a horrible title. The format is subject verb object. Someone does something to or with something else.
Some examples:

• Pilot performs pre-flight safety check.
• Author adjusts plot element timelines.
• Accountant reconciles accounts-receivable ledger.

Description

A brief description of the activity represented in the use case. One to five sentances – no more than a single paragraph – describing the activity or process represented by the use case. The description does not include the full details, but does provide the “next level of insight” into understanding what the use case covers. Someone who is not experienced in the domain can get a base understanding of the contents of the use case from reading a description.
Some examples:

• A pilot peforms an FAA mandated list of equipment and operational inspections prior to every flight. All inspections must pass before the flight is allowed to take off per corporate policy.
• An author will review the timelines, or sequences of events for each plot and subplot within their novel. The author is assuring that the sequence of intended events is internally consistent, and achieves the desired pacing effects for the book. Any needed changes to the timeline are made.
• Accountants regularly validate that the entries in the accounts-receivable ledger are consistent with the recent sales and current payments. This validation is required for all GAAP public reporting on a quarterly basis, and may be required more frequently for internal management accounting purposes.

Primary Actor

The primary actor is the person who is the subject of the use case, performing the verb of the use case on the object. A use case may have multiple actors but has one most important person. The term actor represents the person who takes action – not someone playing a role. Other actors may be involved, either as participants, or as infrequent or secondary performers of the use case.
Some examples:

• Primary Actor: Pilot. Secondary Actors:Flight Crew, Sr. Maintenance Technician
• Primary Actor: Author.
• Primary Actor: Financial Accountant. Secondary Actor: Financial Accounting Manager

Triggers

A trigger is the outside event that triggers the beginning of a use case. This is an event that initiates a use case. Many use cases do not have an obvious trigger – and the trigger will be documented as “User decides to do X.” This invisible decision represents the trigger.

Some examples:

• Pilot receives flight plan.
• Author initiates review of the novel.
• It is the first day of the last month of the quarter.

Preconditions

Preconditions are the set of things that must be true for the use case to occur. These represent a contract of sorts – the use case will not happen unless all of the preconditions are met. If a situation is optional, it is not a precondition.

Some examples:

• Flight plan has been approved.
• None. (In our example, the author can review timelines before during or after completing a draft of the novel.)
• Accounts receivable entries have been made in the ledger.

Post-conditions

Post-conditions represent the other side of the contract. If the normal course (see below) of the use case has been completed, then these things must be true.

Some examples:

• The plane has been confirmed to be safe and ready for the approved flight plan.
• The timeline is updated to reflect the author’s desired result.
• All accounting entries have been reconciled or identified as being incorrect.

Normal Course

The preferred, desired, or most common sequence of events that represent the use case. This sequence of events must be followable in the solution. If no alternate courses are defined, then this sequence of events must be followed (with no alternatives) in the solution. The normal course is documented as a series of single actor, single action sentences, each with a number.

An example:

1. Accountant accesses current quarter accounts-receivable ledger information.
2. Each of the following steps is repeated for every entry in the ledger.
3. Accountant confirms transaction record matches appropriate other ledger (sales, billing, etc).
4. Accountant updates ledger to indicate that the entry was confirmed.

Alternate Courses

Alternate course are supported alternatives to the normal course. Each variation is identified as an alternative, and each step that varies from the normal course will be explicitly defined within the alternative course. Each variant step is identified with a number, and all unidentified steps are explicitly defined to be the same as they are in the normal course.

An example:

A1: Incorrect ledger entry identified.

A1.3. Accountant confirms that the ledger record does not match the appropriate other ledger.

A1.4. Accountant updates ledger to reflect inconsistency.

A1.5. Accountant identifies proper value to reflect in the ledger.

A1.6. Accountant updates the ledger with the proper value.

A common naming convention is to begin the numbering with the letter “A” to represent that it is an alternate course, followed by a number indicating which alternative it is. In the example above, we have a single alternate course, “A1: Incorrect ledger entry identified”. In this example, steps 1 and 2 of the normal course are identical, steps 3 and 4 are replaced with new steps, and steps 5 and 6 have been added.

If another alternate course were defined, it would be numbered “A2″, and if it provided an alternate for step 3 in the normal course, that step would be numbered “A2.3.”

[Update]

Another situation that happens with alternate courses is that a single step in the normal course can be replaced with multiple steps in an alternate course. To represent this, we add a letter, denoting that the steps are all combined to replace a single step. In the example above, we replaced step 4 and added steps 5 and 6. An alternate course may also be documented as follows:

Another example:

A1: Incorrect ledger entry identified.

A1.3. Accountant confirms that the ledger record does not match the appropriate other ledger.

A1.4.a. Accountant updates ledger to reflect inconsistency.

A1.4.b. Accountant identifies proper value to reflect in the ledger.

A1.4.c. Accountant updates the ledger with the proper value.

Exception Courses

An exception course identifies how the system should behave when something goes wrong in the normal course (or an alternate course). If there are multiple exception behaviors, they will be called out as separate exception courses.

An example:

E1: Unable to access accounts receivable ledger

E1.1. Accountant fails to access accounts receivable ledger information.

E1.2. System presents error message to accountant explaining access failure and reported cause of failure.

An exception course is identified with the letter “E”, and otherwise follows the same numbering conventions as the alternate courses.

Includes

Sometimes use cases are broken down into smaller, reusable use cases (such as login steps, using a search window to find an account, etc). A reference to another use case from the includes section indicates that the referenced use case is a subset of this use case.

An example:

The use case of “Accountant Prepares Quarterly Report” could include the use case of “Accountant reconciles accounts-receivable ledger”.

References/Traces

Each use case is written to support one or more market requirements (goals). The specific goals that are being supported will be identified as references, or traces.

Each use case is also supported by one or more functional requirements. References or traces to each of these requirements will be identified in this section. These references allow product / program managers to understand the impacts of changes and delays in implementation on the overall product delivery schedule.

Assumptions

All assumptions that are implicit in the rest of the use case will be explicitly documented here.

Examples:

• Pilot is certified to run a safety-check.
• Airport allows pilot to run her own safety-check.

Notes

Any text that helps the author or potentially helps the reader is documented as notes. These are not requirements documented in the use case, any more than notes in the margin of a book are part of the book.

Conclusion

With this understanding, reading any formal use case will be a breeze.

- – -

Check out the index of the Foundation series posts for other introductory articles.

Functional Testing, also referred to as System Testing of software is the practice of testing the completed software to confirm that it meets the requirements defined for the software. A functional test is typically a test of user interactions, but can also involve communication with external systems. We contrast functional testing with unit testing. We also show how functional testing provides different benefits than unit testing.

This is a relatively long post for a Foundation Series post, so sit back with some coffee and relax. This primer will be worth it if its a new topic. If you know this stuff already, check out the links to other articles that go into more depth on points we make.

An Application is a Series of Flows

We can think of an application from the perspective of a user, as a series of interactions, or flows through the user interface.

People are not usually forced to follow a fixed set of instructions, or a predefined sequence of actions in an application. They can interact with controls in a random order, skip controls entirely, or otherwise do stuff that developers don’t expect.

Unit Tests are Whitebox Tests

Unit testing, as we detailed in our telephone example, provides targeted test coverage of specific areas of the code inside the application. Unit tests are written by developers, to allow them to test that the implementation that they created is behaving as they intended. Unit tests don’t implicitly provide the start-to-finish coverage that functional tests usually provide. Unit tests are whitebox tests that assure that a specific behavior intended by the developer is happening. A weakness of using unit tests alone is that they will not identify when the developer misinterpreted the requirements.

Functional Tests are Blackbox Tests

A functional test, however, is designed without insight into how the implementation works. It is a blackbox test. A functional test represents a set of user interactions with the application. The concept behind a functional test is to validate something about the state of the application after a series of events. According to Aberro Software, 80% of all functional tests are performed manually. That means that the most common functional test involves a tester making selections in a series of controls, and then evaluating a condition. This evaluation is called an assertion. The tester asserts that the software is in a specific state (an output is created, a control is filtered in a specific way, a control is enabled, a navigation option is disabled, etc).

Good functional requirements are written as concisely as possible. A requirement that supports a particular use case might state that the user specifies A, B, and C, and the application responds with D. A functional test designed to validate that requirement will almost always mimic this most common flow of events. The script that the tester follows will be to specify A, then B, then C. The tester will then evaluate the assertion that D is true. If D is false, then the test has failed.

A functional test may not cover the entire set of likely user interactions, but rather a subset of them.

One problem with this approach is that it does not account for a user specifying (A, B, X, C) or (A, C, B). These variations in order of operations might cause the underlying code to execute differently, and might uncover a bug. For a tester to get complete coverage of the requirement (A + B + C => D), he would have to create multiple scripts. This is expensive, tedious, and often redundant. But a tester has no way to know if the multiple scripts are redundant, or required.

Combining Unit Tests and Functional Tests

When we combine both unit testing and functional testing approaches, we are implementing what is called graybox testing (greybox testing). This is also referred to as layered testing. Graybox testing provides two types of feedback into the software development process. The unit tests provide feedback to the developer that her implementation is working as designed. The functional tests provide feedback to the tester that the application is working as required.

Graybox testing is the ideal approach for any software project, and is a key component of any continuous integration strategy. Continuous integration is a process where the software is compiled and tested every day throughout the release cycle – instead of waiting until the end of the cycle to test. Read this plan for implementing continuous integration if you want more details.

Automating Functional Tests

Automating unit testing is both straightforward, and relatively inexpensive. Automating functional testing is more expensive to set up, and much more expensive to maintain. Each functional test represents a script of specific actions. A tester (with programming skills) can utilize software packages like WinRunner to create scripts of actions followed by assertions. This represents an upfront cost of programming a script to match the application, in parallel with the development of the application – and it requires a tester with specialized skills to program the script.

The maintenance cost of automating functional tests is magnified in the early development stages of any application, and throughout the life of any application developed with an agile process. Whenever an element of the user interface is changed, every script that interacts with that element can be broken (depending on the nature of the change). These broken scripts have to be manually updated to reflect these ongoing changes. In periods of heavy interface churn, the cost of maintaining the test suite can quickly become overwhelming.

In the real world, apparently 80% of teams find that this overwhelming cost of automated testing outweighs even the high cost of manual functional testing.

Improved Automation of Functional Tests

We can reduce the maintenance cost of keeping automated scripts current with the user interface by abstracting the script-coding from the script-definition. This is referred to as keyword and table scripting. A set of objects are coded by the tester and given keywords. Each object represents an element in the user interface. Script behavior (sequence of interaction) is defined in terms of these keywords. Now, when a UI element is changed, the keyword-object is updated and all of the scripts that reference it are repaired.

This, however, does not address issues where one control is refactored into two controls, the adding or removing of controls, or changes in the desired flow of interaction. There is still a very large (albeit smaller) maintenance burden. And the applications that use this approach (such as QTP) can cost in the tens of thousands of dollars. Another reason to do functional testing manually.

Conclusion

Functional testing is important to validating requirements. It is an important element of assuring a level of software quality. And it is still expensive with the best of today’s proven solutions. Even with the high cost, it is much cheaper than the risk of delivering a solution with poor quality. Plan on having functional testing as a component of any process to achieve software product success.

- – -

Check out the index of the Foundation series posts which will be updated whenever new posts are added.

Continuous Integration

Continuous Integration is the software development and quality process where all team members merge their code and verify it frequently – at least daily. This verification project includes both an automated build process and automated testing. The main benefits of continuous integration come from risk-reduction and cost-reduction.

Integration has to happen. Making it continuous reduces its cost. There are also efficiencies for developers who can write better code faster when they are writing it in the context of the latest (most up-to-date) version of the code.

Risk is reduced in two ways. First, continuous integration causes fewer bugs to be created, thereby reducing risk. Second, when bugs are created, they are identified at the earliest possible moment (same day as their creation). This maximizes the time available to resolve them. No surprises at the end of the release cycle.

Merging Code

When a single software developer is writing code, she writes her code, saving frequently, and archiving it. But she is the only person working on the code. Other than the developer, no one cares how often she checks it in, as long as she can deliver the software on the release date.

When multiple developers work together, they depend upon each other. On any given day, different developers are writing different pieces of software – usually objects with today’s languages. These objects talk to each other, depend upon each other, or at least co-exist with each other in the completed software. This stuff is all “under the hood” for users, but imperative to understand when trying to manage a development process or team.

Why Merging Matters

After the developers go off into their offices and create their objects independently, some or all of the team members have to stop what they are doing, and integrate all of those objects back into a common code-base (aka ‘the software). When a developer fixes a bug in one or more objects, those fixes need to be incorporated back into the common code-base. With multiple developers, there are multiple elements that all have to be rolled back together again into the code. Developers often refer to this as merging or promoting into the trunk, or tip, or main branch.

Each change has a set of predicted effects on the rest of the software. These changes can be tested by the developer before integrating her code into the trunk. Each change also has a set of unpredicted effects on the rest of the software. And combinations of changes can ‘create’ effects that did not occur with either change individually. ‘Unpredicted effects’ is fancy-talk for ‘bugs’. The more changes we integrate into the trunk at a time, the more bugs we create. And this is an accelerating effect – the complexity of integration increases faster than the number of changes being integrated.

Increased Complexity Drives Higher Costs

As we increase the frequency of integration, we decrease the quantity of changes per integration. This decreases both the cost per integration and the total cost of integration. It is much cheaper to integrate five changes 100 times than to integrate 100 changes five times.

Each object and each connection in these diagrams represent a potential source of error. With 4 objects, we have 10 sources. With 7 objects, we have 30 sources of error. This represents an accelerating increase in the cost of integration.

To minimize the costs, we need to minimize the number of objects being integrated. We do that by minimizing the time between integrations.

Overhead

The main resistance to frequent merging is the cost of testing. Testing involves building the merged code, running the tests, and evaluating the results. When the building and testing (the integrating) are automated, the cost of evaluating test results can be minimized. When test-evaluation is also automated, we can bypass test-result evaluation except when the system notifies us that something is broken.

Continuous integration is only feasible when the overhead of integrating (merging and verifying) is trivialized through automation.

Conclusion

Agile processes depend upon continuous integration, but any software development process is improved with continuous integration. This is one of the enablers of iterative development processes. It reduces the cost of quality (or allows us to achieve higher quality levels at the same cost). It also makes development more enjoyable because developers spend less time on fixing bugs and more time implementing solutions.

- – -

Check out the index of the Foundation series posts for other introductory articles.

PERT = Program Evaluation Review Technique

PERT is a technique for providing definitive estimates of how long it will take to complete tasks. We often estimate, or scope, the amount of time it will take us to complete a task or tasks. PERT allows us to provide not only an estimate, but a measure of how good the estimate is. Good estimates are a critical element in any software planning strategy. In this post, we will present an introduction to using PERT, explain how it works and how to interpret PERT estimates.

Multiple estimates

Imagine we’ve been asked to predict how long it will take us to go to the store and get some vanilla ice cream. “It depends,” we think. If the traffic is light, and there is a short line at the checkout counter, 15 minutes. If traffic is average and we have to wait behind several people at the checkout line, 30 minutes. If there’s an accident on the way and traffic is backed up, it could take an hour.

We may not realize it, but we just created a PERT estimate.

Definition of PERT

A PERT estimate uses three estimates for any given task, to provide both an expected duration for the task and an understanding of how much we might be off in our estimate. To create a PERT estimate, we first create three seperate estimates of the time it will take to complete the task.

1. Optimistic, or best-case scenario. 15 minutes to get ice cream, if everything goes right.
2. Likely scenario. 30 minutes is how long we think it will probably take to get the ice cream.
3. Pessimistic, or worst-case scenario. 60 minutes if everything bad that could reasonably happen happens.

Next, we combine these estimates to create a single number that best predicts how long it will take. We create a weighted-average of the three values, but we count the “likely” estimate as being four-times more likely than either the optimistic or pessimistic estimate. This represents the mean estimate.
mean = (optimistic + (4 * likely) + pessimistic) / 6

The PERT mean estimate for our ice cream delivery is (15 + (4 * 30) + 60) / 6 = 32.5 minutes. We often see PERT estimates documented as all three values, 15/30/60.

Basic PERT interpretation

A PERT estimate includes an approximation of standard deviation (stdev) based on the optimistic and pessimistic values. Stdev provides us with insight into the shape of the probability curve represented by the estimates.

stdev = (pessimistic – optimistic) / 6

The PERT stdev for our ice cream delivery is (60 – 15) / 6 = 7.5 minutes.

The PERT estimate is a representation of a beta distribution. The math is pretty complex, and applying it to a single estimate can give us a false sense of precision. Remember that the numbers we put together in the first place are our best guesses, so doing more precise math on rough estimates doesn’t give us a more precise PERT estimate, just more math.

The beta distribution is very similar to a normal distribution (the familiar bell curve), when it is balanced. This similarity is used to simplify the application of PERT. The standard simplification in using PERT is to treat estimates as if they represented the following probabilities:

• Optimistic: 5% of the time we will complete our task in less than the optimistic time estimate.
• Likely: 50% of the time we will complete our task in less time than the likely time estimate.
• Pessimistic: 95% of the time we will complete our task in less than the pessimistic time estimate.

More advanced PERT analyses

What happens when we want to combine several PERT estimates for our project? Imagine we had to make two trips to buy ice cream, and we want an estimate of how much total time it would take.

Here’s the wrong solution. Add the two PERT estimates. 15/30/60 + 15/30/60 = 30/60/120 with a mean of 65 minutes and a stdev of 15 minutes. This would say that there is a 90% chance that we will spend between 30 and 120 minutes getting ice cream.

Again, without much math, there’s an intuitive reason why this is wrong. We are not taking into account the likelihood that our two trips will take different amounts of time. If we hit bad traffic in one trip, that doesn’t mean we are likely to hit it on the second trip as well. If there is 1 chance in 20 that we hit out pessimistic estimate, then there is only 1 chance in 400 that it would happen twice. So really, there is a 99.5% chance that we would have between 30 and 120 minutes of total travel. We can narrow down this estimate a lot.

To combine the two estimates, we combine the underlying probability distributions that they represent. With the PERT approximations, we would do this as follows:

1. The combined mean is the sum of the two old means. 32.5 + 32.5 = 65 minutes.
2. The combined stdev is the square root of the sum of the squares of the two old stdevs. SQRT(7.5^2 + 7.5^2) = 10.6.
3. The combined optimistic estimate is the mean minus ( 3 * stdev). Optimistic = 65 – (3 * 10.6) = 33 minutes.
4. The combined pessimistic estimate is the mean plus (3 * stdev). Pessimistic = 65 + (3 * 10.6) = 97 minutes.
5. Our combined PERT estimate is 33/65/97 with a mean of 65 minutes and a stdev of 10.6 minutes.

This combined window is more narrow than the “just double everything” approach, and reflects that the law of averages will eventually narrow down the interval for our estimate – because some tasks take longer than estimated, and some take less time. It all averages out in the end, as long as the estimates are good to begin with.

Beyond PERT

• One question that always comes up when planning is how you created your estimates.
• Managing releases with timeboxes.
• Updating an existing release schedule with requirements changes.

Summary

We have enough information to know how to create a PERT estimate for a single task. We also know how to combine those PERT estimates to provide a multi-task estimate. There is a lot more we could cover, such as showing the impact of this statistical technique on critical path analysis, or incorporating an approximation of the correlation of estimates (if this task takes longer than estimated, then that task will probably take longer than we estimated). That’s too much detail for this introductory article – just know that it’s out there. What we cover here represents what the vast majority of project managers understand about PERT (and maybe a bit more).

- – -

Check out the index of the Foundation Series posts which will be updated whenever new posts are added.

Feature driven development (FDD) is one of several agile methodologies for developing software iteratively. Iterative development is the opposite of waterfall development.

In a nutshell

FDD is a process that begins with high level planning to define the scope of the project, which then moves into incremental delivery. Each increment of delivery involves a design phase and an implementation phase. The scope of each increment is a single feature. Extreme programming (XP) is a much better known agile methodology. XP is often described as an emergent design process, in that no one knows what the finished product is going to be until the product is finished. FDD, by comparison, defines the overall scope of the project at the beginning, but does not define the details.

Explained by analogy

Consider the writing of a mystery novel as an analogy.

• Waterfall process. First, our author determines the major characters, the mystery, a detailed plot outline, and outlines for all of the subplots. Then she sketches out all of the minor characters. Finally she writes the novel, following her outline along the way. Immediately after typing ‘the end’ she sends the novel off to the editor. The editor replies with major change suggestions – it seems that the topic might have been in vogue two years ago, but it doesn’t sell very well today. And half the chapters are low-value rambling that are sure to lose the readers. Our author starts over.
• Extreme programming: XP. Our author decides to write a mystery novel, puts a blank sheet of paper in the typewriter and starts typing. As she writes, she realizes that she’d much rather write a romance, so she edits the chapters she’s finished and keeps moving forward. She sends early drafts to her editor of every chapter as she finishes it. The editor suggests that she change the setting from the Alps to the Amazon, and she edits again. After finishing the book, she has a three-book historical fiction series set in the rain forest.
• Feature driven development: FDD. Our author creates an outline for the story, gives names to the major characters and prepares to write chapter one. As she starts each chapter, she writes some details of the subplot, makes some notes about how the characters should develop, and begins writing. She sends her outline to the editor, as well as drafts of each chapter as she completes them. She splits her time between incorporating feedback on previous chapters and outlining/writing the current chapter. At the end of the book, she has a mystery with the same major characters that she expected – but they didn’t develop into exactly the people she expected, and she never would have predicted the sub-plots that created themselves as she wrote.

When we look at these approaches, we see that FDD tries to combine the best part of a waterfall process (good planning) with the best part of XP (continuous improvement through iteration).

More detail

There are five phases in an FDD process. The first three phases are planning phases and the last two phases are iterative phases (they are repeated for each iteration).

Planning phases:

1. Develop an overall model. This is a representation of how the solution will work and what it will do. It is the high-level framework describing the big picture of how everything works together.
2. Build feature list. This is the list of features needed to implement the high level view from phase 1.
3. Plan. Create a rough plan of the entire project. Some proponents also talk about creating detailed plans per feature (as each feature is addressed).

Iterative phases (one feature per iteration)

1. Design the feature. What Alan Cooper would call program design.
2. Implement the feature. Writing code, testing, documentation.

Conclusion

There is little or no discussion about requirements in FDD. Starting with an overall model is great from a developer’s perspective. The challenge is in determining what to place in the model – what requirements are important to the users? How will users interact with the system? Good answers to these questions can make or break an overall model – and a faulty model will yield low-value software.

This approach to agile development can be very effective when augmented with the right requirements management process.

Learning more about FDD

• Brad Appleton has a good post providing several links to other sources and explaining how FDD sits between waterfall and xp processes.
• Jeff De Luca wrote the precursor to FDD (or the original version, as he explains)
• Peter Coad, author of FDD has a great 23 slide pdf presentation providing an overview of FDD.
• www.featuredrivendevelopment.com.

- – -

Check out the index of the Foundation Series posts which will be updated whenever new posts are added.

Very large and complex systems can be very difficult and expensive to test. We inherit legacy systems with multiple man-years of development effort already in place.  These systems are in the field and of unknown quality. With these systems, there are frequently huge gaps in the requirements documentation. Pairwise testing provides a way to test these large, existing systems. And on many projects, we’re called in because there is a quality problem.

We are faced with the challenge of quickly improving, or at least quickly demonstrating momentum and improvement in the quality of this existing software. We may not have the time to go re-gather the requirements, document them, and validate them through testing before our sponsor pulls the plug (or gets fired). We’re therefore faced with the need to approach the problem with blackbox (or black box) testing techniques.

For a complex system, the amount of testing required can be overwhelming. Imaging a product with 20 controls in the user interface, each of which has 5 possible values. We would have to test 5^20 different combinations (95,367,431,640,625) to cover every possible set of user inputs.

The power of pairwise

With pairwise programming, we can achieve on the order of 90% coverage of our code in this example with 54 tests! The exact amount of coverage will vary from application to application, but analysis consistently puts the value in the neighborhood of 90%. The following are some results from pairwise.org.

We measured the coverage of combinatorial design test sets for 10 Unix commands: basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc. […] The pairwise tests gave over 90 percent block coverage.

Our initial trial of this was on a subset Nortel’s internal e-mail system where we able cover 97% of branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.

[…] a set of 29 pair-wise AETG tests gave 90% block coverage for the UNIX sort command. We also compared pair-wise testing with random input testing and found that pair-wise testing gave better coverage.

Got our attention!

How does pairwise testing work?

Pairwise testing builds upon an understanding of the way bugs manifest in software. Usually, a bug is caused not by a single variable causing a bug, but by the unique combination of two variables causing a bug. For example, imagine a control that calculates and displays shipping charges in an eCommerce website. The website also calculates taxes for shipped products (when there is a store in the same state as the recipient, sales taxes are charged, otherwise, they are not). Both controls were implemented and tested and work great. However, when shipping to a customer in a state that charges taxes, the shipping calculation is incorrect. It is the interplay of the two variables that causes the bug to manifest.

If we test every unique combination of every pair of variables in the application, we will uncover all of these bugs. Studies have shown that the overwhelming majority of bugs are caused by the interplay of two variables. We can increase the number of combinations to look at every three, four, or more variables as well – this is called N-wise testing. Pairwise testing is N-wise testing where N=2.

How do we determine the set of tests to run?

There are several commercial and free software packages that will calculate the required pairwise test suite for a given set of variables, and some that will calculate N-wise tests as well. Our favorite is a public domain (free) software package called jenny, written by Bob Jenkins. jenny will calculate N-wise test suites, and its default mode is to calculate pairwise tests. jenny is a command line tool, written in C, and is very easy to use. To calculate the pairwise tests for our example (20 controls, each with 5 possible inputs), we simply type the following:

jenny 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 > output.txt

And jenny generates results that look like the following:

1a 2d 3c 4d 5c 6b 7c 8c 9a 10c 11b 12e 13b 14d 15a 16c 17a 18d 19a 20e
1b 2e 3a 4a 5d 6c 7b 8e 9d 10a 11e 12d 13c 14c 15c 16e 17c 18a 19d 20d
1c 2b 3e 4b 5e 6a 7a 8d 9e 10d 11d 12a 13e 14e 15b 16b 17e 18e 19b 20c
1d 2a 3d 4c 5a 6d 7d 8b 9b 10e 11c 12b 13d 14b 15d 16d 17d 18b 19e 20a
1e 2c 3b 4e 5b 6e 7e 8a 9c 10b 11a 12c 13a 14a 15e 16a 17b 18c 19c 20b
1a 2a 3c 4e 5e 6a 7b 8c 9d 10b 11b 12b 13e 14a 15d 16d 17c 18c 19b 20d [...]

Where the numbers represent each of the 20 controls, and the letters represent each of the five possible selections.

What’s the catch?

There are two obvious catches. First, when you use a tool like jenny, we must run all of the tests that it identifies, we can’t pick and choose. Second, pairwise testing doesn’t find everything. What if our example bug before about taxes and shipping only manifested when the user is a first time customer? Pairwise testing would not catch it. We would need to use N-wise testing with N >= 3. Our experience has been that N=3 is effective for almost all bugs.

There is also a sneaky catch – test generators like jenny assume that the order of variables is irrelevant. Sometimes we are testing dynamic user interfaces, where the order of value selection in controls is relevant. There is a solution to this, and we will update this post with a link to that solution when it is available.

- – -

Check out the index of software testing series posts for more testing articles.

CMMI is the initialism for Capability Maturity Model Integration.

CMMI is a numeric scale used to “rate” the maturity of a software development process or team. Maturity can be thought of like enlightenment. An immature process is not much different from the old “infinite monkeys” yarn – maybe we get it right, but probably not. A fully matured or enlightened process not only does it right, but improves itself over time.

The Software Engineering Institute (SEI) at Carnegie Mellon (Go Tartans! BSME90) created the CMM model for software engineering in the late 80’s and early 90’s. In an effort to consolidate multiple CMM models for different process areas, the SEI team created the CMMI in 2002. In this post, we will understand what each level represents.

Technically, the name of the model is the Capability Maturity Model Integration for Software Engineering, or SW-CMM, but in practice people just use CMM. The 645 page document can be found on the CMU SEI site.

Five levels of maturity in CMMI

The folks at the SEI created five classifications or levels of process maturity. Every software project can be classified into one of the levels. The levels are progressively harder to achieve, and each builds on the previous level. To be classified in a particular level, a process must meet all of the criteria of that level. If a process meets all the criteria of level 2, and some of the criteria of level 3, then that process is considered to be a level 2 process.

Note that there is a level 0, or incomplete. This level represents the absence or incompleteness of activity. If we are developing software, we are at least at level 1.

1. Performed.
2. Managed.
3. Defined.
4. Quantitatively Managed.
5. Optimizing.

CMMI Level 1: Performed

The lowest possible CMMI level (there is no level 0) represents software development essentially in the absence of a process. Specifically, the SEI defines level 1 as being achieved if the goals of the process are met. If we are trying to write software, and we write software, then we are at level 1. In practice, level 1 processes are the ones described as chaotic and unordered. A team with a level 1 process can easily be dependent upon individual effort for success.

Most small software companies, independent developers, as well as many corporate IT departments operate at level 1. If we can point to projects that were “saved” by a superstar on the team, we are probably operating at CMMI level 1. We love having superstars on our teams – we hate being dependent upon them for survival.

CMMI Level 2: Managed

SEI defines CMMI level two as follows:

A managed process is a performed (capability level 1) process that is also planned and executed in accordance with policy, employs skilled people having adequate resources to produce controlled outputs, involves relevant stakeholders; is monitored, controlled, and reviewed; and is evaluated for adherence to its process description.

Our interpretation: take a CMMI level one process, and add a project manager to coordinate the chaos. The distinction is that at this level, the activities are planned, and then managed according to the plan. No additional insight is applied, just orchestration.

CMMI Level 3: Defined

To achieve CMMI level three, we have to take a process that qualifies for CMMI level two, and institute it as a corporate standard. Then we tailor and apply that standard process to individual projects.

CMMI Level 4: Quantitatively Managed

When we incorporate quantitative measurement or statistical analysis tools as part of our process management process, we are operating at CMMI level four. This level represents “hard” introspection that utilizes quantified data to improve the management of the current project within our defined project.

CMMI Level 5: Optimizing

When we analyze the quantitative data from our projects and apply it to refining our processes for the future, we are operating CMMI level five. This level represents an introspective approach to quantitative project management, combined with a continuous improvement objective.

Articles In This Series

• Foundation Series: CMMI Levels Explained
• What CMMI Level Should We Use?
• CMMI Levels and RMM Introduction
• CMMI Levels and RMM Level 1 – Written Requirements
• CMMI Levels and RMM Level 2 – Organized Requirements
• CMMI Levels and RMM Level 3 – Structured Requirements
• CMMI Levels and RMM Level 4 – Traced Requirements
• CMMI Levels and RMM Level 5 – Integrated Requirements
• Don’t forget to take our One Minute Survey on CMMI and RMM.

- – -

Also check out the index of the Foundation Series posts for other introductory articles.

What the heck is UX?

UX, pronounced you-ex, is the shorthand for user-experience. It represents the science and art of tailoring the experience that users have with a product – in our case, software. UX is a relatively new term, rapidly overtaking HCI (human-computer interface) and CHI (computer-human interface) as the acronym du jour. In some circles it is known as human-factors engineering, applied to software design. There are several disciplines within this field, we’ll introduce each of them.

We talk about the different roles within this field in several posts throughout Tyner Blain. The following are introductory explanations for these roles.

Information Architecture (IA)

The study of information and it’s presentation to people. Also the study of how people interact with information. Many software packages allow users to manage complex information. Information can be presented in ways that make it easier for people to absorb and understand.

As a very simple example, imagine a website that allows you to research the cost of living in different cities in the USA. There are thousands of cities in the country. IA helps with designing a user interface that allows users to get information for a specific city. An IA specialist would recognize that cities can be organized by state. In fact, cities in different states can have the same name, like Springfield, Missouri and Springfield, Illinois. But two cities within the same state won’t have the same name. This insight can be applied to present a design where the user selects a state first, which then filters a list of the cities within that state.

A corporate internet may really be the combination of several different standalone websites – a company news bulletin or blog, an interface to the HR system, a download center for installing corporate-approved software, an email directory for the company, etc. IA specialists will determine how to organize all of these functions so that employees can intuitively find what they need and get as much benefit out of the site as possible.

Usability

The study of what makes software easy to use or hard to use. A usability specialist will look at the tasks that a user needs to perform, and analyze the most intuitive or efficient ways to perform them. Think of the sequence of steps that you take when adding a graph of data in Microsoft excel. There is a wizard that walks you through a series of questions in order to create the graph for you. A usability specialist determined the best sequence in which to ask and answer those questions.
Usability specialists will also make holistic assessments of how an application or suite of applications behave. This helps users gain competence or mastery of software more quickly. All of Microsoft’s applications use the same approach for opening and saving files (same menus, same shortcut keys, same dialogs, etc). This is the result of usability analysis.

A usability specialist will also be the person who determines how to make software great for novice and experts alike. This is critical to having successful software – the experts are the people who will promote your software for you, but they won’t become experts unless they survive the novice-user break-in period.

Graphic (or Visual) Design

Some people erroneously think of visual designers as the people who make software sexy. The can certainly do that, but graphic design is as much about creating emotions for the users, consistency of presentation, and establishing elements of brand as it is about sexy. This is what makes a Macintosh look like a Macintosh (while usability specialists make it great to use).

A graphic designer can create a set of consistent icons that make an application feel professional, and make the user feel whatever the designer wants. Graphic designers can make the user interface feel different enough to create a notion of uniqueness and branding (association of the images with the product or company), while also keeping them consistent enough with “everybody else” that users know what to do. Another technique is to create an affordance visually. An affordance is an image or element that suggests an action. A dial says “turn me” while a slider says “slide me.”

This can be very subtle and very powerful.

Think about scrollbars for a second. Most scrollbars have a pretty boring look. There are tiny up and down arrows at the top and bottom – which create an affordance that says “click on me and the window will move up (or down). That’s good design. There’s also a grey bar in the middle. In some user interfaces, the size of that bar is proportional to the amount of the content that is currently visible. This gives the user some insight into how much content is hidden – another good visual design. A user can also click and drag the grey bar up and down to move the contents of the window. There are no visible cues that this would work, a user would have to be shown that this works. Another example of “hidden” functionality is the ability to click in the light grey “background” of a scrollbar – it causes the contents of the window to page up or page down. Again, without training or an errant click, people would not know this.

If we make a tiny change to that scrollbar by adding a few lines in the center, we create a tactile effect – implying that the user can “grab” it with the mouse. This scrollbar screams “grab me”. Subtle, but powerful.

Interaction Design

Interaction Designers are a different breed.  They focus on the software at a higher level, using a goal-driven process to focus on the intent and objectives of the users.

- – -

Check out the index of the Foundation Series posts which will be updated whenever new posts are added.

What are unit tests?

Testing software is more than just manually banging around (also called monkey testing) and trying to break different parts of the software application. Unit testing is testing a subset of the functionality of a piece of software. A unit test is different from a system test in that it provides information only about a particular subset of the software. In our previous Foundation series post on black box and white box testing, we used the inspections that come bundled with an oil change as examples of unit tests.

Unit tests don’t show us the whole picture.

A unit test only tells us about a specific piece of information. When working with a client who’s company makes telephone switches, who’s internal software development team did not use unit tests we discussed the following analogy:
Unit tests let us see very specific information, but not all of the information. Unit tests might show us the following:

A bell that makes a nice sound when ringing.

A dial that lets us enter numbers.

A horn that lets us listen to information.

We learn a lot about the system from these “pictures” that the unit tests give us, but we don’t learn everything about the system.

We knew (ahead of time) that we were inspecting a phone, and with our “unit tests” we now know that we can dial a phone number, listen to the person on the other end of the line, and hear when the phone is ringing. Since we know about phones, we realize that we aren’t “testing” everything. We don’t know if the phone can process sounds originating at our end. We don’t know if the phone will transmit signals back and forth to other phones. We don’t know if it is attached to the wall in a sturdy fashion.

Unit testing doesn’t seem like such a good idea – there’s so much we need to know that these unit tests don’t tell us. There are two approaches we can take. The first is to combine our unit tests with system tests which inspect the entire system – also called end to end tests. The second is to create enough unit tests to inspect all of the important aspects. With enough unit tests, we can characterize the system (and know that it is a working phone that meets all of our requirements).

Software developers can identify which parts of their software need to be tested. In fact, this is a key principal of testing-driven development (TDD) – identify the tests, then write the code. When the tests pass, the code is done.

Why not use system tests?

The system test inspects (or at least exercises) everything in the software. It gives us a big picture view. Ultimately, our stakeholders care about one thing – does the software work? And for them, that means everything has to work. The intuitive way to test, then, is to have tests that test everything. System testing is also known as functional testing.

These comprehensive tests tell us everything we want to know. Why don’t we use them?

There is a downside to system testing. In the long run, it’s more expensive than unit testing. But the right way to approach continuous integration is to do both kinds of testing.

In our Software testing series post on blackbox and whitebox testing we discuss several tradeoffs associated with the different types of testing. For most organizations, the best answer is to do both kinds of testing – do some of each. This is known as greybox testing, or grey box testing.

System tests are more expensive, because they are more brittle and require more maintenance effort to keep the tests running. The more your software changes, the faster these costs add up. Furthermore, with Agile practices, where portions of the system are built and tested incrementally, with changes along the way, system tests can be debilitatingly expensive to maintain.

Because unit tests only inspect a subset of the software, they only incur maintenance costs when that subset is modified. Unit testing is done by the developers, who write tests to assure that sections of the software behave as designed. This is different from functional testing, that assures that the overall software meets the requirements.
There are more articles on software testing in our software testing series.
- – -

Check out the index of the Foundation series posts for other introductory articles.

Blackbox tests and whitebox tests.

These terms get thrown about quite a bit. In a previous post, we referenced Marc Clifton’s advanced unit testing series. If you were already familiar with the domain, his article could immediately build on that background knowledge and extend it.Software testing can be most simply described as “for a given set of inputs into a software application, evaluate a set of outputs.”
Software testing is a cause-and-effect analysis.

The inputs can be user actions, data from external systems, or any combination of the two.

The outputs can be outputs on the computer screen, data output to a file (like a log or report), a communication with other computers (like web services responses), or any other change in state of the system.

As an example of simple inputs and outputs, a calculator program has the requirement: “Allow user to calculate the result of dividing one number by another, non-zero number.” If a/b=c, the inputs would be the dividend (a) and the divisor (b). The outputs would be the quotient (c).

As an example of change in state as an output, consider a requirement: “After the user logs in, the system will make future decisions based upon that user’s predefined preferences.” For that requirement, an input would be “logged-in user id” and an output (or change in state) would be “system loads user’s preferences.”

Black box testing

Black box testing is a stimulus-response analysis of behavior. To run (or define) a black box test, we don’t need to know anything about how the software works. We provide it with a stimulus (user selects “advanced search” button) and inspect for a response (advanced search page input form is presented to the user).
In biology class, we performed black box tests on dead frogs. We hooked up electrodes to their legs and applied current. The legs kicked. We didn’t know anything about how the frogs worked, or we were told that the muscles would respond to small electrical currents by contracting. And we didn’t need to know. We just tested the frogs.

With a calculator program, we don’t need to know the hoops that the cpu jumps through to calculate the quotient. We don’t need to wonder if the program is written in php or python or c++. All we need to do is inspect the output for a given set of inputs.

This highlights the primary benefit of black box testing: a system can be tested by someone with no knowledge of how it works.

This allows us to more easily find people capable of testing our software – the pool of available people with the skills to keep track of what they input and what they output is much larger than that of people who understand the stuff “under the hood”. It also saves our testers from having to learn how it works – they can start testing immediately.

When a team is organized with a dedicated testing (only) staff, the tests they create are typically black box tests – because the team can be staffed more cost effectively.

Blackbox tests are sometimes referred to as opaque tests or closed-box tests. They are sometimes also referred to as behavioral tests – in that they only test the behavior of the system, not how (or how well) it is constructed.

White box testing

A white box test is one that requires insight into how the code is implemented. The test takes advantage of understanding the data structures or flows to provide specific information about how the code is working as an output of the test.

White box tests are also sometimes called clear-box tests or structural tests because they can provide insight into how the code is performing.

When you take your car for a state inspection test, it is a black-box test. The overall performance (breaking distance, emissions levels, no sharp edges, etc) is inspected. When you take your car for an oil change, you usually get a set of white-box test results (air-filter cleanliness check, coolant mixture analysis). These pieces of detailed information (coolant at 60% mix) don’t tell you if the car is “good” – but they give you insight into how it’s running.

Unit testing, or testing a subset of the functionality of a piece of software can use black box or white box testing, but is most commonly done using white box tests. A unit test is a test that provides a piece of specific information (like coolant mix, or testing a connection to a database, or the speed of a SQL query), without neccessarily making a statement about the overall quality of the software or system.

The primary benefit of white box testing is that you can use insight of how the software is constructed to efficiently test it. This testing efficiency comes from having the ability to target specific areas of the code for testing, and also allows more efficient selections of tests to run. The weakness of white box testing is that it requires knowledge of how the software is written in order to design the appropriate tests. Another weakness is that misinterpretation of the requirements can result in whitebox tests of the wrong functionality.

When a team is organized so that developers are responsible for testing their own code, they are more likely to incorporate white box tests, and more specifically unit tests. There are efficiencies to using these types of tests, making the development process easier.

Grey box testing

When we combine black box and white box tests in the same test suite, we get what is called grey box testing, or greybox testing. This systematic approach to testing allows us to combine the benefits of both blackbox testing and whitebox testing in the same test suite.

Test automation

Both black box and white box tests can be automated. Different tools and techniques are applied for the different types of tests, but both are feasible and common. Continuous integration is a development process that leverages test automation to reduce costs and improve quality.

There are more articles on software testing in our Software Testing Series.

What book should I read to learn more?

Software Testing, by Ron Patton.

Here’s a review from Randy Rice “Software Testing Consultant & Trainer” (Oklahoma City, OK)

Software Testing is a book oriented toward people just entering or considering the testing field, although there are nuggets of information that even seasoned professionals will find helpful. Perhaps the greatest value of this book would be a resource for test team leaders to give to their new testers or test interns. To date, I haven?t seen a book that gives a better introduction to software testing with this amount of coverage. Ron Patton has written this book at a very understandable level and gives practical examples of every test type he discusses in the book. Plus, Patton uses examples that are accessible to most people, such as basic Windows utilities.

I like the simplicity and practicality of this book. There are no complex formulas or processes to confuse the reader that may be getting into testing for the first time. However, the important of process is discussed. I also have to say a big THANK YOU to Ron Patton for drawing the distinction between QA and testing! Finally, the breadth of coverage in Software Testing is super. Patton covers not only the most important topics, such as basic functional testing, but also attribute testing, such as usability and compatibility. He also covers web-based testing and test automation ? and as in all topics covered in the book, Patton knew when to stop. If you want to drill deeper on any of the topics in this book, there are other fine books that can take you there!

I love this book because it is practical, gives a good introduction to software testing, and has some things that even experienced testers will find of interest. This book is also a tool to communicate what testing and QA are all about. This is something that test organizations need as they make the message to management, developers and users. No test library should be without a copy of Software Testing by Ron Patton!

- – -

Check out the index of the Foundation Series posts for other introductory articles.

Karl Wiegers wrote the book on structured requirements – Software Requirements, 2nd Edition, Karl E. Wiegers.

If you are involved in managing requirements, you should own this book. Even if you don’t follow his approach to managing requirements, or don’t like how he deals with use cases, you should still read this book – at a minimum, you’ll know more about it than your pointy-haired boss who reads this blog, sees this post, and tells you that you must follow the Wiegers way.

He details his framework, tells you how to use it, and how to manage requirements in it. Karl also has a website, processimpact.com, chock full of resources.

Karl proposes that there are three distinct levels of requirements

1. Business requirements - Goals of the business like “increase profits”, “improve branding”, “become dominant in a market”
2. User requirements – Goals or tasks of the users of software like “create purchase order”, “find a book my wife would like”
3. Functional requirements – Functionality that the software must include like “calculate profit-maximizing price” or “generate Sarbanes-Oxley compliance report”

He also classifies these requirements as either being functional or non-functional.

Functional requirements describe what the system must do.

• Provide a history of transactions for auditing purposes
• Enable users to listen to samples of the music on the CD

Non-functional requirements constrain how the system must do it.

• Most relevant search results will be returned in under 5 seconds
• System will be available 99% of the time between 9 AM and 7 PM Eastern time

Karl then presents these types of requirements with a structured classification. His structure shows different types of requirements driving other types of requirements. In the picture below, we would see that a business requirement (increase profits) drives a user requirement (define product prices) which drives a functional requirement (calculate profit-maximizing price).

I believe a simplified version of this diagram (which is a simplified version of a diagram from page 9 of his book) makes it easier to introduce the concepts.

In a presentation to a class at St. Edwards University last fall, I presented the following single slide.

Summing it all up

Goals are achieved through use cases.

Use cases are enabled by functional requirements.

Functional requirements lead to design and implementation.

Non-functional requirements characterize how functional requirements must work.

Constraints restrict how functional requirements may be implemented.

[Update 2007/02/26]

I’ve refined my thinking about how structured requirements should be represented. In short, I feel that non-functional requirements are under-emphasized in the real world.  I proposed a modified view of structured requirements, designed to increase the level of attention given to non-functional requirements. I go into more detail in the article, Non-Functional Requirements Equal Rights Amendment. Here’s the diagram of the structure that I proposed in that article:

- – -

Check out the index of the Foundation Series posts for other introductory articles.

A common framework is needed to compare processes and their merits. All software development includes the following three steps in some combination:

1. Decide what to do.
2. Do it.
3. Deliver it.

It sounds like an overly simplified list; Decide, Develop, Deliver. We’ll see that it isn’t.

Different process styles are like different dance styles.

Different processes will put different levels of emphasis and effort into each step. Each processes may repeat steps, or take them in a different order.

We can think of each different process as a different style of dance. They are alike and unalike. There are people who prefer one style to another, and there are specialists in one style and generalists across styles.

For any particular song, some styles of dance are intuitive and comfortable while others are awkward. The same is true of applying a software process to a given development scenario.

What works for a global team on a multi-year enterprise software project may not work well for a start-up creating a consumer website. The process that works best for an engineering firm developing embedded software for a pace maker may be a nightmare to apply to an open-source project for creating an RSS reader.

Software processes are usually categorized as waterfall or iterative – although I first heard these categories from proponents of iterative processes (the “new” way) when they were getting a lot of press in the late 90’s. Bias aside, the analogy works.

Waterfall process
A waterfall process is one where all of the work is done by the team, and when it is complete, it is delivered to their customer(s). With a waterfall, the water doesn’t do anything useful until after it falls off the end of the cliff – and by then it’s too late to climb out of the barrel if you change your mind. You don’t know if you’ll hit a rock until you reach the bottom of the waterfall. The larger the waterfall, the larger the risk, and the bigger the mess if you do hit a rock.

The same is true for waterfall software processes – we decide what to do, we develop the solution, and then we deliver it. We don’t revisit our decisions. Once the decisions are made, we’re in the barrel and falling, and we won’t know if our software will succeed until after we deliver it. The risk of failure is proportional to the amount of money and time spent before delivering the software to our customer.

Incremental process
An incremental process is one where we will deliver portions of the software in the smallest reasonable subsets of the ultimate functionality. We will have multiple deliveries to our customer. We will revisit the assumptions and prioritizations for each “next” delivery when we complete each “previous” delivery.

A sailing analogy works well for this approach. To sail in a straight line from start to finish, we set a heading, H1, and begin sailing. After a period of time, we check to see where we are (invariably, we’ve drifted off course a little), correct our course and set a new heading, H2, and continue sailing. We keep doing this (checking position, adjusting heading, sailing some more) incrementally until we reach our destination.

With incremental software development, we do the same thing as when sailing. First we decide on what to do and what to do first. Then we develop and deliver the first set of functionality. We then revisit our decisions (including getting feedback on what we’ve just delivered) and decide what to do next. We then develop and deliver the next set of functionality. We repeat until all of the desired functionality has been delivered.

Which process is best?
Picking the best process, like picking the best dance depends on circumstances. Let’s look at a few “real world” factors that can drive our decisions.

Budgeting and planning (winner: waterfall process)
With a waterfall process, we decide at the start exactly what we intend to accomplish. We can therefore scope and schedule the project. We can also determine staffing and costs. Budget decisions are easy – IRR and ROI can be calculated – we can calculate expected values for both costs and (forecasted) benefits. With an iterative process, we’re saying “I reserve the right to change my mind later.” We fully expect to change the scope of delivery mid-project. We know we will learn more about how to forecast the benefits after each incremental release. Because we know about, and even desire future changes, we can’t reasonably estimate ROI. We can “fix” the budget (aka timebox the project), but we can’t predict the value we will achive within the time allotted.

Return on investment (ROI) (winner: incremental process)
Incremental processes get the nod for return on investment because they start creating value earlier. Imagine software that can be delivered in two releases, where each release provides features that generate $10,000 per month in value. Each release takes 6 months. With waterfall delivery, there will be no value creation until after 12 months – and then it will generate $20,000 per month. With incremental delivery, the first release will generate $10,000 per month from month 6 to month 12, and then it will generate $20,000 per month for the next 12 months. The extra value from incremental delivery is the $60,000 earned from the first release while the second release is being developed.

Delivering a subset of functionality (winner: it depends)
Some projects can’t tolerate partial delivery. A pace maker that only regulates pulse, but can’t be recharged is not useful. However, an RSS reader without archiving can be released now with feed-reading, and updated in the next release to include archiving. Many times we run into users who say things like “Until it does everything, I won’t use it.” This is most common when replacing a legacy application. Users are accustomed to a particular workflow, and have expectations about what they should be able to do. Change management is the discipline that focuses specifically on helping users manage changes in the way they do their work, and the tools they use to do it. We can often help users make a transition, even from a legacy system, in incremental steps. When we can’t, a waterfall style delivery is usually better. The key thing to remember about incremental delivery is that the increments are as small as possible – which may mean “deliver it all in the first release.”

Are there other factors that we’ve overlooked? Add a comment and tell us about them.

Check out the index of the Foundation Series posts for other introductory articles.