We all know the story about the emperor’s new clothes. I’ve been thinking about prioritization and scheduling, and as far as I know, no one is promoting that we maximize value – they (and we) have been promoting that we do the most valuable stuff first. Doing the most valuable things first does not result in getting value the fastest. In this article, we show why not.

Genesis

About a month ago, I read an article by Kelly Waters on how to prioritize intuitively. He presents a magic square diagram, showing both the “how valuable is it” and the “how hard is it to do” axes. I’m oversimplifying, read his article for more details – he incorporates elements of risk, complexity, etc. I really liked that he was addressing the “missing element” of how much work is involved. However, his diagrammatic approach, while presenting this information very well, does not really yield insights into what to do first. Kelly and I had a great discussion over the next couple weeks, exploring the interplay of work and value in prioritization, trying to find a way to encourage value-maximizing decisions.

Prioritize By Value

We have talked about prioritizing the most valuable requirements first repeatedly. And in the last of those links, we accidentally hinted at, but didn’t grasp the real goal:

We will only consider those steps where the profitability of change exceeds our hurdle rate for investment.

We’ve also talked in the past about using use cases as the basis for scheduling, as each use case represents realizable value. For the rest of this article, we’ll talk in the context of scheduling use cases across releases.

Consider a very simple example – you have five use cases, with values of 10, 9, 9, 8, and 7 respectively (the units don’t matter). If you sort those use cases in order by value, from left to right, they would look like the following:

The size of each box shows the relative value of having the use case implemented.

Based on our previous guidance (and everyone else’s), you would implement them in order, from left to right. Makes sense. Do the most valuable thing first. Do the next most valuable thing next. Repeat until the value is not high enough to continue.

What About The Amount of Work?

OK, the amount of work required should play a role too. In our time-boxing article, we describe each release as having a given capacity, which you can visualize in terms of cost (resource) and time (duration of applying resources):

We fill up that capacity with use cases, based upon how much work is involved.

We can estimate the work involved with each use case by any of a number of methods – but the earliest estimates can be developed using use case points.

Consider the following “work” measurements, identified for each of our use cases from above:

We have the same sequence of use case implementation (based on value), but now we can visually see that there are different amounts of work associated with each use case. The area of each box represents the relative level of effort required to implement the use case.

Prioritize By Value Results

The best way to explain the flaw with the classical “prioritize by value” approach is to show what happens after the first release. Consider that you can accomplish 30 units of work in the first release.

We can schedule the first two use cases for this release. The size of the time-box above represents the amount of work that can be accomplished. With the first two use cases scheduled, the time-box will look like the following:

We have completely used up the available capacity of the team (Work = W = 30 = 10 + 20) by delivering the two most valuable use cases. We have delivered 19 units of value (Value = V = 10 + 9 = 19).

Value Maximization

When we consider both the value (V) and the cost (in terms of work, W) of each use case, we see that some use cases generate more value per unit of work than other use cases. If we consider the ratios of value to work (V/W), and sort the use cases based on this approach, we would see the following:

And with the previous specific value and work values:

If we were to organize our delivery of use cases based on this ratio, we would be saying “prioritize the most effective use cases in terms of value per unit of work.” This may seem counter intuitive, but it makes sense – get the most bang for the buck earlier, and you will get more value faster.

Consider what our first release would look like:

We would complete three use cases (using 25 units of available work), and we would deliver 25 units of value. We would also be able to start working on one of the use cases that would be delivered in the next release.

True Maximization

To find the mathematically proven maximal value, we have to do a bunch more work. This prioritization exercise is actually an example of the bin-packing problem, an np-complete computer science puzzle. To make a long story short, we can’t use a simple heuristic and guarantee that in all cases it will be optimal. But we can do better than “most valuable first.”

If we use the scheduling rule as follows:

Schedule the use cases in order based on the highest value-to-work ratio, skipping use cases that are “too big” for the current release.

Then we will get value out of the system as fast as possible. There are a couple problems with this approach:

1. It does not take into account that you can apply work from one release to a use case that is scheduled in a future release. Intuitively, any “remaining time” after scheduling complete use cases should be spent on the next highest-ratio use case. I haven’t proven that mathematically, but it makes sense.
2. Use cases, and their underlying requirements, and the implementation tasks to support those requirements are not actually independent. You may need to introduce one use case before another – the second use case may not be possible without the first one. Implementing a requirement that is shared across use cases will reduce the “remaining work” for those other use cases – causing a need to recalculate ratios. Implementation tasks are often dependent upon one another. The discrete tasks to support a valuable use case may require implementation work that is also leveraged in lower value use cases. Further, some implementation tasks must be performed sequentially. You can’t optimize a query before defining a database schema, for example.

The second problem actually applies to any prioritization approach that incorporates value. The nature of software development introduces constraints (X must be completed before Y can be started). Those constraints narrow the possible scheduling choices. And they make it impractical to determine the “optimal” solution. At least with commercially available tools, excluding expert systems, which can be used to solve this type of problem.

Conclusion

• Sequencing use cases based solely on value does not maximize the delivery of value over time.
• Sequencing those use cases based upon the ratio of value to effort will increase the rate at which value is delivered to customers.
• It is impractical (and possibly marginally valuable) to determine the optimal sequence for scheduling use cases to maximize value.

You should use the “highest ratio first” approach, and when a use case can’t be delivered yet because of interdependencies, skip it. Also – apply judgement to sanity check if you are doing something that seems odd – like delaying a high ratio, high value use case. Explore with the development team if there are ways to adjust their dependencies to allow for a “more valuable” delivery sequence

Today we recorded an interview with James Taylor, co-author of Smart (Enough) Systems, How To Deliver Competitive Advantage by Automating Hidden Decisions. This book, written by James Taylor with Neil Raden comes out on Jun 29th (2007), and is available for pre-order from Amazon today. Our interview covers many of the topics in their book, with a focus on the ideas inside and the benefits you can get from applying them, in just under an hour.

Smart (Enough) Systems

James and Neil introduce a pretty compelling concept – that paying attention to small decisions can have as large an impact for companies as paying attention to large decisions. All successful companies invest in strategic decision making – the big impact, infrequent decisions. But most companies overlook the benefits of investing cost-effectively in the small decisions that happen thousands or millions or even billions of times a day.

In our interview, James gives us some insight into exactly how that can happen, how to approach these small decisions, and how to manage them.

Interview With James Taylor

You can download our interview with James Taylor in mp3 format (18 MB, 52 minutes) and listen at any time. As this is my first recorded interview, please accept my apologies for the “ums and ahs.” James does 99% of the speaking, so it still turned out ok – he did a fantastic job.

James’ book is an excellent read, and eye-opening, revealing what should be obvious but remains hidden about improving our businesses. Every action and interaction a company makes is the result of a decision. Making those decisions accurately, quickly, and cost-effectively is critical to every business. Being able to adapt those decisions (and their resulting processes and customer-interactions) rapidly is the key to success in a world that is changing faster than ever. The flow of ideas is straightforward, and each new idea builds upon the previous ones, allowing readers to quickly conceptualize a better way to run their business. The anecdotes and real-world examples also solidify the concepts, and almost force the reader to think about how to apply it to their own business.

I personally recommend the book, and thank James for the opportunity to read it and speak with him about it.

More Resources on Smart (Enough) Systems

Near the end of our interview, James mentions the web site for the book. The book is available from Amazon now, and can be ordered through the link on the main page. They also have an RSS feed with news on the book, and will be launching a wiki for the book very soon as well.

This approach to integrating business and IT systems is also known as enterprise decision management, or EDM. You can read more about EDM at James’ EDM Blog.

Esoteria

We’ve studiously avoided talking about requirements for expert systems because it is such a small niche of software development. Please let us know in the comments on this post if this is an area you would like to read more about. This post is both a discussion of the main barrier to success for these systems and an introduction to future posts if you ask for them in the comments on this post.

Background

Expert systems can solve some of the hardest problems. Yet AI software has not dominated the software landscape, neither Heinlein’s nor Vinge’s fictions have become real. Why has AI software failed? After over 10,000 hours of gathering, validating and implementing requirements for expert systems, we’re convinced that it’s because programmers are being asked to solve the wrong problems.

The problem isn’t that the hardest problems are too hard to solve, it’s that they often don’t need to be solved at all.

Expert systems are often referred to as knowledge engineering systems. An expert system takes a database containing data, and adds engineering rules to it. This database is then usually known as a knowledge base. An engine can then process inputs into “the system”, and generate outputs in accordance with the data and rules embodied in the knowledgebase.

Catalyst for this post
We recently were linked to from Nilsnet (thanks!), and the first article we read was titled ‘Expert System = program that almost solves a toy version of a non-problem‘ by Nils Davis. In that article, he also links to an article that presents a criticism of AI. This just started the gears turning…

Business applications of expert systems
There are a few ‘common’ problems that are addressed with expert systems:

• Resource planning and allocation: Minimizing the amount of work-in-process inventory that has to be carried to achieve a given production schedule, determining the lowest-cost manufacturing facility to source an order, airplane route scheduling.
• Equipment configuration: Custom-configuration of high end equipment (telecommunications hardware, super-computers and clusters) with validation that the equipment will be operational as ordered.
• Travelling salesman: Minimizing the distance or time spent travelling to multiple locations to save costs in scheduling of deliveries.
• Resource optimization: Resolving otherwise computationally prohibitive constraints to achieve optimal selections of services, software, or hardware.
• Automation of tedious engineering work: Calculating and validating layout and placement information in association with engineering rules (aircraft interiors, card placement in modular systems, traffic routing in a pbx, etc)

Migration projects are the norm
We’ve talked in the past about the migration continuum, here’s a diagram from that post depicting the range of projects

Every expert system project we’ve seen has been a migration project from an existing process. Fewer than 10% have involved major process changes. These projects almost always get relegated to being plug-ins into existing business processes. The client companies simply automate a step in their existing process. In one major engagement, the client was coincidentally re-engineering their entire project, but the expert systems portion was no more than a box in a flowchart from an executive perspective. A multi-million dollar box, true, but still a second-class portion of the project.

Mis-application of technology
If engineering is the application of science to practical problems, then knowledge engineering is the mis-application of science to impractical problems.

Using expert systems to solve problems requires programmers that are “wired differently.” There are two factors that at least correlate with this.

1. The experts in these technologies tend to be unappreciative of the other elements of successful software.
2. The non-experts in most organizations do not understand the technology – neither its capabilities nor its limitations.

Myopic viewpoint

To quote the Gary Martins reference (from Nils’ post):

While all other areas of technology have enjoyed heroic advanages since [1955], AI advocates continue to pick over the same stale chestnuts that seemed so fascinating way back then.

I chuckled when I read this. I remember a project that was undertaken at my previous employer to rewrite/migrate an existing expert system engine to a new platform. The first thing the developer did was create a set of these chesnuts (9-queens, bin-packing, etc.) as his criteria for validating his work. Only after getting feedback did he incorporate real-world problems (pbx trafficing, frame-relay network design, card-slotting). This is exactly the lack of appreciation that prevents the technologists who understand the technology from thinking about valuable applications.

We’ve talked, in Intimate Domains about the importance of operating and communicating in the intersection of these domains, as the following Venn diagram shows.

The expert system experts tend to conspicuously avoid any overlap with the stakeholders/users/marketing group. My favorite quote from an expert system developer about users:

I’d rather pick up garbage than design user interfaces.

Hmmm.

Lack of understanding

Even technically savvy business people don’t have an appreciation of the applicability of expert systems to their problems. Unfortunately, there aren’t many people who can live in both worlds (switch hitters) and provide that synthesis of ideas. Many a glazed eye has ignored a powerpoint presentation attempting to explain the technology.

Lack of collaboration

Ultimately, both of these problems result in a lack of collaboration. A typical conversation might go like this:

Business: We want the software to configure the optimal racked storage system based on our customer’s needs.
Developer: Define optimal. Lowest cost? Highest performance? Highest profit (to our knowledge, no one has implemented a profit-maximizing configurator)?
B: Lowest cost. Can you make it propose upselling suggestions?
D: Of course we could, that’s trivial. But it’s not a good application of this technology. You should focus on the configuration.
B: OK, you’re the expert. How much will this cost? How fast will it run?
D: Optimality is hard. We did a prototype, and we expect it will take a minute to process for a fully maxed out system. Also, it will cost you (really large amount).
B: Well, we need it – go ahead. [Makes note to himself: Replace these guys asap - too expensive]

Here are a couple improvements to the conversation. We start by putting a requirements analyst into the loop.

Business: We want the software to configure the optimal racked storage system based on our customer’s needs.
Analyst: Do you want to optimize on cost to your customer, profits to your company, or some other metric? How important is optimality?
B: Lowest cost. What do you mean important? It’s critical – that’s how we compete!
A: Well, there’s a theoretical lowest cost for any solution. An engineer can figure it out in a day, our software can do it in about a minute. And it will cost you (really large amount). But if we only had to get within 1% of the optimal solution (say $1,000 for a $100,000 storage system), we could do it for half the cost and it would take about a second for our software to find the answer. Besides, our analysis shows that your average system cost your customers 10% more than the “optimal” price. You would be getting 90% of the benefit of the system at half the cost.
B: Our sales reps have the authority to adjust prices up and down 3%, and sales managers can cut them by 5%. We can live with 1%.
…

Maybe if these conversations had taken place, the companies that failed to capitalize on expert systems would still be around. The problem is that the technologists, in their eagerness to please, do whatever is asked of them (if they can). They never learned to question the value of what they were being asked to do. In our example, the customer didn’t need to get the optimal price – any improvement would make them more competitive. And the business person knew it. The technologist didn’t know to ask.

There are many examples like this in the engineering domain as well. A subject matter expert may say “It must be X,Y,Z” and the developer will say “not a problem.” X,Y,Z might be a choice that saves a penny over X,Z,Y. And implementing “X first” might be a lot easier. Furthermore, the SME should have said ” is important.”

Someone who understands the particular expert system, and who understands business and engineering rules needs to act as an intermediary to gather the requirements. This need is no different than with any software project. The problem is there are so few people who can fill it that the industry is languishing (again) before the problem can truly be addressed.

Conclusion

There are a lot of amazing things that can be accomplished with expert systems. The challenge is in knowing when those solutions are unneccesary. Again, please add a comment if you would like to read more (or not) about expert systems at Tyner Blain. Thanks in advance!