Business rules are often hidden in processes as hidden decisions.  Once you discover that hidden decision, how do you communicate the impact of exposing and managing the decision?

Hidden Decision

In our previous article on hidden business rules and enterprise decision management, we looked at a process that had a hidden decision.

First, take a look at the process with the decision still hidden:

Here’s the process (with the decision exposed) from that article:

The decision that was hidden was an implicit decision – “Pick One?” – for which the answer was always “No.”  After making this discovery, and communicating with your team, you then have to decide if you are going to make the decision explicit.  Making the decision explicit will involve defining the business rules for how to make the decision, and (if done in software) implementing the decision, or the ability for a person to make the decision.

Impact of a Decision

The diagrams above allude to the fact that there is money to be made in the process, in the “Continue Automated Process” step.  We need to provide a better understanding of how / when money is made.  A slight expansion of the process flow above (replacing the “Continue Automated Process” with a couple steps, looks like the following:

[click for larger version]

The last step introduces a real-world decision: “Sell products?” and only when the answer is yes does the process “Earn Revenue.”

If you’re analytically minded, you can apply percentages to each branch of each decision, and add everything up to figure out an answer.  That might be reasonable for a simple example like this one, but it can become too complex with a more complicated process.

There’s an easier way.  And since the goal is to communicate the impact to someone, you want an easier way.

Truth Tables and Decisions

A truth tables is a tried and true artifact for describing combined sets of decisions.  You can use it effectively to extract information from a decision-heavy process diagram.  Consider the following application of a truth table to the process above:

[larger version]

Each of the decisions in the process flow has a column.  Each row represents a unique path through the flow.  Each cell contains the result of the decision, for that path.  Note that when a path does not involve a decision, there is no value in the cell.

That provides a comprehensive view of the process, which you could build upon.  However, in this case, you are focusing on the impact of the hidden decision (“Pick One”).  The following paths, from the truth table, involve that decision:

[larger version]

Ignoring the other paths, and re-organizing, you find the following:

[larger version]

50% of the time, when “Many” results are found in the first decision in the process, if you pick one of the results, you will earn revenue.

[larger version]

When you do not pick one, 20% of the time, the customer (or user) abandons the process.  For the users that do not abandon the process, you earn revenue 50% of the time.  Taking both into account, you earn revenue only 40% of the time.

Adding Up the Impacts

When there are “Many” results from search, you can improve the performance of this part of your process by 20% – specifically, you increase revenue by 20%.  But only if you always “Pick One” of the many results. You can extend this analysis to determine the percentage of None/One/Many outputs from the search at the start of the process, and determine an overall impact on the process.  If search returned many results 10% of the time, you can increase revenue (overall) by 1% by always picking one of those results.  That gives you the business justification to calculate ROI on exposing the hidden decision.

The end result is that you have a simple way to communicate the impacts (truth table) in the language of stakeholders (ROI).

A business process is not just a sequence of steps.  A business process is a series of decisions and actions.  Some decisions are obvious and can be actively managed.  Some decisions are hidden, and until you discover them, you can’t manage or improve them.  Here is a real-world example of the discovery of a hidden enterprise decision.

Enterprise Decision Management

Over the past couple of years, I’ve collaborated with and become friends with James Taylor, an enterprise decision management guru.  He and Neil Raden wrote the book, Smart (Enough )Systems , you should get it if you don’t already have it.    It is sitting on my desk right now, and every time I pick it up I have to mark another page with a post-it note so I can find that good idea again later.  If you aren’t convinced, listen to this interview with James about their book.

James and I developed a presentation, Getting It Right: Rules and Requirements in Software last fall for the International Business Rules forum.  Below is a slideshare of a shortened version of the presentation, delivered to the Austin IIBA this spring.

A large portion of that talk was dedicated to discovering hidden business rules within use cases.  The reason we dedicated so much time to rules discovery is that the first step to managing decisions across your enterprise is to discover and isolate those decisions!  In this article, we will look at an example of a decision hidden within a simple business process.

A Simple Business Process

The following is a simple real-world business process.  The details of the steps in the process have been modified to clarify this example.

The process flow example above depicts an automated process that involves searching internal systems for an existing customer (record).  There are three possible outcomes from the search, shown above from left to right: no results are returned from the search, a single result is returned, or multiple results are returned.  When no results are returned, a new customer (record) is created, and the automated process continues.  When a single result is returned, that customer is used and the process continues.  When multiple results are returned, a new customer (record) is created and used, but the automated process is interrupted and a manual intervention is required before the process can continue.

This third path confused me when it was initially presented to me this way by the subject matter expert who explained the process to me. The business creates a new customer (record) even when multiple results are found because they don’t have time to determine which result is the right one, within the automated process.  They would rather create a duplicate customer (record), continue with the automated process, and clean up the mess later.  That’s much better than losing a sale (a very real risk because of delays due to manual oversight.

As part of this elicitation session, I applied a variation on active listening by talking to the diagram above (on a whiteboard) and enhancing it.

Impacts and Decisions

One technique that is effective in reaching agreement, generally, is to inspire the other person to “have your idea” so that it becomes their idea.  Once I understood the process above, I realized that there was an implicit decision in the process.  Instead of making the statement – “you have a hidden decision in your process” and then hoping the business stakeholder (also in the room) would care, I chose to start with the impact.  Presenting the impact of a hidden decision first might expose that decision and trigger acknowledgment of the hidden decision by the stakeholders.  The following diagram shows how I first modified the flow on the whiteboard.

The subject matter experts had told me that when there is a manual intervention in the otherwise automated process, the process is sometimes abandoned.  Imagine customers walking away from an online sale because (for some reason) the sale cannot be completed without an unexpected delay.  I represented this with the “Abandon Process?” decision in the flow.  This decision was never drawn before, because it is the customer’s decision, and it can not be controlled by the business.

I also pointed out that the business makes money only through the last step in the process, “Continue Automated Process.”  My goal was to accentuate that this abandonment decision has a very real value.  The stakeholder and subject matter experts agreed – this was something they had told me previously.  But they had never drawn it directly within their process.

As it turns out, this did not trigger any additional insights.  It did however, set the stage for describing the relevance of the hidden decision.

The Hidden Decision

After setting the stage, I updated the process flow diagram to expose the hidden decision.

The hidden decision is the decision of picking one of the many search results (top right corner of the diagram).  Your first response may be that the business is not picking any of the results.  My response is that choosing none of the provided results is a choice, just as picking any one of them would be.

This view of the process highlights that if the business could automatically choose a “best” result when searching returns multiple results, the business would avoid the risk of abandoned processes due to manual oversight.

With the current process, the business stakeholder acknowledged that the hidden business rule is ” pick none of the results 100% of the time”.  With current project schedules, there is nothing that can be done about that in time for the immediate software release (of the software that embodies this process).

Before our meeting was over, the business stakeholder added determination of future-state business rules to define the “how do we pick one?” decision to the deliverables for the next release.

A Smart (Enough) Decision

James and Neil (I would guess) will tell you that ideally, the decision (should we pick one, and which one should we pick) should be informed by the process.  A feedback loop can be created where we identify the customers (the people who might abandon the process), and determine statistically which ones are most likely to abandon the process.  For those people, it is especially important that we be able to “pick one” from a set of search results, thus completely avoiding the opportunity to abandon the process.  For those people who are unlikely to abandon the process, the “pick one” decision is less important.

If it were easy to “pick one”, we would consider changing the way search is handled to never return multiple results.  In this real world example, that is not an option (I’m intentionally not sharing the reasons, to avoid disclosing too much).  The only option in this case is for the business to respond intelligently (enough) when presented with multiple search results.

Conclusion

This process flow example has two primary outcomes – generate profits, or don’t generate profits.  The first modification of the process flow makes that possibility explicit.  That allows for an analysis to determine if there are any hidden decisions in the flow (there are) that impact the outcomes of the flow.  There may be other hidden decisions, but they do not impact the profitability of this process, so they were not explored.

Traditional requirements happen after a multi-system architecture has been defined.

But what about the requirements that feed into that architecture? The requirements that drive the enterprise architecture decisions in the first place? We haven’t talked about those before.

Setting The Stage

The typical enterprise software requirements project ecosystem looks something like the following:

1. An existing application exists within a network of (logically) interconnected applications that collectively comprise an enterprise architecture.
2. Business processes exist, each of which utilizes one or more of these applications to achieve tangible business objectives.
3. Some of these applications have user interfaces, and therefore have users who are trying to achieve the goals associated with the business processes.

Within that environment, the typical enterprise software requirements project looks very familiar:

1. There are new processes, or capabilities that are desired by the business.
2. An enterprise architect has decided that these capabilities will be embodied into one or more of the existing applications. The end result is that each application has a “defined scope”, and a set of goals.
3. The requirements project for each application under development is initiated with these objectives and boundaries.

What if you are tasked with defining the ecosystem, instead of defining a product within a pre-defined ecosystem? How would you go about it?

• The architect makes design decisions about what each system in the architecture does.
• The developer makes design decisions about what each class in the application does.

Isn’t this just design, writ large? Why should business analysts or product managers be involved? This is just an implementation activity on steroids.

What We Are NOT Talking About

There is a very good article at IBM that defines architectural requirements as application requirements that have significant impact on the architecture of a single application. That is not what we are talking about. Here’s a link, since that is a good article – if that’s what you were hoping to read today.

The Lifecycle of Creating Software

In One Man’s Trash, we looked at the life cycle of developing a software product from the perspective of the contributors to that product.

Our analysis was of a single product – not an application that has to live in an ecosystem with other products. That analysis showed four layers of solving the problem, but only one meta-layer of solving the problem.

In the uselessly big picture view, there are multiple layers of “requirements” and “design” throughout the life cycle. When defining requirements within the “single software product” scope we wrote about before, most of the following is actually a distraction.

1. Companies and individuals have money to invest, and they have goals associated with those investments.
2. They will choose a mix of investments, where that mix is designed to achieve a set of objectives.
   1. Manage volatility
   2. Generate ROI (where the return is utility, but is easiest to simplify as a financial return)
   3. Mitigate risk
3. Within the constraints imposed by that mix strategy, they design a portfolio. That portfolio delegates roles to individual investments. Investment 1 is responsible for generating a high return at a high risk. Investment 2 is responsible is designed to hedge investment 1, mitigating risk while reducing overall return. Investment 3 may be responsible for generating a smaller, but highly reliable return. Each investment thus has a scope and a set of goals.
4. If one of those investments is in a company, it now has a scope and a set of goals – and a large company will develop its own portfolio of investments – choices in how it allocates funds internally. One of those investments may be a choice to operate in a particular industry, where those operations are defined by a set of goals and constraints. [Note - conglomerates may have additional, recursive sets of layers 1 through 4, as they manage a portfolio of companies.]
5. Those operations may manifest as a set of business processes, initiated by people using software, and supported by other systems internally. The choice to use people, or software, and how to organize the systems is one of design.
6. Once that design choice is made, each individual piece of software has a defined scope – a set of responsibilities within the larger architecture. And those responsibilities become goals and constraints within the context of a set of contracts between the systems. The entire chart above lives within this one step in the bigger picture.

When managing software requirements in the typical enterprise software project, this scope and these goals is already defined. Here are a couple examples of an application being defined within an enterprise architecture:

• A company needs to employ direct sales reps to sell their products. The company has a software application that the reps use to sell the products, and that application is integrated with another system that manufacturing uses to fulfill the orders. An accounting system also exists that is responsible for collecting funds due for each of the orders. Sales reps have to be compensated for their work, and a compensation system is identified to calculate sales rep compensation. It must be integrated with the existing systems and is responsible for determining how much the reps should be paid. The compensation system must integrate with the payroll system that is responsible for paying the sales reps.

Each of the applications (sales, fulfillment, payroll, collections, compensation calculation) has a defined role to play in the architecture. A typical requirements project would be to define the requirements for the compensation system. It helps to understand the ecosystem – the context in which this product has defined responsibilities – but that is generally a constant, not a variable. So, most requirements books, tools, and techniques are designed to operate with a presumption of a fixed set of goals and scope. The goals and scope of the product usually need to be discovered and validated, but they are perceived as immutable – at least within the timeframe of the project.

“Standalone software product” product management operates under essentially the same principles – the company is assumed to have a strategy, a target market, goals and a scope for the product. Requirements work for a standalone product is similar to the requirements work for an enterprise application that “knows its place” within an enterprise architecture.

But what about the requirements that drove the enterprise architect to choose an approach that involves separate applications for sales, fulfillment, collections, payroll, and compensation? These are architecture requirements

Defining Architecture Requirements

We know how to define and manage requirements within a predefined environment. Start with goals, and work our way down to “design.” Other people on the team will start at design, and continue through development, test, deployment, and adoption. This is a pretty comfortable environment. It is also an effective environment. When a problem involves the interaction of multiple systems, an efficient way to tackle that problem is through decomposition. Just as an investor decomposes a portfolio into individual investments, a company will decompose enterprise software into multiple discrete systems. And a developer will decompose a single system into multiple classes (or multiple modules, each made of

And we know the importance of requirements – the most successful teams don’t go immediately from goals and constraints to development. Multiple studies demonstrate the cost-effectiveness of using requirements effectively to prevent mistakes in implementation. It stands to reason that we are missing an extremely large opportunity if we don’t apply equivalent rigor to the architectural decisions that lead to the definition of those goals and constraints. So how do we go about defining the requirements that feed into the architecture?

Here’s a really good article by Scott Ambler that provides a more detailed perspective, and practical guidance for enterprise architects. The rest of this article is geared towards business analysts who find themselves supporting enterprise architects pursuing a path like the one Scott describes.

The first step is to identify the business processes – all of the business processes that are relevant. Using our example, these are the compensation-management processes, sales and fulfillment processes, payroll processes, and collection processes.

Part of defining these processes is identifying the stakeholders. We’re dealing with a much larger scope here – so valuation of the different stakeholder goals becomes much harder. It is easier to prioritize “sell bundled products” versus “sell customizable products” than it is to prioritize “pay sales reps” versus “fulfill orders.”

A very high level view of the processes is enough to do an first iteration of an architectural design. This is also sometimes called a straw man architecture, or candidate architecture. Notice that we said “first iteration.”

The next step – and arguably, it is a concurrent step – is to define the boundaries between those systems. Where / when does the order placement system (first system) transfer responsibility to the fulfillment system (second system)? And what does the first system have to provide to the second system for fulfillment (and possibly collections and compensation) systems to do their jobs? And how must those systems respond? What information must they provide to meet user requirements? What information must they provide to satisfy audit requirements or regulatory rules?

While this looks like design, I would argue that this is also feasibility analysis. The architects are designing the enterprise architecture – or the system under development. But that system is also an input into the requirements processes for each of the individual applications in the architecture. The scope of each individual application is defined by the system boundaries. The goals for each individual application are a direct result of the responsibilities that are identified by mapping the business processes to the individual applications.

Now we have a candidate architecture, a set of identified business processes, and a mapping of those processes onto that architecture – defining the responsibilities of each system. This yields a feasibility analysis by the architects. They will take this assessment (and any identified gaps), and modify the architecture – adding or removing systems, changing responsibilities, and otherwise iterating on the candidate architecture. Then they will review and iterate again – until there is a feasible design – just as developers will invest in the design of an application until there is a feasible solution to meet the application requirements. And this process should involve collaboration with stakeholders, and revisiting (or even redesign) of business processes with the candidate architectures in mind.

Conclusion

Developing architecture requirements (and architectures) is an iterative layering exercise. As much as it isn’t “our job”, our contributions to defining the architecture requirements will not only serve to verify the feasibility of the architectural approach, but they will provide clarity about the role of each application in the architecture. And that clarity of purpose sets the stage for a more effective “traditional” requirements process for that application.

This will also cause a significant reduction in requirements churn later – as flaws at the inter-system level will be identified and resolved before application requirements get underway in earnest.

Documenting processes is something most business analysts have to do. The goal of documenting the process is to communicate requirements. By establishing a shared understanding of the process, you can establish the context for the requirements. Easy processes are easy to draw and understand. When documenting a more complex process, you need to provide the same clarity and consistency. In this article we show how to document asynchronous process steps to maximize the clarity of the documentation.

Speak in Your Reader’s Language

The diagrams you choose to use represent a graphical language for communication. It is really no different than choosing a language in which to write your textual requirements. You need to choose a graphical language that the consumers of your document understand.

You can use BPMN (business process modeling notation) to document your processes, and be quite precise in your description of the process. However, if the team you are communicating with does not know BPMN, it may not be the most effective way to communicate. You are faced with a trade-off. You have to either use a language (process flow charts) that is easily understood but hard to extend, or use a language that is precise in interpretation – but only to people who know the language.

Here’s a solution that uses diagrams that are better known than BPMN, leveraging process flow charts and UML sequence diagrams.

Asynchronous Process Definition

An synchronous process is one where two objects or systems talk to each other. The first system sends a message to the second system and waits for a response. An asynchronous process is one where two objects or systems talk to each other – but the first system does not wait for a response from the second.

Imagine making a bank deposit. If you go to the bank, hand your money to the cashier, and wait for a deposit slip, you have just participated in a synchronous process. You did something, and waited for the cashier before doing something else.

Now imagine driving through a toll booth. You drive up to the toll booth, roll down your window, and throw your money into the collection basket. You wait for the machine to count the money, the light turns green, and you drive on. This is also a synchronous process.

Now imagine that you are in a hurry. You roll down your window as you approach the toll booth. You slow down, and as you drive by, you throw the money into the collection basket, and accelerate. You don’t wait for the light to turn green – you trust that you had put enough money in the basket and you move on. The machine still counts the money, and the light still turns green – but you didn’t wait for it. You continued to do what you were doing – driving in a hurry. You just participated in an asynchronous process.

Drawing an Asynchronous Process

Consider a simple asynchronous process – ordering a hamburger at Wendy’s.

Wendy’s has an interesting process for serving “made to order” burgers quickly. They cook a small number of hamburgers in advance, and keep them warm for a short period of time. If someone orders a burger, they take one of the already cooked burgers, and assemble it with the custom toppings. They also put another raw burger on the grill to replace the one they just sold. The challenge for the grill cook at Wendy’s is to have enough burgers ready that no one has to wait – while not having so many burgers precooked that any of them dry out before you order.

The labels in the boxes above are generic, but it could be used to describe ordering a burger at Wendy’s. The box “calling process” represents you ordering a burger. The cashier processes the request, and then the flow splits into two parallel paths. The grill cook immediately prepares your hamburger, and simultaneously starts cooking a new hamburger to replace the one he just used.

After your burger is prepared, the process returns to you – you get your burger and go eat. You don’t wait for the new hamburger to finish cooking.

A process diagram doesn’t normally do a very good job of communicating that a process is asynchronous. We’ve had good success in using a modified box – with a small clock in the lower right corner – to indicate that a process is asynchronous. You could infer that the “delayed activity” is asynchronous because the flow moves directly from the “immediate response” to the “return” symbol. But in a good diagram, you don’t have to infer.

Sometimes, it helps to create an alternate view of the same process. This additional view can make it easier for the consumers of your diagram to understand the process.

An Asynchronous Sequence Diagram

Another way to diagram processes is with UML sequence diagrams. Scott Ambler has an excellent article explaining how to read a sequence diagram. The same burger ordering process could be drawn with the following sequence diagram.

The diagram above uses the same generic labels again, but still applies to our example of ordering a hamburger at Wendy’s. Inside the “par” bounding box, you get the immediate response (preparing your burger), and in parallel, the delayed activity (cooking a replacement hamburger). Note that, in the diagram, you don’t have to wait for the response from the delayed process. You get to eat your burger before the new one has completed cooking.

Summary

Asynchronous processes don’t come up often when learning about diagramming techniques. But you find them all the time in the real world. One of the easiest ways to speed up a process is to find steps that take a long time, and figure out how to do them asynchronously. You don’t wait for paperwork to be processed when signing up for an insurance policy – the coverage is effective immediately. You don’t wait for a loan application to be processed before driving off in your new car. Once the loan is approved, you get the keys. Many different software and manufacturing processes take advantage of queueing (like the Wendy’s example) to allow for faster “real time responses” to requests.

Being able to draw and discuss asynchronous processes is a good skill to have in your repertoire when documenting requirements in the real world.

Product managers are often described as the hub or center of a software development organization. Saeed Khan takes umbrage with this under-appreciative image in an awesome article about information flow, product managers, and the SDLC.

Hat Tip

I love that the community of great people writing about product management and business analysis has grown so richly over the last couple of years. Yet again we get to thank someone else for pointing us to a great article. Steve Johnson (vote for him) points us to the On Product Management blog in a recent article. They’ve quietly amassed over 50 articles in the last three months, and we, thanks to Steve, found one we loved today.

Information Flow

In one of a series of articles on being a great product manager, Saeed discusses the importance of the product manager not only in creating information, but in getting the right information to the people who need it. Here’s how you know you want to read the article. First, scan it. Then click on the two diagrams. The first one shows the network of roles in a typical large software company, showing the more significant information flows between them. The second diagram shows a detailed heatmap of the amount of information flowing through any particular role as a function of where the product is within the SDLC.

As soon as you see these two diagrams, you will immediately walk away with the appreciation that Saeed speaks with perspective, experience, and insight. Now you know you want to read the rest of the article in depth.

Enabling, Not Documenting

One thing I really like about Saeed’s article is that he positions part of the role of the product manager as enabling other people on the team to be more effective. Product managers don’t make all the decisions, don’t do the product implementation, and don’t do the tactical selling activities. Other people do. What product managers do is enable them to do it better, by providing them with the right information.

In another good article from the same blog, Alan tells about the tragic demo from a company pitching software to product managers – and discussing the features, not the value. Unlike in that demo, Saeed is focused on the value of information flow within an organization. He doesn’t talk about the artifacts that are created (if any), he focuses, particularly with the heatmap, on how different roles are involved in different stages of the SDLC.

The article makes good points, and the heatmap drives home that there’s rigor behind the assertions. And as the first article I read from this blog, I’m extremely excited to read more. I would like to see (and will now have to think about) how different roles need information with an agile product development lifecycle. These are the types of inward focused assessments that help us get better at product management, and help our teams achieve greater software product success.