Agile at Scale – Outcome Driven (or Broken)

Taking agile, a process otherwise optimized for small, cross-functional, collaborative teams and making it work at scale is fascinating. You have to change some elements, and retain others, as you redefine the context. Being outcome driven, is one element you must retain – or even elevate in importance, or you fundamentally break the system of delivery

Getting Faster at Building the Wrong Thing

I “went agile” in 2000/2001 – I was writing software and leading teams at an enterprise software company at the time, and we adopted elements of XP and other agile practices in what we branded (internally and externally) as Fast Cycle Time (FCT). It was more than just incremental delivery, we really did incorporate agile principles like pairing and demoing, and engineering practices like automated build and test processes, code-reads, design for testing. It was a brave new world for me at the time and I was hooked forever. Over the next few years my career evolved through people-management, pre-sales, and program management roles into product management.

I spend a lot of time a decade ago as an early agile product manager – testing, learning, applying, and understanding how to be not only a product manager working with agile teams, but also how to be agile as a product manager. Both were then uncommon, and both are still important.

I would regularly say – and I still believe it to be true – that when a team or a smaller organization (say 100 developers or fewer) goes agile, if they are not focused also on product management, all they are doing is getting faster at building the wrong thing. To quote Jon Harmer who I’m fortunate to now call a colleague, “you just crash your car into the wall faster.”

I’ve spent the last half-year in the beginning of my journey understanding how agile product management at scale can work. Seeing things in practice. Helping a mid-size organization make the transition from waterfall (at scale) to agile (at scale). Think of a couple dozen teams (a couple hundred people) working to deliver across a couple hundred systems, for a multi-billion dollar organization. Long time readers at Tyner Blain will know that my writing here is driven in large part by the absorption, adoption, extrapolation and generalization of experiences through patterns and into concepts and applicable ideas.

For small teams, agile development without a focus on the right outcomes is getting faster at building the wrong thing. As a result you just build more of the wrong stuff. A lack of focus on outcomes is no better than a focus on the wrong outcomes. Being output-focused is not being outcome-focused.

For agile at scale, I don’t believe this is true. At scale, if you are not driving your execution through a focus on outcomes, I don’t believe you are actually faster – you’re only different. At scale, to leverage Jon’s cleverness, “you still crash your car into the wall, but not necessarily faster.”

The key difference here is around the opposing system dynamics of encapsulation and coordination. Smaller organizations have much less need for coordination – “everything” happens within the team. Why that matters is worth exploring.

Scale is Built Necessarily on Orchestration and Planning

You might believe planning to be anathema to agility. The classic war between rationalism and empiricism, which makes me want to re-read Brooks’ The Design of Design. I also suspect another luminary I’m thrilled to be working with, Dave Nicolette (@davenicolette) would even suggest that “agile at scale” may be oxymoronic on principle. He’s probably already written about it too – but I get to hear him talk about it :). At the end of the day, it depends on how you define agility, or more precisely how you define “self-directed.” There’s always a context in which self-determination applies. Since this is a heavy name-dropping post, I will once again thank Andy Polaine (@apolaine) for the visual which is permanently grafted onto the concept of nested contexts for me now.

Agreeing that teams have latitude of operation and autonomy of decision making within the context of working together is a tenet of agile. Doing this to achieve a shared desired outcome is the key to harnessing agility as a powerful force for achieving business objectives.

I would propose outcome-driven agile teams are the pry-bar we can use to loosen and potentially untie the Gordian knot of agility and scale being logically consistent. So – as long as we are willing to accept a definition of agile where “self-directed” means autonomy within the context of the role of that team, we only need to articulate the rest of the context.

• Agility at scale requires multiple small teams
• Each team has autonomy within the context of the role of the team
• Each team is working in coordination with other teams to deliver
• Each team is focused on business goals

This focus on the business goals – the “why’s” of investing to build something – is the definition of being outcome driven.

This coordination requires planning, in an ongoing fashion. There is a theoretical continuum from no-coordination to pure synchronous action. The fewer highly-coupled systems, and the fewer bottlenecked shared resources, the less coordination is required. As a technologist, I can imagine a situation where all systems are decoupled, and no teams are over-subscribed; through a combination of architectural and training and staffing decisions. As a consultant, I am comfortable there will never be an organization at scale which can avoid coordination and planning.

Everyone Has a Plan until…

“Everyone has a plan, until they get hit.” – Mike Tyson

There are many quotable versions of this concept, this is the one that grabbed me years ago. Other fighters would prepare for a fight with Mr. Tyson, crafting a plan to dance around him, tire him out over several rounds, and then use their greater endurance and skill to win the fight. Mr. Tyson would regularly make those plans worthless by doing something like knocking his opponent unconscious in the first half of the first round. As Mr. Tyson said “Everyone has a plan, until they get hit.”

This for me is the touchstone to remember that there are many reasons your plan will change. Another colleague of mine, Jeff Howey, is fond of asking the room – “the plan is a lie, right?” To which everyone nods yes. This isn’t something unique to agile – the waterfall plan is a lie too.

About 5 years ago, I explored several reasons why products fail. Among the reasons are variations which all reflect a need to change the plan. You realize (by gaining new information, outside of the development process) that your team is going to fail, so you need to change. A formal mechanism for doing this is a pre-mortem.

1. You picked the wrong customer for whom to solve problems.
2. For the customer you picked, you picked the wrong problem to solve
3. You made the mistake of declaring success without completely solving the problem
4. Your solution approach – you discover – will not solve the problem
5. Your execution / implementation is unable to rise to the needs of your design, and therefore will not solve the problem (even with a viable design)

The last two scenarios, (4) and (5) are actually not examples of selecting the wrong problem, they are examples of failing to solve the selected problem. Usually, these are not problems with the plan (the choice of problem to be solved), although they could be. In the situation where your strategic choices about which problems to solve are both valuable and desirable, it is possible that they are not feasible for your team to solve. It may be that your strategy is one which is beyond the skills of your teams, in which case you need to choose a different problem to solve. I’ve listed them for the sake of thoroughness – although I have only once seen this a driver of failure. Usually, other things are also wrong.

You can run a powerful exercise, called a premortem, to help discover in advance what is wrong with your plan. Proposed by psychologist Gary Klein, it is one of Daniel Kahneman’s favorites. Here’s a 3 minute explanation

Outcomes and Collaboration and Agile

When a small (e.g. “as designed”) agile team discovers that they are working on the wrong problem – for any of the reasons listed – they can easily change. They can change because they are directly collaborating with their business stakeholders. They are directly engaged with their customers acquiring the signals which trigger the discovery that the plan needs to change. From an organizational perspective, you can describe this team as being encapsulated – the ability to change the plan is within their collaborative area of influence.

They can change the plan. They can change the plan because they are working together – both the business side (sponsors / customers / stakeholders / users) and the technology side (the team creating the product or solution). They share an understanding of the value/opportunity/desirability and the cost/complexity/time of solving the problem. They share an understanding of the suitability of the design to potentially solving the problem. They share an understanding of their ability to execute against the design. They can collectively manage technological and market uncertainty about why, what, and how they are doing their work.

This team is outcome driven. And when they get metaphorically punched in the face, they can pivot to a new way to solve the problem, to a new definition of “solved” or to a new problem to solve. Because the team is encapsulated the complexity of this change can go unnoticed. The ripple-effects and new orchestration requirements are resolved “inside” the team.

If the team were not outcome driven, their ability to radically change would still be comparable – the team could be tasked with a new set of assignments, and they would discard their previous work, resolve the dependencies and do the new work. From outside of this encapsulated team, the change process would appear to be just as easy.

This misapprehension – that change is easy for agile teams, regardless of focus on outcomes – is the root cause of the difficulties organizations struggle to adopt agile practices for large ecosystems / organizations.

Agile Without Outcomes

What happens when this team tries to not be outcome driven? OK. Insert the sound of screeching tires (or a phonograph needle dragged across the surface). I have repeatedly challenged other authors for doing the following

1. Describing a bad thing.
2. Assigning the name of a good thing (like roadmaps) to the bad thing.
3. Eloquently arguing about why the bad thing is genuinely bad.
4. Declaring the good thing from (2) to be bad, because the bad thing from (1) is actually bad.

I will try and avoid the same mistake. If the team is not outcome driven, their process is not agile (it is only incremental delivery against a project specification). What this team can do, as I’ve said in the past, is generate more output. This is a more efficient process, but only more efficient – not more effective. It helps you to crash into the wall faster. And that makes it a bad thing.

Effective coordination and execution is irrelevant when you’re solving the wrong problem.

Outcomes and Progressive Elaboration

An interesting characteristic of agile, for product managers, is how “requirements” are managed. These days, I really talk about intentionality because what I’m talking about is the organization’s intention to achieve an outcome. The term intentionality resonates with my audiences, and helps me focus.

One agile principle (I believe I first heard it from Kent Beck back in 2000) is about doing things at the last responsible moment. It is also a characteristic of lean manufacturing. In software development, “doing something” is creating work-in-process inventory. The insights from thinking, designs from designing, code from coding, tests and results from testing are inventory. Inventory. Don’t create it until you have to. YAGNI is one way to remember this – “you ain’t gonna need it (so don’t build it).”

Just as rolling-wave planning encourages us to sketch out rough time-lines and directionally-correct plans into the future, with detailed planning reserved for the immediate future, intentionality is progressively articulated relative to time and scope. We avoid investing to create something likely to be discarded – specific details about the distant future.

There is a long-term goal or outcome you are trying to achieve. In support of that long term goal you may define a handful of strategic initiatives. To advance those initiatives you may also identify a set of deliverable, valuable products or product capabilities. These are steps along the path to the long term goal. The first of those capabilities may get progressively elaborated into features and stories – which are what the team works to deliver in releases and sprints, getting feedback along the way. You do not build out all of the stories which span the entirety of every initiative designed to realize the goal. You progressively elaborate, as needed. At the last responsible moment.

A scannable list of one way to progressively articulate intentionality:

• Strategy – a long term goal, multi-year in scale and organizational in scope
• Opportunity – an initiative supporting the strategy, easily a year or longer, maybe divisional in scope
• Epic – a key component or capability as part of the opportunity, market-positionable, product or service scoped
• Feature – a collection of outcomes from a user’s point of view, atomically deliverable or testable for customers
• Story – one scenario of value delivery for at least one customer in at least one context

With a small team, all of this progressive elaboration does two things. First, it establishes context – the big picture. Second, it drives focus – what the team works to achieve right now. However, it comes at the cost of inextricably coupling requirements and design elements at multiple levels.

Progressive elaboration of intent incorporates elements of design and planning.

• The stories make sense, given the selected feature.
• The feature reflects a set of design decisions (in terms of both interaction and implementation) to realize the intentionality of the epic.
• The epic manifests some hypotheses and assumptions about how best to realize an opportunity.
• The opportunity represents one aspect of one way to succeed with a strategy.

This is more than intentionality, it is also design and planning. They are inextricably linked.

As you cross multiple contexts – progressively getting deeper into the problem space – you are grafting planning elements and design decisions into what the team will do.

As you engage with the market over time – you will discover you need to adapt the plan. As a small agile team, you easily do this. You revisit decisions up and down the elaboration ladder and you adapt. Operating at scale is about orchestrating and coordinating. Planning. This is why an outcome focus is required for agile to work at scale. As desired outcomes change, plans change

Outcomes and Planning at Scale

When you’re orchestrating a large organization working together, there are necessarily elements of decomposition and allocation of responsibility for executing parts of a plan. This is true for waterfall and agile organizations. They both do it, but they both do it differently – an aspect which makes transformation from one to the other difficult.

In the waterfall world, areas of the organization, and then divisions, and teams, will each be “assigned part of the plan” to execute. Someone coordinates that execution and allocation of responsibility across teams. This is how large projects have always been done.

When a not-agile large organization is calling itself “agile” but orchestrating a plan across teams, instead of progressively elaborated intentionality, the system behaves similarly to a large waterfall organization.

When the plan changes, things get interesting – the may we live in interesting times kind of “interesting.”

One of the teams will discover that their particular portion of the plan won’t work. Perhaps they get negative customer feedback on a prototype or the deliverables of the current sprint. The team wants to change the plan, to incorporate this feedback. The problem is that the team only knows the plan and not the intent; they lack the context to effectively deviate from (or modify) the plan to address the very real problem. They have to bureaucratically go “up the chain” with their findings, and initiate discussions around how to adapt the plan. In the meantime, nothing gets done. This is “at scale” but it is not “agile.” It is the same for waterfall and not-agile teams.

For a team to be able to adapt the plan they need to understand the intent. With insight into intent, they can be self-organizing within the context of how they approach delivering their portion of intentionality. They can assess if whatever they learned invalidated the framing of intentionality one-level up as well. And that team can determine how they will adapt, without having to escalate all the way up through the progressive elaboration – except when that is what is required.

The following distinction is key. When each team is working a portion of a plan, they have no insight into cross-team dependencies and must always escalate before changing. There is not autonomy with respect to adapting to changing circumstances. When each team is working to deliver a portion of intent, they have autonomy and flexibility to change what they build to deliver. They only escalate when what they learn invalidates intentionality – and then escalation only goes as far as it needs to.

What is unique is that with a focus on intentionality, at each stage of progressive elaboration, the teams can adapt within their context, without disrupting or invalidating the greater context of intentionality. This is the secret sauce of agility at scale. Without it, you only have waterfall, agile-in-name-only, at scale.

Agile at scale cannot work without intentionality.

Speed of Change Requires Intentionality

Near the start of this article, I asserted that without intentionality, at-scale “agile” organizations are not faster. If there were no changes to the plan, they would be faster than a waterfall organization – avoiding death marches and being generally more efficient is an outcome of a frequent delivery cadence. However, every team will get punched in the face and every plan will have to change.

The need to change the plan – and all of the cross-team dependencies – is what slows down not-agile at scale teams, making them no faster than waterfall teams. Working against progressively elaborated intentionality is what allows agility and adaptation by small teams working in concert as part of a larger organization. Without delays.