Why We Created Agile 2

Most Agile transformations struggle. According to an Allied Market Research study, “63% of respondents stated the failure of agile implementation in their organizations.” This is despite the fact that Agile is increasingly seen as a strategic enabler. According to a McKinsey article, “Done right, [Agile] could reduce risk and improve decision making.”

One of the biggest challenges in an Agile transformation is that executive leadership does not “get it.” In a Forbes article, Peter Bendor-Samuel explains that “in companies with a culture focused on meeting objectives and never failing, executives often proclaim [Agile] experiments and pilots as victories even when the actual results were not a success. In effect, the organization lies to itself…The transformation initiatives then move forward on flawed foundations and incorrect learnings.”

My own experience strongly bears this out. In the majority of my consulting engagements, senior leadership was effectively in the way. Either they had an incorrect understanding of what Agile approaches would work for them, or they applied a traditional approach — what has worked for them their entire career — and treat Agile as a “process rollout,” in which they task analysts with defining the new approach and a set of tasks to implement it.

That does not work, because Agile is mostly about all of the people in an organization learning new ways of thinking, and then applying those new ways of thinking to define their own new ways of working. Thus, one cannot define it up front.

Agile transformation is a journey, and most of that journey consists of people learning and trying new approaches in their own work. Over time, and with the help of Agile coaches, consultants, and forward thinking leaders, various groups integrate their new ways of working, to create a more Agile system in an increasingly larger context. The many Agile clusters combine, creating an Agile whole — an Agile organization.

The problem is, even that approach does not always work well, because there are many Agile ideas that are flawed. Some of the ideas expressed by the Agile Manifesto are oversimplifications, which leads to poor application of those ideas.

Other ideas originate from the Agile community at large or from other sources, such as Extreme Programming; yet some of those ideas are controversial. For example, consider the Extreme Programming practice known as “Test-Driven Development” (TDD), which emphasizes a unit test-centric approach to programming, and should not be confused with other “test-first” methods. Unit test-centric TDD is often viewed as an Agile “best practice”, but many people feel that it is the opposite of how they work best. Unless an Agile coach understands the nuances and diversity of opinions on these issues, they are likely to give inappropriate advice to the team or the organization.

There is also the fact that while many in the Agile community are truly agile in their approach to things, the Agile community as a whole has considerable inertia in its thinking — a great irony, because it means that the Agile community is not very agile. Practices that were championed early are very hard to dislodge from popular Agile thinking. For example, the practice of putting all of the members of an Agile team in an Agile “team room,” all sitting next to each other, often side-by-side at a table, has been shown to make it very difficult for people to focus.

An Agile coach colleague of mine who had long advocated that practice one time had to sit at the team’s table, and work on something herself. After only a few minutes she looked up with an expression of frustration and said “How can anyone concentrate like this?” The ambient noise and nearby conversations made it difficult for her to focus on her work.

Yet the Agile team room persists. It is the “standard” arrangement for Agile teams. Do an Internet search for “open plan office” and see how many search results come up. There will be many — and almost all will be articles that describe the open plan arrangement very negatively. Yet the Agile team room is still the default way to arrange Agile teams. Somehow, the Agile community is deaf to the push-back on the team room idea and shows intransigence on the practice.

Today, the COVID-19 pandemic has finally disrupted the Agile team room. However, I will not be surprised if when people return to the office, that they return to an Agile team room — even though it is now known to be a very bad setup.

Agile Is Broken

It is things like these that contribute to the challenges that organizations have when they try to adopt Agile methods, and there is more — a lot more. Early Agile approaches were entirely focused on a single team, but it often takes many teams to maintain a technology product.

Today many organizations have complex multi-product digital platforms, and those products share a set of core digital services. The challenge is therefore less in how to operate a team, and more in how to maintain a multiproduct ecosystem. Changes to one digital service might affect tens or hundreds of products, each maintained by tens of teams. One bank that I am familiar with has 500 software teams. Do they need 500 teams? That is a valid question: but it is certain that their platform could not be maintained by one team.

On the hardware product side, consider machinery. A modern commercial engine cannot be designed and built by a single team of five to ten people: today’s engines contain many highly engineered parts, many from myriad third party suppliers but others created in-house, and engines also have multiple electronic control units, each containing tens or hundreds of thousands of lines of software and all interconnected by a communication network. To design and build such an engine, one needs many teams.

As a result of these complex multi-team realities, Agile “scaling frameworks” sprang up. Unfortunately, there are several such frameworks, and while at the lowest level they usually are similar, above the team level they are radically different. An organization therefore cannot look for the “best practice” approach to applying Agile ideas at scale: there is no accepted “best” approach, or common scaling model that presents a widely accepted range of choices. Instead, there are many competing proprietary and branded approaches.

Worst of all, early Agile essentially dismissed the need for leadership. The Agile Manifesto stated two things that related to leadership:

1. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
2. The best architectures, requirements, and designs emerge from self-organizing teams.

The first one states without any qualifier that one should “trust” the team — in other words, a team needs no oversight. The second one strongly suggests that teams should not have designated team leads. Nor does it acknowledge that in a self-organizing team, a leader or “in group” will usually emerge — not necessarily a good thing. For self organization to work, one often needs a-lot of external support, and that requires external leadership to train and coach people on how to self organize well as well as provide outlets and oversight for when things go awry.

But the Manifesto says none of that: it merely says or strongly implies: No team lead, and no oversight; and nothing is said about when one has multiple teams, or how those should coordinate, or who forms those teams.

The Agile community rapidly and very early evolved a culture of mistrust of authority. This was for good reason. I myself had some terrible software project managers during my career. I also had some really good ones.

Unfortunately, dismissing the role of leadership does not solve the problem. Good leadership is essential. Throwing out leadership replaces bad leadership with no leadership, or insisting on no one having any authority when in fact authority is needed to create a team in the first place, or to spend an investor’s money, or to resolve a personnel issue.

Agile cannot work or be usable at scale without addressing leadership as a central issue, as well as the many dimensions of leadership, the forms of leadership that are most effective, and when authority is actually needed and when it is not.

Agile Can Work — Really Well

I believe in Agile.

Over time I have increasingly seen articles and blog posts critical of Agile. That is nothing new; but what has changed is that these posts and articles are now coming from people who have a lot of experience using Agile methods. Their posts and articles are insightful. They point to the problems that I described above. And when I see Agile mentioned in programmer forums, the mention is negative in what seems like more than half the time.

There is also the rise of DevOps. I am a DevOps consultant, and I teach an O’Reilly Online course, More Effective DevOps Testing, so I feel that I understand DevOps. My fear is that organizations will perceive a choice between Agile and DevOps, and they will choose DevOps.

The truth is they need both. DevOps has come to focus mostly on the technical side of things, while Agile pivoted very early to focus mostly on the human side. Both are important.

If only Agile’s problems were fixed.

And then COVID-19 struck. People saw that remote teams can work, and work well. Sure, some people prefer working remotely and others prefer being in an office; but the idea that being in person for “face-to-face” meetings is essential for collaboration was exposed as a falsehood. It occurred to me that COVID could become the final nail in the Agile coffin. I realized that if Agile were to be fixed, now was the time.

I needed a team of people. The main criteria were,

• They must not be heavily invested in the status quo: they felt free to challenge Agile.
• They have demonstrated independent thinking.

I posted a notice of the initiative in LinkedIn and examined each response. I also wanted the team to be global and diverse — not a group of “middle aged Western white IT guys” like the group that created the Agile Manifesto (and like me). The group that we formed has experience in Lean product development, business, leadership and org change, program management (Agile and traditional), professional coaching, human resources, system engineering, DevOps, and Agile.

Agile 2 is what we came up with. It is the result of deep and intense collaboration. We decided to publish the underlying ideas that we exchanged in the course of developing Agile 2’s principles and values, so that the thought process can be seen. We also decided to explain each principle at length, because we feel that one of the problems with the Agile Manifesto is that each statement is made without any supporting context or explanation, allowing it to be misinterpreted.

We do not feel that Agile 2 is the last word on the issue of agility. We plan to inspect and adapt Agile 2 in the future through occasional but not too frequent updates, but we also believe that Agile 2 — indeed no single set of ideas — can be comprehensive in its treatment of human collaboration or any aspect of human behavior. Works like ours are merely sets of ideas — nothing more. All sets of ideas should be considered, and thoughtful decisions made.

We hope that Agile 2 contributes positively and that it succeeds in making the case that Agile is still highly relevant and can work and work well.