The Agile Community Embraces an Unworkable Fantasy

I am an Agilist. The company that I co-founded in 1995, with almost 200 employees by the year 2000, adopted eXtreme Programming (XP) that year. My 2005 book High-Assurance Design interpreted secure and high reliability engineering practices in an Agile context, proposing a concrete answer to the question, “How can organizations build highly reliable and secure systems using Agile methods?” Since then I have helped more than ten large organizations to move to Agile and DevOps approaches.

But the Agile community is very off the rails. It teaches people an ideal model that does not work. This model is like a sacred calf: its elements are discussed by Agile coaches as if they are truisms. One challenges it at great peril: if one speaks against the model, one can be labeled an “Agile doubter”, or someone who is “not really Agile” — a mark of death in the Agile community. The Agile community has historically been very closed minded, and not able to debate Agile ideas in an open-minded and healthy way. My editor at Pearson described it as “insular”. It was DevOps that burst through the Agile doors and humbled the community into accepting more open debate.

(Image is from “My Book of Bible Stories”​)

Indeed, writing about this is always a personal risk for me: those who don’t agree will refer to me as “that anti-Agile” guy. I know because it has happened. They are wrong though: I was an early adopter of Agile, and am fully committed to the core ideas of Agile; and I have been immensely successful in implementing Agile — but not in the typical manner of adopting out-of-the-box standard Agile practices.

My approach is more of a DevOps approach: to always look at the uniqueness of each situation and consider it as a system: Consider what behavioral changes will achieve the business goal, whether it is shorter time to market and building the right things, or higher quality and building things that can be trusted to work. I always start from a business driver perspective and draw from Agile ideas, Lean ideas, behavioral therapy ideas, and many others, rather than trying to force-fit the sacred calf idealist model on every situation.

Let’s look at what the Agile sacred calf model is.

If one takes a training course in Agile “basics”, one will be taught the Agile Manifesto — a brilliant and timeless document — but then one will be taught the Scrum process — even though Kanban is arguably a much better fit for today’s DevOps methods. In addition, one will be told the following falsehoods:

1. Self-organizing autonomous teams are the best way to organize programmers.
2. Managers are an “anti-pattern”; ideally there should be no hierarchy, and no one should have explicit authority. All authority and all leaders should emerge through self-organization.
3. No individual should be accountable: only entire teams should be accountable.
4. Failure is the best way to learn.
5. One should always trust the team, even if you have never worked with the team before.
6. All development teams should be feature teams that work across the entire set of technology stacks of the product.
7. Anyone should be able to work on anything.
8. An open team room is the best arrangement, so that people can collaborate most easily and will benefit from “osmotic” communication.
9. More collaboration is always better, and verbal communication is always best.

The Agile Manifesto does not say any of these things. Let me repeat that:

The Agile Manifesto does not say any of the nine things in the list.

While the Manifesto mentions elements of some of these things, it never says “do only this” or “always do that”. Nearly every statement in the Agile Manifesto is written to imply a tradeoff, so that one can decide what approach is best for a situation.

The above nine axioms form a model that is taken as truth by most of the Agile community at large, despite lots of evidence that the model is wrong. People who really understand Agile don’t make the above nine absolutist claims, but those people are a small percent, and they know to not openly criticize the sacred calf model, unless they are speaking privately with someone else who truly understands Agile.

Books debunking the above model abound, although they do so in a careful and politically correct way, tiptoeing around the model and claiming instead that one needs to “augment” typical Agile practices. A recent one is Rethinking Agile: Why Agile Teams Have Nothing To Do With Business Agility, by Klaus Leopold. In that book, Leopold expertly connects the dots on the leadership gaps that tend to exist in organizations that create autonomous self-organizing teams and then expect that they will be Agile as a result.

Despite the attention-getting title, the book does not directly criticize the nine-axiom Agile model, but instead shows how the adoption of that model by organizations leaves huge gaps, and Leopold explains how to fill those gaps.

Agilists (including me) are quick to point out that one cannot simply “insert” Scrum or self-organizing teams: one must “change the organization’s culture”. But what does that mean? The very same people who stand by the above nine prescriptions seldom have a good answer for what a culture change means. They say platitudes like, “One must get managers to embrace failure”, and “The organization needs to learn to experiment”.

Fine, but that kind of advice is too vague, and abstract advice that ends in a platitude tells me that someone does not really know what they are talking about — that they learned something in a book but have not actually solved that problem.

I have helped more than ten organizations “go Agile” or “go DevOps”, and I have seen every permutation of this. I have also seen common patterns that are usually present. These are,

1. Organizations keep their existing market research and product definition methods, which have a long lead time, and which “hand off” product requirements to “IT” to build.
2. Organizations dismantle old style program-level or product-level leadership, but don’t replace it with an Agile equivalent.
3. Organizations try to “roll out” Agile, treating it like a process change, when in fact it is mostly a learning journey.
4. Organizations fail to put sufficient emphasis on the technical side of Agile, which today is “DevOps”.

Many organizations try to “roll out” Agile by adopting SAFe, or some variation of that. This actually helps somewhat because it fills some gaps pertaining to item B above, and possibly even impacts A — although that is less common. But the huge Product Increment planning exercise that SAFe prescribes is unnecessary, in my opinion.

The biggest leadership gaps that I usually see pertain to technical leadership. Organizations might adopt a “Lean portfolio” process (which SAFe recommends) and a product “release train” structure, but they fail to address how the many components of their product integrate. They dismantle their traditional approach of having a separate QA team focus on “system integration testing” but they don’t replace it with an Agile and DevOps equivalent, which ideally is frequent, cadence-based on-demand automated integration testing, backed by “shift-left” local integration testing.

They often put a development manager in charge of the product teams, but the manager has heard the message that teams should be “self-organizing and autonomous”, and so he or she fails to step in and provide the leadership that is needed. The result is that the teams tend to operate in a highly independent manner, and fail to adequately address integration across teams. The sacred calf model is then the direct cause of the ensuing dysfunction.

Agile methods include having regular retrospectives, and most organizations do this at a team level because Scrum prescribes it, but they fail to institute it at a product or program level. Agilsts will point out that this is a symptom of “not having an Agile culture”, because an Agile culture would surely think to have retrospectives at every level. But when an organization is trying to adopt Agile methods, they cannot magically “become an Agile culture”. They need guidance on what practices to install. Eventually those practices will foster more Agile thinking, but that change in thinking will take a long time.

And this is where the Agile community, in general, is confused: the claim is that organizations must change their culture to be Agile. Yet the reality is that to change an organization’s culture, one must first change behavior — not the other way around. Culture change follows behavior change, and if the behavior change is maintained for long enough, the change becomes self-sustaining and becomes the new culture. (In the behavioral therapy community, this change lifecycle is described by the “transtheoretical model”.)

In other words, it is unreasonable to expect organizations to come up with Agile approaches just by learning the Agile Manifesto: they need help in defining new processes and practices, that over time will become the new normal; and in the course of those behavioral changes, they need Agilists to interpret the changes for them, to help them to see the behavioral changes through an Agile lens — that way, their thinking begins to change. But the behavioral changes must come first. People go from concrete to abstract, not abstract to concrete.

Thus, it is the Agile community’s fault for popularizing the overly simple model of autonomous teams and not helping organizations to anticipate all of the new processes and practices that they will need so that those teams will work together. Telling them that they should have a collection of self organizing teams and that they should “start to think Agile” is not enough. The teams need explicit methods for organizing their collective behavior: they will not adequately self organize around a product or value stream, and most development managers will not know how to stimulate the required level of cross-team collaboration: they will stand back and watch, and expect the teams to magically start collaborating — because that is how they have been told Agile works — but it doesn’t happen.

A failure to have retrospectives at a product or program level blocks the improvements that Agile might bring: product feature cycle time cannot be substantially reduced, nor can the rate of escaped product defects, unless one focuses on end-to-end improvement, spanning the many teams of a product or value stream.

Improving individual teams will not unlock the benefits of Agile. And Agile coaches, who tend to be non-technical, shy away from discussing technical issues during retrospectives, so don’t rely on them to help you improve your end-to-end technical flow. They will tell you that you need to identify dependencies and reduce your “WIP”, but they won’t have any ideas about how to manage the dependencies so that integration happens frequently and tests at all levels of integration are sufficient. (See this article on dependency management, and this article series on defining a testing strategy.) Nor do most teams know the CI/CD practices that are needed for rapid integration — they don’t know what they don’t know: thus, the adage “trust the team” fails to account for the DevOps mentoring that teams need.

The fact is, someone needs to be accountable for the product level technical flow. Otherwise, there is no product level leadership around things like end-to-end testing. The leadership style should be servant leadership, so that it is collaborative and enabling — not autocratic and fear based. But someone needs to have their eye on that ball all the time: the integration level issues that come up are complex and numerous, and they will totally block substantial improvement if you don’t address them.

A person who has accountability needs to have actual authority: accountability without authority is a terrible situation to be in. But a servant leader will not exercise their authority most of the time: they will save it for rare occasions when they need to make a decision about something that cannot be resolved collaboratively, such as the need to re-organize some teams. The rest of the time, they are watching for things that are falling through the crack, and making sure that issues are being discussed and resolved — mainly by asking questions and making suggestions, but sometimes — hopefully rarely — by insisting that something be addressed, and asking the team to trust in the other direction.

So much for the falsehood that one should have autonomous self-organizing teams! What teams need is servant leadership, at all levels: the team level, the product level, and the program or value stream level.

Then there is the claim by Agilists that failure is the best way to learn. No one likes to fail. A market facing failure is never a good thing. The idea of “fail early, fail often” pertains to controlled experiments, which might be market-facing if they need to be. SpaceX uses the “fail early, fail often” technique: they build lots of rocket engines and run them to failure, to learn how they fail. They even do that for whole rockets. But the last thing they want is a failure during a mission!

Instead of the very misleading “fail early, fail often”, the maxim should be, “try things — experiment a-lot and expect many experiments to fail — that way you will learn, and you won’t fail when it counts”.

The other items in the list of nine Agile falsehoods can be summarized as extremes that contain good ideas but that usually don’t work in their extreme form — the form that is usually explained during an “Agile Basics” course and which is advocated by a large percent of Agile coaches.

Self-organization? Sure — to a degree, but better to have a true servant leader who can help the team to organize and to have high quality dialectic discussions. Anyone can work on anything? Maybe — but sometimes business-critical core microservices need to be maintained by a single team in order to ensure their design cohesion. Open team room? No one actually seems to like that, but maybe the “coffeehouse” format is a good compromise — thus we come full circle to the old “caves and commons” idea. In-person communication is always best? For simple things yes, but for very complex issues, some people communicate better by writing their thoughts down first: that is a matter of personality.

And so on.

Extremes don’t work, except in extreme circumstances. Most organizations are not in extreme circumstances. The Agile Manifesto was about balance and judgment — not about extremes. Those who teach the nine falsehoods do not really understand Agile: they are cargo-cult Agilists. All those nine ideas are extreme variants of good ideas, but the extremes don’t make any sense most of the time.

Each of the nine maxims should be taught as an idea to be considered, and nothing more, with the caution that the right approach depends on the situation.

It is time to dismantle the false ideal model of extremes that is being taught in Agile Basics courses and that does not actually work, and bring judgment and reality back into Agile.