Why Agile Isn’t Working: Bringing Common Sense to Agile Principles

Much of agile’s success is due to the fact that it “sells” so well by promising solutions to perennial IT concerns: projects that run over budget and time, lack of team effectiveness, lack of true collaboration, poor product quality and dissatisfied customers.

I’ve been involved in a number of agile projects from all perspectives, as a team member, leader architect and overall responsible manager. I’ve concluded that agile has not only failed like other fad methodologies before it but, in fact, is making things worse in IT. Yes, there are certain occasions when agile does work, particularly for proof of concept (POC) work involving already well-integrated teams, but I’m talking about 80 percent of projects here.

3 Ways the Agile Methodology Is Flawed

The agile methodology produces the opposite of the promised effects. There are many reasons for this, but I’ll focus on are the three most destructive agile principles, which I’ve renamed (shown in bold below).

Delivery Over Quality

The first, delivery over quality recasts the agile principle of “early and continuous delivery of valuable software.” Yes, early visibility is an important tool for customer collaboration. What is dangerous, though, is that focusing on continuous delivery has the effect of creating an unmanageable defect backlog while developers work to put something in front of the customer.

In most agile projects I’ve been involved with, the defect list has grown faster than it can be dealt with. On one medium-sized project, a huge defect backlog had developed over the previous 18 months and required a full-time role just to manage it. We routinely needed groups of iterations devoted solely to trying to clear the backlog. Not only did that come at the expense of new features, but it naturally introduced new defects—which were just thrown back onto the same endless pile of things to be fixed.

Obviously, large defect backlogs are nothing new in projects, but what is new is that agile promotes the practice of ignoring defects. This leads to unmanageable backlogs and, eventually, to ineffective development. I’ve watched developers, initially caught up in the excitement of collaboration euphoria, end up psychologically worn down by the perpetual defect list to the point of burnout. Agile projects are responsible for burning out many a good developer, which is far from the agile promise that projects “should be able to maintain a constant pace indefinitely.”

Development Over Planning

The second flaw, development over planning, hits the agile principle of “responding to change over following a plan.” In theory, developers code while collaborating with stakeholders to define, refine and change requirements as the projects goes along. The methodology, however, does not distinguish between big and small changes.

Every change has a cost, but agile does not account for this. The result? People often change really big things late in the game using the rationale that since it’s an agile project, it can handle it. The only way the project can handle this is by adding iterations. As that happens, defects that might have been easy to fix at one point get harder and harder to fix, since the code base keeps changing.

This principle encourages poor and irresponsible planning while hiding its effects. As iterations continue and the defect list grows, the customer becomes more frustrated—not only because of the lack of quality, but because delivery expectations aren’t being met either. This is much different from more traditional practices, where you have a project based on well-defined requirements and changes are managed through a Change Management process—which, while sometimes byzantine, will at least state time and monetary costs more clearly.

Collaboration Over Management

The third flawed agile principle is collaboration over management. Agile emphasizes self-organizing teams that “do the right thing” in some sort of group evolutionary manner. Nothing against empowering individuals—many traditionally managed projects don’t use people well—but agile isnt always consistent with responsible management.

You don’t need a methodology that takes authority away from a single individual. In too many agile projects the Scrum Master is little more than a hapless cowboy waving his arms in despair while his charges go in all different directions at once. You cannot replace accountable and responsible project management with some immature utopian myth of self-organization. That’s no science for success.

Changing Agile Thinking With 3 Common Sense Principles

One thing that’s seductive about agile is the name. We like the idea of being agile in our thinking. Agile as a methodology cannot deliver agile thinking, however, and inevitably ends up preventing it.

Think of “agile” as the ability to take the input of all the variable elements of the project—budget, time, design patterns, reusability, customer needs, corporate needs, precedents, standards, technology innovations and limitations—and come up with a pragmatic approach that solves the problem at hand in such a way that the product is delivered properly.

This sounds like a tall order, but really comes down to three common sense principles.

Prioritization

If there’s one key quality a good project manager needs, it’s prioritization: The ability to take the pressures of all project elements and determine which path to follow based on what’s most important to achieve.

Remember, the goal is to deliver a quality product on time and to budget; as a rule, there are always some elements that have to be sacrificed to fulfil the needs of the others. It’s the How to Develop Next-Generation Project Managers role of the project manager to define and maintain the project priorities so they can function as a decision framework for team members as they carry out their tasks.

Pragmatism

One of the hardest things for many developers is pragmatism. Rather than think practically, they inevitable fall into abstract approaches to problems. I was on one very expensive project based on a purist design in which everything was reusable and could be dynamically coupled together. It was a marvel of abstraction. The only problem was that it didn’t work. It had the worst performance I have ever seen in a Java system and was incapable of supporting the required load.

The core problem there was that the team attempted to solve real-world problems with an abstract approach. No matter how virtual technology gets, it’s still ultimately based on the physical world that we humans operate in.

Physical problems cannot be solved abstractly. Look at any factory machine and see how many oddly shaped parts it contains, each good for just one very vital physical function. Software is no different. Sometimes things are meant for one use only. That’s not a bad thing if it gets the job done and functions properly for many years.

Dynamism

Finally, dynamism means the ability to switch strategies when the current one isn’t working. Many times, despite our best planning, what seems like a good project, design or development strategy hits a brick wall. The most important thing in this situation is to know when to call it off.

If the brick wall is only a brick wall, a bit of brute force—i.e. one or two late nights—will get through it. But if it’s really a mountain of rock, it’s best not to risk the entire project but, instead, find another way around. It sounds like a simple concept, but I’ve watched projects go on for weeks, team members beating their collective heads against the mountain, only to conclude too late that another approach was needed.

Agile promises solutions it cannot deliver. It promotes sloppy requirements, hides the true cost of development and prevents effective management. Contrary to what we’re told to expect, this leads to long-running projects, dissatisfied customers and an overall IT ineffectiveness.

What we hope to find in the methodology, however, is achievable through agile thinking. With this thinking, we can in fact solve the problems of IT project management and learn how to deliver stable products on time and to budget. If the project team knows how to think and work effectively, then we don’t need to look to a methodology to save us from project failures. We can do it ourselves.

Lajos Moczar is a senior technology strategist and consultant, operating as The Consultant CTO. When not consulting, he writes on various IT topics. Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn.

Copyright Lajos Moczar, 2013. Reprinted by permission