The Agile process of software development is often perverted by sick politics

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com

For those of you who don’t want to read this whole essay, here is the summary:

The word “agile” has a plain meaning in standard English, and that meaning was considered something positive by software developers, so much so that the most successful new development process of the last 30 years calls itself “Agile”. However, at many of the companies that I have worked, actual agility is suppressed because of various political factors. Fear wins out over trust. Instead of actual agility, what is venerated is the use of specific tools, such as Basecamp, or PivotalTracker, or Jira, or Sprintly, or Trello, or Asana, or Smartsheet or Agilecraft. Artificial point systems are introduced to track what work is done, and middle-managers who are trying to protect their jobs boast to upper management about the number of points accomplished in a week. Thus using “agile” tools leads back to the kind of corporate pathologies that the Agile movement hoped to escape from.

That is the summary. If you are busy, you can leave now, knowing the gist of what I’m going to say. The rest of this essay adds nuances and anecdotes, and also adds some qualifiers that take into account specific circumstances that sometimes undercut the thesis. I’ll talk about some of the terrible implementations of Agile that I’ve seen, and then I’ll highlight one of the cases where I saw it done really well: when Mark Herschberg was CTO at ShermansTravel.com.

——————————————————————-

Historical Background

In industrial contexts, the word “agile” has evolved over the last 30 years. In the early 1990s, researchers such as Ted Goranson argued that “agility” was going to be as important in the 90s as “quality” had been in the 80s. There was a sense that the Japanese auto manufacturers had hurt the USA auto manufacturers by putting in place a series of processes that ensured the quality of Japanese cars. The USA corporations were slow to realize the importance of structured processes for ensuring quality, and they were slow to catch up to the Japanese. USA researchers, many funded by DARPA, did not want the USA to fall behind where agility was concerned.

Goranson defined “agility” as the ability of large corporations to use small corporations as part of their supply chains. This was especially important in the USA, where the smaller companies tended to be more innovative than the larger companies.

In the world of large corporate software projects, there was a desire to get away from the kinds of disasters that Robert Glass wrote about in his book Software Runaways: Monumental Software Disasters. Among the problems of that era, there was a feeling that there was too much time between planning a feature and delivering it — too much negotiation, too many contracts, too much documentation, all of it based on the distrust that comes from expensive projects being negotiated while those who want the software are separated from those who actually write the software.

The worst of these project failures, the most expensive and the most ambitious, was the attempt made by the Federal Aviation Authority to modernize the computer system it uses to keep track of what planes are in the air. The effort began in 1981 and ended in complete failure in 1994. The government hired IBM to do the actual work, and over the course of 14 years, IBM burned through $3.7 billion dollars. Nothing was accomplished. The project was finally shut down by Congress. Nothing came out of the project, not a single piece of software, nor even a line of code, was ever used for anything. Sadly, this kind of thing still happens, as in Oracle charges$240 million for a website, and then fails to build the website.

It’s obvious why the more thoughtful (and ethical) software developers would want to get away from these kinds of disasters.

In 2001, 17 software developers, who had been working on issues relating to agility, issued this manifesto:

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

There are 17 famous names attached to this manifesto:

Kent Beck

Mike Beedle

Arie van Bennekum

Alistair Cockburn

Ward Cunningham

Martin Fowler

James Grenning

Jim Highsmith

Andrew Hunt

Ron Jeffries

Jon Kern

Brian Marick

Robert C. Martin

Steve Mellor

Ken Schwaber

Jeff Sutherland

Dave Thomas

My guess is that if you are a professional computer programmer, then you probably own at least one book written by one of these individuals. I personally own books by 3 of these individuals. So, certainly, this group has been influential. And yet, are corporations truly internalizing the message that the Agile Manifesto tried to communicate?

These individuals are noteworthy for their idealism and their ethics, however, they are also successful enough that their career could survive a setback. That makes it easier for them to stand by their beliefs.

My own feeling is that we software developers need to be 100% ready to be fired everyday — that is the only way we can have the courage to tell the truth and to take full responsibility for everything we’ve done, both the good and the bad. But I don’t have children and I don’t have a mortgage, so it is easy for me to brave — I realize for many other software developers, they need to be much more careful, if only to protect their families. And this need to be careful is one of the factors that leads to the sickness of corporate politics. But of course, the need to be careful doesn’t start with the software developers — rather, that comes down from the top, from leadership that often remains aloof from the tech team until a project fails, and then the only concern of the top leadership is finding someone to blame.

Common deviations away from the Agile Manifesto Principles

For those who only want a summary, this whole section can be reduced to one word: politics.

Although the Agile Manifesto Principles offer us a set of beautiful ideals to work by, the harsh reality of corporate life often undercuts them. Very briefly, I will review the Principles of the Agile Manifesto, and summarize the most common violations I have seen.

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

At most of the places that I’ve worked, the highest priority for both the managers and the software developers was keeping their job. A cycle of “We failed, someone must be fired, so fire that person over there, they did it, not me” sadly repeats every few months in these organizations. If you like, you can try to pretend that we live in an ideal world where we only need to do a good job and surely the top management will deal with us justly. In the real world, most mid-level managers, and most software developers, know that the punishments that come after failure can be political, personal, and arbitrary.

We welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

Except whenever a software company makes a mistake and fails to deliver — then the first and most common thing the management of the tech team will blame is “changing requirements” — see the Oracle and BBC cases mentioned above.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

This is a 100% brilliant idea and I think everyone agrees with it. And yet, it is often an illusion — a team can agree that it will deliver the X Feature in 2 weeks, and then they do it — but the feature by itself doesn’t do anything, and the rest of the software still doesn’t work, and the system overall may not work for another year. Some projects have long lead times — there is no way around this fact.

Business people and developers must work together daily throughout the project.

But is it the project managers and the business leaders who should meet daily, or should the business people meet with the actual software developers? This is the issue where I’ve seen the least consensus, with good arguments made for each strategy. At some companies, it seems important to protect the computer programmers from the business people, whereas at others, constant conversation is useful.

The ugliest scenario that I’ve seen, which is surprisingly common, is when the CEO or COO or President Of Product goes to a computer programmer and says “This project is our #1 priority” and then 2 weeks later goes to them and says “This other thing is our #1 priority” and then 2 weeks later goes to them and says “This totally different thing over here is our #1 priority” — so 6 weeks go by and nothing is accomplished.

Earlier this year I consulted at a company that had spent $3 million dollars, and 4 years, trying to build an internal CMS (first in PHP, then in Ruby, and first with an external company, then an internal team, and then an external company again) and they’d never come up with anything that actually helped the staff — the COO was a woman who changed her mind often enough that the various groups that worked on the project were never able to accomplish anything truly useful. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The is a brilliant idea but it requires a lot of trust, and trust is a very scarce resource. Many managers feel that their #1 responsibility is to minimize risk for the firm, and this attitude typically starts with the CEO and radiates down through the organization. And the short-term tactics that most managers use to minimize risks tend to undercut a culture of trust. At the most recent place that I worked the Director Of Engineering wanted me to check in with him several times a day, relaying to him fine-grained details of what I was doing: changing an NPM package, changing the structure of some JSON messages, switching our use of Redis from pub/sub to FIFO, changing the worker bots to be async instead of sync. My own feeling was that there was no point hiring someone as experienced as myself, if the management was going to question each decision that I made. Perhaps you disagree with me. Perhaps you feel that management has a right to immediately veto any decision that a developer makes — you have a right to your opinion, but do consider how much you yourself are then going against the Agile spirit of “trust them to get the job done”. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. We all know the history that lead up to this Principle. A big corporation such as IBM would sign a contract with another big company such as United Airlines, promising to deliver a system that could keep track of flight times. Months or years would be spent negotiating the contract, and lawyers would spend as much time reading over the documentation as the software developers did. And given the awful results of this cumbersome process, it is easy to say that face to face conversations are better. But I’ve mostly seen this Principle turned into the morning ritual of having a “stand-up” meeting. When this is done correctly, it can be useful. “Done correctly” means the team is less than 12 people, they meet in the same physical space, the entire meeting takes less than 15 minutes, and useful information is exchanged. But I have, many times, seen teams as large 40 people involved, I have seen this done over telephones and Skype, and I have seen the meeting drag on for 45 minutes. Please be clear about what I am saying: there is nothing wrong with telephones or Skype, and there is nothing wrong with remote work, but it is no longer a “stand up” meeting. If the team is big or remote then it is best to use other means of communication. Just because you call a meeting a “stand up” doesn’t mean it is a “stand up”. Merely having a meeting each morning in no way guarantees that you are living up to the spirit of this Principle. Working software is the primary measure of progress. This is a beautiful ideal but who takes the blame when there is no progress? Some of you may have read my earlier essay What happens when the Board Of Directors begins to panic about a startup where I worked last year. In that case, we worked on one app from May to July, and then the Board Of Directors decided on a Big Pivot, so we threw away our previous work and started over. But the new app went slowly. The Member of the Board who had most strongly argued for the Big Pivot then blamed us developers for the lack of progress, even though we had had no say over the Big Pivot. This might seem unfair to us software developers, but what if the Board was right and the original app was worthless? In that case the mere fact that we could get it working was also worthless. If the leadership has correctly decided that a radical change of direction is wise, then working software can no longer be the primary measure of progress. If an idea is worthless, then working software that enables that idea is also worthless. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. To my mind, this is the most controversial part of the Agile manifesto. This goes against the plain English meaning of the word “agile”. A great athlete can train for agility or train for endurance, but those are two different things. A small startup with 6 months of funding needs to get a working product out the door in 3 months, so it can raise more money before running out in 6 months. In this case “a constant pace indefinitely” is pointless — the team might make slow, steady progress, but the company will go bankrupt. What does matter, in such circumstances, is having great programmers who can work 90 hours a week for 6 months. I’ll point out that almost all of the signatories of the Agile Manifesto worked in very large corporations. This particular Principle seems adapted for life in a big corporation, rather than life in a small startup. Continuous attention to technical excellence and good design enhances agility. This is absolutely true, though this principle is often sabotaged by the same lack of trust that I spoke of above. In particular, there is a tension between the short-term need to meet deadlines and the long-term need to pay down technical debt. There needs to be trust between the team and the high level management, so that the team can say “This next week we are simply paying down technical debt, we are not working on any new features.” If the top leadership is also under pressure, then often this trust simply won’t exist. At the firm where I worked earlier this year the Board Of Directors fired the CEO, and there is a rumor that the Board will soon fire the COO. The woman who is COO wasted$3 million dollars, and 4 years, on a failed attempt to build an internal CMS. With her own career in crisis, she is putting the tech team under a lot of pressure to deliver on short term goals. In such a crisis, the tech team is not allowed to take into consideration those things that will only pay off in the long-term.

Simplicity–the art of maximizing the amount of work not done–is essential.

I like this Principle very much, but it is often thought of as narrowly applying to writing software. The biggest payoff arises when the top leadership can avoid wasted moves. Again, consider the Big Pivot that we did at the startup where I worked last year. We threw away much of the work that we had done in May, June and July. Perhaps the leadership was wise to make the Big Pivot — this can never be known for sure. What is inarguable is that the project would have been better off if the concerns the leadership had, which turned into panic at the end of July, could have been considered more deeply, earlier. Had we done the Big Pivot in June or May, then there would have been much less wasted effort.

The best architectures, requirements, and designs emerge from self-organizing teams.

This is absolutely true, but again, this requires trust, and trust is a scarce resource. How can the leadership trust a team, when the failure of the team might mean the end of the career for the manager? If you are in a leadership position, and you are getting paid \$350,000 a year, would you feel comfortable trusting your entire career to some group of software developers who want to self-organize? You would have to be either very brave, or you would have to have the trust of the leadership above you (and if you are the CEO, you would have to know you have the trust of the Board). This is rare.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

This is sometimes a good idea, though again it requires the trust of the leadership. Also it is a form of Democratic Centralism that tends to force out those with divergent opinions. I am not saying this is necessarily a bad thing, as team cohesion is sometimes a crucial element of success, and some uniformity in code writing is desirable. But this can also be a bad thing. “Team cohesion” tends to also be “team homogeneity”. On the teams where I’ve seen the most self-criticism and self-renewal, I’ve also seen a tendency toward what you might call the “dictatorship of the majority”. Once 60% of the team has decided that habit X is good and everyone should do it, those who hate habit X tend to either quit the team, or sometimes they are forced out. And so you end up with a team where 100% of the team feels that habit X is a good thing. Hopefully habit X really is a positive thing, otherwise the whole team is now in trouble, and with no dissenting voices to say why.

The backlash against Agile is driven by the corruption of Agile

A vast industry of advisors has grown up, all of them promising to teach you how to do Agile the right way. Apparently Agile is simple enough that it can be summarized in 4 short sentences in the Manifesto, yet it’s complicated enough that you really need to hire the right people, with particular credentials, or you will botch the whole thing.

Search Google for “scrum master certified” and you get a sense of what a huge industry Agile has become:

To be clear, I think the right methods can offer real benefits, and I think training can help spread “best practice”. And if ever you find one of those rare project managers who knows how to unleash the full potential of your team, grapple them unto thy soul with hoops of steel. True leadership is rare.

Thankfully, there has been a backlash against some of the bureaucracy that has grown up around this supposedly lightweight methodology.

Dave Thomas was one of the original 17 signers of the Agile Manifesto. But in 2014 he wrote “It’s time to kill Agile“:

…The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products.

…Once the Manifesto became popular, the word agile became a magnet for anyone with points to espouse, hours to bill, or products to sell. It became a marketing term, coopted to improve sales in the same way that words such as eco and natural are. A word that is abused in this way becomes useless — it stops having meaning as it transitions into a brand.

This hurts everyone, but I’m particularly sensitive to the damage it does to developers. It isn’t easy writing code, and developers naturally are on the lookout for things that will help them deliver value more effectively. I still firmly believe that sticking to the values and practices of the manifesto will help them in this endeavor.

I’ve been in a three day long sprint planning meeting for a six week “sprint.” I’ve seen user story cards with nothing but “Make dashboard better” written on them by Certified Scrum Masters. It’s pathetic. Even more pathetic is that I accepted their job offer, but I didn’t know any better. They told me they were Agile, after all.

…Every time I talk with a so-called Agile company about how they work I get a laundry list of SaaS Web applications. Trello, Basecamp, JIRA, Pivotal — none of these tools existed when the Agile Manifesto [was written]

These tools aren’t the solution. Agile doesn’t need a solution, it is the solution, or rather a guiding light to the solution for your particular team. These tools are masquerading around pretending to be the solution to people. People that haven’t learned what Agile is all about. These people just hit the Wikipedia page for Agile after seeing it mentioned in a job post because their degree in Communications couldn’t get them any other gig.

Iteration planning meetings are seriously expensive. Group discussion around design, group estimation, group acceptance, all highly inefficient. These meetings are helpful when the team has problems with commitments or trust, but otherwise, these meetings require a lot of time and are very, very expensive.

I remember just getting bored to tears listening to discussions around stories I wasn’t developing on. Eventually, we had to ask the question, “are these meetings providing the value we are paying for”. Yes, there was some value, but not near enough to warrant so much expense. I’m sitting in a meeting, thinking, “can’t I just start actually developing software”?

…This might be a bit controversial, but the big difference between Scrum and things like Lean Software Development for me were the difference of focusing on process versus delivery. Lean focuses on delivering value, and having a set of approaches on discovering your unique way of doing so, where Scrum has a framework for a process and tells you if you don’t follow this approach, you’re doing it wrong.

Then came that requirement. The one where you try to replace an expert user’s years of experience and intuition with software. What started out as a vague and wooly requirement, soon became a monster as we started to dig into it.

…The business was frustrated by the lack of progress and the vaguely hoped for project delivery dates began to slip. It was all a bit of a mess.

The boss looked at this and decided that we were loose cannons and the ship needed tightening up. He hired a project manager with an excellent CV and a reputation for getting wayward software projects under control. He introduced us to ‘Jira’, a word that strikes fear into the soul of a developer. Now, rather than taking a high level requirement and simply delivering it at some point in the future, we would break the feature into finely grained tasks, estimate each of the tasks, then break the tasks into finer grained tasks if the estimate was more than a day’s work. Every two weeks we would have a day long planning meeting where these tasks were defined. We then spent the next 8 days working on the tasks and updating Jira with how long each one took. Our project manager would be displeased when tasks took longer than the estimate and would immediately assign one of the other team members to work with the original developer to hurry it along. We soon learned to add plenty of contingency to our estimates.

Then a strange thing started to happen. Everything slowed.

Of course we had no way to prove it because there was no data from ‘pre-PM’ to compare to ‘post-PM’, but there was a noticeable downward notch in the speed at which features were delivered.

And I have personally felt the de-motivation that he talks about here:

We became somewhat de-motivated. After losing an argument about how things should be done more than a few times, you start to have a pretty clear choice: knuckle down, don’t argue and get paid, or leave.

And he points out that the new “Agile process” actually made the organization more sick:

[The programmers hired by the new PM] actually seemed to like the finely grained management. As one explained to me, “you take the next item off the list, do the work, check it in, and you don’t have to worry about it.” It relieved them of the responsibility to make difficult decisions, or take a strategic view.

Obviously, a healthy organization is one where the developers take maximum responsibility for all that they do. This so-called “Agile process” was entirely opposed to the spirit of the Agile Manifesto, and entirely opposed to the kind of culture that a sane organization should want.

Our approach, which we call Non-Blocking Development, isn’t so much a methodology as it is a pragmatic approach that optimizes for shipping working software over everything else.

…A very anti-agile pattern in Non-Blocking Development is that the business should move entirely to asynchronous communication. While we agree that face-to-face communication is the most efficient and effective method of communicating, we do not believe it’s pragmatic in today’s working world. Globalization is happening at a frenetic pace. Businesses have teams dispersed across the globe. Strictly adhering to this principle would, in fact, have the opposite effect on businesses.

And I love this quote:

“You can’t ship process.” – Ethan Kaplan, VP of Product at Live Nation

Don’t get me wrong, I’m not suggesting that Agile is responsible for consulting companies with hours to bill and services to sell. There were over-priced consultants before Agile, and there will be over-priced consultants after Agile. But such consultants, to the extent that they act as parasites, extracting money without adding value, are facillitated by whatever weaknesses exist in the companies that we work for.

Over-priced consultants seek customers who are suffering from some internal conflict. And I believe, the healthiest way for us software developers to deal with such situations is to talk with complete honesty about the complicated politics that often exist inside of the companies that we work for. When there is a gap of misunderstanding separating the tech team from the top management, bringing in an expensive consultant is a common mistake. What would work better is complete candor. After World War I, John Maynard Keynes was full of worry about the political situation in Europe, and he wrote to a friend, “I personally despair of results from anything except violent and ruthless truth-telling — which will work in the end, even if slowly.” That should be the motto of every ethical software developer who seriously wants to communicate reality to the top leadership.

The right way to do Agile

The best implementation of Agile that I’ve seen was when I was at ShermansTravel.com and Mark Herschberg was CTO. The system went like this:

Herschberg reckoned that each developer could get 5 hours of real work done each day (the rest of the day got eaten up by email, conversations, research, and other incidentals). We had a crew of 5 programmers, so we had 125 hours a week of real programming. The sprints were 2 weeks long, so we had 250 hours in a sprint.

Each department could submit requests, and as soon as the request came in, one of us developers would estimate the time it would take. So, for instance, the Editorial team might complain about the images being mis-aligned on the blog, and a developer would estimate that it would take 1 hour to fix. And the Advertising team might tell us we have a new client who wanted us to add some massive amount of Javascript to our site, to load a particular ad, and a developer would estimate the work at 5 hours. And the Finance department might complain that our measured click-through rate was in major disagreement with what neutral 3rd parties were reporting, so a developer would estimate that it would take 12 hours to crawl through our reports and find the error.

There would be a great backlog of requests. Advertising might easily have 150 hours of requests pending, and Editorial might have 220 hours of requests pending.

Herschberg gave each department a budget of hours. Perhaps Editorial got 80 and Advertising got 60 and Finance got 70 and Sales got 40. The departments could negotiate among themselves, and trade hours back and forth, just so long as it added up to 250 hours for a 2 week sprint.

We didn’t waste much time on estimation. We didn’t waste too much time on process. This was a fairly lean implementation of Agile.

The crucial thing was that we used Agile as a tool for honesty. We had explicitly political goals: to manage the expectations of the other departments. We weren’t especially focused on measuring our own progress; the biggest benefits was that each department was forced to see that the other departments all had compelling demands, and therefore no department deserved to get the exclusive attention of the tech team. We thus avoided that terrible political situation, which I’ve seen elsewhere, where Presidents of various departments sneak over to the tech team and secretly ask “Please make my department your #1 priority”.

In other words, we used Agile as a tool to promote an honest understanding of what was going on with tech. Each department saw our finite limits, and each department saw the demands of the other departments. The transparency was beneficial to the whole organization.

A final word of caution

Having said all that, I’ll add a caveat: when I got to ShermansTravel, we were maintaining a codebase that was 4 years old, and we were only making incremental changes to it. Agile is at its best in such situations.

Later, we decided that we needed to completely re-write the code for ShermansTravel. We started over in a new language. Instead of one big monolithic code base, we took a microservices approach, and we created 6 small apps that worked together in harmony.

The complete re-write took several months, and for the re-write we were less rigid about using the Agile method, partly because the work was new and original and therefore more difficult to estimate.

I’ll quote Mike Hadlow one more time:

The problem with this approach is that it fundamentally misunderstands the nature of software development. That it is a creative and experimental process. Software development is a complex system of multiple poorly understood feedback loops and interactions. It is an organic process of trial and error, false starts, experiments and monumental cock-ups. Numerous studies have shown that effective creative work is best done by motivated autonomous experts. As developers we need to be free to try things out, see how they evolve, back away from bad decisions, maybe try several different things before we find one that works.

Hadlow goes too far in suggesting that all software development is creative. If you are only making incremental changes to an existing code base, the work tends to be dull and straightforward, and also easy to estimate. And Agile can be very useful for companies that are in that situation, maintaining an old code base.

However, the more innovative the project, the less likely that anyone can form accurate estimates, and therefore the less likely that Agile will be useful. At the startup where I worked last year we built a Conversation Finite State Machine to track the accumulated meaning of sentences parsed with our Natural Language Processing engine. We were doing boldly original work that went beyond anything that Apple or Google or Amazon had yet come out with. Trying to create estimates for such a project is utter fiction. Since the Board Of Directors was willing to mandate what amounted to a pure research project, the Board should have been willing to accept all the uncertainty of a pure research project. And when one of the Board members insisted that we stick to his estimates and complete our 2 week sprints, a great deal of tension developed on that project. Because in such wholly unique situations, what is needed is actual agility, in the plain English sense of the word. Facing a truly novel challenge, a team would be wise to seek agility, not “Agility”.

Source