The Coral Reef Pattern of incremental improvement

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

The Acme Legal Firm needed a small office, and there was little available for rent in their city, so they decided they would build their own place. They hired a builder and asked her what they should do.

“Well, I think we should build this office with wood. I’m a big believer in wood. It is beautiful and fairly strong, and it is easy to hire workers who know how to work with wood. The failure modes of wood are well known.”

“What about the risk of fire?” asked the CEO of Acme Legal Firm.

The builder laughed. “It’s funny, every time I suggest wood as a building material, clients ask me about the risk of fire. But I’ll have you know, modern wood is pressure treated and highly resistant to fire. You won’t have to worry about fire.”

“Okay!” said the CEO. “Let’s do this with wood!”

The office was built and the workers enjoyed the space, but the company grew and eventually the office was too small. The CEO decided they would need more space. Instead of hiring the original builder, or seeking a recommendation from them, the CEO picked a name randomly from an Internet search for “builder” and asked for a consultation.

A gruff, older man came to the office and introduced himself as a builder.

“We need more space,” said the CEO.

“You need to tear down this building and build a new one,” said the builder.

“That isn’t possible,” said the CEO. “I was hoping you could build an addition.”

“But your current office is made of wood. Aren’t you afraid of fire?”

“I think modern wooden structures are safer than they were in the past? At least, that is what I was told.”

“Every wooden structure that I have ever seen eventually burned down!” shouted the builder dramatically.

“I had no idea!” said the CEO in surprise.

“But even if you are not afraid of fire, what about rot from water? The roof will eventually leak, and the wood will rot. Your workers will get sick with mold, and you will go bankrupt because of all the lawsuits.”

“That does sound bad,” admitted the CEO, “We really don’t have the money for a complete rebuild, but I think the addition should use a new material. What do you recommend?”

“Steel,” said the builder. “It resists fire and water, it is incredibly strong, you can expand endlessly, if you ever grow huge and need a 90 story office, steel will get you there.”

“I can’t imagine that we will ever grow so large that we need a 90 story building.”

“But the important thing is, if you ever do need to scale up, steel gives you everything you need to grow.”

“Isn’t it expensive to hire workers who know how to work with steel?”

“Yes, but think of what you are getting! Not some cheap wooden structure knocked together by barely literate hammer swingers who think every problem can be solved with a few bangs, but highly trained craftsmen who are producing a structure that might last a thousand years!”

“That is a good argument!” agreed the CEO. “Okay, we will use steel for the expansion.”

The expansion was built and new hires moved into it and enjoyed working there.

After a few more years, the company had grown considerably, and it needed more office space. Rather than reaching out to either of the previous builders, or even asking them for a recommendation, the CEO again randomly picked a name out of an Internet search for “builder”.

The new builder arrived at the office the next morning. He was young and had very small glasses and a goatee, and also he was sunburned as he’d just come back from a month in the woods at a primitive Earth skills workshop.

“We need more space,” said the CEO.

“You need to tear down this building and build a new one,” said the builder.

“That isn’t possible,” said the CEO. “I was hoping you could build an addition.”

“Aren’t you concerned about what an ugly agglomeration your office is? Who mixes wood and steel as you have done?”

“I am sorry you feel that way, but we don’t have the money to rebuild everything right now. Do you have a recommendation for an addition?”

“Stone,” said the builder. “There is no need for fashionable building materials such as steel. Stone is a classic building material. Our earliest human ancestors built homes out of stone, so it answers a primal need and puts us in touch with our roots.”

“But isn’t a stone building cold and drafty?” asked the CEO.

The builder laughed. “It’s funny, every time I suggest stone as a building material, clients ask me if it’s cold and drafty. But seriously, what do you think grass is for?”

“I guess I never really thought about what grass was for,” admitted the CEO sheepishly.

“Do you think the Neanderthals stood around during the Ice Age, for half a million years, thinking ‘I wish our home wasn’t so cold and drafty.’?”

“That would be a long time to be cold,” admitted the CEO.

“Obviously they used grass. It’s the perfect insulator, and it is very cheap. Think about how much money you can save on your heating bill during the winter, if we use enough grass.”

“I do love saving money,” agreed the CEO.

“An addition built with stone will give your workers a warm place to hide during the winters, as well as answering their primal human need for connection to the Earth.”

“That is a good argument!” agreed the CEO. “Okay, we will use stone for the expansion.”

And so the expansion was built with stone. And new hires worked there, and found their primal need for connection with the Earth was somewhat answered by the stone, though many of them suffered terrible allergies from the large amounts of grass used as insulation.

———————————————
———————————————
———————————————

This may surprise you, but the above story did not actually happen. In fact, it is a parody. This satire is aimed at the insane way that the tech industry has normalized the practice of hiring successive tech leads who have radically different preferences of architecture and language.

I’m writing this in response The Lava Layer (or Lava Flow) anti-pattern. It is a very good essay and everyone should read it. I realize the problem described there has some truth to it. However, I disagree about who should get the blame, and what the cure is. So I have two criticisms.

In terms of blame, that essay seems to suggest that the problem is with individual team leaders who allow their personal preferences to get in the way of what is best for the company. But I would put the blame at a higher level. Who thought it was a good idea to bring in a succession of team leaders, without first discovering that each of them had utterly opposing ideas about how to architect a system? That essay describes a situation where the top leadership has abdicated its role of vetting team leaders to see if they have a philosophy that meshes with the company’s real needs, both for modernization but also for maintaining the legacy systems.

I agree this does happen, much more often than it should. If I was going to change anything, I would change this terrible habit of the management. Business leaders need to be educated about the fact that a certain consistency of vision, regarding technology, is a benefit to the company.

However, I am not arguing for stagnancy. In every organization, there is a constant need to evolve to meet changing circumstances. One of the primary jobs of leadership is finding the right balance between stability and flexibility.

This is my second criticism of that essay: it does not suggest the correct way for an organization to evolve. And the pattern it criticizes is similar to the most successful pattern for change that I’ve seen.

The great French historian Fernand Braudel argued that civilizations build up like coral reefs, with each layer being built on top of the previous layer, and few disasters being so strong that they could actually sweep away all of the underlying layers. A terrible plague or a disastrous war might damage the top layer, but the lower levels are there and eventually the civilization rebounds, altered and yet still recognizable.

Something like this resembles most of the successful efforts I’ve seen to modernize the tech stack at large companies. I call this the Coral Reef Pattern. It is a pattern of incremental improvement.

I once worked with a massive monolithic app which had been written in PHP, using the Symfony framework. It suffered terrible performance issues. The pages were taking 10 to 20 seconds to render on the server, and much longer to serve to the client. I wrote about this in An architecture of small apps.

We began to gradually pull the system apart, keeping the Symfony monolith for the HTML template rendering, but pulling out the real performance blocks and re-writing them in more performant languages. For instance, routing in Symfony is known to be terribly slow, once you have a large number of routes. So we moved most of the routing to Nginx. Also, the code had been making calls directly to the database, and the ORM was slow, so we wrote an API using the Scala language, and we slowly changed the PHP code so it made calls to our API, rather than making calls to the database. The old PHP code was relying on Memcache to cache some of the database calls, but the logic was inconsistent and there were endless problems with stale data. Once the database calls were moved to a smaller Scala app, we built a simpler caching system for the Scala app, and we gained both speed and also simplicity.

An architecture of small helper apps allowed the old PHP monolith to survive. Rather than doing a complete re-write, we kept the old system working by strategically fixing the most urgent issues with small dedicated apps that were specialized to the one task they were given. Like a coral reef, the new layer did not replace the old layer, but added to it, in ways that made the whole system more performant.

One could argue that we made the system much more complex by introducing a new language. Wouldn’t it be simpler if everything was in PHP? Have we not injured the firm by splitting the work between two languages that start with radically different assumptions about computing?

Again, one of the primary jobs of leadership is finding the right balance between stability and flexibility. We should strive to find the simplest path forward that meets all of the needs of the business, but we should not sacrifice the needs of the business by artificially constricting the range of options that will be considered for technology solutions. PHP has the advantage that a lot of frontenders understand it, so it can be useful as a template language (I’m not a fan of PHP, but that is a common argument that I hear for it, and that is one of the few reasonable arguments for it). Scala has several advantages in terms of speed, strong types that can enforce contracts at API boundaries, and access to memory, so one can cache certain items without having to rely on a separate technology such as Memcache (for the sake of brevity, I’m going to skip the debate about when you would want to do this).

A number of big firms have followed the above pattern. Twitter started off with a monolith built in Ruby On Rails, and they were soon confronted with more traffic than could be handled, even with some very talented people trying to make the Rails code work. They then moved to an architecture where Rails handled the frontend, but everything on the backend was re-written to use higher level languages such as Scala. Likewise, Facebook started off as a PHP app, then moved to using PHP only for frontend templates, with everything on the backend being rewritten in more performant languages. They wrote a lot in C, and used the PHP as a thin wrapper around their C code.

Both Twitter and Facebook are cases of Coral Reefs, with new layers being built up that still make use of the old layers.

In the essay “The Lava Layer (or Lava Flow) anti-pattern” the real enemy is the top leadership, not the tech team leads. In every case described, the tech team leads see a need for an incremental improvement, and they begin to work on it, and then the top leadership interrupts the process, so the code becomes an inconsistent mess. There isn’t much that the tech team leads can do to fix the situation — they are doing the right thing, but the top leadership sabotages them.

In other words, the difference between a Lava Layer failure and a Coral Reef success is the behavior of the top leadership. This isn’t something the tech team leads can necessarily control. A growing company should support its tech team while the team builds needed new layers — then the company is on the path to be the next Facebook or Twitter. Or the top leadership can sabotage the tech teams, and end up with a set of conflicting apps that cripple the company.

The tech team should do its best to educate the top leadership about what is at stake. Because the tech team needs to be supported by the top leadership, if the tech team is going to be able to build the new layer that revitalizes the old layers and keeps the company moving forward.

Source