Smash Company Splash Image

May 13th, 2018

In Technology


The Coral Reef Pattern of incremental improvement

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at:

The Acme Legal Firm needed a small office, and there was little available for rent in their city, so they decided they would build their own place. They hired a builder and asked her what they should do.

“Well, I think we should build this office with wood. I’m a big believer in wood. It is beautiful and fairly strong, and it is easy to hire workers who know how to work with wood. The failure modes of wood are well known.”

“What about the risk of fire?” asked the CEO of Acme Legal Firm.

The builder laughed. “It’s funny, every time I suggest wood as a building material, clients ask me about the risk of fire. But I’ll have you know, modern wood is pressure treated and highly resistant to fire. You won’t have to worry about fire.”

“Okay!” said the CEO. “Let’s do this with wood!”

The office was built and the workers enjoyed the space, but the company grew and eventually the office was too small. The CEO decided they would need more space. Instead of hiring the original builder, or seeking a recommendation from them, the CEO picked a name randomly from an Internet search for “builder” and asked for a consultation.

A gruff, older man came to the office and introduced himself as a builder.

“We need more space,” said the CEO.

“You need to tear down this building and build a new one,” said the builder.

“That isn’t possible,” said the CEO. “I was hoping you could build an addition.”

“But your current office is made of wood. Aren’t you afraid of fire?”

“I think modern wooden structures are safer than they were in the past? At least, that is what I was told.”

“Every wooden structure that I have ever seen eventually burned down!” shouted the builder dramatically.

“I had no idea!” said the CEO in surprise.

“But even if you are not afraid of fire, what about rot from water? The roof will eventually leak, and the wood will rot. Your workers will get sick with mold, and you will go bankrupt because of all the lawsuits.”

“That does sound bad,” admitted the CEO, “We really don’t have the money for a complete rebuild, but I think the addition should use a new material. What do you recommend?”

“Steel,” said the builder. “It resists fire and water, it is incredibly strong, you can expand endlessly, if you ever grow huge and need a 90 story office, steel will get you there.”

“I can’t imagine that we will ever grow so large that we need a 90 story building.”

“But the important thing is, if you ever do need to scale up, steel gives you everything you need to grow.”

“Isn’t it expensive to hire workers who know how to work with steel?”

“Yes, but think of what you are getting! Not some cheap wooden structure knocked together by barely literate hammer swingers who think every problem can be solved with a few bangs, but highly trained craftsmen who are producing a structure that might last a thousand years!”

“That is a good argument!” agreed the CEO. “Okay, we will use steel for the expansion.”

The expansion was built and new hires moved into it and enjoyed working there.

After a few more years, the company had grown considerably, and it needed more office space. Rather than reaching out to either of the previous builders, or even asking them for a recommendation, the CEO again randomly picked a name out of an Internet search for “builder”.

The new builder arrived at the office the next morning. He was young and had very small glasses and a goatee, and also he was sunburned as he’d just come back from a month in the woods at a primitive Earth skills workshop.

“We need more space,” said the CEO.

“You need to tear down this building and build a new one,” said the builder.

“That isn’t possible,” said the CEO. “I was hoping you could build an addition.”

“Aren’t you concerned about what an ugly agglomeration your office is? Who mixes wood and steel as you have done?”

“I am sorry you feel that way, but we don’t have the money to rebuild everything right now. Do you have a recommendation for an addition?”

“Stone,” said the builder. “There is no need for fashionable building materials such as steel. Stone is a classic building material. Our earliest human ancestors built homes out of stone, so it answers a primal need and puts us in touch with our roots.”

“But isn’t a stone building cold and drafty?” asked the CEO.

The builder laughed. “It’s funny, every time I suggest stone as a building material, clients ask me if it’s cold and drafty. But seriously, what do you think grass is for?”

“I guess I never really thought about what grass was for,” admitted the CEO sheepishly.

“Do you think the Neanderthals stood around during the Ice Age, for half a million years, thinking ‘I wish our home wasn’t so cold and drafty.’?”

“That would be a long time to be cold,” admitted the CEO.

“Obviously they used grass. It’s the perfect insulator, and it is very cheap. Think about how much money you can save on your heating bill during the winter, if we use enough grass.”

“I do love saving money,” agreed the CEO.

“An addition built with stone will give your workers a warm place to hide during the winters, as well as answering their primal human need for connection to the Earth.”

“That is a good argument!” agreed the CEO. “Okay, we will use stone for the expansion.”

And so the expansion was built with stone. And new hires worked there, and found their primal need for connection with the Earth was somewhat answered by the stone, though many of them suffered terrible allergies from the large amounts of grass used as insulation.


This may surprise you, but the above story did not actually happen. In fact, it is a parody. This satire is aimed at the insane way that the tech industry has normalized the practice of hiring successive tech leads who have radically different preferences of architecture and language.

I’m writing this in response The Lava Layer (or Lava Flow) anti-pattern. It is a very good essay and everyone should read it. I realize the problem described there has some truth to it. However, I disagree about who should get the blame, and what the cure is. So I have two criticisms.

In terms of blame, that essay seems to suggest that the problem is with individual team leaders who allow their personal preferences to get in the way of what is best for the company. But I would put the blame at a higher level. Who thought it was a good idea to bring in a succession of team leaders, without first discovering that each of them had utterly opposing ideas about how to architect a system? That essay describes a situation where the top leadership has abdicated its role of vetting team leaders to see if they have a philosophy that meshes with the company’s real needs, both for modernization but also for maintaining the legacy systems.

I agree this does happen, much more often than it should. If I was going to change anything, I would change this terrible habit of the management. Business leaders need to be educated about the fact that a certain consistency of vision, regarding technology, is a benefit to the company.

However, I am not arguing for stagnancy. In every organization, there is a constant need to evolve to meet changing circumstances. One of the primary jobs of leadership is finding the right balance between stability and flexibility.

This is my second criticism of that essay: it does not suggest the correct way for an organization to evolve. And the pattern it criticizes is similar to the most successful pattern for change that I’ve seen.

The great French historian Fernand Braudel argued that civilizations build up like coral reefs, with each layer being built on top of the previous layer, and few disasters being so strong that they could actually sweep away all of the underlying layers. A terrible plague or a disastrous war might damage the top layer, but the lower levels are there and eventually the civilization rebounds, altered and yet still recognizable.

Something like this resembles most of the successful efforts I’ve seen to modernize the tech stack at large companies. I call this the Coral Reef Pattern. It is a pattern of incremental improvement.

I once worked with a massive monolithic app which had been written in PHP, using the Symfony framework. It suffered terrible performance issues. The pages were taking 10 to 20 seconds to render on the server, and much longer to serve to the client. I wrote about this in An architecture of small apps.

We began to gradually pull the system apart, keeping the Symfony monolith for the HTML template rendering, but pulling out the real performance blocks and re-writing them in more performant languages. For instance, routing in Symfony is known to be terribly slow, once you have a large number of routes. So we moved most of the routing to Nginx. Also, the code had been making calls directly to the database, and the ORM was slow, so we wrote an API using the Scala language, and we slowly changed the PHP code so it made calls to our API, rather than making calls to the database. The old PHP code was relying on Memcache to cache some of the database calls, but the logic was inconsistent and there were endless problems with stale data. Once the database calls were moved to a smaller Scala app, we built a simpler caching system for the Scala app, and we gained both speed and also simplicity.

An architecture of small helper apps allowed the old PHP monolith to survive. Rather than doing a complete re-write, we kept the old system working by strategically fixing the most urgent issues with small dedicated apps that were specialized to the one task they were given. Like a coral reef, the new layer did not replace the old layer, but added to it, in ways that made the whole system more performant.

One could argue that we made the system much more complex by introducing a new language. Wouldn’t it be simpler if everything was in PHP? Have we not injured the firm by splitting the work between two languages that start with radically different assumptions about computing?

Again, one of the primary jobs of leadership is finding the right balance between stability and flexibility. We should strive to find the simplest path forward that meets all of the needs of the business, but we should not sacrifice the needs of the business by artificially constricting the range of options that will be considered for technology solutions. PHP has the advantage that a lot of frontenders understand it, so it can be useful as a template language (I’m not a fan of PHP, but that is a common argument that I hear for it, and that is one of the few reasonable arguments for it). Scala has several advantages in terms of speed, strong types that can enforce contracts at API boundaries, and access to memory, so one can cache certain items without having to rely on a separate technology such as Memcache (for the sake of brevity, I’m going to skip the debate about when you would want to do this).

A number of big firms have followed the above pattern. Twitter started off with a monolith built in Ruby On Rails, and they were soon confronted with more traffic than could be handled, even with some very talented people trying to make the Rails code work. They then moved to an architecture where Rails handled the frontend, but everything on the backend was re-written to use higher level languages such as Scala. Likewise, Facebook started off as a PHP app, then moved to using PHP only for frontend templates, with everything on the backend being rewritten in more performant languages. They wrote a lot in C, and used the PHP as a thin wrapper around their C code.

Both Twitter and Facebook are cases of Coral Reefs, with new layers being built up that still make use of the old layers.

In the essay “The Lava Layer (or Lava Flow) anti-pattern” the real enemy is the top leadership, not the tech team leads. In every case described, the tech team leads see a need for an incremental improvement, and they begin to work on it, and then the top leadership interrupts the process, so the code becomes an inconsistent mess. There isn’t much that the tech team leads can do to fix the situation — they are doing the right thing, but the top leadership sabotages them.

In other words, the difference between a Lava Layer failure and a Coral Reef success is the behavior of the top leadership. This isn’t something the tech team leads can necessarily control. A growing company should support its tech team while the team builds needed new layers — then the company is on the path to be the next Facebook or Twitter. Or the top leadership can sabotage the tech teams, and end up with a set of conflicting apps that cripple the company.

The tech team should do its best to educate the top leadership about what is at stake. Because the tech team needs to be supported by the top leadership, if the tech team is going to be able to build the new layer that revitalizes the old layers and keeps the company moving forward.


Check out my book:


May 21, 2018 2:39 am

From xulin on Docker is the dangerous gamble which we will regret

"Elixir & Erlang..."

May 16, 2018 3:28 pm

From Steffen on Docker is the dangerous gamble which we will regret

"I'll never regret the use of docker cause I never used it (and never will). My advise is the the anti docker ..."

May 15, 2018 1:36 pm

From Clayton Gulick on Docker is the dangerous gamble which we will regret

"Great article. The current Docker craze, imho, is the same old problem where engineers get a hammer and every ..."

May 15, 2018 10:38 am

From lawrence on Docker is the dangerous gamble which we will regret

"Yawar, yes, I have looked at Elixir. And as I mentioned in the article: "They are now examining Go and E..."

May 15, 2018 10:32 am

From lawrence on Docker protects a programming paradigm that we should get rid of

"Agam Brahma, thank you. I have now fixed the typo...."

May 15, 2018 12:11 am

From Agam Brahma on Docker protects a programming paradigm that we should get rid of

"Great article, so true @ "unwillingness to confront the reality of the emerging solution". Minor typo: s/On..."

May 14, 2018 8:41 pm

From Yawar on Docker is the dangerous gamble which we will regret

"Just wondering, have you looked at Elixir? It seems to fit perfectly with the ideals you describe: based on de..."

May 14, 2018 7:00 pm

From lawrence on The Coral Reef Pattern of incremental improvement

"Andrew Whitworth, thank you for writing. About this: "Rails might not have been such a big problem for u..."

May 14, 2018 4:05 pm

From Andrew Whitworth on The Coral Reef Pattern of incremental improvement

"I like the coral reef metaphor, but this and your tangent about using separate languages both suffer from the ..."

May 14, 2018 12:43 pm

From Mickael on Docker is the dangerous gamble which we will regret

"Hi, I agree with some of your points, especially that docker is maybe more a dev tool than a production tool...."

May 14, 2018 6:25 am

From Eduardo Moroni on Docker is the dangerous gamble which we will regret

"Hey dude, Nice work. pretty awesome essay...."

May 14, 2018 6:18 am

From Pietro on Docker is the dangerous gamble which we will regret

"Hi, I wish there were more people like you around. The real mess I see coming, or, I should say, which has ..."

May 14, 2018 5:54 am

From Jongilanga Guma on Docker is the dangerous gamble which we will regret

"Awesome Essay, believing in Technology Hyper or fashion has showed us results that we regret. I think am gonna..."

May 14, 2018 3:27 am

From Nonax on Docker is the dangerous gamble which we will regret

"Hi! really nice article and great analysis! I'm sharing lot of your view concerning docker and its environm..."

May 14, 2018 12:34 am

From Anon on Docker is the dangerous gamble which we will regret

"Docker as a tool breaks down at one of the core linux philosophies. Lots of different tools doing lots of diff..."


Pingback: Docker:一场令人追悔莫及的豪赌 – 梁达标先生的持久战

May 14, 2018
4:05 pm

By Andrew Whitworth

I like the coral reef metaphor, but this and your tangent about using separate languages both suffer from the same vulnerability: At some point you may lose the operational ability to maintain it.

One of the promises of microservices was that you could write each new service in its own language with its own tech stack. A lot of developers took this as an opportunity really go wild with exploration, and some even to pad their resumes. Next thing you know you’ve got a dozen runtimes you need to support and not enough people on your team to keep up with it all. We recently finished rewriting one such service, written in Rails, where none of the remaining members of our team could adequately maintain it after the original developer left. Rails might not have been such a big problem for us if we had a budget to hire a replacement rails dev, and I doubt PHP is going to be a problem for you any time soon, with there still being plenty of PHP devs out there to hire. But then again, with more firms moving away from PHP like you seem to be, at some point many PHP devs are going to want to get out of that and focus on something newer, more exciting and with a more stable future. In 5, maybe 10 years we’re going to be thinking about PHP the way we think of COBOL, FORTRAN, FoxPro, etc: We’re going to have core systems written in those languages that we can’t afford to rewrite, and we don’t have anybody around who can maintain them. Inability to be easily and cheaply maintained is itself a form of brokenness, even if the software continues to do what it is “supposed to” do.

Saying “we’re using the coral reef pattern” doesn’t absolve you from cutting out cancer when you find it, and doesn’t magically make every historical decision you’ve ever made into a correct or sustainable decision. “We didn’t make a mistake, we’ve just moved on to a new layer!”

Using Facebook as an example as using a historical base of PHP and gradually moving to new things is sort of a bad example: They’ve put in extensive standards of features which can and cannot be used for performance and other reasons, and even gone so far as to write their own PHP interpreter to address some of the shortcomings. Sure, it’s still PHP in parts, but it’s a very different beast from what it was when they started. They’ve rewritten the bottom layer of their coral, but kept it looking just as ugly as it used to. I always found that curious.

So long as you have the capacity to maintain the old and develop the new you’ll be fine. But keep your eye on the forecast, because when the winds start to change you’re going to want to be able to avoid this trap.

May 14, 2018
7:00 pm

By lawrence

Andrew Whitworth, thank you for writing. About this:

Rails might not have been such a big problem for us if we had a budget to hire a replacement rails dev

Right, and that is a mistake made by upper management. Again, to be clear, I think the Lava Flow anti-pattern is a real problem, just like the Coral Reef pattern can be a real win, and the difference between the two is almost entirely the wisdom of upper management. Because in the end, change is necessary. That change will either be a well managed move to a new architecture, perhaps with new languages, or it will be a chaotic situation, full of multiple failed attempts to introduce something new.

I wrote this essay in response to the essay about the Lava Flow anti-pattern. That other essay seems to put the blame on the individual team leaders, but in this essay I was trying to make the case that the individual team leads are not really doing anything wrong. The blame should be aimed at the top leadership, who never allow the team leads to modernize the tech stack in a coherent way, and even worse, when one tech leader leaves the firm, the top leadership makes no effort to find a new team lead who shares some of the vision of the previous team lead. I agree this anti-pattern is common, and it is very detrimental. The satire at the start of my post is meant to highlight how ridiculous it is when top leadership fails to find team leaders who share any of the vision of the previous team leader.

(Exceptions would include the situation where the previous team leader was a disaster.)

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>