August 19th, 2018
(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: firstname.lastname@example.org
In reality, there’s a balance between moving fast and and moving slow. It’s difficult to communicate that balance because every type of product demands a different balance. I suppose that intuition comes from experience, which is a terrible answer for someone trying to learn.
What’s a new developer to do?
The natural tendency seems to be asking the internet. It turns out that this is incredibly effective.
It’s also incredibly dangerous. Before I go any further, I’ll continue my story.
This company continued to work with me after that product launch. I reviewed a significant amount of code, helped mentor their developers, and built new projects for them. Everything went swimmingly.
One day, I ran into a section of code that triggered my spidey sense. I could have sworn that I had seen it before. Sure enough, after pasting a line into a search engine, I found the exact section of code in a blog post. Naturally I read the whole thing, right up to the line that said, “Don’t do this in production.“
Yet here it was, tipping its hat at me from the front lines of a production codebase.
It didn’t take long to find many sections of code from similar blog posts. Almost all of the blog posts either wrote a disclaimer or should have written one. They all solved one small piece of a problem, but took many liberties in their solution to make it simpler to read. It’s understandable. Most readers appreciate brevity when learning a concept.
The code from these blog posts had spread through the codebase like a disease, scattering issues here and there without any rhyme or reason. And there wasn’t any obvious cure other than to read everything manually and fix issues as I went along. Without unit tests or automated deployments, this took almost a year. I’m almost certain the cost of fixing the code exceeded the margin on revenue due to writing it in the first place.
But what other option did these developers have? They had to deliver something, and they had never released a production application before. So they did what any sensible person would try to do, and they learned on the job.
Production systems can fail in an incredible number of ways. Without having experienced or read about these failures, it’s difficult to have an intuition about how to prevent them or how to solve them. It’s a tall order to ask a new development team to do this, especially without any guidance.
Before going any further, I want to mention that every person involved in this mess had good intentions. The developers who wrote the code wanted to build a good product and improve themselves. The managers who hired them wanted the same thing. The blog post writers wanted to share useful solutions. Everyone did their best to help one another out, and it’s important to remember that.