Why are large companies so difficult to rescue (regarding bad internal technology)

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

Why are large companies so difficult to rescue?

Studying a bloated, sluggish, rigid behemoth helps you understand what your real advantages are when you’re small. In particular, moving at high speed requires a team where every member trusts the other members. This is possible when everyone in the company fits in a single room, but it is impossible when you have several thousand people spread across multiple countries.

I worry there is a lot of glib, superficial rhetoric coming out of Silicon Valley about the importance of being agile in one’s development processes. There are too many assumptions being made about the ease of introducing agile techniques. Especially maddening is the assumption that a large corporation can be agile if it simply wants to — as if becoming agile is simply a matter of willpower.

Why would this issue be relevant to the entrepreneur who is running a small startup? Mostly because if your startup is growing, you will witness some loss of agility within two or three years. If we consider an extreme case, a very large corporation, it might be easier to put your own troubles in perspective. Though there are many wonderful benefits that come from growing your company, it is also true that communication becomes more difficult. Moreover, you will increasingly be confronted with the consequences of decisions that you made in the past. So let’s consider how much scale limits speed.

Over the last 22 years, I’ve mostly worked with small- and mediumsized companies. When I was working with Time Out in 2012, the organization might have had a few thousand people interacting with it, but that’s only if you count every employee working at the affiliates and franchises. Yet at that time it was the largest company I’d worked with.

In 2019, I was working with an agency that was acting as a kind of out-sourced Research & Development hub for a company that had 11,000 employees and operated in 180 companies. It was, and is, a famous company that most of you would know, and many of you have been its customers. In this chapter, I will call it SuperRentalCorp.

A lot has been written on the theme, “startups are agile but big companies are dinosaurs that can’t get anything done.” This perhaps overlooks the big companies that remain adaptive to change, but for now, let’s focus on the question, why is it often difficult to rescue the technology situation at a big company? SuperRentalCorp is a great example of some of the problems that come up.

Here is how I was originally told that they needed help. The head of the agency that I was at said “Hey, this company needs help setting up an API. Do you know how to set up an API? Can you help them with that?”

An API is a portal through which software outside the company can reach the data in the company database. They are easy to build, so I was surprised when he told me that SuperRentalCorp had been trying to build an API for two years but so far had failed.

I thought to myself, “Two years to build an API! For God’s Sake, it is 2019, there are a million tools that make it easy to set up an API. How can they possibly think this is difficult? A good engineer can knock this out in a day. Why are they struggling?”

But my initial thinking assumed the type of scenario where only one database was being used. At most of the small startups where I have worked, it really is possible to build a simple API in a day or two, and to provision it with backups and failovers and redundancy with another day or two of effort. Absolutely it can be done in less than a week. But there are two big problems that plague rescue efforts at big companies: history and trust. I’ll talk about history first.

When I say “history,” I don’t mean simply dealing with legacy apps or obsolete code. I also mean dealing with the consequences of decisions that different CEOs have made over the last 30 or 40 years, as the company adapted to the changing winds of each decade’s zeitgeist.

One hundred years of solitude: the independence of the old colonial outposts

SuperRentalCorp got started more than 100 years ago, when large parts of the world were governed by a few empires run from Europe, and most corporations ran worldwide operations from their homes in some Western country. But then in the era between 1947 and 1980 all of the old empires broke apart and were replaced by over a hundred new countries, each of which was protective of its new independence. So SuperRentalCorp decided to adopt a decentralized structure. It set up subsidiaries for most countries, and the subsidiaries operated as independent companies. Sometimes partial ownership of the subsidiary was sold. This meant that when the first wave of big databases arrived in the late 1970s, this company had no central database, no central IT department, no CTO or CIO.

A little later, as the political situation seemed safer, the benefits of consolidating some services became obviousso some of the subsidiaries merged, creating regional companies. There was one corporation for the MidEast and North Africa, one for Europe, one for North America, one for Asia, one for South America. With such a structure the company entered the 1990s, and this is when the company got serious about managing all services through networked databases.

Facing intense global competition in the 1990s, the company decided to grow by buying its most successful competitors. If I told you all of the name brands, you would recognize most of them, but you’d probably be surprised to learn that all of them are now owned by a single corporation. I was surprised, as these were several companies that I had thought of as competitors. But they are no longer competitors.

Then, around 2005, the Web 2.0 moment arrived, leading to an explosion of nimble competitors who were using the Internet to offer a service similar to SuperRentalCorp but in new ways. Again, SuperRentalCorp bought several of these startups to squash the competition by absorbing it. Many of these startups only exist in one country, or a single market, such as the EU.

Over the last few years the new CEO decided it would be best to unify the company. Several of the international subsidiaries, which had been partly sold off, have now been 100% re-purchased and are now being restructured so they will operate as departments inside of the company, rather than as independent corporations.

As part of this new focus on unification, SuperRentalCorp would like to create a single unified API, so the outside world thinks the company has an internally unified tech architecture. That is, SuperRentalCorp would like to create the illusion that the company has a single database and that interacting with said database is easy.

A feat for magicians: making twenty databases appear as if they were one

But what is the reality? SuperRentalCorp has 20 major databases, run by 20 different teams, in at least ten different countries. Many of these teams enjoy a long history of operating as an independent company, each team guarding their data, partly out of security fears, partly out of concerns about local laws regarding user privacy, and partly out of sheer stubbornness.

As with any database operations, there are two concerns here: putting data into the database (the writes) and pulling data out of the database (the reads).

Pulling data out of the databases is not that hard. A single engineer, given a few weeks or months to work on it, could write some code to pull data out of each of the 20 databases and then store that data in the central API. Hypothetically, we could refresh the data every five minutes. A customer in New York, or in Paris, or in Tokyo, could go to the SuperRentalCorp website and search for a car — then the website would ping the API, and the API would respond with a list of cars available in that city. And the information would probably be accurate, since it is, at most, five minutes out of date. That’s the easiest way to implement such an API. With a bit more effort, we could have the 20 databases automatically update the API whenever anything changes in any one of those databases. That would eliminate the fiveminute delay and ensure that the search results are always 100% accurate.

Well, not really. In fact, sometimes people rent a car and promise to bring it back on a certain day, but then they do not return the car on that day. This happens often enough that car rental companies generally cannot guarantee you any particular car. The situation remains fluid right up to the moment that you arrive at some particular location and ask “Where is my car?” They then hopefully have some car that they can rent you. Since the behavior of the customers is a bit chaotic, the database ends up being chaotic, and with each individual database being chaotic, the combined unified database is also a bit of a mess.

All the same, creating this central source of data, to be read from, is not impossible.

But wait, suppose the customer is looking at the website, and then the customer decides to rent a car, so the website sends the results to … uh … where? Let’s say the website sends the results to the API, since the API acts as the company’s public portal. Then the question is, where does the API send the data? Does the API somehow know which of those 20 databases needs to be updated?

At first this might be simple. If the customer is in Tokyo, then the code should update the database in Japan; if the customer is in Paris, then the code should update the database in the EU. But wait, what if the customer in Tokyo is going on vacation to New York City, so the car being rented is in the U.S.? I guess we then update the database in the U.S.? But the payment is made in Japan, paid in yen, and might have to pay some taxes that are specific to Japan, so we might have to create two different records: one to handle the payment in Japan, and one to handle the car in theU.S. And if the customer brings back the car with damage, we might have to impose additional charges, so that would be another entry in the database in Japan, and probably also an update in the U.S., and in addition we now need to update a database of inventory and car maintenance ? And we have to create rules like this for every country from which customers might come, and also rules for every country where those customers might be renting cars. But wait, we have more than one database per country and we also have multiple brands. Was the customer using the regular service, which is expensive, or were they renting from SuperBudgetCarCorp, one of our many subsidiaries? We need more rules for each brand, in each country.

Eventually, it gets complicated. This is no longer a project that a single engineer knocks out in a few weeks.

So, in total, SuperRentalCorp spent three years and $25 million dollars on building this new API. They took their time with it and added in some clever bells and whistles, which perhaps makes sense when you realize this is the infrastructure on which they plan to build all future technology at the company.

What might happen if they get this wrong? What happens if the database becomes a chaotic mess, such that SuperRentalCorp is unable to keep track of where its cars are located? Well, speaking hypothetically, SuperRentalCorp could start to have some of the problems that Hertz has had:

Hertz customers claim they were arrested, some jailed and even held at gunpoint after false theft reports

More people have come forward to claim they were accused, arrested or even jailed after Hertz reported cars they had legitimately rented were stolen.

Carrie Gibbs was arrested at a California gas station in 2019 after Hertz told police she’d stolen the truck.

The truck had been rented for her from Hertz by an insurance company after an accident, but video obtained exclusively by CBS News shows police arresting her. They took her to jail, and she was charged with felony car theft.

The case was later dismissed but Gibbs said in court papers she lost her real estate license for a year.

https://www.cbsnews.com/news/hertz-claims-false-arrests/

Even Sisyphus would quit

Perhaps it says something about the level of effort involved that after two years the lead engineer on the project felt burned out and quit. During his last week at the company, I spoke with him and he left me with the impression that the most stressful thing about the project wasn’t on the technical side. Rather, the real stress was the other people. Every one of the 20 independent database teams jealously guarded its data, and they fought against his efforts to centralize that data. And at a certain point he was tired of arguing with them. For two years, his most important skill was, not his knowledge of databases, but his sense of diplomacy.

A big effort to revamp the technology stack at an old company will run into many problems, and these will be partly technical and partly psychological. Each team has to cede some power and then trust a process that is out of their control. And then what happens when the top leadership changes? What if a new CEO arrives and they decide to break up the company again? How much can the teams trust the durability of the current corporate strategy? Should they leave themselves the capabilities to go back to the old way they used to do things?

So far I’ve only talked about the problems arising from internal databases and internal processes. In a sense, those are all easy compared to the difficulties that arise from reliance on external suppliers of servicesnone of which were under the control of SuperRentalCorp.

Starting in the 1990s, there arose a management idea which argued that a company should focus on its “core competencies” and outsource everything else. The basic idea is that if you are a newspaper, for example, you don’t need to hire janitors to keep your office clean; instead, you outsource cleaning to a company that specializes in cleaning corporate offices. Focus on what you are good at, and leave everything else to someone else. If you try to do everything yourself, then you are guilty of “NotInventedHere Syndrome.”

There is a lot in this argument with which I agree, though I’ve also come to see the downsides. Outsourcing limits your flexibility, since you end up in long-term relationships with external companies that may not evolve with your needs. And though it might seem easy to fire one cleaning service and hire another, there are some kinds of services that are very difficult to replace.

In the 1990s, SuperRentalCorp decided to outsource the management of its customer loyalty program, as that was seen as a financial function, and SuperRentalCorp is not a finance company. The company they outsourced too is astoundingly primitive and behind the times — that company does not offer a public API for their service. So now, when SuperRentalCorp would like to make the loyalty system embeddable in a variety of CRM (CustomerRelationship Management) and POS (Point Of Sale) systems, SuperRentalCorp cannot do so, because it has no control over the technology decisions being made at the company that controls the loyalty program. Yes, SuperRentalCorp could end their relationship with that other company and develop their own technology for managing their loyalty program, but they are already fighting a great many tech battles. For now, ending their relationship with the company that manages their loyalty program is not considered an option. All of which helps explain why rescuing the technology at bigger, older companies can be so difficult. One is constantly fighting against history.

Can 11,000 people trust each other like dear friends?

There is one other issue, and it is all about trust. Multi-billion dollar companies are constantly dealing with both internal and external actors who are acting in bad faith. This isn’t theoretical; this is a daily reality. Glib rhetoric from places like Silicon Valley, suggesting that corporate agility is always desirable and easy, misses the reality of the situation. As much as I love the startup community, I feel like too much of the writing that comes out of Silicon Valley simply assumes away the problems that bigger companies are facing. In particular, much of the writing assumes that issues of trust are silly, rather than important and real. Some of the most superficial advice tends to “assume goodwill,” as if multi-billion dollar companies are just like Wikipedia.

As blind as this advice is regarding the weaknesses of large companies, it can also blind us to some of the real advantages entrepreneurs of small firms have. When the whole team is five people, and you can all look each other in the eye, you can double-check your coworkers when they seem to be acting aberrant. That isn’t possible when you have 11,000 employees in 180 countries. To a large extent, “be agile” is synonymous with “able to trust each other.”

Do you like Sam Adams beer? It’s a great beer, and it’s made by a great company. Founded in 1984 by Jim Koch and Rhonda Kallman, Boston Beer has had healthy margins and a great product for almost 40 years. At one point Jim Koch was interviewed by Inc magazine and he said, “Our advantage is that we’re small. Our suppliers are small. We close $5 million contracts with a handshake. We don’t need multiple rounds of legal review.”

That is the advantage that you also have, during the first few years, while your startup is growing. Some of the ideas that I’ve discussed are only possible in environments where people can trust each other. If someone from a Fortune 500 company were to read this book, on almost every page they might ask, “But what would the lawyers think of this? Shouldn’t we worry about lawsuits?” But during the first few years of your company, you may not know every one of your employees, but you will know every one of your managers, which makes it easier to proceed in a trustful manner.

Having said that, it is sad but true that at a certain scale it becomes difficult to continue to operate with the flexibility that comes from trust. Until someone can figure out the magic spell that allows vast groups of people in different countries, with different cultures, speaking different languages, to all trust each other as if they were good friends, then people in the startup community need to rein in their flippant recommendations that larger companies should be more agile. [2, see footnote]

Post external references

  1. 1
    https://www.cbsnews.com/news/hertz-claims-false-arrests/
Source