Smash Company Splash Image

March 12th, 2019

In Business

5 Comments











If you enjoy this article, see the other most popular articles




















If you enjoy this article, see the other most popular articles




















If you enjoy this article, see the other most popular articles

Why are large companies so difficult to rescue (regarding bad internal technology)

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

I worry there is a lot of glib, superficial rhetoric coming out of Silicon Valley about the importance of being agile in one’s development processes. There are too many assumptions being made about the ease of introducing agile techniques, and which problems are solved by agile techniques. This essay is my attempt to offer a corrective.

Over the last 20 years I was the technical co-founder at three startups, two of which grew to modest size and then were sold (I talked about this a little in the first book that I wrote). I have done some consulting at medium to large sized companies, by which I mean, companies that had a few offices and maybe, at most, 3,000 people. Especially after I moved to New York City (in 2009) I worked as a rescue agent, trying to save older media companies that had fallen behind the times and were trying to catch up.

Now I am working with the largest company that I have ever worked with. It has 11,000 employees and it operates in 180 companies. It is a famous company that most of you would know, and many of you have been its customers. In this essay, I will call it SuperRentalCorp.

A lot has been written on the theme “startups are agile but big companies are dinosaurs that can’t get anything done.” And a lot of business writers have written interesting books about how big companies can restructure to be a bit more agile. For instance, Eric Ries offers some interesting ideas in his book The Startup Way.

But why is it so hard to rescue the technology situation at a big company?

SuperRentalCorp is a great example of some of the problems that come up.

Here is how I was originally told that they needed help. A friend of mine who was already consulting with them said “Hey, this company needs help setting up an API. Do you know how to setup an API? Can you help them with that?”

He told me that the company had been trying to build an API for two years, but so far they had failed.

I thought to myself, “Two years to build an API! For God’s Sake, it is 2019, there are a million tools that make it easy to set up an API. How can they possibly think this is difficult? A good engineer can knock this out in a day. Why are they struggling?”

But my initial thinking sort of assumed type of scenario where you have one database, and you are just putting the API between the database and the outside world, which is a common scenario at the early stage startups that I’ve worked at. With a greenfield Ruby On Rails project, you can seriously build this kind of API in an hour.

There are two big problems that plague rescue efforts at big companies: history and trust. I’ll talk about history first.

When I say “history” I don’t mean simply dealing with legacy apps, I also mean dealing with the consequences of decisions that different CEOs have made over the last 30 or 40 years, as the company adapted to the changing winds of each decade’s zeitgeist.

SuperRentalCorp got started more than 100 years ago, when large parts of the world were governed by a few empires run from Europe, and most corporations ran worldwide operations from their homes in some Western country. But then in the era between 1948 and 1980 all of the old empires broke apart, and were replaced by over a hundred new countries, each of which was protective of its new independence. So SuperRentalCorp decided to adapt a decentralized structure. It setup subsidiaries for most countries, and the subsidiaries operated as independent companies. Sometimes partial ownership of the subsidiary was sold. This meant that when the first wave of big databases arrived in the late 1970s, this company had no central database, no central IT department, no CTO or CIO.

A little later, as the political situation seemed safer, the benefits of consolidating some services became obvious, so some of the subsidiaries merged, creating regional companies. There was one corporation for the MidEast and North Africa, one for Europe, one for North America, one for Asia, one for South American. With this structure, the company entered the 1990s, and this is when the company got serious about managing all services through networked databases.

Facing intense global competition in the 1990s, the company decided to grow by buying its most successful competitors. If I told you all of the name brands, you would recognize most of them, but you’d probably be surprised to learn that all of them are now owned by a single corporation. I was surprised, as these were several companies that I thought of as competitors. But they are no longer competitors.

Then, around 2005, the Web 2.0 moment arrived, leading to an explosion of nimble competitors who were using the Internet to offer a service similar to SuperRentalCorp, but in new ways. Again, SuperRentalCorp bought several of these startups to squash the competition by absorbing it. Many of these startups only exist in one country, or a single market, such as the EU.

Very recently, the new CEO decided it would be best to unify the company. Several of the international subsidiaries have been 100% purchased and are now being restructured so they will operate as departments inside of the company, rather than independent corporations.

As part of this new focus on unification, SuperRentalCorp would like to create a single unified API, so the outside world thinks the company has an internally unified tech architecture. That is, SuperRentalCorp would like to create the illusion that the company has a single database, and interacting with that database is easy.

But what is the reality? SuperRentalCorp has 20 major databases, run by 20 different teams, in at least 10 different countries, many with a history of operating as an independent company, each team guarding their data, partly out of security fears, partly out of concerns about local laws regarding user privacy and the regulation of international transfers of user data, and partly out of sheer stubbornness.

As with any database operations, there are two concerns here, the reads and the writes. The reads are not that hard. We could pull the (necessary) data from 20 different databases, store the data in a centralized database that would act as a cache, and put an API between that database and the rest of the world. There would be minor issues about stale data, and we’d have to experiment to figure which data is high priority and needs to be copied over in a matter of seconds. Less important data could be copied over every 5 minutes, or on a trigger when updated.

The reads are not that difficult (not easy, but not impossible).

The writes are another thing. If a customer in London wants to rent a resource from the subsidiary we have in London, the rent request (the database write) needs to go to the central API, but does that mean the central API has to know which internal database that particular write is supposed to go to? Likewise the write requests happening in Nigeria, and Germany, and Brazil, each of which will go to different databases. This becomes a bit of a nightmare. Twenty years ago this line of thinking lead to the creation of the Enterprise Service Bus architecture, but ESB is now going out of favor, because it was too complex and rigid and unwieldy.

Two years ago, SuperRentalCorp decided to become a customer of MuleSoft, to help create their new API. They have so far spent about $25 million on their efforts involving MuleSoft. MuleSoft has some great tools for building APIs but those tools seem to help with the reads much more than with the writes. Which is to say, MuleSoft helps with the easy stuff, but not so much the hard stuff. (Having said that, I’ll add that there are engineers working for SuperRentalCorp who love MuleSoft.)

In terms of the best integration architecture, what seems to me the only long-term solution is something like the unified log architecture that Jay Kreps wrote about back in 2013. All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log. However, SuperRentalCorp has retail outlets with POS (point of sale) systems which talk directly to specific databases, and the path of that write (straight from the POS to the database) is hardcoded in ways that will be difficult to change, so it will be a few years before the company can have a single write-point. For now, each database team needs to be accepting writes from multiple sources. But a unified log is the way to go in the long-term. And that represents a large change of process for every one of those 20 teams. Which helps explain why the company has spent 2 years and $25 million trying to build an API, and so far they have failed.

The problem is partly technical and partly psychological. Each team has to cede some power and then trust a process that is out of their control. And what happens when the company gets a new CEO? What if she decides to break up the company again? How much can the teams trust the durability of the current corporate strategy? Should they leave themselves the space to go back to the way they used to do things?

Having said all that, I am pleased to say that this new API project is now in beta testing. After two years, it is finally going to be rolled out to the public this year.

So far I’ve only talked about the problems arising from internal databases, and internal processes. In a sense, those are all easy, compared to the reliance on external suppliers of services, none of which are under the control of SuperRentalCorp. Starting in the 1990s there arose a management idea that argued that a company should focus on its “core competencies” and outsource everything else. The idea is if you are a newspaper, you don’t need to hire janitors to keep your office clean, but rather, you outsource cleaning to a company that specializes in cleaning corporate offices. Focus on what you are good at, and leave everything else to someone else. If you try to do everything yourself, then you are guilty of “Not Invented Here Syndrome”. There is a lot in this argument that I agree with, though I’ve come to see the downsides. Outsourcing limits your flexibility, since you end up in long-term relationships with external companies that may not evolve with your needs. And though it might seem easy to fire one cleaning service and hire another, there are some kinds of services that are very difficult to replace. Back in the 1990s SuperRentalCorp decided to outsource the management of its customer loyalty program, as that was seen as a financial function, and SuperRentalCorp is not a finance company. The company they outsourced too is astoundingly primitive and behind the times — that company does not offer a public API for their service. So now, when SuperRentalCorp would like to make the loyalty system embeddable in a variety of CRM (customer relationship management) and POS systems, SuperRentalCorp can not do so, because it has no control over the technology decisions being made at the company that controls the loyalty program. Yes, SuperRentalCorp could end their relationship with that other company, and develop their own technology for managing their loyalty program, but they are already fighting a great many tech battles. For now, ending their relationship with the company that manages their loyalty program is not considered an option.

All of which helps explain why technology rescues at bigger, older companies are so difficult. One is constantly fighting against history.

There is one other issue, and it is all about trust. Multi-billion dollar companies are constantly dealing with both internal and external actors who are acting in bad faith. This isn’t theoretical, this is a daily reality. Glib rhetoric about how they should be more agile is not very helpful, as they are up against real questions of corporate structure, ownership, and strategy. As much as I love the startup community, I feel like too much of the writing that comes out of Silicon Valley simply assumes away the problems that bigger, older companies are facing. In particular, much of the writing assumes that issues of trust are silly, rather than important and real. Some of the superficial advice tends to “assume goodwill” as if multi-billion dollar companies are just like Wikipedia. The reality is that large companies are constantly facing the risk that they will be destroyed by the greed of people outside and inside the company. Startups have an easier time with the issue of trust, because when the whole team is 5 people, and you can all look each other in the eye, you can double-check your co-workers when they seem to be acting aberrant. That isn’t possible when you have 11,000 employees in 180 countries. To a large extent “be agile” is almost synonymous with “trust each other.” If you’re wondering why large companies have trouble being agile, it is partly because it is impossible for 11,000 people to trust each other the way 5 people can. That is simply reality. Until someone can figure out the magic spell that allows vast groups of people, in different countries, with different cultures, speaking different languages, to all trust each other as if they were good friends, then people in the startup community need to be a lot more careful about how carelessly they recommend that larger companies should be more agile.

[ [ UPDATE 2019-06-24 ] ]

Good conversation about this at Hacker News:

https://news.ycombinator.com/item?id=20260114

Source



Check out my books:
"I wish I could go back," said Anna. "I guess I thought it would always be there, and I could go back and learn more when I was older. But now I'm older and it's gone."

"All the great art scenes are like that," said Mariah. "Renoir's career was half over before the term Impressionism caught on. And Fitzgerald and Hemingway had given up on the Left Bank long before the place was overrun by talentless hacks who wanted to imitate the Lost Generation lifestyle. And the Beats had mostly left San Francisco before busloads of visitors started to do tours of the Haight-Ashbury. When Johnny Rotten couldn't work with the Sex Pistols anymore, he left and the London punk scene began to die. Later on, he said he regretted his decision to leave. Everyone thinks they can go away and come back later, but they never can. When Joan Didion and her husband left New York, she quipped that some other couples were staying too late at the party, but that gets it all backward. The party ends whether you want it to or not, and it takes an unusual arrogance to celebrate the end of an era that some people will remember as the best years of their life. Hemingway lived in Paris during his twenties, but he didn't write about his experience in Paris until he was in his sixties. No one ever knows they're part of an art movement; it's something you only see afterward."

"But if we only see it in retrospect, then how can we find the next great art scene?" asked Anna. "What do I look for?"




Also read this true story about a startup I worked at in 2015:




RECENT COMMENTS

September 2, 2019 11:58 am

From lawrence on Docker is the dangerous gamble which we will regret

"Chrisco, thank you, this is a great comment. You raise the point of MySQL in Docker, but you have to provide a..."

September 1, 2019 8:12 pm

From chrisco on Docker is the dangerous gamble which we will regret

"I live in the Java world. Since about 2000 all my web apps have been deployed into what have been known as "ap..."

August 29, 2019 5:39 pm

From Brandon on How I recovered from Lyme Disease: I fasted for two weeks, no food, just water

"This is a fantastic story. There's something deeply harrowing in the sentence "[then] I took Amoxicillin for 1..."

August 27, 2019 1:53 pm

From lawrence on High Availability is not compatible with a MVP

"Joshua Hoover, I strongly agree. I've previously advocated for Heroku, which was the pioneer in the serverless..."

August 26, 2019 10:39 pm

From Joshua Hoover on High Availability is not compatible with a MVP

"Agreed on HA not being a part of MVP. It's waste. The MVP is one step closer to discovering you have to do som..."

August 21, 2019 8:47 am

From Jorge Castro on Docker is the dangerous gamble which we will regret

"Hi there and I agree completely but I can resume as follow: Docker promises simplicity, i.e. IT IS EASY. ..."

August 20, 2019 2:29 pm

From lawrence on If you want to go dancing in New York City, consider Silvana

"Which is fine. Like I said, there are dance scenes that have strict “no alcohol” rules. That might appeal to y..."

August 20, 2019 1:35 pm

From Just An Observer on If you want to go dancing in New York City, consider Silvana

""Promise of an early bed" - the whiff of danger keeps me away from many venues like the one you describe...."

August 20, 2019 12:22 am

From lawrence on If you want to go dancing in New York City, consider Silvana

"I think any time you go to any club there is the possibility of running into an angry person, maybe a person w..."

August 19, 2019 7:56 pm

From Just An Observer on If you want to go dancing in New York City, consider Silvana

"I'm confused. You and your friends went out, had a fight, and it's still a great place to go to? Maybe..."

August 18, 2019 8:57 pm

From Michael L on Americans increasingly hate each other

"You seem to have little patience for people who choose different tech paths than you. Although it looks like o..."

August 18, 2019 8:53 pm

From Michael L on Many of my Stackoverflow questions have been marked as duplicates even though they were not

"You don't suffer slights well, do you? Others who choose to waste time with dumb tech, do they keep you up at ..."

August 18, 2019 8:34 pm

From Michael L on Docker protects a programming paradigm that we should get rid of

"Me again. I've worked for a company that focuses on containerized applications for some time now. There is abs..."

August 18, 2019 8:00 pm

From Michael L on Docker protects a programming paradigm that we should get rid of

"To build on my last statement, I'm not trying to show that I'm "smarter." I'm probably not, or if I am, who gi..."

August 18, 2019 7:48 pm

From Michael L on Docker protects a programming paradigm that we should get rid of

"You think that containerization is going anywhere? I agree that it isn't strictly necessary, but you mistake y..."

5 COMMENTS

Pingback: Why are large companies so difficult to rescue (regarding internal technology) – INDIA NEWS

Pingback: Why are large companies so difficult to rescue (regarding internal technology) - Elect Area

June 24, 2019
3:49 am

By RASHMI GUPTA

History and Trust..wonderfully summarized

July 1, 2019
2:09 pm

By Amanda

Having made human centered design and technology lead transformation my main focus for the last 3 years or so, I can’t say enough about your final point: trust has to be built and maintained in a programmatic way like any other resource. People have to understand what the corporate values are and that everyone up their chain and in their unit align with each employee’s personal interpretation of the values. No small feat!
It’s a huge difference in the operating constraints of startups and big corporations. It would be great to have another article that digs into this aspect more deeply.

Pingback: Link: Why are large companies so difficult to rescue (regarding bad internal technology) – Coté

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>