Buy versus Build: mistakes I have made

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

I saw this question on Hacker News and thought I would share my own mistakes. I wrote about this at length in my book, and the following is an excerpt.

What I got wrong: I thought this work needed to come in-house, but in the end “in-house versus outsourcing” was not the crucial issue. The crucial issue was building a trusting, long-term relationship with the team, and that team could have been an out-sourced team.

Open Verse Media

When I first started, in 2016, at Open Verse Media, an ebook publisher, they asked me to look at their content management system (CMS/CRM). The staff had to rely on it, but it was very slow. The COO, whom I’ll call Robin, had overseen the creation of this app. The actual work of creating the software was outsourced to one firm, but after two years Robin felt they were too expensive. She fired that first firm and then hired a firm in Ohio, which I’ll call MegaStars.

The app had been built using a popular software framework called Ruby On Rails. Whenever Robin felt that a new feature was needed, she would ask MegaStars to add the feature. MegaStars billed $500 an hour, and over the course of seven years, a total of $3 million had been spent on the creation of this app.

The staff hated the app. When the head of marketing wanted to bring up the top 100 best-selling books, they would click on a link, and it would take a full 60 seconds for the page to come up. The staff had gotten used to the fact that they always needed to be engaged in two tasks, that is, something to keep them busy while they waited for the pages in the CMS to render. An advanced search, with multiple filters, could take up to five minutes to render a report. Many of the lower-level staff would simply go into Slack and engage in gossip with their peers while waiting for each page to slowly appear.

So on my first day I logged into our main web server, and right away I could see that the app was generating several thousand errors each hour, all of which were being written to a log file. Since this app was single-threaded, the work of writing the errors to the log file had to happen while each page was rendering. This was one reason why it was so slow.

This arms-length relationship needed to be closer.

Why did this app have so many problems? Well, when Robin requested a new feature, MegaStars would tell her exactly how much time was needed to get that feature done. If they felt a new feature needed 30 hours to build, they would simply quote $15,000 as the price tag. Sometimes the new work conflicted with old work and generated new problems, but that wasn’t in the estimate and therefore the new problems needed to be ignored as much as possible. This tactic of ignoring new problems had been going on for many years. Additionally, much of the code base was now out of date and suffered version conflicts whenever some parts of the system needed to use newer libraries of code (which in Ruby On Rails are called “gems”).

MegaStars could have said, “Pay us $100,000 and we will clean up all of these problems.” But then Robin might ask, “Why did you allow these problems to exist? What are we paying you for?” It might seem like a scam, if MegaStars asked for more money to fix the problems that they themselves had created.

Here was the central dynamic of the situation: Robin felt she held power because she could terminate the relationship at any time. In fact, all of the problems in the relationship were because she could end the relationship at any time and was leaning on that fact as her main way of getting compliance. MegaStars was unwilling to commit to the long-term health of the software while Robin was constantly threatening to fire them.

When you work with an outside agency, they typically can’t or won’t go back and clean up the code, because the customer is not willing to pay $500 an hour for that work. Some of the better agencies try to include the clean-up work in the overall price, but then those agencies seem expensive — and they get undercut by other agencies that are willing to do the absolute minimum, even if that means writing poor-quality code full of errors.

More one-on-one meetings would have helped

In many ways, the situation was worse than what I’ve already described. “Robin asked MegaStars to add a new feature” – what does this really mean? As a practical matter, the real process was something like this:

1. The staff hated the CMS.
2. Occasionally the frustration was so intense that it bubbled up to Robin.
3. Robin would convene a large meeting, including all team leads and their assistants.
4. Robin would give a speech emphasizing the need to control the budget, plus various warnings she had received from MegaStars – without doing a full re-write, MegaStars felt there was a limited amount they could do. Plus a full re-write would be too expensive.
5. Then Robin opened the floor to suggestions.
6. Everyone threw out some ideas, but without any knowledge of how much a feature might cost, and no real idea of what the budget was, the staff tended to engage in self-censorship.
7. Robin would pick three or four ideas that seemed interesting, then send them in written form to MegaStars.
8. MegaStars would send back a cost estimate.
9. Robin would then approve whatever items she felt were within the budget.
10. A new contract would be signed between Robin and MegaStars, regarding the next batch of work.
11. MegaStars would deliver the work, but without cleaning up some of the long-standing problems.

Please note, this is not a rant about out-sourcing. I’ve seen companies have great results while working with an outside agency. The real issue is this: if your company depends on an outside relationship, then that relationship needs to be a close, long-term, trusting relationship.

There were several factors that caused things to get so bad at Open Verse:

1. The CEO was an industry legend, but rather elderly, so she pushed most of her responsibilities onto her COO. Robin was therefore spread thin with too many responsibilities.
2. The CEO and COO had spent much of their careers in print publishing, and were slow to realize how different ebooks were. (Books that sold well were more topical, less based on the prestige of the writer.)
3. Robin was very slow to realize how much the organization depended on the CMS. She herself didn’t use it, so perhaps she didn’t realize how painful it was for staff to have to wait 60 seconds for a page to render.
4. Robin thought her power, regarding MegaStars, lay in the fact that she could fire them. In fact, this was a source of weakness in the relationship.

It does not matter if your company has an internal tech team or works with an external agency, if you are the COO, be prepared to have long one-on-one conversations with whoever is heading up your software development. Obviously the COO is going to push day-to-day management of the tech team to someone else (a CTO or a project manager who can operate at a high level) but then the COO needs to be in frequent contact with that person.

Who should accumulate requests for new features in the software? That should be the CTO or project manager, not the COO. It should be a regular, on-going process, not an occasional ad-hoc event. It needs to be someone who has the time to sit with those making the requests, talk to them one-on-one, and translate what they claim to need into what they really need.

One way or another, the only path forward for Open Verse Media was to find someone who could manage the software on a day-to-day basis. There were two possibilities:

1. Hire a project manager and let her manage the relationship with the outside agency. The project manager could focus on building a close, trusting relationship with that agency.
2. Hire a CTO, plus several software developers, and bring all software development in house.

Open Verse Media decided to go with the latter option. They fired MegaStars and instead hired a CTO plus several software developers. This should have given Open Verse Media the ability to move forward with faster and better software, as well as the ability to imagine software projects much more ambitious than anything that had been possible in the awkward and distrustful relationship with MegaStars.

As it happened, Open Verse Media hired the wrong person to be CTO. This person was an egomaniac and very controlling. This irritated the software developers, and after eight months there was a mass exodus where the whole tech team quit. If the goal was to get a team that could care about the software over the long-term, choosing the wrong person to be the team leader undermined the intent — a fact that keeps us from drawing any easy or simple conclusions from this story, regarding the benefits of out-sourcing versus in-sourcing. It evidently isn’t true that bringing the work in-house ensures the project will go smoothly. There remain other factors that can sabotage the situation. However, the fact remains that the relationship between the COO and MegaStars was unable to be productive because of distrust between the parties.

(The above was an excerpt from my book.)

Source