The ethics of being a high level tech consultant (a Fractional CTO)

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

Let’s say you write code for most of two decades, lead some teams, and have your own startup for a few years. Eventually you decide you want to be what Jon Williams calls a “Fractional CTO” — that is, you want to share your wisdom with CEOs so their companies can do well. Perhaps you want to specialize in helping startups, especially during that long phase before they need a full-time CTO.

What are the ethics of such a role? At a basic level, many of these rules are things that even a child would understand:

1. Be honest when giving advice. Sometimes your advice will not be popular. In fact, at times your advice will be so unpopular that you risk losing the account — but that is a risk you always have to take if you’re going to do your job ethically. Please note, you should never be rude nor pick a fight, rather, you should always be polite and respectful. Put another way, you need to be assertive without being aggressive. But your client deserves your honest opinion, and you need to have the courage to give it even when they’re clearly unhappy with what you’re saying. Remember, no one is asking you to do anything that might be genuinely brave, like fight in a war, or racially integrate a Woolworth’s lunch counter in Greensboro in 1960, or “Speak now, against the day.” You’re simply being asked to be honest with your client when you think they’re about to do something stupid with money.

2. Be honest about the limits of your competence; don’t advise on subjects where you lack knowledge. Be especially honest if you have an opinion you know is controversial.

3. Be honest about the limits on your time. Also be careful that you don’t over-commit yourself.

4. Don’t reveal the company’s financial data or trade secrets, neither in public nor in private. Individual staff, as well as their strengths and weaknesses, can be discussed in private but not in public.

5. Pertinent to the above: Don’t poach the talent. My standard consulting contract has a “Don’t poach the talent” clause, which lasts during the time I work with a company and for two years afterwards. But this whole issue can raise very personal feelings of betrayal, so even after two years, you need to be careful. While it is natural to want to rehire people you’ve worked with before, stealing them from a former client can decimate the relationship you spent time building with that client. So far I’ve been lucky; I’ve never faced a competitive situation over staff. Every time I’ve rehired someone who I worked with before, they had already quit or the startup had folded.

6. Don’t steal money; in the case of an adviser, this mostly means don’t bill unnecessary hours and don’t invent work for yourself if that work is not needed.

7. Bill for every hour you work. Cheating yourself is almost as bad as cheating the client; in the long run it is unsustainable. If you end up poor because of a client, you won’t be able to do your best work. In addition, be sure you are charging a reasonable hourly rate. Right now in New York City, there are two types of advisors, those who are cheap enough that they still do some implementation (charging between $200 an hour and $400 an hour) and those who only advise, some of whom charge more than $1,000 an hour. In the lower tier, the best devops person I know in NYC charges $300 an hour, with a discount for long-term contracts that commit to a fixed block of hours per month. The rates are influenced by factors such as skill and reputation, but the largest factor is probably just the size of the company. Small companies might balk at paying $1,000 an hour, whereas for big companies that kind of money is trivial.

8. Try to protect your client from unnecessary costs.

9. Fire yourself as soon as you’ve built a team that can manage without you.

10. Don’t hurt people and don’t ask people to hurt themselves. If you allow a crisis to develop to the point that the only way to save the company is for the software developers to work 70 hours a week then you should be ashamed of yourself and you should resign. If your client wanted bad estimates and over-optimistic deadlines they could have hired any one of a million inexperienced project managers, but instead they hired you. Personally, I believe in scenario planning and it helps my clients: I’ll go on a long walk and try to imagine every scenario that might ambush us over the next year, then I plan what we should do to limit those risks (with the understanding that it is impossible to fully eliminate all risks, and so the challenge is to find what mitigation strategies are economic).


The above is the executive summary, the tl;dr. If you are in a hurry, you can stop reading right now — because I’m about to write a lot of words that merely repeat the above, along with a few real-life examples.


————————


Be especially honest if you have an opinion you know is controversial.

One of my more controversial opinions is that I generally advocate only for those AWS services that can be duplicated elsewhere. That is, I try to avoid the services that are unique to AWS, as that would lock clients into its continued use. (Andreessen Horowitz recently published a white paper about the real costs of the cloud, The Cost of Cloud, a Trillion Dollar Paradox.)

So am I giving my client good advice? Some advisers might say that I’m misleading my client, that I’m being unprofessional, that I’m not following “best practices.” That is, they would insist that AWS cloud computing is the absolute best available service for the price and I’m therefore doing my clients a disservice when I advocate for alternatives. Still, every good engineer I know has at least one or two controversial tech opinions, so in a sense, we all face this issue eventually.

In terms of the ethics, I handle this potential conflict simply by being upfront with the client, offering both sides of the argument, saying something akin to, “I don’t think we should use the AWS API Gateway, because if we ever move, we’re going to have to recreate that whole set of permissions/filters on our own. So if we need them, let’s just do it in our own code, it’s not difficult. But hey, listen, a lot of other advisers would tell you that AWS API Gateway is the way to go, partly because it can handle infinite scale. I discount that argument, because you’re a business-to-business entity, and so you’re never going to see the kinds of viral spikes that you might see if you were building something public, something business-to-consumer. So we don’t need infinite scale.”


Honest communication is the main way I handle the ethically gray areas.

Does that mean, however, that I should warn the CEO about every controversial opinion I have? No. For instance, if you look at the Wikipedia page for Object Oriented Programming, you’ll see that I’m listed as one of the critics, mostly because of an essay I wrote back in 2014. I can spend all day talking about the Functional paradigm versus the OOP paradigm. Is it necessary for the client to know this? Absolutely not. Styles of development can be worked out among members of the tech team. Most experienced engineers, most of the time, are just two beers away from an angry rant about why their favorite programming language is better than your favorite programming language. Should the CEO ever know about this? God no. Protect them from this kind of ephemera.

The only technical issues that matter to the CEO are the ones that involve large amounts of money, such as what AWS services we might use. Arguably, this is one area where there is a difference between a real CTO and a Fractional CTO. I’ve been at companies where the CTO decided to use Docker/Kubernetes plus related AWS services, yet never mentioned it to the CEO because such a decision is clearly under the purview of a CTO. As a Fractional CTO, knowing you’ll probably move on and leave the company in a few months, you typically don’t have an independent budget — so on big spending issues you check with the CEO more often.


Don’t bill unnecessary hours and don’t invent work for yourself.

This sounds simple but is actually maddeningly difficult in practice. Deciding what is necessary versus unnecessary is not always obvious. For instance, I want to be immediately helpful and yet I need to ask a huge number of questions up front before my advice is correctly tailored to the unique situation that my new client is actually facing. How many questions should you ask your client up front? The questioning process is essential, but does every question justify its inclusion in your billable hours?

For a while Time Warner was setting up a website for each one of the Showtime shows, and they were using WordPress to execute this project. They eventually decided they needed a better CMS than WordPress, so they hired PricewaterhouseCoopers to come up with a recommendation. PricewaterhouseCoopers interviewed a number of executives at Time Warner. Then they did a formal analysis of the strengths and weaknesses of various CMS platforms. Finally they gave their recommendation: get rid of WordPress and replace it with Drupal. For this recommendation, they billed $7 million.

Is that excessive? If someone had asked me, I would have given a highly educated opinion for a lot less than $7 million. Though I will admit I’ve never (while working alone) had a client as large as Time Warner, and I realize consultants dealing with big companies have to manage the complex politics of large bureaucracies. Still, even with that consideration, $7 million seems a bit excessive. Keep in mind, the $7M is just for the recommendation. The actual implementation of the idea was left to the various tech teams scattered throughout the Time Warner system.

Why didn’t the executives at Time Warner simply ask their own tech team leads for an opinion? Well, there was the risk that the existing teams would simply recommend the technologies with which they were already comfortable, whereas PwC was perceived as being able to give an objective choice. Also, the executive who had to authorize the money (in order to convert all WordPress sites to Drupal) was buying some insurance — that is, if the project turned into a quagmire, the executive could say, “This wasn’t my idea, it was the recommendation we got from PricewaterhouseCoopers.”

So again: is $7 million excessive? This is an ambiguous situation, it’s genuinely difficult to say. But perhaps it is the wrong question. Why didn’t the executives at Time Warner trust the leaders of their tech teams? Of all the team leads, there wasn’t even one who was considered both smart enough and objective enough to give a recommendation? Such a serious lack of trust suggests some extreme internal situation, and so the real mistake here was allowing that internal situation to develop to such a point of crisis. Compared to that, paying $7 million to PwC was minor.


Educating the CEO: how much is billable?

Again, don’t invent work for yourself. But is one inventing unnecessary work when one educates the CEO? It really depends on the situation, and here I think reasonable people will often disagree about how far one should go.

For instance, technical terms: there are many the client does not need to know. The people I advise are typically very intelligent and can quickly learn anything, but they are also very busy, so I should not waste their time forcing them to learn something unnecessarily.

So when does the client need to know some technical terms? Mostly when they need to make a decision, and especially if there is a lot of money involved.

I had a client who had previously decided to outsource the creation of their business intelligence tool. The outside agency attempted to describe how they were hoping to optimize the reports with a non-traditional approach, in which they would have a lot of tables that wouldn’t rely on other tables for fact lookups. I believe they were trying to raise the issue of normalization versus denormalization, but their language was so ambiguous that I couldn’t even figure out what they were saying.

For various reasons, including many other projects being undertaken at the company, I wanted all data centralized in the MongoDB database that the company was already using for other purposes. I asked for a meeting that would involve me, the CEO, and the representatives of the outside agency. At that meeting I assumed that I and the representatives would probably use technical terms such as “normalization” and “denormalization.” It therefore made sense to educate the CEO about these words, because otherwise they would have difficulty understanding what was being said.

If you’re going to explain a term, to what level of detail do you explain it? Should I offer the biography of Edgar Codd and what led him to invent SQL back in the 1960s? Should I explain what algebraic data types are? My preference is to explain the minimum that I think the CEO will need to know, so they’ll understand what kinds of trade-offs are involved in each decision. I sent this in an email, in preparation for our meeting:

A normalized schema is one where every piece of data has one raw, true, honest, absolute, canonical source; the natural place where it should live, with no redundancy.

A denormalized schema is one where the data might be either redundant or simplified, or both, for ease of query.

From the point of view of computer science or data science, it’s the normalized data that is real, whereas the denormalized form is something that can be thrown away. It’s disposable. Why do we even allow denormalized data? Because it’s often useful to designers and frontend programmers and people creating dashboards: it can make software much faster, dashboards more responsive, and it makes life easier for frontend programmers.

(The idea of normalized data was invented by Edgar Codd in the 1960s, and he actually proposed 6 levels of normalization, but I’m going to skip over that level of detail.)



That’s not exactly the computer-science definition of the terms, but I think it communicates enough that the CEO could then eavesdrop on our conversation and understand it.


Try to protect your client from unnecessary costs.

Much of this is common sense, so it does not raise any ethical challenges. The ethical difficulties come up when the client is already emotionally or intellectually committed to a particular strategy and you feel it’s important to change their mind. Then the ethical challenge becomes being honest about how much you disagree with them.

Two years ago I wrote Don’t waste $1 million on devops infrastructure that you’ll never need. That startup had already spent more than $1 million developing its main app and the DevOps infrastructure to guarantee that it would enjoy better than 99.999999% uptime. In that essay, I complained that instead of a minimal viable product they had developed something more like a maximal viable product. A business guru such as Steve Blank would have told them that their goal should have been to iterate quickly through a series of minimal viable products until they achieved “product/market fit.” Instead, they had wasted huge amounts of time and money on an advanced DevOps setup that they never actually used, because their product never became popular. In that case the damage was done before I first met them. Had I met them sooner, I would have advocated for much less investment in their DevOps setup. They could have simply uploaded their app to Heroku; that would have been a fine solution, giving them all they needed for months or even years. And it would have saved them at least $500,000.

Likewise, some parts of the tech industry have become infected with a mania for Kubernetes and Docker/containers. I have written a satire about some of the absurd conversations I’ve had on the subject. In general, when working with startups, I urge my clients to use the simplest DevOps setup that makes sense for them.

The basic idea here applies broadly: don’t use uncommon technologies when common technologies will do just fine. For instance, don’t hire computer programmers who work with a niche language (Haskell, Clojure, Arc, Scheme, Groovy) if your client is doing ordinary corporate work. For a straightforward website, stick with something in widespread use, like Python, Javascript, Ruby, Java (or a Microsoft technology, if the rest of their stack will be with Microsoft).

I anticipate the possibility of someone accusing me of being a hypocrite and citing my book How To Destroy A Tech Startup In Three Easy Steps
because we used Clojure there. But that was an advanced AI/NLP project; right from the start we knew we were hiring software developers who had specialized skills, so in that case using a niche language like Clojure was in keeping with the characteristics of the project. Using the Stanford NLP library was also highly specialized, as was our deep knowledge of the Salesforce API. That project had many failures, but the use of Clojure was not one of them.

However, a startup like that is an exception. Most companies are doing boring, ordinary stuff the majority of the time, and for those mundane tasks, you should stick to mundane technologies. Using widely used languages and frameworks allows you to hire from a larger pool of software developers, and this helps you get very good people without having to pay the kinds of salaries that specialists expect.

Again, most of this is just common sense. It only becomes an ethical issue when you feel the CEO is going down the wrong road, and you need to find the courage (and the diplomacy) to convince them to change course. I’m using words like “only,” but in fact, these dynamics pop up rather frequently; while I’ve been consulting, I’ve run into situations like this at every place where I’ve been called in.


Fire yourself as soon as you’ve built a team that can manage without you.

Once in a while you might fall in love with a startup and want to stay, but that is rare. For the most part, a part-time Fractional CTO is called in to help build something from scratch or to rescue a situation that has gone awry. In both cases, your goal should be to build a strong team and then leave. How long does this take? Every company is different. You might advise one CEO for a month and another for a year. Just remember you should be hiring and then training your eventual replacement.

But what if you wanted to stay and become full-time?

For the sake of this conversation, there are two types of startups — those built around a new innovation in software (MongoDB, Dropbox, Confluent), and those built around some innovation in design, marketing, finance, security, transport, agriculture, or anything else that is not software. With the first category of startups, the CTO is typically one of the founders, possibly even the inventor of the new technology, so it is very unlikely that you will be hired as CTO. But for all other startups, I give this reply:

You do not currently need a full-time CTO. You need high-level analysis and you need help hiring a tech team, but I can do all that and bill less than 15 to 20 hours a week. As to when you do need a full-time CTO, I offer the following rule of thumb:

◾ Once you have more than 20 engineers, you need a full-time CTO.

However, that is not really accurate, because it measures the wrong thing. A more complex but also more accurate rule of thumb goes like this:

◾ Since the CTO is literally a C-level executive, the hiring of such depends on the maturity of all of your other C-level functions. Do you have a Chief Marketing Officer and/or a Chief Product Officer, and do they have fully staffed teams?

When a startup has, for instance, six or seven or eight software developers, typically they will have a leader. The startup can hire someone or pick an existing member of the team. If you have a few good engineers, often one or two of them will possess both the technical and personal skills to win the trust of the others and become the natural leader of the team. Management can accept this; it is indeed a valid way to grow a team.

But only up to a point.

As the other C-level functions mature, it becomes very important to have an actual CTO (either hired from outside or promoted from within). This is important because a team leader often gets pulled down into mundane day-to-day work, lacking the time to think about long-term strategy. But it’s also important because of the nature of the other C-level functions.

It’s absolutely critical for the CTO to be assertive and put some limits on how the other C-level functions relate to the tech team. Since the tech team will always be a scarce resource, there needs to be a structured way that the product team, the marketing team, the business intelligence team, and the sales team all make requests and have those requests prioritized. Otherwise a startup will get into the difficult situation where, over the course of a week, the tech team hears, from different teams, “This is the startup’s number one priority!” and then from someone else, “No, this is the startup’s number one priority!” and then from someone else, “No, this is the startup’s number one priority!”

Once the other C-level functions have matured, someone who is merely in the role of informal leader of the tech team will lack the heft needed to push back against the demands of the Chief Marketing Officer and the Chief Product Officer and the Chief Sales Officer.

One person who I saw handle this well was Mark Herschberg. During the era from 2001 to 2011, ShermansTravel had operated without a CTO, even though they had a CMO and a CFO and a CSO and a Chief Content Officer. The result was exactly the kind of chaos that I described above. The company had five software developers on the tech team, and those developers faced constant pressure from the various C-level officers. As everyone worked in one big room, the CMO could simply walk over to one of the software developers and say, “I need you to drop everything and focus on this one particular task, because Marketing really needs it,” and then two hours later the CCO might walk over and say, “Hey, we really need this for Content, please focus on this and nothing else.” I could write a whole book about how this undermined the company. But in early 2011 the company hired Herschberg as the CTO, and he brought order out of chaos. He developed a point system in which every C-level executive had a budget of a certain number of points, and every request became a ticket that had an estimate of points on it, and, for any particular two-week sprint, the various competing C-level officers could use their budget of points to buy the tickets that they wanted to see happen during that sprint. The crucial element to this was that Herschberg, being a CTO, could speak to the other C-level executives as an equal. The system worked impressively well.


In the end, there’s no substitute for integrity.

The only absolute rule is that when you communicate with a client, you should always be doing so in good faith. We can discuss various rules that should be followed by any tech consultant who aspires to be known as ethical, but in practice these rules will lead to decisions that have trade-offs where reasonable people might disagree. In such cases, the best way forward is often to simply describe those trade-offs to the client so that they understand why you think their time and money should be invested in a particular way.

The occasional misjudgement is inevitable, that is, the occasional decision which later on we will wish we had made differently. But I suspect that so long as we always communicate in good faith, the nature of our intentions will come across to the client — and we can continue forward in a productive partnership that leads to more success for everyone.


For a clear example of how much good a Fractional CTO can do, please read “When A Quick Comment Is Worth $100 Million“.

Post external references

  1. 1
    https://www.linkedin.com/in/jonwill/
  2. 2
    https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap-cloud-lifecycle-scale-growth-repatriation-optimization/
  3. 3
    https://en.wikipedia.org/wiki/Object-oriented_programming
  4. 4
    https://www.pwc.com/
  5. 5
    https://www.amazon.com/Destroy-Tech-Startup-Three-Steps-ebook/dp/B0772FJQ1T/
  6. 6
    https://www.linkedin.com/in/hershey/
  7. 7
    https://www.shermanstravel.com/
Source