June 11th, 2016
(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: firstname.lastname@example.org
Sean Hull references a conversation that he and I had in Slack. I would like to expand on the argument that I made then. You might want to read his essay first, where he makes these points about AWS:
1. Over 70 services offered
2. Still complex to build high availability
3. Need a dedicated devops
4. Orchestration involves many moving parts
5. Troubleshooting failed deployments is difficult
At the time of our conversation I had crashed an AWS instance and we were having trouble fixing it. Among the points I made to Sean was that the problem was specific to AWS. If we’d been in the Rackspace cloud, I simply would have rolled back to yesterday’s image. A problem that would have taken 5 minutes to fix instead dragged on for 2 weeks. We became embroiled in a mystery no one could quite figure out. I even went to the great popup office that Amazon now maintains in New York (a fantastic resource), and I spoke in person with 2 AWS engineers, and they were unable to come up with a definite answer about what was wrong. To his enormous credit, Sean had the best guess of anyone, and when I followed his advice, the problems stopped.
At the last few startups when I’ve worked I’ve insisted on using the Rackspace cloud. It is fairly simple and straightforward. The interface is simple. A mediocre developer can figure it out in 20 minutes. This simplicity is a powerful feature! We should not underestimate it!
Though I strive to be the most amazing programmer that I can ever be, I am not striving to be the most amazing devops guy in the world. And yet, at very small startups, the lead developer is typically also the lead devops guy. I’ve already written of the drama that went on at Celolot last year, yet despite the many, many problems that we had there, we never had any devops problems. And that is because I was in charge of devops, and our needs were simple, and I managed everything through Rackspace. Rackspace made it easy for me to create an image of my servers each night, and when we needed additional development servers, we just spun up a new one from an existing image. Simple.
I can handle basic Linux devops. But AWS is not Linux devops. AWS has become its own specialty. With over 70 services offered, only a full-time AWS specialist can properly understand how to build something in AWS.
Some of you, reading this, will be tempted to say “You are only complaining because you are ignorant about AWS!” But that is exactly my point! Knowledge of Linux devops does not translate into knowledge of AWS, and for most small teams, they only need basic Linux devops. What AWS offers is overkill!
Last year we built a system with an architecture similar to what Matthias Nehlsen has written about — we had some Clojure apps that talked to each other over Redis. So I set up a simple Redis server. Please note: I did not setup a cluster. We didn’t need that yet. We just needed a simple Redis server. So I spun up a basic Linux server, installed some packages, spun up Redis, secured it, and I was done. It took less than hour, perhaps as little as 30 minutes.
Recently I was working with a team that had to build a data ingester, to pull data from multiple 3rd parties. I was working with an inexperienced team, so I taught them the basics of how such a system would work. I created this humorous illustration, which we reviewed together:
Again, even though I am not a devops guy, I took on the role of devops to set this system up. But this team had already committed itself to AWS, so I figured I would do this the official AWS way. How do I set up Redis in AWS? Immediately I run into this:
Ugh! It’s a bit more than I need right now!
I’m not saying AWS is bad! I think AWS is wonderful! For a big company with a dedicated devops guy, AWS is awesome. Or if a medium sized company has the budget to call in someone like Sean, then AWS is awesome. But for a small team with no dedicated devops guy, AWS is inappropriate. Because over and over again it forces you to think about complexities that you don’t yet want or need to deal with.
Last week I was talking with my friend Chris Clarke. He runs devops at Parsely. Their servers are all at AWS. Their frontend is written in Angular, and it makes calls to ElasticSearch. Chris told me he was running a 62 node ElasticSearch cluster, which stores over 100 terabytes of data. No doubt, when you are working at that scale, then AWS is very useful. But you need someone with the incredible skills of Chris Clarke, and that is a rare thing. And specialized. Small startups don’t normally have/need/want a devops guy of that skill level, because the devops needs of small startups tend to be minimal.
And of course, there has been a trend to reduce devops in small startups to nothing at all. Heroku started that trend, mostly for the Rails crowd, but then for other technologies. More recently, there have been companies such as LSQ.io, run by the very brilliant Antonio Pellegrino. If you are working with NodeJS, then you should be using LSQ — they give you a fantastic way to test, deploy and monitor your NodeJS apps.
And yet, the serverless movement is not quite 100% there yet. Heroku and LSQ are awesome, but if you need a Cassandra database then you need to set that up yourself. Someone like me, if I was running a small startup, I would offload what I could to such services as these, but for any special thing that my startup needed, I would spin up some servers in the Rackspace Cloud and I would install whatever packages I needed, and I would customize it myself. I am not a devops guy, but I can do all the basics that are typically needed at a small startup.
What I can’t do is deal with the vast complexity of AWS. I don’t have the time. I’m busy building software, and I need to learn the algorithms and technologies needed for my software. Learning AWS is, at this point, its own specialty, and therefore AWS is only useful to companies once the company is big enough to hire a full time devops AWS specialist, or at least big enough to have the budget to bring in someone like Sean on a regular basis.