Smash Company Splash Image

October 5th, 2017

In Technology

1 Comment











If you enjoy this article, see the other most popular articles




















If you enjoy this article, see the other most popular articles




















If you enjoy this article, see the other most popular articles

AWS does not protect you from devops

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

Once upon a time, people made fun of Linux GUIs because they would give you a graphical way to edit a configuration file, but you still had to know everything in the configuration file, and what the settings did. People made the reasonable point that slapping a pretty graphical interface on a configuration file did not make it any easier to work with Linux.

Nowadays, the clients I interact with say things like “We want to use AWS because then we won’t have to spend so much money on devops.” This is a victory for Amazon’s marketing. The reality is that Amazon offers a graphic interface for configuring complex systems, but you still need to understand the complex systems. Amazon is not actually hiding the complexity from you, Amazon simply makes it a little prettier.

I wrote about this a year ago in AWS is inappropriate for small startups because its complexity demands a specialist.

That post sparked some interesting conversations on Hacker News.

I’ll expand upon the point here, based on work I’ve been doing this week.

Recently I’ve been working on an app to write a few million small JSON documents to ElasticSearch on AWS. I get a lot of SocketTimeout errors and, more surprising, JVM OutOfMemoryError errors. To be clear, it is AWS that is throwing the JVM OutOfMemoryError errors. ElasticSearch is written in Java, and it is the ElasticSearch instance on AWS that is throwing the JVM OutOfMemoryError errors. Apparently the task of figuring out the appropriate level of hardware needed for my app is entirely on me. AWS does not auto-scale the ElasticSearch service. Which is to say that the crucial devops work is still my responsibility. AWS is not helping here.

It took me days to track down the AWS documentation that explains why I was seeing this error.

So I reconfigured the ElasticSearch cluster:

And I set up a bunch of dashboards and alerts to let me know when something goes wrong with this cluster:

The expensive thing about devops is that you need to learn a lot about configuration. That takes time. Time is expensive. And when I use AWS, I still need to learn a lot about configuration. For example, how many instances, of what size, do I need to use before I stop getting JVM OutOfMemoryError errors? The only way I can find out is trial and error. Amazon is not helping in any way here. I’m still spending time on devops tasks.

Among my frustrations, Amazon does not emphasize the errors that I was seeing. Check out the Troubleshooting page:

I would guess that there are more developers who have made my mistake than who have been tripped up by Kibana. Writing too much information too quickly and getting SocketTimeouts and OutOfMemory errors is probably a common mistake, especially since one or two million small JSON documents really should not be a lot for AWS. There are surely a few developers out there who started their projects with the naive belief that a default AWS cluster would be enough for a fairly minor task?

More so, Amazon does put some interesting obstacles in my way which would not exist if I was working with raw hardware. For instance, I have long known that Amazon has a load balancing service, for websites, and that load balancing service dislikes sharp spikes — its much better at gradual changes in volume, rather than sharp changes. I wouldn’t have thought they would use the same balancing in front of its ElasticSearch servers, but that does seem to be true — when my app first starts writing to ElasticSearch, the app faces frequent disconnects and so I see SocketTimeout errors in the logs, but after 3 or 4 minutes the app can write about 15,000 documents a minute. And this implies that they are load balancing:

“Your performance can also be impacted if your application isn’t sending enough I/O requests. This can be monitored by looking at your volume’s queue length and I/O size. The queue length is the number of pending I/O requests from your application to your volume. For maximum consistency, HDD-backed volumes must maintain a queue length (rounded to the nearest whole number) of 4 or more when performing 1 MiB sequential I/O. ”

There might be reasons to use AWS, but please don’t use it because you think it’s going to save you a money that you would otherwise waste on devops. If you use AWS, your contractors will still end up doing a great deal of devops.

===========================
===========================
===========================

Follow the conversation on Hacker News.

Source



Check out my book:





RECENT COMMENTS

August 28, 2020 6:37 pm

From Aleksandar on Why I hate Ruby On Rails

"I am in RoR about 7 years, but I don't like it (I do it on my primary job). In free time (freelance) I work mo..."

July 14, 2020 8:53 am

From lawrence on How I recovered from Lyme Disease: I fasted for two weeks, no food, just water

"In my experience, yes. I have made it common to do a 3 day fast a few times a year. Anecdotally, I would say i..."

July 13, 2020 3:07 pm

From Tes on How I recovered from Lyme Disease: I fasted for two weeks, no food, just water

"Thank you so much for sharing your story. I have recently been diagnosed with Lyme disease and want to treat i..."

May 31, 2020 9:03 am

From Gábor Hidvégi on What does PHP offer now?

"Dear Lawrence, I can agree with you partially, but not in the whole at all. I agree with you that even P..."

May 22, 2020 12:43 pm

From Frank Hileman on Object Oriented Programming is an expensive disaster which must end

"Good article. However I hate to see so called "SOLID Principles" conflated with OOD. SOLID is a form of cargo ..."

May 22, 2020 10:29 am

From ByterBit on Object Oriented Programming is an expensive disaster which must end

"The first time I looked closely at OOP I was astonished at the – idiocy, the absence of any usefulness, the ad..."

May 13, 2020 5:03 pm

From Just+An+Observer on His Girl Friday

"Closest match to the intensity of this is the movie "Twentieth Century" with John Barrymore and Carole Lombard..."

October 19, 2019 3:08 am

From Bernd Schatz on Object Oriented Programming is an expensive disaster which must end

"I really enjoyed your article. But i can't understand the example with the interface. The example is reall..."

October 17, 2019 4:50 pm

From Anderson Nascimento Nunes on The conventional wisdom among social media companies is that you can’t put too much of the onus on users to personalize their own feeds

"Can't speak for anyone else, but on my feed reader: 5K bookmarked feeds, 50K regex on the killfile to filter o..."

October 9, 2019 3:08 pm

From Dan Campbell on Object Oriented Programming is an expensive disaster which must end

"Object-Oriented Programming is Bad https://www.youtube.com/watch?v=QM1iUe6IofM..."

October 4, 2019 8:44 pm

From lawrence on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"Gorgi Kosev, I am working to clean up some of my Packer/Terraform code so I can release it on Github, and then..."

October 4, 2019 5:14 pm

From Gorgi Kosev on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"> Packer, sometimes with some Ansible. The combination of Packer and Terraform typically gives me what I ne..."

October 4, 2019 12:40 pm

From lawrence on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"Gorgi Kosev, about this: "I would love if you could point out which VM based system makes it simpler and..."

October 4, 2019 7:31 am

From Gorgi Kosev on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"I won't list anything concrete that you missed, because that will just give you ammunition to build the next a..."

October 4, 2019 1:39 am

From lawrence on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"Gorgi Kosev, also, I don't think you understand what a "straw man argument" is. This is a definition from Wiki..."

1 COMMENT

August 21, 2018
11:48 am

By Nanacy

Very good post.

Please have look for AWS and DevOps Training online or corporate http://www.a2info.com.

Leave a Reply

Your email address will not be published. Required fields are marked *