Sometimes I make a bad decision

It’s very, very, very rare, almost unheard of, but sometimes I make a mistake.

I was helping at Open Road this spring. Sean Hull was also helping. Open Road runs a bunch of WordPress sites, such as The Line Up. All of the images for these sites were stored on a central NFS server. All of the machines were starving for hard disk space. The previous Director Of Technology had made many, many bad decisions, including the decision to save money by keeping the hard disks dangerously small. Sometimes the disks filled up and everything broke, at which point the previous Director Of Technology would increase them slightly. Of all the places where he could have economized, this was the stupidest. Eventually he was quit/fired.

Sean very much felt we should move to S3. But I felt we were overwhelmed with too many other disasters, so I said we should just make the NFS server bigger. Much bigger. It had been 100 gigs, and I wanted it to become 1.5 terabytes. I figured that would give us breathing room. The previous Director Of Technology had made so many bad decisions that I was constantly dealing with some ambush. For instance, we had some important ingestion scripts that got us financial data, and he had “daemonized them with Screen”. I would laugh, except I’m crying. And then I killed the server and needed to restart it, but, oh, the daemons weren’t real daemons, so I had to re-write them as cron scripts.

Anyway, I asked Sean to increase the NFS server. He made a good argument that NFS is antiquated and we should move to S3. I 100% agree with him. I very much feel that Open Road should eventually move to S3. But I don’t think we had the resources to do it in April, when everything was falling apart. We would have had to re-write parts of the PHP code in WordPress to point to S3. Every image tag in every article would need to be changed. I didn’t have the energy to deal with all that.

But yes, Open Road should move to S3. We recently re-wrote some of the old apps as new React / Relay / GraphQL apps. As soon as the new apps are deployed, I think Open Road can think about moving to S3.

Keep that in mind when read Sean’s remark about Open Road:

So you find a lot of shops implementing things wrong because they didn’t understand a feature. In other words AWS already solved the problem.

OpenRoad comes to mind. They’ve got media files on the filesystem, when S3 is plainly Amazon’s purpose-built service for this.

He is 100% correct, but I don’t think we had the energy to make the change in April. So did I make a bad decision? Only time will tell. It’s a question of, will the NFS be good enough till a new system can be built.

I also made the case to Sean that small companies should not use AWS because AWS is now complicated enough that it has become the realm of specialists. Only when a company is big enough to justify a full time devops person does it make sense to move to AWS.

From Sean’s post: