Refactor async work in Ruby

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at:

Kevin Buchanan makes an interesting point:

But, that’s starting to seem like a lot of behavior, and maybe that behavior is crucial enough to our application that we want to have more control over it, or want one place to go to change how we retry asynchronous tasks in our application. If we consider this retry, backoff, failure logic a key feature of our app, we probably don’t want to be coupled to using Sidekiq for this. What if tomorrow we want to switch to a different background processing framework? What if we want to perform the file import outside of our Sidekiq worker? What if we want to start our retry process without queueing a job first?

I still recognize that as a valid point, though I’ll admit I’m growing weary of some aspect of the conversation. “Do you really want to depend on this technology when it might go obsolete and then you have to replace it?” If I followed that logic to its extreme then I would never be able to use any technology, since it will all go obsolete at some point. I recall 10 years ago I felt strongly that any outside dependency should be wrapped, but that caused me to write tons of wrapping code and my apps got verbose.

Maybe things have changed for me nowadays because I now follow a microservices where all of my apps are very small, so I don’t really have a problem throwing away any particular app when I feel like its got a problem. And I try to keep outside dependencies to specific apps.

All the same, its a good essay with some interesting points on how to refactor. He ends up with this:

file = PaymentFile.create(data: params[:data]){ time: + }).perform(FileImport, file){ time: 10.minutes }).perform(FileImport, file){}).perform(FileImport, file){ retries: 10 }).perform(FileExport, file)

and then he sums up:

There are a few benefits here. First, we now have a coherent interface for performing an abstract piece of work at an abstract time—either asynchronously, or synchronously with asynchronous retries. We’ve now completely hidden the specifics of our job queueing implementation in Sidekiq behind our own domain objects. We could swap out Sidekiq for something else tomorrow and know that our core app logic is going to stay the same, without having to make more changes. We also get the benefit of making it clear to everyone working in our codebase how our retry logic works, and where to go if we want to change how we backoff or handle failed jobs.

There are more pieces here, and the overhead might not be worth it for domains that aren’t very dependent on this behavior.