How to restart public JVM services

How do have zero-downtime restarts and upgrades? This is interesting:

We’re going to use Stuart Sierra’s excellent component project to manage the lifecycle of our service which, for the moment, will simply store a random number on initialisation and serve that back in response to any request. Getting Jetty to start on a random, operating system assigned, port is simply a matter of passing zero as the desired port number. If we then communicate this port number in some way that nginx can pick this up we have the ability to run multiple instances of our service at one time and switch the reverse proxy.

Our etcd will be running on the host machine and not clustered: there is no need to communicate this service information outside of the host machine. To communicate the port number from our service to etcd we will employ etcd-clojure, and use a known key, uswitch/experiment/port/current.

(ns etcd-experiment.system
(:require
[com.stuartsierra.component :refer [Lifecycle]]
[etcd-experiment.util :refer [etcd-swap-value]]
[ring.adapter.jetty :refer [run-jetty]]))

(defrecord JettyComponent [root-key routes]
Lifecycle
(start [component]
(let [server (run-jetty routes {:join? false
:port 0})
server-port (.getLocalPort (first (.getConnectors server)))]
(etcd-swap-value (str root-key “/port”) server-port)
(clojure.pprint/pprint {:service-port server-port})
(-> component
(assoc :server server)
(assoc :server-port server-port))))

(stop [component]
(let [server (:server component)]
(.stop server)
component)))

(defn jetty-component
[root-key routes]
(map->JettyComponent {:root-key root-key
:routes routes}))

We’ll build a component that will store the PID of the service into uswitch/experiment/pid/current and ensure that it is dependent on the service component itself.

(ns etcd-experiment.pid-manager
(:require
[com.stuartsierra.component :refer [Lifecycle]]
[clj-pid.core :as pid]
[etcd-experiment.util :refer [etcd-swap-value]]))

(defrecord PidManager [root-key service]
Lifecycle
(start [component]
(let [pid (pid/current)]
(etcd-swap-value (str root-key “/pid”) pid)
(clojure.pprint/pprint {:service-pid pid})
(assoc component :pid pid)))

(stop [component]
component))

(defn pid-manager-component
[root-key]
(map->PidManager {:root-key root-key}))

This Gist brought to you by gist-it. view raw pid_manager.clj
We will also retain the previous values for both of these keys in uswitch/experiment/port/previous and uswitch/experiment/pid/previous, which is supported in our code by etcd-experiment.util/etcd-swap-value

The advantage of random port assignment is not only the ability to run the same service multiple times but also the port number is only available after the service has started. Hence we will only write the port and, by the component dependency, the pid information to etcd after the service has been successfully deployed and started.