Daemontools and Ruby on Rails

Dan J Bernstein’s (djb) daemontools is a set of programs to help you manage unix services. It provides a flexible, secure and convenient way of starting, stopping and sending signals to background processes. Combined with his ucspi-tcp tools, it can be used as an awesome replacement to inetd (it’s most often used in this way to run qmail, a secure and high-performance mta). It can be fiddly to set up and has a bit of a steep learning curve but I already use daemontools for various other stuff, so it was just natural for me to use it for Ruby on Rails deployment.

Djb has some interesting ideas about how we should be doing things on unix, namely: very differently from how we’re doing things now. As his own software follows these rules, it tends to seem a bit odd to a new user, especially the filesystem layout. He also doesn’t allow modified versions of his software to be distributed, so it’s a bit tricky for GNU/Linux distros to “fix” it. Most distros get around this by distributing a source-only package, which automatically patches the code and builds you your own package – you just can’t then distribute this package to others.

This makes djb’s software technically non-free in the FSF sense. Some argue that it’s these license limitations that have kept his software so secure all these years. I’m side stepping the issue for now – I’m either a pragmatist or I just don’t care about your freedom. Try to decide which it is, and whether there is actually a difference.

Anyway, follow the docs for your distro and get daemontools up and running then I can focus on the Ruby on Rails aspect. This post is getting a little too long and windy already.

Rails

So, the usual way to deploy a FastCGI RoR app is to run the spawner script to start it, then run the script every 5 minutes or so to restart the process if it crashes. This maybe be convenient but it’s just laughable. It delays startup on boot, leaves you high and dry for 5 minutes if ruby crashes and wastes resources checking. And the reaper script, for restarting and stopping the process, just looks in the process list to find the pid. More chuckling. What decade are we in?

We’re going to run the RoR FastCGI process under daemontools. It takes a little initial setup, and with multiple daemons some duplication of configs, but it will start immediately (or asap) on boot, will be restartable/reloadable without guessing pids and will restart automatically if ruby crashes. We can even do “advanced” security stuff such as dropping secondary groups, setting resource limits and even logging that can’t be manipulated by an attacker compromising ruby.

Daemontools setup

Our rails app is called shop and it’s located on the filesystem at /home/web/shop/current. We are going to run it as the user web on port 12000. We need the spawn-fcgi tool from lighttpd too. We’re going to run the log process as the user rorlog. You need root access on the deployment box but that shouldn’t be a problem for a serious deployment. I do have a way of configuring daemontools to allow your users to safely setup and control their own processes, but I’ll save that for another post.

1. Make a new service directory for this service:

   mkdir /var/lib/ror-shop

2. Create the service run script (and make it executable):

   cat > /var/lib/ror-shop/run
   #!/bin/sh
   export RAILS_ENV=production
   exec /usr/bin/spawn-fcgi -u web -g web -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi
   

3. Create the service log directory:

   mkdir -p /var/lib/ror-shop/log/main

4. Create the log service run script (and make it executable):

   cat > /var/lib/ror-shop/log/run
   #!/bin/sh
   exec setuidgid rorlog multilog t ./main
   

5. Symlink the new service into the daemontools services directory (usually /service):

   ln -s /var/lib/ror-shop /service

Within 5 seconds, daemontools will see the new service and start both the ror and the log run scripts.

If you need to run a few processes on different ports, just make a service for each process and modify the run script to change the port.

By default, the logs rotate at 99999 bytes and keep 10 files archive. See the multilog docs for arguments to change this

Controlling your processes

To control the process you use the svc command. For example to restart the service:

svc -t /service/ror-shop

-d to stop it, -u to start it, -h to reload the app, -k to kill it if ruby goes haywire (daemontools will restart it immediately).

I’m using spawn-fcgi to set the uid/gid because I need to preserve secondary groups in my setup and it does this. If you don’t need this, I’d recommend using the setuidguid tool that comes with daemontools:

exec setuidguid web /usr/bin/spawn-fcgi -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi

To set some resource limits you can use the softlimit tool. To limit rails to 30M of ram and 300 file descriptors, use something like:

exec softlimit -m 31457280 -o 300 setuidguid web /usr/bin/spawn-fcgi -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi

As you can see, daemontools can be a bit tricky for a new user and a bit fiddly for a the initial setup but it is reliable, secure and flexible enough to cover most Ruby on Rails deployments for sure.