Rubinius has support (as of today!) for running multiple instances of it’s VM within one process, each VM on it’s own *native* thread, each VM running many ruby green threads. Each VM has it’s own heap and so each VM could load different apps that wouldn’t interfere with each other. We have plans for a mod_rubinius for apache that takes full advantage of this feature. Stay tuned ;)

- Ezra Zygmuntowi on a comment on Ruby Inside.

Very interesting stuff. Why bother making Rails thread safe when you have an awesome Ruby VM such as Rubinius. I’d like to see Mongrel (or FastCGI! Bring back FastCGI!) make use of this somehow, running multiple Rails instances itself in one process and distributing requests between them. Interested in knowing how it’d deal with memory leaks in external libraries though (like rmagick suffers from).

Still, you lose finer grained access to most of the nice UNIX process management stuff though then, like limiting memory usage with ulimits, but nobody seems to be using that for Ruby deployment anyway. It’s all fiddling around with Monit and such instead (why always with the steps backward!).

So, say I have my code stored in a subversion repository on my local disk, say file:///project/trunk. That’s fine for when Capistrano is querying the latest revision, but the remote servers need to use the repository url svn+ssh://mymachine/project/trunk.

Without modifying the code, this was impossible with Capistrano v1. With Capistrano v2, you can prefix any scm configuration variable with local_ and it will be used for local operations:

set :repository, "svn+ssh://mymachine/project/trunk"
set :local_repository, "file:///project/trunk"

Djb has some interesting ideas about how we should be doing things on unix, namely: very differently from how we’re doing things now. As his own software follows these rules, it tends to seem a bit odd to a new user, especially the filesystem layout. He also doesn’t allow modified versions of his software to be distributed, so it’s a bit tricky for GNU/Linux distros to “fix” it. Most distros get around this by distributing a source-only package, which automatically patches the code and builds you your own package – you just can’t then distribute this package to others.

This makes djb’s software technically non-free in the FSF sense. Some argue that it’s these license limitations that have kept his software so secure all these years. I’m side stepping the issue for now – I’m either a pragmatist or I just don’t care about your freedom. Try to decide which it is, and whether there is actually a difference.

Anyway, follow the docs for your distro and get daemontools up and running then I can focus on the Ruby on Rails aspect. This post is getting a little too long and windy already.

Rails

So, the usual way to deploy a FastCGI RoR app is to run the spawner script to start it, then run the script every 5 minutes or so to restart the process if it crashes. This maybe be convenient but it’s just laughable. It delays startup on boot, leaves you high and dry for 5 minutes if ruby crashes and wastes resources checking. And the reaper script, for restarting and stopping the process, just looks in the process list to find the pid. More chuckling. What decade are we in?

We’re going to run the RoR FastCGI process under daemontools. It takes a little initial setup, and with multiple daemons some duplication of configs, but it will start immediately (or asap) on boot, will be restartable/reloadable without guessing pids and will restart automatically if ruby crashes. We can even do “advanced” security stuff such as dropping secondary groups, setting resource limits and even logging that can’t be manipulated by an attacker compromising ruby.

Daemontools setup

Our rails app is called shop and it’s located on the filesystem at /home/web/shop/current. We are going to run it as the user web on port 12000. We need the spawn-fcgi tool from lighttpd too. We’re going to run the log process as the user rorlog. You need root access on the deployment box but that shouldn’t be a problem for a serious deployment. I do have a way of configuring daemontools to allow your users to safely setup and control their own processes, but I’ll save that for another post.

1. Make a new service directory for this service:

   mkdir /var/lib/ror-shop

2. Create the service run script (and make it executable):

   cat > /var/lib/ror-shop/run
   #!/bin/sh
   export RAILS_ENV=production
   exec /usr/bin/spawn-fcgi -u web -g web -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi
   

3. Create the service log directory:

   mkdir -p /var/lib/ror-shop/log/main

4. Create the log service run script (and make it executable):

   cat > /var/lib/ror-shop/log/run
   #!/bin/sh
   exec setuidgid rorlog multilog t ./main
   

5. Symlink the new service into the daemontools services directory (usually /service):

   ln -s /var/lib/ror-shop /service

Within 5 seconds, daemontools will see the new service and start both the ror and the log run scripts.

If you need to run a few processes on different ports, just make a service for each process and modify the run script to change the port.

By default, the logs rotate at 99999 bytes and keep 10 files archive. See the multilog docs for arguments to change this

Controlling your processes

To control the process you use the svc command. For example to restart the service:

svc -t /service/ror-shop

-d to stop it, -u to start it, -h to reload the app, -k to kill it if ruby goes haywire (daemontools will restart it immediately).

I’m using spawn-fcgi to set the uid/gid because I need to preserve secondary groups in my setup and it does this. If you don’t need this, I’d recommend using the setuidguid tool that comes with daemontools:

exec setuidguid web /usr/bin/spawn-fcgi -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi

To set some resource limits you can use the softlimit tool. To limit rails to 30M of ram and 300 file descriptors, use something like:

exec softlimit -m 31457280 -o 300 setuidguid web /usr/bin/spawn-fcgi -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi

As you can see, daemontools can be a bit tricky for a new user and a bit fiddly for a the initial setup but it is reliable, secure and flexible enough to cover most Ruby on Rails deployments for sure.

As with all 500 status codes, the 503 code is an error code but it signifies a temporary error. The client should try again later (in fact you can specify how much later using the Retry-After header).

A 200 code tells the client everything is normal and OK. So the user gets your nice maintenance page telling them of a temporary outage, whereas their browser gets told that everything is fine. Now this might not be a problem for a user, but if the client is a search engine or a caching proxy then it will assume the maintenance page is the new valid content for the request.

If the Google crawler hits your site when you have the maintenance page up, it will update its search index with your “we’re down for now” message, rather than your cash prizes blog content. Your page rank will drop, your fat Adsense cheque will diminish and you’ll have to go back to your regular nine to five job in the city with people you don’t like in clothes you hate wearing.

So, as you can see, it’s important to return the correct status code. Here’s how to do it with Lighty and mod_magnet:

Mod_magnet

Mod_magnet allows you to control request handling in Lighttpd using the Lua programming language (it replaces mod_cml and adds some bells and whistles). Here’s how to do a maintenance page with it:

Ensure the module is loaded:

server.modules += ("mod_magnet")

Tell it where to find your Lua script:

magnet.attract-physical-path-to = (server.docroot + "/magnet.lua")

Write a lua script to check for the maintenance page file and display it instead of the usual content if it exists.

if (lighty.stat(lighty.env["physical.doc-root"] .. "/maintenance-live.html")) then
                lighty.content = { { filename = lighty.env["physical.doc-root"] .. "/maintenance-live.html" } }
                lighty.header["Content-Type"] = "text/html"
        return 503
end

Create your maintenance page as maintenance.html in your document root. When you want it displayed, symlink it to maintenance-live.html and it will automatically be displayed.

I find it useful to be able to switch back and forth easily without restarts in and around the actual maintenance period but don’t need it day to day. So if you worry about the extra file stat operation (and the Lua) overhead during normal periods then just comment the Lua code out when you’re done with the maintenance feature for a while. mod_magnet will detect that you modified the Lua script and recompile it automatically.

You’ll notice that the return 503 actually returns a pre-generated HTML page as well as setting the status code. This extra bit of html gets tagged onto the end of your nice maintenance page, making it a bit ugly. You’ll just have to put up with this for now. I see this as a bug in Lighty which needs addressing. There should be some way to set a status code with Lua without returning the pre-generated html too.

Debug access

It might be convenient to disable the maintenance page for certain client IP addresses: if you’re upgrading the site, you’ll want to test it’s working yourself before disabling the maintenance page, right? I couldn’t figure out how to get the source address from within the Lua script, so I did it as a conditional in the Lighty config:

  $HTTP["remoteip"] != "10.0.0.1" {
    magnet.attract-physical-path-to = ( server.document-root + "/magnet.lua" )
  }

Using mod_magnet might be overkill for just this task, but if you’re already using it for other request processing stuff, then it’s ideal for this. Or maybe its convenience will be enough to sway you. Just remember to get the status code right!

See nixCraft for another example using PHP.