I got to have a bit of a nosey around the Manchester BBC building too – though I was worried I’d open the wrong door and end up on TV. Didn’t fancy having to apologise to Jeremy Paxman.

Brightbox also sponsored some pizza, and gave away t-shirts and stickers like candy (there was no candy though).

My slides are available here, and contain a little example file system indexer. I made my slides with webby and S6 if you’re interested.

Expect unstructured discussion of Ruby, Ruby on Rails and other random stuff plus nice people, great beer and coffee and geeky tshirts.

The balcony back room of Mr Foley’s has been booked.  Announce that you’re coming on the upcoming page.

Oh, and we now have a website: http://leedsrubything.org/

No clear plan yet either, but expect unstructured discussion of Ruby and Ruby on Rails at least.

Thursday 7th February 2008 at 7pm in the Victoria Hotel pub. All welcome!

More details here: http://upcoming.yahoo.com/event/423116

I’ll be talking about SANs, Centos, Ubuntu, Xen, Apache, Lighty, NGINX, MySQL and other goodies. Heck, I might even mention Ruby, which would be nice considering it’s a Ruby user group.

My business partner Jeremy will be nattering about the business side and various other things.

Update: A couple of photos here and here.

Rubinius has support (as of today!) for running multiple instances of it’s VM within one process, each VM on it’s own *native* thread, each VM running many ruby green threads. Each VM has it’s own heap and so each VM could load different apps that wouldn’t interfere with each other. We have plans for a mod_rubinius for apache that takes full advantage of this feature. Stay tuned ;)

- Ezra Zygmuntowi on a comment on Ruby Inside.

Very interesting stuff. Why bother making Rails thread safe when you have an awesome Ruby VM such as Rubinius. I’d like to see Mongrel (or FastCGI! Bring back FastCGI!) make use of this somehow, running multiple Rails instances itself in one process and distributing requests between them. Interested in knowing how it’d deal with memory leaks in external libraries though (like rmagick suffers from).

Still, you lose finer grained access to most of the nice UNIX process management stuff though then, like limiting memory usage with ulimits, but nobody seems to be using that for Ruby deployment anyway. It’s all fiddling around with Monit and such instead (why always with the steps backward!).

I could implement some locking in my application, but it’s always good to avoid as much new code as possible so, in the good old *NIX fashion, I cobbled together a short bash script taking advantage of existing tools. What this does is executes the given rake task in the given rails root using the Debian/Ubuntu tool start-stop-daemon (provided by the dpkg package, which is therefore always installed). start-stop-daemon uses a pid file to keep track of the rake program for the given task, so it will never run a second concurrent instance of rake for this task. Cron just keeps trying to run it every 5 minutes or whatever, but it only runs once concurrently.

Before running the rake task, my script also sets the maximum amount of cpu time can be used to the given number of seconds using the ulimit command. So if anything goes a bit mad, taking ages to complete and uses loads of cpu time, it will be killed and then Cron will start it again on the next period. I should do the same for ram usage too, but haven’t done yet.

It’s not completely fool-proof: for one, if a rake task hangs using no cpu it will never end and never be started again but this has not happened before and is still better than all tasks failing to run because the machine is unresponsive.

It’s executed thus (I set different nice values for different tasks using the nice directly in my crontab):

run-rake.sh /home/john/railsroot taskname:whatever maxcpuseconds

And here is the script:

#!/bin/sh
railsdir=$1
raketask=$2
pidfile=$railsdir/log/$raketask.pid
cpuseconds=$3
export RAILS_ENV=production

ulimit -t $cpuseconds
exec /sbin/start-stop-daemon -d $railsdir -b -o -m -p $pidfile --start --startas /usr/bin/rake $raketask

I tried the RadRails addon thing out for developing Rails. It has some nice features but is rather a big jump from vim, which I’ve been using up until now. And to the best of my memory, vim has never crashed once. Whereas Eclipse has already crashed about 10 times in 24 hours.

UPDATE: I used Eclipse for all my Ruby on Rails development for almost two weeks but I’ve now given up.  Even with leaky old Firefox and the monolith that is OpenOffice running concurrently I would rarely notice swapping, but Eclipse has decimated my swap partition.  It’s resource requirements (mostly RAM) have turned my brand new laptop into a 486 DX66 with 4MB RAM and a broken CPU fan.  It would regularly crash too.  Any time saving it’s features might have offered were well cancelled out by all the lost work.

I am using the Aptana Ruby on Rails Eclipse addons, so maybe you can blame that, though the Haskell addons were misbehaving too. Ridiculous stuff.

I’m just going to learn how to use some of the more advanced features of VIM.  I’m giving GVIM a go too.  Oh VIM, how I missed you so.

If you use Ferret anywhere that allows users to execute queries, those users can crash the Ruby process with a specially crafted query.  This was quite serious for a number of my sites (not to mention slowing development of a current app) so I applied the fix to the released 0.11.4 source and repackaged it as 0.11.4.1.

Obviously this isn’t in any way official, but it works for me and I’m sharing here for anyone else affected. Gem, tgz and zip here and just the patch available here (derived from the author’s changeset to trunk).

The patch is against the release source, as the subversion repository seems to be down atm (I got the changeset from the web bases subversion viewer).

Get upgrading!

So, say I have my code stored in a subversion repository on my local disk, say file:///project/trunk. That’s fine for when Capistrano is querying the latest revision, but the remote servers need to use the repository url svn+ssh://mymachine/project/trunk.

Without modifying the code, this was impossible with Capistrano v1. With Capistrano v2, you can prefix any scm configuration variable with local_ and it will be used for local operations:

set :repository, "svn+ssh://mymachine/project/trunk"
set :local_repository, "file:///project/trunk"

I’ll be the tall one with the curly hair… stood at the front… talking about Ruby on Rails.

Directions and stuff to be found on the WYLUG website.

Ferret offers huge improvements over the original MySQL full-text search method, and I’m looking forward to adding some fancy keyword statistics graphs in the future – perhaps showing censorship patterns in bbc comments with certain keywords.

Because News Sniffer is distributed across a number of servers, I used DRb (distributed Ruby) to allow them all to update one central Ferret index.  DRb seems to work very well generally, and is amazingly simple to use, but I ran into a few problems with recycled objects and invalid references whilst using Ferret across it, apparently due to the garbage collector on the service side collecting things still in use on the client side.  I think I eliminated most of them but they still crop up once in a while – I’ll be looking into this further.

I also moved from using memcached for cache fragment storage to FileStore.  This allows me to expire fragments using  regular expressions, which lets me use fragment caching more easily and more often (such as with paged listing).  FileStore is rather slower than memcached, especially when expiring using these regular expressions, but being able to use it more often outweighed the performance hit.  FileStore is obviously not distributed unless you have a shared file system, so I used DRb here too.

It would be nice to add regular expression expiry to memcached, but I think this goes against the original design spec for memcached.  I’m considering adding configurable memory limits to the Rails MemoryStore fragment store, where it’ll remove least recently used fragments when the limit is approached (currently it would just keep allocating ram until your OS killed your Ruby process).

I also found a (easily fixable on Linux/BSD) race condition in FileStore where you could theoretically retrieve a corrupted fragment when it’s used in a multi-process shared storage setup (though not a multi-thread setup, so my DRb’ed FileStore should be safe).

Hopefully, with the improved searching due to Ferret and the  higher performance due to FileStore, News Sniffer will now be more useful.

Djb has some interesting ideas about how we should be doing things on unix, namely: very differently from how we’re doing things now. As his own software follows these rules, it tends to seem a bit odd to a new user, especially the filesystem layout. He also doesn’t allow modified versions of his software to be distributed, so it’s a bit tricky for GNU/Linux distros to “fix” it. Most distros get around this by distributing a source-only package, which automatically patches the code and builds you your own package – you just can’t then distribute this package to others.

This makes djb’s software technically non-free in the FSF sense. Some argue that it’s these license limitations that have kept his software so secure all these years. I’m side stepping the issue for now – I’m either a pragmatist or I just don’t care about your freedom. Try to decide which it is, and whether there is actually a difference.

Anyway, follow the docs for your distro and get daemontools up and running then I can focus on the Ruby on Rails aspect. This post is getting a little too long and windy already.

Rails

So, the usual way to deploy a FastCGI RoR app is to run the spawner script to start it, then run the script every 5 minutes or so to restart the process if it crashes. This maybe be convenient but it’s just laughable. It delays startup on boot, leaves you high and dry for 5 minutes if ruby crashes and wastes resources checking. And the reaper script, for restarting and stopping the process, just looks in the process list to find the pid. More chuckling. What decade are we in?

We’re going to run the RoR FastCGI process under daemontools. It takes a little initial setup, and with multiple daemons some duplication of configs, but it will start immediately (or asap) on boot, will be restartable/reloadable without guessing pids and will restart automatically if ruby crashes. We can even do “advanced” security stuff such as dropping secondary groups, setting resource limits and even logging that can’t be manipulated by an attacker compromising ruby.

Daemontools setup

Our rails app is called shop and it’s located on the filesystem at /home/web/shop/current. We are going to run it as the user web on port 12000. We need the spawn-fcgi tool from lighttpd too. We’re going to run the log process as the user rorlog. You need root access on the deployment box but that shouldn’t be a problem for a serious deployment. I do have a way of configuring daemontools to allow your users to safely setup and control their own processes, but I’ll save that for another post.

1. Make a new service directory for this service:

   mkdir /var/lib/ror-shop

2. Create the service run script (and make it executable):

   cat > /var/lib/ror-shop/run
   #!/bin/sh
   export RAILS_ENV=production
   exec /usr/bin/spawn-fcgi -u web -g web -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi
   

3. Create the service log directory:

   mkdir -p /var/lib/ror-shop/log/main

4. Create the log service run script (and make it executable):

   cat > /var/lib/ror-shop/log/run
   #!/bin/sh
   exec setuidgid rorlog multilog t ./main
   

5. Symlink the new service into the daemontools services directory (usually /service):

   ln -s /var/lib/ror-shop /service

Within 5 seconds, daemontools will see the new service and start both the ror and the log run scripts.

If you need to run a few processes on different ports, just make a service for each process and modify the run script to change the port.

By default, the logs rotate at 99999 bytes and keep 10 files archive. See the multilog docs for arguments to change this

Controlling your processes

To control the process you use the svc command. For example to restart the service:

svc -t /service/ror-shop

-d to stop it, -u to start it, -h to reload the app, -k to kill it if ruby goes haywire (daemontools will restart it immediately).

I’m using spawn-fcgi to set the uid/gid because I need to preserve secondary groups in my setup and it does this. If you don’t need this, I’d recommend using the setuidguid tool that comes with daemontools:

exec setuidguid web /usr/bin/spawn-fcgi -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi

To set some resource limits you can use the softlimit tool. To limit rails to 30M of ram and 300 file descriptors, use something like:

exec softlimit -m 31457280 -o 300 setuidguid web /usr/bin/spawn-fcgi -n -p 12000 -f /home/web/shop/current/public/dispatch.fcgi

As you can see, daemontools can be a bit tricky for a new user and a bit fiddly for a the initial setup but it is reliable, secure and flexible enough to cover most Ruby on Rails deployments for sure.

Currently, all the forum and news article downloading and scraping happens on a different machine to the web server. It has a VPN connection to the database and memcache servers, but I’d like to integrate the Ferret text indexing system for better searching capabilities. To centralise Ferret, I have a three options:

1. regularly reindex new content from the database on the web server;
2. DRb a Ferret Object;
3. or use ActiveResource to access the models via the web service.

DRb-ing a Ferret Object would be quite elegant, but using ActiveResource would also replace the need for a database and memcache connection (and I could do much better fragment caching actually).

Anyway, I searched high and low for some docs – lots of blog entries about how great it is, but no real api docs. When I searched through the Rails code and found nothing either, I got suspicious. Finally I found a couple of blog entries stating that ActiveResource was dropped from Rails 1.2. It seems to be planned for Rails 2.0. Not sure how I missed this. I guess my search-foo is lacking.

I’ll be investigating other options. I’d much prefer not to build a SOAP or XMLRPC interface. Ugh.

As you might imagine, it’s getting trabigillions of hits right now. Lucky I wrote it in Ruby on Rails with its built in page caching which (after first render) serves as fast as the webserver can serve. And since I’m using Lighttpd too, that is very fast.

For example Tea and Flour can be put on the compost heap, but Bindweed and Walnuts cannot. And it’s not always a good idea to put Orange peel on there either.

For geeks: I wrote Compost This using Ruby on Rails, which is one of the best web frameworks I’ve used, and I’m really starting to love the Ruby language too. I’ll release the code soon as an example.

The problem with this from a performance standpoint is that a thread/process of the web application has to be running for the entire duration of the download. With a busy webserver serving many concurrent downloads, this is an immense overhead. The web server itself should be orders of magnitude faster at serving files directly than via a web application, but you can’t just stick the files in a different directory and hope nobody finds the secret urls. The new web server on the block, Lighttpd, has some clever solutions for this problem.

mod_secdownload

Lighttpd has a module called mod_secdownload which allows a web application to generate a time limited authenticated url to a file.

Basically, you stick all your files in a directory outside the web root. Then configure lighttpd to tell it the path to this dir, a secret password and how long to keep a generated url valid. Now you code your web application to receive a request, check the session auth stuff, then generate this time limited url (authenticated with the same secret password as lighttpd) and redirect the browser to it. The browser then makes a new request using this url but is this time served directly by lighttpd without invoking your web application, and therefore goes super awesome fast. Another benefit is that your web application doesn’t need to be on the same server as your files, so you can load balance your file hosting server around the globe without them needing access to your database or anything complicated like that.

Anyone eavesdropping on the session could get the file url, but it would only be valid for as long as you configured lighttpd. If security is key, then configure this period to be very short. If your web application is on a different server to the one holding the files, then you need to ensure the clocks are synced on them. If your files aren’t sensitive and you’re just trying to prevent hot linking, then choose a larger period.

The lighttpd setup is well explained on the mod_secdownload documentation page, along with an example of how to generate urls in PHP.

I wrote the following Ruby on Rails helper for something I’m working on. You should be able to modify it for your own purposes.

   def asset_url(asset)
           secret = "mysecretpassword" # same as in lighttpd
           uri_prefix = "/asset_files/" # same as in lighttpd
           filename = "/#{asset.id}"
           t = Time.now.to_i.to_s( base=16 ) # unixtime in hex
           hash = Digest::MD5.new( "#{secret}#{filename}#{t}" )
           "#{uri_prefix}#{hash}/#{t}#{filename}"
   end

In my case I’m just preventing hotlinking, so urls are valid for 10 minutes. Since time has a habit of increasing every second, you get new urls to the same files every second which sucks for caching. I fixed this by only generating a new URL every 5 minutes. So, before converting to hex:
t = t - t.modulo(300) # modulo must be half secdownload.timeout

X-LIGHTTPD-send-file header

There is another, easier option. Have your web application do the session magic checks, then simply return a special header to lighttpd telling it serve a file from the filesystem to the browser. Your web app terminates and lighttpd takes over.

X-LIGHTTPD-send-file: /var/assets/filename

Obviously this only works if your files are being served from the same web server as the web application. Also, you need to be careful about sending user defined data to the header, as the web server will serve up any file it has permissions to if asked. And you need to make sure lighttpd is configured to accept these headers too.

Additionally, I’ve not tried this method myself so I might have just made it all up in my head and the feature doesn’t exist.

More about all this can be found on the lighttpd wiki.

Today, after no further conversion, I got this message from him:

(16:14:48) Sid: Hey John!! Just wanted to say thanks for introducing me to Ruby on Rails.. I’ve picked up on it and its changed my life. Now I’m working a contract for the government and dating a hot american chick from new york. btw – like the photo. its class.
(16:16:16) Sid logged out.

He used to be a Coldfusion developer. After finding Ruby on Rails he must look back on Coldfusion and laugh up hard matter from his lower intestine.

Anybody else want to comment on how I’ve changed their life? If you only met me once and had to spend the rest of your life avoiding me it still counts.