Lighttpd and Ruby on Rails: Secure and Fast Downloading

When controlling access to files on a webserver developers often use the web application itself as a file server. The request comes in, the script checks for some session authentication variable or something, then streams the file from disk (hopefully from outside the webroot) to the browser.

The problem with this from a performance standpoint is that a thread/process of the web application has to be running for the entire duration of the download. With a busy webserver serving many concurrent downloads, this is an immense overhead. The web server itself should be orders of magnitude faster at serving files directly than via a web application, but you can’t just stick the files in a different directory and hope nobody finds the secret urls. The new web server on the block, Lighttpd, has some clever solutions for this problem.

mod_secdownload

Lighttpd has a module called mod_secdownload which allows a web application to generate a time limited authenticated url to a file.

Basically, you stick all your files in a directory outside the web root. Then configure lighttpd to tell it the path to this dir, a secret password and how long to keep a generated url valid. Now you code your web application to receive a request, check the session auth stuff, then generate this time limited url (authenticated with the same secret password as lighttpd) and redirect the browser to it. The browser then makes a new request using this url but is this time served directly by lighttpd without invoking your web application, and therefore goes super awesome fast. Another benefit is that your web application doesn’t need to be on the same server as your files, so you can load balance your file hosting server around the globe without them needing access to your database or anything complicated like that.

Anyone eavesdropping on the session could get the file url, but it would only be valid for as long as you configured lighttpd. If security is key, then configure this period to be very short. If your web application is on a different server to the one holding the files, then you need to ensure the clocks are synced on them. If your files aren’t sensitive and you’re just trying to prevent hot linking, then choose a larger period.

The lighttpd setup is well explained on the mod_secdownload documentation page, along with an example of how to generate urls in PHP.

I wrote the following Ruby on Rails helper for something I’m working on. You should be able to modify it for your own purposes.

   def asset_url(asset)
           secret = "mysecretpassword" # same as in lighttpd
           uri_prefix = "/asset_files/" # same as in lighttpd
           filename = "/#{asset.id}"
           t = Time.now.to_i.to_s( base=16 ) # unixtime in hex
           hash = Digest::MD5.new( "#{secret}#{filename}#{t}" )
           "#{uri_prefix}#{hash}/#{t}#{filename}"
   end

In my case I’m just preventing hotlinking, so urls are valid for 10 minutes. Since time has a habit of increasing every second, you get new urls to the same files every second which sucks for caching. I fixed this by only generating a new URL every 5 minutes. So, before converting to hex:
t = t - t.modulo(300) # modulo must be half secdownload.timeout

X-LIGHTTPD-send-file header

There is another, easier option. Have your web application do the session magic checks, then simply return a special header to lighttpd telling it serve a file from the filesystem to the browser. Your web app terminates and lighttpd takes over.

X-LIGHTTPD-send-file: /var/assets/filename

Obviously this only works if your files are being served from the same web server as the web application. Also, you need to be careful about sending user defined data to the header, as the web server will serve up any file it has permissions to if asked. And you need to make sure lighttpd is configured to accept these headers too.

Additionally, I’ve not tried this method myself so I might have just made it all up in my head and the feature doesn’t exist.

More about all this can be found on the lighttpd wiki.