Tag: http

Rate limiting with Apache and mod-security

Rate limiting by request in Apache isn’t easy, but I finally figured out a satisfactory way of doing it using the mod-security Apache module. We’re using it at Brightbox to prevent buggy scripts rinsing our metadata service. In particular, we needed th e ability to allow a high burst of initial requests, as that’s our normal usage pattern. So here’s how to do it.

Install mod-security (on Debian/Ubuntu, just install the libapache2-modsecurity package) and configure it in your virtual host definition like this:

SecRuleEngine On

<LocationMatch "^/somepath">
  SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog
  SecAction "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog"
  SecRule IP:SOMEPATHCOUNTER "@gt 60" "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog"
  SecAction "phase:2,pass,setvar:ip.somepathcounter=+1,nolog"
  Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>

ErrorDocument 509 "Rate Limit Exceeded"

(more…)

15/5/2012 15 Comments

Maintenance pages status codes and Lighttpd

I’ve noticed a few very broken maintenance page Lighttpd config examples around, including the one on the mod_magnet documentation page. They all manage to display the maintenance page ok, but they return a HTTP 200 status code to the client, rather than the more appropriate HTTP 503 code.

As with all 500 status codes, the 503 code is an error code but it signifies a temporary error. The client should try again later (in fact you can specify how much later using the Retry-After header).

A 200 code tells the client everything is normal and OK. So the user gets your nice maintenance page telling them of a temporary outage, whereas their browser gets told that everything is fine. Now this might not be a problem for a user, but if the client is a search engine or a caching proxy then it will assume the maintenance page is the new valid content for the request.

If the Google crawler hits your site when you have the maintenance page up, it will update its search index with your “we’re down for now” message, rather than your cash prizes blog content. Your page rank will drop, your fat Adsense cheque will diminish and you’ll have to go back to your regular nine to five job in the city with people you don’t like in clothes you hate wearing.

So, as you can see, it’s important to return the correct status code. Here’s how to do it with Lighty and mod_magnet:

(more…)

10/1/2007 2 Comments

Referrer Securer

Did you know that Firefox (and Epiphany) don’t send referrers when following a link from an SSL encrypted site? The target site cannot tell whether you clicked a link or typed the url in directly.

I don’t know about other browsers, but this seems like a sane behaviour.

16/8/2006 1 Comment

Lighttpd and Ruby on Rails: Secure and Fast Downloading

When controlling access to files on a webserver developers often use the web application itself as a file server. The request comes in, the script checks for some session authentication variable or something, then streams the file from disk (hopefully from outside the webroot) to the browser.

The problem with this from a performance standpoint is that a thread/process of the web application has to be running for the entire duration of the download. With a busy webserver serving many concurrent downloads, this is an immense overhead. The web server itself should be orders of magnitude faster at serving files directly than via a web application, but you can’t just stick the files in a different directory and hope nobody finds the secret urls. The new web server on the block, Lighttpd, has some clever solutions for this problem.
(more…)

16/3/2006 3 Comments