Rate limiting with Apache and mod-security

Rate limiting by request in Apache isn’t easy, but I finally figured out a satisfactory way of doing it using the mod-security Apache module. We’re using it at Brightbox to prevent buggy scripts rinsing our metadata service. In particular, we needed th e ability to allow a high burst of initial requests, as that’s our normal usage pattern. So here’s how to do it.

Install mod-security (on Debian/Ubuntu, just install the libapache2-modsecurity package) and configure it in your virtual host definition like this:

SecRuleEngine On

<LocationMatch "^/somepath">
  SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog
  SecAction "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog"
  SecRule IP:SOMEPATHCOUNTER "@gt 60" "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog"
  SecAction "phase:2,pass,setvar:ip.somepathcounter=+1,nolog"
  Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>

ErrorDocument 509 "Rate Limit Exceeded"

This does a few things and has a couple of knobs for you to tweak depending on your requirements. The first SecAction initializes the state, in this case by IP address. You can do this by a cookie if you like, but I needed it done by IP address (if you’re using a reverse proxy of some kind then get the IP from the X-Forwarded-For header here instead).

The second SecAction deprecates the counter by 1 every 1 second. This is setting the base rate of our rate limit, one per second. I’ve named the counter somepathcounter here, feel free to call it what you want and use different names for different rate limiting different parts of your site.

The SecRule checks to see if the counter is greather than 60 and if so it sleeps 300ms, sets the RATELIMITED environment variable and returns a 509 code response. This is setting the burst rate of our rate limit, to 60. Any IP can do a burst of 60 requests as fast as it likes and it then limited to 1 per second. If it makes no further requests for 60 seconds then the counter is decremented back to 0, which means their burst has been fully recharged.

The last SecAction increments the counter for every successful request (the previous SecRule skips this line if the request was rate limited).

Then the Header definition ensures a header is set whenever a request is rate limited, giving a hint to the client that they shouldn’t try again for 10 seconds. This is obviously just a guide and a lot of clients don’t implement it (and it’s really only valid on a 503 status anyway) so it’s a little bit of wishful thinking really.

Then we define a neat ErrorDocument for the 509 response to give a better clue to the client about what is happening.

509 HTTP Response

I chose to use the 509 HTTP code which isn’t a standard (it’s Apache’s own “Bandwidth Limit Exceeded” code). Technically a 503 (with the Retry-After header) is perhaps a better choice but I already use 503 for maintenance pages and I wanted to differentiate between the two responses in the logs. Twitter invented their own status 420 “Enhance Your Calm” code, but that appears to mostly be an elaborate marijuana reference. RFC 6586 defines 429 “Too Many Requests” for this purpose but it’s very new and Apache won’t let you set an ErrorDocument for it yet. And it’s debatable whether this is a 4xx error mode or a 5xx error mode.

mod-security state data

mod-security needs to store the rate limit state between restarts so you need to tell it where to write that data. I create a directory in /var/lib/mod-security but you can just stick it in /tmp if you like:

SecDataDir /tmp

If you don’t define this, mod-security just seems to silently not apply the rate limiting, so you need it even if you don’t care about state between restarts.

It doesn’t appear possible to store the mod-security state in a shared database so you can’t rate limit accurately when you have multiple load balanced web servers.

Sleeping considered bad

This implementation sleeps 300ms per request when the rate has been exceeded but sleeping here is generally not a great idea. It ties up the Apache worker for the duration of the sleep, so whilst it will relieve the load on your backend app (which would otherwise likely use lots of CPU) it arguably makes it easier to tie up all of Apache’s workers and take your site offline.

There are easier ways to tie up a web server though, and without a sleep many clients will just immediately retry the request, spamming the logs with useless messages and using HTTP parsing cpu etc. And without the rate limiting at all, your app might be even slower and could tie things up just as easily. So feel free to tune this as you like.

Basically, remember that this is not a malicious denial of service protection system – we’re just using it enforce a basic policy, most usually breached due to a mistake rather than an attack.

Other options

mod_cband sounds pretty good but I can’t find documentation on how to rate limit requests, just bandwidth, and it’s not packaged for Debian/Ubuntu.

mod_evasive sounds like something you might want to use for DoS protection.

If you’re using NGINX then the HTTP limit req module sounds nice and simple.

Netfilter‘s hashlimit matcher is very powerful and has the benefit of being in a layer before Apache entirely, so you can save even more CPU cycles. Netfilter also has the ability to rate limit logging too, so you won’t fill your disk with useless logs. If you want to return a 509 response, instead of just dropping or rejecting the SYN packet, then you can redirect rate limited connections to an Apache virtual host that just returns 509s (put it on a different port and use the DNAT Netfilter target).