Rate limiting with Apache and mod-security

Rate limiting by request in Apache isn’t easy, but I finally figured out a satisfactory way of doing it using the mod-security Apache module. We’re using it at Brightbox to prevent buggy scripts rinsing our metadata service. In particular, we needed th e ability to allow a high burst of initial requests, as that’s our normal usage pattern. So here’s how to do it.

Install mod-security (on Debian/Ubuntu, just install the libapache2-modsecurity package) and configure it in your virtual host definition like this:

SecRuleEngine On

<LocationMatch "^/somepath">
  SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog
  SecAction "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog"
  SecRule IP:SOMEPATHCOUNTER "@gt 60" "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog"
  SecAction "phase:2,pass,setvar:ip.somepathcounter=+1,nolog"
  Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>

ErrorDocument 509 "Rate Limit Exceeded"


This does a few things and has a couple of knobs for you to tweak depending on your requirements. The first SecAction initializes the state, in this case by IP address. You can do this by a cookie if you like, but I needed it done by IP address (if you’re using a reverse proxy of some kind then get the IP from the X-Forwarded-For header here instead).

The second SecAction deprecates the counter by 1 every 1 second. This is setting the base rate of our rate limit, one per second. I’ve named the counter somepathcounter here, feel free to call it what you want and use different names for different rate limiting different parts of your site.

The SecRule checks to see if the counter is greather than 60 and if so it sleeps 300ms, sets the RATELIMITED environment variable and returns a 509 code response. This is setting the burst rate of our rate limit, to 60. Any IP can do a burst of 60 requests as fast as it likes and it then limited to 1 per second. If it makes no further requests for 60 seconds then the counter is decremented back to 0, which means their burst has been fully recharged.

The last SecAction increments the counter for every successful request (the previous SecRule skips this line if the request was rate limited).

Then the Header definition ensures a header is set whenever a request is rate limited, giving a hint to the client that they shouldn’t try again for 10 seconds. This is obviously just a guide and a lot of clients don’t implement it (and it’s really only valid on a 503 status anyway) so it’s a little bit of wishful thinking really.

Then we define a neat ErrorDocument for the 509 response to give a better clue to the client about what is happening.

509 HTTP Response

I chose to use the 509 HTTP code which isn’t a standard (it’s Apache’s own “Bandwidth Limit Exceeded” code). Technically a 503 (with the Retry-After header) is perhaps a better choice but I already use 503 for maintenance pages and I wanted to differentiate between the two responses in the logs. Twitter invented their own status 420 “Enhance Your Calm” code, but that appears to mostly be an elaborate marijuana reference. RFC 6586 defines 429 “Too Many Requests” for this purpose but it’s very new and Apache won’t let you set an ErrorDocument for it yet. And it’s debatable whether this is a 4xx error mode or a 5xx error mode.

mod-security state data

mod-security needs to store the rate limit state between restarts so you need to tell it where to write that data. I create a directory in /var/lib/mod-security but you can just stick it in /tmp if you like:


SecDataDir /tmp

If you don’t define this, mod-security just seems to silently not apply the rate limiting, so you need it even if you don’t care about state between restarts.

It doesn’t appear possible to store the mod-security state in a shared database so you can’t rate limit accurately when you have multiple load balanced web servers.

Sleeping considered bad

This implementation sleeps 300ms per request when the rate has been exceeded but sleeping here is generally not a great idea. It ties up the Apache worker for the duration of the sleep, so whilst it will relieve the load on your backend app (which would otherwise likely use lots of CPU) it arguably makes it easier to tie up all of Apache’s workers and take your site offline.

There are easier ways to tie up a web server though, and without a sleep many clients will just immediately retry the request, spamming the logs with useless messages and using HTTP parsing cpu etc. And without the rate limiting at all, your app might be even slower and could tie things up just as easily. So feel free to tune this as you like.

Basically, remember that this is not a malicious denial of service protection system – we’re just using it enforce a basic policy, most usually breached due to a mistake rather than an attack.

Other options

mod_cband sounds pretty good but I can’t find documentation on how to rate limit requests, just bandwidth, and it’s not packaged for Debian/Ubuntu.

mod_evasive sounds like something you might want to use for DoS protection.

If you’re using NGINX then the HTTP limit req module sounds nice and simple.

Netfilter‘s hashlimit matcher is very powerful and has the benefit of being in a layer before Apache entirely, so you can save even more CPU cycles. Netfilter also has the ability to rate limit logging too, so you won’t fill your disk with useless logs. If you want to return a 509 response, instead of just dropping or rejecting the SYN packet, then you can redirect rate limited connections to an Apache virtual host that just returns 509s (put it on a different port and use the DNAT Netfilter target).

Comments

Keith Kim says:

Thank you. This is very helpful!

Chris T says:

Hello John,
Nice info (even it is from 2 years back). I tried your rule with modsec 2.7 and i get the error that there is no id on the rules. Could you update your post to match the current version of modsec?
Thanks in advance.
Chris

PF says:

I just had the same problem with modsec 2.7. They even have a FAQ entry that explains that the error means you’re using an outdated syntax – but the idiots fail to explain in what way the syntax has changed or at least provide a link to some sort of documentation for the new syntax. That’s probably because the documentation is generally nonexistant, which doesn’t exactly fille me with confidence.
After some googling: You essentially need to add “id=random-but-unique-numberhere” to each line, e.g.:
SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog,id:11
SecAction “phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:12”

me says:

Tried this but failed :(
SecRuleEngine On

SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog,id:887
SecAction “phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:888”
SecRule IP:SOMEPATHCOUNTER “@gt 60” “phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,889”
SecAction “phase:2,pass,setvar:ip.somepathcounter=+1,nolog”
Header always set Retry-After “10” env=RATELIMITED

ErrorDocument 509 “Rate Limit Exceeded”

ErrorLog /var/log/httpd/integrator_error_log

someone says:

Problem: Mod_security is writing the counter to a file with a lock or something. If you get your rate high enough, mod security cannot increase the counter fast enough, so you can break your rate limit easily. So this solution do not protect against DDos. Here you need something like fail2ban, cband or a firewall

Sophie says:

Hi,

Thanks lots for this code.

I dropped the 300 m/s down to 40 m/s so I could see the rule being hit in modsec_audit.log and saw this error. IP address obfuscated.


–677a4369-H–
Message: collections_remove_stale: Failed deleting collection (name “ip”, key “44.40.42.49_fec7e2f938f90a9a3b4b78f875c997a73a9444d9”): Internal error
Apache-Handler: proxy-server
Stopwatch: 1437655358027830 198518 (- – -)
Stopwatch2: 1437655358027830 198518; combined=337801, p1=134, p2=329, p3=20, p4=30, p5=168653, sr=47, sw=37673, l=0, gc=130962
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.6.
Server: Apache/2.2.15 (CentOS)
Engine-Mode: “DETECTION_ONLY”

Unsure if this is good.

I amended your rule because it had syntactical errors on my modsec.


SecRuleEngine On
#

SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog,id:11
SecAction “phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:12”
SecRule IP:SOMEPATHCOUNTER “@gt 60” “phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,id:13”
SecAction “phase:2,pass,setvar:ip.somepathcounter=+1,nolog,id:14”
Header always set Retry-After “10” env=RATELIMITED

ErrorDocument 509 “Rate Limit Exceeded”

Any advice on the errors I saw?

Kind regards,
Sophie.

Vadim says:

ErrorDocument 509 is not supported in Apache 2.4, so it is better to put 429 that is recognized and is able to be set using ErrorDocument directive

sixthousand says:

This 509 ErrorDocument definition will indeed break Apache 2.4 configuration, and I agree that 429 instead is the most sensible status code. Deeper dive here: https://stackoverflow.com/questions/33032500/apache-2-4-custom-error-responses

Alparslan says:

is there any way to add diferrent rate limit for diferrent ip adresses.

I want to write like this;

1.x.x.x rate limit is 30
2.2.2.2 rate limit is 60
last rule every ip rape limit is 10

SPI says:

Great topic, but how would you implement a rule telling that the rules must apply to any distinct resource in your site. I mean,
/my-resource/123 and /my-resource/456 are two separate resources. I need to allow 10 requests per minute per resource, ie. hitting 10 times the resource 123 should block it, but must not block the resource 456? All my attempts fail miserably…

Daniel says:

To prevent getting a huge file “/tmp/ip.pag” and also the error “Modsecurity: Failed deleting collection…”, I added the command expirevar, here the updated script (counts only html, txt, xml and pdf):

SecAction initcol:ip=%{X-Forwarded-For},pass,nolog,id:4444446
SecAction “phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:4444447”
SecRule IP:SOMEPATHCOUNTER “@gt 60” “id:4444448,phase:2,pause:300,deny,status:429,setenv:RATELIMITED,skip:1,noauditlog”
SecAction “id:4444449,phase:2,pass,setvar:ip.somepathcounter=+1,expirevar:ip.somepathcounter=600,nolog”

Regards, Daniel

Cyril says:

Hi,

Excellent post ! Exactly what i needed !

I am using mod_security 2.8 so i had to add the ‘id’ at the end of each line

Now i am trying to put an additionnal condition ; i want to rate limit POST request. Unfortunately i can’t get it work =(

Does any one can help me out with this ?

Here is what i tried :

SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog,id:22000100
SecAction “phase:5,deprecatevar:ip.ttcounter=1/60,pass,nolog,id:22000101”
SecRule IP:TTCOUNTER “@gt 10” “phase:2,deny,status:403,setenv:RATELIMITED,skip:1,log,id:22000102,chain”
SecRule REQUEST_METHOD “^POST$” “id:22000103”
SecAction “phase:2,pass,setvar:ip.ttcounter=+1,nolog,id:22000104”
Header always set Retry-After “10” env=RATELIMITED

But here is what i get from the apache configtest

AH00526: Syntax error on line 5 of /etc/modsecurity/modsecurity_testing_35_post_throttling.conf:
ModSecurity: Metadata actions (id, rev, msg, tag, severity, ver, accuracy, maturity, logdata) can only be specified by chain starter rules.
Action ‘configtest’ failed.
The Apache error log may have more information.

Thank you for your help

Mike Q says:

Hello, I think you should preface this with all with the a strong urge to turn on ‘limit’ for ufw and create a link to how to set that up (which is much easier). I tried running ab tool with the solution in this article and it worked (but with a different config I poached from github for Ubuntu 16.04 https://gist.github.com/josnidhin/91d1ea9cd71fde386c27a9228476834e) but things did slow down a bit. A quick config setup with ufw (using a script I created) did a better job .

Rafael says:

Hello, I followed this article and everything works very well, except if my *.conf files have some SecRuleRemoveById directive. Everything is related in https://github.com/SpiderLabs/ModSecurity/issues/2325. Do you know why these strategy doesn’t work in this conditions?

Thanks

Leave a Reply to Rafael Cancel reply