Category: Security

Heroku tcp session information leakage

The Linux kernel exposes lots of interesting information via the /proc filesystem. For example, /proc/net/tcp and /proc/net/udp expose information about all tcp and udp sessions on the server.

In the usual Linux kernel style, these files are freely readable by all users on the Linux system. This becomes a bit of a problem on a multi-tenant system like Heroku. Heroku deploy many customers apps on each of their Amazon EC2 servers. So if you create an app skeleton, deploy to Heroku and shell out and you can see all the other apps tcp and udp sessions:


$ heroku run console
Running `console` attached to terminal... up, run.1971
irb(main):001:0> system("netstat | grep ESTAB | head -n 10")
tcp 0 0 22d87d80-deb7-415:33113 ip-10-40-86-97.ec:27018 ESTABLISHED
tcp 0 0 22d87d80-deb7-415:57256 ip-10-60-122:postgresql ESTABLISHED
tcp 0 0 22d87d80-deb7-415:59268 domU-12-31-3:postgresql ESTABLISHED
tcp 0 0 22d87d80-deb7-415:38536 collector6.newrelic:www ESTABLISHED
tcp 0 0 22d87d80-deb7-415:54459 ip-10-189-243-92.:27017 ESTABLISHED
tcp 0 0 22d87d80-deb7-415:50495 ip-10-151-25-11.ec:6242 ESTABLISHED
tcp 0 0 22d87d80-deb7-415:53192 ip-10-114-25:postgresql ESTABLISHED
tcp 0 0 22d87d80-deb7-415:47405 72.21.215.154:https ESTABLISHED
tcp 0 0 22d87d80-deb7-415:39011 collector6.newrelic:www ESTABLISHED
tcp 0 0 22d87d80-deb7-415:52237 ip-10-218-31-147.:42002 ESTABLISHED
=> true
irb(main):002:0>

If the netstat command (or system method) is not available, you can just read and parse the file in Ruby.

It’s an information leak, so I think it is quite low risk but still a hint that Heroku’s process separation still isn’t as strict as one might hope (I still remember David Chen’s pretty shocking discovery back in 2011)

There are a few ways to fix this kind of problem. Mandatory access control systems, like AppArmor can prevent processes reading these files. The Grsecurity security patches have lots of protections against proc based information leaking too, these files included.

Heroku’s response

I reported my findings to Heroku’s security team back in September 2012. I had some problems getting timely responses at first but after whining on Twitter (in March 2013) I got lots of attention from them (and an apology for lack of response). Complaining on Twitter is a clearly a powerful tool and to be used only wisely; and maybe sometimes when drunk.

Anyway, it seems they had been working hard on this and just failing to update me; it is quite a major change for them and a low priority bug imo. They’ve fixed it by properly virtualising networking too. It’s mentioned on their change log but it doesn’t go into too much detail. Important point is that it’s fixed.

1/5/2013 Add Comment

Rate limiting with Apache and mod-security

Rate limiting by request in Apache isn’t easy, but I finally figured out a satisfactory way of doing it using the mod-security Apache module. We’re using it at Brightbox to prevent buggy scripts rinsing our metadata service. In particular, we needed th e ability to allow a high burst of initial requests, as that’s our normal usage pattern. So here’s how to do it.

Install mod-security (on Debian/Ubuntu, just install the libapache2-modsecurity package) and configure it in your virtual host definition like this:

SecRuleEngine On

<LocationMatch "^/somepath">
  SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog
  SecAction "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog"
  SecRule IP:SOMEPATHCOUNTER "@gt 60" "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog"
  SecAction "phase:2,pass,setvar:ip.somepathcounter=+1,nolog"
  Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>

ErrorDocument 509 "Rate Limit Exceeded"

(more…)

15/5/2012 15 Comments

Segfault in Ruby Ferret query parser

Whilst working with the Ruby text search engine library Ferret, I came across a segfault in the query parser. It had already been reported and fixed, but I realised it can lead to a denial of service.

If you use Ferret anywhere that allows users to execute queries, those users can crash the Ruby process with a specially crafted query.  This was quite serious for a number of my sites (not to mention slowing development of a current app) so I applied the fix to the released 0.11.4 source and repackaged it as 0.11.4.1.

Obviously this isn’t in any way official, but it works for me and I’m sharing here for anyone else affected. Gem, tgz and zip here and just the patch available here (derived from the author’s changeset to trunk).

The patch is against the release source, as the subversion repository seems to be down atm (I got the changeset from the web bases subversion viewer).

Get upgrading!

13/9/2007 Add Comment

Encrypted partitions with Ubuntu/Debian

I figured out how to set up an encrypted partition on Ubuntu the other day. There are a bunch of ways of doing it but I found this to be the simplest. It should work on Debian too, since all the relevant packages are Debian ones anyway. In my example I’m encrypting an LVM partition (logical volume), but it should work with any device, including removable USB keys (see end notes). UPDATE: This is broken in Edgy but I figured out a simple fix, see below.

(more…)

6/12/2006 17 Comments

Opt-out of centralised NHS records

The government are centralising our medical information onto something called the “NHS Spine”. So our entire NHS medical histories will be moved to this system opening it up to general access for millions more employees of:

  • various government agencies including the police and social workers
  • private investigators, media organisations and other commercial entities.

Well, you apparently have the legal right to opt out of this “data rape”:

In June 2005, FIPR developed an opt-out letter to send to the Secretary of State. People who sent this off have been fobbed off. We now recommend that you opt out via your GP. Ask your GP to enter into your record the code 93C3 (“refused consent for upload to national shared electronic record”). You can also ask for your address and phone number to be kept off the NHS internal directory, and for your hospital records also to not be uploaded to central systems: see here for details. We encourage you to opt out even if you have nothing to hide; if only people who do have something embarrassing in their records opt out, then doing so will carry a stigma.

4/11/2006 Add Comment

Referrer Securer

Did you know that Firefox (and Epiphany) don’t send referrers when following a link from an SSL encrypted site? The target site cannot tell whether you clicked a link or typed the url in directly.

I don’t know about other browsers, but this seems like a sane behaviour.

16/8/2006 1 Comment

Windows popup spam

Whilst closely watching the traffic to a server here at work (I had a good reason, I don’t just find it fun) (yes I do) I noticed a firewall batting away a bunch of incoming Microsoft Messenger Service NetrSendMessage. These are UDP packets destined for port 1026. The contents of the messages seem to be spyware and spam tricks. “Your system needs updating, click here to purchase the patch” etc.etc.

I’ve not come across this before, but it seems to be wide spread. In 2 hours I collected over a dozen to one particular host, all from different source IPs and nearly all with different messages and urls in them. Here are some excerpts, notice that as URLs aren’t clickable in message boxes they have to leave instructions to type the url in.

UPDATE: For all the non-techies, these messages are NOT the result of a virus or worm or anything like that. They are just network messages sent over the internet by scammers, a bit like spam. You can safely ignore them. If you want them to go away, install some firewall software or follow the instructions by Manimo to turn off the messaging service.

(more…)

22/9/2005 11 Comments

Black Hat, Amsterdam

I leave for Amsterdam on Wednesday where I’m attending the Black Hat Briefings. I was at DefCon in Las Vegas a few years ago so I’m interested to see what the BHB are like in comparison. I hope it’s not just a big ugly advertis-a-thon. I’m there for a few days courtesy of work and will have photies to post when I get back I expect.

My new Laptop arrived today too (not got it in my hands though). The ickle IBM Thinkpad X40 is very portable, but I’ve been using it for more of a desktop replacement than a portable troubleshooter, hence the new Viao one. Big 17inch widescreen LCD, crazy CPUness (for Doom3 and Half Life 2 fun), and 1G RAM. I expect it’ll weigh more than two Terri Schiavos* but I’m a big guy.

* – Please note: topical reference.

29/3/2005 Add Comment

grsecurity and selinux

I’m playing with the grsecurity patches for Linux. Unfortunately 2.6.8 changed in a way that causes major headache for the grsec team, so no planned release date for a new patch. Having some problems with strange enforcements of rlimits, potentially linked to the rlimit auditing code. I’ll hopefully get time to tinker with SELinux too.

28/9/2004 Add Comment

air gap switch security

Whale communications have invented something very secure, and very special. To the naked non-technical eye, their marketing material seems misleading and misguided, but this is the state of the art of security technology. It does some stuff to ensure undefined things do or do not happen.

Quote: “The patent-pending air gap switch keeps sensitive systems and data physically disconnected from untrusted networks and users, and transfers application-level data in real time. It is a high-speed, solid-state analog switch that connects a 512K memory bank to one SBC at a time via a SCSI interface. The air gap switch contains no Operating System, no TCP/IP address, no programmable units, all of which protects the appliance from being compromised. It hides internal addresses, preventing hackers’ mapping of internal network and any tunnelling threat. It protects confidential information such as private keys and configuration data by placing them behind the “air gap.”

It’ll also apparently cure all known ailments, gives you a full head of hair, and a long life free of pain.

7/7/2003 1 Comment

horribly ported

My port of that pptpd exploit to Linux was apparently so horrendous that it prompted ‘r4nc0rwh0r3’ of ‘blightninjas’ to take the time to do it properly. In my defence, the original code really sucked, and I myself only needed the testing part to work (which seemed to work for me). It also compiled fine for me with gcc 3.2.3 (worksforme(tm)). And I in no way proclaim myself to be a good C programmer! Anyway, my laziness and lameness was thoroughly ridiculed by them here. Find their own fixed version here. At least I got my name on Bugtraq. Roll on fame and the big dollar.

26/4/2003 Add Comment

PPTP vulnerability

A buffer overflow vulnerability has been found in the PoPToP PPTP server. The daemon is commonly run as root, therefore this can be pretty serious. A sample exploit was released for windows, and I’ve ported it to Linux (gcc). The problem on our own firewalls was mitigated because, although we have to run the crappy software to provide simple windows VPN access, we have some stack protection in our kernels.

22/4/2003 Add Comment