Killing and butchering a chicken

Posted on 22/12/2011 by john

At the weekend I killed a chicken, and Louisa and I plucked it and then Louisa butchered it. It was one of a few chicks that turned out to be male, so his fate was to be killed and eaten by us.

This was the second chicken I’ve killed myself (for food or otherwise) – the first one took a bit of mental preparation but this one was a bit easier.

He has a pretty good free range life, was killed quickly and we’ll waste very little of him (we’ve already had a soup made from cooking his carcass in the slowcooker).

Louisa has written up the experience in more detail on her blog.

Here are some photos – you might consider them a little grisly.

 

Posted in Personal, Photography | Tagged , , | Leave a comment

Inside Google Plus

Posted on 12/10/2011 by john

Steven Levy interviewed Google’s Bradley Horowitz about Google+:

Wired: Some users are chafing at Google’s insistence that they provide real names. Explain the policy against pseudonyms.

Horowitz: Google believes in three modes of usage—anonymous, pseudonymous, and identified, and we have a spectrum of products that use all three. For anonymity, you can go into incognito mode in Chrome and the information associated with using the browser is not retained. Gmail and Blogger are pseudonymous—you can go be captainblackjack@gmail.com. But with products like Google Checkout, you’re doing a financial transaction and you have to use your real name.For now, Google+ falls into that last category. There are great debates going on about this—I saw one comment yesterday that claimed that pseudonyms protect the experience of women in the system. I felt compelled to respond, because I’ve gotten feedback from women who say that the accountability of real names makes them feel much more comfortable in Google+.

Notice that Horowitz did not answer the question, and what he did say was just ridiculous nonsense. Steven Levy at Wired didn’t seem to notice, or care.

Horowitz tries to make us think that we need our real name when making a financial transaction.  Thousands of years of currency proves that is not the case.

Horowitz then goes on to blurrily equate making a financial transaction with sharing videos of cats on Google+.

And then the cherry on the top: Google+ protects women.

This was the closest there was to a serious question in the whole interview and Horowitz just laughed out of his arse at it.

Continue reading →

Posted in Politics, Tech | 4 Comments

Ceph at London Devops, 25th July 2011

Posted on 22/7/2011 by john

I’ll be down London way on Monday 25th July giving a talk about Ceph at the London Devops meetup. Come along and learn about petabyte scale distributed filesystems, or just come along and drink beer with us!

Posted in Tech | Tagged , , , , , , , , | 2 Comments

Redirecting outgoing mail with Postfix

Posted on 11/6/2011 by john

We have a various staging deployments of our systems at Brightbox and need to test that the emails they send are correct. We have a bunch of test accounts registered with various email addresses and we wanted them all to go to our dev team, rather than the original recipient.

Rather than write support for this into our apps, we used Postfix to redirect the mail to our devs.

In our case, our staging deployments use a local installation of Postfix and the systems are generally not used by anything else, which makes this dead easy.

Firstly, write a rewrite map file, with the following one line of content. Call it /etc/postfix/recipient_canonical_map:

/./	devteam@example.com

Then configure Postfix like this (in /etc/postfix/main.cf):


recipient_canonical_classes = envelope_recipient
recipient_canonical_maps = regexp:/etc/postfix/recipient_canonical_map

Now all mail going through this relay will be redirected to devteam@example.com. It rewrites only the envelope, so the important headers are not changed.

Posted in GNU/Linux, Tech | Tagged , , , , , , | Leave a comment

Puppet dependencies and run stages

Posted on 29/5/2011 by john

I’m using Puppet to manage some apt repositories on Ubuntu and have had a dependency problem. I want to write the source configs before running apt-get update and I want to run that before installing any packages.  Otherwise, a manifests that tries to install a package from a custom repository will fail, either because the repository is not configured or the apt metadata hasn’t been retrieved yet.

Due to Puppet changes being idempotent, this is usually solvable by running puppet a few times (ew). Or you can do this properly by diligently setting all the dependencies for all of your packages on your apt-get update command, and having that depend on your source configs, but that’s pretty fiddly.

Continue reading →

Posted in Tech | Tagged , , , , | 5 Comments

Indexing syslog messages with solr

Posted on 3/3/2011 by john

I’ve been thinking about centralized indexing and searching of logs for a while and the other day I came across a project called Graylog2 that does just that. It provides a service to receive messages over the network (in a couple of formats, including syslog) and writes them into mongodb. It then has a rails application that lets you browse and search the logs.

It’s neat but I wasn’t quite happy with the search options – I’ve always thought logs should be indexed with a real full text indexer. So, I knocked up a couple of scripts to do just that, as a proof of concept.

It uses rsyslog to receive the messages and write them to a named pipe.  A small ruby script called rsyslog-solr reads from the other end of the pipe and writes batches of the incoming messages to the full text indexer. I chose solr as the full text indexer as it has some very good options for scaling up, which will be necessary when indexing lots of logs.

Solr indexes, compresses and stores the messages sent to it, so we can retrieve the full text without having to store the original log. I wrote a custom schema definition optimized for this.

Then another script, rsyslog-solr-search, is used to query Solr and display the matching messages.

Querying is fun, for example I’ve searched all ssh authentication failures across all hosts and then searched on the originating IPs to see what other probes they made.

You don’t have to do advanced searches though, you can just display all logs from the last hour, or day or whatever.

One important note, any user that can generate logs that are sent to the system can cause a denial of service attack by sending specially malformed messages. This can be fixed by moving the formatting of the log entries from rsyslog into the ruby script, but I’ve not done it yet.

I’ve pushed the code to github under the MIT license. Feel free to improve it.

Posted in Tech | Tagged , , , , , , , , , , | 2 Comments

Breaking my blog embargo

Posted on 19/2/2011 by john

I often find myself in the situation where I’ve not blogged for a long time which makes it difficult to write a new one, as if the long delay means the next blog has to be weighty and impressive.

I’ve realised this is an illusion. Nobody is sitting in wait for my next blog entry. Nobody else has noticed I’ve not blogged in ages. Only I know I’ve not blogged in ages. And I certainly shouldn’t care about what I think.

So I’m breaking my accidental self-imposed blog embargo with this mundane entry which says nothing of import.

If you were hoping for something of more consequence then I make no apology, though your hopes disprove my above realisation, which is irksome to say the least.

Posted in Personal | Tagged , | 2 Comments

The cost of free

Posted on 27/9/2010 by john

Helienne Lindvall writes in the Guardian:

Cory Doctorow [will] cost you $25,000 (£15,800) to get him to speak at your conference…

But what does Doctorow speak about? Well, ironically, he’s a proponent of giving away content for free as a business model – and for years he’s been telling the music industry to adapt to it. Am I the only one to see the irony in this?

I don’t see the irony. This is exactly what Doctorow recommends. Give your content away and charge to perform it. Give your music away and charge for your gigs. I bet the content of his slides is creative commons, and I bet the recordings of this talks are creative commons even. But if watching a video of him isn’t enough for you and you want him in person, then you pay for it.

It seems that Helienne Lindvall does not understand even the basic ideas of free culture.

UPDATE: Helienne Lindvall seems to have been misinformed anyway, as per this tweet from Doctorow himself:

@helienne, I’m afraid you were badly misinformed. I don’t have a “booker”, I don’t charge anything like the sum quoted, most talks are free

UPDATE: Doctorow has since written an interesting article rebutting Lindvall.

Posted in Tech | Tagged , , , | Leave a comment

ipq.co: create dns records instantly

Posted on 10/9/2010 by john

ipq.co is a new service I put together to lower the barrier for dns management. It’s the tinyurl of the dns world – provide an IP address and you get a random dns record for it (or you can choose your own, if it’s available).  Looking at other dns management systems, I was surprised this hadn’t been done before (and by how awful most of the dns interfaces are out there!)

I wrote it in Ruby using the Rails 3 framework, with the dns records being served by the PowerDNS MySQL back end (though I’ll likely be switching it to use a custom back end using my powerdns_pipe library for more flexibility).

We’re building a big new cloud system over at Brightbox and we’ve been thinking how to provide convenient dns records for our customers.  We already have some basic integration but the resulting records are quite a mouthful. ipq.co is just a bit of an experiment to explore other ways of solving the problem.  There has already been some discussion over on Hacker News about possible applications (and implications) of the service – I’m interesting in how people will use it.

I’ve got some plans for other features which I’ll be adding over the next few weeks, and then I’ll be selling it to Google for low 7 figures, so watch this space.

UPDATE: Some ipq.co records were used to point at some phising sites and Google blacklisted the entire site.  I’ve requested a delisting with Google but that might take some time.  Any thoughts on how to avoid this in future?  I’m thinking check the IP with some well established  banlists on create (and possibly check them all regularly after that too).

Posted in Tech | Tagged , , , , , | 7 Comments

LVM snapshot performance

Posted on 18/6/2010 by john

The Linux Logical Volume Manager (LVM) supports creating snapshots of logical volumes (LV) using the device mapper. Device mapper implements snapshots using a copy on write system, so whenever you write to either the source LV or the new snapshot LV, a copy is made first.

So a write to a normal LV is just a write, but a write to a snapshotted LV (or an LV snapshot) involves reading the original data, writing it elsewhere and then writing some metadata about it all.

This quite obviously impacts performance, and due to device mapper having a very basic implementation, it is particularly bad.  My tests show synchronous sequential writes to a snapshotted LV are around 90% slower than writes to a normal LV.

Continue reading →

Posted in GNU/Linux, Tech | Tagged , , , , , , , , | 4 Comments

Testing XML with rspec, xpath and libxml

Posted on 6/4/2010 by john

I’m currently working with the virtualization API libvirt which uses XML to represent virtual machines and I’m generating this XML using Ruby.  I’m using rspec to test my code and wanted to test that my output was as I expected.  I started out with rspec-hpricot-matchers which worked fine until I started testing slightly more complex xml, which hpricot wasn’t handling well.

So I wrote a have_xml matcher using the rspec dsl which uses the libxml library to do the testing.  It’s so simple it’s not really worthy of a gem, so here it is (licensed under public domain).  The text check is optional and, to be honest, doesn’t belong here really.  It should be a separate matcher.


require 'libxml'

Spec::Matchers.define :have_xml do |xpath, text|
  match do |body|
    parser = LibXML::XML::Parser.string body
    doc = parser.parse
    nodes = doc.find(xpath)
    nodes.empty?.should be_false
    if text
      nodes.each do |node|
        node.content.should == text
      end
    end
    true
  end

  failure_message_for_should do |body|
    "expected to find xml tag #{xpath} in:\n#{body}"
  end

  failure_message_for_should_not do |response|
    "expected not to find xml tag #{xpath} in:\n#{body}"
  end

  description do
    "have xml tag #{xpath}"
  end
end

So, add that somewhere (usually spec/spec_helper.rb) and use it like this:


it "should include the xen_machine_id" do
  @xml.should have_xml('/domain/name', 'bb-example-001')
end

it "should include the network devices" do
  @xml.should have_xml "/domain/devices/interface[1]/ip[@address='1.2.3.4']"
  @xml.should have_xml "/domain/devices/interface[1]/mac[@address='aa:00:01:02:03:04']"
  @xml.should have_xml "/domain/devices/interface[1]/script[@path='/etc/xen/scripts/vif-bridge']"
  @xml.should have_xml "/domain/devices/interface[1]/source[@bridge='inetbr']"
end
Posted in Ruby, Tech | Tagged , , , , , | 2 Comments

Chat Roulette: Eye Vagina

Posted on 10/3/2010 by john

Chat Roulette is a web site that hooks you up to a random person. It streams their webcam video and audio to you, and your’s to them.  When you’re done, you click next and get another random person. That’s the whole thing.  It’s fun, for a short period of time.

Anyway, whilst holding my webcam to different parts of my body (if you ever use my webcam, wash your hands) I discovered that my eye, on its side, with the right lighting, and right shadows, and bad focus, through a webcam… looks kinda, possibly, a bit like girl bits.

It’s probably fair to say that, for a large proportion of the random strangers on Chat Roulette, the “Next” button is usually clicked in the hope of seeing a girl flashing some part of her body.

Combine these two seemingly unconnected facts together, and you get some of the reactions you see in my Eye Vagina video!  The music is “My Vagina” by NOFX. I edited out roughly 300 people jerking off.  The vid has had more than half a million hits on you tube. I’m expecting my share of their fat advertising profits any day now.

I recorded it using recordmydesktop and edited it using Pitivi (which actually had some very annoying audo sync problems I had to jump through hoops to avoid, which was a shame).

Posted in Personal | Tagged , , , , , | 2 Comments