Install mod-security (on Debian/Ubuntu, just install the libapache2-modsecurity package) and configure it in your virtual host definition like this:

SecRuleEngine On

<LocationMatch "^/somepath">
  SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog
  SecAction "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog"
  SecRule IP:SOMEPATHCOUNTER "@gt 60" "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog"
  SecAction "phase:2,pass,setvar:ip.somepathcounter=+1,nolog"
  Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>

ErrorDocument 509 "Rate Limit Exceeded"

This does a few things and has a couple of knobs for you to tweak depending on your requirements. The first SecAction initializes the state, in this case by IP address. You can do this by a cookie if you like, but I needed it done by IP address (if you’re using a reverse proxy of some kind then get the IP from the X-Forwarded-For header here instead).

The second SecAction deprecates the counter by 1 every 1 second. This is setting the base rate of our rate limit, one per second. I’ve named the counter somepathcounter here, feel free to call it what you want and use different names for different rate limiting different parts of your site.

The SecRule checks to see if the counter is greather than 60 and if so it sleeps 300ms, sets the RATELIMITED environment variable and returns a 509 code response. This is setting the burst rate of our rate limit, to 60. Any IP can do a burst of 60 requests as fast as it likes and it then limited to 1 per second. If it makes no further requests for 60 seconds then the counter is decremented back to 0, which means their burst has been fully recharged.

The last SecAction increments the counter for every successful request (the previous SecRule skips this line if the request was rate limited).

Then the Header definition ensures a header is set whenever a request is rate limited, giving a hint to the client that they shouldn’t try again for 10 seconds. This is obviously just a guide and a lot of clients don’t implement it (and it’s really only valid on a 503 status anyway) so it’s a little bit of wishful thinking really.

Then we define a neat ErrorDocument for the 509 response to give a better clue to the client about what is happening.

509 HTTP Response

I chose to use the 509 HTTP code which isn’t a standard (it’s Apache’s own “Bandwidth Limit Exceeded” code). Technically a 503 (with the Retry-After header) is perhaps a better choice but I already use 503 for maintenance pages and I wanted to differentiate between the two responses in the logs. Twitter invented their own status 420 “Enhance Your Calm” code, but that appears to mostly be an elaborate marijuana reference. RFC 6586 defines 429 “Too Many Requests” for this purpose but it’s very new and Apache won’t let you set an ErrorDocument for it yet. And it’s debatable whether this is a 4xx error mode or a 5xx error mode.

mod-security state data

mod-security needs to store the rate limit state between restarts so you need to tell it where to write that data. I create a directory in /var/lib/mod-security but you can just stick it in /tmp if you like:

SecDataDir /tmp

If you don’t define this, mod-security just seems to silently not apply the rate limiting, so you need it even if you don’t care about state between restarts.

It doesn’t appear possible to store the mod-security state in a shared database so you can’t rate limit accurately when you have multiple load balanced web servers.

Sleeping considered bad

This implementation sleeps 300ms per request when the rate has been exceeded but sleeping here is generally not a great idea. It ties up the Apache worker for the duration of the sleep, so whilst it will relieve the load on your backend app (which would otherwise likely use lots of CPU) it arguably makes it easier to tie up all of Apache’s workers and take your site offline.

There are easier ways to tie up a web server though, and without a sleep many clients will just immediately retry the request, spamming the logs with useless messages and using HTTP parsing cpu etc. And without the rate limiting at all, your app might be even slower and could tie things up just as easily. So feel free to tune this as you like.

Basically, remember that this is not a malicious denial of service protection system – we’re just using it enforce a basic policy, most usually breached due to a mistake rather than an attack.

Other options

mod_cband sounds pretty good but I can’t find documentation on how to rate limit requests, just bandwidth, and it’s not packaged for Debian/Ubuntu.

mod_evasive sounds like something you might want to use for DoS protection.

If you’re using NGINX then the HTTP limit req module sounds nice and simple.

Netfilter‘s hashlimit matcher is very powerful and has the benefit of being in a layer before Apache entirely, so you can save even more CPU cycles. Netfilter also has the ability to rate limit logging too, so you won’t fill your disk with useless logs. If you want to return a 509 response, instead of just dropping or rejecting the SYN packet, then you can redirect rate limited connections to an Apache virtual host that just returns 509s (put it on a different port and use the DNAT Netfilter target).

Rather than re-implement the wheel, riak-syslog expects that a syslog daemon will handle receiving syslog messages and will be able to provide them in a specific format – there is documentation on getting this running with rsyslog on Ubuntu.

I’ve used it to gather and store a few hundred gig of syslogs over the last several months on an small internal Riak cluster on Brightbox Cloud and it’s working well (which can’t be said of a similar setup I did with Solr which caved in after a while and needed some fine tuning!)

There is documentation on getting it set up in the README, and some examples of how to conduct searches too.

If you want to play with Riak, you can build a four node cluster spanning two data-centres in five minutes on Brightbox Cloud.

You might also be interested in my post about indexing syslog messages with Solr.

Understandably tired after a long day of sleeping.

let’s assume you have a client called foo and a server called bar

or command examples like:

mysqldump -h server1 | mysql -h server2

When I write documentation, I prefer to tell a story. What is the client called, Steven? Are we taking a mysqldump of a production server and writing it to a staging server?

Human brains like stories. It’s much easier to keep track of facts if they have some kind of meaning. Many memory improvement techniques use stories to link things together. And when you’re reading documentation, you’re usually learning some new concept anyway – so you’re adding unnecessary cognitive load by using abstract labels like Foo and Bar or A and B.

In my Git submodules post I name the two example projects your_project and other_project and use it consistently throughout. You never have to rememeber whether “Foo” is the remote project or not.

One of my own favourites is an old heartbeat cluster guide I wrote. It involves two clusters, each of which consisted of two servers working together. I named the first cluster JuliusCaesar, naming the two nodes Julius and Caesar. The second cluster is called MarcusAurelius. Throughout the documentation, I’m able to refer to any server just by it’s name and you can know where it is in the network.

It’s part of why I like using rspec to do testing, because it encourages you to tell a story rather than to just test arbitrary values.

So, put some thought into your examples. Tell a story. Make it easier for the reader to keep track of all this new stuff they’re learning.

This was the second chicken I’ve killed myself (for food or otherwise) – the first one took a bit of mental preparation but this one was a bit easier.

He has a pretty good free range life, was killed quickly and we’ll waste very little of him (we’ve already had a soup made from cooking his carcass in the slowcooker).

Louisa has written up the experience in more detail on her blog.

Here are some photos – you might consider them a little grisly.

Wired: Some users are chafing at Google’s insistence that they provide real names. Explain the policy against pseudonyms.

Horowitz: Google believes in three modes of usage—anonymous, pseudonymous, and identified, and we have a spectrum of products that use all three. For anonymity, you can go into incognito mode in Chrome and the information associated with using the browser is not retained. Gmail and Blogger are pseudonymous—you can go be captainblackjack@gmail.com. But with products like Google Checkout, you’re doing a financial transaction and you have to use your real name.For now, Google+ falls into that last category. There are great debates going on about this—I saw one comment yesterday that claimed that pseudonyms protect the experience of women in the system. I felt compelled to respond, because I’ve gotten feedback from women who say that the accountability of real names makes them feel much more comfortable in Google+.

Notice that Horowitz did not answer the question, and what he did say was just ridiculous nonsense. Steven Levy at Wired didn’t seem to notice, or care.

Horowitz tries to make us think that we need our real name when making a financial transaction.  Thousands of years of currency proves that is not the case.

Horowitz then goes on to blurrily equate making a financial transaction with sharing videos of cats on Google+.

And then the cherry on the top: Google+ protects women.

This was the closest there was to a serious question in the whole interview and Horowitz just laughed out of his arse at it.

I emailed Steven Levy and asked him why, given the opportunity of interviewing Horowitz, he didn’t ask anything close to a serious question. It’s been over a week and I’ve had no response.

TL;DR: Don’t trust Steven Levy to report honestly about Google.

Hi Steven,

I hope you're well.

I read your article "Inside Google Plus"[1] the other day and I have a
couple of questions I hope you'll have time to answer,

I was surprised to notice that you didn't ask any questions about
privacy - Google+ unequivocally raises numerous serious privacy
concerns.  Why did you decide not to put any of these questions to
Horowitz?

Regarding your question about pseudonyms, Horowitz didn't really answer
your question. He kind of claimed that financial transactions cannot be
anonymous and that Google+ is somehow like a financial transaction.  To
be frank, his answer seemed just hand waving - why didn't you challenge
him on this harder?

Best regards,

John Leach

[1] http://www.wired.com/magazine/2011/09/ff_google_horowitz/

Rather than write support for this into our apps, we used Postfix to redirect the mail to our devs.

In our case, our staging deployments use a local installation of Postfix and the systems are generally not used by anything else, which makes this dead easy.

Firstly, write a rewrite map file, with the following one line of content. Call it /etc/postfix/recipient_canonical_map:

/./ devteam@example.com 

Then configure Postfix like this (in /etc/postfix/main.cf):

 recipient_canonical_classes = envelope_recipient recipient_canonical_maps = regexp:/etc/postfix/recipient_canonical_map 

Now all mail going through this relay will be redirected to devteam@example.com. It rewrites only the envelope, so the important headers are not changed.

Due to Puppet changes being idempotent, this is usually solvable by running puppet a few times (ew). Or you can do this properly by diligently setting all the dependencies for all of your packages on your apt-get update command, and having that depend on your source configs, but that’s pretty fiddly.

New versions of Puppet now have a feature called run stages that can be used to solve this problem. You put your source configs and apt update command in a run stage and tell puppet to run everything in that run stage before the other stages (there is an implicit run stage called main where everything goes by default).

This seems quite neat at first, but only the new parameterized classes can be set to be in certain run stages – so you end up putting things into a class just to put it into a run stage. It’s really not much better than being diligent with your dependencies for this problem – worse in many ways. (There is a long discussion about the implementation of run stages in the Puppet issue tracker that might help you understand the use case for them).

But there is a new relationship syntax to make setting dependencies much easier which I’m using to mass-depend all packages on my apt-get update command:

class apt {
  exec { "apt-update":
    command => "/usr/bin/apt-get update"
  }

  # Ensure apt is setup before running apt-get update
  Apt::Key <| |> -> Exec["apt-update"]
  Apt::Source <| |> -> Exec["apt-update"]

  # Ensure apt-get update has been run before installing any packages
  Exec["apt-update"] -> Package <| |>
}

Remember, Puppet is declarative – the dependencies set there get applied to all Apt::Key, Apt::Source and Package resources even if they’ve not yet been defined. (This example also assumes the only packages you’re defining are apt ones btw!)

In summary, run stages are hard and fiddly. You probably don’t need them. Learn how to use the new relationship syntax.

It’s neat but I wasn’t quite happy with the search options – I’ve always thought logs should be indexed with a real full text indexer. So, I knocked up a couple of scripts to do just that, as a proof of concept.

It uses rsyslog to receive the messages and write them to a named pipe.  A small ruby script called rsyslog-solr reads from the other end of the pipe and writes batches of the incoming messages to the full text indexer. I chose solr as the full text indexer as it has some very good options for scaling up, which will be necessary when indexing lots of logs.

Solr indexes, compresses and stores the messages sent to it, so we can retrieve the full text without having to store the original log. I wrote a custom schema definition optimized for this.

Then another script, rsyslog-solr-search, is used to query Solr and display the matching messages.

Querying is fun, for example I’ve searched all ssh authentication failures across all hosts and then searched on the originating IPs to see what other probes they made.

You don’t have to do advanced searches though, you can just display all logs from the last hour, or day or whatever.

One important note, any user that can generate logs that are sent to the system can cause a denial of service attack by sending specially malformed messages. This can be fixed by moving the formatting of the log entries from rsyslog into the ruby script, but I’ve not done it yet.

I’ve pushed the code to github under the MIT license. Feel free to improve it.

I’ve realised this is an illusion. Nobody is sitting in wait for my next blog entry. Nobody else has noticed I’ve not blogged in ages. Only I know I’ve not blogged in ages. And I certainly shouldn’t care about what I think.

So I’m breaking my accidental self-imposed blog embargo with this mundane entry which says nothing of import.

If you were hoping for something of more consequence then I make no apology, though your hopes disprove my above realisation, which is irksome to say the least.

Cory Doctorow [will] cost you $25,000 (£15,800) to get him to speak at your conference…

But what does Doctorow speak about? Well, ironically, he’s a proponent of giving away content for free as a business model – and for years he’s been telling the music industry to adapt to it. Am I the only one to see the irony in this?

I don’t see the irony. This is exactly what Doctorow recommends. Give your content away and charge to perform it. Give your music away and charge for your gigs. I bet the content of his slides is creative commons, and I bet the recordings of this talks are creative commons even. But if watching a video of him isn’t enough for you and you want him in person, then you pay for it.

It seems that Helienne Lindvall does not understand even the basic ideas of free culture.

UPDATE: Helienne Lindvall seems to have been misinformed anyway, as per this tweet from Doctorow himself:

@helienne, I’m afraid you were badly misinformed. I don’t have a “booker”, I don’t charge anything like the sum quoted, most talks are free

UPDATE: Doctorow has since written an interesting article rebutting Lindvall.

I wrote it in Ruby using the Rails 3 framework, with the dns records being served by the PowerDNS MySQL back end (though I’ll likely be switching it to use a custom back end using my powerdns_pipe library for more flexibility).

We’re building a big new cloud system over at Brightbox and we’ve been thinking how to provide convenient dns records for our customers.  We already have some basic integration but the resulting records are quite a mouthful. ipq.co is just a bit of an experiment to explore other ways of solving the problem.  There has already been some discussion over on Hacker News about possible applications (and implications) of the service – I’m interesting in how people will use it.

I’ve got some plans for other features which I’ll be adding over the next few weeks, and then I’ll be selling it to Google for low 7 figures, so watch this space.

UPDATE: Some ipq.co records were used to point at some phising sites and Google blacklisted the entire site.  I’ve requested a delisting with Google but that might take some time.  Any thoughts on how to avoid this in future?  I’m thinking check the IP with some well established  banlists on create (and possibly check them all regularly after that too).

So a write to a normal LV is just a write, but a write to a snapshotted LV (or an LV snapshot) involves reading the original data, writing it elsewhere and then writing some metadata about it all.

This quite obviously impacts performance, and due to device mapper having a very basic implementation, it is particularly bad.  My tests show synchronous sequential writes to a snapshotted LV are around 90% slower than writes to a normal LV.

Once copied and written, writes to the same chunk are only 15% slower.  Reads are super fast, only a 5% speed impact.

Still, not many usage patterns involve huge full speed sequential writes to a filesystem, so LVM is still useful in most circumstances.

I did some tests to see how writes to one snapshotted LV impacted the performance of writes to a completely separate normal LV. Does a snapshotted LV ruin the performance of all your other LVs? Yes, especially if you’re using the cfq disk scheduler. Switching to the deadline scheduler made things considerably better for the normal LV (but slowed writes to the snapshotted LV a little further).

I did these tests on a 12 disk hardware RAID10 system. The test is a synthetic benchmark so I urge you to do your own tests, but it’s safe to say that device mapper does not implement clever snapshotting like btrfs or zfs – don’t expect great performance from it.

Improving LVM Snapshot performance

There are a few ways to improve performance of LVM snapshots.  The most obvious one is the chunk size, which can be tweaked when creating the snapshot.  This controls the size of the data that will be copied and written on write operations.  The best setting will depend on lots of stuff, such as your RAID stripe size and your usage patterns.

There is an as-yet uncommitted patch that improves snapshot write performance a bit by being a bit clever about the disk queuing, but it’s still slow.

Also, device mapper supports non-persistent snapshots (i.e: lost after reboot), which should avoid having to write the change metadata to disk (which will save a lot of seeks and writes) but LVM doesn’t seem to support creating these yet.

Putting the snapshot device on a separate disk would help too – I’m not sure it’s possible with LVM, but device mapper does support it.

So I wrote a have_xml matcher using the rspec dsl which uses the libxml library to do the testing.  It’s so simple it’s not really worthy of a gem, so here it is (licensed under public domain).  The text check is optional and, to be honest, doesn’t belong here really.  It should be a separate matcher.

require 'libxml'

Spec::Matchers.define :have_xml do |xpath, text|
  match do |body|
    parser = LibXML::XML::Parser.string body
    doc = parser.parse
    nodes = doc.find(xpath)
    nodes.empty?.should be_false
    if text
      nodes.each do |node|
        node.content.should == text
      end
    end
    true
  end

  failure_message_for_should do |body|
    "expected to find xml tag #{xpath} in:\n#{body}"
  end

  failure_message_for_should_not do |response|
    "expected not to find xml tag #{xpath} in:\n#{body}"
  end

  description do
    "have xml tag #{xpath}"
  end
end

So, add that somewhere (usually spec/spec_helper.rb) and use it like this:

it "should include the xen_machine_id" do
  @xml.should have_xml('/domain/name', 'bb-example-001')
end

it "should include the network devices" do
  @xml.should have_xml "/domain/devices/interface[1]/ip[@address='1.2.3.4']"
  @xml.should have_xml "/domain/devices/interface[1]/mac[@address='aa:00:01:02:03:04']"
  @xml.should have_xml "/domain/devices/interface[1]/script[@path='/etc/xen/scripts/vif-bridge']"
  @xml.should have_xml "/domain/devices/interface[1]/source[@bridge='inetbr']"
end

Anyway, whilst holding my webcam to different parts of my body (if you ever use my webcam, wash your hands) I discovered that my eye, on its side, with the right lighting, and right shadows, and bad focus, through a webcam… looks kinda, possibly, a bit like girl bits.

It’s probably fair to say that, for a large proportion of the random strangers on Chat Roulette, the “Next” button is usually clicked in the hope of seeing a girl flashing some part of her body.

Combine these two seemingly unconnected facts together, and you get some of the reactions you see in my Eye Vagina video!  The music is “My Vagina” by NOFX. I edited out roughly 300 people jerking off.  The vid has had more than half a million hits on you tube. I’m expecting my share of their fat advertising profits any day now.

I recorded it using recordmydesktop and edited it using Pitivi (which actually had some very annoying audo sync problems I had to jump through hoops to avoid, which was a shame).

Funding your content through advertising is hugely inefficient. Of the people who visit your site, usually only a tiny proportion click on (or notice) an advert, and only a tiny proportion of those then spends any money.  So a tiny, tiny proportion of your visitors give any money to your advertisers. So money filters down this system in tiny margins.  Then, at the bottom of the system, a tiny amount of the profits from the income covers the cost of advertising.  Then this money moves back up the system to you, usually via your advertising agent who takes a nice cut (I’ve heard Google pass as little as one twelfth onto the publisher in some cases).

And this doesn’t consider the costs of the advertiser choosing and designing the ad or the tonnes of bandwidth and gatrillions of CPU cycles used to serve the actual adverts.

It also does not consider externalities, such as pollution. Advertising is mind pollution. Advertising is designed to affect the behaviour of people for the benefit of the advertiser.  Why would anyone willingly expose themselves to something designed to steal their attention?

You might argue that advertising creates value – some viewers choose to buy when otherwise they wouldn’t have. But what of the huge proportion of people who just had their attention stolen? No value was created there.

Because not everyone is suckered in by it, advertising squanders billions of hours of attention every day to produce nothing.

Begging

Walking down a street in town I might get approached by a beggar who needs money to eat.  In the UK we have the Vagrancy Act of 1824 which prevents these people from harassing me. They can be punished by detention, hard labour and whipping apparently.

Advertising is just a company harassing me for money to eat.  They’re just better funded.  I believe we should be able to detain and whip their advertising departments.

Unobtrusive payment

But seriously, advertising is a broken method of paying for stuff.  If we could unobtrusively pay for content on the Internet, I’m sure enough people would do so to more than cover costs of production.

We need good, unobtrusive payment methods and a change in culture to pay for good content would follow.  We can work on changing the culture now though: support sites that are ad-free (or have ad-free subscriptions).  I pay for a couple of ad-free subscriptions myself – be the change you want to see in the world.

I look forward to the day when a site with advertising is a clear signal that nobody would pay for it otherwise. My ad-blocker could then just block the whole site to save my wasting my attention.

Full Disclosure

My company does a (very small) amount of advertising, though I’m not involved directly in it. I don’t know how well it performs. We also sponsor conferences and events, which is of course advertising. I also help run a couple of web sites that make money from advertising.

From a viewer’s perspective, I hate advertising. From a publisher’s perspective, I can make a few quid from it and kinda just hope the pollution isn’t so bad (we do not allow annoying flashing adverts, and block ads from particularly evil corporations whenever we can but frankly, I’m mostly getting by on cognitive dissonance).  I’ve not thought about it until now, but from an advertisers perspective, it’s of course nice to get new customers (though I’m not sure of the “quality” of the custom we get via advertising – I’m now interested in investigating this further).

I’m in no way dependent on any of my income from advertising, so it’s hard to speak from these perspectives.

Update: Poor people

I’m basically saying that because adverts are not well targetted, the majority of advert views are wasted. They’re mind pollution.

But in order for adverts to get more accurate, the ad companies need to collect personal information about us: what we do online, what we like etc. So we’re supposed to hand over our privacy, just so we can ethically view “free” stuff on the Internet?

Let’s suppose advertising becomes perfectly targeted. Every advert you see is something you really can’t do without and something you can afford. Wouldn’t this mean you buy everything you get shown? Wouldn’t this mean you’d run out of money?

Is it unethical for poor people to view ad-supported online content if they can’t afford anything being advertised? However well targeted the ads are, they have no money to spend so it’s completely fruitless. Perhaps ad supported websites should ban public library Internet addresses – poor people are reading for free!

Discussions Elsewhere

• Rob Sayre: Why adblockers work
• Brian Carpet: Advertising is devastating to my well being
• Comments on Hacker News

Being a self-confessed full text indexing nerd and a Ruby-lover, I wrote Xapian Fu: a library to provide access to Xapian that is more in line with “The Ruby Way”.

I started writing Xapian Fu exactly a year ago today but left it for a couple of months, then restarted work on it on the train on the way back from the 2009 Scotland on Rails conference.  Development was test driven, so it’s got an extensive test suite (using rspec).  Documentation is in rdoc and is quite detailed.  As of the latest version, it supports Ruby 1.9 too.

Xapian Fu basically gives you a Hash interface to Xapian – so you get a persistent Hash with full text indexing built in (and ACID transactions!).

Example

For example, create a database called example.db, put three documents into it and search them and print the results:

  require 'xapian-fu'
  include XapianFu
  db = XapianDb.new(:dir => 'example.db', :create => true,
                    :store => [:title, :year])
  db << { :title => 'Brokeback Mountain', :year => 2005 }
  db << { :title => 'Cold Mountain', :year => 2004 }
  db << { :title => 'Yes Man', :year => 2008 }
  db.flush
  db.search("mountain").each do |match|
    puts match.values[:title]
  end

There are of course a whole bunch more examples in the documentation.

Schema-less

The hard work of full text indexing and storage is of course done by the Xapian library, but I have added a couple of useful features.  One in particular is the ability to use symbols (or strings) as field names. Xapian has no real concept of fields, but you can store arbitrary data that it calls values in a numbered slot alongside each document.  Instead of making you deal with field numbers, Xapian Fu uses a hash function to convert your field names into numbers.  This means Xapian Fu is schema-less - you can add or omit fields whenever you like.  It’s useful to define fields when opening databases so that Xapian Fu can recognise them in searches or to give Xapian Fu some clues on the type of data you’ll be using, but it’s not necessary.

Efficient storage of fields for ordering

If you tell Xapian Fu what type of data you’ll be storing in your fields, it can store them more efficiently.  For example, if you don’t specify the type, integers will be converted to strings as is, so 354,441,945,266,899 becomes “354441945266899″ – that’s  fifteen bytes!  When you tell Xapian Fu that your field is going to be an Integer,  it will store them in double precision floating point format which is 8 bytes and can represent up to about 16 decimal digits.  Also, it’s stored in big-endian format, so Xapian can still use the field for ordering results. XapianFu will store Time objects like this too, so again, it’s size efficient and can be used for ordering results.

Since Xapian Fu now knows what type the field is, it can convert it back when you access it too, so you get an Integer or a Time object (rather than a String, which is how Xapian represents it internally). It currently supports Integer, Fixnum, Bignum (up to a certain size), Float, Time and Date.  You can add your own types easily by decorating your instances with special methods.

Stemming and stopping

Xapian has stemming support for loads of languages (via the Snowball library), but no stop word lists.  Xapian Fu uses the appropriate stemmer when you specify a language for your document or database and comes with stop word lists for 13 languages (also automatically used).  This means Xapian doesn’t have to index these common stop words, so you get faster indexing and search times, a smaller database and more relevant search results.

Xapian Fu also knows that your searches won’t work right unless you stem them too! It automatically stems queries using the database language (though this does fall down a bit if you have different documents with different languages in your database at the moment, but it can be disabled (and isn’t too difficult to add support)).

Will Paginate support

will_paginate is a pagination library for ActiveRecord (and other DB abstraction layers).  It has helpers for drawing page list interfaces.  Xapian Fu supports will_paginate by using the same method names in result sets (such as :current_page and :total_pages).  You can pass a Xapian Fu result set to the will_paginate helpers and you’ll get lovely page list interfaces (you need to handle accepting the parameters in your action and setting up the next search of course!)

Active Record support

Xapian Fu does not yet have an Active Record plugin (I’ll add one soon) but as Xapian Fu uses the :id field as the Xapian primary key by default, it’s trivial to use it in your Rails app right now. See the “ActiveRecord Integration” section of the README for code examples.  In this case, you probably don’t need to actually store any data in the Xapian database, just the index information (and the :id field of course, but that’s stored by default) – so you get a smaller database (though you still need to store fields that you want to group by (collapse) or order results with).

Multi-master replicated full text indexing service

Xapian Fu doesn’t do this. I’m designing something that might though :)

Getting Xapian Fu

It’s available in gem form from Rubyforge/cutter.  The code is on github here.  You’ll need the Xapian Ruby bindings installed – on Debian/Ubuntu this is the libxapian-ruby1.8 package.  The gem named xapian claims to provide Xapian and the Ruby bindings but it failed form me on 64bit.  The gem named xapian-full claims to provide the Ruby bindings without Xapian (you’ll obviously need to build and install Xapian yourself) but I’ve not used that either.  RPMs, source files and other downloads are listed on the Xapian downloads page.

There is also this weird kinda splash page I made, in some kind of attempt to host something about Xapian Fu on my own domain. Not really sure what real purpose it serves.