So a write to a normal LV is just a write, but a write to a snapshotted LV (or an LV snapshot) involves reading the original data, writing it elsewhere and then writing some metadata about it all.

This quite obviously impacts performance, and due to device mapper having a very basic implementation, it is particularly bad.  My tests show synchronous sequential writes to a snapshotted LV are around 90% slower than writes to a normal LV.

Once copied and written, writes to the same chunk are only 15% slower.  Reads are super fast, only a 5% speed impact.

Still, not many usage patterns involve huge full speed sequential writes to a filesystem, so LVM is still useful in most circumstances.

I did some tests to see how writes to one snapshotted LV impacted the performance of writes to a completely separate normal LV. Does a snapshotted LV ruin the performance of all your other LVs? Yes, especially if you’re using the cfq disk scheduler. Switching to the deadline scheduler made things considerably better for the normal LV (but slowed writes to the snapshotted LV a little further).

I did these tests on a 12 disk hardware RAID10 system. The test is a synthetic benchmark so I urge you to do your own tests, but it’s safe to say that device mapper does not implement clever snapshotting like btrfs or zfs – don’t expect great performance from it.

Improving LVM Snapshot performance

There are a few ways to improve performance of LVM snapshots.  The most obvious one is the chunk size, which can be tweaked when creating the snapshot.  This controls the size of the data that will be copied and written on write operations.  The best setting will depend on lots of stuff, such as your RAID stripe size and your usage patterns.

There is an as-yet uncommitted patch that improves snapshot write performance a bit by being a bit clever about the disk queuing, but it’s still slow.

Also, device mapper supports non-persistent snapshots (i.e: lost after reboot), which should avoid having to write the change metadata to disk (which will save a lot of seeks and writes) but LVM doesn’t seem to support creating these yet.

Putting the snapshot device on a separate disk would help too – I’m not sure it’s possible with LVM, but device mapper does support it.

So I wrote a have_xml matcher using the rspec dsl which uses the libxml library to do the testing.  It’s so simple it’s not really worthy of a gem, so here it is (licensed under public domain).  The text check is optional and, to be honest, doesn’t belong here really.  It should be a separate matcher.

require 'libxml'

Spec::Matchers.define :have_xml do |xpath, text|
  match do |body|
    parser = LibXML::XML::Parser.string body
    doc = parser.parse
    nodes = doc.find(xpath)
    nodes.empty?.should be_false
    if text
      nodes.each do |node|
        node.content.should == text
      end
    end
    true
  end

  failure_message_for_should do |body|
    "expected to find xml tag #{xpath} in:\n#{body}"
  end

  failure_message_for_should_not do |response|
    "expected not to find xml tag #{xpath} in:\n#{body}"
  end

  description do
    "have xml tag #{xpath}"
  end
end

So, add that somewhere (usually spec/spec_helper.rb) and use it like this:

it "should include the xen_machine_id" do
  @xml.should have_xml('/domain/name', 'bb-example-001')
end

it "should include the network devices" do
  @xml.should have_xml "/domain/devices/interface[1]/ip[@address='1.2.3.4']"
  @xml.should have_xml "/domain/devices/interface[1]/mac[@address='aa:00:01:02:03:04']"
  @xml.should have_xml "/domain/devices/interface[1]/script[@path='/etc/xen/scripts/vif-bridge']"
  @xml.should have_xml "/domain/devices/interface[1]/source[@bridge='inetbr']"
end

Anyway, whilst holding my webcam to different parts of my body (if you ever use my webcam, wash your hands) I discovered that my eye, on its side, with the right lighting, and right shadows, and bad focus, through a webcam… looks kinda, possibly, a bit like girl bits.

It’s probably fair to say that, for a large proportion of the random strangers on Chat Roulette, the “Next” button is usually clicked in the hope of seeing a girl flashing some part of her body.

Combine these two seemingly unconnected facts together, and you get some of the reactions you see in my Eye Vagina video!  The music is “My Vagina” by NOFX. I edited out roughly 300 people jerking off.  The vid has had more than half a million hits on you tube. I’m expecting my share of their fat advertising profits any day now.

I recorded it using recordmydesktop and edited it using Pitivi (which actually had some very annoying audo sync problems I had to jump through hoops to avoid, which was a shame).

Funding your content through advertising is hugely inefficient. Of the people who visit your site, usually only a tiny proportion click on (or notice) an advert, and only a tiny proportion of those then spends any money.  So a tiny, tiny proportion of your visitors give any money to your advertisers. So money filters down this system in tiny margins.  Then, at the bottom of the system, a tiny amount of the profits from the income covers the cost of advertising.  Then this money moves back up the system to you, usually via your advertising agent who takes a nice cut (I’ve heard Google pass as little as one twelfth onto the publisher in some cases).

And this doesn’t consider the costs of the advertiser choosing and designing the ad or the tonnes of bandwidth and gatrillions of CPU cycles used to serve the actual adverts.

It also does not consider externalities, such as pollution. Advertising is mind pollution. Advertising is designed to affect the behaviour of people for the benefit of the advertiser.  Why would anyone willingly expose themselves to something designed to steal their attention?

You might argue that advertising creates value – some viewers choose to buy when otherwise they wouldn’t have. But what of the huge proportion of people who just had their attention stolen? No value was created there.

Because not everyone is suckered in by it, advertising squanders billions of hours of attention every day to produce nothing.

Begging

Walking down a street in town I might get approached by a beggar who needs money to eat.  In the UK we have the Vagrancy Act of 1824 which prevents these people from harassing me. They can be punished by detention, hard labour and whipping apparently.

Advertising is just a company harassing me for money to eat.  They’re just better funded.  I believe we should be able to detain and whip their advertising departments.

Unobtrusive payment

But seriously, advertising is a broken method of paying for stuff.  If we could unobtrusively pay for content on the Internet, I’m sure enough people would do so to more than cover costs of production.

We need good, unobtrusive payment methods and a change in culture to pay for good content would follow.  We can work on changing the culture now though: support sites that are ad-free (or have ad-free subscriptions).  I pay for a couple of ad-free subscriptions myself – be the change you want to see in the world.

I look forward to the day when a site with advertising is a clear signal that nobody would pay for it otherwise. My ad-blocker could then just block the whole site to save my wasting my attention.

Full Disclosure

My company does a (very small) amount of advertising, though I’m not involved directly in it. I don’t know how well it performs. We also sponsor conferences and events, which is of course advertising. I also help run a couple of web sites that make money from advertising.

From a viewer’s perspective, I hate advertising. From a publisher’s perspective, I can make a few quid from it and kinda just hope the pollution isn’t so bad (we do not allow annoying flashing adverts, and block ads from particularly evil corporations whenever we can but frankly, I’m mostly getting by on cognitive dissonance).  I’ve not thought about it until now, but from an advertisers perspective, it’s of course nice to get new customers (though I’m not sure of the “quality” of the custom we get via advertising – I’m now interested in investigating this further).

I’m in no way dependent on any of my income from advertising, so it’s hard to speak from these perspectives.

Update: Poor people

I’m basically saying that because adverts are not well targetted, the majority of advert views are wasted. They’re mind pollution.

But in order for adverts to get more accurate, the ad companies need to collect personal information about us: what we do online, what we like etc. So we’re supposed to hand over our privacy, just so we can ethically view “free” stuff on the Internet?

Let’s suppose advertising becomes perfectly targeted. Every advert you see is something you really can’t do without and something you can afford. Wouldn’t this mean you buy everything you get shown? Wouldn’t this mean you’d run out of money?

Is it unethical for poor people to view ad-supported online content if they can’t afford anything being advertised? However well targeted the ads are, they have no money to spend so it’s completely fruitless. Perhaps ad supported websites should ban public library Internet addresses – poor people are reading for free!

Discussions Elsewhere

• Rob Sayre: Why adblockers work
• Brian Carpet: Advertising is devastating to my well being
• Comments on Hacker News

Being a self-confessed full text indexing nerd and a Ruby-lover, I wrote Xapian Fu: a library to provide access to Xapian that is more in line with “The Ruby Way”.

I started writing Xapian Fu exactly a year ago today but left it for a couple of months, then restarted work on it on the train on the way back from the 2009 Scotland on Rails conference.  Development was test driven, so it’s got an extensive test suite (using rspec).  Documentation is in rdoc and is quite detailed.  As of the latest version, it supports Ruby 1.9 too.

Xapian Fu basically gives you a Hash interface to Xapian – so you get a persistent Hash with full text indexing built in (and ACID transactions!).

Example

For example, create a database called example.db, put three documents into it and search them and print the results:

  require 'xapian-fu'
  include XapianFu
  db = XapianDb.new(:dir => 'example.db', :create => true,
                    :store => [:title, :year])
  db << { :title => 'Brokeback Mountain', :year => 2005 }
  db << { :title => 'Cold Mountain', :year => 2004 }
  db << { :title => 'Yes Man', :year => 2008 }
  db.flush
  db.search("mountain").each do |match|
    puts match.values[:title]
  end

There are of course a whole bunch more examples in the documentation.

Schema-less

The hard work of full text indexing and storage is of course done by the Xapian library, but I have added a couple of useful features.  One in particular is the ability to use symbols (or strings) as field names. Xapian has no real concept of fields, but you can store arbitrary data that it calls values in a numbered slot alongside each document.  Instead of making you deal with field numbers, Xapian Fu uses a hash function to convert your field names into numbers.  This means Xapian Fu is schema-less - you can add or omit fields whenever you like.  It’s useful to define fields when opening databases so that Xapian Fu can recognise them in searches or to give Xapian Fu some clues on the type of data you’ll be using, but it’s not necessary.

Efficient storage of fields for ordering

If you tell Xapian Fu what type of data you’ll be storing in your fields, it can store them more efficiently.  For example, if you don’t specify the type, integers will be converted to strings as is, so 354,441,945,266,899 becomes “354441945266899″ – that’s  fifteen bytes!  When you tell Xapian Fu that your field is going to be an Integer,  it will store them in double precision floating point format which is 8 bytes and can represent up to about 16 decimal digits.  Also, it’s stored in big-endian format, so Xapian can still use the field for ordering results. XapianFu will store Time objects like this too, so again, it’s size efficient and can be used for ordering results.

Since Xapian Fu now knows what type the field is, it can convert it back when you access it too, so you get an Integer or a Time object (rather than a String, which is how Xapian represents it internally). It currently supports Integer, Fixnum, Bignum (up to a certain size), Float, Time and Date.  You can add your own types easily by decorating your instances with special methods.

Stemming and stopping

Xapian has stemming support for loads of languages (via the Snowball library), but no stop word lists.  Xapian Fu uses the appropriate stemmer when you specify a language for your document or database and comes with stop word lists for 13 languages (also automatically used).  This means Xapian doesn’t have to index these common stop words, so you get faster indexing and search times, a smaller database and more relevant search results.

Xapian Fu also knows that your searches won’t work right unless you stem them too! It automatically stems queries using the database language (though this does fall down a bit if you have different documents with different languages in your database at the moment, but it can be disabled (and isn’t too difficult to add support)).

Will Paginate support

will_paginate is a pagination library for ActiveRecord (and other DB abstraction layers).  It has helpers for drawing page list interfaces.  Xapian Fu supports will_paginate by using the same method names in result sets (such as :current_page and :total_pages).  You can pass a Xapian Fu result set to the will_paginate helpers and you’ll get lovely page list interfaces (you need to handle accepting the parameters in your action and setting up the next search of course!)

Active Record support

Xapian Fu does not yet have an Active Record plugin (I’ll add one soon) but as Xapian Fu uses the :id field as the Xapian primary key by default, it’s trivial to use it in your Rails app right now. See the “ActiveRecord Integration” section of the README for code examples.  In this case, you probably don’t need to actually store any data in the Xapian database, just the index information (and the :id field of course, but that’s stored by default) – so you get a smaller database (though you still need to store fields that you want to group by (collapse) or order results with).

Multi-master replicated full text indexing service

Xapian Fu doesn’t do this. I’m designing something that might though :)

Getting Xapian Fu

It’s available in gem form from Rubyforge/cutter.  The code is on github here.  You’ll need the Xapian Ruby bindings installed – on Debian/Ubuntu this is the libxapian-ruby1.8 package.  The gem named xapian claims to provide Xapian and the Ruby bindings but it failed form me on 64bit.  The gem named xapian-full claims to provide the Ruby bindings without Xapian (you’ll obviously need to build and install Xapian yourself) but I’ve not used that either.  RPMs, source files and other downloads are listed on the Xapian downloads page.

There is also this weird kinda splash page I made, in some kind of attempt to host something about Xapian Fu on my own domain. Not really sure what real purpose it serves.

Ruby’s case statement calls the === method on the argument to each of the when statements

So, this example:

case my_number
  when 6883
    :prime
end

Will execute 6883 === my_number

This is all fine and dandy, because the === method on a Fixnum instance does what you’d expect in this scenario.

However, the === method on the Fixnum class does something different. It’s an alias of is_a?

That is cute, because it allows you to do this:

case my_number
  when Fixnum
    "Easy to memorize"
  when Bignum
    "Hard to memorize"
  end

But it won’t work as you might expect in this scenario:

my_type = Fixnum
case my_type
  when Fixnum
    "Fixed number"
end

This won’t work because Fixnum === Fixnum returns false because the Fixnum class is not an instance of Fixnum.

My workaround for this is to convert it to a string first. Not sure if that’s the best solution, but it works for me(tm).

my_type = Fixnum
case my_type.to_s
  when "Fixnum"
    "Fixed number"
end

go = Proc.new { sleep 24.hours }
self.wants :sedatation
begin ; nil ; end
case go ; where "no" ; nil ; end
self.wants :sedatation
self.get '/airport'
self.put '/airport/plane'
before self.insane? do
  3.times { hurry! }
end
return if self.can_control? :fingers
return if self.can_control? :brain
5.times { "no" }

I recorded me singing it, which is kinda stupid tbh.

I used mencoder to convert this to something Youtube found tasty. Like this:

mencoder -ss 15 -endpos 1:18 -vf pp=al:f,scale=480:360 -oac mp3lame -ovc lavc -lavcopts vcodec=libx264:mbd=1:vbitrate=2000 MOV01362.MPG -o MOV01362.x264


Also, pimp for another Geek/Ukelele project: Ukepedia, all 3 million Wikipedia articles one song at a time

More connections will use more RAM, but how much?  We don’t want to overcommit, as the connection tracker uses unswappable memory and things will blow up. If we set aside 512MB for connection tracking, how many concurrent connections can we track?

There is some Netfilter documentation on wallfire.org, but it’s quite old. How can we be sure it’s still correct without completely understanding the Netfilter code? Does it account for real life constraints such as page size, or is it just derived from looking at the code? A running Linux kernel gives us all the info we need through it’s slabinfo proc file.

We can peek at how the kernel is using RAM using the proc file /proc/slabinfo and clear this up.  The nf_conntrack entry from here tells us, on one particular firewall, that there are 26,702 active entries (or objects), that each object is 304 bytes in size and 13 of them fit in each slab (and that each slab is 1 kernel page).  So we know that conntrack entries take up 304 bytes each.  But if we’re going to be accurate, then we have to account for the overhead of the kernel page size.

The Linux kernel uses a slab memory allocator, so rather than allocating 304 bytes every time a conntrack entry is needed, they are allocated in “slabs” of one or more kernel pages which reduces memory fragmentation and improve performance.  When they’re done with, they’re not immediately freed – instead the memory is reused the next time another object of the same type is needed.

In the kernel we’re using, the page size is 4096 bytes. As slabinfo told us, 13 nf_conntrack objects fit in each slab and each slab takes up 1 page. 13 objects of 304 bytes is 3952 bytes in total, which leaves 144 bytes of waste per slab.  So every 13 objects we waste 144 bytes. So a nf_conntrack object consumes about 316 bytes on this box, giving us almost 1.7 million entries for our 512MB.

You can get your kernel’s page size with the command: getconf PAGESIZE. The slabtop program, installed on most modern GNU/Linuxes, shows the info from /proc/slabinfo it in a pretty table and lets you sort the values.

For those of your with short attention spans, I finally get started with the talk at about 2mins 30, and start singing the first article, Otitis Media, at about 7mins.

Hugh’s answer genuinely made me think this was an early April Fool’s joke. In fact I’m still not sure Medialens aren’t making me look like an idiot:

No proof, but circumstantial evidence and reasonable presumption of AQI [al-Qaeda in Iraq] involvement – very much their modus operandum. Suicide attacks are their signature method, and this was a dramatic detonation suggesting a lot of explosive – again, very AQI.

And…who else would do this?

So, process of elimination, history of AQI attacks in Diyala etc.

And the logic of it Sunni Arab vs Iraqi Kurds. As a man in Jalawla told Reuters:

“Al-Qaida is targeting the Kurds because it believes that
we are involved in the political process and collaborating
with the Americans.”

This blows my mind. “very AQI” and “a man in Jalawla told Reuters”. “Who else would do this?”

As Medialens point out, the BBC claim they are “committed to evidence-based journalism”. Except they pick and choose when their committment applies, such as when they refused to report the use of banned weapons by US forces in their November 2004 assault on Falljuah.

I got to have a bit of a nosey around the Manchester BBC building too – though I was worried I’d open the wrong door and end up on TV. Didn’t fancy having to apologise to Jeremy Paxman.

Brightbox also sponsored some pizza, and gave away t-shirts and stickers like candy (there was no candy though).

My slides are available here, and contain a little example file system indexer. I made my slides with webby and S6 if you’re interested.

The main gripe appears to be with “women only” conferences, such as the Women on the Web conference, organised by a group called Forward Ladies, or the Geek Girl dinners.

I think a fair summary of his, and some other commenters, opinion is that these “women-only” events don’t help the effort to get more women involved in technology. Comparing it to positive discrimination in many ways.

Women Friendly

The way I see these events is more “women-friendly”, rather than “women-only”.  With Geek Girl dinners, this is explicit, as men can attend at the invite of a women. A simple, but generally effective heuristic to select for friendliness to women.

With Women on the Web, it’s less clear, but nowhere does it state men are not welcome.  The same goes for the Forward Ladies membership terms and conditions. I’m not sure how Dom knows this event is women only (I’ve emailed Forward Ladies for more info).

Until I’m shown otherwise, I’m assuming the Women on the Web conference is women-friendly, not women-only.  Of course, if you browse the site you see photos of women, and all the past speakers appear to be women, and this might not be that inviting to men – but how inviting do you think Infosec is to women, with photos of rooms full of men, and women in skimpy clothes giving out leaflets? Perhaps all the speakers are women because they just don’t get many men interested – maybe men just  need to be thicker skinned and ask to be involved.

As an amusing side note, after checking the Infosec url was correct I caught sight of this year’s branding – hundreds of what I assume are just people, but they happen to be using the pretty well established symbol for “men” (let’s not get into that though).

Demand

You only have to see the turnout of women at Leeds Geek Girl dinners compared to the turnout of women at Leeds Geekup to know there is a demand for these women-friendly events. For whatever reasons, some women are more likely to go an explicitly women-friendly geek event than another random geek event. Of course women should be encouraged to come to all events, but not all events are equally friendly to women, and it’s often difficult to assess how friendly they are from outside.

Social Groups

Women are a social group  – they’ll often share some common experiences and outlooks due to their sex – due to society’s treatment of them as a group of people. This goes for men too. And gay people. And geeks. Etc.  When you see women in this way, you’d expect some of them to organise and attend these types of events. Most social groups do.

Geek Girl Dinners just have a convenient way of selecting women-friendly people: women, or men invited by women. Of course, not all women are women friendly  – that is just a stereotype, but it seems to work for them.

Largely though, I think if you name things and brand things right, you get the people you’re hoping for. Maybe that’s why Infosec is a sausage-fest.

Update

Forward Ladies has confirmed that men are welcome to Women on the Web, and are welcome as members of Forward Ladies too. They also run 50/50 events “to which men are specifically invited”.

More Leeds Market photos here on my Flickr profile.

From the site:

“Techietubbies is a weekly podcast covering a multitude of subjects, from a round up of the week’s tech news, live callers, competitions, questions and answers… and beer :)”

Though I’m driving, so no tech news for me. I think it’s recorded if you can’t see the live thing.  It’ll be broadcast live here via ustream.tv

I realised that having tinkered with computers heavily almost my entire life, a lot of my “computer skills” have shifted into the intuitive, automatic systems.  I obviously (hopefully) use the rational systems a great deal, but underlying it is definitly intuition – the gut feeling of where to go next to solve the problem.  I regularly come up seemingly random avenues of investigation that lead to gold and I couldn’t say with any certainty why I thought of it.  I’m assuming this is the same for most computer geeks (and chess geeks, cooking geeks, music geeks etc. :).  It’s become a native language for us.

I don’t think the average rational system can easily deal with very complex problems.  It’s great for some more-linear concentrated work or planning, but for big stuff with lots of parts – hard work.  I think I usually research and “pre-process” a bunch of material around a problem using my rational system, then my automatic system gets to work mulling over the bigger picture.  Then when I’m making rational decisions about it, I’m heavily informed by the intuition. Or sometimes just when I’m showering.

Anyway, not sure where I was going with this other than a “aren’t I great” blog post. The summary would be, don’t rely on your rational systems so much. Give the intuitive some good mulling time. And shower regularly.

Louisa and I put this together in just a few days, me coding and Louisa designing. Was a fun little project to do.

“We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.”

That’s pretty funny. She’s basically saying

“It’s better to build important things with tools you can’t examine for yourself.”

SAS claim to have over 40,000 customer sites worldwide.  The news article claim 250,000 people use R regularly.  The difference here isn’t in the numbers of users, it’s that, with R, every user is a potential developer.  SAS can’t possibly compete with that.

My mate Paul Brook is talking about RAID on Linux too.

Come along for the talk, or the beer, or the socialising – or all three.