CPU benchmarks

Here are some geekbench readings (32bit tryout version) for some of our servers and for comparison some Amazon EC2 images.

The method here is pretty simple. Take the default OS install, copy geekbench 32 bit tryout edition onto the machine. Run it and record the results.

It’s important to remember that geekbench performs a mixture of tests, some of which don’t parallelise. This means a server with a fast core will receive a higher score than one with lots of slower cores. As a result the sandybridge and ivybridge machines score very highly because the servers will increase the performance of a single core if the other cores are idle.

Disk benchmarks

We have several disk subsystems available. Single disk, dual disk mirrored software RAID, dual disk mirrored hardware RAID, 8 disk array hardware RAID and PCI-E SSD accelerator card.

Read only benchmarks

The benchmark here is carried out with iops, a small python script that does random reads.

4kb reads

1MB reads

It’s worth noting that with 64MB reads the Intel 910 delivers 1.2GB/sec, the hi.4xlarge instance 1.1GB/sec (curiously the Amazon machine was quicker with 16MB blocks). At the smaller block sizes the machine appears to be bottlenecked on CPU rather than the PCI-E accelerator card. The RAID10 array had a stripe size of 256kB so the 1MB read requires a seek on every disk – hence performance similar to that of a single disk as the limitation is seek rather than transfer time. There’s a reasonable argument that a more sensible setup is RAID1 pairs and then LVM striping to have much larger stripe sizes than the controller natively supports.

We’re not sure why the SAS array benchmarks so slowly, it is an old machine (five years old) but is set up for performance not reliability.

Write only benchmarks

I went back to rate.c, a synchronous disk benchmarking tool we wrote when investigating and improving UML disk performance back in 2006. What I did was generate a 2G file, run random sized synchronous writes into it and then read out the performance for 4k and 1M block sizes. The reasoning for a 2GB file is that our Linode instance is a 32bit OS and rate.c does all the benchmarking into a single file limited to 2GB.

Write performance

I think there’s a reasonable argument that this is reading high for small writes on the BBU controllers (including the VMs & Linode VM). It’s entirely possible that the controllers manage to cache the vast majority of writes in RAM and the performance wouldn’t be sustained in the longer term.

Real world test

We presented these results to one of our customers who has a moderately large database (150GB). Nightly they take a database backup, post process it then reimport it to another database server in order to do some statistical processing on it. The bottleneck in their process is the database import. We borrowed their database and this is the timing data for a postgresql restore. The restore file is pulled from the same media the database is written to.

The postgres import is mostly single threaded – usually the servers sit at 100% CPU on one core with the others idle with only occasionaly bursts of parallelism. Consequently usually the CPU is bursting to 2.5Ghz (Sandy Bridge) or 3.3Ghz (Ivy Bridge). The Ivy Bridge RAID1 machine is actually a Mac Mini. In many ways this is an application perfectly suited to ‘the cloud’ because you’d want to spin up a fast VM, import the database then start querying it. It’s important to note that the estimated lifetime of the Intel 520 RAID 10 array in this test is six months, the performance gain there over the 910SSD is entirely due to faster single threaded performance on the CPU.

Bias

Whilst I’ve tried to be impartial obviously these results are biased. When Mythic Beasts choose hardware for our dedicated and virtual server platforms we deliberately search out the servers that we think offer the best value, so to some extent our servers have been chosen because historically they’ve performed well at the type of benchmarks we test with. There’s also publication bias, if the results said emphatically that our servers were slow and overpriced we’d have fixed our offering, then rewritten the article based on the newer faster servers we now had.

Notes

The real world test covers two scenarios, the delay in getting a test copy of the database for querying for which temporary storage may be fine, plus in the event of something going hideously wrong a measure of the downtime of your site until it comes back up again in which case persistent storage is terrifically important.

I plan to add an m2.xlarge + EBS instance, and a hi1.4xlarge instance. I originally didn’t include the hi1.4xlarge because they don’t have EBS optimised volumes for persistent storage. I might also add some test Mythic Beasts VMs with both safe and unsafe storage (i.e. you cache all writes in the host RAM ignoring sync calls) which is a cheap and easy way to achieve instance storage with a huge performance benefit. I excluded the Linode VM from the final test as it’s too small.

One of our customers decided to send us a more unusual christmas card in the form of a puzzle. It was this,

if you do

cd /dev
mv *. *..

how do you restore the system to a working state?

To add hypothetical pressure to this hypothetical question imagine that the users of the hypothetical server had just done a major press launch.

Our answer was, if your hypothetical panicking system administrator had bought our managed service we could restore it from backup. As a favour we’d even restore the backup to a new virtual server allowing us to keep the filesystem on the old machine so we’d have a chance of pulling out any missing updates that had happened since the backup that morning. We could then merge it all back on to the original platform at a slightly less stressful time.

Happily our backup and restore procedures do get tested, so we were able to fairly straightfowardly restore from the backup server, build the new VDS and bring it up running the site.

One thing our newest employee didn’t know was about the usefulness of netcat so I was able to make him one of todays lucky 10,000.[*] If you’re running a debian install CD you’ll notice it doesnt have ssh or rsync but does have netcat. Consequently you can transfer your data nice and quickly as follows,

backup# iptables -I INPUT 1 -p tcp -s -J ACCEPT
backup# cat backed-up-filesystem.tar.gz | nc -p xxx -l
target# nc -p xxxx | gunzip | tar xv
backup# iptables -D INPUT 1

As always, if you read this and think “that’s clever, I wonder if I’d have thought of that” you should take a look at our managed services. If you’d definitely done it better[**] then we invite you to show off at our puzzle.

[*] Scaling for the UK birth rate rather than the US one he’s one of todays lucky 1,600. You’d have to be pedantic even by the standards of system administrators to complain about that though.

[**] Before some points out that tar has a z flag we are aware of that. However computers have lots of processors and this way tar / gunzip / nc can all sit on different cores and make it go faster which is important if you’ve people waiting for the site to come back up.

One of our managed server customers came to us with a straightforward sounding problem. They needed to do a massive data import to their database which was going to take several days to complete. They hadn’t managed to test their full import because they didn’t have a test machine with enough memory, but the procedure looked it looked very solid on smaller datasets. After some discussion we came up with a plan where we’d put together a faster replacement server for them, they could repeatedly try their import until the database was complete and tested, then we’d move the full site over to the new faster machine, reclaiming the old one back into our pool.

As this seemed an ideal opportunity to test our backup and restore procedures and to give training to our newest staff member, instead of copying their server over to the new one, we restored the new one from backup. Happily we restored the state of the server fairly easily, did a little bit of troubleshooting the boot process, a summary of the expected issues in getting the new machine up are as follows,

• udev remembers your network card mac address, so on the first boot your ethernet card comes up as eth2/eth3 instead of eth0/eth1 which means the network doesn’t start, handily we have serial console / recovery mode on our servers so this is easy to fix.
• /dev can’t be backed up so you need to manually generate a minimal set of device nodes.
• it’s important to make sure you’ve generated /proc /sys /tmp /run mount points with the right permissions.
• when installing grub in a chroot you need to make sure you’ve done a bind mount of /proc /sys /dev or you’re going to get very confused
• you can’t back up a mysql database purely by copying it’s files, we take a mysql dump which we restore from, this means you need to start from a blank mysql database and do the restore into it. In order to let the debian mysql scripts work you need to move /etc/mysql/debian.cnf out of the way, create your blank mysql database, do the restore, stop the database, move debian.cnf back in in order to make your system-maintainer passwords match the database at each stage.
• On our replacement server we’ve installed LVM (the original was just a partition) so we needed to install the LVM tools into the chroot to build the initramfs to boot the system.
• We needed to edit /etc/fstab because the UUIDs have changed to point the machine at the correct partitions to boot from.

This is all fine and expected. We handed the machine to our customer, they were happy and it all seemed done. Two days later they came back, mysql on the new server was much slower than the previous server. They gave us a sample query and wondered why it answered in a fraction of a second on their development and current production server. We started investigating, the new server was running 100% CPU, no disk IO, the entire database source files cached so we hadn’t done something obviously stupid. We then started looking at the mysql query planner, it had a different query plan for the same query on the same database on two different machines.

How odd we thought.

Looking further at the indexes we discovered that cardinality of the indexes for the tables in question was different. Further investigation shows that the indexes are built differently if you do INSERT, INSERT, INSERT as the original database was created compared to a single multi-row insert that the restore from backup did.

The solution is to run

optimize table table_name

which finishes building the indexes properly, the query plan changes and the queries then run quickly. We have now added mysqloptimize –all-databases to our recovery procedure and handily put it into this blog post in the hope of saving someone else the investigation we’ve just done.

If your database backup procedure doesn’t have this step in and attempting to figure it out sounds hard you should consider our Managed hosting services. If investigating this sort of thing sounds like a fun way to spend your working days you should consider applying for a job.

We’ve updated our dedicated server range. Our entry level hardware RAID server has had a RAM and CPU boost to 2.5Ghz Ivy bridge and 16GB of RAM. Our RAID10 servers have been boosted to 32GB of RAM or 128GB of RAM and we’ve now got some machines with 256GB of RAM available.

We can also by arrangement now offer VMs with 120GB of RAM (we’re using the 128GB dedicated servers to expand the next phase of our virtual server platform).

We’re now running a mirror for the Raspberry Pi download server and the Raspbian apt repository on a Raspberry Pi.

The first problem was obtaining a Raspberry Pi as buying one was tricky: firstly the online shops were down then the queue for a Pi was rather long. As any banker can tell you occasionally crime does pay so I abducted Mooncake, a cat which owns Liz and Eben. I then ransomed the feline back in exchange for a Raspberry Pi.

The hardware setup starts with a power supply with an IEC14 connector connected to the masterswitch for remote power cycling. This PSU connects to a powered USB hub, with a USB lead connecting to the power connector on the Raspberry Pi. The Pi is then connected back to the hub on the data cable with a 1TB USB external hard disk attached to that. There’s a 100Mbit ethernet cable which connects up to the core Mythic Beasts network and out to the Internet. Technically the switch port on the other end is 1Gbit but the Raspberry Pi isn’t fast enough to use that.

The yellow fibre in the background is a very large Internet exchange with over a terabit of bandwidth, the Pi isn’t connected directly (that’d be too stupid even for us) and the packets travel from the Pi to the exchange a couple of feet away via another building. However with LONAP and LINX within 2ms and AMSIX a mere 8ms away it’s still rather well connected.

The software setup on the Pi is fairly straightforward. We started with the Debian squeeze image, ssh / apache enabled, munin enabled (graphs here). We’ve changed the password (obviously), the ssh keys (shipping the same ssh key on every OS image isn’t optimal), and moved /var/www and /var/log to the USB disk so as not to fill the SSD card. rpi-update was needed to make the USB/network setup stable under load as the initial image kept crashing. We also set the RAM split to 224MB for Linux as we really aren’t using the GPU.

It’s up and running both IPv4 (93.93.128.128) and IPv6 (2a00:1098:0:80:1000:13:0:3), as the core Raspbian server is also running IPv6 and the main Raspberry Pi server is also IPv6 at present this machine has seen more than 50% of its traffic over IPv6. I suspect this will change as people download images from it though.

At present the Raspberry Pi is devoting nearly half its CPU to drawing munin graphs so I need to benchmark the new Raspbian distribution to see if the hard float debian build improves the anti-aliasing, as presently all the calculations are done without using the FPU on the arm core in the Pi. Benchmarking suggests that we can deliver 35-50Mbps of file downloads reasonably comfortably at present.

Is this sensible? We’ve had a few customers ask us if the Raspberry Pi would be a sensible device for hosting on as it’s very cheap and very low power. Unfortunately it’s also very slow for this kind of application and the supporting hardware is very bulky. The i7 quad core mac mini occupies less space than the Pi + hub + disk + PSU, uses about fives times as much electricity, costs about five times as much once you include the supporting hardware but is hundreds of times faster. So revolutionising the hosting industry isn’t going to happen with the Raspberry Pi, at least not until they build a PoE one with gigabit ethernet and more RAM.

It’s not currently sensible to do this with shelves full of Raspberry Pis because the performance per Watt isn’t good enough. But we’re working on it.

Thanks go to Liam Fraser and Mike Thompson for adding us to the official mirror lists for the Pi and Raspbian. Additional thanks go to Eben and Liz for paying the ransom fee of one Raspberry Pi in exchange for the safe return of Mooncake who looks a lot like Raffles.

This evening I joined Eben & Liz Upton at the Nominet Internet Awards in my role as a contributor to the Raspberry Pi project. We were moderately surprised when the project won the Outstanding Contribution award. We’d like to offer our congratulations to everyone nominated for an award and to thank Nominet for an excellent evening.

People with excellent eyesight may be able to spot me in this photo taken by Dr Black.

There’s a well publicised bug in the linux kernel which makes it unhappy with leap seconds and usually Java and Mysql. If you’ve a linux box running at high CPU for no explicable reason after the leap second at the weekend you were probably affected.

The fix is,

/etc/init.d/ntp stop
date; date `date +"%m%d%H%M%C%y.%S"`; date
/etc/init.d/ntp start

This is what it did to the power monitoring in one of our racks with a large number of affected machines.

There is a reason we allocate the whole power allocation for a server and not just the typical idle usage.

100 years ago Alan Turing was born, in his sadly short lifetime he achieved a number of fairly impressive things, broke the enigma cipher, invented modern computing and invented the Turing test. On his day off he ran up and down the river in Cambridge and narrowly missed a place in the Olympic marathon team.

We haven’t done anything quite as impressive as that, although to be fair I don’t think anyone else in the last century has either. However some time ago at the Cambridge Beer Festival team Mythic entered a code cracking challenged based on a piece of ciphertext printed on a poster.

BLEQXVXEKPNRMWBRMVMVERNMWMZNQTTMRKMTMRHP

This is a substitution cipher which team Mythic cracked on the first day. You can decipher it with this perl one liner,

echo ‘BLEQXVXEKPNRMWBRMVMVERNMWMZNQTTMRKMTMRHP’ | perl -pe ‘tr/NMWZQTRKHPEBLXV/sonfidtcukaemlb/;’

which outputs

emailblackstonetobobatsonofsiddotcodotuk

We did, and we now have to get onto the more difficult task of finding a convenient day to have a barbeque and drink the beer we won.

If you think that working for a company that goes to beer festivals and solves problems by writing little bits of code is fun our jobs page is here: Mythic Jobs. If you’re running a web facing company and thinking that this is terribly clever you might want to think about our managed hosting.

We’ve now enabled IPv6 by default for all customers hosted on either onza or yali. The control panels for these machines have been running IPv6 for over a year, we’ve now enabled it for all customer websites too.

Behind the scenes we’ve been expanding and improving our VDS platform. We’ve just added a second cluster (Telecity HEX) that you can now order VMs in and we’re about to add a third cluster (Cambridge) too. You’ll be able to specify which zone your virtual server runs in if you have a preference and each one is independently routed.

We’ve had a long association with the Cambridge Cycle Campaign and Cyclestreets. Cyclestreets are borrowing a 64GB VM in our HEX cluster to run some of their data import jobs and to test out how well it works. Shortly after that we’ll be launching bigger VM sizes, 16GB, 24GB, 32GB, 48GB, 64GB and 80GB with corresponding disk and bandwidth increases. Initially these will be available in the London zones only (SOV/HEX) but we expect to follow through with Cambridge in the near future.