Some more benchmarks

CPU benchmarks

Here are some geekbench readings (32bit tryout version) for some of our servers and for comparison some Amazon EC2 images.

The method here is pretty simple. Take the default OS install, copy geekbench 32 bit tryout edition onto the machine. Run it and record the results.

It’s important to remember that geekbench performs a mixture of tests, some of which don’t parallelise. This means a server with a fast core will receive a higher score than one with lots of slower cores. As a result the sandybridge and ivybridge machines score very highly because the servers will increase the performance of a single core if the other cores are idle.

Disk benchmarks

We have several disk subsystems available. Single disk, dual disk mirrored software RAID, dual disk mirrored hardware RAID, 8 disk array hardware RAID and PCI-E SSD accelerator card.

Read only benchmarks

The benchmark here is carried out with iops, a small python script that does random reads.

4kb reads

1MB reads

It’s worth noting that with 64MB reads the Intel 910 delivers 1.2GB/sec, the hi.4xlarge instance 1.1GB/sec (curiously the Amazon machine was quicker with 16MB blocks). At the smaller block sizes the machine appears to be bottlenecked on CPU rather than the PCI-E accelerator card. The RAID10 array had a stripe size of 256kB so the 1MB read requires a seek on every disk – hence performance similar to that of a single disk as the limitation is seek rather than transfer time. There’s a reasonable argument that a more sensible setup is RAID1 pairs and then LVM striping to have much larger stripe sizes than the controller natively supports.

We’re not sure why the SAS array benchmarks so slowly, it is an old machine (five years old) but is set up for performance not reliability.

Write only benchmarks

I went back to rate.c, a synchronous disk benchmarking tool we wrote when investigating and improving UML disk performance back in 2006. What I did was generate a 2G file, run random sized synchronous writes into it and then read out the performance for 4k and 1M block sizes. The reasoning for a 2GB file is that our Linode instance is a 32bit OS and rate.c does all the benchmarking into a single file limited to 2GB.

Write performance

I think there’s a reasonable argument that this is reading high for small writes on the BBU controllers (including the VMs & Linode VM). It’s entirely possible that the controllers manage to cache the vast majority of writes in RAM and the performance wouldn’t be sustained in the longer term.

Real world test

We presented these results to one of our customers who has a moderately large database (150GB). Nightly they take a database backup, post process it then reimport it to another database server in order to do some statistical processing on it. The bottleneck in their process is the database import. We borrowed their database and this is the timing data for a postgresql restore. The restore file is pulled from the same media the database is written to.

The postgres import is mostly single threaded – usually the servers sit at 100% CPU on one core with the others idle with only occasionaly bursts of parallelism. Consequently usually the CPU is bursting to 2.5Ghz (Sandy Bridge) or 3.3Ghz (Ivy Bridge). The Ivy Bridge RAID1 machine is actually a Mac Mini. In many ways this is an application perfectly suited to ‘the cloud’ because you’d want to spin up a fast VM, import the database then start querying it. It’s important to note that the estimated lifetime of the Intel 520 RAID 10 array in this test is six months, the performance gain there over the 910SSD is entirely due to faster single threaded performance on the CPU.

Bias

Whilst I’ve tried to be impartial obviously these results are biased. When Mythic Beasts choose hardware for our dedicated and virtual server platforms we deliberately search out the servers that we think offer the best value, so to some extent our servers have been chosen because historically they’ve performed well at the type of benchmarks we test with. There’s also publication bias, if the results said emphatically that our servers were slow and overpriced we’d have fixed our offering, then rewritten the article based on the newer faster servers we now had.

Notes

The real world test covers two scenarios, the delay in getting a test copy of the database for querying for which temporary storage may be fine, plus in the event of something going hideously wrong a measure of the downtime of your site until it comes back up again in which case persistent storage is terrifically important.

I plan to add an m2.xlarge + EBS instance, and a hi1.4xlarge instance. I originally didn’t include the hi1.4xlarge because they don’t have EBS optimised volumes for persistent storage. I might also add some test Mythic Beasts VMs with both safe and unsafe storage (i.e. you cache all writes in the host RAM ignoring sync calls) which is a cheap and easy way to achieve instance storage with a huge performance benefit. I excluded the Linode VM from the final test as it’s too small.