Over the past week or so we’ve given the Science Media Centre a hand in moving their WordPress site into a virtual machine hosted by Mythic Beasts. They’re a charity who work with journalists, scientists and engineers to try and improve the quality of science reporting and removing the misleading rubbish that otherwise gets written. Mythic Beasts is a company founded by science graduates who are very easily angered by terrible science articles in the papers. We’re hoping the saving on destroyed laptops and monitors will easily cover all the management and consultancy services we’ve donated.

If we have fewer idiotic articles proving that Coffee cures cancer^* and Coffee causes cancer^* and rather more articles that our talented university friends pioneer new cancer treatments we’ll consider the time and effort we’ve put in to helping them well spent.

* Actual links removed in the name of good taste. Here’s something more interesting to read, and if you’re still curious, you can look up coffee in the index.

Today is Guy Fawkes Night, when traditionally in the UK we launch fireworks to commemorate not blowing up King James I, the House of Lords and a large chunk of central London. Most modern firework displays use rather less gunpowder than the original plot in 1605.

At Mythic Beasts we think with the advent of four hundred years additional technology we should be aiming a bit higher. That’s why instead of buying fireworks we’ve donated some hosting to Southampton University Spaceflight. Certainly I (Pete) was incredibly inspired as a child by space flight and I remember watching the first untethered spacewalk as a child which motivated me strongly to learn all about rockets and space ships and from there to reverse engineer 8 bit computer games ultimately leading to running a hosting company.

This is of course the second university space program we’re helping out with – some of the tracking for Cambridge University Spaceflight is also done on servers hosted with us.

If you want to hear what we’ve got to say, but can only cope with 140 characters at a time then you can now follow us on Twitter.

We may use Twitter to give updates on service status when we’ve got problems, but the official source for such information remains our status page.

So our newest Mythic Beast started working for us today. The first task is to start installing your new laptop and reading and signing employment contracts. Today we had our newest employee fail at the first hurdle.

The laptop in question is a shiny Toshiba z930. This one came with Windows 8 and a fully charged battery. On first powering it on it comes up with the Windows 8 licence. This has a tickbox option for ‘I accept the license’ and a big button labelled ‘Accept’ to click on.

If you don’t tick the box, it tells you you have to. There’s no option to reject the license.

If you press the power button the laptop suspends itself. If you press and hold the power button the laptop still suspends itself. Ctrl-Alt-Delete doesn’t work. You can’t remove the battery as it’s built in. In frustration our newest employee suggested pouring his coffee over the damn thing to make it power cycle. This was a really stupid idea, not only does the laptop have a spill proof keyboard he’d also then have no coffee.

The best plan we could come up with was to wait for the batteries to run out which requires pressing a key about every five minutes to stop the thing suspending itself.

We’ve just done a performance comparison of one of our little dedicated servers versus a dual core VM hosted on VMware through another provider.

The VMware machine has two cores of an Intel Xeon E5530 (Westmere) at 2.4Ghz, we have four hyperthreaded cores of an i7-3615QM (Ivy Bridge) at 2.3Ghz.

Both machines are running the same operating system install, same application code so we ran siege for 30s at a time with different concurrency levels as a benchmark to find out if our machine was faster and by how much.

The initial comparison was the dual core VMware service (green), versus our VM (red). At very low concurrency (1-2 simultaneous requests) our machine is slightly slower to render each page. Beyond this the existing machine has exactly the predicted load curve in that it slows linearly with additional simultaneous users – the new machine appears to slow only very slightly with minimal performance degradation.

By default we’re running the ondemand cpu scheduler which means when idle the cores are clocked at 1.2Ghz. The page render time remains almost constant to four cores as the host spreads the load around the four 1.2ghz cores keeping the render time constant. Beyond this the scheduler starts to turn up the core speed as the load rises, so at 8 cores we’re still rendering pages in the same average time because each core is now clocked at 2.3Ghz almost twice as fast – we’ve doubled the amount of CPU available. Only then does the performance begin to drop off and then sub-linearly.

On the existing host the performance is much more predictable – it takes a constant amount of CPU to render each page request and as you double the concurrency the render time doubles.

If you turn off the power-saving on the i7 and set it to performance mode it gives the expected linear performance decrease with increasing load. Interestingly it’s slightly slower at maximum CPU utilisation, I think (but haven’t confirmed) this is because it can’t use the turbo boost feature to increase the clock-speed as much as the power-saving option because it’s always running at a warmer temperature as it doesn’t cool down as much between each benchmark run.

We’re going to leave the machine in ondemand mode, whilst it’s slightly slower in normal use, it uses less electricity so it cheaper to run and less harmful to the polar bears, it also has significantly better performance for short peaks – it has a stockpile of cold that it can borrow against for short periods of time.

I wonder if they should start teaching thermodynamics in computer science courses.

CPU benchmarks

Here are some geekbench readings (32bit tryout version) for some of our servers and for comparison some Amazon EC2 images.

The method here is pretty simple. Take the default OS install, copy geekbench 32 bit tryout edition onto the machine. Run it and record the results.

It’s important to remember that geekbench performs a mixture of tests, some of which don’t parallelise. This means a server with a fast core will receive a higher score than one with lots of slower cores. As a result the sandybridge and ivybridge machines score very highly because the servers will increase the performance of a single core if the other cores are idle.

Disk benchmarks

We have several disk subsystems available. Single disk, dual disk mirrored software RAID, dual disk mirrored hardware RAID, 8 disk array hardware RAID and PCI-E SSD accelerator card.

Read only benchmarks

The benchmark here is carried out with iops, a small python script that does random reads.

4kb reads

1MB reads

It’s worth noting that with 64MB reads the Intel 910 delivers 1.2GB/sec, the hi.4xlarge instance 1.1GB/sec (curiously the Amazon machine was quicker with 16MB blocks). At the smaller block sizes the machine appears to be bottlenecked on CPU rather than the PCI-E accelerator card. The RAID10 array had a stripe size of 256kB so the 1MB read requires a seek on every disk – hence performance similar to that of a single disk as the limitation is seek rather than transfer time. There’s a reasonable argument that a more sensible setup is RAID1 pairs and then LVM striping to have much larger stripe sizes than the controller natively supports.

We’re not sure why the SAS array benchmarks so slowly, it is an old machine (five years old) but is set up for performance not reliability.

Write only benchmarks

I went back to rate.c, a synchronous disk benchmarking tool we wrote when investigating and improving UML disk performance back in 2006. What I did was generate a 2G file, run random sized synchronous writes into it and then read out the performance for 4k and 1M block sizes. The reasoning for a 2GB file is that our Linode instance is a 32bit OS and rate.c does all the benchmarking into a single file limited to 2GB.

Write performance

I think there’s a reasonable argument that this is reading high for small writes on the BBU controllers (including the VMs & Linode VM). It’s entirely possible that the controllers manage to cache the vast majority of writes in RAM and the performance wouldn’t be sustained in the longer term.

Real world test

We presented these results to one of our customers who has a moderately large database (150GB). Nightly they take a database backup, post process it then reimport it to another database server in order to do some statistical processing on it. The bottleneck in their process is the database import. We borrowed their database and this is the timing data for a postgresql restore. The restore file is pulled from the same media the database is written to.

The postgres import is mostly single threaded – usually the servers sit at 100% CPU on one core with the others idle with only occasionaly bursts of parallelism. Consequently usually the CPU is bursting to 2.5Ghz (Sandy Bridge) or 3.3Ghz (Ivy Bridge). The Ivy Bridge RAID1 machine is actually a Mac Mini. In many ways this is an application perfectly suited to ‘the cloud’ because you’d want to spin up a fast VM, import the database then start querying it. It’s important to note that the estimated lifetime of the Intel 520 RAID 10 array in this test is six months, the performance gain there over the 910SSD is entirely due to faster single threaded performance on the CPU.

Bias

Whilst I’ve tried to be impartial obviously these results are biased. When Mythic Beasts choose hardware for our dedicated and virtual server platforms we deliberately search out the servers that we think offer the best value, so to some extent our servers have been chosen because historically they’ve performed well at the type of benchmarks we test with. There’s also publication bias, if the results said emphatically that our servers were slow and overpriced we’d have fixed our offering, then rewritten the article based on the newer faster servers we now had.

Notes

The real world test covers two scenarios, the delay in getting a test copy of the database for querying for which temporary storage may be fine, plus in the event of something going hideously wrong a measure of the downtime of your site until it comes back up again in which case persistent storage is terrifically important.

I plan to add an m2.xlarge + EBS instance, and a hi1.4xlarge instance. I originally didn’t include the hi1.4xlarge because they don’t have EBS optimised volumes for persistent storage. I might also add some test Mythic Beasts VMs with both safe and unsafe storage (i.e. you cache all writes in the host RAM ignoring sync calls) which is a cheap and easy way to achieve instance storage with a huge performance benefit. I excluded the Linode VM from the final test as it’s too small.

One of our customers decided to send us a more unusual christmas card in the form of a puzzle. It was this,

if you do

cd /dev
mv *. *..

how do you restore the system to a working state?

To add hypothetical pressure to this hypothetical question imagine that the users of the hypothetical server had just done a major press launch.

Our answer was, if your hypothetical panicking system administrator had bought our managed service we could restore it from backup. As a favour we’d even restore the backup to a new virtual server allowing us to keep the filesystem on the old machine so we’d have a chance of pulling out any missing updates that had happened since the backup that morning. We could then merge it all back on to the original platform at a slightly less stressful time.

Happily our backup and restore procedures do get tested, so we were able to fairly straightfowardly restore from the backup server, build the new VDS and bring it up running the site.

One thing our newest employee didn’t know was about the usefulness of netcat so I was able to make him one of todays lucky 10,000.[*] If you’re running a debian install CD you’ll notice it doesnt have ssh or rsync but does have netcat. Consequently you can transfer your data nice and quickly as follows,

backup# iptables -I INPUT 1 -p tcp -s -J ACCEPT
backup# cat backed-up-filesystem.tar.gz | nc -p xxx -l
target# nc -p xxxx | gunzip | tar xv
backup# iptables -D INPUT 1

As always, if you read this and think “that’s clever, I wonder if I’d have thought of that” you should take a look at our managed services. If you’d definitely done it better[**] then we invite you to show off at our puzzle.

[*] Scaling for the UK birth rate rather than the US one he’s one of todays lucky 1,600. You’d have to be pedantic even by the standards of system administrators to complain about that though.

[**] Before some points out that tar has a z flag we are aware of that. However computers have lots of processors and this way tar / gunzip / nc can all sit on different cores and make it go faster which is important if you’ve people waiting for the site to come back up.

Our shell account servers sphinx and onza now support mosh (the mobile shell), which is an alternative to using ssh for command line logins. The key feature of mosh is that it uses its own UDP-based protocol, which means that a mosh login can persist during network outages, and even if the client changes its IP address! You can start mosh on your wifi at home in the morning, jump on a train and switch to 3G, then plug into the Ethernet at the office and the mosh login will still be working.

To get started, simply install mosh on your client (it’s already in the major Linux distributions) and type the appropriate command to log in to your shell account:

mosh myname@sphinx.mythic-beasts.com
mosh myname@onza.mythic-beasts.com

It’s relatively young software, so there may be a few teething problems (it doesn’t play well with gnome-terminal unfortunately, although there are alternative terminal emulators such as mrxvt that work much better). Still, our staff are already completely hooked.

Over the past couple of weeks we have been migrating the final customers from Bluelinux (a hosting company we acquired) to our own hosting platform. This process is now complete. Any customers having issues should contact us at support@mythic-beasts.com.

Customers with shell accounts have been moved to onza, for which the control panel is here whereas customers with non-shell accounts have been moved to yali, for which the control panel can be found here. These control panels offer a flexible interface for web and email configuration.

Customers have more space, more software available to them and upgraded versions of software previously available at no additional cost.

One of our managed server customers came to us with a straightforward sounding problem. They needed to do a massive data import to their database which was going to take several days to complete. They hadn’t managed to test their full import because they didn’t have a test machine with enough memory, but the procedure looked it looked very solid on smaller datasets. After some discussion we came up with a plan where we’d put together a faster replacement server for them, they could repeatedly try their import until the database was complete and tested, then we’d move the full site over to the new faster machine, reclaiming the old one back into our pool.

As this seemed an ideal opportunity to test our backup and restore procedures and to give training to our newest staff member, instead of copying their server over to the new one, we restored the new one from backup. Happily we restored the state of the server fairly easily, did a little bit of troubleshooting the boot process, a summary of the expected issues in getting the new machine up are as follows,

• udev remembers your network card mac address, so on the first boot your ethernet card comes up as eth2/eth3 instead of eth0/eth1 which means the network doesn’t start, handily we have serial console / recovery mode on our servers so this is easy to fix.
• /dev can’t be backed up so you need to manually generate a minimal set of device nodes.
• it’s important to make sure you’ve generated /proc /sys /tmp /run mount points with the right permissions.
• when installing grub in a chroot you need to make sure you’ve done a bind mount of /proc /sys /dev or you’re going to get very confused
• you can’t back up a mysql database purely by copying it’s files, we take a mysql dump which we restore from, this means you need to start from a blank mysql database and do the restore into it. In order to let the debian mysql scripts work you need to move /etc/mysql/debian.cnf out of the way, create your blank mysql database, do the restore, stop the database, move debian.cnf back in in order to make your system-maintainer passwords match the database at each stage.
• On our replacement server we’ve installed LVM (the original was just a partition) so we needed to install the LVM tools into the chroot to build the initramfs to boot the system.
• We needed to edit /etc/fstab because the UUIDs have changed to point the machine at the correct partitions to boot from.

This is all fine and expected. We handed the machine to our customer, they were happy and it all seemed done. Two days later they came back, mysql on the new server was much slower than the previous server. They gave us a sample query and wondered why it answered in a fraction of a second on their development and current production server. We started investigating, the new server was running 100% CPU, no disk IO, the entire database source files cached so we hadn’t done something obviously stupid. We then started looking at the mysql query planner, it had a different query plan for the same query on the same database on two different machines.

How odd we thought.

Looking further at the indexes we discovered that cardinality of the indexes for the tables in question was different. Further investigation shows that the indexes are built differently if you do INSERT, INSERT, INSERT as the original database was created compared to a single multi-row insert that the restore from backup did.

The solution is to run

optimize table table_name

which finishes building the indexes properly, the query plan changes and the queries then run quickly. We have now added mysqloptimize –all-databases to our recovery procedure and handily put it into this blog post in the hope of saving someone else the investigation we’ve just done.

If your database backup procedure doesn’t have this step in and attempting to figure it out sounds hard you should consider our Managed hosting services. If investigating this sort of thing sounds like a fun way to spend your working days you should consider applying for a job.