Our shell account servers sphinx and onza now support mosh (the mobile shell), which is an alternative to using ssh for command line logins. The key feature of mosh is that it uses its own UDP-based protocol, which means that a mosh login can persist during network outages, and even if the client changes its IP address! You can start mosh on your wifi at home in the morning, jump on a train and switch to 3G, then plug into the Ethernet at the office and the mosh login will still be working.

To get started, simply install mosh on your client (it’s already in the major Linux distributions) and type the appropriate command to log in to your shell account:

mosh myname@sphinx.mythic-beasts.com
mosh myname@onza.mythic-beasts.com

It’s relatively young software, so there may be a few teething problems (it doesn’t play well with gnome-terminal unfortunately, although there are alternative terminal emulators such as mrxvt that work much better). Still, our staff are already completely hooked.

Over the past couple of weeks we have been migrating the final customers from Bluelinux (a hosting company we acquired) to our own hosting platform. This process is now complete. Any customers having issues should contact us at support@mythic-beasts.com.

Customers with shell accounts have been moved to onza, for which the control panel is here whereas customers with non-shell accounts have been moved to yali, for which the control panel can be found here. These control panels offer a flexible interface for web and email configuration.

Customers have more space, more software available to them and upgraded versions of software previously available at no additional cost.

One of our managed server customers came to us with a straightforward sounding problem. They needed to do a massive data import to their database which was going to take several days to complete. They hadn’t managed to test their full import because they didn’t have a test machine with enough memory, but the procedure looked it looked very solid on smaller datasets. After some discussion we came up with a plan where we’d put together a faster replacement server for them, they could repeatedly try their import until the database was complete and tested, then we’d move the full site over to the new faster machine, reclaiming the old one back into our pool.

As this seemed an ideal opportunity to test our backup and restore procedures and to give training to our newest staff member, instead of copying their server over to the new one, we restored the new one from backup. Happily we restored the state of the server fairly easily, did a little bit of troubleshooting the boot process, a summary of the expected issues in getting the new machine up are as follows,

• udev remembers your network card mac address, so on the first boot your ethernet card comes up as eth2/eth3 instead of eth0/eth1 which means the network doesn’t start, handily we have serial console / recovery mode on our servers so this is easy to fix.
• /dev can’t be backed up so you need to manually generate a minimal set of device nodes.
• it’s important to make sure you’ve generated /proc /sys /tmp /run mount points with the right permissions.
• when installing grub in a chroot you need to make sure you’ve done a bind mount of /proc /sys /dev or you’re going to get very confused
• you can’t back up a mysql database purely by copying it’s files, we take a mysql dump which we restore from, this means you need to start from a blank mysql database and do the restore into it. In order to let the debian mysql scripts work you need to move /etc/mysql/debian.cnf out of the way, create your blank mysql database, do the restore, stop the database, move debian.cnf back in in order to make your system-maintainer passwords match the database at each stage.
• On our replacement server we’ve installed LVM (the original was just a partition) so we needed to install the LVM tools into the chroot to build the initramfs to boot the system.
• We needed to edit /etc/fstab because the UUIDs have changed to point the machine at the correct partitions to boot from.

This is all fine and expected. We handed the machine to our customer, they were happy and it all seemed done. Two days later they came back, mysql on the new server was much slower than the previous server. They gave us a sample query and wondered why it answered in a fraction of a second on their development and current production server. We started investigating, the new server was running 100% CPU, no disk IO, the entire database source files cached so we hadn’t done something obviously stupid. We then started looking at the mysql query planner, it had a different query plan for the same query on the same database on two different machines.

How odd we thought.

Looking further at the indexes we discovered that cardinality of the indexes for the tables in question was different. Further investigation shows that the indexes are built differently if you do INSERT, INSERT, INSERT as the original database was created compared to a single multi-row insert that the restore from backup did.

The solution is to run

optimize table table_name

which finishes building the indexes properly, the query plan changes and the queries then run quickly. We have now added mysqloptimize –all-databases to our recovery procedure and handily put it into this blog post in the hope of saving someone else the investigation we’ve just done.

If your database backup procedure doesn’t have this step in and attempting to figure it out sounds hard you should consider our Managed hosting services. If investigating this sort of thing sounds like a fun way to spend your working days you should consider applying for a job.

Today we have made a number of changes Onza and Yali.

• Each user can now read their own web logs, which are stored in /home/user/www/log
• PHP scripts no longer need the execute bit set to run
• All scripts in cgi-bin directories have been moved to /home/user/cgi-bin to conform with Apache’s security recommendations

We hope that these changes will make it easier for people to run and debug their code.

We’ve updated our dedicated server range. Our entry level hardware RAID server has had a RAM and CPU boost to 2.5Ghz Ivy bridge and 16GB of RAM. Our RAID10 servers have been boosted to 32GB of RAM or 128GB of RAM and we’ve now got some machines with 256GB of RAM available.

We can also by arrangement now offer VMs with 120GB of RAM (we’re using the 128GB dedicated servers to expand the next phase of our virtual server platform).

We’re now running a mirror for the Raspberry Pi download server and the Raspbian apt repository on a Raspberry Pi.

The first problem was obtaining a Raspberry Pi as buying one was tricky: firstly the online shops were down then the queue for a Pi was rather long. As any banker can tell you occasionally crime does pay so I abducted Mooncake, a cat which owns Liz and Eben. I then ransomed the feline back in exchange for a Raspberry Pi.

The hardware setup starts with a power supply with an IEC14 connector connected to the masterswitch for remote power cycling. This PSU connects to a powered USB hub, with a USB lead connecting to the power connector on the Raspberry Pi. The Pi is then connected back to the hub on the data cable with a 1TB USB external hard disk attached to that. There’s a 100Mbit ethernet cable which connects up to the core Mythic Beasts network and out to the Internet. Technically the switch port on the other end is 1Gbit but the Raspberry Pi isn’t fast enough to use that.

The yellow fibre in the background is a very large Internet exchange with over a terabit of bandwidth, the Pi isn’t connected directly (that’d be too stupid even for us) and the packets travel from the Pi to the exchange a couple of feet away via another building. However with LONAP and LINX within 2ms and AMSIX a mere 8ms away it’s still rather well connected.

The software setup on the Pi is fairly straightforward. We started with the Debian squeeze image, ssh / apache enabled, munin enabled (graphs here). We’ve changed the password (obviously), the ssh keys (shipping the same ssh key on every OS image isn’t optimal), and moved /var/www and /var/log to the USB disk so as not to fill the SSD card. rpi-update was needed to make the USB/network setup stable under load as the initial image kept crashing. We also set the RAM split to 224MB for Linux as we really aren’t using the GPU.

It’s up and running both IPv4 (93.93.128.128) and IPv6 (2a00:1098:0:80:1000:13:0:3), as the core Raspbian server is also running IPv6 and the main Raspberry Pi server is also IPv6 at present this machine has seen more than 50% of its traffic over IPv6. I suspect this will change as people download images from it though.

At present the Raspberry Pi is devoting nearly half its CPU to drawing munin graphs so I need to benchmark the new Raspbian distribution to see if the hard float debian build improves the anti-aliasing, as presently all the calculations are done without using the FPU on the arm core in the Pi. Benchmarking suggests that we can deliver 35-50Mbps of file downloads reasonably comfortably at present.

Is this sensible? We’ve had a few customers ask us if the Raspberry Pi would be a sensible device for hosting on as it’s very cheap and very low power. Unfortunately it’s also very slow for this kind of application and the supporting hardware is very bulky. The i7 quad core mac mini occupies less space than the Pi + hub + disk + PSU, uses about fives times as much electricity, costs about five times as much once you include the supporting hardware but is hundreds of times faster. So revolutionising the hosting industry isn’t going to happen with the Raspberry Pi, at least not until they build a PoE one with gigabit ethernet and more RAM.

It’s not currently sensible to do this with shelves full of Raspberry Pis because the performance per Watt isn’t good enough. But we’re working on it.

Thanks go to Liam Fraser and Mike Thompson for adding us to the official mirror lists for the Pi and Raspbian. Additional thanks go to Eben and Liz for paying the ransom fee of one Raspberry Pi in exchange for the safe return of Mooncake who looks a lot like Raffles.

This evening I joined Eben & Liz Upton at the Nominet Internet Awards in my role as a contributor to the Raspberry Pi project. We were moderately surprised when the project won the Outstanding Contribution award. We’d like to offer our congratulations to everyone nominated for an award and to thank Nominet for an excellent evening.

People with excellent eyesight may be able to spot me in this photo taken by Dr Black.

All Mythic Beasts domain registrations include access to our DNS service, which includes a number of powerful features.  One of these is an API that allows you to programmatically make changes to the DNS records for your domain.  This has a number of uses including:

• Automating DNS-based failover
• Scripting the redirection of requests to another server to display a holding message during maintenance periods
• Implementing “dynamic DNS” so that you can have a static hostname for a dynamically allocated IP address

The DNS API provides a simple, HTTP-based API, that is designed to be trivial to script.  We’ve recently made some improvements to the API.  The first is that where multiple commands are included in a single request, they are treated as an atomic transaction, meaning that they either all succeed or all fail, and there is no risk of the changes being made live with only some of the commands completed.  This means that you can change a DNS record by including a DELETE and an ADD command in a single request.

The other change is that the API now makes use of HTTP error codes to reflect the results of the transaction.  If you get a 200 response, all of the commands succeeded.  If you get anything else, then none of the commands will take effect.

You can find full details of the API, including some sample Perl code on our support pages. It’s also pretty easy to script it on the command line. Here’s an example of how to change a CNAME record using cURL:

curl --data 'domain=example.com&password=s3cr3t&command=DELETE www 86400 CNAME live.example.com.&command=DELETE www 86400 CNAME backup.example.com.'   https://secure.mythic-beasts.com/control/fcgi/customer/primarydnsapi

There’s a well publicised bug in the linux kernel which makes it unhappy with leap seconds and usually Java and Mysql. If you’ve a linux box running at high CPU for no explicable reason after the leap second at the weekend you were probably affected.

The fix is,

/etc/init.d/ntp stop
date; date `date +"%m%d%H%M%C%y.%S"`; date
/etc/init.d/ntp start

This is what it did to the power monitoring in one of our racks with a large number of affected machines.

There is a reason we allocate the whole power allocation for a server and not just the typical idle usage.

100 years ago Alan Turing was born, in his sadly short lifetime he achieved a number of fairly impressive things, broke the enigma cipher, invented modern computing and invented the Turing test. On his day off he ran up and down the river in Cambridge and narrowly missed a place in the Olympic marathon team.

We haven’t done anything quite as impressive as that, although to be fair I don’t think anyone else in the last century has either. However some time ago at the Cambridge Beer Festival team Mythic entered a code cracking challenged based on a piece of ciphertext printed on a poster.

BLEQXVXEKPNRMWBRMVMVERNMWMZNQTTMRKMTMRHP

This is a substitution cipher which team Mythic cracked on the first day. You can decipher it with this perl one liner,

echo ‘BLEQXVXEKPNRMWBRMVMVERNMWMZNQTTMRKMTMRHP’ | perl -pe ‘tr/NMWZQTRKHPEBLXV/sonfidtcukaemlb/;’

which outputs

emailblackstonetobobatsonofsiddotcodotuk

We did, and we now have to get onto the more difficult task of finding a convenient day to have a barbeque and drink the beer we won.

If you think that working for a company that goes to beer festivals and solves problems by writing little bits of code is fun our jobs page is here: Mythic Jobs. If you’re running a web facing company and thinking that this is terribly clever you might want to think about our managed hosting.