Programmer art, just say no.

If you’ve had a look at the Raspbian website today you’ll have noticed the big red !!!FAILOVER TEST!!! logo at the top right corner. That’s because today is officially unimportant for Raspberry Pi, whereas in three weeks time it will be officially very important. Historically Christmas day sees our highest traffic loads as people unwrap their new Raspberry Pis and try them out. The most critical things for us to worry about are some of the educational and getting started resources on the website, and Raspbian and the mirror director so people can download new packages for their existing Raspberry Pis.

The majority of the website has a relatively small amount of data, so pulling an image from backup and redeploying is a relatively quick operation. Raspbian however is a bit harder – it’s a big image with around 4TB of data.

So we picked today to schedule a failover of Raspbian from it’s normal dedicated server to a VM hosted in the Raspberry Pi cloud. This is aiming to check

• Is the failover server up to date and does it work?
• Is the failover setup fast enough to keep up with the traffic load?
• Does every service successfully fail over?

So far we’ve had a very smooth operation, we’ve had to add a couple of missing packages that had been overlooked during setup and testing, but basically we did a DNS flip and the whole site moved over.

If you like to discover that your disaster recovery system works before you have a disaster, have a look at our Managed Services or get in touch – sales@mythic-beasts.com.

Pi Zero, 2p for scale, not included with prize.

Thanks to a recent visit to Pi Towers, we’re in possession of a very difficult to get hold of Raspberry Pi Zero. Within Mythic Beasts we don’t have an immediate need for a Pi Zero, so we thought we’d give it away to someone more deserving. So here’s a competition.

Our recruitment page frequently foxes experienced programmers but teenagers often have little difficulty,

http://sphinx.mythic-beasts.com/cgi-bin/job.pl

This competition is open until 9th December at 17:00. In order to enter you must have been born on or after 1st September 1997. Send us a successful answer, and we’ll pick the one we like best and send the winner a Pi Zero.

People over the age of 18 will have to be satisfied merely with the respect of their peers and can go buy their own Pi Zero, for example from our customer Pi Supply.

Detailed Rules


SPONSOR

A number of people noticed that Raspberry Pi had launched their $5 Pi Zero yesterday. We had advance warning that something was going to happen, even if we didn’t know exactly what. When the Pi2 launched we had some difficulties keeping up with comment posting and cache invalidation. We gave a very well received talk on the history and launch at the UK Network Operators Forum which you can see below.

• Containment: One part of the site can’t significantly impact the performance of another.
• Scalability: We can pull VMs into our public cloud and duplicate them if required.
• Flexibility: We no longer have to have a single software stack that supports everything.

For the Pi 2 launch we sustained around 4500 simultaneous users before we really started struggling with comment posting and cache invalidation. So our new plan was to be able to manage over 5,000 simultaneous site users before we needed to start adding more VMs. This equates to around 1000 hits per second.

In order to do this, we need to make sure we can serve any of the 90% of the most common requests without touching the disks or the database; and without using more than 10ms of CPU time. We want to reserve all our capacity for pages that have to be dynamic – comment-posting and forums, for example – and make all the common content as cheap as possible.

So we deployed a custom script staticify. This automatically takes the most popular and important pages, renders them to static HTML and rewrites the webserver configuration to serve the static pages instead. It runs frequently so the cache is never more than 60 seconds old, making it appear dynamic.  It also means that we serve a file from filesystem cache (RAM) instead of executing WordPress. During the day we improved and deployed this same code to the MagPi site including some horrid hackery to cache popular GET request combinations.

Some very vague back-of-the-envelope calculations give us:

It’s fair to say that we exceeded our target of 5,000 simultaneous users,

Liz Upton was quite pleased:

Not to mention a certain amount of respect from our peers

If you deployed the blog unoptimised to AWS and just had auto-magic scaling, we’d estimate the monthly bills to be many tens of thousands of dollars per month, money that instead can be spent on education. In addition you’d still need to make sure you can effortlessly scale to thousands of cores without a single bottleneck somewhere in the stack causing them all to lie idle. The original version of the site (with hopeless analytics plugin that processed the complete site logs on every request) would consume more computer power than has ever existed under the traffic mentioned above. At this scale optimisation is a necessity, and if you’re going to optimise, you might as well optimise well.

That said, we think some of our peers possibly overstated our importance in the big scheme of things,

A scary beast that lives in the Fens.

Fenrir is the latest addition to the Mythic Beasts family. It’s a virtual machine in our Cambridge data centre which is running our blog. What’s interesting about it, is that it has no IPv4 connectivity.

eth0 Link encap:Ethernet HWaddr 52:54:00:39:67:12
     inet6 addr: 2a00:1098:0:82:1000:0:39:6712/64 Scope:Global
     inet6 addr: fe80::5054:ff:fe39:6712/64 Scope:Link
     UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

It is fronted by our Reverse Proxy service – any connection over IPv4 or IPv6 arrives at one of our proxy servers and is forwarded on over IPv6 to fenrir which generates and serves the page. If it needs to make an outbound connection to another server (e.g. to embed our Tweets) it uses our NAT64 service which proxies the traffic for it.

All of our standard management services are running: graphing, SMS monitoring, nightly backups, security patches, and the firewall configuration is simpler because we only need to write a v6 configuration. In addition, we don’t have to devote an expensive IPv4 address to the VM, slightly reducing our marketing budget.

For any of our own services, IPv6 only is the new default. Our staff members have to make a justification if they want to use one of our IPv4 addresses for a service we’re building. We now also need to see how many addresses we can reclaim from existing servers by moving to IPv6 + Proxy.

Dear LazyWeb,

Yesterday we did a routine disk replacement on a machine with software RAID. It has two mirrored disks, sda and sdb with a RAID 1 partition with software RAID, /dev/md1 mirrored across /dev/sda3 and /dev/sdb3. We took the machine offline and replaced /dev/sda. In netboot recovery mode we set up the partition table on /dev/sda, then set the array off rebuilding as normal:

mdadm --manage /dev/md1 --add /dev/sda3


This expects to take around three hours to complete, so we told the machine too boot up normally and rebuild in the background while being operational. This failed – during bootup in the initrd, the kernel (Debian 3.16) was bringing up the array with /dev/sda3, but not /dev/sdb3, claiming it didn’t have enough disks to start the array and refusing to boot.

Within the initrd if I did:

mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3


the array refused to start claiming that it didn’t have sufficient disks to bring itself online, but if I did:

mdadm --assemble /dev/md1 /dev/sdb3
mdadm --manage /dev/md1 --add /dev/sda3


within the initrd it would bring up the array and start it rebuilding.

Our netboot recovery environment (same kernel) meanwhile correctly identifies both disks, and leaves the array rebuilding happily.

To solve it we ended up leaving the machine to rebuild in the network recovery mode until the array was fully redundant at which point the machine booted without issue. This wasn’t an issue – it’s a member of a cluster so downtime wasn’t a problem – but in general it’s supposed to work better than that.

It’s the first time we’ve ever seen this happen and we’re short on suggestions as to why – we’ve done hundreds of software RAID1 disk swaps before and never seen this issue.

Answers or suggestions in an email or tweet.

With today being Back To The Future day, it’s worth reflecting on two pieces of advice I received in the mid 1980s. The best piece of advice was definitely from the film:

‘If you put your mind to you, you can accomplish anything’.

the worst was my mother:

‘Stop playing on your Spectrum and go and do your piano practice’

I’m not certain this generalises, I think that largely your parents do give better advice than Hollywood script writers.

it’s a server graph!

One of the outstanding tasks for full IPv6 support within Mythic Beasts was to make our graphing server support IPv6 only hosts. In theory this is trivial, in practice it required a bit more work.

Our graphing service uses munin, and we built it on munin 1.4 nearly five years ago; we scripted all the configuration and it has basically run itself ever since. When we added our first IPv6 only server it didn’t automatically get configured with graphs. On investigation we discovered that munin 1.4 just didn’t support IPv6 at all, so the first step was to build a new munin server based on Debian Jessie with munin 2.0.

Our code generates the configuration file by printing a line for each server to monitor which includes the IP address. For IPv4 you print the address as normal, 127.0.0.1, for IPv6 you have to encase the address in square brackets [2a00:1098:0:82:1000:0:1:1]. So a small patch later to spot which type of address is which and we have a valid configuration file.

Lastly we needed to add the IPv6 address of our munin server into the configuration file of all the servers that might be talked to over IPv6. Once this was done, as if by magic, thousands of graphs appeared.

Unless you really like maths, you’re probably better off just looking at the pictures.

Today is Ada Lovelace day, where we celebrate the achievements of women in the traditionally male dominated fields of Science, Technology, Engineering and Maths.

Mythic Beasts came about as a side project of a bunch of students, most of whom studied at Clare College Cambridge. As a Cambridge student you receive supervisions, hour long tutorials in sets of two or three. You also live in the college for three years and some fellows of the college also have rooms in the same accommodation. As luck would have it, our director Pete was partnered with one of our other founders, Richard for supervisions in Physics, and in the second year they were jointly supervised by Professor Cathie Clarke and it turned out that her college room was directly opposite Pete’s in Clare Old Court.

This led to a slightly unusual arrangement, rather than everyone trekking into the department for supervisions they decided to hold them in Pete’s room at 8am, usually accompanied by a bacon sandwich and strong black coffee before heading off to lectures. Cathie was a superb teacher, neatly covering dynamics, orbits and effortlessly showing why practically the whole of spaceflight involves pointing your rocket motor in obviously the wrong direction in order to get to where you want to go. She also, neatly answered other questions including electro-magnetism that had stumped pretty much the whole year in Clare and all the supervisors in that subject, despite having had about sixty seconds notice and only half a cup of coffee.

Pete’s room was on the top right hand corner of this photograph, the bacon cooking kitchen roughly above the passageway.

There was a particularly memorable supervision, where Richard overslept and arrived very late to Pete’s room, anxious he’d missed the supervision. On waking Pete up, they jointly discovered the note on the door from Cathie apologising that she’d overslept and the supervision would need to be rescheduled. A co-incidence for which all parties were grateful.

So whilst her impressive CV gives away the huge publication list, professorship and that she’s the course coordinator for astrophysics; in person we had the privilege of knowing that she is also a superb prize-winning teacher, gifted researcher and somehow on top of all that, a lovely human being who occasionally oversleeps just like the rest of us.

We’re still installing iOS9 for testing reasons onto this Apple Device

tl;dr iOS9 applications only work with the newest SHA-256 certificates. If your iOS9 application or website is showing certificate errors and you’d like some help, contact support@mythic-beasts.com

iOS9 was recently released which brings a number of changes. In addition to the widely publicised changes about IPv6 (iOS9 prefers IPv6 and all applications in the App Store must function without issue on an IPv6 only network), Apple has forced obsolescence of older types of SSL certificate.

SSL certificates use hashing functions to provide security. The Secure Hash Algorithm 1 (SHA-1), was published by the NSA in 1995 as the standard for secure authentication. The first theoretical attacks were shown in 2005 leading to a recommendation in 2010 that we abandon SHA-1 and move to SHA-256. In 2014 Google put a sunset date for SHA-1 of December 2016 – if your website trusts an SHA-1 certificate past this date Chrome refuses to regard your site as secure.

With iOS9, Apple pulled the date at which everyday software stops working with SHA-1 forward. If your website or application is secured with a SHA-1 certificate, iOS9 gives warnings and errors. The fix is easy, we can provide or re-issue your existing certificate with an iOS9 compatible – and more importantly more secure – SHA-256 certificate.

September 2015: 2.5% of the UK has native IPv6

Yesterday was the second meeting of the UK IPv6 Council. Eleven months ago Mythic Beasts went along to hear what the leading UK networks were doing about IPv6 migration. Mostly they had plans, and trials. However, the council is clearly useful: Last year, Nick Chettle from Sky promised that Sky would be enabling IPv6 in 2015. His colleague, Ian Dickinson gave a follow-up talk yesterday and in the past two months UK IPv6 usage has grown from 0.2% to 2.6%. We think somebody had to enable IPv6 to make his graph look good for today’s presentation…

In the meantime, Mythic Beasts has made some progress beyond having an IPv6 website, email and our popular IPv6 Health Check. Here’s what we’ve achieved, and not achieved in the last twelve months.

Customer Facing Successes

• IPv6 support for our control panel.
• IPv6 support for our customer wiki.
• Offered IPv6-only hosting services that customers have actually bought.
• Added NAT64 for hosted customers to access other IPv4 only services.
• Added multiple downstream networks, some of which are IPv6 only.
• Raspberry Pi has a large IPv6 only internal network – 34 real and virtual servers but only 15 IPv4 addresses, and integrations with other parts of their ecosystem (e.g. Raspbian) are also IPv6.
• IPv6 for all DNS servers, authoritative and resolvers
• IPv6 for our single sign on authentication service (this was one of the hardest bits).
• Our SMS monitoring fully supports IPv6 only servers (this was quite important).
• Our backup service fully supports IPv6 only servers (this was very important).
• Direct Debits work over IPv6, thanks to GoCardless

Internal Successes

• IPv6 on our own internal wiki, MRTG, IRC channels
• Full Ipv6 support for connectivity to and out of our gateway server.
• IPv6 rate limiting to prevent outbound spam being relayed.
• Everything works from IPv6 with NAT64.

Prototypes

• Shared IPv4/IPv6 load balancer for providing v4 connectivity to v6 only hosted services.

Still to do

• Our card payment gateway doesn’t support IPv6.
• Our graphing service doesn’t yet support IPv6-only servers (edit – implemented 15th October 2015).
• Automatic configuration for an IPv6-only primary DNS server which slaves to our secondary DNS service.
• Billing for IPv6 traffic.
• One shared hosting service still has incomplete IPv6 support. One shared hosting service has optional instead of mandatory IPv6 support.
• Automatic IPv6 provisioning for existing server customers.
• Make sure everything works from IPv6 with no NAT64.

Waiting on others

• The management interfaces for our DNS wholesalers don’t support IPv6.
• Nor our SSL certificate providers.
• Nor our SMS providers.
• Nor our card payment gateway.