Nid wyf yn y swyddfa ar hyn o bryd. Anfonwch unrhyw waith i’w gyfieithu.

NGINX version

First issue: you need NGINX >= 1.9.10, as there was a bug with using proxy_protocol on IPv6 listeners. If you’re on Debian Jessie, you can get a suitable version from Jessie backports.

PROXY protocol version

Second issue: NGINX only speaks PROXY protocol v1 and our proxy was attempting to speak v2.

v1 is a human readable plain text protocol, whereas v2 is binary. If you see something like this in the error log:

2016/05/09 11:11:30 [error] 6058#6058: *1 broken header: "

QUIT
!
 ]Y??.????PGET / HTTP/1.1

Then that’s a good sign that you’ve got a v2 reverse proxy talking to you.

We’ve now changed our proxy to only speak PROXY protocol v1 by default. We will look into making this a configurable option in the future. The Apache module seems happy speaking either version.

Whilst we’re here, here are some other failure modes you might see. This in the access log, is v2 PROXY protocol being spoken to NGINX which is not configured for PROXY protocol at all.

2a00:1098:0:82:1000:3b:1:1 - - [09/May/2016:11:08:55 +0100] "\x00" 400 172 "-" "-"

And this is v1 PROXY protocol being spoken to NGINX which is not configured for it:

2a00:1098:0:82:1000:3b:1:1 - - [09/May/2016:11:39:30 +0100] "PROXY TCP4 93.89.134.240 46.235.225.189 64221 80" 400 173 "-" "-"

We’re increasingly using our IPv4 to IPv6 reverse proxy to host websites on IPv6-only virtual machines. One of the downsides of proxying is that your server doesn’t get to see the client’s real IP address. For non-SSL connections, the proxy can insert an “X-Forwarded-For” header, but SSL is increasingly becoming the norm, and one of the nice things about an SNI-aware reverse proxy is that it doesn’t need to do SSL off load: we don’t need your certificates on our proxy and your traffic stays encrypted until it hits your server. Of course, this means that we can’t go inserting any headers into your connection either.

Fortunately, there is a solution: PROXY protocol. This is a protocol-agnostic mechanism for passing information from a reverse proxy to a server, including the client IP address.

We’ve just added support for PROXY protocol to our reverse proxy:

Turning this on allows your server to get the client IP address, but as it’s an additional protocol, not part of HTTP, your server must be expecting it: turning this on and pointing it at a standard HTTP server will result in a broken website.

Most web servers have support for this. NGINX has support built in, and just needs “proxy_protocol” adding after the listen directive:

server {
    listen 80   proxy_protocol;
    listen 443  ssl proxy_protocol;
    ...
}

You will probably also want some additional configuration to actually set the IP address that gets used for logs etc., and also to ensure that you only trust proxy information from the real proxy servers.

For Apache, support is provided by mod_proxy_protocol, which needs to be installed manually. Once done, configuration is easy:

<VirtualHost *:443>
  ...
  ProxyProtocol On

  CustomLog ${APACHE_LOG_DIR}/access.log "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

The CustomLog line instructs Apache to use the real client IP rather than the proxy. You should now see v4 addresses being happily logged on your IPv6 server:

root@vm1:~# tail -n 1 /var/log/apache2/access.log
93.93.130.44 - - [29/Apr/2016:14:05:32 +0100] "GET / HTTP/1.1" 200 321 "-" "curl/7.26.0"

Unfortunately the module doesn’t currently provide a way to restrict enablement to trusted proxies only. As such, you’ll probably want to install a firewall to restrict HTTP/HTTPS traffic to only come from our proxies, as otherwise clients could easily fake their IP address.

One thing to watch out for is that although this is applied within a VirtualHost configuration, it’ll actually apply to all virtual hosts on the same IP address and port. This is an unavoidable side effect of the fact that the proxy information is sent before we start talking HTTP. Of course, with IPv6, throwing another IP address at the problem isn’t an issue.

Last week at the UK Network Operators Forum Pete gave a talk about our IPv6 only hosting, progress we’ve made and barriers we’ve overcome.

It’s now available to view online

Let’s Encrypt on a v6-only host

One of the much requested features for Let’s Encrypt free SSL certificates is support for IPv6-only hosts. Whilst this is promised in the very near future we’re happy to say that IPv6-only hosts behind our NAT64 & Proxy services work out of the box with Let’s Encrypt.

To test it we took the traditional dogfood approach, this website is run on an IPv6-only VM and we’ve just enabled Lets Enrypt SSL support on our own blog. As soon as Let’s Encrypt offer SSL certificates for IPV6-only hosts with no proxy and no NAT64 we’ll give that a try too.

DNS-based domain validation (dns-01)

An alternative approach would be to use dns-01 validation using our DNS API. Our API speaks native v6, so that should work just fine on a truly single-stack IPv6 host.

It’s with great sadness we learned of the death of Prof Sir David MacKay, FRS. He taught three of the Mythic Beasts founders information theory in 1999–2000, a fascinating and stunningly well-lectured course. The textbook Information Theory, Inference, and Learning Algorithms is freely available to download. Prof MacKay believed it was possible to make the world a better place.

Energy policy done by the numerate.

David’s other textbook Sustainable Energy, is not only fascinating, but vital reading for anyone interested in energy policy. It’s excellent, not because it provides answers, but because it teaches the tools to work things out. Can we power the UK only from wind energy? No. Can it make a substantial contribution? Yes, If we built lots and lots of wind how much can it contribute? Hopefully around 25% if you include a lot of offshore.

If more people read and understood the book we’d have fewer articles about the amazing solar panels that create power from rain, when the abstract tells us they create micro amps and milli-volts, throwing away 180W/m² of solar to create 0.000000001W/m² from rain.

David’s website withouthotair.com is prepaid far in advance and will remain available for many years to come.

At the end of March we migrated the Raspberry Pi website from a very big multi-core server to a tiny cluster of eight Raspberry Pi 3s. Here’s a bit more detail about how it worked.

The Pi rack not fooling anyone on April 1st

Booting

For the Raspberry Pi 3 launch we tried out some Pis running in a data centre environment with high load using the SD card for the root filesystem. They kept crashing, if you exceed the write capability of the card the delays make the kernel think the storage has failed and the system falls over. We also want to be able to remotely rebuild the filesystem so we can fix a broken Pi remotely. So we’ve put the root filesystem on a network file server, which is accessed over NFS.

The Raspberry Pi runs the latest kernel, 4.1.18-v7+ and boots from the SD card with a configuration as follows:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/nfs rootfstype=nfs
  ip=10.46.189.2::10.46.189.1:255.255.255.252::eth0:off 
  nfsroot=10.46.189.1:/export/10.46.189.2 elevator=deadline 
  fsck.repair=yes rootwait

This brings up a block of 4 IP addresses on eth0. One address for the network, one for broadcast, one for the Pi and one for the network fileserver. It then mounts the NFS filesystem at:

nfsroot=10.46.189.1:/export/10.46.189.2

and uses that as the root filesystem.

Overly simple introduction to VLANs

On a traditional switch, you plug things and any ethernet port can talk to any other ethernet port. If you want to have two different networks you need two different switches, and any computer that needs to be on both networks needs two network ports. In our case we’re trying to have a private network for storage for each Raspberry Pi, so each Pi requires its own switch and the fileserver needs it’s own network port for every Raspberry Pi connected to keep them separate. This is going to get expensive very quickly.

Instead we turn on virtual LANs (VLAN). We connect our fileserver to port 24 and create a VLAN for ports 1 & 24, another for 2&24, etc. The switch configuration for the fileserver port specifies these VLANs as “tagged”, meaning our switch adds a header to the front of every packet from a Raspberry Pi port that allows the fileserver to tell which VLAN, and therefore which Raspberry Pi, the packet came from. The fileserver can reply with the same header, and that packet will only be sent to that specific Raspberry Pi. It behaves as if each Raspberry Pi has its own switch.

Network on the fileserver

The fileserver sees each VLAN as a separate network card, named eth0.N where N identifies the VLAN. We can configure them like any other network interface:

auto eth0.10
iface eth0.10 inet static
	address 10.46.189.1
	netmask 255.255.255.252

auto eth0.11
iface eth0.11 inet static
	address 10.46.189.5
	netmask 255.255.255.252

eth0.10 and eth0.11 appear to be network cards with a tiny network with one Raspberry Pi on the end, but in reality there’s a single physical ethernet connection underneath all of them.

Network on the Raspberry Pi

On the Raspberry Pi, eth0 is already configured on the Raspberry Pi by the boot line above to talk to the fileserver. On our switch configuration, we specify that private network is “untagged” on Raspberry Pi port, which means that it won’t have a VLAN header on it and we can access it as “eth0” rather than “eth0.N” as we did on the fileserver.

In order to do anything useful, we also need to give the Raspberry Pis access to the public network. On our network, the public network is accessible on VLAN 131. We configure this to be a “tagged” VLAN on the Raspberry Pi port, meaning it becomes accessible on the eth0.131 interface. We can configure this in the normal way, and in keeping with other back-end servers on the Raspberry Pi setup, it only has an IPv6 address:

auto eth0.131
iface eth0.131 inet6 static
	address	2a00:1098:0:84:1000:1::2
	netmask 64
	gateway	2a00:1098:0:84::1

Effectively the Raspberry Pi believes it has two network cards, one on eth0 which is a private network shared with the fileserver, one on eth0.131 which has an IPv6 address and is connected to the real internet.

Why all that configuration?

In an ideal world we’d have a single IPv6 address for each Pi, and mount the network filesystem with it. However, with an NFS root filesystem, potentially another user on the LAN who can steal your IPv6 address can access your files. There’s a second complication, IPv4 is built into the standard kernel on the Raspberry Pi and the differences per Pi are constrained to just the kernel command line, with IPv6 we’d have to build it into an initrd which would load up the IPv6 modules and set up the NFS mounts.

Planning for the future we’ve spoken to Gordon about how PXE boot on the Raspberry Pi will work and it’s extremely likely that it’s going to require IPv4 to pull in the bootloader, kernel and initrd. Whilst there is native IPv6 in the Raspberry Pi office, there isn’t any IPv6 on their test lan for developing the boot code and it’s a currently not a major priority for the Pi despite around 5% of the UK having native IPv6.

So if we want to make this commercial, each Pi needs its own storage network and it needs IPv4 on the storage network.

Power over Ethernet

We’ve added a Power over Ethernet HAT to our Raspberry Pis. This means that they receive power over the ethernet cable in addition to the two separate networks. As well as reducing the amount of space used by power bricks, it also means you can power cycle a Raspberry Pi just by re-configuring the switch.

Software

Each Raspberry Pi runs Raspbian with Apache2 installed. We’ve pulled in PHP7 from Debian Stretch to improve PHP performance and then copied all the files for the Raspberry Pi website onto the NFS root for each Raspberry Pi (so the fileserver effectively has 8 copies – one for each Pi). We then just added the IPv6 addresses of the Raspberry Pis into the site’s load balancer, deleted the addresses for the main x86 servers and waited for everything to explode.

Did it work?

Slightly to our surprise, yes and well. We had a couple of issues – the Pi is much slower than the x86 servers, not only clock speed but also the speed of the network card used to access the filesystem and the database server. Some rarely used functions, such as registering a new Raspberry Jam, weren’t really quick enough under the new setup and gave people some error pages as the connections timed out. Uploading images for new WordPress posts was similarly an issue as receiving a 3MB file and distributing eight copies on a 100Mbps network isn’t very fast. But mostly it worked.

Did power cycling the Pis via the switch work?

We never tested it in production, every Pi remained up and stable for the whole 3.5 day duration we had the system in use. In testing it’s been fine.

Can I buy one?

Not yet. At present you can still break a Pi by destroying the flash, and the enclosure doesn’t allow for replacement without taking the whole shelf (which in production would contain 96 Pis) offline. Once we have full netboot for the Pi, it is a service we could offer.

Can I register my interest to buy a Pi in the cloud?

Sure – email us at sales@mythic-beasts.com and we’ll add you to a list to keep you up to date.

Here at Mythic Beasts, we’ve been busily undermining sales of our SSL certificates by rolling out support for free certificates from Let’s Encrypt, partly because we think that the internet should be secure by default, but mostly because we’re lazy and Let’s Encrypt makes it easy to fully automate certificate issue and deployment.

Domain validated certificates

The majority of SSL certificates in use today are “Domain Validated” certificates. These are issued automatically by a certificate authority once you have completed some action that proves that you are in control of the domain for which the certificate is being requested. This can include responding to an email send to an address at your domain, or posting a file to a specific location on your website.

Let’s Encrypt DNS challenge

One of the options for validation offered by Let’s Encrypt is a DNS challenge (known as “dns-01”), whereby you prove ownership of your domain by adding a specific entry to its DNS zone. This option is quite interesting, as it allows you to avoid meddling in any way with your web server configuration and, if your DNS is hosted with Mythic Beasts, you can automate the addition of the necessary records using our DNS API.

Automating via our DNS API

In order to support this, we’ve developed a hook script that works with the letsencrypt.sh client.

We’ve also written a step-by-step guide to configuring dns-01 validation using our DNS API.

Please note, if you’re a hosting account customer, you don’t need to worry about any of this. You can get an SSL certificate for your website simply by hitting a button in the control panel.

Thanks go to David Earl for testing this and providing the initial implementation of the hook script..

We’ve spent even more time than usual in data centres recently as we’ve been kitting out our new cage in the Meridian Gate data centre.

Much of the new capacity is being deployed as “managed racks”.  Racks are generally supplied with the bare essentials of electricity, cooling and locked doors.  At Mythic Beasts, we transform them into managed racks, including all the features you need to effectively administer your equipment remotely, including:

Logging serial consoles

• Internet connectivity – we’ve got 10Gbps connections onto both LINX networks, connecting at different sites.  We’ve also got multiple transit providers, and are present on the LoNAP peering exchange.   Our network has native IPv6 support, and if you have your own address space, we can provide you with BGP feeds from our routers. We can also offer private LANs, both as VLANs or as physically separate networks.
• Remote power management – power cycle your server immediately, at any time using our customer control panel.
• Serial connectivity – a 115.2kbps serial connection may seem a bit old fashioned in an age when we’re wiring our switches together at 40Gbps, but they remain an extremely effective mechanism for out-of-band control of servers and other equipment, particularly when coupled with our logging serial console software.
• On-site support – all of our London facilities have 24/7 access to the data centres’ on-site engineers.  We are also able to arrange for our own staff to carry out routine maintenance, such as replacing failed hard drives.

Meridian Gate is the third London data centre in which we have a presence, along with Sovereign House and Harbour Exchange, with the three sites connected by our own dark fibre ring.

After the Raspberry Pi 2 launch in February 2015, we had a review of how to improve and scale the hosting setup for Raspberry Pi. There were two components that caused us pain during the Pi 2 launch: the main site, running WordPress, and the forums, powered by phpBB.

The first question from our review was whether we should be putting effort into scaling a WordPress site. WordPress is estimated to be powering as many as a quarter of all websites, and it’s popular for a reason: it makes site development very easy. WordPress is easily extensible through themes and plugins, it’s supported by a vast array of existing third party plugins, and it provides a good built-in framework for delegating and moderating authoring roles.

Unfortunately, this ease of development is at least in part down to a very simplistic execution model, with each page being dynamically generated, executing code from every installed plugin, and typically resulting in multiple database queries. When the Raspberry Pi site gets busy, it’s usually down to a huge number of visitors hitting just a page which is essentially a static news story. WordPress provides no built-in mechanism for caching such content, so by default, we’re dynamically generating many copies of identical, or near identical pages.

Losing the flexibility and ease of development offered by WordPress just to cope with the handful of days when the site gets very busy would be unfortunate, so we decided to put effort into making the existing site scalable.

Caching

For pretty much every WordPress problem you can imagine, there’s at least one plugin offering to solve it for you. For site performance, there are a number of plugins such as WP Supercache, but as WordPress itself provides no framework for identifying cacheable parts of page, these can only take a very simple and typically over-cautious, page-based approach.

For example, if you’re a logged in user, you might get served a page that is in someway tailored to you, so Supercache bypasses its cache and serves you a dynamic page. Similarly, if Supercache sees a request that looks like a comment being posted, the cache is invalidated, and a dynamic page is served, and cached for future requests.

During the Pi 2 launch, we saw significant problems with load spikes when comments were posted. Clearly, small delays in comments being visible on the site is a minor annoyance compared to thousands of visitors being served an error page, so we set about making our caching more aggressive.

We wrote a small hack called staticify. This fetches the key pages from the blog every 60 seconds and renders them to static HTML. That way we always have a page in our static cache, and because we’re selecting the pages that we cache we can afford to be more brutal with our caching: we know that there’s no user-specific content on these pages, so we serve up the same cached page even if you’re logged in.

More virtualisation

An important goal after the Pi 2 launch review was to split out different parts of the site onto separate virtual servers. For example, having the WordPress blog and the forums software on different VMs made it much easier to experiment with using Hip Hop VM which offered a significant performance boost to the blog, but is incompatible with the forum software.

Although the Raspberry Pi setup runs as a private cloud on a single host machine, having different components split onto separate VMs makes it much easier to balance resources between them, and if necessary, spin up extra capacity quickly using our public cloud.

IPv6

When Raspberry Pi was hit by DDOS we built an IPv6-only backend network for the machines to communicate with each other. In the new setup all access to the back-end VMs comes from either one of four front-end load balancers, or a “gateway” VM. So we thought we’d remove IPv4 connectivity from the VMs entirely. For example this is ifconfig on one of the blog PHP VMs :

eth0      Link encap:Ethernet  HWaddr 52:54:00:3f:8a:5a  
          inet6 addr: 2a00:1098:0:82:1000:x:y:z/64 Scope:Global
          inet6 addr: fe80::5054:ff:fe3f:8a5a/64 Scope:Link

The VM needs to occasionally call out over IPv4. For example, akismet and Twitter don’t yet have full IPv6 support, so these requests go through a NAT64 gateway, provided by Mythic Beasts that proxies the connections so it appears almost seamless to the VM. This is part of the Mythic Beasts IPv6 education project, backward ISPs claim there is no demand for IPv6, whereas we provide multiple services from IPv6-only servers and give discounts if you use IPv6-only services.

SSL

Officially we enabled SSL because we wanted to improve our Google ranking but handy side effects include irritating the security services and preventing third party networks injecting adverts or corrupting downloads. The SSL decryption is done on the front-end load balancers and as they have lots of spare CPU incurs no performance issue. The only thing that isn’t is the image downloads because of incompatibilities with the current version of NOOBs. We hope to eventually resolve this.

Pi Zero Launch

November 26th at 7am, the Pi Zero is launched, a $5 computer given away on magazines. The bandwidth graph for the Raspberry Pi server does this:

Launch day bandwidth graph

It’s very exciting and quickly exceeds our previous records for the launch of Pi 2. The two VMs that generate all the webpages for the blog and deliver all the content are humming along at 10-25% capacity. The database VM is almost completely idle, we’ve successfully cached almost everything,our database server only sees load when a comment is being posted or the cache is being updated. Meanwhile we neatly exceed the 4500 users we had for the Pi2 breaking 10,000 simultaneous users at our peak.

A quick back-of-the-envelope calculation and we conclude that our staticify script avoided executing WordPress a large number of times and the following slightly dubious claim is mostly true:

The MagPi site was a bit more difficult, it hadn’t had the same level of optimisation and went through a number of changes throughout the day to accelerate it. However, the VM setup meant that the excess load was contained to specific virtual machines- under our original flat hosting setup the load from the MagPi would have taken everything offline and made identifying the underlying cause much harder.

Raspbian

We now run a full mirror of the main Raspbian site, and we’ve even done a test to make sure that the failover works.

Scheduled Raspbian hosting failover test in operation. Mail support@mythic-beasts.com if you find anything b0rked.

— Mythic Beasts (@Mythic_Beasts) December 6, 2015

The mirror director is a critical piece of infrastructure, without it package downloads will fail and updates can’t complete. So in the event of a failure we need to bring the mirror director back up much more quickly than we can restore 4TB+ of data from backup. As a result of this work we now have a hot spare, which has been fully tested.

Does it work and is WordPress still a good idea?

We weathered the PiZero and Christmas Day traffic peaks with ease and we think we can probably double or triple the number of people using the sites at peak times before we have to think much more or add hardware. The result is we’ve a really useful and very busy site, that supports our multiple contributors, moderators and users with a relatively minimal amount of engineering and administration time, on a comparatively small server setup.

It’s been a long time coming, but we’re now pleased to announce that we’ve got DNSSEC support in public beta, and you can enable it for your domain at the click of a button.

What is DNSSEC?

DNSSEC is a set of extensions to the DNS protocol that ensures that you can trust the IP addresses that you get back from the DNS system. For example, if you visit www.yourbank.com, the first thing that happens is that your browser uses a DNS server to find out the IP address of your bank’s web server. But how do you know that you can trust the address that you get back? Your request will probably get bounced through multiple DNS servers, such as your home router, your ISPs servers, and finally the authoritative server for the domain. If any one of those gets compromised (and let’s face it, home routers have a terrible security record) it could easily insert a different IP address and direct your request to an entirely different server.

DNSSEC means that all responses are signed with encryption keys that have been lodged with the registry, so you can’t inject bogus responses just by compromising an intermediate server.  Of course, the system only works if the systems making the requests check the signatures of the responses that they receive, something which certainly doesn’t happen everywhere yet.

Sounds complicated?

Yes it is, particularly as it is recommended that the encryption keys that you use are changed (or “rotated”) regularly. Fortunately, we’ve now automated all the hard stuff, and if you’ve got your domain registration and DNS hosting with Mythic Beasts, you can make DNSSEC go just by hitting a big green button.  We’ll take care of the rest:

Unlike some people, we believe that the internet should be a safe place to do business by default, so this service is, and will continue to be, provided at no extra cost.

If you want to try it out, simply visit our control panel, find the domain under “My Domains” and follow the “DNSSEC” link.