Some months ago we announced that we were planning an even cheaper version of our entry-level VPS Lite virtual servers.

It took a little longer than planned, but we’re now pleased to announce the launch of our IPv6-only VPS Lite, for the ludicrously low price of £5+VAT/month (or even less if you pay annually). This virtual server comes with 4 billion IPv6 addresses, but no IPv4 connectivity.

The world is running out of IPv4 addresses, and whilst we’ve got an allocation that isn’t going to run out any time soon, issuing an IPv4 address with every server we sell will ultimately be a barrier to our growth. If you’re happy to have a server without an IPv4 address, we’re happy to give you a discount, and that’s exactly what we’ve done with our IPv6-only VPS Lite.

IPv6 usage is currently running at about 3% and doubling roughly annually, so now is a great time to get familiar with IPv6.

Whilst these servers probably aren’t suitable for fronting a website just yet, we’ve already got a number of customers who run IPv6-only backend networks, with everything behind their load balancers on single-stack IPv6.

Although these Virtual Servers themselves don’t have IPv4 connectivity, the host server does, which means that you can get to the admin console, virtual serial console and VNC over an IPv4 connection.

If you’re a technology professional and have no idea what any of this means, we suggest you start your training by placing an order here.

Given just how much of our lives involves using apps, websites or embedded computers, you might think that an initiative to teach children the basics of how to create software and not just how to use it would be uncontroversial.

Well, that’s the stated goal of Year of Code, and whilst you might be able to find fault with its execution, that’s not the angle that Jeremy Paxman chose to take when interviewing Year of Code’s director, Lottie Dexter, on Newsnight last week. Instead, Paxman decided to ridicule the very notion using the curious argument that because he couldn’t understand some computer code, it was therefore meaningless, and by implication, worthless. That he can’t get his head around what code is, let alone what it means, rather confirms the importance of this initiative.

Here at Mythic Beasts we have some pretty strong views on starting coding young, so Pete took his Raspberry Pi and attempted to explain what coding is in terms that even Jeremy might understand, through the medium of a musical e-Card and via a few lines of Perl that can generate just about every pop hit ever. To read Pete’s views, or just to see a video of him playing the piano, click here.

We’ve just rolled out some improvements to our monitoring service. All server products, including virtual servers, get access to our basic ping monitoring service, allowing you to receive SMS and email alerts if your server goes off-line. For £5/month you can add enhanced monitoring, which allows you to confirm that individual services such as a web server are working correctly. Customers of our Managed Hosting service not only get access to enhanced monitoring, but also get the benefit of having our staff respond to the alerts for them.

The new features are:

Temporary silencing of alerts

You can now silence alerts for a set period. This is handy if you’re doing maintenance, and don’t want to be constantly pestered by alerts, but also don’t want the risk of forgetting to turn the monitor back on again afterwards.

Prowl notifications

Prowl is a notification system for iOS, allowing you to receive alerts on an iPhone or iPad. The advantage of Prowl notifications over SMS is that they’re not limited to 141 characters, so we can include a more verbose message, including direct links for silencing the alert. They’re also delivered over the internet, rather than the mobile network, so will work if you have a wifi connection, but no mobile signal.

Support for an Android equivalent (such as Notify My Android) is on the to-do list.

Improved email alerts

The email alerts previously included the same dense text that we use for SMS alerts. The new style notifications are now more verbose, and include links for silencing the monitor, and in the case of web alerts, a link to the page that failed.

Monitoring of arbitrary TCP ports

We provide monitors for most common services, including HTTP, SMTP, IMAP and POP3. You can now also monitor any TCP port. This check simply confirms that the host is accepting connections on this port, and then closes the connection.

Edit: we’ve now received a report from Telecity, so have updated this report to take account of this.

Further edit: explanation for extended outage in one rack added.

Summary

• A power interruption occurred at around 8:09am on Saturday 25th January, affecting multiple floors in Sovereign House.
• For the most part, the interruption was momentary (around 500ms), but long enough to cause a reboot of affected equipment.
• One of our racks was without power until 10:38am, due to a tripped circuit breaker.
• Our staff were onsite at 11:15am, and then worked to restore services that had not come back up cleanly. One such server was our SOV DHCP server which will have affected any virtual servers configured to boot via DHCP.

Details

The power outage was caused by an interruption to the external mains power supply, followed by a failure of the DRUPS (Diesel Rotary Uninterruptible Power Supply) system that is supposed to ensure that power to the data centre is maintained during such a power cut.

The DRUPS system contains three separate units with sufficient capacity to cope with the failure of any one unit. Unfortunately, in this event, the unit that failed did so in a manner that triggered a shutdown of the other two. From the Telecity report:

… one of the units on DRUPS System 1 experienced a fault on its synchronisation card. This fault caused the unit to go into overload which, in turn, had a direct impact on the remaining two units. During the overload condition, the faulty unit back-fed the other two units which, for protection and per design, automatically shut down.

At this point the system went into raw mains bypass mode (i.e. bypassing the UPS systems, and connecting the data centre load directly to the mains). This occurred around 2 minutes after the original mains supply failure, by which point the mains supply had been restored, but there was a 500ms interruption as the bypass occurred.

This much is consistent with our observations, which is that in all but one rack, the logs on our remote PDUs did not record an outage, but the vast majority of equipment attached to them did: the management interfaces in these PDUs draw very little electricity and are known to be able to survive very short power supply interruptions.

As noted above, one of our racks experienced a more extended outage. This was due to the circuit breaker on the power bar being tripped. This was noticed and rectified by data centre staff inspecting racks following the initial outage.

At this point, the faulty DRUPS unit is out of service, meaning that whilst the power supply is protected, there is no redundancy until the unit is repaired and tested.

Conclusion

Whilst we are certainly unhappy about the outage, at this point we have no cause to question our choice of data centre provider. Sovereign House is a major UK internet hub, and is a purpose-built 6 floor data centre, built to the highest industry standards. With the best will in the world, there will always be faults that can take an entire DC, or significant parts of it, off-line, and for this reason, we would always recommend that mission-critical applications are served from multiple sites. Independent routing ensured that our facilities at other sites were unaffected by the Sovereign House outage.

That said, the aftermath of the outage has revealed some areas in which we can improve. In particular, the extended outage of one rack had a knock on effect to connectivity of others. Following Sunday’s scheduled maintenance work, we’re now in a position to improve our network topology to make it more resilient. We are also planning improvements to our Virtual Server hosts and database servers to ensure that they can recover more quickly following such an outage, and we have already made changes to our support systems to make them more resilient.

Beyond directly fixing the affected units, Telecity are also planning improvements to their communications during such an incident. This will help us direct our efforts more effectively.

Notes

For the avoidance of doubt, this interruption was completely unrelated to the network upgrades scheduled for Sunday evening, which went ahead as planned.

Finally, thank you to all customers who monitored our status page during the outage.

At the end of last year we took the decision to significantly upgrade our two connections to LINX – our busiest connections to the outside world.

This turned out to be a good plan as Mythic Beasts got a Christmas present in the form of a new company bandwidth record, thanks to two customers, Blinkbox Music and Raspberry Pi getting a substantial spike in hits as people unwrapped their Christmas presents.

And it seems that the excitement of all the presents hasn’t worn off, as the Christmas day record has just been toppled by a new all time high yesterday. With the Blinkbox apps very high in the free music app charts, we’re not expecting it to stand for long.

We aim to give our users the choice of a range of anti-spam measures. One of the options we provider is sender verify, a simple check whereby before you accept a mail, you check that the sender of that email exists, and would accept mail from you. You can argue about how effective this is as an anti-spam measure, but it seems a perfectly reasonable check to want to make, in the same way that many people choose to not answer their phone to those who withhold caller ID.

Unfortunately, some people object to you asking the question.

We recently had some complaints from users who said that they couldn’t receive mail from people with addresses hosted on Microsoft’s Hotmail servers, and sure enough, Hotmail have blacklisted one of our servers’ IPs for daring to enquire about whether particular sender addresses were valid. This affects not just hotmail.com, but various other Microsoft domains.

Sadly, Microsoft aren’t going to change their policy for us, so we needed to whitelist them. This isn’t entirely trivial as what matters is where the sender’s email address is hosted, which means looking up the MX records for that domain. Fortunately, Exim makes this easy enough, provided that you’re not offended by curly brackets. Adding the following condition to a sender verify ACL will disable the check for Hotmail hosted domains:

!condition = ${if forany{${lookup dnsdb{>: mxh=$sender_address_domain}{$value}fail}}{match {$item}{\Nmx.\.hotmail\.com\N}}}

I should note that for quite some time, we’ve used a dedicated IP address for performing our sender verify checks in order to minimise the impact of exactly this type of blacklisting. If we hadn’t done this, the blacklist would have made it impossible for any users to send mail to Hotmail-hosted addresses too. As it was, the problem only affected users who had elected to use sender verify on their domains.

You can now configure reverse DNS for IPv6 through our customer control panel. If you’ve previously been handling reverse DNS for your allocation through delegation and would prefer to use the control panel, then please get in touch.

If you’ve got a server with us and are interested in trying IPv6 and don’t already have an allocation then please email support and we’ll be happy to provide you with a block of addresses.

Our network has supported IPv6 for a while, but recently we’ve been making a concerted effort to enable IPv6 on more of our servers. What we’ve learned (mostly the hard way) is that the challenge in doing this is not so much in enabling specific services, such as making your webserver speak IPv6, but in the less obvious side effects of bringing up an IPv6 address on the server in question. Once you do this, the server will start making outgoing connections over IPv6 where possible, and that’s when you find out all the places that you’ve got IP-based access controls squirreled away.

One that caught us out recently when we brought up IPv6 addresses on our mail servers was an SPF record that listed our outgoing servers by their IP (v4) addresses. In hindsight, including IP addresses in an SPF record was never a great idea. It would be much better to use the “mx” or “a” SPF terms, referring to mail servers by name rather than address.

To help others avoid making the same mistake, we’ve added SPF record checking to our IPv6 Health Check. The rules on this are necessarily a bit arbitrary: if you have an explicit reference to an IPv4 address, it expects you to have at least one reference to an IPv6 address. In addition, any time that you use an MX term, it expects that MX to have both IPv4 and IPv6 addresses.

For an example of this, compare the results for twitter.com with the results for google.com. We fail twitter.com because of the “mx:one.textdrive.com” term. There are other parts of Twitter’s SPF that don’t appear to have IPv6 equivalents (e.g. “_netblocks.zdsys.com”) but there’s no easy way to determine which IPv6 address block corresponds to each IPv4 address block. Suggestions for better ways to categorise these test results gratefully received.

Last night we quietly upgraded the disks in our Sphinx shell server to a pair of SSD drives. Sphinx has been suffering under heavy I/O load for a while now, and it’s safe to say that the SSDs have resolved that problem for the foreseeable future.

The upgrade was without downtime, using the magic of LVM’s pvmove command.

It’s been upgraded with a pair of fiendishly expensive server-grade SSDs. We’re not normally ones to pay too much attention to whether kit is designated as “server-grade” but in the case of SSDs it really matters due to the limited number of write cycles on SSDs. The new disks are good for 8TB of writes per day for 5 years, whereas the equivalent consumer grade version is only rated for 20GB/day, which wouldn’t last very long in Sphinx.

Sphinx has a special place in our hearts as it’s the machine on which the company was founded nearly 14 years ago, and it’s been in pretty much continuous service ever since. Of course, the current hardware has absolutely nothing in common with the dual Celeron BP6 that we deposited in a Fulham datacentre back in 2000, and it now lives in Docklands, but it’s still the same machine (right?) which is why it still says:

[pdw@sphinx ~]$ rpm -q redhat-release
redhat-release-6.1-1

(don’t worry, that’s probably the only package from RH 6.1 that we’re still using…)

As if it’s not bad enough that we have to waste a huge amount of time, not to mention a non-trivial amount of hardware, bandwidth and electricity, trying to deal with spam, we also waste a fair amount of time dealing with the “ingenious” ways that the anti-spam brigade come up with to stop legitimate mail from getting through.

This week’s contribution was so special that it took three of us to confirm that they really were doing something as silly as it first appeared.

First some background: the IP address that you get given by your ISP for your broadband connection is usually dynamically allocated. This means that it may change every time you reconnect, and may be reallocated to other users when you’re not using it. This obviously makes it impossible to selectively blacklist the users of such addresses in response to spam complaints, so it is common practice for mail servers to block connections from all IPs that are known to be allocated on this basis, using something like the PBL. Users of such IP addresses are expected to use their ISP or hosting provider’s mail servers to send outgoing mail, and the administrators of those servers take responsibility for policing their customers (on pain of having their mail servers blacklisted).

Today, a customer complained that a legitimate email being sent via our server in this manner (using authenticated SMTP) was being blocked. On closer inspection, it turned out that the IP addresses that the receiving server was objecting to was not our server’s IP address, but the sender’s IP address on the basis of it having a “poor reputation”. Well duh – it’s a dynamically allocated IP: there’s a decent chance that at some point in its life it’s been allocated to an infected computer and used to send spam. They’s why you don’t accept mail from them directly, right?

The stupid thing is that the only way that the receiver knows the originating IP address is through a “Received” header that we add. That’s right, we could trivially defeat this anti-spam measure by configuring our mail server to not add the header.

Of course we’re not going to do that, as it would break the incredibly useful trace that is provided by the received headers, and is one of the few things that helps keep sane mail admins sane.