We’re pleased to announce that Mythic Beasts has acquired VMHaus, a virtual server provider with facilities in London and Los Angeles. We will continue to run VMHaus as a separate brand selling low-cost, prepaid virtual servers, which we believe will complement our own virtual server products well. We’re also pleased to announce that VMHaus co-founder Basil Fillan has joined Mythic Beasts as a full time employee. Basil has been responsible for the development of the VMHaus technical infrastructure, and will be ideally placed to help us provide support to VMHaus customers.

In the short term, VMHaus customers will see no changes to their services. Payments and invoicing will continue to be through VMHaus Ltd, and we will continue to accept new orders for VMHaus products. In the medium term, we’re planning improvements to both the VMHaus platform and our own virtual server infrastructure, based on our combined experiences of developing the two systems.

On the VMHaus side we hope to be able to start selling virtual servers in Amsterdam early in the new year, and also be able to offer IPv6-only virtual servers at a discounted rate. VMHaus customers will also be able to take advantage of our other services such as domain registration and backups.

On the Mythic Beasts side, we expect to be able to offer service upgrades thanks to the economies of scale resulting from the acquisitions of VMHaus and of BHost this summer.

OpenWRT is awesome, as it allows you to run proper Linux tools on your home router. I’m currently using a very old, underspecced TP Link box, with 32MB of RAM, but just 4MB of flash storage. This is just enough to get what I need installed, but one thing I’ve always wanted to do is use iftop to quickly see what’s using all the bandwidth. Unfortunately iftop, with its dependencies on libpcap and libncurses, just won’t fit into a 4MB image.

I recently stumbled across opkg’s install-to-RAM option, allowing me to use the 32MB of RAM to install the package, with the minor and obvious downside that it gets uninstalled when the router gets rebooted. For something like iftop, which is used for ad-hoc diagnostics, this isn’t a big issue.

Installing to RAM puts the packages under /tmp, so a little effort is required to make sure that libraries and other resources can be found. I now have the following shell script which installs iftop if it isn’t already, sets some environment variables and invokes iftop:

#!/bin/sh

if [ ! -f /tmp/usr/bin/iftop ] ; then
  opkg update
  opkg install -d ram iftop
fi

export LD_LIBRARY_PATH=/tmp/usr/lib
export TERM=xterm
export TERMINFO=/tmp/usr/share/terminfo/

/tmp/usr/bin/iftop $@

Fortunately I do have enough free space on flash storage to store the above script.
Obviously a similar approach could be used with other packages that are only needed “on demand”.

Managed customers of Mythic Beasts with libssh installed will have just received a notification that we updated it without warning or testing.

This is obviously bad practice, so what were we thinking?

A security advisory for libssh has just come out which is very bad. To paraphrase,

libssh -> hello new user
user -> can I have a root shell
libssh -> can you authenticate?
user -> yes but I'm not going to
libssh -> okay, have a root shell

This is completely secure, unless the client is prepared to lie in order to exploit your system. In the late 1990s some of our founders might have once exploited an online quiz in exactly the same way to get perfect scores. Don’t trust the client.

In our risk analysis, the risk of breakage to a customer site though a botched patch is vastly lower than giving an attacker a root shell, which is why we pushed an emergency update within a few hours of updated packages being available.

If this is the first you’ve heard about the issue, we suggest you’d benefit from our Managed Services

At Mythic Beasts we rotate staff members around different roles. This is to protect the company from the unlikely event that a staff member is abducted by aliens and someone else has to take over at short notice.

With great sadness we have to report that Toby Goodwin, our first full time employee was not abducted by aliens. Much worse, he had an undiagnosed asymptomatic heart problem and passed away unexpectedly and painlessly last week.

Back in 2010 Toby had been running a bookshop in Cambridge with a quirky and eclectic selection of books. That business had come to an end and Toby was wondering about dusting off his UNIX skills and looking for work. At the same time Mythic Beasts had grown too large for the two then-active founders to effectively keep up and after an interview over a beer in the Devonshire Arms, Toby joined Mythic Beasts.

We didn’t initially realise how lucky we were because Toby had the perfect blend of skills. An experienced UNIX hacker from his days at Cygwin, he quickly figured out most of the technical operations to keep Mythic running. Meanwhile his experience at the bookshop gave him incredible patience and empathy for confused customers. He took it on himself to continuously improve our operations introducing radical new ideas like helper scripts having consistent names to make them easy to find, continuous integration and automated testing of our control panel.

Toby implemented the bulk of our managed server update system. When he started, we had tens of managed customers and updating packages was starting to become time consuming. Gradually this became a highly reliable and flexible system which means we can audit and update thousands of servers quickly and efficiently, whilst correctly notifying every affected customer in a timely fashion. Toby was always modest about his achievements and never suffered from being defensive about his code. When our summer students discovered a significant security flaw in a piece of configuration, he congratulated them and worked with them to resolve it quickly.

After working with us for a few years in Cambridge, Toby met Heather and moved with her to her native Scotland where they married and brought into the world a highly reliable early morning alarm clock called Zachary. Toby would regularly work early in the morning before taking some time out to deliver Zachary to nursery or work with him on significant structural engineering projects.

.

Goodnight Toby. We’ll miss you.

Hot on the heels of acquiring Retrosnub, we’ve also bought the customers and assets of BHost. BHost are a virtual server provider with services in London, Amsterdam and California based on OpenVZ and KVM.

We’re excited about this acquisition as it provides us with a great opportunity to expand our network using BHost’s Amsterdam infrastructure. At the same time, we’re confident that we can provide some immediate and longer term improvements to the BHost service, not least through our larger support team being able to offer more timely and helpful responses to customer queries.

Although handover officially happened today, BHost customers have had access to our control panel for several weeks, mostly so that we could start tackling EU VAT bureaucracy. BHost are a US-registered business. We’re a VAT-registered business in the UK. Thanks to VAT MESS, it’s actually much harder for us to sell to EU-based consumers than it was for BHost, as we’re required to collect an unreasonable amount of evidence of customer location.

The good news for BHost customers is that we’re matching BHost’s current pricing with our UK VAT-inclusive price. This means that EU VAT-registered businesses, and all non-EU customers will see a significant reduction in the price that they pay.

If you’re a BHost customer and you’ve not already done so, please log in to our customer control panel using your BHost username (email address) and password and confirm your contact details.

Network Expansion

BHost run a network from London/Amsterdam with multiple 10Gbps uplinks and some peering in each site. We will be moving the BHost London network behind our own so that BHost customers can take advantage of our larger capacity uplinks and significantly better peering arrangements, which includes transit-free connections to every major UK ISP.

We’re also taking the opportunity to significantly improve the connectivity to our Cambridge data centre. We currently have two uplinks via different London data centres. We will replace one of these links with a direct connection to Amsterdam, and bring both up to 10Gbps. Combined with BHost’s existing London/Amsterdam connection, this will create a 10Gbps ring around London, Cambridge and Amsterdam, complementing our 50Gbps ring around our three London sites. This will provide increased bandwidth and improved resiliency for our Cambridge customers, whilst also providing a second London/Amsterdam link to improve resilience within the BHost network.

BHost Amsterdam customers will gain direct UK connectivity through our extensive London peering. We will gain the Amsterdam Internet Exchange connection (AMSIX) from BHost, bringing improved European connectivity to all London customers. We expect to be able to substantially increase the number of AMSIX peers, improving EU connectivity for all customers.

Cloud expansion

BHost’s London presence is in the Meridian Gate (MER) data centre. We already have a significant footprint in MER, although it’s not currently available as a zone in our public cloud. We’re investing in new hardware to deploy in Meridian Gate which is both substantially faster and more power efficient than the current hosts. We’ll be deploying this into our existing suite in MER, and then migrating BHost servers into it. BHost customers will see a small window of scheduled downtime as we migrate each server, but should then seen significantly improved performance on the new hardware.

Our Amsterdam and US presences will give additional options to customers that need to be physically hosted within the (post-Brexit) EU or US. We expect this to become more relevant after Brexit when the UK and EU may have diverging regulatory requirements.

Additional services

All BHost customers can now take advantage of additional Mythic Beasts services including management services for virtual servers, domain registration and DNSSEC-enabled, API-backed DNS hosting.

Support

Mythic Beasts have a larger support team and we’re very well placed to provide significantly improved customer service to all of our new customers. Of course, we do expect the period immediately after the transition to be very busy as customers become familiar with the new billing arrangements, and we get to grips with supporting BHost’s services. We will have additional staff during this period, but please be patient if support responses are a little slower than usual.

We’re looking to expand our team by employing a part-time finance and administrative assistant for between 12 and 20 hours a week. The duties of the role will likely include dealing with invoicing queries from customers, reconciling bank transfers with invoices, uploading receipts to Receipt Bank and Xero, chasing up overdue invoices, dealing with some paper mail to the company, taking cheques to the bank, filing paperwork and keeping a calendar of administrative deadlines. Previous experience in a similar role is desirable, as is experience with Xero accounting software.

Mythic Beasts don’t have an office, so the job primarily involves working from home. We’ll provide you with a laptop and cover reasonable home-working expenses. Hours are flexible, but we would normally expect you to be available in Cambridge on Wednesday afternoons. You will be entitled to 30 days of holiday a year plus bank holidays, pro rata for the hours worked. Salary is subject to experience. You will be eligible for company health insurance and membership of our employee share scheme after the qualifying period.

If you’re interested or would like to know more, drop us an email.

Just before Christmas we were approached by Malcolm Scott, director of Retrosnub, a small cloud hosting provider in Cambridge. His existing connectivity provider had run out of IPv4 addresses. They’d decided to deal with this issue by adding charges of £2 per IPv4 address per month to encourage existing customers to return unused IPv4 addresses to them. As a cloud hosting provider with a substantial number of virtual machines (VMs) on a small number of hosts this had the result of tripling the monthly colocation bill of Retrosnub.

Aware of my presentation on IPv6-only hosting at UKNOF, Malcolm knew that opportunities for significant expansion were severely limited due to the difficulty of obtaining large amounts of IPv4 address space. Retrosnub faced a future of bankruptcy or remaining a very niche provider. His connectivity providers seemed strongly in favour of Retrosnub going bust so they could reclaim and re-sell the IPv4 space for higher margin services.

As a larger provider with our own address space, we had sufficient spare capacity in our virtual machine cloud to absorb the entire customer base of Retrosnub with no additional expenditure. Our work in supporting IPv6-only virtual machines will also make it easier to significantly reduce the number of IPv4 addresses required to support Retrosnub services. We formed a deal and agreed to buy the customer base of Retrosnub.

Combining operations

Since agreeing the deal, we’ve been working hard to merge our operations with minimum disruption.

The top priority was the domain name services because domains expire if you don’t renew them. Doing a bulk transfer of domain names between registrars is something which Nominet, the body responsible for UK domains, makes extremely easy, as it just requires changing the “tag” on all the domains.

Unfortunately, just about all other TLDs follow a standard ICANN process, which requires that a domain be renewed for a year at the time of transfer, and that the owner of the domain approves the process. If you were designing a process to destroy competition in a market by making it hard for resellers to move between registrars, it would look quite like this.

We’ve now got the bulk of domains transferred, and the next steps will be to migrate the DNS records from Retrosnub to Mythic Beasts so that our control panel can be used to change the records.

At the same time, we rapidly formulated a plan to migrate all the virtual machines in to stem the financial losses. Moving the VMs required an unavoidable change in IP address, and we also wanted to get them migrated from their current platform (Citrix Xenserver with para-virtualisation) to our own platform (KVM with full hardware virtualisation).

In order to ease the transition, we arranged for a pair of servers to do IP forwarding: a server in our cloud that forwarded the new IP to the VM in the Retrosnub cloud until it was migrated in, and another in the Retrosnub cloud that forwarded the old IP after the server had been moved. By doing this we were able to give customers a one week window in which to complete their IP migration, rather than forcing it to be done at the time that we actually moved the VM.

In the process of this migration, all customers received a significant bandwidth upgrade and majority received disk, RAM and CPU upgrades too.

We completed this on schedule before the quarterly colocation bill arrived, so instead of paying the much increased bill, we cancelled the contract and removed the servers from the facility.

Next steps

Our next step will be to migrate all the web and email hosting customers into our standard shared hosting environment. This has some time pressure as Google have plans for Chrome to start marking all non-HTTPS websites as insecure. We offer one click HTTPS hosting using Let’s Encrypt on all of our hosting accounts.

We use Let’s Encrypt for SSL certificates, and our preferred client for obtaining certificates is the simple but effective dehydrated shell script, not least because it’s packaged for Debian.

On Sunday, we started getting some alerts relating to a failure to automatically re-issue Let’s Encrypt certificates. A quick bit of digging yielded this error:

+ Creating fullchain.pem…
  + ERROR: An error occurred while sending get-request to http://cert.int-x3.letsencrypt.org/ (Status 301)

Let’s Encrypt have started including an HTTP redirect as part of the certificate issue process and dehydrated doesn’t pass the necessary option to curl to follow the redirect. This can be fixed by patching dehydrated (and a packaged fix for Debian Stretch is now available via Debian backports), but it can also be solved with a simple config change:

echo 'CURL_OPTS="-L"' > /etc/dehydrated/conf.d/curl.sh


Naturally, customers of our managed hosting services and customers using the free HTTPS option on our hosting accounts need not worry about this issue. Our managed hosting includes monitoring all HTTPS websites for certificates nearing expiry, so we become aware of any issues well before your users do.

Today is Pi Day where we celebrate all things mathematical. Today is a super special Pi day, because a new Raspberry Pi has been released.

It takes the previously excellent Raspberry Pi 3 (or 3B, to give it its full name) and upgrades it with an extra 200Mhz of CPU speed and gigabit ethernet over USB 2. It fixes many of the netboot issues which Pete highlighted at the last big Pi Birthday Party and will soon have a new smaller and cheaper Power over Ethernet HAT. These new features are of particular interest for our Raspberry Pi Cloud service, as we use netbooted Pis, with network file storage and Power over Ethernet to enable remote powercycling.

We’ve had one to play with, and we’ve run our favourite benchmark – Raspberry Pi’s own website. We installed the full stack (MySQL, WordPress & PHP7) under Debian Stretch onto a Pi 3B and a Pi 3B+, and tried it out with 32 concurrent connections. We’re running near identical setups on the two servers: both have their files stored over the network on an NFS file server and it’s the same operating system and applications; only the kernel differs.

Model	Pages/second
Raspberry Pi 3B	3.15
Raspberry Pi 3B+	3.65

The new model is about 15% faster than the old one which is almost exactly as expected from the boost in clock speed; WordPress is CPU limited.

Checksumming the 681MB database file shows up the gigabit ethernet rather effectively. All our storage is over the network so reading files is a benchmark of the network speed.

Model	Elapsed time	Data rate
Raspberry Pi 3B	54.4s	11.25MB/s
Raspberry Pi 3B+	28.1s	22.1MB/s

This is very nearly twice as fast as the previous model.

When is it coming to the Raspberry Pi Cloud?

The Raspberry Pi 3B+ is an obvious upgrade for our Raspberry Pi Cloud. We need to wait for the PoE HAT to become available. That will allow us better density and lower capital costs. However, the 3B+ consumes more power than the 3B so we need to do some thermal and airflow work before we can make it generally available.

Flatpak provides Linux desktop applications in a secure sandbox which can be installed and run independently of the underlying Linux distribution. Application developers can produce one Flatpak and select the versions of libraries that their application is built, tested and run with so it’s easy for users on any Linux OS to get whatever was intended by the application developer.

Flathub is a distribution service to make sure that Flatpaks are available for popular Linux desktop applications, and at its heart is a private cloud running BuiltBot which builds popular Linux and free/open source desktop apps in Flatpak format. This lives in Mythic Beasts’ Cambridge data centre.

At Mythic Beasts we like this idea so much we offered them lots of free bandwidth (100TB) to help get them started. We’ve now upgraded this with a pair of virtual machines in our core Docklands sites to provide redundancy and more grunt for traffic serving.

Are Flathub downloads going a bit faster for folks today? @Mythic_Beasts have been helping speed up the @FlatpakApps revolution this week. 🙂

— Robert McQueen (@ramcq) 23 February 2018

Some of their users noticed and were appreciative immediately:

2017-02-23 16:30:00irc wow! Flathub is *so* much faster i’m getting like 10 MB/s compared to less than 1 this morning … and the search is now instant

2017-02-26 11:35PersiFlathub is _really_ fast now, great job to whoever is responsible
🙂