With our successful move out of Harbour Exchange last year we’ve been making all our services available in Telehouse. We have now deployed a virtual server cluster in Telehouse and those of you who avidly watch our order forms have been ordering virtual servers there for a few months.

The addition of virtual servers in Telehouse means that we can now offer three London locations for users wishing to run distributed clusters.

We’ve previously offered a choice between SSD storage for performance, and HDD storage for capacity. In the last few years, almost all new virtual servers orders have been for the SSD option so in Telehouse we took the decision to simplify our operations and only offer SSD-based virtual servers using high performance NVMe storage. Storage is the limiting factor on our virtual server hosts, so faster storage means we can run more VPSs on each host without compromising performance. We never over-sell RAM on our VPS platform — a 4GB VPS is backed by 4GB of real RAM — so our newest host servers now have 512GB of RAM and 5th Generation Intel Scalable Xeon CPUs. Storage is mirrored pairs of enterprise NVMe drives, and the servers have dual power supplies and dual 10G uplinks. Higher capacity host servers allow us to pack more virtual servers into the data centre space, and simplifying our configuration requires us to hold fewer spares on-site to cover hardware failures.

We have also rolled out more large VPS hosts into our other data centres increasing our total capacity (both SSD and HDD) and making larger VM sizes more readily available across all our UK sites. We’ve also completed decommissioning all of our older hosts with less than 256GB of RAM.

As of yesterday, new support tickets are now being handled in our new support system, which is powered by Request Tracker (RT). Hopefully from the outside, things look exactly as they did before: you email support@mythic-beasts.com, your request gets assigned a ticket number, and you get a timely and helpful response from our non-dedicated support team.

From the inside, this change is pretty daunting. Our support system is absolutely critical to our day-to-day work, and learning a new tool, and adapting all of our customer-facing process to use it is a big change. There are lots of little things like the canned “snippets” that we use when composing replies to common questions, and the processes to be followed when contacting customers in response to automated alerts. Doing this from a clean slate would be hard enough, but we also have to provide continuity for existing tickets, which can remain active for weeks, months or very occasionally, years.

We’ve been tackling the migration as incrementally as possible. Most internal, and certain external tickets have been handled in RT for some weeks now, but yesterday was the day that we flipped the switch on the support@ fire hose.

Our previous system has served us well for the 16 years that we’ve been using it, but it suffers from being closed source. There have been various enhancements that we’d like to have made, but we couldn’t, and ultimately our migration away from it has been forced by the fact that the standalone product is no longer maintained, having been replaced by SaaS version – an option that is out of the question for us.

Of course, not having access to the source code has made migrating away from it harder.

Open Source or bust

We actually evaluated RT a bit over twenty years ago, and decided that it wasn’t right for us, based mostly on some requirements that in hindsight seem a bit odd (RT has also matured a bit since then!) In selecting RT this time, we carried out a thorough evaluation of available ticketing systems, with one absolutely non-negotiable requirement: must be open source and hosted in house. We simply cannot afford for such a critical and hard-to-migrate part of our business to be locked in to closed source software, let alone a SaaS service.

Our primary goal with the new system so far has been to replicate the functionality that we had before, but once it is bedded in, we plan to make further enhancements to improve integration with our other software. Whilst in the short term we hope that you’ll see no change, in the long-term we hope that customers will see improvements to how we handle support.

We’re considering adding Request Tracker to our list of managed applications. Please drop us an email if this is something that you might be interested in.

We’ve long said that the greatest achievement of the modern tech industry is to lower standards so far that providing a service that works at all is regarded as exceptional service. One thing this implicitly includes is service cancellation. We believe in the extremely simple principle that if you don’t want or no longer require a service you can cancel it. If you don’t like the service you can move it to another provider – we implemented easy DNS import and export to make it easy to arrive or leave as a customer.

Our control panel allows you to cancel any or all of your services at the end of the billing period, with an optional note to explain why.

There’s no retentions team, you don’t have to fill in a paper form, there’s no huge contract lock in period, you don’t need a doctors note or a letter from your Mum or to be passed around five different departments and we won’t find a mysterious term in a 150 page long contract that insist on you paying for another two decades. We also don’t waste your time with seemingly infinite customer satisfaction surveys where you can rate us from one smiley to nine stars.

We welcome the new US government legislation to require all providers to match our simple cancellation process. As a company owned by founders and staff we are still guided by our desire to offer services good enough that we would willingly want to pay for them.

At the end of July, Debian 10 (Buster) reached end-of-life, and with it, all mainstream support for Python 2. The Python Software Foundation actually ended support for Python 2 on 1st January 2020, but it’s remained in Debian until now because a small number of important packages depend on it.

One of those packages is Mailman 2, a widely-used mailing list manager.  For various reasons, many projects using Mailman 2 have resisted upgrading to its successor, Mailman 3.  With Debian 10 reaching end-of-life, we’re seeing renewed interest in this migration.

One of the barriers to migrating to Mailman 3 is that the upgrade breaks links to messages in the mailing list archives.  There are links all over the internet to messages in Mailman 2 list archives, and for many projects, breaking these links would be a significant loss.

We’ve recently done some Mailman 2 to 3 migrations, and as part of this, we developed our own solution to preserving archive URLs.  We created a script that trawls Mailman 2 archives and creates a map of old URLs in the Mailman 2 archives to the corresponding URL in the Mailman 3 archives.  This can be used by Apache’s mod_rewrite to generate redirects for the old URLs.   The map can be converted to a DBM file for more efficient lookups.  This is important for archives containing many thousands of URLs.

We’ve made the mailman-archive-mapper script freely available on GitHub.

We offer mailman as a managed application.

 

In March 2004 we moved all three of our servers into a single rack in the 6/7 Harbour Exchange data centre, operated at the time by Redbus.  The data centre has changed hands several times, and merged with the building next door to become what is now Equinix LD8. We’ve been continuously present for 20 years and 1 month. Normally moving out of a data centre is a difficult, expensive and time consuming operation that is best avoided, but Equinix offered us terms that made doing so make sense. In September 2023 we opened our new core point of presence in Telehouse South.

We’re happy to report this project is now complete and our footprint in Equinix LD8 is now reduced to an optical-only point of presence forwarding 10Gbps waves to our core site at City Lifeline.

Our new space in Telehouse South offers a considerable upgrade over what we could offer in LD8. All servers now have remotely switchable dual power feeds and with dual 10Gbps uplinks. We are able to offer offer cross-connects to anywhere in the Telehouse London campus and 10Gbps wavelengths back to our other sites. We already have some new colocation customers taking advantage of these additional services. We still include serial for out-of-band server management.

During this move, we live migrated our virtual server cloud to hosts in either City Lifeline or Sovereign House. Apart from a few special cases supporting very old virtual servers or ones with BGP transit services, this was done without interruption to the client. Dedicated servers and colocation customers moved in a series of windows to minimise downtime while the servers were relocated.

We brought on additional network capacity as part of the move including 10Gbps and 100Gbps links to transit providers and private peers within the Telehouse London campus. This provides a significant upgrade in connected external capacity.

Mythic Beasts is now a verified Green Hosting Provider according to the Green Web Foundation.

We’ve demonstrated to the Green Web Foundation that all our UK and EU data centres buy as much renewable electricity as they use. This hasn’t changed our operations; internally we met this requirement in 2018. What’s changed is that we’ve now provided all the documentation to meet the certification standards of the Green Web Foundation.

Of course this isn’t quite the same as saying that all the electricity we use comes from renewable power. Ultimately, the electrical energy from a wind farm isn’t tagged to flow directly to the data centres we use and there is also no requirement that the electricity is bought at exactly the same time it is used. Similarly, the data centres have fossil-fueled generator backup which means small amounts of fossil energy are still used.

That said, we do believe that this is an important and useful step in the right direction. By getting verified under this scheme we, and the 429 other verified companies, apply pressure on the data centre suppliers to buy and use renewable energy which strongly encourages the marketplace to build more renewable generation.

Some of our data centre providers are very large well-resourced companies and they place very large long term orders for renewable power. This means renewable power providers can secure funding to build out renewable power generation. When they want to build a data centre, they also have to fund the building of an equivalent amount of renewable generation to power it.

Yesterday, the following not-so-subtle notice appeared on the admin interface of all Mastodon instances:

The Mastodon team announced on Monday that this release was coming, so we were ready for it:

Details of the vulnerability are still limited, but from what we do know it sounds serious (“Remote account takeover“).

All our managed Mastodon instances were safely patched just over an hour after the new packages dropped. One instance gave us a bit of trouble, as the new version appeared to tickle a bug in Elasticsearch causing ES to consume all CPU on the server. After we eventually pinned down the cause, it was resolved by an upgrade of Elasticsearch. Turns out the ES upgrade didn’t fix it, and we’re still working with our customer to get this resolved.

Managed open source hosting

Open source software such as Mastodon, GitLab and Nextcloud can offer a great alternative to the lock-in associated with proprietary cloud equivalents, but the effort associated with hosting them can be significant: backups, monitoring, security patching, and the investigation and debugging required when a supposedly innocuous software upgrade leaves your CPU usage wedged at 100%.

Our managed open source hosting provides the best of both worlds: the convenience of a “cloud” solution, but without the lock-in. Your data is yours, and if you don’t like our service you can take your data and host it somewhere else (although we’re confident you won’t want to). And because there’s no lock-in, you get straightforward pricing based on the resources you’re using, rather than loss-leaders followed by price hikes once you’re hooked.

Read more about our managed hosting, or drop us an email at for more information.

 

Last year we enhanced our web hosting service with the ability to choose your own PHP verison. You can choose a different PHP version for each website hosted with us, so you can upgrade your staging site and test before you upgrade the production one. With PHP 8.0 about to go end-of-life, the addition of PHP 8.2 provides more options for migrating production applications.

Since the initial roll-out, we’ve added more PHP versions to help with moving and upgrading older applications. Not only is the newest version PHP 8.2 available, but you can also select the older 7.3 and 7.4 versions. We’re proud to sponsor Ondřej Surý who creates the debian packages we rely on.

Our hosting accounts still support unlimited websites, have free and automatic SSL through Let’s Encrypt to keep your sites secure, and include MariaDB databases.

 

On Saturday the Debian team released the latest version of Debian, Bookworm. We’re pleased to announce that this is now available on our virtual and dedicated servers.

Bookworm is a fully supported operating system for our managed hosting and we already have it running on some of our internal production servers. Our preferred open source server management system, Sympl, has also been updated to support Bookworm. Other feature enhancements include much more control over PHP versions and settings. Our virtual server cloud has pre-built images for standard Bookworm and Bookworm with Sympl pre-installed.

There are many improvements in Bookworm, with PHP 8.2 support being the most anticipated by our customers. We would like to thank the Debian team for all their hard work in making this release.

We’ve been offering IPv6-only hosting for eight years now, and have demonstrated that many websites can forego the expense of an IPv4 address pretty easily. You can read more about how we do this on this blog post from 2020. This blog post itself is being served from an IPv6-only server!

A key part of this is our IPv4-to-IPv6 proxy. This listens for incoming traffic on a shared IPv4 address and forwards it to your IPv6-only server. In order to use the proxy, you need to tell it which hostnames to listen for, and which server or servers to forward traffic to. This can be done using our control panel, and as of today, it can also be done via an API.

Having an API for proxy configuration makes it possible to automatically add or remove backend servers, allowing you to spin up additional servers, or take servers out of service for failover or maintenance.

You can also use the API to add and remove hostnames handled by the proxy, and so can be used to automate the provisioning of new services.

Fine-grained access controls

As for our DNS API and Domain API, the Proxy API provides fine-grained access control for API keys. For example, you can create an API key that only has access to a specified domain or hostname, or you can create a read-only API key if you only need to read the current configuration.

Getting started

Our IPv4-to-IPv6 proxy is available to all customers with a Mythic Beasts server, including virtual servers, Raspberry Pi servers, dedicated and colo. You can find more information on the proxy service, and the Proxy API on our support pages.