After careful consideration, we have reluctantly taken the decision that the use of Cloudflare will not be supported on our Web and Email Hosting service. First two clarifications:

• This only applies to our Web and Email Hosting services. It does not, and will not, apply to virtual servers (VPS), dedicated servers or Raspberry Pi servers.
• We have never formally supported or encouraged the use of Cloudflare with our Web and Email service. Until now, we have generally discouraged it, but we’ve tried to accommodate users who chose to use it.

The problem

As we’ve written about previously, we’ve seen a huge increase in the amount of abusive bot traffic hitting our web servers — much of it badly behaved AI scrapers — and frequently the volume of traffic is such that it overwhelms the server, and makes websites unavailable. At times we’ve seen over 95% of traffic coming from AI scrapers. Our Web and Email Hosting service is what’s often referred to as “shared hosting” meaning that we have websites for many customers on a single server. This is a very cost effective way to provide web hosting, but it does mean that any load issues caused by traffic to one website can affect other sites on the same server.

Our primary tool for dealing with abusive traffic is to identify the source of traffic and block the IPs that it’s coming from.

Cloudflare provides a service that seeks to protect websites by blocking abusive traffic. It does this by operating a “reverse proxy”; rather than web traffic arriving directly at the web server, it is instead sent to Cloudflare’s servers. Cloudflare inspects the traffic, filters out the abusive traffic, and then forwards the legitimate traffic to the actual web server. This has the effect that all traffic arriving at the web server appears to come from Cloudflare’s IP addresses, rather than the actual client IPs.

Some of our customers have chosen to front websites hosted on our shared hosting servers with Cloudflare. The problem is that Cloudflare isn’t perfect; it doesn’t succeed in filtering out all abusive traffic. This is particularly true of the free tier that we tend to see used in conjunction with our Web and Email Hosting service.

Unfortunately, when we see a large volume of abusive traffic arriving via Cloudflare, we are faced with a choice: either we block Cloudflare’s IP addresses, knowing that this will take all websites using Cloudflare completely offline, or we accept the traffic, which potentially has an impact on all websites hosted on that server. With the growth in volume of abusive traffic, we are being forced to make this choice increasingly often.

We have now taken the decision that we will treat Cloudflare’s IPs like any other IPs: if we see abusive traffic from them, we will block them. In the future, we may introduce a permanent block, or redirect traffic to a support page on our site that explains why Cloudflare is not supported on the service in order to avoid customers inadvertently using an unsupported configuration, but we will contact customers who appear to be using Cloudflare prior to taking this step.

FAQs

Obviously these questions are not frequently asked as this is the first announcement of this change, but whatever; here are some questions and answers:

What have you got against Cloudflare?

None of this is a criticism of the service that Cloudflare provides. The same would apply to any reverse proxy service placed in front of our servers that prevents us from seeing the actual source IP, thereby removing our ability to effectively filter traffic ourselves.

We do have concerns about any service that breaks the end-to-end encryption of web traffic, but that’s unrelated to this issue. Cloudflare (and any other such service) relies on terminating the TLS connection from the client on their servers, inspecting the traffic, and then making a new secure connection to the target web server.

Why do you think you’re better at this than Cloudflare?

We’re not, but when a particular attack is affecting our ability to provide our web hosting service to hundreds of customers, we’ve got a much stronger incentive to resolve it quickly.

Cloudflare definitely has some very impressive technology, but we suspect that much of it isn’t provided in the free tier of their service. Upgrading to a paid-for tier, or careful configuration of the free tier, might yield better filtering, but in our case, we’re affected by the most permissive configuration; it only takes one customer to point Cloudflare at our server with minimal filtering and we’ve got a mix of legitimate and abusive traffic arriving from the same IPs and we’re back where we started.

Cloudflare includes the client IP address in an HTTP header – why don’t you filter on that?

Filtering based on the source IP address of a TCP/IP connection can be done very efficiently because you only need to look at the packet header. Filtering based on an HTTP header requires accepting the connection, setting up a TLS connection and decrypting the content, and then parsing the headers, which is a significant overhead, and in extreme cases, the load from doing this is prohibitive.

Can I use Cloudflare on my VPS or dedicated server?

Yes, absolutely. The issues described here that make Cloudflare such a problem for our Web and Email Hosting service don’t apply to VPSs and dedicated servers, so if you want to put Cloudflare — or any other reverse proxy — in front of your server, you are free to do so. Of course, dealing with any abusive traffic that slips through is your responsibility.

Can I use Cloudflare on my managed server?

If you have a managed server with us then we will attempt to resolve any load issues caused by abusive traffic as part of this service. Please do not install Cloudflare (or similar) in front of your managed server without discussing it with us first, as it will hamper our ability to respond to any incidents. A special case is our WordPress Shared tier of managed WordPress hosting. As the name suggests, this is implemented on our shared hosting platform, and so we cannot support the use of Cloudflare with this service.

In 2018 Mythic Beasts acquired VMHaus, a small provider of very low cost virtual servers.

As an independent virtual server hosting provider, VMHaus was not financially viable. Post acquisition, we significantly reduced the costs of running VMHaus by using economies of scale from Mythic Beasts. We could recycle retired servers and disks from Mythic Beasts into VMHaus making their hardware effectively free. We provided rack space and transit from Mythic Beasts data centre space at cost price, taking advantage of Mythic Beasts economy of scale and buying power. However, an energy crisis and high inflation post-Covid meant that VMHaus would likely never become financially viable and in 2024 we took the decision to close the company. We gave all VMHaus customers six months notice to migrate to another server, with an offer of discounted hosting in the Mythic Beasts cloud.

VMHaus ran used a pre-payment model; customers had to buy credits in advance, then use up the pool of credits by running virtual servers. If your credits ran out, the servers stopped working and it was your responsibility to refill the meter before this happened. They also had some neat technical features we’ve incorporated into the main Mythic Beasts cloud – per-second billing, cloud-init for customising installs at boot and fully private network segments for each virtual server with IP address portability.

The pre-payment model made the shutdown of VMHaus a bit more complicated as VMHaus held funds that would not be used prior to the shutdown of the company. In order to refund unused credit, we needed customers to tell us where to refund it to, so in order to put a time limit on the wrap-up, we gave customers with positive balances a choice: get a refund, or donate the balance to the NLNet Foundation, defaulting to the latter if we don’t hear from you. The credit balances were typically very small, and in many cases, the accounts had been inactive for a number of years. Donated balances were each rounded up to the nearest dollar, reflecting the fact that this option saved us PayPal payment fees.

The NLNet Foundation funds projects that create and maintain key internet infrastructure – the sort of software that VMHaus and Mythic Beasts rely on.

We’re very pleased to say a large number of customers actively asked us to donate their balance, and combined with the balances from customers who didn’t respond to our many email reminders,we ended up with a final balance of $5240. We rounded this up to €4550 and sent it to NLNet Foundation.

Software-as-a-Service has transformed the way that we use computers. The ability to access services anywhere, on any device, without having to install, manage or upgrade applications has obvious advantages. You can be up-and-running on GitHub, Slack or Box in a matter of minutes, and very often there’s a free entry-level tier that requires no up-front payment at all.

Even when you move into the paid-for tiers, replacing the cycle of licence fees and upgrade fees with a simple ongoing service fee can also be attractive.

But SaaS comes with a major drawback: lock-in.

You’re at the mercy of a single provider, and your only recourse if you’re not happy with the service is a potentially complex and expensive migration to a completely different platform. It may not be easy, or even possible, to export your data from your old provider.

And those “free” entry-level tiers have to be paid for somehow. As your usage increases, costs can quickly become very expensive. Or the unprofitable cheap and free tiers get withdrawn or restricted as the business needs to move its income stream from investors to actual customers.

Self-hosted open source

Open source alternatives to many popular services exist, and the obvious alternative to SaaS is self-hosting: run your own server and install the software yourself. You’re completely in control, your data is on your servers and there’s no risk of lock-in.

But self-hosting loses many of the advantages of SaaS; you’re now responsible for running the servers, applying security patches, maintaining backups, and monitoring for issues.

Managed Open Source Services

At Mythic Beasts, we offer an alternative: MOSS – Managed Open Source Services. With MOSS, you get the convenience of Software as a Service, but without the lock-in. We manage the service, apply security updates, provide 24/7 monitoring and take regular backups, but your data remains under your control: we’ll happily give you full access to all the data on your server at any time.

By using open source software, you’re not tied to a single provider. If you’re not happy with our service you can migrate to another provider, or even switch to self-hosting, without the cost of having to familiarise users with different software.

And if you decide you do want to change software, open source puts you in a much stronger position for planning a migration.

Our pricing structure is straightforward, and tied to what it costs us to provide the service. With no free, entry-level tiers that need to be subsidised by the higher tiers, prices scale with your usage, rather than leveraging your lock-in.

We’ve been hosting exclusively on open source software for nearly 25 years, and provide Managed Open Source Services for a range of applications, including GitLab, Mattermost, and NextCloud. For more information, please see our Managed Applications page, or contact us to discuss your requirements.

A History Lesson

Back in the mists of time, not that long ago, you could happily forward email to another account sending all your email to a single mailbox and replying through your internet provider’s SMTP server with whatever address you wanted as the sender.

Unfortunately, back in the 1970s, someone realised you could send that new fancy ‘electronic mail’ to people you didn’t know. The few hundred users affected were rather negative about it, and it didn’t really catch on. But, within a couple of decades, the ARPANET became the Internet, and unsolicited email started to become annoying.

With the ‘Information Superhighway’ appearing in the mid 90s and the growth of personal computers, the Internet started to grow rapidly, and ‘spamming’ (a term adopted from Usenet) became more common.

Blacklists and whitelists were developed, with lists of known spammer email addresses and IPs which were known to send spam, but the fact is that email addresses could be trivially faked, and there were always other servers to send mail through.

Over the years there have been various different approaches to authenticating email senders, but it’s a continual arms race. Spam remains a problem, as does the plethora of scams and fraud that are enabled by the ease with which email can be forged.

SPF & SRS

In the early 2000s,  ‘Sender Policy Framework’ (SPF) started to gain traction. SPF works by publishing a DNS record that declares which servers are allowed to send mail from your domain.

Unfortunately, SPF had two fatal flaws:

1. Pretty much by design, it breaks email forwarding. You can’t, for example, use our servers to simply forward mail to your Gmail account, because our servers aren’t allowed to send mail from the original sender’s domain.
2. SPF restricts the envelope sender, which is the hidden address that bounce messages get sent to. The one that users actually see in their mail client – the “From” address – is still unrestricted!

(1) could be fixed by another Three Letter Acronym: SRS or ‘Sender Rewriting Scheme’. As the name suggests, this replaces the original sender address with a different address in your own domain, and because this sender is usually hidden, the received mail looks no different. We’ve supported SRS forwarding on our servers for a long time, and until recently, it worked well.

(2) was a trickier problem, which we’ll come back to…

DKIM

In the 2010s and with processing becoming cheaper, ‘DomainKeys Identified Mail’ (DKIM) also appeared, which electronically signs the mail so that the remote server can verify the signature. The public keys are published in the sender’s domain’s DNS, so recipients can verify that mail came from the claimed sender.

DMARC

The issue with DKIM is that it’s not universally adopted, so whilst you can check if a signed mail is legitimate, you can’t assume that an unsigned email isn’t. Enter yet another acronym: DMARC (‘Domain-based Message Authentication, Reporting and Conformance’).  DMARC allows a domain-owner to publish yet another DNS record which says “all my mail will either be DKIM signed, or pass SPF, or both, and if it isn’t, bin it!”

Adoption of DMARC has been slow, not helped by the fact that publishing a strict DMARC policy in 2014 broke just about every mailing list of the time. Fast-forward ten years, mailing lists have learned to adapt, and the big mail providers are pushing the world towards DMARC. Gmail, probably the largest free email provider, now requires either SPF or DKIM to be enabled on a domain in order for them to accept any mail from it, and if you’re sending bulk email, you need a DMARC record too.

The Effect On Forwarding Mail

While these measures have all helped greatly to combat spam, with a large proportion of it now coming from compromised servers or mailboxes, it’s had some down-sides, the most obvious of which is difficulty in forwarding email.

Lots of our customers like to register a domain with us, set up some email addresses in that domain, and forward them to existing email accounts at other providers, such as Gmail or Yahoo. We also host lots of clubs and other organisations that have role-based addresses (like “treasurer@myclub.org”) that forward to the person currently doing that role, and mail expanders like “committee@myclub.org” that forward to a handful of people.

DMARC alignment

SPF is now widely adopted and, as noted above, you used to be able to work around the issue that it creates for forwarding using SRS.

But, one of the subtle features of DMARC is that it quietly fixes problem (2) with SPF. In order to pass SPF in a way that makes DMARC happy, the hidden sender address that SPF restricts has to be in the same domain as the visible “From” address (this is called “alignment”).

• If you forward without SRS, the messages that you forward will fail an SPF check.
• If you forward with SRS, the messages will technically pass SPF, but DMARC will call it a fail because the sender and from addresses don’t match.

In other words, there is no way to pass a DMARC check using SPF when forwarding mail. You have to rely on DKIM, and hope that forwarding mail (with or without SRS) doesn’t break DKIM.

You would hope that senders wouldn’t publish a strict DMARC policy without first ensuring that all of their outbound mail is DKIM-signed, but you may be disappointed, and with mail providers increasingly enforcing DMARC policies, mail forwarding has become less and less reliable.

Yet another acronym – ARC – promises to fix mail forwarding, but it’s not a magic bullet. We’ll discuss that another time.

Of course, forwarding mail works fine if you’re in control of the server you’re forwarding to, and can whitelist the server doing the forwarding, but that’s not common. Most people want to forward to providers like Gmail where they have no such control.

For some platforms there is a workaround; rather than forwarding mail, have the platform collect it from a mailbox using POP3 or IMAP.  For example, Gmail have their “check messages from other accounts” functionality in their mailboxes, which picks up you mail from a mailbox as if it’s been received by the Gmail mailbox directly, and other platforms are adding similar options.

This works well enough for personal email addresses, but it’s a lot more hassle for things like role-based addresses at emails at clubs and societies, and doesn’t work at all for mail expanders where multiple recipients need to collect the mail.

This is why we can’t have nice things.

With our successful move out of Harbour Exchange last year we’ve been making all our services available in Telehouse. We have now deployed a virtual server cluster in Telehouse and those of you who avidly watch our order forms have been ordering virtual servers there for a few months.

The addition of virtual servers in Telehouse means that we can now offer three London locations for users wishing to run distributed clusters.

We’ve previously offered a choice between SSD storage for performance, and HDD storage for capacity. In the last few years, almost all new virtual servers orders have been for the SSD option so in Telehouse we took the decision to simplify our operations and only offer SSD-based virtual servers using high performance NVMe storage. Storage is the limiting factor on our virtual server hosts, so faster storage means we can run more VPSs on each host without compromising performance. We never over-sell RAM on our VPS platform — a 4GB VPS is backed by 4GB of real RAM — so our newest host servers now have 512GB of RAM and 5th Generation Intel Scalable Xeon CPUs. Storage is mirrored pairs of enterprise NVMe drives, and the servers have dual power supplies and dual 10G uplinks. Higher capacity host servers allow us to pack more virtual servers into the data centre space, and simplifying our configuration requires us to hold fewer spares on-site to cover hardware failures.

We have also rolled out more large VPS hosts into our other data centres increasing our total capacity (both SSD and HDD) and making larger VM sizes more readily available across all our UK sites. We’ve also completed decommissioning all of our older hosts with less than 256GB of RAM.

As of yesterday, new support tickets are now being handled in our new support system, which is powered by Request Tracker (RT). Hopefully from the outside, things look exactly as they did before: you email support@mythic-beasts.com, your request gets assigned a ticket number, and you get a timely and helpful response from our non-dedicated support team.

From the inside, this change is pretty daunting. Our support system is absolutely critical to our day-to-day work, and learning a new tool, and adapting all of our customer-facing process to use it is a big change. There are lots of little things like the canned “snippets” that we use when composing replies to common questions, and the processes to be followed when contacting customers in response to automated alerts. Doing this from a clean slate would be hard enough, but we also have to provide continuity for existing tickets, which can remain active for weeks, months or very occasionally, years.

We’ve been tackling the migration as incrementally as possible. Most internal, and certain external tickets have been handled in RT for some weeks now, but yesterday was the day that we flipped the switch on the support@ fire hose.

Our previous system has served us well for the 16 years that we’ve been using it, but it suffers from being closed source. There have been various enhancements that we’d like to have made, but we couldn’t, and ultimately our migration away from it has been forced by the fact that the standalone product is no longer maintained, having been replaced by SaaS version – an option that is out of the question for us.

Of course, not having access to the source code has made migrating away from it harder.

Open Source or bust

We actually evaluated RT a bit over twenty years ago, and decided that it wasn’t right for us, based mostly on some requirements that in hindsight seem a bit odd (RT has also matured a bit since then!) In selecting RT this time, we carried out a thorough evaluation of available ticketing systems, with one absolutely non-negotiable requirement: must be open source and hosted in house. We simply cannot afford for such a critical and hard-to-migrate part of our business to be locked in to closed source software, let alone a SaaS service.

Our primary goal with the new system so far has been to replicate the functionality that we had before, but once it is bedded in, we plan to make further enhancements to improve integration with our other software. Whilst in the short term we hope that you’ll see no change, in the long-term we hope that customers will see improvements to how we handle support.

We’re considering adding Request Tracker to our list of managed applications. Please drop us an email if this is something that you might be interested in.

We’ve long said that the greatest achievement of the modern tech industry is to lower standards so far that providing a service that works at all is regarded as exceptional service. One thing this implicitly includes is service cancellation. We believe in the extremely simple principle that if you don’t want or no longer require a service you can cancel it. If you don’t like the service you can move it to another provider – we implemented easy DNS import and export to make it easy to arrive or leave as a customer.

Our control panel allows you to cancel any or all of your services at the end of the billing period, with an optional note to explain why.

There’s no retentions team, you don’t have to fill in a paper form, there’s no huge contract lock in period, you don’t need a doctors note or a letter from your Mum or to be passed around five different departments and we won’t find a mysterious term in a 150 page long contract that insist on you paying for another two decades. We also don’t waste your time with seemingly infinite customer satisfaction surveys where you can rate us from one smiley to nine stars.

We welcome the new US government legislation to require all providers to match our simple cancellation process. As a company owned by founders and staff we are still guided by our desire to offer services good enough that we would willingly want to pay for them.

At the end of July, Debian 10 (Buster) reached end-of-life, and with it, all mainstream support for Python 2. The Python Software Foundation actually ended support for Python 2 on 1st January 2020, but it’s remained in Debian until now because a small number of important packages depend on it.

One of those packages is Mailman 2, a widely-used mailing list manager.  For various reasons, many projects using Mailman 2 have resisted upgrading to its successor, Mailman 3.  With Debian 10 reaching end-of-life, we’re seeing renewed interest in this migration.

One of the barriers to migrating to Mailman 3 is that the upgrade breaks links to messages in the mailing list archives.  There are links all over the internet to messages in Mailman 2 list archives, and for many projects, breaking these links would be a significant loss.

We’ve recently done some Mailman 2 to 3 migrations, and as part of this, we developed our own solution to preserving archive URLs.  We created a script that trawls Mailman 2 archives and creates a map of old URLs in the Mailman 2 archives to the corresponding URL in the Mailman 3 archives.  This can be used by Apache’s mod_rewrite to generate redirects for the old URLs.   The map can be converted to a DBM file for more efficient lookups.  This is important for archives containing many thousands of URLs.

We’ve made the mailman-archive-mapper script freely available on GitHub.

We offer mailman as a managed application.

 

In March 2004 we moved all three of our servers into a single rack in the 6/7 Harbour Exchange data centre, operated at the time by Redbus.  The data centre has changed hands several times, and merged with the building next door to become what is now Equinix LD8. We’ve been continuously present for 20 years and 1 month. Normally moving out of a data centre is a difficult, expensive and time consuming operation that is best avoided, but Equinix offered us terms that made doing so make sense. In September 2023 we opened our new core point of presence in Telehouse South.

We’re happy to report this project is now complete and our footprint in Equinix LD8 is now reduced to an optical-only point of presence forwarding 10Gbps waves to our core site at City Lifeline.

Our new space in Telehouse South offers a considerable upgrade over what we could offer in LD8. All servers now have remotely switchable dual power feeds and with dual 10Gbps uplinks. We are able to offer offer cross-connects to anywhere in the Telehouse London campus and 10Gbps wavelengths back to our other sites. We already have some new colocation customers taking advantage of these additional services. We still include serial for out-of-band server management.

During this move, we live migrated our virtual server cloud to hosts in either City Lifeline or Sovereign House. Apart from a few special cases supporting very old virtual servers or ones with BGP transit services, this was done without interruption to the client. Dedicated servers and colocation customers moved in a series of windows to minimise downtime while the servers were relocated.

We brought on additional network capacity as part of the move including 10Gbps and 100Gbps links to transit providers and private peers within the Telehouse London campus. This provides a significant upgrade in connected external capacity.

Mythic Beasts is now a verified Green Hosting Provider according to the Green Web Foundation.

We’ve demonstrated to the Green Web Foundation that all our UK and EU data centres buy as much renewable electricity as they use. This hasn’t changed our operations; internally we met this requirement in 2018. What’s changed is that we’ve now provided all the documentation to meet the certification standards of the Green Web Foundation.

Of course this isn’t quite the same as saying that all the electricity we use comes from renewable power. Ultimately, the electrical energy from a wind farm isn’t tagged to flow directly to the data centres we use and there is also no requirement that the electricity is bought at exactly the same time it is used. Similarly, the data centres have fossil-fueled generator backup which means small amounts of fossil energy are still used.

That said, we do believe that this is an important and useful step in the right direction. By getting verified under this scheme we, and the 429 other verified companies, apply pressure on the data centre suppliers to buy and use renewable energy which strongly encourages the marketplace to build more renewable generation.

Some of our data centre providers are very large well-resourced companies and they place very large long term orders for renewable power. This means renewable power providers can secure funding to build out renewable power generation. When they want to build a data centre, they also have to fund the building of an equivalent amount of renewable generation to power it.