Modern email involves a confusing array of different acronyms. Most of these are attempts to fix the problem of email being fundamentally insecure, with no way to authenticate the sender of an email. The remainder are attempts to fix the new problems created by attempting to fix the first problem.

This blog post tries to provide a concise glossary of these different technologies, and the associated DNS records and URLs needed to make them go.

SPF: Sender Policy Framework

Publish a list of servers that are allowed to send mail from your domain.

SPF is published as a DNS TXT record for the domain itself (an apex record), which states which servers are allowed to send mail from your domain. For example:

example.com. IN TXT "v=spf1 include:_spf.mythic-beasts.com ~all"

SPF breaks email forwarding, unless you use SRS. SPF only restricts the “envelope sender”, which is not normally visible to end users.

SRS: Sender Rewriting Scheme

Rewrite sender addresses when forwarding mail in order to avoid failing SPF checks.

SRS is a technique used when forwarding email that replaces the original sender address with an address in your own domain. SRS only affects the envelope sender, which is not normally visible to end users. SRS allows email forwarding to work with SPF.

DKIM: DomainKeys Identified Mail

Digitally sign email envelopes.

Public keys are published in DNS records, allowing recipients to verify that the email is authentic. Public keys are published as TXT records within the _domainkey subdomain. For example:

mythic-beasts-k1._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG...."

The first part of the hostname is a unique identifier for the key. The signature is added to the email as a DKIM-Signature header. The header includes a field (the s field) with the name of the key used to sign the message.

DMARC: Domain-based Message Authentication, Reporting and Conformance

Tell recipients to reject email from your domain if it isn’t DKIM signed and doesn’t pass SPF.

DMARC is a mechanism that allows a domain owner to assert that all email sent will pass either DKIM or SPF validation, and if it doesn’t recipients should reject it. Subtly changes SPF behaviour so that it binds the “From” address (which is the one that users usually see) rather than the envelope sender. Breaks email forwarding even with SRS unless messages are DKIM signed. Example DMARC record:

_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=postmaster@example.com"

ARC: Authenticated Received Chain

Enable mailing lists to forward DKIM-signed emails.

ARC allows a system that forwards mail to provide a digitally-signed summary of the results of SPF, DKIM and DMARC validation at the point thta it was received by the forwarding server. This enables the intermediary system to make changes to the message (such as adding a mailing list footer) that will break the original DKIM signature, but still allow the final receiver to verify its integrity. ARC relies on recipients trusting the intermediary. ARC uses the same DNS-published DomainKeys as DKIM.

DANE: DNS-based Authentication of Named Entities

Publish TLS certificates in DNS, and require TLS when connecting to your servers.

DANE is not specific to email, but it can be used to enforce the use of secure TLS connections when mail servers talk to each other. In the absence of DANE, mail servers will generally try to use TLS if possible, but fall back on an insecure connection if it doesn’t work. By publishing a TLSA DNS record, domains can enforce that TLS is used when delivering mail to the servers listed in its MX records. DANE relies on DNSSEC, and also provides an alternative to Certificate Authority-based authentication of TLS certificates.

MTA-STS: Mail Transfer Agent Strict Transport Security

Require TLS when connecting to your servers by publishing a policy at a well known URL.

MTS-STS allows you to require secure TLS connections when mail servers connect to your server by publishing a machine-readable policy at a well-known URL (https://example.com/.well-known/mta-sts.txt). MTA-STS is an HTTPS-based alternative to the TLS-enforcement part of DANE.

Early in 2023, we started providing sponsored hosting to the Open Rights Group (ORG), in order to support their campaign for digital rights in the UK.  Our motivation for sponsoring them is simple: we require security in order to provide our services, and our encryption requirements have only increased since we last mocked the government’s plans to ban it. The current government’s latest efforts on this front underline the ongoing importance of the ORG’s work.

Encrypted data can only be accessed by the holder of the key. This locks out everyone else – including the government. Successive governments keep trying for a magical solution which gives them, and only them, access too, despite the fact that such an approach only serves to undermine the security of the legitimate, everyday uses of encryption. It turns out that when faced with implementing the impossible, providers will simply turn off security entirely, as Apple have just done by disabling Advanced Data Protection for UK and only UK users.

We continue to support the Open Rights Group and their work to try and allow UK users to secure their own data.  Our sponsored services provide public facing servers for the ORG website, the @openrightsgroup@social.openrightsgroup.org fediverse presence and the blocked.org.uk service which tracks which sites have been blocked by major UK ISPs. We also host internal  Nextcloud, Collabora Office, Matrix and email services.

 

Post by @openrightsgroup@social.openrightsgroup.org

View on Mastodon

 

A History Lesson

Back in the mists of time, not that long ago, you could happily forward email to another account sending all your email to a single mailbox and replying through your internet provider’s SMTP server with whatever address you wanted as the sender.

Unfortunately, back in the 1970s, someone realised you could send that new fancy ‘electronic mail’ to people you didn’t know. The few hundred users affected were rather negative about it, and it didn’t really catch on. But, within a couple of decades, the ARPANET became the Internet, and unsolicited email started to become annoying.

With the ‘Information Superhighway’ appearing in the mid 90s and the growth of personal computers, the Internet started to grow rapidly, and ‘spamming’ (a term adopted from Usenet) became more common.

Blacklists and whitelists were developed, with lists of known spammer email addresses and IPs which were known to send spam, but the fact is that email addresses could be trivially faked, and there were always other servers to send mail through.

Over the years there have been various different approaches to authenticating email senders, but it’s a continual arms race. Spam remains a problem, as does the plethora of scams and fraud that are enabled by the ease with which email can be forged.

SPF & SRS

In the early 2000s,  ‘Sender Policy Framework’ (SPF) started to gain traction. SPF works by publishing a DNS record that declares which servers are allowed to send mail from your domain.

Unfortunately, SPF had two fatal flaws:

1. Pretty much by design, it breaks email forwarding. You can’t, for example, use our servers to simply forward mail to your Gmail account, because our servers aren’t allowed to send mail from the original sender’s domain.
2. SPF restricts the envelope sender, which is the hidden address that bounce messages get sent to. The one that users actually see in their mail client – the “From” address – is still unrestricted!

(1) could be fixed by another Three Letter Acronym: SRS or ‘Sender Rewriting Scheme’. As the name suggests, this replaces the original sender address with a different address in your own domain, and because this sender is usually hidden, the received mail looks no different. We’ve supported SRS forwarding on our servers for a long time, and until recently, it worked well.

(2) was a trickier problem, which we’ll come back to…

DKIM

In the 2010s and with processing becoming cheaper, ‘DomainKeys Identified Mail’ (DKIM) also appeared, which electronically signs the mail so that the remote server can verify the signature. The public keys are published in the sender’s domain’s DNS, so recipients can verify that mail came from the claimed sender.

DMARC

The issue with DKIM is that it’s not universally adopted, so whilst you can check if a signed mail is legitimate, you can’t assume that an unsigned email isn’t. Enter yet another acronym: DMARC (‘Domain-based Message Authentication, Reporting and Conformance’).  DMARC allows a domain-owner to publish yet another DNS record which says “all my mail will either be DKIM signed, or pass SPF, or both, and if it isn’t, bin it!”

Adoption of DMARC has been slow, not helped by the fact that publishing a strict DMARC policy in 2014 broke just about every mailing list of the time. Fast-forward ten years, mailing lists have learned to adapt, and the big mail providers are pushing the world towards DMARC. Gmail, probably the largest free email provider, now requires either SPF or DKIM to be enabled on a domain in order for them to accept any mail from it, and if you’re sending bulk email, you need a DMARC record too.

The Effect On Forwarding Mail

While these measures have all helped greatly to combat spam, with a large proportion of it now coming from compromised servers or mailboxes, it’s had some down-sides, the most obvious of which is difficulty in forwarding email.

Lots of our customers like to register a domain with us, set up some email addresses in that domain, and forward them to existing email accounts at other providers, such as Gmail or Yahoo. We also host lots of clubs and other organisations that have role-based addresses (like “treasurer@myclub.org”) that forward to the person currently doing that role, and mail expanders like “committee@myclub.org” that forward to a handful of people.

DMARC alignment

SPF is now widely adopted and, as noted above, you used to be able to work around the issue that it creates for forwarding using SRS.

But, one of the subtle features of DMARC is that it quietly fixes problem (2) with SPF. In order to pass SPF in a way that makes DMARC happy, the hidden sender address that SPF restricts has to be in the same domain as the visible “From” address (this is called “alignment”).

• If you forward without SRS, the messages that you forward will fail an SPF check.
• If you forward with SRS, the messages will technically pass SPF, but DMARC will call it a fail because the sender and from addresses don’t match.

In other words, there is no way to pass a DMARC check using SPF when forwarding mail. You have to rely on DKIM, and hope that forwarding mail (with or without SRS) doesn’t break DKIM.

You would hope that senders wouldn’t publish a strict DMARC policy without first ensuring that all of their outbound mail is DKIM-signed, but you may be disappointed, and with mail providers increasingly enforcing DMARC policies, mail forwarding has become less and less reliable.

Yet another acronym – ARC – promises to fix mail forwarding, but it’s not a magic bullet. We’ll discuss that another time.

Of course, forwarding mail works fine if you’re in control of the server you’re forwarding to, and can whitelist the server doing the forwarding, but that’s not common. Most people want to forward to providers like Gmail where they have no such control.

For some platforms there is a workaround; rather than forwarding mail, have the platform collect it from a mailbox using POP3 or IMAP.  For example, Gmail have their “check messages from other accounts” functionality in their mailboxes, which picks up you mail from a mailbox as if it’s been received by the Gmail mailbox directly, and other platforms are adding similar options.

This works well enough for personal email addresses, but it’s a lot more hassle for things like role-based addresses at emails at clubs and societies, and doesn’t work at all for mail expanders where multiple recipients need to collect the mail.

This is why we can’t have nice things.

With our successful move out of Harbour Exchange last year we’ve been making all our services available in Telehouse. We have now deployed a virtual server cluster in Telehouse and those of you who avidly watch our order forms have been ordering virtual servers there for a few months.

The addition of virtual servers in Telehouse means that we can now offer three London locations for users wishing to run distributed clusters.

We’ve previously offered a choice between SSD storage for performance, and HDD storage for capacity. In the last few years, almost all new virtual servers orders have been for the SSD option so in Telehouse we took the decision to simplify our operations and only offer SSD-based virtual servers using high performance NVMe storage. Storage is the limiting factor on our virtual server hosts, so faster storage means we can run more VPSs on each host without compromising performance. We never over-sell RAM on our VPS platform — a 4GB VPS is backed by 4GB of real RAM — so our newest host servers now have 512GB of RAM and 5th Generation Intel Scalable Xeon CPUs. Storage is mirrored pairs of enterprise NVMe drives, and the servers have dual power supplies and dual 10G uplinks. Higher capacity host servers allow us to pack more virtual servers into the data centre space, and simplifying our configuration requires us to hold fewer spares on-site to cover hardware failures.

We have also rolled out more large VPS hosts into our other data centres increasing our total capacity (both SSD and HDD) and making larger VM sizes more readily available across all our UK sites. We’ve also completed decommissioning all of our older hosts with less than 256GB of RAM.

As of yesterday, new support tickets are now being handled in our new support system, which is powered by Request Tracker (RT). Hopefully from the outside, things look exactly as they did before: you email support@mythic-beasts.com, your request gets assigned a ticket number, and you get a timely and helpful response from our non-dedicated support team.

From the inside, this change is pretty daunting. Our support system is absolutely critical to our day-to-day work, and learning a new tool, and adapting all of our customer-facing process to use it is a big change. There are lots of little things like the canned “snippets” that we use when composing replies to common questions, and the processes to be followed when contacting customers in response to automated alerts. Doing this from a clean slate would be hard enough, but we also have to provide continuity for existing tickets, which can remain active for weeks, months or very occasionally, years.

We’ve been tackling the migration as incrementally as possible. Most internal, and certain external tickets have been handled in RT for some weeks now, but yesterday was the day that we flipped the switch on the support@ fire hose.

Our previous system has served us well for the 16 years that we’ve been using it, but it suffers from being closed source. There have been various enhancements that we’d like to have made, but we couldn’t, and ultimately our migration away from it has been forced by the fact that the standalone product is no longer maintained, having been replaced by SaaS version – an option that is out of the question for us.

Of course, not having access to the source code has made migrating away from it harder.

Open Source or bust

We actually evaluated RT a bit over twenty years ago, and decided that it wasn’t right for us, based mostly on some requirements that in hindsight seem a bit odd (RT has also matured a bit since then!) In selecting RT this time, we carried out a thorough evaluation of available ticketing systems, with one absolutely non-negotiable requirement: must be open source and hosted in house. We simply cannot afford for such a critical and hard-to-migrate part of our business to be locked in to closed source software, let alone a SaaS service.

Our primary goal with the new system so far has been to replicate the functionality that we had before, but once it is bedded in, we plan to make further enhancements to improve integration with our other software. Whilst in the short term we hope that you’ll see no change, in the long-term we hope that customers will see improvements to how we handle support.

We’re considering adding Request Tracker to our list of managed applications. Please drop us an email if this is something that you might be interested in.

We recently went to the UK IPv6 Council annual meeting, ten years since the first one. In the intervening time, IPv6 usage in the UK has grown from 0.2% of connections to 48% today; almost half the country is IPv6-enabled. There was lots of interesting material about IPv6-only and “IPv6 mostly” networks, in addition to dual-stack networks.

IPv6 Mostly

“IPv6 mostly” networks are dual stack networks that provide NAT64 and DNS64 servers. DNS64 provides synthesised IPv6 addresses for IPv4-only resources, and the NAT64 service then provides translation between the two. Some software is incompatible because it tries to talk directly to IPv4 addresses which can’t be reached. Modern computers and phones offer CLAT which bridges this gap. A network client using CLAT in a network that offers NAT64 and DNS64 no longer needs to be dual stack and can turn direct IPv4 off.

DHCP has a new option: Option 108: ‘IPv6-Only preferred’. About 75% of clients – mostly phones, tablets and OSX devices – will specify this option. When present on both server and client, the client won’t request an IPv4 address from the DHCP server, and will operate only with IPv6 addresses. Imperial College London have rolled this out on their wifi network. Of the 71,000 devices using their network, only 16,000 request an IPv4 address. 77% are IPv6-only.

IPv4 as a service

Sky rolled out a network in Italy which is internally IPv6-only, and IPv4 traffic is layered on top using MAP-T. This means the broadband box translates all IPv4 traffic into IPv6 as it enters the network, then a MAP box turns it back into IPv4 as it leaves the Sky network to get to the origin. IPv6 traffic skips both transitions. As a large eyeball network, they have network cache devices in their network. If the traffic flow is IPv4, it is terminated on the cache box in one of the four points key points-of-presence. IPv6 flows can terminate on cache boxes anywhere in the network – crucially closer and faster to the end user.

Other updates

Vodafone started and has nearly finished dual-stacking their network. The motivation was to reduce the IPv4 and carrier grade NAT costs. Today 75% of their customers have IPv6 and 38% of their traffic flows over IPv6. Microsoft talked about all their work on the operating system side to support and proxy IPv6, with the consensus being very clear that CLAT and DHCP option 108 was the most important thing they should have delivered last year.

We’ve long said that the greatest achievement of the modern tech industry is to lower standards so far that providing a service that works at all is regarded as exceptional service. One thing this implicitly includes is service cancellation. We believe in the extremely simple principle that if you don’t want or no longer require a service you can cancel it. If you don’t like the service you can move it to another provider – we implemented easy DNS import and export to make it easy to arrive or leave as a customer.

Our control panel allows you to cancel any or all of your services at the end of the billing period, with an optional note to explain why.

There’s no retentions team, you don’t have to fill in a paper form, there’s no huge contract lock in period, you don’t need a doctors note or a letter from your Mum or to be passed around five different departments and we won’t find a mysterious term in a 150 page long contract that insist on you paying for another two decades. We also don’t waste your time with seemingly infinite customer satisfaction surveys where you can rate us from one smiley to nine stars.

We welcome the new US government legislation to require all providers to match our simple cancellation process. As a company owned by founders and staff we are still guided by our desire to offer services good enough that we would willingly want to pay for them.

We’ve always taken a boringly old-fashioned approach to our domain registration business. We take the price that our supplier charges us, add a little bit more than it costs us to provide the service and sell them on. The price we charge covers the cost of providing support, DNS, and other things like foreign exchange and payment fees.

We don’t do introductory “loss-leaders” – selling domains at a loss for the initial period and then ramping up the price for renewals^*. Our domain business isn’t a loss leader for other services, and we don’t offer barely-profitable prices that need to be subsidised by high margin add-ons that you don’t really need.

In fact, our domain pricing structure is so boring, that we wrote a simple script to do it for us. It pulls in wholesale prices from our supplier via an API, takes the current exchange rate, adds our margin, and sets our list prices.

When we set our current price structure, there were only a small number of Generic Top-Level Domains (gTLDs) and they were mostly a similar price. Country-code TLDs (ccTLDs) are often more expensive, but they’re also often more bureaucratic and more expensive to support.

These days there’s a huge number of gTLDs with a huge range in price, with some gTLDs costing hundreds of pounds per year and the assumption that the more expensive domains are those with greater bureaucracy no longer holds.

In recognition of this, we’ve just rolled out a significant update to our pricing structure that better models our costs. The net result is that we’ve reduced the price of all non-UK domains with a one-year cost of more than £30+VAT and we’ve also increased the discount we offer for multi-year renewals and registrations across all domains.

You can view our updated prices at mythic-beasts.com/domains

The price reductions apply to renewals and transfers as well as new registrations, and as always, existing customers get the same prices as new customers.

All domain registrations include our DNS service, with a powerful DNS API, and support for DNSSEC (where supported by the gTLD).

Our sustainable pricing model means we have no need for nasty lock-in schemes. We’ve never charged transfer-out fees, and our DNS service makes it really easy to import and export your DNS records.

^* Registries do sometimes offer promotional first year pricing.  In this case, we will pass on these discounts, but make the normal price clear in our price list.

This year the back-end system for domain names ending in .gov.uk has migrated from JISC (the organisation that provides connectivity services to Higher Education in the UK) to Nominet (the organisation responsible for nearly all other .uk domains).

This brings a few significant changes for registrars who offer .gov.uk domain names:

• There are enhanced security requirements for registrars. Nominet requires Cyber Essentials.
• The wholesale costs have been reduced.
• Registration periods other than 2 years are now possible.
• All the processes and procedures are different: somethings are now easier, others are harder.

Mythic Beasts were already Nominet members and have now added the ability to register .gov.uk domain names through Nominet. All of our existing .gov.uk domain name customers have been migrated across transparently. We have also applied a price decrease, renewals are now £15+VAT rather than £75+VAT, and new domain registrations have also been reduced from £150+VAT for the first two years, to £89.50+VAT.

We’re aware that a number of registrars have opted not to continue offering .gov.uk domains due to the procedure changes and security requirements. We are able to accept transfers of .gov.uk domains in, and there is no charge for transferring a domain name to us. We can also accept payment by bank transfer on account which is extremely helpful for government bodies that register .gov.uk domains.

The transfer process is very simple, set up an account at the Mythic Beasts control panel and go to transfer domain in. It will ask you for the authorisation code and we’ll complete the transfer for you.

As of March 2023 there were 3854 .gov.uk domains in use – if every registrar follows our lead that should reduce the government deficit by around £200,000/year.

At the end of July, Debian 10 (Buster) reached end-of-life, and with it, all mainstream support for Python 2. The Python Software Foundation actually ended support for Python 2 on 1st January 2020, but it’s remained in Debian until now because a small number of important packages depend on it.

One of those packages is Mailman 2, a widely-used mailing list manager.  For various reasons, many projects using Mailman 2 have resisted upgrading to its successor, Mailman 3.  With Debian 10 reaching end-of-life, we’re seeing renewed interest in this migration.

One of the barriers to migrating to Mailman 3 is that the upgrade breaks links to messages in the mailing list archives.  There are links all over the internet to messages in Mailman 2 list archives, and for many projects, breaking these links would be a significant loss.

We’ve recently done some Mailman 2 to 3 migrations, and as part of this, we developed our own solution to preserving archive URLs.  We created a script that trawls Mailman 2 archives and creates a map of old URLs in the Mailman 2 archives to the corresponding URL in the Mailman 3 archives.  This can be used by Apache’s mod_rewrite to generate redirects for the old URLs.   The map can be converted to a DBM file for more efficient lookups.  This is important for archives containing many thousands of URLs.

We’ve made the mailman-archive-mapper script freely available on GitHub.

We offer mailman as a managed application.