Robocon is a competition run by Hills Road Sixth Form College in Cambridge. Teams of secondary school students build autonomous robot dragons which rampage around the area stealing sheep and gems and taking them back to their lair. The more things your robot dragon steals the better. As soon as we heard there were Robots and Dragons we sponsored one of the teams: the Little Devils.

The Little Devils are five primary school girls who entered a competition for secondary students because the rules didn’t forbid it, and somehow everyone forgot to tell them it was too hard for them. Which is just as well: it wasn’t.

The contest

The goal of the contest is to build a robot and program it to complete a set of challenges autonomously (without human input) within a specially designed arena. Each team gets a quarter of the arena as their home area. At the back of the area is their lair. The arena contains sheep and gems (cardboard cubes) and points are awarded for each sheep returned to the team’s home area, with extra points for placing a sheep or gem on top of the lair. This involves picking the cube up and placing it on a 15cm high shelf.

2D barcodes are displayed around the arena, on the lairs, and on sheep and gems.

Each team is supplied with a ‘brain’ as a starting point for building their robot. This is a Raspberry Pi Zero with an attached camera. This comes with OpenCV and some libraries that tell you every time the camera sees a 2D barcode, along with an estimate of how far away the barcode is and what angle it’s at. The brain also has motor controllers, and general purpose IO.

Collecting sheep and gems is made even trickier by the fact that once you get close to one, the camera can’t focus on it, so the robot has to guess exactly where to stop in order to pick it up. Once picked up, the robot can then search for the marker for its lair and drive towards it to drop the item off. If they taught trigonometry in primary school they could write more sophisticated software that would work out how to travel based on any marker they can see. We’re already looking forward to next year.

This year’s competition took place on 9th and 10th April, and we were pleased to see a variety of robot designs and programming approaches. Some teams tried to round up as many sheep into their space as possible. The Little Devils’ approach to collecting sheep and gems was a vacuum grabber which would try to pick up the cubes and then drive them back to their lair to get the bonus points.

The Robot

One of the most important skills in professional programming is the rubber duck technique. This robot has a Redundant Array of Independent Ducks such that programming can continue at full speed even if a duck is lost in the battle. The ducks were a late addition, and resulted in some last-minute code changes; in testing the Dragon wasn’t laden with ducks and moved faster, so some constants in the code had to be tweaked in order to drive the motors harder to cope with the additional weight from the ducks.

The robot has a software-controlled robot arm to pick up the sheep and gems. There’s a vacuum pump that switches on when the robot thinks it has a gem. Once the pump is on, it lifts up the arm high in order to clear the camera’s view. There are two powered wheels; selectively driving one or both allows the robot to turn or travel forwards or backwards. All of the software was written by the team members with the robot manufacturing done under adult supervision.

The result

We’re really pleased to report that the Little Devils took the third place trophy. We are somewhat concerned that this appears to be the plot for an origin story where a set of super villains destroy humanity with their robot army. Perhaps we should have put “you must not take over the world” in the sponsorship agreement.

As many other folks have reported in the last few weeks, we have also been seeing a huge increase in the amount of traffic from abusive web crawlers.

Automated blocking of abusive traffic has long been a necessary evil. We already block a number of badly behaved SEO and AI crawlers on our shared hosting servers and, on-request, some customer servers. We also have a number of automatic tools to block abusive clients. These are typically attempting to brute force passwords or run web security scanners, and we firewall IP addresses out after a number of suspicious requests. These crawlers and bad-actors can already outnumber real organic traffic but the scale of the recent activity, along with the lengths taken to frustrate automated blocking, are on another level.

This new attack comes from a great many IP addresses, each making a tiny number of requests – often just one – from viable-looking but randomly generated User-Agents. We’ve had some success detecting and blocking these but this has not been without some problems. There have been periods where some of our servers have been struggling under the sheer number of connections they’ve had to deal with and some of the blocks we’ve put in place have impacted some legitimate users, especially those on very old computers. If this is you then we’re sorry you’ve been caught up in this.

To give you some idea of the scale, one of our shared hosting servers has in the last month been averaging over 1.5 million fraudulent requests from 290,000 unique IP addresses per day. These are addresses that we have a very high confidence are not making legitimate requests. We’ve identified 5.1 million unique IP addresses during this period and 3.4 million of those have only made a single request, which has made it very difficult for us to block them proactively.

From these IPs, we’ve seen 2.4 million unique User-Agents, and again 1.9 million of these have only been seen once. We’d certainly be surprised if there are as many Windows 95 users left as we’ve seen in these logs. Especially with the ability to talk to a modern TLS-enabled website.

The vast majority of these requests are from consumer ISP networks from a wide variety of countries with Brazil being the biggest contributor by far. UK networks only make up around 2% of the attacks we’ve seen but it’s the same pattern – it’s the big consumer ISPs we see the most. We’ve been careful to exclude any IP addresses that also show what looks like legitimate activity here, so it’s possible this is undercounting, especially with the growth of CGNAT. Around 5% of the requests are from IPv6 addresses.

The rumours that this is a botnet mostly made up of compromised Android SetTop Boxes that’s been leased out to an AI crawler that’s trying to avoid being blocked seem likely to us, but we’re not impressed. This has been a significant waste of staff time over the last few weeks as we’ve worked to mitigate the impact on our customers.

If you’re an ISP that wants to know if you’re part of this botnet please check to see if your ASN is in this file then contact support for a copy of our logs. There are over 22,000 distinct ASNs in our data, with more than 200 of those networks based in the UK.

Thanks to bgp.tools for the network and country data.

Modern email involves a confusing array of different acronyms. Most of these are attempts to fix the problem of email being fundamentally insecure, with no way to authenticate the sender of an email. The remainder are attempts to fix the new problems created by attempting to fix the first problem.

This blog post tries to provide a concise glossary of these different technologies, and the associated DNS records and URLs needed to make them go.

SPF: Sender Policy Framework

Publish a list of servers that are allowed to send mail from your domain.

SPF is published as a DNS TXT record for the domain itself (an apex record), which states which servers are allowed to send mail from your domain. For example:

example.com. IN TXT "v=spf1 include:_spf.mythic-beasts.com ~all"

SPF breaks email forwarding, unless you use SRS. SPF only restricts the “envelope sender”, which is not normally visible to end users.

SRS: Sender Rewriting Scheme

Rewrite sender addresses when forwarding mail in order to avoid failing SPF checks.

SRS is a technique used when forwarding email that replaces the original sender address with an address in your own domain. SRS only affects the envelope sender, which is not normally visible to end users. SRS allows email forwarding to work with SPF.

DKIM: DomainKeys Identified Mail

Digitally sign email envelopes.

Public keys are published in DNS records, allowing recipients to verify that the email is authentic. Public keys are published as TXT records within the _domainkey subdomain. For example:

mythic-beasts-k1._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG...."

The first part of the hostname is a unique identifier for the key. The signature is added to the email as a DKIM-Signature header. The header includes a field (the s field) with the name of the key used to sign the message.

DMARC: Domain-based Message Authentication, Reporting and Conformance

Tell recipients to reject email from your domain if it isn’t DKIM signed and doesn’t pass SPF.

DMARC is a mechanism that allows a domain owner to assert that all email sent will pass either DKIM or SPF validation, and if it doesn’t recipients should reject it. Subtly changes SPF behaviour so that it binds the “From” address (which is the one that users usually see) rather than the envelope sender. Breaks email forwarding even with SRS unless messages are DKIM signed. Example DMARC record:

_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=postmaster@example.com"

ARC: Authenticated Received Chain

Enable mailing lists to forward DKIM-signed emails.

ARC allows a system that forwards mail to provide a digitally-signed summary of the results of SPF, DKIM and DMARC validation at the point thta it was received by the forwarding server. This enables the intermediary system to make changes to the message (such as adding a mailing list footer) that will break the original DKIM signature, but still allow the final receiver to verify its integrity. ARC relies on recipients trusting the intermediary. ARC uses the same DNS-published DomainKeys as DKIM.

DANE: DNS-based Authentication of Named Entities

Publish TLS certificates in DNS, and require TLS when connecting to your servers.

DANE is not specific to email, but it can be used to enforce the use of secure TLS connections when mail servers talk to each other. In the absence of DANE, mail servers will generally try to use TLS if possible, but fall back on an insecure connection if it doesn’t work. By publishing a TLSA DNS record, domains can enforce that TLS is used when delivering mail to the servers listed in its MX records. DANE relies on DNSSEC, and also provides an alternative to Certificate Authority-based authentication of TLS certificates.

MTA-STS: Mail Transfer Agent Strict Transport Security

Require TLS when connecting to your servers by publishing a policy at a well known URL.

MTS-STS allows you to require secure TLS connections when mail servers connect to your server by publishing a machine-readable policy at a well-known URL (https://example.com/.well-known/mta-sts.txt). MTA-STS is an HTTPS-based alternative to the TLS-enforcement part of DANE.

Early in 2023, we started providing sponsored hosting to the Open Rights Group (ORG), in order to support their campaign for digital rights in the UK.  Our motivation for sponsoring them is simple: we require security in order to provide our services, and our encryption requirements have only increased since we last mocked the government’s plans to ban it. The current government’s latest efforts on this front underline the ongoing importance of the ORG’s work.

Encrypted data can only be accessed by the holder of the key. This locks out everyone else – including the government. Successive governments keep trying for a magical solution which gives them, and only them, access too, despite the fact that such an approach only serves to undermine the security of the legitimate, everyday uses of encryption. It turns out that when faced with implementing the impossible, providers will simply turn off security entirely, as Apple have just done by disabling Advanced Data Protection for UK and only UK users.

We continue to support the Open Rights Group and their work to try and allow UK users to secure their own data.  Our sponsored services provide public facing servers for the ORG website, the @openrightsgroup@social.openrightsgroup.org fediverse presence and the blocked.org.uk service which tracks which sites have been blocked by major UK ISPs. We also host internal  Nextcloud, Collabora Office, Matrix and email services.

 

Post by @openrightsgroup@social.openrightsgroup.org

View on Mastodon

 

A History Lesson

Back in the mists of time, not that long ago, you could happily forward email to another account sending all your email to a single mailbox and replying through your internet provider’s SMTP server with whatever address you wanted as the sender.

Unfortunately, back in the 1970s, someone realised you could send that new fancy ‘electronic mail’ to people you didn’t know. The few hundred users affected were rather negative about it, and it didn’t really catch on. But, within a couple of decades, the ARPANET became the Internet, and unsolicited email started to become annoying.

With the ‘Information Superhighway’ appearing in the mid 90s and the growth of personal computers, the Internet started to grow rapidly, and ‘spamming’ (a term adopted from Usenet) became more common.

Blacklists and whitelists were developed, with lists of known spammer email addresses and IPs which were known to send spam, but the fact is that email addresses could be trivially faked, and there were always other servers to send mail through.

Over the years there have been various different approaches to authenticating email senders, but it’s a continual arms race. Spam remains a problem, as does the plethora of scams and fraud that are enabled by the ease with which email can be forged.

SPF & SRS

In the early 2000s,  ‘Sender Policy Framework’ (SPF) started to gain traction. SPF works by publishing a DNS record that declares which servers are allowed to send mail from your domain.

Unfortunately, SPF had two fatal flaws:

1. Pretty much by design, it breaks email forwarding. You can’t, for example, use our servers to simply forward mail to your Gmail account, because our servers aren’t allowed to send mail from the original sender’s domain.
2. SPF restricts the envelope sender, which is the hidden address that bounce messages get sent to. The one that users actually see in their mail client – the “From” address – is still unrestricted!

(1) could be fixed by another Three Letter Acronym: SRS or ‘Sender Rewriting Scheme’. As the name suggests, this replaces the original sender address with a different address in your own domain, and because this sender is usually hidden, the received mail looks no different. We’ve supported SRS forwarding on our servers for a long time, and until recently, it worked well.

(2) was a trickier problem, which we’ll come back to…

DKIM

In the 2010s and with processing becoming cheaper, ‘DomainKeys Identified Mail’ (DKIM) also appeared, which electronically signs the mail so that the remote server can verify the signature. The public keys are published in the sender’s domain’s DNS, so recipients can verify that mail came from the claimed sender.

DMARC

The issue with DKIM is that it’s not universally adopted, so whilst you can check if a signed mail is legitimate, you can’t assume that an unsigned email isn’t. Enter yet another acronym: DMARC (‘Domain-based Message Authentication, Reporting and Conformance’).  DMARC allows a domain-owner to publish yet another DNS record which says “all my mail will either be DKIM signed, or pass SPF, or both, and if it isn’t, bin it!”

Adoption of DMARC has been slow, not helped by the fact that publishing a strict DMARC policy in 2014 broke just about every mailing list of the time. Fast-forward ten years, mailing lists have learned to adapt, and the big mail providers are pushing the world towards DMARC. Gmail, probably the largest free email provider, now requires either SPF or DKIM to be enabled on a domain in order for them to accept any mail from it, and if you’re sending bulk email, you need a DMARC record too.

The Effect On Forwarding Mail

While these measures have all helped greatly to combat spam, with a large proportion of it now coming from compromised servers or mailboxes, it’s had some down-sides, the most obvious of which is difficulty in forwarding email.

Lots of our customers like to register a domain with us, set up some email addresses in that domain, and forward them to existing email accounts at other providers, such as Gmail or Yahoo. We also host lots of clubs and other organisations that have role-based addresses (like “treasurer@myclub.org”) that forward to the person currently doing that role, and mail expanders like “committee@myclub.org” that forward to a handful of people.

DMARC alignment

SPF is now widely adopted and, as noted above, you used to be able to work around the issue that it creates for forwarding using SRS.

But, one of the subtle features of DMARC is that it quietly fixes problem (2) with SPF. In order to pass SPF in a way that makes DMARC happy, the hidden sender address that SPF restricts has to be in the same domain as the visible “From” address (this is called “alignment”).

• If you forward without SRS, the messages that you forward will fail an SPF check.
• If you forward with SRS, the messages will technically pass SPF, but DMARC will call it a fail because the sender and from addresses don’t match.

In other words, there is no way to pass a DMARC check using SPF when forwarding mail. You have to rely on DKIM, and hope that forwarding mail (with or without SRS) doesn’t break DKIM.

You would hope that senders wouldn’t publish a strict DMARC policy without first ensuring that all of their outbound mail is DKIM-signed, but you may be disappointed, and with mail providers increasingly enforcing DMARC policies, mail forwarding has become less and less reliable.

Yet another acronym – ARC – promises to fix mail forwarding, but it’s not a magic bullet. We’ll discuss that another time.

Of course, forwarding mail works fine if you’re in control of the server you’re forwarding to, and can whitelist the server doing the forwarding, but that’s not common. Most people want to forward to providers like Gmail where they have no such control.

For some platforms there is a workaround; rather than forwarding mail, have the platform collect it from a mailbox using POP3 or IMAP.  For example, Gmail have their “check messages from other accounts” functionality in their mailboxes, which picks up you mail from a mailbox as if it’s been received by the Gmail mailbox directly, and other platforms are adding similar options.

This works well enough for personal email addresses, but it’s a lot more hassle for things like role-based addresses at emails at clubs and societies, and doesn’t work at all for mail expanders where multiple recipients need to collect the mail.

This is why we can’t have nice things.

With our successful move out of Harbour Exchange last year we’ve been making all our services available in Telehouse. We have now deployed a virtual server cluster in Telehouse and those of you who avidly watch our order forms have been ordering virtual servers there for a few months.

The addition of virtual servers in Telehouse means that we can now offer three London locations for users wishing to run distributed clusters.

We’ve previously offered a choice between SSD storage for performance, and HDD storage for capacity. In the last few years, almost all new virtual servers orders have been for the SSD option so in Telehouse we took the decision to simplify our operations and only offer SSD-based virtual servers using high performance NVMe storage. Storage is the limiting factor on our virtual server hosts, so faster storage means we can run more VPSs on each host without compromising performance. We never over-sell RAM on our VPS platform — a 4GB VPS is backed by 4GB of real RAM — so our newest host servers now have 512GB of RAM and 5th Generation Intel Scalable Xeon CPUs. Storage is mirrored pairs of enterprise NVMe drives, and the servers have dual power supplies and dual 10G uplinks. Higher capacity host servers allow us to pack more virtual servers into the data centre space, and simplifying our configuration requires us to hold fewer spares on-site to cover hardware failures.

We have also rolled out more large VPS hosts into our other data centres increasing our total capacity (both SSD and HDD) and making larger VM sizes more readily available across all our UK sites. We’ve also completed decommissioning all of our older hosts with less than 256GB of RAM.

As of yesterday, new support tickets are now being handled in our new support system, which is powered by Request Tracker (RT). Hopefully from the outside, things look exactly as they did before: you email support@mythic-beasts.com, your request gets assigned a ticket number, and you get a timely and helpful response from our non-dedicated support team.

From the inside, this change is pretty daunting. Our support system is absolutely critical to our day-to-day work, and learning a new tool, and adapting all of our customer-facing process to use it is a big change. There are lots of little things like the canned “snippets” that we use when composing replies to common questions, and the processes to be followed when contacting customers in response to automated alerts. Doing this from a clean slate would be hard enough, but we also have to provide continuity for existing tickets, which can remain active for weeks, months or very occasionally, years.

We’ve been tackling the migration as incrementally as possible. Most internal, and certain external tickets have been handled in RT for some weeks now, but yesterday was the day that we flipped the switch on the support@ fire hose.

Our previous system has served us well for the 16 years that we’ve been using it, but it suffers from being closed source. There have been various enhancements that we’d like to have made, but we couldn’t, and ultimately our migration away from it has been forced by the fact that the standalone product is no longer maintained, having been replaced by SaaS version – an option that is out of the question for us.

Of course, not having access to the source code has made migrating away from it harder.

Open Source or bust

We actually evaluated RT a bit over twenty years ago, and decided that it wasn’t right for us, based mostly on some requirements that in hindsight seem a bit odd (RT has also matured a bit since then!) In selecting RT this time, we carried out a thorough evaluation of available ticketing systems, with one absolutely non-negotiable requirement: must be open source and hosted in house. We simply cannot afford for such a critical and hard-to-migrate part of our business to be locked in to closed source software, let alone a SaaS service.

Our primary goal with the new system so far has been to replicate the functionality that we had before, but once it is bedded in, we plan to make further enhancements to improve integration with our other software. Whilst in the short term we hope that you’ll see no change, in the long-term we hope that customers will see improvements to how we handle support.

We’re considering adding Request Tracker to our list of managed applications. Please drop us an email if this is something that you might be interested in.

We recently went to the UK IPv6 Council annual meeting, ten years since the first one. In the intervening time, IPv6 usage in the UK has grown from 0.2% of connections to 48% today; almost half the country is IPv6-enabled. There was lots of interesting material about IPv6-only and “IPv6 mostly” networks, in addition to dual-stack networks.

IPv6 Mostly

“IPv6 mostly” networks are dual stack networks that provide NAT64 and DNS64 servers. DNS64 provides synthesised IPv6 addresses for IPv4-only resources, and the NAT64 service then provides translation between the two. Some software is incompatible because it tries to talk directly to IPv4 addresses which can’t be reached. Modern computers and phones offer CLAT which bridges this gap. A network client using CLAT in a network that offers NAT64 and DNS64 no longer needs to be dual stack and can turn direct IPv4 off.

DHCP has a new option: Option 108: ‘IPv6-Only preferred’. About 75% of clients – mostly phones, tablets and OSX devices – will specify this option. When present on both server and client, the client won’t request an IPv4 address from the DHCP server, and will operate only with IPv6 addresses. Imperial College London have rolled this out on their wifi network. Of the 71,000 devices using their network, only 16,000 request an IPv4 address. 77% are IPv6-only.

IPv4 as a service

Sky rolled out a network in Italy which is internally IPv6-only, and IPv4 traffic is layered on top using MAP-T. This means the broadband box translates all IPv4 traffic into IPv6 as it enters the network, then a MAP box turns it back into IPv4 as it leaves the Sky network to get to the origin. IPv6 traffic skips both transitions. As a large eyeball network, they have network cache devices in their network. If the traffic flow is IPv4, it is terminated on the cache box in one of the four points key points-of-presence. IPv6 flows can terminate on cache boxes anywhere in the network – crucially closer and faster to the end user.

Other updates

Vodafone started and has nearly finished dual-stacking their network. The motivation was to reduce the IPv4 and carrier grade NAT costs. Today 75% of their customers have IPv6 and 38% of their traffic flows over IPv6. Microsoft talked about all their work on the operating system side to support and proxy IPv6, with the consensus being very clear that CLAT and DHCP option 108 was the most important thing they should have delivered last year.

We’ve long said that the greatest achievement of the modern tech industry is to lower standards so far that providing a service that works at all is regarded as exceptional service. One thing this implicitly includes is service cancellation. We believe in the extremely simple principle that if you don’t want or no longer require a service you can cancel it. If you don’t like the service you can move it to another provider – we implemented easy DNS import and export to make it easy to arrive or leave as a customer.

Our control panel allows you to cancel any or all of your services at the end of the billing period, with an optional note to explain why.

There’s no retentions team, you don’t have to fill in a paper form, there’s no huge contract lock in period, you don’t need a doctors note or a letter from your Mum or to be passed around five different departments and we won’t find a mysterious term in a 150 page long contract that insist on you paying for another two decades. We also don’t waste your time with seemingly infinite customer satisfaction surveys where you can rate us from one smiley to nine stars.

We welcome the new US government legislation to require all providers to match our simple cancellation process. As a company owned by founders and staff we are still guided by our desire to offer services good enough that we would willingly want to pay for them.

We’ve always taken a boringly old-fashioned approach to our domain registration business. We take the price that our supplier charges us, add a little bit more than it costs us to provide the service and sell them on. The price we charge covers the cost of providing support, DNS, and other things like foreign exchange and payment fees.

We don’t do introductory “loss-leaders” – selling domains at a loss for the initial period and then ramping up the price for renewals^*. Our domain business isn’t a loss leader for other services, and we don’t offer barely-profitable prices that need to be subsidised by high margin add-ons that you don’t really need.

In fact, our domain pricing structure is so boring, that we wrote a simple script to do it for us. It pulls in wholesale prices from our supplier via an API, takes the current exchange rate, adds our margin, and sets our list prices.

When we set our current price structure, there were only a small number of Generic Top-Level Domains (gTLDs) and they were mostly a similar price. Country-code TLDs (ccTLDs) are often more expensive, but they’re also often more bureaucratic and more expensive to support.

These days there’s a huge number of gTLDs with a huge range in price, with some gTLDs costing hundreds of pounds per year and the assumption that the more expensive domains are those with greater bureaucracy no longer holds.

In recognition of this, we’ve just rolled out a significant update to our pricing structure that better models our costs. The net result is that we’ve reduced the price of all non-UK domains with a one-year cost of more than £30+VAT and we’ve also increased the discount we offer for multi-year renewals and registrations across all domains.

You can view our updated prices at mythic-beasts.com/domains

The price reductions apply to renewals and transfers as well as new registrations, and as always, existing customers get the same prices as new customers.

All domain registrations include our DNS service, with a powerful DNS API, and support for DNSSEC (where supported by the gTLD).

Our sustainable pricing model means we have no need for nasty lock-in schemes. We’ve never charged transfer-out fees, and our DNS service makes it really easy to import and export your DNS records.

^* Registries do sometimes offer promotional first year pricing.  In this case, we will pass on these discounts, but make the normal price clear in our price list.