A rack of Pi 3s… possibly the only cloud computers immune to Spectre?

There’s been a lot of activity in the news regarding two new security issues called Meltdown and Spectre.

The security issues are newsworthy because they’re different to any security issues we’ve seen before. They’re not an issue in software, but in your computer itself. As a result the vulnerabilities cross multiple operating systems – Windows/Linux/OSX and multiple devices – Laptop/Desktop/Server/Phone, and they’re also a lot harder to fix.  Meltdown affects only Intel processors.  Spectre also affects AMD, Power, RISC and some ARM CPUs.

If you’d like to know how the vulnerabilities work, Eben Upton wrote up a clear explanation for Raspberry Pi – the only common functional computer that isn’t affected.

At present the fixes for Meltdown are effective but can cause significant slowdowns. Fixes for Spectre are incomplete and we have had reports that they can cause instability in Haswell and Broadwell families of Intel CPUs (which we own). Spectre is difficult and slow to exploit because it relies on reading memory one bit at a time.  At 1500bytes/second a full memory dump of one of our virtual server hosts (256GB RAM) would take around six years to complete.

Impact

Both issues allow information leakage so that lower priority processes on a server can read secret data from higher priority processes on the same CPU.  Any computer that accepts instructions from an untrustworthy source is at risk.  We’ve reviewed the impact across all of our services, and have applied or will be applying patches as required.  The impact on live hosting platforms is as follows:

Shared hosting servers

Our web hosting and shell account hosting platforms may have untrustworthy users on them. These servers have already been fully patched against Meltdown and fixes for Spectre will be applied as they become available.

Virtual server hosts

Our Virtual Server Cloud uses KVM with hardware virtualisation which is not vulnerable to Meltdown. Spectre patches are being worked on for the kernel which require new microcode for the CPU. KVM will also need to be updated to fully patch. When these updates are available and have been demonstrated to be stable we will be applying them to our host servers.

This will require a restart of our VM hosts and all guest VMs. Customers will be notified in advance of requiring a restart and each of our datacentres will be restarted at a different time to minimise disruption to customers with split site services.

Virtual server guests

Whilst the use of KVM with hardware virtualisation ensures that Meltdown cannot be used to break the isolation between virtual server guests, virtual servers themselves are potentially vulnerable to both Meltdown and Spectre.   Customers should ensure that their servers are patched and rebooted if they have untrusted users or execute untrusted code.

Dedicated servers

Dedicated servers are at no significant risk unless you allow untrusted third parties to upload and execute code. If that’s the case managed customers can contact support@mythic-beasts.com and we’ll apply the Meltdown and Spectre fixes and reboot as a mutually convenient time.

Raspberry Pi 3 servers

As mentioned above, the Raspberry Pi is not affected and no action is needed.

All other systems

We have reviewed the risk to all other systems and are applying patches as required.  This has included patching, as a high priority, all staff desktops and laptops; websites are allowed to execute javascript which can be used to execute a successful Meltdown attack.

Learn more about XSS with Google

For a long time we’ve sponsored Gwiddle, a project that outgrew its hosting on Microsoft Azure, providing free hosting accounts for students. They’ve now become a fully fledged charity, The Gwiddle Foundation, and we’ve had to upgrade the servers we donated to accommodate their ever expanding user base.

Part of their security team is the very talented Aaron Esau (15), who recently applied his penetration testing skills to our website and picked up a difficult to exploit bug.

On our page that allows you to search for domain names, our code embedded the search terms in the results page without appropriately escaping the content. This is a classic cross site scripting bug. Exploiting this bug was far from trivial, as the search term had to be short and from a restricted character set.

Aaron managed to craft an exploit using an ingeniously short payload to extract a session cookie and has posted a full write-up of the vulnerability and exploit.

If you had recently logged into our control panel, not logged out, and then visited a malicious page with this exploit, then the attacker could steal a cookie which would, in theory, give the attacker access to your control panel pages. However, we practise defence in depth, and our cookies are tied to an IP address so simply stealing the session cookie doesn’t give you access unless you also share a source IP address. This is an example where NAT and IPv4 is less secure than having IPv6.

Once Aaron brought the bug to our attention we swiftly fixed the page, thanked him for notifying us and sent him an Amazon voucher to thank him for his time and responsible disclosure.

We should emphasise that we do not believe that anyone has ever attempted to exploit this bug, and that the IP restrictions on session cookies mean that the consequences were fully mitigated.

Nonetheless, it’s embarrassing for us to have such a stupid bug in our code and we’ve been investigating how it occurred. It seems that the reason it crept in is because the domain ordering pages use a different form framework from everything else. Most of our pages have HTML generated by a template, and wherever dynamic data is included, it’s run through a filter to escape any HTML characters. The domain ordering pages use a different approach with much of the HTML being generated by a form module which we then include verbatim into our output. Obviously the HTML in this data mustn’t be escaped, as it would break the form; the form module is responsible for escaping any user input. Unfortunately, there are some other parts of the page which don’t come from the form module, and so do need to be escaped. It’s not very clear from the template code which is which, leading to the bug of not escaping some fields.

A handful of the hundreds of different organisations, all of whom must be trustworthy.

Everybody knows that SSL is a good idea. It secures communications. At the heart of SSL is a list of certificate authorities. These are organisations that the confirm the identity of the SSL certificate. For example, if GeoTrust says that Raspberry Pi is Raspberry Pi we know that we’re talking to the right site and our communications aren’t being sniffed.

However, the list of certificate authorities is large and growing and as it stands, you’ve got to trust all of them to only issue certificates to the right people. Of course, through incompetence or malice, they can make mistakes.

CAA records are a relatively new mechanism that aims to stop this happening, making it harder to impersonate secure organisations, execute bank robberies and steal peoples’ identities.

CAA records enable you to list in your domain’s DNS the certificate authorities that are allowed to issue certificates for your domain. So, Google has a record stating that only Google and Symantec are allowed to issue certificates for google.com. If someone manages to persuade Comodo they are Google and should be issued a google.com certificate, Comodo will be obliged to reject the request based on the CAA records.

Of course, in order to be of any use, you need to be able to trust the DNS records. Fortunately, these days we have DNSSEC (dns security).

How does it work?

A typical CAA record looks something like this:

example.com. IN CAA 3600 0 issue "letsencrypt.org"

This states that only Let’s Encrypt may issue certificates for example.com or its subdomains, such as www.example.com.

Going through each part in turn:

• example.com – the name of the hostname to which the record apply. In our DNS interface, you can use a hostname of “@” to refer to your domain.
• IN CAA – the record type.
• 3600 – the “time to live” (TTL). The amount of time, in seconds, for which this record may be cached.
• 0 – any CAA flags
• issue– the type of property defined by this record (see below)
• "letsencrypt.org" – the value of the property

At present, there are three defined property types:

• issue – specifies which authorities may issue certificates of any type for this hostname
• issuewild – specifies which authorities may issue wildcard certificates for this hostname
• iodef – provides a URL for authorities to contact in the event of an attempt to issue an unauthorised certificate

CAA records can be added using the new section at the bottom of the DNS management page in our control panel:

The @ in the first field denotes a record that applies to the domain itself.

At Mythic Beasts, we’re a bit skeptical about the value of CAA records. In order to protect against the incompetence of CAs, they rely on CAs competently checking the CAA records before issuing certificates. That said, they do provide a straightforward check that CAs can build into their automated processes to detect and reject unauthorised requests, so publishing CAA records will raise the bar somewhat for anyone looking to fraudulently obtain a certificate for your domain.

---

We refuse to bid for government IT work because we can’t handle the incompetence.

At Mythic Beasts we make use of free secure encryption all the time. Like all powerful tools such as roads, trains, aeroplanes, GPS navigation, computers, kitchen knives, vans and Casio watches; things that are very useful for day to day life are also useful for criminals and terrorists. It’s very popular for our politicians and the Home Office, especially our current prime minister King Canute Theresa May and leader of The Thick Party, to suggest that fully secure encryption should be banned and replaced with a weaker version that will reveal all of your secrets but only to the UK security services.

We disagree and think this is a terrible idea. There’s the basic technical objection that a backdoor is a backdoor and keeping knowledge of the backdoor secret is essentially impossible. There’s a recent practical demonstration of this: the NSA knew of an accidental backdoor in Windows and kept it secret.  It was leaked, resulting in the thankfully not very effective WannaCry virus which disabled substantial fractions of the NHS. The government is very good at scope creep: the Food Standards Agency refused to disclose why it needs the power to demand your entire internet history. We think it fundamentally wrong that MPs excluded themselves from the Investigatory Powers act. Then there’s simple commercial objections: it’s a slight commercial disadvantage if every UK product has an ‘Insecure By Order of The Home Office’ sticker on the front when your foreign competitors products don’t.

However, Mathematics does not care what our politicians wish and refuses to change according to their desires. Strong cryptography is free, available on every computer, and can be given away on the front of a magazine. Taking away secure cryptography is going to involve dragging a playstation out of your teenagers’ hands, quite literally stealing from children. Of course secure communications will still be available to any criminal who can illegally access some dice and a pencil.

At Mythic Beasts we make extensive use of open source free cryptography. OpenSSH protects our administrative access to the servers we run and the customers we manage. OpenSSL protects all our secure web downloads, including last month’s million or so copies of Raspbian ensuring that children with a Raspberry Pi don’t have their computer compromised. We make extensive use of free certificates through Let’s Encrypt and we’ve deployed tens of thousands of upgraded packages to customers which are securely verified by GnuPG.

Without these projects, vast quantities of the internet would be insecure. So we’ve made donations to OpenSSH, GnuPG and Let’s Encrypt to support their ongoing work. We’d like to donate to OpenSSL but we can’t see easily how to pay from a UK bank account.

We drew the architecture diagram for the beer festival on a beer mat.

Today marks the first day of the Cambridge Beer Festival, the longest running CAMRA beer festival, one of the largest beer festivals in the UK and in our obviously correct opinion, by far the best. Not only have we run the web back-end for many Cambridge CAMRA websites for many years, this year we’ve been involved with Cambridge App Solutions who run the iPhone Beer Festival App. They’d been having some trouble with their existing hosting provider for the back-end. In frustration they moved it to their Cloud Raspberry Pi which worked rather better. They then suggested that we keep the production service on the Raspberry Pi, despite it being a beta service.

Preparing for production

We’ve set up all our management services for the hosted Pi in question, including 24/7 monitoring and performance graphing. We then met up with Craig, their director in the pub to discuss the app prior to launch. The Pi 3 is fronted by CloudFlare who provide SSL. However, the connection to the Pi3 from Cloudflare was initially unencrypted. We took Craig through our SSL on a Raspberry Pi hosting guide and about a minute later we had a free Let’s Encrypt certificate to enable full end-to-end data security.

The iPhone app that runs the Cambridge Beer Festival (also found at Belfast and Leestock)

The iPhone Beer Festival App tracks which beers are available and the ratings for how good they are. Availability is officially provided, ratings are crowd sourced.  The app is continuously talking to the back end to keep the in app data up to date. All this data is stored and served from the Raspberry Pi 3 in the cloud.

Proximity

The festival also has some Estimote beacons for proximity sensing which use Bluetooth Low Energy to provide precise location data to the phone. On entry to the beer festival the app wakes up and sends a hello message.

Last year we rolled out one-click HTTPS hosting for our hosting accounts using free Let’s Encrypt certificates.  We’ve been making some further improvements to our control panel so that once you have enabled and tested HTTPS hosting, it’s also easy to redirect all HTTP traffic to your HTTPS site.

We’ve also added an option to enable HTTP Strict Transport Security (HSTS).  This allows you to use HTTPS on your website and commit that you’re not going to stop using it any time soon (we use 14 days by default).  Once a user has visited your site their browser will cache the redirect from HTTP to HTTPS and will automatically redirect any future requests without even visiting the HTTP version of your site.

HSTS makes it harder for an attacker to impersonate your site as even if they can intercept your traffic, they won’t be able to present an non-HTTPS version of your site to any user that has visited your site within the last 14 days.

HTTPS and HSTS control panel settings

We believe that the web should be secure by default, and hope that these latest changes will make it that little bit easier to secure your website.  These features are available on all of our web and email hosting accounts.  We’ll also happily enable this as part of the service for customer of our managed server hosting.

Over the past twelve months we’ve implemented Domain Name Security Extensions, initially by allowing the necessary records to be set with the domain registries, and then in the form of a managed service which sets the records, signs the zone files, and takes care of regular key rotation

Our beta program has been very successful, lots of domains now have DNSSEC and we’ve seen very few issues. We thought that we should do some wider testing with a larger number of users than our own website, so we asked some friends of ours with a busy website if they felt brave enough to give it a go

Eben Upton> I think this would be worth doing.
Ben Nuttall> I'll go ahead and click the green button for each domain.
-- time passes --
Ben Nuttall> Done - for all that use HTTPS.

So now we have this lovely graph that indicates we’ve secured DNS all the way down the chain for every request. Mail servers know for definite they have the correct address to deliver mail to, Web requests know they’re at the correct webserver.

The only remaining task is to remove the beta label in our control panel.

Raspberry Pi DNSSEC visualisation, click for interactive version

Let’s Encrypt on a v6-only host

One of the much requested features for Let’s Encrypt free SSL certificates is support for IPv6-only hosts. Whilst this is promised in the very near future we’re happy to say that IPv6-only hosts behind our NAT64 & Proxy services work out of the box with Let’s Encrypt.

To test it we took the traditional dogfood approach, this website is run on an IPv6-only VM and we’ve just enabled Lets Enrypt SSL support on our own blog. As soon as Let’s Encrypt offer SSL certificates for IPV6-only hosts with no proxy and no NAT64 we’ll give that a try too.

DNS-based domain validation (dns-01)

An alternative approach would be to use dns-01 validation using our DNS API. Our API speaks native v6, so that should work just fine on a truly single-stack IPv6 host.

Here at Mythic Beasts, we’ve been busily undermining sales of our SSL certificates by rolling out support for free certificates from Let’s Encrypt, partly because we think that the internet should be secure by default, but mostly because we’re lazy and Let’s Encrypt makes it easy to fully automate certificate issue and deployment.

Domain validated certificates

The majority of SSL certificates in use today are “Domain Validated” certificates. These are issued automatically by a certificate authority once you have completed some action that proves that you are in control of the domain for which the certificate is being requested. This can include responding to an email send to an address at your domain, or posting a file to a specific location on your website.

Let’s Encrypt DNS challenge

One of the options for validation offered by Let’s Encrypt is a DNS challenge (known as “dns-01”), whereby you prove ownership of your domain by adding a specific entry to its DNS zone. This option is quite interesting, as it allows you to avoid meddling in any way with your web server configuration and, if your DNS is hosted with Mythic Beasts, you can automate the addition of the necessary records using our DNS API.

Automating via our DNS API

In order to support this, we’ve developed a hook script that works with the letsencrypt.sh client.

We’ve also written a step-by-step guide to configuring dns-01 validation using our DNS API.

Please note, if you’re a hosting account customer, you don’t need to worry about any of this. You can get an SSL certificate for your website simply by hitting a button in the control panel.

Thanks go to David Earl for testing this and providing the initial implementation of the hook script..

It’s been a long time coming, but we’re now pleased to announce that we’ve got DNSSEC support in public beta, and you can enable it for your domain at the click of a button.

What is DNSSEC?

DNSSEC is a set of extensions to the DNS protocol that ensures that you can trust the IP addresses that you get back from the DNS system. For example, if you visit www.yourbank.com, the first thing that happens is that your browser uses a DNS server to find out the IP address of your bank’s web server. But how do you know that you can trust the address that you get back? Your request will probably get bounced through multiple DNS servers, such as your home router, your ISPs servers, and finally the authoritative server for the domain. If any one of those gets compromised (and let’s face it, home routers have a terrible security record) it could easily insert a different IP address and direct your request to an entirely different server.

DNSSEC means that all responses are signed with encryption keys that have been lodged with the registry, so you can’t inject bogus responses just by compromising an intermediate server.  Of course, the system only works if the systems making the requests check the signatures of the responses that they receive, something which certainly doesn’t happen everywhere yet.

Sounds complicated?

Yes it is, particularly as it is recommended that the encryption keys that you use are changed (or “rotated”) regularly. Fortunately, we’ve now automated all the hard stuff, and if you’ve got your domain registration and DNS hosting with Mythic Beasts, you can make DNSSEC go just by hitting a big green button.  We’ll take care of the rest:

Unlike some people, we believe that the internet should be a safe place to do business by default, so this service is, and will continue to be, provided at no extra cost.

If you want to try it out, simply visit our control panel, find the domain under “My Domains” and follow the “DNSSEC” link.