A Cambridge start-up approached us with an interesting problem. In this age of virtualisation, they have a new and important service, but one which can’t be virtualised as it relies on trusted hardware. They know other companies will want to use their service from within their private networks within the big cloud providers, but they can’t co-locate their hardware within Amazon or Azure.

This picture is a slight over simplification of the process

The interesting thing here is that the solution is simple. It is possible to link directly into Amazon via Direct Connect and to Azure via Express Route. To use Direct Connect or Express Route within the UK you need to have a telco circuit terminating in a Telecity data centre, or to physically colocate your servers. As many of you will know, Mythic Beasts are physically present in three such data centres, the most important of which is Telecity Sovereign House, the main UK point of presence for both Amazon and Microsoft.

So our discussion here is nice and straightforward. Our future customer can co-locate their prototype service with Mythic Beasts in our Telecity site in Docklands. They can then connect to Express Route and Direct Connect over dedicated fibre within the datacentre when they’re ready to take on customers. Their customers then have to set up a VPC Peering connection and the service is ready to use. This is dedicated specialised hardware from the inside of ‘the cloud’, and it’s something we can offer to all manner of companies, start-up or not, from any dedicated or colocated service. You only need ask.

Expect to see these around for some time to come

Currently most network connectivity is 1Gbps ethernet over Cat5e copper which stretches up to 100m. There is an infrequently used standard for 10Gbps over Cat6 copper to 55m for higher speeds.

Now demand is starting to appear for faster than 1Gbps speeds, and it’s very attractive to do this without replacing the installed base of Cat5e and Cat6 cabling. There are new standards in the pipeline for 2.5Gbps and 5Gbps ethernet over Cat5e/Cat6 cabling.

In the data centre it’s common to have 10Gbps over SFP+ direct attach for short interconnects (up to 10m) and 1Gbps/10Gbps/40Gbps/100Gbps over fibre for longer distances. 1Gbps and 10Gbps are widely adopted. 40Gbps and 100Gbps are a different design, implemented by combining multiple lanes of traffic at 10Gbps to act as a single link. 100Gbps has changed to be 4 lanes at 25Gbps rather than 10 at 10Gbps.

The more lanes you have in use, the more switches and switching chips you need – but effectively this means that 40Gbps has the same cost in port count as 100Gbps. The next generation of 100Gbps switching hardware will consist of a large number of lanes that run at either 10Gbps or 25Gbps. With current interfaces, you’d use 4 lanes for 100Gbps, 4 lanes for 40Gbps or 1 lane for 10Gbps. The obvious gap is using a single lane for 25Gbps standard so you can connect vastly more devices at greater than 10Gbps speeds.

So in the near future, we’re expecting to see 2.5Gbps and 25Gbps ethernet becoming available, and in the longer term work has now started on 400Gbps standards.

For a long time at Mythic Beasts we’ve had a fairly healthy dislike for managed switches. The configuration method of switches is akin to a database with auto-commit on every command – you can’t batch a series of configuration changes into an atomic update. This means that you not only need to think about your starting and end configurations, but you also need to think about all the intermediate configuration too and make sure you don’t accidentally explode everything with an unexpected switch loop. Switches are also expensive and it’s always rankled that we’re paying a lot of money in order to use a network operating system that’s user unfriendly. Some of them are often less stable than the servers they connect to and they seem to manage excellent vendor lock-in – there is no end of advice that you can’t plug standards compatible switches from different manufacturers into each other because you risk inter-operability issues.

We’ve recently started trying Linux switches — commodity switches running Cumulus Linux.

Cumulus Linux makes your switch appear like a standard-ish debian server, with a lot of NICs.
The interfaces on our “1G” model are:

The switch is configured via /etc/network/interfaces, and uses bridges, VLANs and bonds to set up the configuration.

Linux has lots of advantages as a switch operating system. For a start if you need to patch ssh, under linux you download a replacement digitally signed openssh package and restart the process, on a traditional switch you download a whole new firmware over insecure tftp and reboot the switch – unlucky for the people connected to the switch.

The first obvious difference when configuring these switches is that by default, the switch doesn’t switch any traffic until some configuration is put in.

We can set up a simple network:

 # The primary network interface
 auto eth0
 iface eth0 inet static
        address xxx
        gateway xxx

 auto br0
 iface br0
         bridge-ports glob swp1-48
         bridge-stp on
         setmcsnoop 0

 auto br1
 iface br1
         bridge-ports glob swp49-52
         bridge-stp on
         setmcsnoop 0

This sets up the 1G ports (1-48) as a single VLAN, the 10G ports (49-52) as second VLAN, a management interface on the management port (eth0).

In this case we have an uplink on port 48 to a different network. So to migrate the uplink from our 1G network to our 10G network we would write out a new configuration file:

 auto br0
 iface br0
         bridge-ports glob swp1-47
         bridge-stp on
         setmcsnoop 0

 auto br1
 iface br1
         bridge-ports glob swp48-52
         bridge-stp on
         setmcsnoop 0

then bring the interfaces up with

 ifup -a

Note that ifup under Cumulus is different to standard Debian. It links to ifupdown2 which can inspect the current running state and apply only changes, rather than having to take an interface down and up on a standard server.

One deeply troubling thing about Cumulus Linux is it includes a minimal vi, but not a full implementation of vim.

But there are many other advantages that make up for this inexplicable oversight: being Debian-ish it has sudo, so you can give arbitrary permissions to multiple users rather than just show / enable / configure. You can easily update things with ssh. You can configure your switch with puppet. You can easily back up the entire configuration with rsync, version control it with etckeeper and bzr (sadly no git!). You can write code and run it directly on the switch which allows all kinds of options for monitoring and configuration.

We now have a few Cumulus Linux switches in production for private client networks. Here’s one providing lots and lots of bandwidth:

When we started in 2000, 80Gbps was about the whole internet. Now a private link for one customer. pic.twitter.com/hB8CzmJfgr

— Mythic Beasts (@Mythic_Beasts) September 2, 2015

Even complex configurations can be handled relatively easily. For example, we have a customer with a private cloud who wants to run 20Gbps into each host, exposing different 10 different VLANs to their virtual servers, and then routing between them. This can be done on a 10G switch by bonding pairs of interfaces together, and then bridging the required VLANs on each of the bonded interfaces.

This config turns out to be nice and simple to write, and has the advantage of looking very similar on the switch and the server:

auto bond13            
iface bond13           
  bond-slaves swp1 swp2         
  bond-mode 802.3ad             
  bond-miimon 100               
  bond-use-carrier 1            
  bond-lacp-rate 1              
  bond-min-links 1              
  bond-xmit-hash-policy layer3+4
                       
auto bond14            
iface bond14           
  bond-slaves swp3 swp4         
  ....

auto br-tag130
iface br-tag130
  bridge-ports bond13.130 bond14.130 ...

auto br-tag2544
iface br-tag2544
  bridge-ports bond0.2544 bond1.2544  ...

A customer of ours has been extending their private cloud. This adds another 160 cores, 160Gbps, 2TB of RAM and over half a petabyte of storage. On the left you can see the black mains cable, then the serial for out of bound configuration, then red cabling for 1Gbps each to our main network, then 20Gbps per server to the very secure private LAN on SFP+ direct attach.

The out of place yellow cable is network for the serial server above, and the out of place black one is serial to the 720Gbps switch which isn’t quite long enough to route neatly.

There’s a few more bits and pieces to add, but soon it will join their OpenStack cloud and substantially increase the rate at which their data gets crunched.

Happy Incorporation Day to Us
Happy Incorporation Day to Us
Happy Incorporation Day to Mythic Beasts
Happy Incorporation Day to Us

Fifteen years ago today someone with a boring job processed the paperwork and Mythic Beasts Ltd sprang into existence as a legal entity.

We had existed informally for a bit longer than that, we had registered the mythic-beasts.com that April, and our shell server, sphinx, had been running for a while, although it wouldn’t be until early 2001 that we sent our first invoice.

As we all work on t’internet, it’s difficult to all meet in the same pub this Friday for a celebratory drink. That will have to wait until our next full company meetup in September. Slamming the bedroom door, staying in and watching Brazil seems a more apt way for a teenage company to celebrate its bureaucratic anniversary.

We had a plan to post this last year, but we forgot.

On August 13th 2004, ten eleven years ago, our first ever invoice for a virtual dedicated server was paid.

Mythic Beasts Ltd.                http://www.mythic-beasts.com/
103 Beche Road
Cambridge
CB5 8HX

To:



────────────────────────────────────────────────────────────────
Invoice date                                        02-Jul‐2004
Invoice number                                                         
────────────────────────────────────────────────────────────────

────────────────────────────────────────────────────────────────
Ref         Date            Description                 Amount
────────────────────────────────────────────────────────────────
 xxxx     2004‐07‐02  VDS256 virtual dedicated         £400.00
                      server server-name 2004‐07‐02 to
                      2005‐07‐01
────────────────────────────────────────────────────────────────
                                      total            £400.00
────────────────────────────────────────────────────────────────

This VM ran until 2009 at which point it upgraded to a much faster dedicated server. It’s still operational today.

Over the intervening years, our basic service has evolved through three different virtualisation technologies and the virtual machines are now thirty two times the size that they started.

• 256MB, User Mode Linux
• 1024MB, User Mode Linux
• 1024MB, Xen
• 4GB, KVM
• 8GB, KVM with SSD

As computers have become much larger and faster it’s increasingly hard to find a single application that can fill the capacity of a single machine, meanwhile service oriented architecture means that even simple applications are now built out of lots of lighty loaded servers, virtualisation is the magic that means that fifty applications built from tens of servers each can fit into a handful of physical servers. Whilst managing the hardware has become much simpler the number of instances to manage has exploded.

Increasingly these days not only are the servers virtual but the entire infrastructure, routers and all. We now have entirely virtual networks existing within our VM cloud using virtual routers to route traffic to virtual machines.

If you read security lists, you will already be aware that we’re expecting a new release of OpenSSL tomorrow to fix a high severity vulnerability.

We will be reviewing the details as soon as the vulnerability is released, and will be patching the affected servers shortly after the updated packages are released, if necessary we will be contacting customer to reissue keys as we did after the now infamous Heartbleed vulnerability.

If you have any questions, or would like to upgrade to a manged service so we catch these kinds of issues for you, you can contact us at support@mythic-beasts.com.

One of the Mythic Beasts, Rhosyn has written an article on filtering for trefor.net, a widely read blog on technology and networking.

The part we particularly like this this quote,

As a long standing customer of Mythic Beasts 
(shameless plug; outstanding service and support, 
so good that I switched companies recently to work 
for them)

tldr; Apple say IPv6 support vital, we offer £20pa off any VM that is IPv6 only.

Apple have announced that as of iOS 9, all apps require support for IPv6 and must run on an IPv6 only network. The motivation is fairly clear, T-Mobile USA runs an IPv6 only network, Comcast is deploying IPv6 and on IPv6 launch day in Finland 5% of users had IPv6 enabled simultaneously. Google sees around 7% of all traffic as IPv6 now.

Like it or not, IPv6 is here and the predictions of a lengthy period of being dual stack were wrong. Nobody bothered to turn on IPv6 until IPv4 ran out, then instead of IPv6 and Network Address Translation we’re skipping quickly to IPv6 only. If your application doesn’t work on an IPv6 only network an increasing fraction of users simply can’t use it.

At Mythic Beasts we’ve been using IPv6 for a long time. Two years ago we rebuilt the hosting infrastructure for Raspberry PI to be IPv6 only for all internal connections. A future article will explain our scale up to vastly more VMs, many IPv6 only. IPv6 at Mythic Beasts isn’t an add-on, if our IPv6 connectivity breaks, customers go offline. We’re steadily working on spreading IPv6 connectivity throughout other providers.

We’ve been offering developers IPv6 only Virtual Machines for experimentation for a while, and have one of the most comprehensive IPv6 connectivity checkers for hosted software which is very good at demonstrating that enabling a v6 address isn’t quite enough.

Every single connection to this website uses IPv6.

The best way to build the hosting infrastructure today, is to have an IPv6 only network for the whole thing and a single IPv4 address on the load balancer for ‘legacy’ IPv4 connections. To give everyone an incentive to do it right, today we’re extending our IPv6 only VM offer – all virtual machines that are IPv6 only will be discounted for the lifetime of the rental.

If you’re really interested, this presentation at the North American Operators Group about the largest US ISPs moving straight to IPv6 only deployments including the information that over 20% of US users have native IPv6.

SSL Certificates do two things. They encrypt the traffic between the end user and the website, and they provide authentication that confirms the website is who they say they are. As we previously wrote about at present the authentication step is done using a piece of maths called SHA-1.

What the SHA-1 function does, is to provide a signature that says ‘The Certificate Authority confirms that the public key for Mythic Beasts is ….’. It’s extremely important that nobody else can forge this signature, otherwise anybody could present their public key instead of the Mythic Beasts public key and intercept all of the data.

Now SHA-1 has been subject to a lot of analysis by people attempting to forge keys, and slowly progress has been made. SHA-1 has not been “broken”, but thanks to improvements in mathematics and computing, the estimated cost of forging a certificate has steadily fallen from more-money-than-exists to a-large-country-could-do-it and in the next 5 years is likely to reach script-kiddy-with-a-botnet-could-do-it.

So Google, Firefox and others now refuse to accept SHA-1 based certificates that will last into 2017. Whilst you can’t forge them now, in two years time it’s likely that well funded organizations may be able to do so. As a result, the Internet has had to migrate to SHA-2, a new function that achieves the same as SHA-1 for proving authenticity but has no known attacks: forging a SHA-2 signature is currently believed to be entirely infeasible. Google’s announcement of their intention to deprecate SHA-1 was greeted with dismay and anger, but in the end had the desired effect. The certification authorities moved quite quickly to make SHA-2 the default.

At Mythic Beasts this week, we replaced our SSL certificate for all our servers. As expected, the new certificate we were issued was SHA-2 based. Deployment of the new certificate went smoothly, sufficiently smoothly that not a single customer noticed. A short time later we realised that we now didn’t seem to receive any support mail at all.

Our ticket tracking system runs on top of mono, an open source reimplementation of .NET. The older version of mono it uses doesn’t have support for SHA-2 certificates, so our tracker was seeing the secure connection, failing to authenticate and refusing to send or receive email. Briefly we worked around this by turning encryption off for the support system – as the traffic is entirely within our network we aren’t so worried about it being intercepted.

However, we know that our end-users use a wide variety of different clients for email, some of which are quite old and obscure. So we thought it was rather likely that we were breaking email functionality for existing customers with the SHA-2 certificate. We decided the sensible thing to do would be to use the new SHA-2 certificate just for websites, and obtain a new SHA-1 certificate for mail applications.

We will face the same issue again in 12 months. (Except we don’t even know if the certification authority will still offer the choice of getting a SHA-1 certificate then.) We’re hoping that a year will force a number of updates to mail clients and system libraries such that next year we can deploy SHA-2 everywhere. Eventually, we will have to draw a line, and say that if our customers’ clients don’t support SHA-2, they will have to upgrade them, or use unencrypted access.

In a little known fact, here are two old men singing about SSL security beginning with a limited understanding of SHA hashes. It delightfully uses the metaphor of a journey to meet their loved one to show how the process of security is a continuous process that can never be fully achieved.