How to bring up remote desktop on Mac OS X without having a mouse so that you can operate the machine remotely:

option + spacebar, type terminal

This opens spotlight and starts a terminal.

sudo systemsetup -setremotelogin on

This enables ssh so you can log in remotely.

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -on -restart -agent -privs -all

This enables remote desktop so you can log in from another mac, and use the mouse on that one to finish the rest of your configuration.

The Tower Bridge Lifeboat station put up a picture on Twitter of what London would look like today if the Thames Barrier wasn’t closed.

I’ve annotated it with the four biggest nodes from the London Internet Exchange.

It’s not known exactly what fraction of UK Internet traffic passes through these four buildings, but almost every major ISP or content provider exchanges traffic there, Mythic Beasts, Facebook, Twitter, Google, BT, Virgin, Yahoo, Microsoft, Akamai, Netflix, Talktalk, BSkyB, Vodafone, and hundreds of others. Very slowly regional exchanges are starting to be built, largely by LINX itself.

This morning one of our routers in our Cambridge data centre stopped reporting bandwidth data to our billing system. We investigated and whilst it was still routing packets without issue, it appeared to be experiencing hardware failure.

We’ve powered the router down, pending full investigation on our data centre visit this afternoon. Currently all traffic from our Cambridge site is being handled by our other router. This seamlessly failed over with no customer impact.

Depending on your choice of terminology ‘Redundancy has been reduced to N’, or ‘The network is at-risk’. In Mythic Beasts we like to speak English so this translates to, if something else fails before the router is restored to service, there is a risk of a network outage to our Cambridge data centre.

Update : Friday 31st we fully restored our network to it’s usual redundant configuration by replacing the router with a similarly over specified replacement. Customers may have received free bandwidth for some of this period.

Here’s an unsolicited customer review of a migration of a dedicated server to one of our managed virtual machines from Alex at raspi.tv who’s building a 9inch HDMI 1080p screen.

New Year, New Server At mythic Beasts

You can find the original twitter conversation at @Mythic_Beasts.

Our latest blog post is on the Raspberry Pi website. Coping with Christmas

Today we’ve upgraded both of our connections to the London Internet Exchange (LINX) from 1Gbps to 10Gbps.

Over the past few weeks we’ve repeatedly broken the company bandwidth record. And since we’ve recently secured more peering agreements — including every major UK connectivity provider — a greater proportion of our traffic is now going out over LINX. So at peak times our bandwidth usage has been enough that in the unlikely event of a failure of one of the LINX LANs, we would have come close to running out of capacity on our other link. Clearly an upgrade was in order!

Our network engineers performed the upgrade this morning, with no disruption as traffic was automatically and transparently rerouted during the brief down time. After the upgrade, we have 10Gbps from our data centre in Telecity Sovereign House to LINX Juniper; and 10Gbps from our Harbour Exchange data centre to LINX Extreme.

In the event of the failure of either link or router, traffic will automatically reroute around our internal fibre ring to our other site and out to the peering exchange via our other connection. And, for the time being, we have plenty of capacity to spare.

After cloning a server for a customer we noticed that something was a little bit odd:

# md5sum /etc/sudoers

worked fine but:

# sudo -l

responded with:

sudo: unable to stat /etc/sudoers: Permission denied

How odd we thought. More odd was:

# su - username
Cannot execute /bin/bash: Permission denied

A bit of time with Google and strace revealed that we’d managed to set the permissions on / wrongly:

drwx------  27 root root  4096 Jun  4 11:48 ..

rather than:

drwxr-xr-x  27 root root  4096 Jun  4 11:48 ..

What amazed us was not that the machine didn’t work properly but that we could log in at all.

If this is the sort of problem you’d be able to fix, you should look at our jobs page. If you’d like someone else to fix it for you then our Managed hosting is probably of a lot more interest.

Over the past week or so we’ve given the Science Media Centre a hand in moving their WordPress site into a virtual machine hosted by Mythic Beasts. They’re a charity who work with journalists, scientists and engineers to try and improve the quality of science reporting and removing the misleading rubbish that otherwise gets written. Mythic Beasts is a company founded by science graduates who are very easily angered by terrible science articles in the papers. We’re hoping the saving on destroyed laptops and monitors will easily cover all the management and consultancy services we’ve donated.

If we have fewer idiotic articles proving that Coffee cures cancer^* and Coffee causes cancer^* and rather more articles that our talented university friends pioneer new cancer treatments we’ll consider the time and effort we’ve put in to helping them well spent.

* Actual links removed in the name of good taste. Here’s something more interesting to read, and if you’re still curious, you can look up coffee in the index.

Today is Guy Fawkes Night, when traditionally in the UK we launch fireworks to commemorate not blowing up King James I, the House of Lords and a large chunk of central London. Most modern firework displays use rather less gunpowder than the original plot in 1605.

At Mythic Beasts we think with the advent of four hundred years additional technology we should be aiming a bit higher. That’s why instead of buying fireworks we’ve donated some hosting to Southampton University Spaceflight. Certainly I (Pete) was incredibly inspired as a child by space flight and I remember watching the first untethered spacewalk as a child which motivated me strongly to learn all about rockets and space ships and from there to reverse engineer 8 bit computer games ultimately leading to running a hosting company.

This is of course the second university space program we’re helping out with – some of the tracking for Cambridge University Spaceflight is also done on servers hosted with us.

We’ve had an IPv6 aware network for quite some time, and we’ve been gradually rolling it out to our services with the aim of eventually having every service we offer fully available over IPv6 and IPv4. We host the Raspberry Pi website which has an IPv6 only internal network, IPv6 only virtual machines and IPv4 on the front end to help out those of you with the ‘legacy’ internet.

A quick skim over the logfiles suggests that about 96% of you still access the site through the legacy IPv4 network – about 4% of hosts are now connecting over IPv6 which is starting to become a non trivial fraction of the traffic. Of course this is much higher than typical sites, Raspberry Pi users are much more technically aware than the general population.

Yesterday we had our first real connectivity problem to investigate – an end user within Ja.net (the UK academic network) was unable to access files from the Raspberry Pi download server on about half of the occasions. Further investigation showed that they could access the load balancers in our Sovereign House site with connectivity via the London Internet Exchange Juniper LAN, but not the load balancers in our Harbour Exchange site with connectivity over the London Internet Exchange Extreme LAN.

When we started investigating we confirmed that it seemed to be a problem with the Extreme LAN, if we forced the connectivity via the Juniper LAN it worked from both sites, if we forced it via the Extreme LAN it failed from both sites. Odder and odder though, a packet dump on our LINX interface didn’t show us passing the packets on.

Our IPv4 peering worked fine, this was IPv6 specific.

We then started looking at the routing table on the router. Over IPv4 it looks like

131.111.0.0/16 via 195.66.236.15 dev eth0

and over IPv6 it looks like

2001:630::/32 via fe80::5e5e:abff:fe23:2fc2 dev eth0

That gives us the netblock, and the next hop to send the packet to.

So the next step is to check you can reach the gateway happily enough.

# ping 195.66.236.15
PING 195.66.236.15 (195.66.236.15) 56(84) bytes of data.
64 bytes from 195.66.236.15: icmp_seq=1 ttl=64 time=0.220 ms


and

# ping6 fe80::5e5e:abff:fe23:2fc2
connect: Invalid argument


Odd. Then I realised that fe80:: in IPv6 means a link local address – the address is specific to the network card so to ping it you have to specify the destination address and the network interface.

# ping6 fe80::5e5e:abff:fe23:2fc2 -I eth9
PING fe80::5e5e:abff:fe23:2fc2(fe80::5e5e:abff:fe23:2fc2) from fe80::21b:21ff:fe65:a4c5 eth9: 56 data bytes
64 bytes from fe80::5e5e:abff:fe23:2fc2: icmp_seq=1 ttl=64 time=0.451 ms


Then the penny dropped. The routing table has eth0 in it but we’re actually connected to eth9. Under IPv4 this is fine because the next-hop address is globally unique and only accessible over eth9 so we send the packets out of eth9 and they go to the correct destination. Under IPv6 it’s a link local address and therefore valid over any interface, so we obey the routing table and throw the packets out of eth0 whereupon they fall onto the floor because there’s no fibre connected.

Fixing the config to put the right interface description in made it all work, and our end user is happily able to access all the load balancers on all the v6 addresses in all of the buildings.

Obviously if you’re a Mythic Beasts customer and you don’t already have an IPv6 allocation for your real or virtual server, drop us an email and we’ll hand you your own address space to play with.