Enabling Anycast DNS with Esgob

Nat Morris, UK Network Operators Forum director recently gave a presentation to DNS Operations, Analysis and Research centre, which included this remarkably nice slide:

What is Anycast?

Normally a server has a globally unique IP address, and the Internet knows how to send traffic from any other machine in the world to that IP address. With Anycast we share a single address across multiple machines, and your traffic is sent to the nearest machine with that address. This means that UK customers can be answered from a server in the UK and Australian customers from a server in Australia allowing you to have very fast responses to things like DNS queries because you’re always served by a server that’s close by, rather than your query having to travel half way around the world.

To set up an Anycast network, you need your own address space, your own network number (ASN), multiple BGP-aware routers that can announce your address space, and multiple servers that can answer the queries. Typically this would require a pretty hefty budget, but if you’re Nat Morris and you know what you’re doing with software routing on Linux, and you know all the right providers then you can bring up a global Anycast network with 10+ servers and sites on an annual budget of well under $1,000.

The key to doing this is finding ISPs, ideally well-connnected ISPs in key internet hubs, who will provide you with a BGP feed to your hosted server. That’s where a UK clueful hosting company comes into the picture having excellent connectivity, inexpensive virtual machines (VMs) and a willingness to support customers with more unusual configurations.

Quick introduction to BGP and routing

Normally when you have a VM you get a default route, which looks like this:

# ip route 
...
default via 93.93.128.1 dev eth1 

which says that to get to anywhere on the internet, send packets to our router at 93.93.128.1.

Over BGP, instead we send you the whole routing table:

# ip route 
...
1.0.7.0/24 via 5.57.80.128 dev eth3.4  proto zebra  metric 1 
1.0.20.0/23 via 93.93.133.46 dev eth6.220  proto zebra  metric 142080 
... 
500,000 more lines like this

For every block on the whole internet you have a different gateway depending on what you’ve decided is the preferred route. At today’s count this is about 490,000 entries in the routing table. Don’t type ‘route’ if you’re logged in over 3G!

So for this VM, instead of having a default route, Nat has four full BGP sessions, two to each of our two routers to the site. On each router, one session provides 490,000 IPv4 routes, the other provides 18,000 IPv6 routes, and the VM gets to decide which router to send data to.

The other side of the BGP relationship, and the important bit for Anycasting, is that we receive an advert from Nat’s VM for his /24 of IPv4 space and /48 of IPv6 space, which we then advertise out to the world. The 10+ other providers in this Anycast setup will do the same, and hosts will direct traffic to whichever is nearest.

Filtering

As Paul Vixie pointed out in the first question to Nat, the main customers of VMs with BGP are spammers who hijack address space for nefarious usage. At Mythic Beasts we filter our announcements and our customer routes, so if Nat messes up his configuration and accidentally announces that his VM is responsible for the whole of Youtube we’ll drop the announcement rather than expecting one very small VM to handle one fifth of the internet.

BGP on a virtual or dedicated server

If you’re a DNS provider or a content delivery network, you’ll probably want to have an Anycast setup at some point. At Mythic Beasts we remember what it was like to be the little guy which is why we offer full BGP routing (including IPv6 BGP) as an option to any virtual server, dedicated server, colocated server or router. Providing you own your own ASN and IP space we can transit it for you and we can keep the start-up costs very low and scale with you. You can locate your VM or server directly with us in Telecity, mere tens of metres from LINX and LoNAP for minimal latency and maximal available bandwidth.

If you’ve no idea what an ASN, BGP, LIR, RIPE are, we can help arrange your ASN, IP space and BGP config.