What is Anycast and How it works
If you try to access slashroot.in from US the request will be routed to the Linode London data center. If you try to access slashroot.in from Asia, again the request will be routed to the same Linode London Data Center. Take one more case, and try to access slashroot.in from Europe (Ireland) the request will be again routed to Linode London.
The bottom line is, You will be routed to Linode London, no matter from where you access.
There are two primary problems associated with this architecture. They are mentioned below.
- What if something happens to Linode London Data Center, and it goes down due to some problem (this wont happen, i love linode, and they are the best VPS solution providers out there, I will write a detailed post about my review regarding Linode. Just take this as an example.)
- Second problem is if a user from Asia (say India for example), access my site, that user has to unnecessarily suffer a latency of few hundred milliseconds. The problem is with everyone. Say a person accessing my site from US, he will still suffer a little latency as his packets needs to travel all the way to London.
This kind of an architecture where there is only one destination server for all sources (no matter from where the request comes from) is called as unicasting. This is the highly used networking solution on the internet today. There is one destination server, with an ip address assigned to it, and it will answer all the requests.
Traceroute is an awesome tool to check the path your packet is traveling. If you are new to traceroute, i will recommend reading the below post to understand how Traceroute works.
Read: How Traceroute Works?
I will show you the traceroute to my site (slashroot.in) from two different locations. It will all reach linode London DC.
1 1 ms <1 ms <1 ms 192.168.0.1 2 1 ms 1 ms 1 ms ras.beamtele.net [126.96.36.199] 3 50 ms 1 ms 1 ms ras.beamtele.net [188.8.131.52] 4 10 ms 1 ms 1 ms 184.108.40.206.STATIC-Hyderabad.vsnl.net.in [121. 240.252.1] 5 41 ms 14 ms 14 ms 172.25.81.134 6 * 15 ms 14 ms ix-0-100.tcore1.MLV-Mumbai.as6453.net [180.87.38 .5] 7 120 ms * 120 ms if-9-5.tcore1.WYN-Marseille.as6453.net [80.231.2 17.17] 8 121 ms 120 ms 121 ms if-8-1600.tcore1.PYE-Paris.as6453.net [80.231.21 7.6] 9 121 ms 123 ms 121 ms 220.127.116.11 10 120 ms 120 ms 123 ms prs-bb1-link.telia.net [18.104.22.168] 11 134 ms 254 ms 132 ms ldn-bb1-link.telia.net [22.214.171.124] 12 134 ms 136 ms 136 ms ldn-b3-link.telia.net [126.96.36.199] 13 131 ms 131 ms 135 ms telecity-ic-150799-ldn-b3.c.telia.net [80.239.16 7.94] 14 132 ms 132 ms 132 ms 188.8.131.52 15 131 ms 130 ms 131 ms li646-103.members.linode.com [184.108.40.206]
See the 11th, 12th and 13th lines from the above output. It adds a location information. Ex: ldn-b3-link.telia.net. ldn stands for London. Most of the routers in between adds a location information to its host name for easier identification.
Now i will do a traceroute to my server from US. The traceroute output is shown below.
traceroute to slashroot.in (220.127.116.11), 30 hops max, 60 byte packets 1 ec2-79-125-0-134.eu-west-1.compute.amazonaws.com (18.104.22.168) 0.810 ms 0.491 ms 0.710 ms 2 22.214.171.124 (126.96.36.199) 0.993 ms 0.967 ms 0.934 ms 3 188.8.131.52 (184.108.40.206) 1.087 ms 220.127.116.11 (18.104.22.168) 0.946 ms 22.214.171.124 (126.96.36.199) 1.204 ms 4 dln-b2-link.telia.net (188.8.131.52) 1.575 ms dln-b2-link.telia.net (184.108.40.206) 1.207 ms dln-b2-link.telia.net (220.127.116.11) 1.330 ms 5 ldn-bb2-link.telia.net (18.104.22.168) 18.700 ms ldn-bb2-link.telia.net (22.214.171.124) 20.626 ms 20.580 ms 6 ldn-b3-link.telia.net (126.96.36.199) 15.042 ms ldn-b3-link.telia.net (188.8.131.52) 17.349 ms ldn-b3-link.telia.net (184.108.40.206) 14.906 ms 7 telecity-ic-150799-ldn-b3.c.telia.net (220.127.116.11) 12.357 ms 12.402 ms 12.349 ms 8 18.104.22.168 (22.214.171.124) 16.203 ms 12.862 ms 12.822 ms 9 li646-103.members.linode.com (126.96.36.199) 16.122 ms 12.516 ms 12.565 ms
Make a note of 6th and 7th line in the above traceroute from US. Its still reaching at London.
So its the same London DC, no matter from any country in the globe.
This kind of an architecture works well for small to medium range of websites. But once your website becomes too large and starts getting heavy traffic, it will have a sever performance impact, and need to look for other optimum solutions.
Imagine server's like the DNS root servers. There are 13 of them, which needs to be available all the time. Root servers are one of the most critical infrastructures on the internet. Due to this reason, anybody accessing it needs to get the result in the possible minimum latency. Hence DNS root servers uses Anycasting for this purpose. Apart from achieving a better latency, anycasting also provides redundancy. we will be discussing all of that in some time.
Read: DNS Root Servers
Here is where anycasting comes into the picture. Although in the beginning anycasting was only used for DNS. These days Anycasting is used for normal HTTP web sites. We will look into that part later.
What is Anycast?
You might have already heard the below terms in networking. These are very commonly used.
- Unicast (one sender and one receiver. This is the most commonly used type. While you are reading this web page, you are using unicast type.)
- Broadcast & Multicast (This includes one sender and multiple receivers. A good example of broadcast is the ARP request send by a computer on the network to all others. In Multicasting one or more senders can send data to multiple receivers. Multicasting in IPv4 is implemented by a special class of IP addresses reserved for this purpose. All those hosts that are part of that specific class of multicast address will receive the message send to that multicast address.)
Anycasting is completely different from the above two types. A good definition of anycasting is given below.
Anycasting is a method used to advertise one IP address from multiple points in the network topology, and with the help of dynamic routing method, the traffic is delivered to the nearest point.
Although anycast works by having multiple receivers, only one receiver is selected from all the available ones. Hence in a way its a point to point communication with the nearest address. Also the sender does not care which receiver from the possible list is selected (as all of the receivers will be providing the same service and will be mirrors of each other.).
A single anycast address is assigned to multiple hosts providing the service. And the routers in between does the job of selecting the best and nearest destination. A sender will send a request with the anycast address in it packet header and the routers then run the entire show of delivering it to the nearest location.
But the main point to note here is the criteria used to select the receiver from the available list of multiple anycast destinations. There are different schemes/criteria that can be used for this selection. Before getting into that part, let's have a look at a diagram that represents the anycast basic network topology.
In the above shown example topology diagram, two servers are shown. Both these servers are part of anycast group and are assigned with the same ip address of 10.1.1.10.
When client 1 needs to access the server its routed to the nearest server by Router 1. And when client 2 wants to access the server its routed to the nearest location (through router 3 and router 5).
It is now clear from the above shown basic diagram, that there must be some criteria to select the destination server. There are different criteria/scheme that can be used in anycast for determining the best destination server. Let's see two main schemes that can be used in anycast.
- IP Anycast or say Network Layer Anycast: If the destination server is selected by the routing method, in other words users directed towards a destination server that needs few number or router hops in between, then it is called as network layer anycast.
- Application Layer Anycast: If the destination server is selected by calculating the availability of the server, current number of connections, response times etc, then its called as application layer anycasting. But in this method, does not depend on the network but depends on an external source which continuously monitor the statistics (like current number of connections, response time etc.)
Network layer anycast is the highly used type of anycast in the internet. In the beginning anycasting was only used for DNS. Why only DNS ? The reason is described below.
In the previously shown diagram, which describes the basic working of anycast, you can see that the same service is provided by multiple instances in different locations reachable by different paths. The idea behind anycasting is to reduce latency and increase redundancy. DNS works on UDP (which is a connection less protocol.)
TCP is a connection oriented protocol, which means services using TCP requires a successful connection to be established first. Even protocols like HTTP works over TCP (yeah HTTP itself is not stateful, which means each request is independent of the other.). But a connection should be made to the destination server. In case of anycasting if something happens to one of the anycasted instance, the routers will stop sending traffic to them, and will select the next nearest anycast node.
But in that case, if suppose a user was already browsing a web page (consider an HTTP anycast in this case), the already established TCP connection to the node will break (as that node is not available now), and routers will forward the traffic to the next nearest node (but as this is a different node, with the same ip address, the tcp connection needs to be established again.)
Due to this problem, anycasting is mostly used for single request single response protocols. DNS is a single request response protocol. It does not matter which server replies you, because each request is independent, and no tcp connection is required.
If you are new to DNS, please refer the below articles to understand some basics of DNS and its related topics.
Read: How DNS works
An important thing to note, before going ahead....
Anycast was not designed for load balancing. The main purpose of anycast is to achieve reduced latency and redundancy. However depending upon on the way it is configured it gets a slight load balancing effect.
There is no mechanism by which you can identify by looking at the ip address, whether its an anycast ip address or unicast. For example, 188.8.131.52 is an anycasted ip address and 184.108.40.206 is a unicast ip address. Both are ipV4 addresses. A special range of ip address is reserved for anycast in IPv6. But in IPv4 the normal unicast address space is used for anycast purpose. Hence by looking at the ip address you cannot identify whether its anycast or unicast. The only mechanism to identify it is to do traceroute from different locations.
The ip address 220.127.116.11 is an anycasted IP address. How am i too sure about this?. Am sure about this because of two reasons.
1. Its the IPv4 address of F DNS root server.
2. Tracerouting from different locations around the world will make you reach the same IP in different locations. (mostly the nearest from the source.)
Traceroute to DNS F root server from India is shown below.
Tracing route to f.root-servers.net [18.104.22.168] over a maximum of 30 hops: 1 3 ms <1 ms <1 ms 192.168.0.1 2 1 ms 1 ms 1 ms ras.beamtele.net [22.214.171.124] 3 2 ms 1 ms 1 ms ras.beamtele.net [126.96.36.199] 4 1 ms 1 ms 3 ms 188.8.131.52.STATIC-Hyderabad.vsnl.net.in [121 .240.252.37] 5 * * * Request timed out. 6 53 ms 14 ms 13 ms ix-4-2.tcore1.CXR-Chennai.as6453.net [180.87.36. 9] 7 79 ms 93 ms 81 ms if-5-2.tcore1.SVW-Singapore.as6453.net [180.87.1 2.53] 8 81 ms 80 ms 78 ms if-2-2.tcore2.SVW-Singapore.as6453.net [180.87.1 2.2] 9 78 ms 78 ms 80 ms if-1-2.tcore1.HK2-HongKong.as6453.net [180.87.11 2.1] 10 86 ms 86 ms 87 ms isc2-FE.hkix.net [184.108.40.206] 11 88 ms 119 ms 91 ms f.root-servers.net [220.127.116.11]
See the 9th and 10th output above. It clearly shows its going to HongKong.
Let's now do a traceroute to the same f root server from US.
traceroute to 18.104.22.168 (22.214.171.124), 30 hops max, 60 byte packets 1 ip-10-161-95-1.ec2.internal (10.161.95.1) 0.772 ms 1.111 ms 1.386 ms 2 100.68.105.11 (100.68.105.11) 1.381 ms 100.68.105.10 (100.68.105.10) 0.745 ms 100.68.105.8 (100.68.105.8) 1.356 ms 3 100.68.105.25 (100.68.105.25) 0.471 ms 100.68.105.42 (100.68.105.42) 0.519 ms 100.68.105.24 (100.68.105.24) 0.436 ms 4 100.65.114.6 (100.65.114.6) 0.433 ms 100.65.162.5 (100.65.162.5) 0.611 ms 100.65.130.7 (100.65.130.7) 0.433 ms 5 100.65.43.182 (100.65.43.182) 1.212 ms 100.65.35.182 (100.65.35.182) 0.937 ms 100.65.117.50 (100.65.117.50) 0.944 ms 6 100.64.196.169 (100.64.196.169) 0.892 ms 100.64.199.25 (100.64.199.25) 0.855 ms 100.64.196.3 (100.64.196.3) 0.872 ms 7 100.64.192.232 (100.64.192.232) 1.215 ms 100.64.192.204 (100.64.192.204) 1.243 ms 100.64.192.76 (100.64.192.76) 0.704 ms 8 100.64.59.60 (100.64.59.60) 0.438 ms 100.64.59.74 (100.64.59.74) 0.450 ms 100.64.59.78 (100.64.59.78) 0.445 ms 9 100.64.22.15 (100.64.22.15) 1.483 ms 100.64.21.113 (100.64.21.113) 4.283 ms 100.64.21.239 (100.64.21.239) 0.993 ms 10 126.96.36.199 (188.8.131.52) 1.228 ms 184.108.40.206 (220.127.116.11) 1.435 ms 18.104.22.168 (22.214.171.124) 1.382 ms 11 100.64.2.241 (100.64.2.241) 0.932 ms 100.64.2.179 (100.64.2.179) 1.653 ms 100.64.2.21 (100.64.2.21) 1.202 ms 12 100.64.0.52 (100.64.0.52) 1.846 ms 100.64.0.248 (100.64.0.248) 1.220 ms 100.64.0.24 (100.64.0.24) 1.589 ms 13 100.64.16.71 (100.64.16.71) 0.600 ms 100.64.16.33 (100.64.16.33) 0.671 ms 100.64.16.7 (100.64.16.7) 0.713 ms 14 126.96.36.199 (188.8.131.52) 1.602 ms 184.108.40.206 (220.127.116.11) 0.662 ms 18.104.22.168 (22.214.171.124) 0.564 ms 15 126.96.36.199 (188.8.131.52) 3.669 ms 1.368 ms 3.649 ms 16 v416.core1.ash1.he.net (184.108.40.206) 1.575 ms v315.core1.ash1.he.net (220.127.116.11) 1.521 ms 18.104.22.168 (22.214.171.124) 1.858 ms 17 10ge1-2.core1.atl1.he.net (126.96.36.199) 13.821 ms v416.core1.ash1.he.net (188.8.131.52) 13.078 ms 10ge1-2.core1.atl1.he.net (184.108.40.206) 14.128 ms 18 isc.gige-g2-1.core1.atl1.he.net (220.127.116.11) 80.024 ms 80.015 ms 10ge1-2.core1.atl1.he.net (18.104.22.168) 14.023 ms 19 f.root-servers.net (22.214.171.124) 79.806 ms isc.gige-g2-1.core1.atl1.he.net (126.96.36.199) 80.069 ms 79.805 ms
My traceroute is going to Atlanta US (see the 17th and 18th lines above). Isn't that amazing?. The same ip at different locations on the internet. The F root servers are located at around 50 locations around the globe. You can find the entire list at the below link.
F root server locations (You will find the Atlanta and HongKong locations in this list.)
To understand how routing on the internet works, we need to first understand a little about certain terms and protocols which makes internet work. Internet works on a trust relationship. My network will depend on an another network (managed by somebody else)to reach some other network. To understand how routing in internet works we need to first understand two things. Domains and inter domain routing. Consider domains as a network under an independent administrative authority.
Inter domain routing is described in terms of the number of domains it has to transit or (pass through) to reach a destination address of interest. This entire show of inter domain routing is handled by a protocol called as BGP (Border Gateway protocol).
How packets travel inside a domain is up to the domain administration. In other words the domain authority can use any protocol to provide communication and proper delivery inside the domain. As things that happen inside the domain is internet, the protocols that will be used inside the domain is classified as Interior Gateway Protocol's
As a domain is independent and is handled by an independent administrative authority, this domain is broadly called as Autonomous System's. You will rarely hear the term domain, because its broadly classified and generally referred to as Autonomous System's. Or AS
Am quite sure that you have already heard about the below definition of the internet.
Internet is a large network made by interconnecting small networks. Or network of networks.
As we now know what an Autonomous System is, we can also say that internet is a large collection of interconnected Autonomous systems. But yeah there must be some mechanism by which you can uniquely identify the domains or say AS (autonomous systems). For that Autonomous system's are numbered.
AS numbers are integers ranging from 0 to 4,294,967,295. IANA(Internet Assigned Numbers Authority ) maintains the entire list of AS number's. You can have a look at AS number's in the below link.
As we discussed earlier, BGP is the protocol which runs the entire show in inter domain routing. And BGP makes use of AS number's. Let's see how AS number's are used in BGP.
In the above shown diagram ROUTER 1 receives two different paths to reach 172.16.200.0/24 network. ROUTER 1 can reach the server 172.16.200.200 through two different paths (please note that the servers are different.).
If you see the routing table of ROUTER 1, you will see it has two routes for the subnet 172.16.200.0/24. The routes look something like the below.
- 172.16.200.0/24 AS path (2)
- 172.16.200.0/24 AS path (3,4)
The left most number in the AS path is the AS number from which the router got the route announcement information. So AS path (3,4) means the router received the address advertisement from AS 3(which is its neighbor), and the right most number indicates the AS from which the address advirtisement originated.
Now the normal behavior of BGP(if it has multiple routes to reach the same address prefix) is to select the path that is the shortest to reach the destination. Shortest in terms of AS's it has to go through...
In anycast the same address prefix is announced from multiple locations. And most of the time a router has multiple paths to reach that address. The important point is that in anycasting most of these paths will lead to a different server with the same service. Hence if something happens to one of your anycasted server, you can withdraw the routes to that particular location, and the traffic will then start flowing to the next nearest location.
Please note the fact that explaining BGP and AS path selection is beyond the scope of this article...The things explained here will only be useful to get an idea about how anycasting works in the real world.
What are the main advantages of using Anycast?
- One of the major advantage is to reduce latency in response. A user accessing the service from a particular location will be directed to the nearest end point providing the service.
- Higher service uptime. An issue or technical glitch in one of the anycasted resource will not affect the other one in another locations, due to which users can be routed to that location.
- Better resistance against Distributed Denial Of Service Attacks.
Large scale attacks which consume the server bandwidth and resources are generally executed from multiple locations around the world. The main motive of such attacks is to make a service or website unavailable to a legitimate client.
If the attack originates from a single source address, it can be blocked to a certain extent. But what if the attack originates from 100 different geographical locations and all sending junk traffic to one particular service(for this purposes attackers normally use some thing called as botnet, which are computers infected by malicious programs, and are under control of the attacker.). A normal unicasted service will go unavailable in such conditions.
Attackers use botnet to send junk traffic, because they themselves does not require a large bandwidth to send it to the victim server. A botnet under control of an attacker can generate large amount of traffic (which is basically combined traffic from all infected hosts around the globe). But if the target is using Anycast, such attacks can be mitigated to a larger extent.
Because the attack volume gets distributed to all anycast nodes (As the infected hosts sending the traffic are also from different geographical locations, the traffic will be received by their respective nearest anycast address.)
Disadvantages of Anycast:
- Anycast is not suitable for Long Lived session protocols. For example, its not suitable for TCP and connection oriented stuff. Although if the routing is stable and no fluctuation occurs then it can also be used for TCP protocols like HTTP. I must say some major CDN (Content Delivery Networks) providers already use anycast as their primary mechanism for improving performance and reliability of HTTP.
- A strategic distribution of nodes around the world must be made, with reliable upstream providers for a perfect anycaste architecture.