What is TCP MULTIPATH and how does MULTIPATH in TCP work

Sarath Pillai's picture
Multipath TCP

Every day we read some or the other technical stuff online that we feel like will be a boom in the coming days. Most of the times the stuff we stumble upon will only be a modified version of an already existing thing, or say a re-engineered version of an already existing one.

 

But in recent days I stumbled upon a major improvement in the networking field, that will surely become a major leap in the way communication systems are designed today.

 

Before getting into the details of the technology, we will begin with the basics so that even non technical readers can understand the importance of the proposed technology. I must not say proposed, because some of the major players who are in the forefront of technology, have already implemented it and some even went to an extent of providing it as a feature in their newly released heavy marketed software versions. We will discuss more about who implemented this in the later part of this article.

 

To understand this technology, we need to understand some basics behind what happens when we communicate over the internet. We will begin this with what is multi homed and multi addressed in terms of networking.

 

What is multi homing ?

 

Generally a computer device, or any communication device is attached to one single network. However, these days we have our computing devices that is attached to multiple networks, at one time.

 

A good example of multi homing is a tablet or a multimedia phone that is attached to both mobile networks (either through GPRS or a 3G connection) & a local WiFi network.

 

A better example of multi homing is a server machine/ or a networking router, that is connected to two Internet Service Providers. These routers and server machines often purchase multiple network connection links from multiple Internet Service Providers, to achieve multi homing functionality.

 

The major advantage of having a multi homing server or a router is that even if something happens to one network link (or say something happens to one ISP), the server/router will still remain connected through the other ISP.

 

To summarize this, multi homing is primarily done for achieving better redundancy in networks. Please don't forget the fact that multi homing will increase your cost because you will have to pay multiple ISP's for maintaining your redundant links.

 

Now as I have told, multi homing is primarily designed for better redundancy (which means if one link goes down due to some reason, the other can be used to reach the target destination). But you can never use both the links simultaneously at one time to communicate.

 

The major road block that the networking world has faced till now was that, we were unable to use these two networking links (multi homed), simultaneously for one TCP connection. Which means if I have to communicate with yahoo, even-though I have two network connection, I have to use either of them to communicate. And I have to use that one connection till the end of that TCP session.

 

If you think logically, its simply waste of networking resources. Because I can achieve better connectivity and throughput if I was able to use both of the connections together at one time to communicate with yahoo in a single TCP session.

 

You might think what is the purpose of using multiple links or say multiple paths together at one time for one TCP connection. The reason is that we have come a long way using internet. And the use case we have today is much different that the one we used to have during the designing of these communication protocols.

 

  • We conduct video calls and conferences over internet

  • We use the internet for phone calls

  • We use internet for location services like GPS

All of the above mentioned use cases need consistent and better network throughput and connectivity. Especially voice data needs more throughput and connectivity than others. Let's have a look at what happens when a normal TCP connection takes place.

 

We know the fact that unlike UDP, TCP requires a connection to be established before any data transfer occurs. But through the complete series of action there are several things that play a major role in establishing the connection, keeping track of the data transferred, error detection, and re-ordering of data at the receiver end.

 

Data is broken into smaller chunks and are send one by one..Each packet send has a sequence number attached to it. The beginning sequence number is randomly selected, and is incremented by one for each packet sent. The receiving side rebuilds the original data by assembling each packet received with the help of order of sequence numbers.

 

A normal single path TCP connection gets established by a series of 3 messages exchanged between the server and the client. Its called as a "Three way handshake", the name comes from the three message types exchanged. I will recommend reading the below article about TCP three way handshake to understand how a normal TCP connection is established. 

Read: TCP Three way Handshake Explained

 

In simple terms the steps involved in establishing a normal TCP connection is explained below.

 

Step 1: The machine who wants to initiate a connection sends a SYN request with initiation request and its beginning sequence number.

Step 2: The receiving machine replies back to the SYN message with a SYN-ACK message. This message contains its beginning sequence number and acknowledgment of the SYN message received.

Step 3: On receiving the SYN-ACK message, the initiator sends a final ACK, containing the acknowledgement of the SYN-ACK message, and the connection is considered to be established.

Now let's see the design limitations of single path TCP (The current protocol, we all are using. Also is used by you while reading this article.)

 

 

  • TCP uses only one single path to establish a connection, and that single path will be used for that entire TCP session (till the data transfer and connection purpose is completed)
  • Imagine a situation where a connection is made, and the path through which the connection is established is congested (don't forget the fact that the same network bandwidth is used by other users also, all must have an equal share of bandwidth). So if you are using a connection and 3 other users are also using the same path then each of them will have only 33% of the total available bandwidth. This is the case even if you have another connection that's available (as mentioned before single path TCP will never use the other connection/link available)
  • The situation of congestion will become worse, if the router devices in between the source and destination has some kind of QoS enabled. QoS stands for Quality of Service, its nothing but a method used to provide higher preference to a specific set of protocols over the other.

 

Is there any method to use multiple paths with the two connection links available, in a single connection ?

 

The answer to that question is YES. Both the connections can be used together for one single connection (i must not say can be used, because some of them have already started to use this.)

 

The technology that can be used to achieve this is called MPTCP or Multi Path TCP.

 

There is a huge benefit of using both the connections at one time together. Two of those main benefits as far as an end user is concerned are outlined in the below bullet points.

 

1.  MPTCP (Multi path TCP) will increase the bandwidth

It will increase the bandwidth because two connection links with two separate paths are used in a single connection. Due to congestion if one path is only providing a small percentage of its bandwidth, the other path can also be utilized. Hence the total bandwidth for a MPTCP connection will be the combined bandwidth used by both the paths/links

        2.  MPTCP ( Multi Path TCP ) provides Better Redundancy

Multi path TCP provides a better connection redundancy, because your connection will not be affected even if one link goes down. An example use case is suppose you are watching a video online and you are over your WiFi connection. Even if you walk out of your WiFi connection range, the video streaming should not be affected because it should automatically stop sending data through WiFi connection and should now only use cellular network. The end user must never feel that there was a connection loss of some kind.

 

Technical details of a proposed technology are very easy to describe theoretically. But a proper implementation of that in the real world is quite difficult. The main difficulty in launching any protocol is that, it should not break an already existing one, and should be capable of working at ease with the already deployed infrastructure.

 

Some major considerations that were made during the design of multi path TCP are as below.

 

  • It should work with an already exisiting internet architecture. And should not require any kind of modifications on the routers that comes in between the communication path
  • A modification of computer applications must not be required
  • Middle boxes such as NAT devices should never destroy the multi path tcp connection (Dont forget the fact that NAT devices are designed to modify communication packets that flow through them)
  • It should be backward compatible. Means if due to any reason a successful multi path tcp connection cannot be set up, it must always fall back to the normal TCP connection.

 

Who first suggested the idea of having a multi path TCP connection to over come the limitations in TCP?

 

The idea was first suggested by Christian Huitema during early 1990's. However it took quite a long time to see idea in action, due to the challenges involved in designing such a protocol. He is part of the engineering team in Microsoft. His involvement is majorly in the field of networking. You can find a shoft biography of his from the below link

Christian Huitema

Multi path TCP is implemented at Layer 4. Layer 4 is the Trasport layer, which comes in between the application and network layer. Below the Transport layer are Network Layer, Data Link Layer, and the Physical layer. As Multi path TCP is implemented in the transport layer, no major changes needs made to the routing devices that comes in between the communication path, because these routing devices primarily use network layer data (however there are still some complications if you have a device that filters segments based on unknown flags).

 

Multipath TCP and transport layer

 

As mentioned in the above figure, Multi Path TCP is an added extention to TCP to improve its functionalities. Let's discuss the series of events that occur when establishing a Multi Path TCP connection.

 

Multi Path TCP handshake

 

 

Let's now understand the series of steps described in the above shown Multi Path TCP handshake.

There are two machines that are used in illustrating a multi path tcp handshake. Machine 1 & Machine 2. These two machines have two interfaces each.

MA1= First interface on machine 1

MA2= Second Interface on machine 1

MB1= First interface on machine 2

MB2= Second interface on machine 2

 

Step 1: Step 1 is sending a SYN request to the destination. This is very much similar to the normal TCP SYN request, but it contains an extra flag that mentions the server's capability to use Multi Path TCP. This extra flag is MP_CAPABLE (which is shown in the figure above)

 

Step 2: Step 2 is a SYN ACK reply that also contains an MP_CAPABLE flag set. Now please note the fact that if machine 2 replies a SYN ACK without a MP_CAPABLE flag, that means machine 2 does not support Multi Path TCP.

 

Step 3: This step is a final ACK reply from machine 1, also with an MP_CAPABLE flag set. Please note the fact that all these three messages contain a key that verifies the messages (machine 1 sends its keys during SYN, machine 2 sends its keys during SYN ACK, and machine 1 sends back both the keys during ACK as a final message)

 

Now the initial TCP connection is set, with both the machines informing each other's capability to use Multi Path TCP. The above established connection was from interface 1 on machine 1 to interface 1 on machine 2. Hence for the completion of a multi path connection we still need to have another connection (second path), to be established from the second interface.

 

Step 1: The second connection that will be made is called as a subflow in terms of Multi Path TCP. Machine 1 sends the first SYN with a token which is created from the keys exchanged in the first connection. This token is used to authenticate the subflow session. Also a flag of MP_JOIN is set on the SYN request. This flag helps in joining this connection to the already created one

 

Step 2: On receiving the SYN with MP_JOIN flag, machine 2 will respond back with a SYN ACK (which will also contain a MP_JOIN flag set). This message also contains MAC & Address ID's

 

Step 3: On receiving the SYN ACK, machine 1 will reply back with an ACK message, which also contains a MP_JOIN flag set (This also contains MAC)

 

Step 4: The final message is an ACK message from machine 2, which acts a confirmation that it recieved the MAC and ACK message from machine 1. This final ACK from machine 2 established the sublow connection

 

Some awesome features of TCP Multi PATH

 

  • If one interace has both ipv4 & ipv6 then two seperate subflows are created. One for ipv4 & one for ipv6
  • If you have more than 2 connection interfaces, more that two paths are made
  • Paths that do not work are dropped after a certain time out interval
  • If one of the path fails, then the other available working path is used to inform the reciever that a particular path has failed, rather than waiting for the timeout to complete

 

Which devices and Operating systems have Multipath TCP already deployed?

 

Latest versions of Linux kernel supports Multi Path tcp, provided both the side of the connection supports the feature. The latest implementation of multipath tcp for linux can be found from the below link.

MultiPath TCP v0.87

 

Also the latest heavy marketed apple mobile operating system iOS 7 is the first operating system to push the wide use of multipath TCP. So those who have updated their iOS devices to the latest version, knowingly or unknowingly use this technology.

 

Hope this article was helpful in getting a basic understanding of what Multi Path TCP is, and how it works. Kindly please let me know about your views and suggestions.

 

Rate this article: 
Average: 4 (4 votes)

Comments

Excellent document. I have the following doubt. you have mentioned the below

"Latest versions of Linux kernel supports Multi Path tcp, provided both the side of the connection supports the feature."

What do you mean by "provided both the side of the connection supports the feature.".

For example if i plan to implement this feature of "MultiPath TCP" on my Linux Router which has 2 ISP connections, do you mean the other end Router (i.e. the ISP's end) also should have this feature of "MultiPath TCP".

Please clarify.

Thanks

Sarath Pillai's picture

Hi,

Glad to know that you liked the article!!

What i meant by "provided both the side of the connection supports the feature" was that Multipath tcp will only work if both the sides supports it. Otherwise how will the other side of the connection know that a particular traffic belonged to which connection etc.

So normally if you try to establish a multipath tcp to a target server, and it does not support it, the connection will fallback to normal tcp.

The latest ios7(Apple's mobile operating system) supports Multipath tcp connections. But reports claim that multipath tcp is only used by ios7 while using siri(and not normal web browsing). This is because siri central server must be supporting multipath tcp, while other normal websites we visit is normal tcp for sure. The main plus point of multipath tcp is that, the client can always send info saying i support multipath tcp, but if the server does not respond with multipath tcp then the client will fallback to normal tcp. This is in fact a good feature because we can slowly move to multipath tcp and others who does not support are not affected at all.

So even if you enable mutlipath tcp on your linux server, the other server/client to which you will be connecting should also support multipath tcp packets.

Please let me know.

Thanks

Hi Sarath,

Thanks for the quick response with the explanation. As my task is to setup a redundant Internet link with 2 ISP connections, I thought I can achieve it by MPTCP which is really great. As we never know whether the ISP end supports MPTCP, I think this option will not work for my requirement.

Can I achieve this by using Ethernet Channel bonding on Linux, pl. advise. or is there any mechanism which can be used using a Linux Router.

Thanks

Sarath Pillai's picture

Hi George John,

Sorry for the delayed reply(i was busy with some other stuff). Anyways..If you want to set up redundancy in network link using multiple ISP then NIC bonding is not a good solution...

There are several small tools available to make this work. Its always good to have three NIC cards on your server(1 for your LAN, 2nd for first ISP and 3rd for second ISP.). I personally have not tested these(anyways its a good point that you asked, i myself have to test these..if success i will surely spin up an article with my findings.)

1. The first solution is, you can load balance the two connections with simply adding two weighted routes on the server (give more weight to the ISP which is having more bandwidth). There is a script available for this as well (https://code.google.com/p/netsane/)

2. Second is to have a fail-over testing script to remove and add routes if one failes. And example script can be found here....https://github.com/Evanlec/config/blob/master/bin/gwping

3. There is another tool solely made for this purpose called as ispunity. As i told before, i have not tested this, but can be helpful to you(a few of my friends reported to have used this to a certain level of success.). http://www.ispunity.com/. Ispunity addresses both the load-balencing and failover issues. I would recommend you to try this first.

Please let me know if it works for you..

Cheers
Sarath

Hi Sarath,

If you could also explain TCP slow start. I have been trying to wrap my head around that concept.

Thanks,
Kiran

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.