MPLS - what is that?
In the past years, I learned a lot about the technologies used in the internet and apart from some people dealing with them every day, nobody knows. One of these technologies is MPLS. I read a lot about it but never tried to set it up myself.
Time has come to change that.
What is MPLS and what does it?
Request: If something is wrong here, please let me know, so I don't spread faulty knowledge. Thank you!
MPLS is the abbreviation for Multiprotocol Lable Switching. The basic idea behind MPLS is directing data by inspecting very short labels rather than network addresses. This might not seem to be very important in small networks, but it is in big provider networks. MPLS also offers the possibility to define a path through the network, hop by hop. This might be interesting to ensure a constant low latency for VoIP calls or other time critical applications.
As far as I know (this is backed by en.wikipedia.org/wiki/Multiprotocol_Label_Swicthing), MPLS is mainly used to forward IP packets or ethernet frames (in conjunction with VPLS).
MPLS is also referred as "L2.5" protocol and in fact, it is not possible to make a sharp cut between layer 2 and layer 3 regarding the functionality of MPLS.
Test network setup
The test network consists of four MikroTik RouterBoards (RB750 series), connected in a square as shown in the image below. Each router is connected to its neighbours over a /30 transfer net. The router's main IP address is the first address of the respective /28 net, so A has 10.10.1.129, B 10.10.1.145, C 10.10.1.161 and D 10.10.1.177.
Ignore the net 10.10.1.236/30 net on Router A ether3, this is the interconnect to my LAN at home and is not necessary to proceed with this how-to.
To reproduce my setup, the first step is to configure the IP addresses according the following images. The interface "loopback" is a bridge without any ports. The loopback interface comes handy to address the routers without using any of the /30 net addresses. Configuring a loopback interface is also known best practice as it is a bridge interface which is always up, regardless what's the state of the physical interfaces. So you always have the possibility to address the router without the need to know which interface is up and which IP you can use. This also is a precondition to have OSPF working properly.
To ensure that MPLS will work correctly, we need to make routing between the routers work flawlessly. Therefore we use OSPF, as it is simple to configure - at least is is more comfortable than configuring static routes. The OSPF interface and network configuration is the same on all routers and therefor I only made screenshots from router A. The OSPF instance configuration is different for every router (router ID = loopback address). See images for details:
The next step is finally configuring MPLS. As we know, MPLS uses labels to route traffic. In contrast, conventional routing needs routes (=addresses and prefixes) to decide where to direct a packet. In conventional routing, a routing protocol, for example OSPF, is used to distribute the routes. An equivalent mechanism exists for MPLS: LDP - the Label Distribution Protocol.
We are going to configure LDP general settings as seen in the image below. The LSR ID (Label Switch Router ID) and the transport address is the IP address of the loopback interface. You must repeat this step for every router with the respective IP adress.
Now we configure the interfaces where LDP should work on. You must configure LDP for every ethernet port you want to use MPLS on. Please make sure to check "Accept Dynamic Neighbours". As this configuration is the same on every router, there is only one screenshot.
Now, I am telling you something I struggled a lot with: L2MTU. I wanted to configure a VPLS tunnel over MPLS. The VPLS tunnel should be able to carry unfragmented IP packets, which have a regular size up to 1500 bytes (not talking about jumbo frames here). MPLS in conjunction with VPLS adds another 26 bytes. If you want to use VLAN tags, you will need another 4 bytes, ending up with 1530 bytes in total on layer 2. This fact is exactly the thing I am talking about: It is not sufficient to only configure the MTU of the MPLS interface to 1530 bytes, your physical interface must also be able to transport 1530 bytes at once. In my first experiment, I used a RB750G in the MPLS path, which only has 1522 bytes L2MTU and I did not notice it. So I got the VPLS tunnel working, but it was fragmenting IP packets, resulting in poor performance.
Further information about RouterBoard L2MTU can be found here: http://wiki.mikrotik.com/wiki/Manual:Maximum_Transmission_Unit_on_RouterBoards
As I am now using RouterBoards that can handle L2MTU of 1530 bytes (RB750 and RB750GL), I got the expected result by configuring correct MPLS interface MTU, which is 1530 bytes for me.
You must configure each MPLS interface an every router. Furthermore, you should ensure that every component on the MPLS path is able to process the configured MPLS MTU. Best practice is to configure same MPLS MTU on every device in the MPLS cloud.
Now, the routers exchange their labels, which can be seen in "LDP Neighbors" tab.
Okay, MPLS should be up and running. The final step is to configure the VPLS tunnel. The following steps must be configured on both VPLS endpoints.
In my case, the VPLS tunnel is set up between router A and router C:
To transport plain L2 traffic over the VPLS tunnel from and to physical interfaces, you must create a bridge and add the vpls1 interface and for example the ether3 interface. After you have done this on both devices, you can pass L2 traffic from ether3 on Router A to ether3 an Router B. It will transparently be passed via MPLS and VPLS.
Update 2015-06-13: MPLS Performance
I just ran some performance tests:
I meassured a single FTP transfer between router A (RB750GL, 400MHz) and router C (RB750GL, 400MHz). The traffic is routed over a VPLS tunnel. Router A is the ingress router, router C the egress router. The CPU of the ingress router is maxed out at around 220MBit/s, the egress router's CPU load is around 65%. I think this is due to the heavy encapsulation work that has to be done on the ingress router. The routers inbetween (router B and router D) don't have noticeable CPU load. Due to the label switching going on, they do not need to encapsulate that much.
Another Update on that:
I noticed the ingress router A was rebooting from time to time during transfer. This was caused by the watchdog running on the router. As the CPU gets maxed out, watchdog reboots the router. As the CPU is maxed out almost all the time during transfer, the router keeps rebooting. To prevent this, you could consider disabling watchdog. Keep in mind: This might render the device unaccessable if something really breaks, because it won't reboot automatically.
Update 2015-07-06: Use STP / RSTP
While playing around with the MPLS setup, I noticed strange traffic flows on the VPLS intrerface which is connected to my home network. So I clicked on the "Torch" button of the VPLS interface and saw that there was VoIP traffic from my PBX flowing.
After some thinking, I came to the conclusion, that I have built a loop. Enabling STP on every bridge in the path fixed the problem.
Conslusion: Enable STP or RSTP on the bridge on which you connect the VPLS interface with a physical port to prevent loops!
If the information I provided was helpful to you, I would really appreciate if you have a look on my Amazon wishlist.
I'm not begging for anything and I will continue to share my knowledge but of course I would be really happy to see some packages arriving ;-)