ECMP routing - equal-cost multi-path routing
The jumpNet is a hub and spoke architecture. There are two central VPN servers to which all clients are connected simultanously. Furthermore, all routes between the clients are exchanged with OSPF. Unfortunatelly, I needed to adjust the interface costs on one of the two VPN servers, to create a preferred path, as I had trouble to get ECPM working in the first place.
Point of origin:
- The jumpNet is founded on the address space 10.10.0.0/16
- Each client network gets a 10.10.X.0/24
- On the VPN servers, there is a stateful firewall
- The VPN servers are MikroTik Routerboards running RouterOS
Configuring ECMP is very easy: Just configure a equal-cost path between two points in your network with OSPF and your're done. For each flow the decision will be taken which next-hop to use. So for multiple flows, the path usage will be balanced.
The problem which will occur consists of two parts:
- Asymmetric routing will be possible as there are now two equal-cost paths available. This is perfectly fine, as routing is in fact stateless.
- We're using a stateful firewall, which will allow established and related connections. For this to work, you must assure, that the firewall can "see" the "new" connection part aswell as the "established" connection part. This is a contradiction to 1.
See the following diagram:
All routing in the internet should be considered as potentially beeing asymmetric and this is, as I already said, perfectly fine.
Look at the diagram: The router R1 is originating a TCP connection to R4. R1 has two available paths to R4. The upper path routes the traffic over R2, the lower path over R3. In this example, the TCP SYN flagged packet is routed over R2.
R4 also has two available paths to reach R1. R4 receives the TCP SYN flagged packet and answers with an TCP SYN ACK flagged packet. This time, the traffic is routed over R3.
Imagine there are stateful firewalls configured on R2 and R3. In this case, as the TCP SYN flagged packet from R1 gets routed via R2, R2 sees this as new connection because of the SYN flag. Assuming there are accept rules for "new", "established" and "related" connections from R1, an entry in R2 firewall's state table will be created. R2 passes the TCP SYN flagged packet to R4 and R4 replies with TCP SYN ACK packet in reply. But this time, the traffic is routed over R3. Again, assuming that there are accept rules for "new", "established" and "related" connections from R4, the firewall will block the TCP SYN ACK packet as it never saw the corresponding TCP SYN flagged packet. From the perpespective of R3, the single TCP SYN ACK packet belongs to a TCP conenction with an invalid connection state.
Hum. So there is a goal conflict between asymmetric routing (which is stateless) and a stateful firewall.
How to solve this? The simple solution would be: Disable connection tracking. This is not a practical solution as you'll lose firewall protection, we must develop a more sophisticated solution.
The correct solution would be, to ensure, that traffic from the desired sources doesn't get tracked by the firewall. In the case of the jumpNet, I have to except the 10.10.0.0/16 range from beeing tracked on the VPN servers. To do so, we need two rules in the firewall's RAW table and an additional rule in the forwarding filter chain:
/ip firewall raw add action=accept chain=prerouting comment=\ "force connection tracking for local net sourcing" src-address-list=\ conntracked add action=notrack chain=prerouting comment="ECMP for VPN" dst-address-list=\ !conntracked src-address=10.10.0.0/16
In the address-list "conntracked", there are the local IP ranges which still should get tracked.
/ip firewall filter add action=accept chain=forward connection-state=untracked
Make sure, this rule is the very first rule in the forward chain!
That's it. Now, remote packets from the 10.10.0.0/16 range will be excpeted from connection tracking and all other traffic, especially locally originated traffic still gets tracked.
With this setup, you'll be unable to use MikroTik's FastTrack on the VPN servers. At least I couldn't manage to get it working together with the firewall configuration above.
With this configuration, we can benefit from the stateful firewall on the one hand and on the other hand, we can also benefit from ECMP over the VPN connections.
Of course this will also work for non VPN conenctions.
I hope you'll find this usefull. Please let me know your thoughts and of course let me know if I missed something. Just write an e-mail to email@example.com
Thanks for reading! Next time, I'll handle traffic engineering over redundant paths.
Further information on that topic
If you like to read on: