NTH load balancing with masquerade

[patris1]

http://wiki.mikrotik.com/wiki/NTH_load_balancing_with_masquerade

Introduction

This example is improved (different) version of round-robin load balancing example. It adds persistent user sessions, i.e. a particular user would use the same source IP address for all outgoing connections. Consider the following network layout:

Quick Start for Impatient

Configuration export from the gateway router

/ ip address
add address=192.168.0.1/24 network=192.168.0.0 broadcast=192.168.0.255 interface=Local 
add address=10.111.0.2/24 network=10.111.0.0 broadcast=10.111.0.255 interface=wlan2
add address=10.112.0.2/24 network=10.112.0.0 broadcast=10.112.0.255 interface=wlan1

/ ip firewall mangle
add chain=prerouting src-address-list=odd in-interface=Local action=mark-connection \
  new-connection-mark=odd passthrough=yes 
add chain=prerouting src-address-list=odd in-interface=Local action=mark-routing \
  new-routing-mark=odd passthrough=no
add chain=prerouting src-address-list=even in-interface=Local action=mark-connection \
  new-connection-mark=even passthrough=yes 
add chain=prerouting src-address-list=even in-interface=Local action=mark-routing \
  new-routing-mark=even passthrough=no
add chain=prerouting in-interface=Local connection-state=new nth=2,1 \ 
    action=mark-connection new-connection-mark=odd passthrough=yes
add chain=prerouting in-interface=Local action=add-src-to-address-list \
  address-list=odd address-list-timeout=1d connection-mark=odd passthrough=yes 
add chain=prerouting in-interface=Local connection-mark=odd action=mark-routing \ 
    new-routing-mark=odd passthrough=no
add chain=prerouting in-interface=Local connection-state=new nth=2,2 \ 
    action=mark-connection new-connection-mark=even passthrough=yes
add chain=prerouting in-interface=Local action=add-src-to-address-list \
  address-list=even address-list-timeout=1d connection-mark=even passthrough=yes 
add chain=prerouting in-interface=Local connection-mark=even action=mark-routing \ 
    new-routing-mark=even passthrough=no

/ ip firewall nat 
add chain=srcnat out-interface=wlan1 action=masquerade
add chain=srcnat out-interface=wlan2 action=masquerade

/ ip route 
add dst-address=0.0.0.0/0 gateway=10.111.0.1 scope=255 target-scope=10 routing-mark=odd
add dst-address=0.0.0.0/0 gateway=10.112.0.1 scope=255 target-scope=10 routing-mark=even 
add dst-address=0.0.0.0/0 gateway=10.112.0.1 scope=255 target-scope=10

Explanation

First we give a code snippet and then explain what it actually does
IP Addresses

/ ip address 
add address=192.168.0.1/24 network=192.168.0.0 broadcast=192.168.0.255 interface=Local
add address=10.111.0.2/24 network=10.111.0.0 broadcast=10.111.0.255 interface=wlan2 
add address=10.112.0.2/24 network=10.112.0.0 broadcast=10.112.0.255 interface=wlan1

The router has two upstream (WAN) interfaces with the addresses of 10.111.0.2/24 and 10.112.0.2/24. The LAN interface has the name "Local" and IP address of 192.168.0.1/24
Mangle

/ ip firewall mangle 
add chain=prerouting src-address-list=odd in-interface=Local action=mark-connection \
  new-connection-mark=odd passthrough=yes 
add chain=prerouting src-address-list=odd in-interface=Local action=mark-routing \
  new-routing-mark=odd

All traffic from customers having their IP address previously placed in the address list "odd" is instantly marked with connection and routing marks "odd". Afterwards the traffic is excluded from processing against successive mangle rules in prerouting chain

/ ip firewall mangle 
add chain=prerouting src-address-list=even in-interface=Local action=mark-connection \
  new-connection-mark=even passthrough=yes 
add chain=prerouting src-address-list=even in-interface=Local action=mark-routing \
  new-routing-mark=even

Same stuff as above, only for customers having their IP address previously placed in the address list "even".

/ ip firewall mangle 
add chain=prerouting in-interface=Local connection-state=new nth=2,1 \ 
    action=mark-connection new-connection-mark=odd passthrough=yes
add chain=prerouting in-interface=Local action=add-src-to-address-list \
  address-list=odd address-list-timeout=1d connection-mark=odd passthrough=yes 
add chain=prerouting in-interface=Local connection-mark=odd action=mark-routing \ 
    new-routing-mark=odd passthrough=no

First we take every second packet that establishes new session (note connection-state=new), and mark it with connection mark "odd". Consequently all successive packets belonging to the same session will carry the connection mark "odd". Note that we are passing these packets to the second and third rules (passthrough=yes). Second rule adds IP address of the client to the address list to enable all successive sessions to go through the same gateway. Third rule places the routing mark "odd" on all packets that belong to the "odd" connection and stops processing all other mangle rules for these packets in prerouting chain.

/ ip firewall mangle 
add chain=prerouting in-interface=Local connection-state=new nth=2,2 \ 
    action=mark-connection new-connection-mark=even passthrough=yes
add chain=prerouting in-interface=Local action=add-src-to-address-list \
  address-list=even address-list-timeout=1d connection-mark=even passthrough=yes 
add chain=prerouting in-interface=Local connection-mark=even action=mark-routing \ 
    new-routing-mark=even passthrough=no

These rules do the same for the remaining half of the traffic as the first three rules for the first half of the traffic.
The code above effectively means that each new connection initiated through the router from the local network will be marked as either "odd" or "even" with both routing and connection marks.
The above works fine. There are however some situations where you might find that the same IP address is listed under both the ODD and EVEN scr-address-lists. This behavior causes issues with apps that require persistent connections. A simple remedy for this situation is to add the following statement to your mangle rules

add chain=prerouting in-interface=Local connection-state=new nth=2,2 \ 
    src-address-list=!odd action=mark-connection new-connection-mark=even \
    passthrough=yes

This will ensure that the new connection will not already be part of the ODD src-address-list. You will have to do the same for the ODD mangle rule thus excluding IP's already part of the EVEN scr-address-list

/ ip firewall nat 
add chain=srcnat out-interface=wlan1 action=masquerade
add chain=srcnat out-interface=wlan2 action=masquerade

Fix the source address according to the outgoing interface

/ ip route 
add dst-address=0.0.0.0/0 gateway=10.111.0.1 scope=255 target-scope=10 routing-mark=odd 
add dst-address=0.0.0.0/0 gateway=10.112.0.1 scope=255 target-scope=10 routing-mark=even

For all traffic marked "odd" (consequently having 10.111.0.2 translated source address) we use 10.111.0.1 gateway. In the same manner all traffic marked "even" is routed through the 10.112.0.1 gateway

/ ip route
add dst-address=0.0.0.0/0 gateway=10.112.0.1 scope=255 target-scope=10

Finally, we have one additional entry specifying that traffic from the router itself (the traffic without any routing marks) should go to 10.112.0.1 gateway

---

موضوعات مشابه:

• مشکل با load balancing و اینتر نت مخابرات
• Load Balancing Exchange 2007 Client Access Servers using Windows Network Load-Balancing Technology
• مشكل كانفيگ كردن Load Balancing
• Qos , Load Balancing

---

[patris1]

http://wiki.mikrotik.com/wiki/NTH_load_balancing_with_masquerade_%28another_approach%29

NTH load balancing with masquerade (another approach)

From MikroTik Wiki

Contents

• 1 Summary
• 2 Fast forward
• 3 Detailed explanation
• 4 Packet route logic
• 5 How to expand this example to more WANs

Summary

To load balance using Nth can be done in several different ways. This approach focuses on performance, meaning, less rules packet takes, less attributes rule have to check. At current state majority of packets will take N rules, where N is number of connections you try to load balance using this example. This example is easy to expand from 2 connections to any number you wish, but consequences are: more connections, more rules packets have to pass, greater load on router, bigger latency.
In this example N=2
NOTE: this example assumes you have empty mangle. after rules have been processed, packets are accepted with routing-mark rules.
Fast forward

for those impatient

/ip address add address=10.0.0.1/24 interface=first-Out
/ip address add address=10.0.1.1/24 interface=second-Out
/ip address add address=172.16.0.1/24 interface=ether3-Local
/ip address add address=172.16.1.1/24 interface=ether4-Local

/ip firewall address-list add address=172.16.0.0/24 list=local
/ip firewall address-list add address=172.16.1.0/24 list=local

/ip route add gateway=10.0.0.2
/ip route add gateway=10.0.0.2 routing-mark=first
/ip route add gateway=10.0.1.2 routing-mark=second

/ip firewall nat add chain=srcnat out-interface=first-Out action=masquerade
/ip firewall nat add chain=srcnat out-interface=second-Out action=masquerade

/ip firewall mangle add action=add-src-to-address-list address-list=first address-list-timeout=0s chain="mark new unseen" disabled=no nth=2,1
/ip firewall mangle add action=add-src-to-address-list address-list=second address-list-timeout=0s chain="mark new unseen" disabled=no nth=2,2
/ip firewall mangle add action=add-src-to-address-list address-list=seen address-list-timeout=0s chain="mark new unseen" disabled=no
/ip firewall mangle add action=jump chain="mark new unseen" disabled=no jump-target="mark connection"
/ip firewall mangle add action=mark-connection chain="mark connection" disabled=no new-connection-mark=first_conn passthrough=yes src-address-list=first
/ip firewall mangle add action=mark-connection chain="mark connection" disabled=no new-connection-mark=second_conn passthrough=yes src-address-list=second
/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=first_conn disabled=no new-routing-mark=first passthrough=no
/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=second_conn disabled=no new-routing-mark=second passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=first_conn disabled=no new-routing-mark=first passthrough=no src-address-list=first
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=second_conn disabled=no new-routing-mark=second passthrough=no src-address-list=second
/ip firewall mangle add action=jump chain=prerouting connection-state=new disabled=no jump-target="mark connection" src-address-list=local
/ip firewall mangle add action=jump chain=prerouting connection-state=new disabled=no jump-target="mark new unseen" src-address-list=local

Detailed explanation

• Adding ip addresses to interfaces

I am assuming i have 2 outgoing WAN and 2 Local LAN. I assume that addresses on WAN are public (not in example)

/ip address add address=10.0.0.1/24 interface=first-Out
/ip address add address=10.0.1.1/24 interface=second-Out
/ip address add address=172.16.0.1/24 interface=ether3-Local
/ip address add address=172.16.1.1/24 interface=ether4-Local

• Creating address list of possible local addresses

We will need this list in our configuration, so only traffic from local interfaces are marked with routing marks. You can also use in interface if there are just one incoming LAN interface on the router

/ip firewall address-list add address=172.16.0.0/24 list=local
/ip firewall address-list add address=172.16.1.0/24 list=local

• Adding routes

Default route for unmarked traffic, and 2 routes for marked routes.
NOTE: connections to router will only work to 10.0.0.1 address. Connections to other WAN address will always fail, that is configurable, but it is out of scope of this document

/ip route add gateway=10.0.0.2
/ip route add gateway=10.0.0.2 routing-mark=first
/ip route add gateway=10.0.1.2 routing-mark=second

• Masquerade rules

So our local addresses can access internet addresses

/ip firewall nat add chain=srcnat out-interface=first-Out action=masquerade
/ip firewall nat add chain=srcnat out-interface=second-Out action=masquerade

• Mangle rules

Where the whole marking is made. I am dividing mangle in 5 sections (A-E)

• Section A

These 4 rules adds address to address list, as result, we are dividing all internal addresses currently active to dynamic address lists first and second these will be correspondingly routed through corresponding gateways. When that is done, address for simplicity is added to one more address list - seen so we know that we have seen this address and do not have to check more than once. When everything is done we jump to mark connection and set routing-mark for packet we are working with. Here we are working just with new packets that we have not seen yet.
After this section finishes, these packets are not different from those that are matched in Section D, so they are passed to Section B for further processing

/ip firewall mangle add action=add-src-to-address-list address-list=first address-list-timeout=0s chain="mark new unseen" disabled=no nth=2,1
/ip firewall mangle add action=add-src-to-address-list address-list=second address-list-timeout=0s chain="mark new unseen" disabled=no nth=2,2
/ip firewall mangle add action=add-src-to-address-list address-list=seen address-list-timeout=0s chain="mark new unseen" disabled=no
/ip firewall mangle add action=jump chain="mark new unseen" disabled=no jump-target="mark connection"

• Section B

Next 4 rules are marking connection of both, new packets from hosts we have not seen yet and with new packets from seen hosts. First, mark connection, then add routing-mark

/ip firewall mangle add action=mark-connection chain="mark connection" disabled=no new-connection-mark=first_conn passthrough=yes src-address-list=first
/ip firewall mangle add action=mark-connection chain="mark connection" disabled=no new-connection-mark=second_conn passthrough=yes src-address-list=second
/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=first_conn disabled=no new-routing-mark=first passthrough=no
/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=second_conn disabled=no new-routing-mark=second passthrough=no

• Section C

Next 2 rules are setting up routing-mark on packets that have connection-mark set. As result majority of packets are passing though just these 2 rules

/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=first_conn disabled=no new-routing-mark=first passthrough=no src-address-list=first
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=second_conn disabled=no new-routing-mark=second passthrough=no src-address-list=second

• Section D

This rule caches new connection packets that come from our "seen" clients, eg, client initiated new http download session (opening web page). Packets are passed to Section B where they are marked

/ip firewall mangle add action=jump chain=prerouting connection-state=new disabled=no jump-target="mark connection" src-address-list=local

• Section E

If client ip address is not in our seen list, then address is passed to Section A where it is added to address list and after that is ready to be processed

/ip firewall mangle add action=jump chain=prerouting connection-state=new disabled=no jump-target="mark new unseen" src-address-list=local

Packet route logic

• New packet from unseen addressee

When router is booting up it have no seen list, and no clients are assigned to gateways. Or packet is received from previously unseen client. When first packet arrives it is checked in Section C, as it does not match there, it is passed over to Section D and then to Section E where it is finally matched and passed for processing on Section A. In Section A packet is matched and assigned to either of 2 address lists (first and second) and then added to seen address-list. After that is done, packed is passed to Section B where its connection is marked and then packet receives its routing mark and is accepted.

• New packet from seen addressee

Packet is passed through Section C to Section D where it is matched and passed to Section C where connection is marked and accepted

• Packet from seen addressee

Packet arrives in Section C and is matched there and accepted.


How to expand this example to more WANs

To have more WANs you have to add additional IP address and additional route with routing-mark, eg, third
Then you have have to edit Sections A-C

• Changes in Section A

here we have to adjust nth field value first value is what number of packed we are looking for, usually it is equal to your WAN count. And add additional rule as in example below

/ip firewall mangle add action=add-src-to-address-list address-list=first address-list-timeout=0s chain="mark new unseen" disabled=no nth=3,1
/ip firewall mangle add action=add-src-to-address-list address-list=second address-list-timeout=0s chain="mark new unseen" disabled=no nth=3,2
/ip firewall mangle add action=add-src-to-address-list address-list=third address-list-timeout=0s chain="mark new unseen" disabled=no nth=3,3
/ip firewall mangle add action=add-src-to-address-list address-list=seen address-list-timeout=0s chain="mark new unseen" disabled=no
/ip firewall mangle add action=jump chain="mark new unseen" disabled=no jump-target="mark connection"

• Changes in Section B

here we will have to add 2 new rules, to mark connections that source address is in third address-list, and after that mark routing corresponding to connection mark

/ip firewall mangle add action=mark-connection chain="mark connection" disabled=no new-connection-mark=first_conn passthrough=yes src-address-list=first
/ip firewall mangle add action=mark-connection chain="mark connection" disabled=no new-connection-mark=second_conn passthrough=yes src-address-list=second
/ip firewall mangle add action=mark-connection chain="mark connection" disabled=no new-connection-mark=third_conn passthrough=yes src-address-list=thrid
/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=first_conn disabled=no new-routing-mark=first passthrough=no
/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=second_conn disabled=no new-routing-mark=second passthrough=no
/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=third_conn disabled=no new-routing-mark=third passthrough=no

• Changes in Section C

Here have to add rule just like in section B just change chain to prerouting as all other rules in this section.

/ip firewall mangle add action=mark-routing chain="mark connection" connection-mark=third_conn disabled=no new-routing-mark=third passthrough=no

/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=first_conn disabled=no new-routing-mark=first passthrough=no src-address-list=first
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=second_conn disabled=no new-routing-mark=second passthrough=no src-address-list=second
/ip firewall mangle add action=mark-routing chain="prerouting" connection-mark=third_conn disabled=no new-routing-mark=third passthrough=no