HOWTO BONDING

NIC Bonding

Sometimes this is called port trunking and may be called for something else too, but we will use bonding. But what is bonding? It's shortly making X number of NICs to work as one, with the purse of increasing the throughput (HT), increase the network availability (HA) or a combination of both.

It's possible to use different brands and models of NICs, in a HA setup you can have different speeds (the bond will adapt to the slowest). Even if a NIC supports jumbo frames, it may not always work well in a bond together with jumbo frames.

Before you begin with setting up your bond, check that all of the components used in your bond are working properly, for broken hardware and bad cables will be slightly more difficult to detect when you are setting up your bond for the first time.

Example

This example will include 3 servers, all using 3 NICs for their bond (The servers could have more NICs or/and bonds) and they have a RedHat like Linux which uses network-scripts to configure network settings.

- Server 1: NFS server (ip: 10.0.0.1)
- Server 2: NFS client (ip: 10.0.0.2)
- Server 3: NFS client (ip: 10.0.0.3)

You have to decide if we want to use mii or arp monitoring of the "ports", mii is done locally and won't detect if something stopped to work remotely. Arp has the disadvantage that not all NIC drivers supports features needed for this to work.

You also need to pick a mode how your bond should work, mode 0 - 3 should work with most switches, while mode 4 will require features you won't find in home switches and mode 5 - 6 will require that your NICs driver has ethtool support.

In the /etc/modprobe.conf file add the following (mii):

alias bond0 bonding
options bond0 miimon=80 mode=0

In the /etc/modprobe.conf file add the following (arp, server 1):

alias bond0 bonding
options bond0 arp_interval=80 arp_ip_target=10.0.0.2,10.0.0.3 mode=0

You must specify between 1 and 16 ip-numbers, the more ip-numbers listed in the arp_ip_target the less risk that the "port" will be taken down when the remote machine reboots, all addresses is separated with a comma.

Create the /etc/sysconfig/network-scripts/ifcfg-bond0 (server 1):

DEVICE=bond0
IPADDR=10.0.0.1
NETMASK=255.255.255.0
NETWORK=10.0.0.0
BROADCAST=10.0.0.255
GATEWAY=
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

The ifcfg-bond0 don't really differ from a traditional ifcfg-eth0, and it may have gateway specified

Change the /etc/sysconfig/network-scripts/ifcfg-eth1 to (all servers):

DEVICE=eth1
HWADDR=c6:73:4b:1b:ba:45
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes

Do always specify the hardware address, or else you will never know which NIC is eth0, eht1 and so on and will cause you problems if you have more than one bond or you have an NIC not part of the bond. Do a similar modification for the eth2 and eth3

Now you can restart the network service and you will have a new entry when you run ifconfig, the bond0, it will have the same MAC adress as eth1 and that applies also to eth2 and eth3. If you want to change the mode used, you need to unload the bonding module and change the setting and then load the module again, this can cause some problems if you do it remotely.

If you decide to remove a NIC from the bond, either you take it down manually with ifconfig, or stop the network change the ifcfg-ethX file so that the NIC ain't part of the bond and then start the network again, if you change the file and then restart the network, you will still have the NIC as part of the bond.

Modes

It's possible to assign the mode number or the mode name when selecting the mode in the kernel module option.

       0 or balance-rr

               Round-robin policy: Transmit packets in sequential
               order from the first available slave through the
               last.  This mode provides load balancing and fault
               tolerance. (This is the default mode if no mode specified)

       1 or active-backup

               Active-backup policy: Only one slave in the bond is
               active.  A different slave becomes active if, and only
               if, the active slave fails.  The bond's MAC address is
               externally visible on only one port (network adapter)
               to avoid confusing the switch.

               In bonding version 2.6.2 or later, when a failover
               occurs in active-backup mode, bonding will issue one
               or more gratuitous ARPs on the newly active slave.
               One gratutious ARP is issued for the bonding master
               interface and each VLAN interfaces configured above
               it, provided that the interface has at least one IP
               address configured.  Gratuitous ARPs issued for VLAN
               interfaces are tagged with the appropriate VLAN id.

               This mode provides fault tolerance.  The primary
               option, documented below, affects the behavior of this
               mode.

       2 or balance-xor 
               XOR policy: Transmit based on the selected transmit
               hash policy.  The default policy is a simple [(source
               MAC address XOR'd with destination MAC address) modulo
               slave count].  Alternate transmit policies may be
               selected via the xmit_hash_policy option, described
               below.

               This mode provides load balancing and fault tolerance.

       3 or broadcast

               Broadcast policy: transmits everything on all slave
               interfaces.  This mode provides fault tolerance.

       4 or 802.3ad

               IEEE 802.3ad Dynamic link aggregation.  Creates
               aggregation groups that share the same speed and
               duplex settings.  Utilizes all slaves in the active
               aggregator according to the 802.3ad specification.

               Slave selection for outgoing traffic is done according
               to the transmit hash policy, which may be changed from
               the default simple XOR policy via the xmit_hash_policy
               option, documented below.  Note that not all transmit
               policies may be 802.3ad compliant, particularly in
               regards to the packet mis-ordering requirements of
               section 43.2.4 of the 802.3ad standard.  Differing
               peer implementations will have varying tolerances for
               noncompliance.

               Prerequisites:

               1. Ethtool support in the base drivers for retrieving
               the speed and duplex of each slave.

               2. A switch that supports IEEE 802.3ad Dynamic link
               aggregation.

               Most switches will require some type of configuration
               to enable 802.3ad mode.

       5 or balance-tlb

               Adaptive transmit load balancing: channel bonding that
               does not require any special switch support.  The
               outgoing traffic is distributed according to the
               current load (computed relative to the speed) on each
               slave.  Incoming traffic is received by the current
               slave.  If the receiving slave fails, another slave
               takes over the MAC address of the failed receiving
               slave.

               Prerequisite:

               Ethtool support in the base drivers for retrieving the
               speed of each slave.

       6 or balance-alb

               Adaptive load balancing: includes balance-tlb plus
               receive load balancing (rlb) for IPV4 traffic, and
               does not require any special switch support.  The
               receive load balancing is achieved by ARP negotiation.
               The bonding driver intercepts the ARP Replies sent by
               the local system on their way out and overwrites the
               source hardware address with the unique hardware
               address of one of the slaves in the bond such that
               different peers use different hardware addresses for
               the server.

               Receive traffic from connections created by the server
               is also balanced.  When the local system sends an ARP
               Request the bonding driver copies and saves the peer's
               IP information from the ARP packet.  When the ARP
               Reply arrives from the peer, its hardware address is
               retrieved and the bonding driver initiates an ARP
               reply to this peer assigning it to one of the slaves
               in the bond.  A problematic outcome of using ARP
               negotiation for balancing is that each time that an
               ARP request is broadcast it uses the hardware address
               of the bond.  Hence, peers learn the hardware address
               of the bond and the balancing of receive traffic
               collapses to the current slave.  This is handled by
               sending updates (ARP Replies) to all the peers with
               their individually assigned hardware address such that
               the traffic is redistributed.  Receive traffic is also
               redistributed when a new slave is added to the bond
               and when an inactive slave is re-activated.  The
               receive load is distributed sequentially (round robin)
               among the group of highest speed slaves in the bond.

               When a link is reconnected or a new slave joins the
               bond the receive traffic is redistributed among all
               active slaves in the bond by initiating ARP Replies
               with the selected mac address to each of the
               clients. The updelay parameter (detailed below) must
               be set to a value equal or greater than the switch's
               forwarding delay so that the ARP Replies sent to the
               peers will not be blocked by the switch.

               Prerequisites:

               1. Ethtool support in the base drivers for retrieving
               the speed of each slave.

               2. Base driver support for setting the hardware
               address of a device while it is open.  This is
               required so that there will always be one slave in the
               team using the bond hardware address (the
               curr_active_slave) while having a unique hardware
               address for each slave in the bond.  If the
               curr_active_slave fails its hardware address is
               swapped with the new curr_active_slave that was
               chosen.

Use together with KVM

This isn't something you are meant to do within your KVM guests, but on the host, you can assign the bridge to the bond instead of the traditional eth0, this way you will have HA, HT or HA/HT setup.

Problem with Bridge + Bonding

There is a known ARP problem for bridge on a bonded interface. Ref:

Please let me know if you know a solution.