Configuring an LACP bonded Interface with Cisco switches

We have received enough interest from our user community about XenServer and network configuration techniques that we felt it might be useful to provide some of the information we have assembled for all.  Please forward all comments/corrections to info@quorumsoft.com

The default bonding mode for Citrix XenServer 5.5/5.6 is "1", or active-backup (slb).  This mode only provides fault tolerance, since only one NIC in the bond is active at a time.  For more heavily utilized production systems, an active-active configuration is preferable.  The two bonding modes that provide full send/receive balancing across all NICs are 4 (802.3ad) and 6 (alb).  Mode 4 provides link aggregation according to the 802.3ad specification, and requires an etherchannel to be configured on the switch.

The following is an example of how to configure an LACP/802.3ad bond with XenServer 5.5 and a Cisco 3750 switch running IOS.  


First, configure your switchports for an LACP etherchannel.  Note: in this case, we are making the etherchannel/Bond a trunk port which will only allow VLANs 2 - 11.  We have two NICs from our XenServer that make up the bond0 interface, plugged int Gi1/0/1 and Gi2/0/1.


// Configure the EtherChannel first

interface Port-channel1
 description Etherchannel Team for XenServer1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 2
 switchport trunk allowed vlan 2-11
 switchport mode trunk
 spanning-tree portfast
 spanning-tree bpduguard enable
!

// now configure the individual switchports

interface GigabitEthernet1/0/1
 description XenServer1 - eth0
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 2
 switchport trunk allowed vlan 2-11
 switchport mode trunk
 speed 1000
 duplex full
 channel-protocol lacp
 channel-group 2 mode active
!

interface GigabitEthernet2/0/1
 description XenServer1 - eth1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 2
 switchport trunk allowed vlan 2-11
 switchport mode trunk
 speed 1000
 duplex full
 channel-protocol lacp
 channel-group 2 mode active
!

// now make sure things are configured properly:
show etherchannel 2 summary

// this should output something like the following:
...

Number of channel-groups in use: 4
Number of aggregators:           4

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
2      Po2(SU)         LACP      Gi1/0/1(P) Gi2/0/1(P)

// Note, you may see the two nics in (I) instead of (P) until you configure and reboot your XenServer



Now, make the following changes to your XenServer host.

1. Create a Bonded interface from XenCenter.  Select your server, then the NICs tab and choose 'Create Bond'.  Add the PIFs you want

2.  The Citrix recommended option.  Run the following command on the XenServer host console.  Make sure to use the UUID of the Bonded interface.
    xe pif-param-set uuid= other-config:bond-mode=802.3ad
    (note: to find your PIF's uuid: xe pif-list host-uuid= )
    (note: to find your Host's uuid: xe host-list )


2a. Some of our users have reported problems getting the offical recommendation working, and suggested an alternate method:

On the XenServer host, edit the following file:
    /opt/xensource/libexec/interface-reconfigure

find the following line:
        "mode":   "balance-slb",
and replace it with
        "mode":   "802.3ad",

Then save and restart the XenServer.  This will change the default bonding method for all Xen bonds to "802.3ad".


3. To make sure things are working as expected, run (replace X with your bond number):
    cat /proc/net/bonding/bondX

    You should see something similar to this:
Ethernet Channel Bonding Driver: v3.1.2 (January 20, 2007)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 31000
Down Delay (ms): 200

802.3ad info
LACP rate: slow
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 2
        Partner Mac Address:

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr:
Aggregator ID: 1

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr:
Aggregator ID: 1


4.  Test and verify.  At this point, your new team should be up and functional.