strict warning: Only variables should be passed by reference in /var/www/sites/www.netomata.com/sites/all/themes/clean/template.php on line 126.

VLANs and Bonding and Xen, oh my!

Updated 4 Jun 09 to add missing 'pre-up ip link set vlanXX up' lines in 'vlanXX' entries in /etc/network/interfaces. Oops, sorry; I'm not sure how those lines disappeared between my actual working configs and this blog post! -Brent

Some things seem like they ought to be straightforward:

  • You're using Xen for virtual hosting
  • Your dom0 server has two physical interfaces, connected to two different upstream switches for maximum reliability
  • Your environment includes multiple VLANs
  • You want to set things up so that any given Xen domU guest can be easily attached to any one of those VLANs.

So, what you need to do is:

  1. Bond the dom0's physical ethernet interfaces (eth0 and eth1) into a "bond0" interface
  2. Set up VLANs via the "bond0" interface.
  3. Make those VLANs available to Xen domU guests via virtual bridges, one per VLAN

The hard way (trust me on this): use some variation/modification of Xen's "network-bridge" script to try to set everything up. Good luck. It will take you days, if you can get it working at all.

The easy way: set it all up in the dom0's /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto bond0
iface bond0 inet manual
        slaves eth0 eth1
        # It's important to use active-backup mode when you've got 2 separate
        # upstream switches. The various other bonding modes only work when
        # you're connecting to the SAME upstream switch; they are useful for
        # increasing bandwidth, but not for failover protection against an
        # upstream switch failure (or local interface failure, cabling failure,
        # and the like).
        bond_mode active-backup
        # "bond_miimon 100" checks the CARRIER status of the physical
	# interfaces every 100ms, and switches from the primary to the
	# backup interface if the carrier fails.  So, if the upstream
	# switch hangs but doesn't drop carrier, you're screwed, because
        # bond_miimon won't detect that.
        bond_miimon 100
        # Given that bond_miimon won't detect certain upstream switch
	# failures, it might seem like you want to use bond_arp_ip_target
	# monitoring instead, which purports to make sure you can actually
	# move traffic by sending ARP requests and looking for replies. 
	# Unfortunately, in this configuration, the bond0 interface doesn't
	# actually have an IP address (the vlanNN VLAN bridges that are
	# established below do, but not the bond0 interface itself), so
	# the ARP requests/replies won't work and you can't use
	# bond_arp_ip_target.
        #       bond_arp_ip_target 10.5.1.2 10.5.16.3
        #       bond_arp_interval 100

# Set up the bonding subinterface for VLAN16
auto bond0.16
iface bond0.16 inet manual

# Set up the bonding subinterface for VLAN32
auto bond0.32
iface bond0.32 inet manual

# Set up the bonding subinterface for VLAN48
auto bond0.48
iface bond0.48 inet manual

# Now set up the bridge for VLAN16, attach the bond0.16 interface to it,
# and assign the dom0 host's IP address on this subnet
auto vlan16
iface vlan16 inet static
        address 10.5.16.31
        netmask 255.255.255.0
        broadcast 10.5.16.255
        # Our default route is via this subnet to the router at 10.5.16.1
        gateway 10.5.16.1
        pre-up ip link set bond0.16 down
        pre-up brctl addbr vlan16
        pre-up brctl addif vlan16 bond0.16
        pre-up ip link set bond0.16 up
        pre-up ip link set vlan16 up
        post-down ip link set vlan16 down
        post-down brctl delbr vlan16

# And set up the bridge for VLAN32, attach the bond0.32 interface to it,
# and assign the dom0 host's IP address on this subnet
auto vlan32
iface vlan32 inet static
        address 10.5.32.31
        netmask 255.255.255.0
        broadcast 10.5.32.255
        # no "gateway" line, so no default route via this interface
        pre-up ip link set bond0.32 down
        pre-up brctl addbr vlan32
        pre-up brctl addif vlan32 bond0.32
        pre-up ip link set bond0.32 up
        pre-up ip link set vlan32 up
        post-down ip link set vlan32 down
        post-down brctl delbr vlan32

# Here's how to set up a bridge for a VLAN that the dom0 server does NOT
# have an IP address on.  This enabled you to attach domU guests to this VLAN
# without them being able to contact the dom0 server (and vice versa); the dom0
# server acts strictly as a dumb bridge for this VLAN.
# Note that the last word on the "iface" line is "manual" instead of "static".
auto vlan48
iface vlan48 inet manual
        pre-up ip link set bond0.48 down
        pre-up brctl addbr vlan48
        pre-up brctl addif vlan48 bond0.48
        pre-up ip link set bond0.48 up
        pre-up ip link set vlan48 up
        post-down ip link set vlan48 down
        post-down brctl delbr vlan48

Now, when you're setting up a Xen domU guest, to attach it to a particular VLAN, you simply add a "bridge=vlanXX" parameter to the "vif" line in the guest's .cfg file:

vif         = [ 'ip=10.5.32.99,bridge=vlan32,mac=00:16:3E:69:76:02' ]

The guest will see this as interface "eth0", with the IP address as assigned (make sure it matches what's in the guest's own /etc/network/interfaces file, or that the guest gets via DHCP).

If you want to attach a Xen domU guest to multiple VLANs, simply use multiple "vif" entries, like so:

vif         = [
                'ip=10.5.16.99,bridge=vlan16,mac=00:16:3E:64:1C:A7',
                'ip=10.5.32.99,bridge=vlan32,mac=00:16:3E:64:1C:A8',
              ]

The guest will see these as interfaces "eth0" (on VLAN 16) and "eth1" (on VLAN 32).

Note that you do NOT configure the domU guest itself for VLAN support. As far as the domU guest is concerned, there is no VLAN involved, and it is attached directly to the network in question. It's the same as if the domU guest were a physical machine, and were attached to a Cisco switch with "switchport access vlan XX" and "switchport mode access" set for the port the guest is attached to. The Xen dom0 server takes care of all the VLAN and bonding magic, simplifying the domU configurations.