Xen, RHEL, VLANs, NICs, bridges and virtual interfaces

Based on what I’ve read this must have been a pickle for a lot more people than just me. If you’re looking for a virtualization solution but are not keen on shelling out thousands of dollars to VMWare you are gonna explore Xen. It looks nice, comes cheap and should be very flexible… Yeah right! I got myself a RHEL5 license that comes with Xen 4 built-in and got started.

First of all there’s this idea that Xen actually supports VLANs. While this might be true in away, you’ll have a *very* hard time to get this to work. As Mike Neir points out, VLAN interfaces that are created on the Xen host are broken down when Xen is launched. Because they cannot be bridged correctly by Xen you end up with a fictional bridge that won’t function. Note that he did got it to work as described on this wiki but the script is still unstable on shutdown. Christopher also made a script that is supposed to do the trick, it is explained in more detail by a Red Hat employee in this pdf
I eventually gave up and suggest you do the same for now. VMWare is much more mature in this area but comes at a hefty price tag. Still, imagine you do want to use Xen and have multiple guests that you want to connect to different networks there are other solutions at hand. In the end I settled for a solution where I bridge every Xen guest to 1 or more NIC’s which seems to work very well for now. This is how I did it.
First of all it helps to understand how Xen works. Basically Xen tears down the existing network interfaces (eth0, eth1 and so on) and binds them to itself (also referred to as Dom0) in the form of virtual interfaces which are in turn bridged to the interfaces in the guests. This is all pretty well documented here so I won’t explain this again.
What we do need to know is how to set this all up. It took me a long while before I figured it out but here are some basic guidelines.
1) Do not evn attempt to setup a new host with the correct interfaces during installation. Xen does its magic through a ‘default’ interface (the only option you can select when setting up a host anyway) that bridges to the primary NIC of the Xen host. Make sure you have a install source ready and run through the installation process (if it asks to use DHCP on the interface during installation just say yes, it will continue anyway and find your networked installation resource anyway). Afterwards it’s much easier to change the NIC as I’ll explain below.
2) When you finished your installation and have a running Xen host it’s time to prepare to add the correct NIC to your host. However we can’t do that before we make some adjustments to the Xen networking setup. First of all make sure that your NIC doesn’t have an IP address set, I got several IP conflicts because the ARP table on your LAN will not release quickly enough, thus disabling your interface. You can set the IP address in the host (and the correct gateway and so on) which is what you want anyway. I set my NIC on inactive on launch. Now here comes the tricky part. In the official Red Hat manual (get it here) there’s a good explanation from page 111 and onwards. Basically you divert Xen from using its default script (called network-bridge) and instruct it to call it script multiple times for every NIC we want to use in Xen. This is done with the example script (called network-xen-multi-bridge) in the manual. Feel free to add 2 more NIC’s to the example script, my version of RHEL did not allow me to use more than 4 NIC’s in total. This would look something like this:
$script start vifnum=3 bridge=xenbr3 netdev=eth3
$script start vifnum=2 bridge=xenbr2 netdev=eth1
$script start vifnum=1 bridge=xenbr1 netdev=dev29392
$script start vifnum=0 bridge=xenbr0 netdev=eth0 
vifnum is the number of the internal interface, bridge is the value you’ll use in the configuration file of your guest and netdev is the value of the physical device in your pc (I added a NIC later, hence the weird dev29392 name).
3) With this information at hand you can do as the manual says. Copy network-bridge to network-bridge.xen, create the network-xen-multi-bridge script and edit xend-config.sxp. This is where the manual makes an ambiguous note about editing the script. Just make sure (network-script network-xen-multi-bridge) is printed somewhere in the file and that other lines starting with (network-script… are commented out. Do make sure that you chmod +x the network-xen-multi-bridge script, otherwise it won’t load on SELinux setups!
4) Now that we instructed Xen to create 4 NICs we can start handing them out to our hosts. Edit your host configuration file (located in /etc/xen) and find this line:
vif = [ “mac=00:16:3e:1b:b2:a4,bridge=xenbr3” ]
You can just replace the text after bridge= with the bridge variable that belongs to the NIC that you want to use as I did here. In my example eth3 will be used.
5) If you want to use 2 NICs you’re in for a treat. If you do as the manual says you should edit your config file so it looks like this:
vif = [ “mac=00:16:3e:5a:52:be,bridge=xenbr1,script=vif-bridge”,”mac=00:16:3e:79:a5:18,bridge=xenbr2,script=vif-bridge” ]
I don’t know if you need to add the script=vif-bridge part, it should be automatically called for every bridge you create but you can never be thorough enough. What you will find out soon enough is that the 2nd NIC won’t show up in your host. This puzzled me but you’ll just need to add it to /etc/modprobe.conf manually. Add ‘eth1 xennet’ and you should see it as a new hardware device in your network configuration in your guest that you can use.
Hope this helps and a big thanks to the people out there who took the time to write down their stories so I could compile this report.

Comments are closed.

%d bloggers like this: