如何给KVM上的VM配置VLAN?
- 2014.07.16
最近在给KVM上的VM配置vlan属性时，发现报文从KVM host发出去时， VLAN tag中的COS值被重置为0 （KVM 上的vm发出的报文为带VLAN tag，且cos值非0，但是报文被发出KVM host时，cos值被置为0。） 原因是因为KVM host 会将从vm收到的tag剥掉并重新打上vlan tag，数据流量虽通，但cos值却被修改为默认。
如何配置才能使KVM host保持vm本身发出报文的vlan 属性呢， 请参考如下配置
正确配置如下
# vconfigadd virbr2 2
# ifconfig virbr2.2 up
# brctladd if virbr2 eth0
# brctladd if virbr2 vnet1
# brctl show
bridge name bridge id STP enabled interfaces
virbr2 8000.001018d5c740 yes eth0
virbr2-nic
vnet1
ps:
eth0 is physical uplink port;
vnet1 is nic of vm, packets from vm carriy vlan id 2;
默认情况下，若不使用vlan， 直接将vnet1 和 eth0 绑到vbridge2即可；
具体细节可参考如下blog：
http://blog.davidvassallo.me/2012/05/05/kvm-brctl-in-linux-bringing-vlans-to-the-guests/
----------------------------------------------------------------------------
KVM& BRCTL IN LINUX – BRINGING VLANS TO THE GUESTS
Irecently had the opportunity to setup a KVM machine running on a Centosmachine. Tools such as Virtual Machine Manager (VMM) helpimmensely
in the provisioning and administration of virtual machine guestswithin KVM, and bring the KVM solution on par with other solutions such as Xen,VMWare and VirtualBox.
Networkingis one of my main tasks when setting up such an environment, and my particularscenario presented a bit of a challenge. Most articles regarding KVM and VLANsdeal with having the guests in “access” mode – that is they
can only access asingle VLAN. In my particular scenario (a virtual guest hosting a pfsenseinstall) I needed to preserve the VLAN tagging across the virtual bridge, inother words, having the guest in “trunking” mode, making it vlan-aware.
Thisarticle explores my understanding of the whole setup above using linux’sinbuiltuml-tools, a.k.a. brctl. I will go over the “normal”access mode that is presented in most articles, as well as
a way to put theguests in trunking mode, which is very much less well documented on the web.When reading through the article pay particular attention to the order of theindividual components that make up the solution, as when troubleshooting thiswill be invaluable
(hence my many colorful, if inept, diagrams in this article.
Ifound some very good documentation on several sites, though it took a while tofind what I needed, the following was of particular interest:
Iwill be using CentOS rather than Ubuntu in my article. However, do make note ofthe following settings:
net.bridge.bridge-nf-call-ip6tables= 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
These lines disable any filtering of frames on the bridge devices.
Theabove are found under /proc/net/bridge in Centos
STARTINGPOINT.
Ourstarting point will be simply bridging all physical and virtual interfacestogether. Throughout this article we will be discussing 4 main components:
-The physical network interface (eth0)
-Two virtual network interfaces, attached to two virtual guests (vNIC1 andvNIC2)
-A virtual bridge (BR0)
Wewould be aiming for something along the lines of:
Atthis stage, the objective is to get eth0, vNIC1 and vNIC2 talking together. Iwont focus on creating the actual KVM guests, plenty of documentation elsewhereon how to do that. We’ll keep our focus on the bridge, BR0. Creating
the bridgeis simple as is adding the member interfaces:
brctl add br0
brctl addif br0 eth0
brctl addif br0 vNIC1
brctl addif br0 vNIC2
Withthis setup, all three member interfaces can communicate together, like so:
ADDINGVLANS TO THE MIX – THE USUAL GUEST ACCESS MODE
Beforecontinuing further, from this stage onwards make sure to have the 8021q moduleloaded in the kernel. At this stage, this is what we’re end up with:
It’sa busy diagram so I’ll take some of your time to explain it. the physicalinterface, eth0, will be subdivided into vlan subinterfaces using the vconfig command.Tagged and untagged (native) traffic will enter
on eth0. If a subinterface isdefined for a particular vlan, that traffic is stripped of it’s vlan header,and presented untagged to the bridge. Any virtual machines which should haveaccess to this vlan should be members of this bridge. So to achieve the abovewe
first define the two eth0 vlan subinterfaces, for vlan 1 (eth0.1) and vlan 5(eth0.5):
vconfig add eth0 1
vconfig add th0 5
ifconfig eth0.1 up
ifconfig eth0.5 up
Youcan check that the vlans have been correctly setup by running the command:
cat /proc/net/vlan/conf
Nextwe have the two virtual bridges, one for vlan1 traffic (BR01) and the other forvlan5 (BR05):
brctl add br01
brctl add br05
Wethen combine place the appropriate interfaces into the correct bridge:
brctl addif br01 eth0.1
brctl addif br01 vNIC1
brctl addif br05 eth0.5
brctl addif br05 vNIC2
Afterbringing up the interface, you should have KVM GUEST 1 as a member of vlan 1,and KVM GUEST 2 as a member of vlan 2
HOMERUN:GOING TO GUEST TRUNK MODE
Important,the method presented above, and the method about to be presented are mutuallyexclusive…. you must choose between one or the other. Having them both willgive unexpected results (as I learnt the hard way). See the troubleshootingsection
below for some details.
We’llbe aiming for something like this:
So,to explain the above: Tagged traffic comes through on eth0, which doesnot have any vlan subinterfaces defined as before. Traffic hits thevirtual bridge, which is divided into vlan subinterfaces using vconfig.Basically,
the vlan subinterfaces are defined directly on the bridge ratherthan on eth0.
Thedifference is that when subinterfaces are defined on eth0, as notedpreviously Linux will strip the vlan tag, but when defined on the bridge, thevlan tags are kept. The vNICs are both members of the bridge, with the resultthat
the tagged traffic is presented directly to them, with the VLAN taggingintact Notice that if the bridge does not have a particular vlansubinterface (eg vlan 8 in my diagram above – there is no corresponding br0.8)that vlan traffic will be dropped.
Toachieve the above first define the bridge:
brctladd br0
thenwe define the vlan subinterfaces on the bridge:
vconfigadd br0 1
vconfigadd br0 5
ifconfigbr0.1 up
ifconfigbr0.5 up
andwe finally tie it all together
brctladd if br0 vNIC1
brctladd if br0 vNIC2
Andnow KVM GUEST 1 and 2 will be able to see tagged traffic from vlan 1 and 5. Ofcourse the guest must be able to terminate the vlans themselves as they are nowvlan aware.
TROUBLESHOOTINGNOTES
Whiledoing the above, if you run into problems, keep in mind the following:
1.Make sure all interfaces are up. Running ifconfig should showall interfaces, bridges and subinterfaces as “UP”
2.Traffic flow. Refer to the above diagrams. Though I only show the incomingtraffic path (from the physical world to the VMs) in my diagrams, the outboundtraffic flow (from the VMs to the physical world) is exactly the same, but
inreverse.
Thisis essential so as to know where to run packet captures and where to know whatis going wrong.
Togive a real example of what happened in my case… I had defined all the above asin the second scenario presented above, but neglected to remove all the eth0vlan subinterfaces that I had left there as a result of testing the firstscenario.
This resulted in tagged traffic leaving the VM, hitting the bridgecorrectly, and being sent out the physical world through eth0 correctly. But onit’s return, since the traffic was tagged and the kernel had eth0.1 defined, itwas attempting to use those subinterfaces,
and removing the vlan tags. Removingthe eth0 subinterfaces meant that the tagged traffic could hit the bridgedirectly and the tagging was kept intact
3.TCPDUMP… when dumping vlan traffic remember that by default tcpdump will onlymonitor untagged traffic. To monitor tagged traffic from, say, vlan 5, you’dneed:
tcpdump –i eth0 vlan5
----------------------------------------------------------------------------