From Fedora Project Wiki
No edit summary
No edit summary
 
Line 1: Line 1:
Libvirt, bridging and bonding - a primer.
= Libvirt, bridging and bonding - a primer =


Recently, Adam Williamson wrote up a nice long [http://www.happyassassin.net/2014/07/23/bridged-networking-for-libvirt-with-networkmanager-2014-fedora-21/ blog post] on getting bridging to work with the current state of Fedora. His write up is deficient in two areas - he uses the GUI to configure some of it, and he does not address bonding (which is generally only required for a server, anyhow). So, I dove in to solve this problem for a headless server that is doing some virtualization on the side. We will be using nmcli to configure our bond, bridge, and physical ethernet devices. On the example system, I also have a management interface for out-of-band access to the system.
== Introduction ==
Recently, Adam Williamson wrote up a nice long [http://www.happyassassin.net/2014/07/23/bridged-networking-for-libvirt-with-networkmanager-2014-fedora-21/ blog post] on getting bridging to work with the current state of Fedora. His write up is targeted at desktop users, so he didn't cover two areas - he uses the GUI to configure some of it, and he does not address bonding (which is generally only required for a server, anyhow). So, I dove in to solve this problem for a headless server that is doing some virtualization on the side. We will be using nmcli to configure our bond, bridge, and physical ethernet devices. On the example system, I also have a management interface for out-of-band access to the system.


== Procedure ==
=== User: root ===
First, you will need to be root for this - it just makes it that much easier.
First, you will need to be root for this - it just makes it that much easier.
<pre>
<pre>
Line 8: Line 12:
</pre>
</pre>


When I started down this path, my networking was all fubbed up - nothing really was working properly.
=== Example non-working configuration ===
 
When I started down this path, my networking was severely mis-configured - nothing really was working properly. It had reached this point by using the intuitive tools provided by [http://virt-manager.org/ Virtual Machine Manager] (VMM) and <code>virsh</code>. Once I worked through the "suggested" process with those tools, I ended up with a broken network.
<pre>
<pre>
# nmcli con show conf
# nmcli con show conf
Line 62: Line 68:


As you can see, the only IP address I had was my management interface, and libvirt didn't see any interfaces at all. So, I decided to start from scratch.
As you can see, the only IP address I had was my management interface, and libvirt didn't see any interfaces at all. So, I decided to start from scratch.
=== Reset ===
The first step is to remove all configuration from the system.


<pre>
<pre>
Line 75: Line 84:


# nmcli con delete 6338b494-9e37-4677-b3e4-837938cfd860
# nmcli con delete 6338b494-9e37-4677-b3e4-837938cfd860
# nmcli con delete 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85
# nmcli con delete 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85
# nmcli con delete ad33d8b0-1f7b-cab9-9447-ba07f855b143
# nmcli con delete ad33d8b0-1f7b-cab9-9447-ba07f855b143
# nmcli con delete 5dd47203-fffb-671a-4fd0-4cff98347a3b
# nmcli con delete 5dd47203-fffb-671a-4fd0-4cff98347a3b
# nmcli con delete 1dad842d-1912-ef5a-a43a-bc238fb267e7
# nmcli con delete 1dad842d-1912-ef5a-a43a-bc238fb267e7
</pre>
That deleted all of my current network configuration except for my management interface.
=== Verify starting point ===
<pre>
# nmcli con show conf  
# nmcli con show conf  
NAME                UUID                                  TYPE            TIMESTAMP-REAL                   
NAME                UUID                                  TYPE            TIMESTAMP-REAL                   
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:24:46 PM CDT  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:24:46 PM CDT  
</pre>
</pre>
So, now I have a clean slate to work with. From here, I want to verify that my management interface is configured properly and won't interfere with the real task at hand:
So, now I have a clean slate to work with. From here, I want to verify that my management interface is configured properly and won't interfere with the real task at hand:
<pre>
<pre>
Line 126: Line 146:
the key thing we were looking for there were <code>ipv4.never-default: yes</code> and <code>ipv4.may-fail: yes</code>. This ensures that the system does not try to use the management network as it's default route, and that if the connection isn't present it won't affect the system.
the key thing we were looking for there were <code>ipv4.never-default: yes</code> and <code>ipv4.may-fail: yes</code>. This ensures that the system does not try to use the management network as it's default route, and that if the connection isn't present it won't affect the system.


Next, reboot (if possible) to clean every thing up. Here's where it gets tricky. The key thing to remember here is that the Physical interfaces are subordinate to the bond, which is subordinate to the bridge. So, we configure the bridge first:
Finally, reboot (if possible) to clean every thing up.  


<pre>
== Configuring the interfaces ==
# nmcli con add con-name br0 type bridge
This is where it gets tricky. The key thing to remember here is that the physical interfaces are subordinate to the bond, which is subordinate to the bridge. So, we configure the bridge first.
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) successfully added.
{{admon/tip|Interface Chain|System -> bridge -> bond -> physical -> network}}
# nmcli con edit ed441360-b0b6-49f2-a2c8-599c4d3d4b92


===| nmcli interactive connection editor |===
=== Bridge interface ===
The bridge is the main interface to the system. This is what will hold the IP information and be presented to the kernel network stack and libvirt.


Editing existing 'bridge' connection: 'ed441360-b0b6-49f2-a2c8-599c4d3d4b92'
{{admon/note|nmcli usage|I am showing everything on one line here for brevity's sake, but you can also stop after defining the con-name or if-name, then entering the nmcli shell to finish the configuration}}


Type 'help' or '?' for available commands.
<pre>
Type 'describe [<setting>.<prop>]' for detailed property description.
# nmcli con add con-name br0 ifname br0 type bridge autoconnect yes stp off ip4 172.16.4.3/24 gw4 172.16.4.1
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) successfully added.


You may edit the following settings: connection, bridge, 802-3-ethernet (ethernet), ipv4, ipv6
# nmcli con show ed441360-b0b6-49f2-a2c8-599c4d3d4b92
nmcli> print
===============================================================================
                              Connection details
===============================================================================
connection.id:                          br0
connection.id:                          br0
connection.uuid:                        ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.uuid:                        ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.interface-name:              nm-bridge
connection.interface-name:              br0
connection.type:                        bridge
connection.type:                        bridge
connection.autoconnect:                yes
connection.autoconnect:                yes
Line 158: Line 175:
connection.secondaries:                 
connection.secondaries:                 
connection.gateway-ping-timeout:        0
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
ipv4.method:                            manual
bridge.interface-name:                  nm-bridge
ipv4.dns:                              172.16.4.2
bridge.stp:                            yes
ipv4.dns-search:                       
ipv4.addresses:                        { ip = 172.16.4.3/24, gw = 172.16.4.1 }
ipv4.routes:                           
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                  no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                    --
ipv4.never-default:                    no
ipv4.may-fail:                          yes
bridge.interface-name:                  br0
bridge.stp:                            no
bridge.priority:                        128
bridge.priority:                        128
bridge.forward-delay:                  15
bridge.forward-delay:                  15
Line 166: Line 194:
bridge.max-age:                        20
bridge.max-age:                        20
bridge.ageing-time:                    300
bridge.ageing-time:                    300
-------------------------------------------------------------------------------
nmcli> set bridge.stp no
nmcli> save
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) sucessfully saved.
nmcli> quit
</pre>
</pre>
I don't have the infrastructure to require STP, and it causes problems with my HP switches, so I turn it off. Also, note the <code>bridge.interface-name</code> - you'll need this for the next step.
 
=== Bond Interface ===
Bonding is used on servers to provide a higher bandwidth, fault tolerance, and/or load balancing. To get the benefit of the faster bandwidth, a switch that supports 802.3AD Link Aggregation is required. To take advantage of the fault tolerance or certain load balancing options, specific hardware is not required. For this example, I am setting up an 802.3AD aggregate link to my switch in order to establish a 2Gbps circuit to the server.
 
<pre>
<pre>
# nmcli con add con-name bond0 type bond
# nmcli con add con-name bond0 ifname bond0 type bond autoconnect yes mode 802.3ad miimon 100
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) successfully added.
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) successfully added.
</pre>
Now the command line options for the bond configuration do not allow me to set it as a bridged interface, so we will need to drop into the nmcli shell to finish up the config.
<pre>
# nmcli con edit 58fe56dd-a30b-483a-9e35-3727ccc6ab0e
# nmcli con edit 58fe56dd-a30b-483a-9e35-3727ccc6ab0e


Line 192: Line 222:
connection.id:                          bond0
connection.id:                          bond0
connection.uuid:                        58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.uuid:                        58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name:              nm-bond
connection.interface-name:              bond0
connection.type:                        bond
connection.type:                        bond
connection.autoconnect:                yes
connection.autoconnect:                yes
Line 204: Line 234:
connection.gateway-ping-timeout:        0
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
bond.interface-name:                    nm-bond
bond.interface-name:                    bond0
bond.options:                          miimon=100,mode=balance-rr
bond.options:                          miimon=100,mode=balance-rr
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
nmcli> set connection.master nm-bridge
nmcli> set connection.master br0
nmcli> set connection.slave-type bridge
nmcli> set connection.slave-type bridge
nmcli> set bond.options miimon=100,mode=802.3ad
nmcli> print
===============================================================================
                              Connection details
===============================================================================
connection.id:                          bond0
connection.uuid:                        58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name:              nm-bond
connection.type:                        bond
connection.autoconnect:                yes
connection.timestamp:                  0
connection.read-only:                  no
connection.permissions:               
connection.zone:                        --
connection.master:                      nm-bridge
connection.slave-type:                  bridge
connection.secondaries:               
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
bond.interface-name:                    nm-bond
bond.options:                          miimon=100,mode=802.3ad
-------------------------------------------------------------------------------
nmcli> save
nmcli> save
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) sucessfully saved.
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) sucessfully saved.
nmcli> quit
nmcli> quit
</pre>
</pre>
You now have a bond inteface bridged with your bridge interface. Now we need to add the physical interfaces to the bond.
 
You now have a bond interface bridged with the bridge interface we set up previously. Now we need to add the physical interfaces to the bond.
 
=== Physical Interfaces ===
Bonded interfaces are pseudo interfaces just like bridges - they need to have physical interfaces slaved to them in order to be operational. Here we will do this all from the command line. You need to know what device name udev gave the interfaces, the easiest way to do this is with the old <code>ifconfig</code> command - it will spit out the udev device names as well as any configuration already applied. Using the udev device name of your physical interfaces, add them as slaves to the bond0 interface we just created.
 
<pre>
<pre>
# nmcli con add con-name eth0 ifname em1 type bond-slave master nm-bond
# nmcli con add con-name eth0 ifname em1 type bond-slave master bond0
Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added.
Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added.


# nmcli con add con-name eth1 ifname p4p1 type bond-slave master nm-bond
# nmcli con add con-name eth1 ifname p4p1 type bond-slave master bond0
Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.
Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.
</pre>


== Verification and Activation ==
The configuration should be complete at this point. To verify our setup, we will use <code>nmcli</code>, <code>brctl</code>, <code>ip address</code>, and <virsh>.
<pre>
# nmcli con show conf
# nmcli con show conf
NAME  UUID                                  TYPE            TIMESTAMP-REAL                   
NAME  UUID                                  TYPE            TIMESTAMP-REAL                   
Line 250: Line 267:
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            never                           
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            never                           
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          never                           
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          never                           
</pre>
Now we need to add the IP address to the bridge interface.
<pre>
# nmcli con edit ed441360-b0b6-49f2-a2c8-599c4d3d4b92
===| nmcli interactive connection editor |===


Editing existing 'bridge' connection: 'ed441360-b0b6-49f2-a2c8-599c4d3d4b92'
Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.
You may edit the following settings: connection, bridge, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> goto ipv4
You may edit the following properties: method, dns, dns-search, addresses, routes, ignore-auto-routes, ignore-auto-dns, dhcp-client-id, dhcp-send-hostname, dhcp-hostname, never-default, may-fail
nmcli ipv4> set method manual
nmcli ipv4> set addresses 172.16.4.3/24 172.16.4.1
nmcli ipv4> set dns 172.16.4.2
nmcli ipv4> print
['ipv4' setting values]
ipv4.method:                            manual
ipv4.dns:                              172.16.4.2
ipv4.dns-search:                       
ipv4.addresses:                        { ip = 172.16.4.3/24, gw = 172.16.4.1 }
ipv4.routes:                           
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                  no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                    --
ipv4.never-default:                    no
ipv4.may-fail:                          yes
nmcli ipv4> save
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) sucessfully saved.
nmcli ipv4> quit
</pre>
Almost there, now we need to turn them on.
<pre>
# nmcli con up br0
# nmcli con up br0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)
Line 307: Line 287:
eth0  c58e2397-2a9d-409f-b3b9-f6667fd44274  em1        no      no  /org/freedesktop/NetworkManager/Devices/6  
eth0  c58e2397-2a9d-409f-b3b9-f6667fd44274  em1        no      no  /org/freedesktop/NetworkManager/Devices/6  
eth1  56e6e53f-5039-4e10-bb3f-8b4b66d41d31  p4p1      no      no  /org/freedesktop/NetworkManager/Devices/6  
eth1  56e6e53f-5039-4e10-bb3f-8b4b66d41d31  p4p1      no      no  /org/freedesktop/NetworkManager/Devices/6  
[root@hermes ~]# nmcli con show conf
NAME  UUID                                  TYPE            TIMESTAMP-REAL                 
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:40:20 PM CDT
eth1  56e6e53f-5039-4e10-bb3f-8b4b66d41d31  802-3-ethernet  Mon 04 Aug 2014 06:43:44 PM CDT
eth0  c58e2397-2a9d-409f-b3b9-f6667fd44274  802-3-ethernet  Mon 04 Aug 2014 06:42:46 PM CDT
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            Mon 04 Aug 2014 06:41:30 PM CDT
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          Mon 04 Aug 2014 06:40:20 PM CDT
</pre>
At this point, I reinstalled the virtualization group and rebooted to make sure everything was up to par.
<pre>
# yum group install Virtualization
</pre>
Now, for the verification stage, to make sure everything is as we desired.
<pre>
# nmcli con show conf
NAME  UUID                                  TYPE            TIMESTAMP-REAL                 
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            Mon 04 Aug 2014 06:47:01 PM CDT
eth0  c58e2397-2a9d-409f-b3b9-f6667fd44274  802-3-ethernet  Mon 04 Aug 2014 06:47:03 PM CDT
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:47:01 PM CDT
eth1  56e6e53f-5039-4e10-bb3f-8b4b66d41d31  802-3-ethernet  Mon 04 Aug 2014 06:47:04 PM CDT
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          Mon 04 Aug 2014 06:47:01 PM CDT
# nmcli con show act
NAME  UUID                                  DEVICES    DEFAULT  VPN  MASTER-PATH                             
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  nm-bridge  yes      no  --                                       
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  nm-bond    no      no  /org/freedesktop/NetworkManager/Devices/2
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  p2p1      no      no  --                                       
eth0  c58e2397-2a9d-409f-b3b9-f6667fd44274  em1        no      no  /org/freedesktop/NetworkManager/Devices/1
eth1  56e6e53f-5039-4e10-bb3f-8b4b66d41d31  p4p1      no      no  /org/freedesktop/NetworkManager/Devices/1


# ip addr
# ip addr
Line 355: Line 305:
4: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
4: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
     link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
     link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
5: nm-bond: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master nm-bridge state UP group default  
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master nm-bridge state UP group default  
     link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
     link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
     inet6 fe80::a8e4:e3ff:fe2b:12af/64 scope link  
     inet6 fe80::a8e4:e3ff:fe2b:12af/64 scope link  
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
6: nm-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default  
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default  
     link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
     link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
     inet 172.16.4.3/24 brd 172.16.4.255 scope global nm-bridge
     inet 172.16.4.3/24 brd 172.16.4.255 scope global nm-bridge
Line 369: Line 319:
     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
       valid_lft forever preferred_lft forever
# brctl show
bridge name    bridge id              STP enabled    interfaces
br0            0080.0023aeab2054      no              bond0
virbr0          8000.000000000000      yes


# virsh iface-list
# virsh iface-list
Line 374: Line 329:
--------------------------------------------
--------------------------------------------
lo                  active    00:00:00:00:00:00
lo                  active    00:00:00:00:00:00
nm-bridge            active    00:23:ae:ab:20:54
br0                  active    00:23:ae:ab:20:54
</pre>
</pre>


Libvirt is now bridged with the bonded interfaces to the network, 2GB pathway to and from the virtual machines to be built.
Libvirt is now bridged with the bonded interfaces to the network, 2GB pathway to and from the virtual machines to be built.

Latest revision as of 20:31, 5 August 2014

Libvirt, bridging and bonding - a primer

Introduction

Recently, Adam Williamson wrote up a nice long blog post on getting bridging to work with the current state of Fedora. His write up is targeted at desktop users, so he didn't cover two areas - he uses the GUI to configure some of it, and he does not address bonding (which is generally only required for a server, anyhow). So, I dove in to solve this problem for a headless server that is doing some virtualization on the side. We will be using nmcli to configure our bond, bridge, and physical ethernet devices. On the example system, I also have a management interface for out-of-band access to the system.

Procedure

User: root

First, you will need to be root for this - it just makes it that much easier. 

$ sudo su -

Example non-working configuration

When I started down this path, my networking was severely mis-configured - nothing really was working properly. It had reached this point by using the intuitive tools provided by Virtual Machine Manager (VMM) and virsh. Once I worked through the "suggested" process with those tools, I ended up with a broken network. 

# nmcli con show conf
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:04:46 PM CDT 
System em1          1dad842d-1912-ef5a-a43a-bc238fb267e7  802-3-ethernet  never                           
System p4p1         5dd47203-fffb-671a-4fd0-4cff98347a3b  802-3-ethernet  never                           
System bond0        ad33d8b0-1f7b-cab9-9447-ba07f855b143  802-3-ethernet  never                           
Wired connection 2  f5984df2-83d8-42e9-bd8c-7830680cb708  802-3-ethernet  Mon 04 Aug 2014 05:39:46 PM CDT 
Bridge br1          2ee981ca-5ff4-4f9b-03fe-32879aa3dc85  bridge          Mon 04 Aug 2014 05:47:56 PM CDT 
kvm-br0             6338b494-9e37-4677-b3e4-837938cfd860  bridge          Mon 04 Aug 2014 04:17:38 PM CDT 

# brctl show
bridge name     bridge id               STP enabled     interfaces
nm-kvm-br0              8000.000000000000       no
virbr0          8000.000000000000       yes

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:76ff:fef4:5c38/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::223:aeff:feab:2054/64 scope link 
       valid_lft forever preferred_lft forever
4: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1b:21:ab:d6:a1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::21b:21ff:feab:d6a1/64 scope link 
       valid_lft forever preferred_lft forever
5: nm-kvm-br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether 82:2c:4a:76:7f:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::802c:4aff:fe76:7f16/64 scope link 
       valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 12:27:18:b3:35:1d brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

# virsh iface-list 
Name                 State      MAC Address
--------------------------------------------
lo                   active     00:00:00:00:00:00

As you can see, the only IP address I had was my management interface, and libvirt didn't see any interfaces at all. So, I decided to start from scratch.

Reset

The first step is to remove all configuration from the system. 

# nmcli con show conf
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:19:46 PM CDT 
System em1          1dad842d-1912-ef5a-a43a-bc238fb267e7  802-3-ethernet  never                           
System p4p1         5dd47203-fffb-671a-4fd0-4cff98347a3b  802-3-ethernet  never                           
System bond0        ad33d8b0-1f7b-cab9-9447-ba07f855b143  802-3-ethernet  never                           
Wired connection 2  f5984df2-83d8-42e9-bd8c-7830680cb708  802-3-ethernet  Mon 04 Aug 2014 05:39:46 PM CDT 
Bridge br1          2ee981ca-5ff4-4f9b-03fe-32879aa3dc85  bridge          Mon 04 Aug 2014 05:47:56 PM CDT 
kvm-br0             6338b494-9e37-4677-b3e4-837938cfd860  bridge          Mon 04 Aug 2014 04:17:38 PM CDT 

# nmcli con delete 6338b494-9e37-4677-b3e4-837938cfd860

# nmcli con delete 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85

# nmcli con delete ad33d8b0-1f7b-cab9-9447-ba07f855b143

# nmcli con delete 5dd47203-fffb-671a-4fd0-4cff98347a3b

# nmcli con delete 1dad842d-1912-ef5a-a43a-bc238fb267e7

That deleted all of my current network configuration except for my management interface.

Verify starting point

# nmcli con show conf 
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:24:46 PM CDT

So, now I have a clean slate to work with. From here, I want to verify that my management interface is configured properly and won't interfere with the real task at hand: 

# nmcli con show conf 56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.id:                          Wired connection 1
connection.uuid:                        56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.interface-name:              --
connection.type:                        802-3-ethernet
connection.autoconnect:                 yes
connection.timestamp:                   1407194686
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          yes
802-3-ethernet.mac-address:             00:04:76:F4:5C:38
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.mac-address-blacklist:   
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            
ipv4.method:                            manual
ipv4.dns:                               
ipv4.dns-search:                        
ipv4.addresses:                         { ip = 10.0.0.3/24, gw = 0.0.0.0 }
ipv4.routes:                            
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.never-default:                     yes
ipv4.may-fail:                          yes

the key thing we were looking for there were ipv4.never-default: yes and ipv4.may-fail: yes. This ensures that the system does not try to use the management network as it's default route, and that if the connection isn't present it won't affect the system.

Finally, reboot (if possible) to clean every thing up.

Configuring the interfaces

This is where it gets tricky. The key thing to remember here is that the physical interfaces are subordinate to the bond, which is subordinate to the bridge. So, we configure the bridge first.

Idea.png
Interface Chain
System -> bridge -> bond -> physical -> network

Bridge interface

The bridge is the main interface to the system. This is what will hold the IP information and be presented to the kernel network stack and libvirt.

Note.png
nmcli usage
I am showing everything on one line here for brevity's sake, but you can also stop after defining the con-name or if-name, then entering the nmcli shell to finish the configuration
# nmcli con add con-name br0 ifname br0 type bridge autoconnect yes stp off ip4 172.16.4.3/24 gw4 172.16.4.1
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) successfully added.

# nmcli con show ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.id:                          br0
connection.uuid:                        ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.interface-name:              br0
connection.type:                        bridge
connection.autoconnect:                 yes
connection.timestamp:                   0
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
ipv4.method:                            manual
ipv4.dns:                               172.16.4.2
ipv4.dns-search:                        
ipv4.addresses:                         { ip = 172.16.4.3/24, gw = 172.16.4.1 }
ipv4.routes:                            
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.never-default:                     no
ipv4.may-fail:                          yes
bridge.interface-name:                  br0
bridge.stp:                             no
bridge.priority:                        128
bridge.forward-delay:                   15
bridge.hello-time:                      2
bridge.max-age:                         20
bridge.ageing-time:                     300

Bond Interface

Bonding is used on servers to provide a higher bandwidth, fault tolerance, and/or load balancing. To get the benefit of the faster bandwidth, a switch that supports 802.3AD Link Aggregation is required. To take advantage of the fault tolerance or certain load balancing options, specific hardware is not required. For this example, I am setting up an 802.3AD aggregate link to my switch in order to establish a 2Gbps circuit to the server. 

# nmcli con add con-name bond0 ifname bond0 type bond autoconnect yes mode 802.3ad miimon 100
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) successfully added.

Now the command line options for the bond configuration do not allow me to set it as a bridged interface, so we will need to drop into the nmcli shell to finish up the config. 

# nmcli con edit 58fe56dd-a30b-483a-9e35-3727ccc6ab0e

===| nmcli interactive connection editor |===

Editing existing 'bond' connection: '58fe56dd-a30b-483a-9e35-3727ccc6ab0e'

Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.

You may edit the following settings: connection, bond, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> print
===============================================================================
                              Connection details
===============================================================================
connection.id:                          bond0
connection.uuid:                        58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name:              bond0
connection.type:                        bond
connection.autoconnect:                 yes
connection.timestamp:                   0
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
bond.interface-name:                    bond0
bond.options:                           miimon=100,mode=balance-rr
-------------------------------------------------------------------------------
nmcli> set connection.master br0
nmcli> set connection.slave-type bridge
nmcli> save
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) sucessfully saved.
nmcli> quit

You now have a bond interface bridged with the bridge interface we set up previously. Now we need to add the physical interfaces to the bond.

Physical Interfaces

Bonded interfaces are pseudo interfaces just like bridges - they need to have physical interfaces slaved to them in order to be operational. Here we will do this all from the command line. You need to know what device name udev gave the interfaces, the easiest way to do this is with the old ifconfig command - it will spit out the udev device names as well as any configuration already applied. Using the udev device name of your physical interfaces, add them as slaves to the bond0 interface we just created. 

# nmcli con add con-name eth0 ifname em1 type bond-slave master bond0
Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added.

# nmcli con add con-name eth1 ifname p4p1 type bond-slave master bond0
Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.

Verification and Activation

The configuration should be complete at this point. To verify our setup, we will use nmcli, brctl, ip address, and <virsh>. 

# nmcli con show conf
NAME   UUID                                  TYPE            TIMESTAMP-REAL                  
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:35:20 PM CDT 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  802-3-ethernet  never                           
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  802-3-ethernet  never                           
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            never                           
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          never                           

# nmcli con up br0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)

# nmcli con up bond0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)

# nmcli con up eth0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)

# nmcli con up eth1
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/9)

# nmcli con show act
NAME   UUID                                  DEVICES    DEFAULT  VPN  MASTER-PATH                               
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  p2p1       no       no   --                                        
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  nm-bridge  yes      no   --                                        
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  nm-bond    no       no   /org/freedesktop/NetworkManager/Devices/5 
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  em1        no       no   /org/freedesktop/NetworkManager/Devices/6 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  p4p1       no       no   /org/freedesktop/NetworkManager/Devices/6 

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:76ff:fef4:5c38/64 scope link 
       valid_lft forever preferred_lft forever
3: em1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
4: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master nm-bridge state UP group default 
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a8e4:e3ff:fe2b:12af/64 scope link 
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet 172.16.4.3/24 brd 172.16.4.255 scope global nm-bridge
       valid_lft forever preferred_lft forever
    inet6 fe80::5c18:76ff:fef5:e166/64 scope link 
       valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 42:59:71:b6:95:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             0080.0023aeab2054       no              bond0
virbr0          8000.000000000000       yes

# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
lo                   active     00:00:00:00:00:00
br0                  active     00:23:ae:ab:20:54

Libvirt is now bridged with the bonded interfaces to the network, 2GB pathway to and from the virtual machines to be built.

Retrieved from "https://fedoraproject.org/w/index.php?title=User:Dmossor/libvirt-bridge&oldid=383514"