Features/VHostNet

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
(Created page with '= Enable kernel acceleration for kvm networking = == Summary == Enable kernel acceleration for kvm networking == Owner == * Name: Mst == Current status == * Targeted release:...')
 
(s/-netdevice/-netdev/)
 
(23 intermediate revisions by 4 users not shown)
Line 6: Line 6:
  
 
== Owner ==
 
== Owner ==
* Name: Mst
+
* Name: [[User:Mst|Michael S. Tsirkin]]
  
 
== Current status ==
 
== Current status ==
* Targeted release: [[Releases/12|  Fedora 12]]  
+
* Targeted release: [[Releases/13|  Fedora 13]]  
* Last updated: 2009-06-06                     
+
* Last updated: 2010-01-26
* Percentage of completion: 20%                
+
* Percentage of completion: 100%
 +
** code complete, changes in response to upstream review are being incorporated
  
 
== Detailed Description ==
 
== Detailed Description ==
Line 17: Line 18:
 
vhost net moves the task of converting virtio descriptors to skbs and back
 
vhost net moves the task of converting virtio descriptors to skbs and back
 
from qemu userspace to the kernel driver.                                 
 
from qemu userspace to the kernel driver.                                 
 +
It is activated by using -netdev options (instead of -net)
 +
and adding vhost=on flag.
  
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
  
 
Using a kernel module reduces latency and improves packets per second for small packets.
 
Using a kernel module reduces latency and improves packets per second for small packets.
 
  
 
== Scope ==
 
== Scope ==
  
The work is all upstream in the kernel and qemu.  Guest code is already
+
The work is all upstream in the kernel and qemu.  Guest code is already upstream. Host/qemu work is in progress.  For Fedora 13 will likely have to backport some of it.                                                       
upstream. Host/qemu work is in progress.  For Fedora 12 will likely have to
+
backport some of it.                                                       
+
  
Milestones:
+
=== Milestones Reached ===
  
 
- Guest Kernel:
 
- Guest Kernel:
Line 39: Line 39:
 
   socket polling                     
 
   socket polling                     
 
   virtio transport with copy from/to user
 
   virtio transport with copy from/to user
 
  <- at this point can be used in production the rest are optimizations
 
    we will most likely need                                         
 
 
  mergeable buffers
 
 
   TX credits using destructor (or: poll device status)
 
   TX credits using destructor (or: poll device status)
 
   TSO/GSO                                             
 
   TSO/GSO                                             
  pin memory with get user pages                     
 
 
   profile and tune                                     
 
   profile and tune                                     
  
 
- qemu:
 
- qemu:
 
   MSI-X support in virtio net
 
   MSI-X support in virtio net
  raw sockets support in qemu, promisc mode
 
 
   connect to kernel backend with MSI-X     
 
   connect to kernel backend with MSI-X     
  migration                               
 
 
   PCI interrupts emulation                 
 
   PCI interrupts emulation                 
 
  <- at this point can be used in production
 
    the rest are optimizations we will most likely need
 
 
  programming MAC
 
 
   TSO/GSO         
 
   TSO/GSO         
 
   profile and tune
 
   profile and tune
  
 +
=== In progress ===
 +
- finalize qemu command line
 +
- qemu: migration                               
 +
 +
Code posted, but won't be upstream in time and probably not important enough to backport
 +
  raw sockets support in qemu, promisc mode
 +
 +
Delayed, will likely not make it by F13 ship date
 +
  mergeable buffers
 +
  programming MAC/vlan filtering
  
 
== Test Plan ==
 
== Test Plan ==
 +
 +
Guest:
 +
* WHQL networking tests
  
 
Networking:
 
Networking:
- Various MTU sizes
+
* Various MTU sizes
- Broadcasts, multicasts,
+
* Broadcasts, multicasts,
- Ethtool                 
+
* Ethtool                 
- Latency tests           
+
* Latency tests           
- Bandwidth tests         
+
* Bandwidth tests         
- UDP testing             
+
* UDP testing             
- guest to guest communication
+
* Guest to guest communication
- More types of protocol testing
+
* More types of protocol testing
- guest vlans
+
* Guest vlans
- Tests combination of multiple vnics on the guests
+
* Tests combination of multiple vnics on the guests
- With/without {IP|TCP|UDP} offload
+
* With/without {IP|TCP|UDP} offload
  
 
Virtualization:
 
Virtualization:
- Live migration
+
* Live migration
  
 
Kernel side:
 
Kernel side:
- Load/unload driver
+
* Load/unload driver
  
 
== User Experience ==
 
== User Experience ==
  
Users should see faster networking at least in cases of SRIOV or a dedicated
+
Users should see faster networking at least in cases of SRIOV or a dedicated per-guest network device.
per-guest network device.
+
  
 
== Dependencies ==
 
== Dependencies ==
Line 98: Line 97:
  
 
== Documentation ==
 
== Documentation ==
 +
* vhost net is activated by adding vhost=on to netdev option.
 +
 +
* For non-vhost:                                                                               
 +
 +
<pre>                                                                                               
 +
-net nic,model=virtio,netdev=foo -netdev                                                     
 +
tap,id=foo,ifname=msttap0,script=/home/mst/ifup,downscript=no
 +
</pre>                                 
 +
                                                                                               
 +
* For vhost:                                                                                   
 +
<pre>                                                                                               
 +
-net nic,model=virtio,netdev=foo -netdev                                                     
 +
tap,id=foo,vhost=on,vhostfd=<fd>,ifname=msttap0,script=/home/mst/ifup,downscript=no           
 +
</pre>
 +
                                                                                               
 +
* In other words, vhost=on is added to enable vhost, and vhostfd= gets the file descriptor for device.                                             
 +
                                                                                               
 +
* To check that vhost is enabled:                                                               
 +
1. Do lsmod and see that vhost_net module has reference count>  1
 +
                           
 +
(this might happen even when vhost was disabled in the end,so please do also one of the following)
 +
 +
                                                 
 +
2. look up your device in guest (assuming it is the only virtio device, virtio0):           
 +
 
 +
    cat /sys/bus/virtio/devices/virtio0/features                                             
 +
bit 15 should be 0                                                                       
 +
 +
3. run netperf from/to host, run top in host, you will see vhost thread running in host
 +
 +
* Migration implications:
 +
vhost net does not support mergeable rx buffers, so to make
 +
migration to non-vhost device safe, you must disable
 +
mergeable rx buffers on non-vhost. You can do this by adding
 +
<pre>
 +
mrg_rxbuf=off
 +
</pre>
  
 
== Release Notes ==
 
== Release Notes ==
----
+
* KVM networking is enhanced by kernel acceleration
 +
 
 +
This was shown to reduce latency by a factor of 5,
 +
and improve bandwidth from 90% native to 95% of native
 +
on some systems [[http://www.linux-kvm.org/page/VhostNet]]
 +
 
 +
== Comments and Discussion ==
 +
* See [[Talk:Features/VHostNet]]
 +
[[Category:FeatureAcceptedF13]]
 +
 
 +
 
 +
[[Category:Virtualization|VHostNet]]
 +
[[Category:F13 Virt Features|VHostNet]]

Latest revision as of 15:37, 19 February 2010

Contents

[edit] Enable kernel acceleration for kvm networking

[edit] Summary

Enable kernel acceleration for kvm networking

[edit] Owner

[edit] Current status

  • Targeted release: Fedora 13
  • Last updated: 2010-01-26
  • Percentage of completion: 100%
    • code complete, changes in response to upstream review are being incorporated

[edit] Detailed Description

vhost net moves the task of converting virtio descriptors to skbs and back from qemu userspace to the kernel driver. It is activated by using -netdev options (instead of -net) and adding vhost=on flag.

[edit] Benefit to Fedora

Using a kernel module reduces latency and improves packets per second for small packets.

[edit] Scope

The work is all upstream in the kernel and qemu. Guest code is already upstream. Host/qemu work is in progress. For Fedora 13 will likely have to backport some of it.

[edit] Milestones Reached

- Guest Kernel:

 MSI-X support in virtio net

- Host Kernel:

 iosignalfd, irqfd, eventfd polling
 finalize kernel/user interface    
 socket polling                    
 virtio transport with copy from/to user
 TX credits using destructor (or: poll device status)
 TSO/GSO                                             
 profile and tune                                    

- qemu:

 MSI-X support in virtio net
 connect to kernel backend with MSI-X     
 PCI interrupts emulation                 
 TSO/GSO        
 profile and tune

[edit] In progress

- finalize qemu command line - qemu: migration

Code posted, but won't be upstream in time and probably not important enough to backport

 raw sockets support in qemu, promisc mode

Delayed, will likely not make it by F13 ship date

 mergeable buffers
 programming MAC/vlan filtering

[edit] Test Plan

Guest:

  • WHQL networking tests

Networking:

  • Various MTU sizes
  • Broadcasts, multicasts,
  • Ethtool
  • Latency tests
  • Bandwidth tests
  • UDP testing
  • Guest to guest communication
  • More types of protocol testing
  • Guest vlans
  • Tests combination of multiple vnics on the guests
  • With/without {IP|TCP|UDP} offload

Virtualization:

  • Live migration

Kernel side:

  • Load/unload driver

[edit] User Experience

Users should see faster networking at least in cases of SRIOV or a dedicated per-guest network device.

[edit] Dependencies

  • kernel acceleration is implemented in the kernel rpm and depends on changes in qemu-kvm to work correctly.

[edit] Contingency Plan

  • We don't turn it on by default if it turns out to be unstable.

[edit] Documentation

  • vhost net is activated by adding vhost=on to netdev option.
  • For non-vhost:
                                                                                                
 -net nic,model=virtio,netdev=foo -netdev                                                       
 tap,id=foo,ifname=msttap0,script=/home/mst/ifup,downscript=no
  • For vhost:
                                                                                                
 -net nic,model=virtio,netdev=foo -netdev                                                       
 tap,id=foo,vhost=on,vhostfd=<fd>,ifname=msttap0,script=/home/mst/ifup,downscript=no            
  • In other words, vhost=on is added to enable vhost, and vhostfd= gets the file descriptor for device.
  • To check that vhost is enabled:

1. Do lsmod and see that vhost_net module has reference count> 1

(this might happen even when vhost was disabled in the end,so please do also one of the following)


2. look up your device in guest (assuming it is the only virtio device, virtio0):

    cat /sys/bus/virtio/devices/virtio0/features                                               

bit 15 should be 0

3. run netperf from/to host, run top in host, you will see vhost thread running in host

  • Migration implications:

vhost net does not support mergeable rx buffers, so to make migration to non-vhost device safe, you must disable mergeable rx buffers on non-vhost. You can do this by adding

mrg_rxbuf=off

[edit] Release Notes

  • KVM networking is enhanced by kernel acceleration

This was shown to reduce latency by a factor of 5, and improve bandwidth from 90% native to 95% of native on some systems [[1]]

[edit] Comments and Discussion