KVM - User contributions [en]

VFIO vs virtio

2015-05-12T02:54:39Z

Robert: Robert moved page VFIO vs virtio to 10G NIC performance: VFIO vs virtio: More accurate name

#REDIRECT [[10G NIC performance: VFIO vs virtio]]

10G NIC performance: VFIO vs virtio

2015-05-12T02:54:39Z

Robert: Robert moved page VFIO vs virtio to 10G NIC performance: VFIO vs virtio: More accurate name

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

== Test Topology ==

2 Intel Grantley-EP platforms (Xeon E5-2697 v3) connected by 10G link; memory 96 G. 
NIC: Intel 82599ES [http://ark.intel.com/products/41282/Intel-82599ES-10-Gigabit-Ethernet-Controller] 
Test Tool: iperf 
OS: RHEL 7.1 

== Result summary ==
*In native environment, iperf can get '''9.4''' Gbps throughput. Since iperf is a SW packet generator and normal process shall it be, this a reasonable number.
*With VFIO passthrough, network performance is also '''9.4''' Gbps; i.e., we cannot observe overhead in virtualization environment with VFIO passthrough method, in context of typical SW network user application.
*With virtio approach, if proper configured (details see below), network performance can also achieve '''9.4''' Gbps; otherwise, poor performance will be '''3.6''' Gbps.

=== Some references first ===
SR-IOV[http://www.intel.com/content/www/us/en/network-adapters/virtualization.html] 
vt-d assignment[[How_to_assign_devices_with_VT-d_in_KVM]]
----

Here is the details about each kind of configuration.

== VFIO passthrough VF (SR-IOV) to guest ==
=== Requirements ===
#You NIC supports SR-IOV (how to check? see below)
#driver (usually igb or ixgb) loaded with 'max_vfs=<num>' (better to modinfo to check accurate parameter name)
#kernel modules needed: NIC driver, vfio-pci module, intel-iommu module

=== Check if your NIC supports SR-IOV ===
'''<nowiki>lspci -s <NIC_BDF> -vvv | grep -i "Single Root I/O Virtualization"</nowiki>'''

=== Assign the VF to a guest ===
Unbind from igbvf driver and Bind to VFIO driver
#unbind from previous driver (take igbvf device for example)
#;echo <vf_BDF> > /sys/bus/pci/device/<vf_BDF>/driver/unbind
#;lspci -s <vf_BDF> -n //to get its number
#://it will return like below
#;<nowiki>0a:13.3 0200: 8086:1520 (rev 01)</nowiki>
#://8086 1520 is its numeric number
#bind to vfio-pci driver
#;echo 8086 1520 > /sys/bus/pci/drivers/vfio-pci/new_id

Now you can see this device is bound to vfio-pci driver 
'''lspci -s <vf_BDF> -k'''

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -net none -device vfio-pci,host=81:10.0 -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio -nographic</nowiki>'''
 '-net none' tells qemu not emulate network devices
 '-device vfio-pci,host=' designate a vfio-pci device and the device's host BDF

== Virtio ==
=== Requirements ===
virtio compiled in kernel (RHEL7.1 native kernel already have them)
CONFIG_VIRTIO=m
CONFIG_VIRTIO_RING=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_BLK=m
CONFIG_VIRTIO_NET=m

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -device virtio-net-pci,netdev=net0 -netdev tap,id=net0,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''

 
==== Poor performance configuration ====
 '''<nowiki>qemu-kvm -m 16G -smp 8 -net nic,model=virtio -net tap,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''
 If not use <nowiki>'-device virtio-net-pci'</nowiki> option, performance will be 3.6 Gbps.

10G NIC performance: VFIO vs virtio

2015-05-12T02:53:15Z

Robert:

User talk:Robert

2015-05-12T02:49:30Z

Robert:

Hello {{PAGENAME}},
Thank you for your recent additions. However, note that due to incessant spam, we had to disable posting of http links by default and hence, any edits that had http links have automatically been rejected :( I have added the necessary permissions to your account so that you can post http henceforth. Please get in touch if you have any questions.
Thanks
[[User:Bsd|bsd]] ([[User talk:Bsd|talk]]) 16:56, 11 May 2015 (UTC)
: Thanks. I was wondering why my contents cannot be saved sometimes. I see now. --[[User:Robert|Robert]] ([[User talk:Robert|talk]]) 02:49, 12 May 2015 (UTC)

10G NIC performance: VFIO vs virtio

2015-05-11T11:00:16Z

Robert: finalize

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

== Test Topology ==

2 Intel Grantley-EP platforms (Xeon E5-2697 v3) connected by 10G link; memory 96 G. 
NIC: Intel 82599ES 
Test Tool: iperf 
OS: RHEL 7.1 

== Result summary ==
*In native environment, iperf can get '''9.4''' Gbps throughput. Since iperf is a SW packet generator and normal process shall it be, this a reasonable number.
*With VFIO passthrough, network performance is also '''9.4''' Gbps; i.e., we cannot observe overhead in virtualization environment with VFIO passthrough method, in context of typical SW network user application.
*With virtio approach, if proper configured (details see below), network performance can also achieve '''9.4''' Gbps; otherwise, poor performance will be '''3.6''' Gbps.

----

Here is the details about each kind of configuration.

== VFIO passthrough VF (SR-IOV) to guest ==
=== Requirements ===
#You NIC supports SR-IOV (how to check? see below)
#driver (usually igb or ixgb) loaded with 'max_vfs=<num>' (better to modinfo to check accurate parameter name)
#kernel modules needed: NIC driver, vfio-pci module, intel-iommu module

=== Check if your NIC supports SR-IOV ===
'''<nowiki>lspci -s <NIC_BDF> -vvv | grep -i "Single Root I/O Virtualization"</nowiki>'''

=== Assign the VF to a guest ===
Unbind from igbvf driver and Bind to VFIO driver
#unbind from previous driver (take igbvf device for example)
#;echo <vf_BDF> > /sys/bus/pci/device/<vf_BDF>/driver/unbind
#;lspci -s <vf_BDF> -n //to get its number
#://it will return like below
#;<nowiki>0a:13.3 0200: 8086:1520 (rev 01)</nowiki>
#://8086 1520 is its numeric number
#bind to vfio-pci driver
#;echo 8086 1520 > /sys/bus/pci/drivers/vfio-pci/new_id

Now you can see this device is bound to vfio-pci driver 
'''lspci -s <vf_BDF> -k'''

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -net none -device vfio-pci,host=81:10.0 -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio -nographic</nowiki>'''
 '-net none' tells qemu not emulate network devices
 '-device vfio-pci,host=' designate a vfio-pci device and the device's host BDF

== Virtio ==
=== Requirements ===
virtio compiled in kernel (RHEL7.1 native kernel already have them)
CONFIG_VIRTIO=m
CONFIG_VIRTIO_RING=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_BLK=m
CONFIG_VIRTIO_NET=m

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -device virtio-net-pci,netdev=net0 -netdev tap,id=net0,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''

 
==== Poor performance configuration ====
 '''<nowiki>qemu-kvm -m 16G -smp 8 -net nic,model=virtio -net tap,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''
 If not use <nowiki>'-device virtio-net-pci'</nowiki> option, performance will be 3.6 Gbps.

10G NIC performance: VFIO vs virtio

2015-05-11T10:56:03Z

Robert: More fine tune

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

== Test Topology ==

2 Intel Grantley-EP platforms (Xeon E5-2697 v3) connected by 10G link; memory 96 G. 
NIC: Intel 82599ES 
Test Tool: iperf 
OS: RHEL 7.1 

== Result summary ==
*In native environment, iperf can get 9.4 Gbps throughput. Since iperf is a SW packet generator and normal process shall it be, this a reasonable number.
*With VFIO passthrough, network performance is also 9.4 Gbps; i.e., we cannot observe overhead in virtualization environment with VFIO passthrough method, in context of typical SW network user application.
*With virtio approach, if proper configured (details see below), network performance can also achieve 9.4 Gbps.

----

Here is the details about each kind of configuration.

== VFIO passthrough VF (SR-IOV) to guest ==
=== Requirements ===
#You NIC supports SR-IOV (how to check? see below)
#driver (usually igb or ixgb) loaded with 'max_vfs=<num>' (better to modinfo to check accurate parameter name)
#kernel modules needed: NIC driver, vfio-pci module, intel-iommu module

=== Check if your NIC supports SR-IOV ===
lspci -s <NIC_BDF> -vvv | grep -i "Single Root I/O Virtualization"

=== Assign the VF to a guest ===
Unbind from igbvf driver and Bind to VFIO driver
#unbind from previous driver (take igbvf device for example)
#;echo <vf_BDF> > /sys/bus/pci/device/<vf_BDF>/driver/unbind
#;lspci -s <vf_BDF> -n //to get its number
#://it will return like below
#;<nowiki>0a:13.3 0200: 8086:1520 (rev 01)</nowiki>
#://8086 1520 is its numeric number
#bind to vfio-pci driver
#;echo 8086 1520 > /sys/bus/pci/drivers/vfio-pci/new_id

Now you can see this device is bound to vfio-pci driver 
'''lspci -s <vf_BDF> -k'''

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -net none -device vfio-pci,host=81:10.0 -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio -nographic</nowiki>'''
 '-net none' tells qemu not emulate network devices
 '-device vfio-pci,host=' designate a vfio-pci device and the device's host BDF

== Virtio ==
=== Requirements ===
virtio compiled in kernel (RHEL7.1 native kernel already have them)
CONFIG_VIRTIO=m
CONFIG_VIRTIO_RING=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_BLK=m
CONFIG_VIRTIO_NET=m

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -device virtio-net-pci,netdev=net0 -netdev tap,id=net0,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''

 
==== Poor performance configuration ====
 '''<nowiki>qemu-kvm -m 16G -smp 8 -net nic,model=virtio -net tap,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''
 If not use <nowiki>'-device virtio-net-pci'</nowiki> option, performance will be 3.6 Gbps.

10G NIC performance: VFIO vs virtio

2015-05-11T10:50:22Z

Robert: virtio part complete

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

== Test Topology ==

2 Intel Grantley-EP platforms (Xeon E5-2697 v3) connected by 10G link; memory 96 G.

NIC: Intel 82599ES

Test Tool: iperf

OS: RHEL 7.1

== Result summary ==
*In native environment, iperf can get 9.4 Gbps throughput. Since iperf is a SW packet generator and normal process shall it be, this a reasonable number.
*With VFIO passthrough, network performance is also 9.4 Gbps; i.e., we cannot observe overhead in virtualization environment with VFIO passthrough method, in context of typical SW network user application.
*With virtio approach, if proper configured (details see below), network performance can also achieve 9.4 Gbps.

----

Here is the details about each kind of configuration.

== VFIO passthrough VF (SR-IOV) to guest ==
=== Requirements ===
#You NIC supports SR-IOV (how to check? see below)
#driver (usually igb or ixgb) loaded with 'max_vfs=<num>' (better to modinfo to check accurate parameter name)
#kernel modules needed: NIC driver, vfio-pci module, intel-iommu module

=== Check if your NIC supports SR-IOV ===
lspci -s <NIC_BDF> -vvv | grep -i "Single Root I/O Virtualization"

=== Assign the VF to a guest ===
Unbind from igbvf driver and Bind to VFIO driver
#unbind from previous driver (take igbvf device for example)
#;echo <vf_BDF> > /sys/bus/pci/device/<vf_BDF>/driver/unbind
#;lspci -s <vf_BDF> -n //to get its number
#://it will return like below
#;<nowiki>0a:13.3 0200: 8086:1520 (rev 01)</nowiki>
#://8086 1520 is its numeric number
#bind to vfio-pci driver
#;echo 8086 1520 > /sys/bus/pci/drivers/vfio-pci/new_id

Now you can see this device is bound to vfio-pci driver 
'''lspci -s <vf_BDF> -k'''

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -net none -device vfio-pci,host=81:10.0 -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio -nographic</nowiki>'''
 '-net none' tells qemu not emulate network devices
 '-device vfio-pci,host=' designate a vfio-pci device and the device's host BDF

== Virtio ==
=== Requirements ===
virtio compiled in kernel (RHEL7.1 native kernel already have them)
CONFIG_VIRTIO=m
CONFIG_VIRTIO_RING=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_BLK=m
CONFIG_VIRTIO_NET=m

Create guest with direct passthrough via VFIO framework 
'''<nowiki>qemu-kvm -m 16G -smp 8 -device virtio-net-pci,netdev=net0 -netdev tap,id=net0,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''

==== Poor performance configuration ==== 
'''<nowiki>qemu-kvm -m 16G -smp 8 -net nic,model=virtio -net tap,script=/etc/qemu-ifup -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio –nographic</nowiki>'''
If not use <nowiki>'-device virtio-net-pci'</nowiki> option, performance will be 3.6 Gbps.

10G NIC performance: VFIO vs virtio

2015-05-11T10:44:56Z

Robert: VFIO method completes

10G NIC performance: VFIO vs virtio

2015-05-11T10:00:02Z

Robert:

10G NIC performance: VFIO vs virtio

2015-05-11T09:52:44Z

Robert: fine tune

10G NIC performance: VFIO vs virtio

2015-05-11T09:51:24Z

Robert: format it

10G NIC performance: VFIO vs virtio

2015-05-11T09:48:38Z

Robert: VFIO

10G NIC performance: VFIO vs virtio

2015-05-11T09:44:02Z

Robert: Result summary

10G NIC performance: VFIO vs virtio

2015-05-11T09:40:59Z

Robert: So how about this?

10G NIC performance: VFIO vs virtio

2015-05-11T09:39:23Z

Robert:

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

I mean this.

10G NIC performance: VFIO vs virtio

2015-05-11T09:38:21Z

Robert:

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

Test
Now

KVM:VFIO V.S. virtio

2015-05-11T09:37:59Z

Robert: Robert moved page KVM:VFIO V.S. virtio to VFIO vs virtio

#REDIRECT [[VFIO vs virtio]]

10G NIC performance: VFIO vs virtio

2015-05-11T09:37:59Z

Robert: Robert moved page KVM:VFIO V.S. virtio to VFIO vs virtio

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

VFIO V.S. virtio

2015-05-11T09:24:42Z

Robert: Robert moved page VFIO V.S. virtio to KVM:VFIO V.S. virtio

#REDIRECT [[KVM:VFIO V.S. virtio]]

10G NIC performance: VFIO vs virtio

2015-05-11T09:24:42Z

Robert: Robert moved page VFIO V.S. virtio to KVM:VFIO V.S. virtio

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

10G NIC performance: VFIO vs virtio

2015-05-11T09:04:42Z

Robert: 10G NIC performance

We did some experiment trying to measure network performance overhead in virtualization environment, comparing between VFIO passthrough and virtio approaches.

10G NIC performance: VFIO vs virtio

2015-05-11T09:02:55Z

Robert: Init

We did some experiment comparing KVM guest network performance between VFIO

whywhy why

Networking

2015-05-11T08:47:21Z

Robert: /* performance */

= Setting guest network =

Guest (VM) networking in kvm is the same as in qemu, so it is possible to refer to other documentations about networking for qemu. This page will try to explain how to configure the most frequent types of network needed.

== User Networking ==

'''Use case:'''
* You want a simple way for your virtual machine to access to the host, to the internet or to resources available on your local network.
* You don't need to access your guest from the network or from another guest.
* You are ready to take a huge performance hit.
* Warning: User networking does not support a number of networking features like ICMP. Certain applications (like ping) may not function properly.

'''Prerequisites:'''
* You need kvm up and running
* If you don't want to run as root, the user you want to use needs to have rw access to /dev/kvm
* If you want to be able to access the internet or a local network, your host system must be able to access the internet or the local network

'''Solution:'''
* simply run your guest without specifying network parameters, which by default will create user-lever (a.k.a slirp) networking:
qemu-system-x86_64 -hda /path/to/hda.img

'''Notes:'''
* The IP address can be automatically assigned to the guest thanks to the DHCP service integrated in QEMU
* If you run multiple guests on the host, you don't need to specify a different MAC address for each guest
* The default is equivalent to this explicit setup:
qemu-system-x86_64 -hda /path/to/hda.img -netdev user,id=user.0 -device e1000,netdev=user.0
* user.0 identifier above is just to connect the two halves into one, you may use any identifier you wish, such as "n" or "net0".
* Use rtl8139 instead of e1000 to get 8139-series NIC.
* You can still access one specific port on the guest using the "hostfwd" option. This means e.g. if you want to transport a file with scp from host to guest, start the guest with "-device e1000,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::5555-:22". Now you are forwarding the host port 5555 to the guest port 22. After starting up the guest, you can transport a file with e.g. "scp -P 5555 file.txt root@localhost:/tmp" from host to guest. Or you can also use other address of the host to connect to.

== private virtual bridge ==

'''Use case:'''
* You want to set up a private network between 2 or more virtual machines. This network won't be seen from the other virtual machines nor from the real network.

'''Prerequisites:'''
* You need kvm up and running
* If you don't want to run as root, the user you want to use needs to have rw access to /dev/kvm
* You need the following commands installed on your system, and if you don't want to run as root, the user you want to use needs to be able to sudo the following command:
/sbin/ip
/usr/sbin/brctl
/usr/sbin/tunctl

'''Solution:'''

* You need to create a bridge, e-g:
sudo /usr/sbin/brctl addbr br0

* You need a qemu-ifup script containing the following:
#!/bin/sh
set -x

switch=br0

if [ -n "$1" ];then
/usr/bin/sudo /usr/sbin/tunctl -u `whoami` -t $1
/usr/bin/sudo /sbin/ip link set $1 up
sleep 0.5s
/usr/bin/sudo /usr/sbin/brctl addif $switch $1
exit 0
else
echo "Error: no interface specified"
exit 1
fi

* Generate a MAC address, either manually or using:
#!/bin/bash
# generate a random mac address for the qemu nic
printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

* Run each guest with the following, replacing $macaddress with the value from the previous step
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0

'''Notes:'''
* If you don't want to run as root, the qemu-ifup must be executable by the user you want to use
* You can either create a system-wide qemu-ifup in /etc/qemu-ifup or use another one. In the latter case, run
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0,script=/path/to/qemu-ifup

* Each guest on the private virtual network must have a different MAC address

== public bridge ==

'''WARNING:''' The here shown method, will not work with most(all?) wireless drivers, as these do not support bridging.

'''Use case:'''

* You want to assign an IP address to your virtual machines and make them accessible from your local network
* You also want performance out of your virtual machine.

'''Prerequisites:'''
* You need kvm up and running
* If you don't want to run as root, the user you want to use needs to have rw access to /dev/kvm
* You need the following commands installed on your system, and if you don't want to run as root, the user you want to use needs to be able to sudo the following command:
/sbin/ip
/usr/sbin/brctl
/usr/sbin/tunctl

* Your host system must be able to access the internet or the local network

'''Solution 1: using distro sysconfig script'''
{|border="1"
!RedHat's way
!Debian's way
!SuSE's way
|-
|
* Edit /etc/sysconfig/network-scripts/ifcfg-eth0
** comment out BOOTPROTO
** Add BRIDGE=br0
* Create /etc/sysconfig/network-scripts/ifcfg-br0
** The content should be:
DEVICE=br0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Bridge
|
/etc/network/interfaces

# Replace old eth0 config with br0
auto <strike>eth0</strike> br0

# Use old eth0 config for br0, plus bridge stuff
iface br0 inet dhcp
bridge_ports eth0
bridge_stp off
bridge_maxwait 0
bridge_fd 0

|
* Start YaST
* Go to Network Configuration
* Add new device -> Bridge
* Tick your existing network device
* done
|}

* /etc/init.d/networking restart
* The bridge br0 should get the ip address (either static/dhcp) while the physical eth0 is left without ip address.

'''VLANs'''

Please note that the rtl8139 virtual network interface driver does not support VLANs. If you want to use VLANs with your virtual machine, you must use another virtual network interface like virtio.

When using VLANs on a setup like this and no traffic is getting through to your guest(s), you might want to do:
# cd /proc/sys/net/bridge
# ls
bridge-nf-call-arptables bridge-nf-call-iptables
bridge-nf-call-ip6tables bridge-nf-filter-vlan-tagged
# for f in bridge-nf-*; do echo 0 > $f; done

'''Solution 2: manual'''

* You need to create a bridge, e-g:
sudo /usr/sbin/brctl addbr br0

* Add one of your physical interface to the bridge, e-g for eth0:
sudo /usr/sbin/brctl addif br0 eth0

* You need a qemu-ifup script containing the following:

#!/bin/sh
set -x

switch=br0

if [ -n "$1" ];then
/usr/bin/sudo /usr/sbin/tunctl -u `whoami` -t $1
/usr/bin/sudo /sbin/ip link set $1 up
sleep 0.5s
/usr/bin/sudo /usr/sbin/brctl addif $switch $1
exit 0
else
echo "Error: no interface specified"
exit 1
fi

* Generate a MAC address, either manually or using:

#!/bin/sh
# generate a random mac address for the qemu nic
printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

* Run each guest with the following, replacing $macaddress with the value from the previous step
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0

'''Notes:'''
* If you don't want to run as root, the qemu-ifup must be executable by the user you want to use
* You can either create a system-wide qemu-ifup in /etc/qemu-ifup or use another one. In the latter case, run
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0,script=/path/to/qemu-ifup

* Each guest on the network must have a different MAC address

== iptables/routing ==

you can also connect your guest vm to a tap in your host. then setting iptables rules in your host to become a router + firewall for your vm.

Routing would be done simply by creating the default route on the client to the IP of the host (and allowing IP forwarding) and setting a route to the tap? device of the client on the host.

Test the setup beforehand:

*Hostside: Allow IPv4 forwarding and add route to client (could be put in a script - route has to be added after the client has started):

sysctl -w net.ipv4.ip_forward=1 # allow forwarding of IPv4
route add -host <ip-of-client> dev <tap-device> # add route to the client

*Clientside: Default GW of the client is of course then the host (<ip-of-host> has to be in same subnet as <ip-of-client> ...):

route add default gw <ip-of-host>

*Clientside v2: If you host IP is not on the same subnet as <ip-of-client>, then you must manually add the route to host before you create default route:

route add -host <ip-of-host> dev <network-interface>
route add default gw <ip-of-host>

== vde ==

Another option is using vde (virtual distributed ethernet).

== performance ==

Data on benchmarking results should go in here.
There's now a page dedicated to ideas for improving
[[Networking Performance]].

Some 10G NIC performance comparison between VFIO passthrough and virtio [[VFIO vs virtio]]

== Compatibility ==

There's another, old and obsolete syntax of specifying network for virtual machines. Above examples uses -netdev..-device model, old way used -net..-net pairs. For example,

-netdev tap,id=net0 -device e1000,netdev=net0,mac=52:54:00:12:34:56

is about the same as old

-net tap,vlan=0 -net nic,vlan=0,model=e1000,macaddr=52:54:00:12:34:56

(note mac => macaddr parameter change as well; vlan=0 is the default).

Old way used the notion of "VLANs" - these are QEMU VLANS, which has nothing to do with 802.1q VLANs. Qemu VLANs are numbered starting with 0, and it's possible to connect one or more devices (either host side, like -net tap, or guest side, like -net nic) to each VLAN, and, in particular, it's possible to connect more than 2 devices to a VLAN. Each device in a VLAN gets all traffic received by every device in it. This model was very confusing for the user (especially when a guest has more than one NIC).

In new model, each host side correspond to just one guest side, forming a pair of devices based on -netdev id= and -device netdev= parameters. It is less confusing, it is faster (because it's always 1:1 pair), and it supports more parameters than old -net..-net way.

However, -net..-net is still supported, used widely, and mentioned in lots of various HOWTOs and guides around the world. It is also a bit shorter and so faster to type.

10G NIC performance: VFIO vs virtio

2015-05-11T07:35:50Z

Robert: Created page with "We did some experiment comparing KVM guest network performance between VFIO"

We did some experiment comparing KVM guest network performance between VFIO

Networking

2015-05-11T06:49:53Z

Robert: /* performance */

= Setting guest network =

Guest (VM) networking in kvm is the same as in qemu, so it is possible to refer to other documentations about networking for qemu. This page will try to explain how to configure the most frequent types of network needed.

== User Networking ==

'''Use case:'''
* You want a simple way for your virtual machine to access to the host, to the internet or to resources available on your local network.
* You don't need to access your guest from the network or from another guest.
* You are ready to take a huge performance hit.
* Warning: User networking does not support a number of networking features like ICMP. Certain applications (like ping) may not function properly.

'''Prerequisites:'''
* You need kvm up and running
* If you don't want to run as root, the user you want to use needs to have rw access to /dev/kvm
* If you want to be able to access the internet or a local network, your host system must be able to access the internet or the local network

'''Solution:'''
* simply run your guest without specifying network parameters, which by default will create user-lever (a.k.a slirp) networking:
qemu-system-x86_64 -hda /path/to/hda.img

'''Notes:'''
* The IP address can be automatically assigned to the guest thanks to the DHCP service integrated in QEMU
* If you run multiple guests on the host, you don't need to specify a different MAC address for each guest
* The default is equivalent to this explicit setup:
qemu-system-x86_64 -hda /path/to/hda.img -netdev user,id=user.0 -device e1000,netdev=user.0
* user.0 identifier above is just to connect the two halves into one, you may use any identifier you wish, such as "n" or "net0".
* Use rtl8139 instead of e1000 to get 8139-series NIC.
* You can still access one specific port on the guest using the "hostfwd" option. This means e.g. if you want to transport a file with scp from host to guest, start the guest with "-device e1000,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::5555-:22". Now you are forwarding the host port 5555 to the guest port 22. After starting up the guest, you can transport a file with e.g. "scp -P 5555 file.txt root@localhost:/tmp" from host to guest. Or you can also use other address of the host to connect to.

== private virtual bridge ==

'''Use case:'''
* You want to set up a private network between 2 or more virtual machines. This network won't be seen from the other virtual machines nor from the real network.

'''Prerequisites:'''
* You need kvm up and running
* If you don't want to run as root, the user you want to use needs to have rw access to /dev/kvm
* You need the following commands installed on your system, and if you don't want to run as root, the user you want to use needs to be able to sudo the following command:
/sbin/ip
/usr/sbin/brctl
/usr/sbin/tunctl

'''Solution:'''

* You need to create a bridge, e-g:
sudo /usr/sbin/brctl addbr br0

* You need a qemu-ifup script containing the following:
#!/bin/sh
set -x

switch=br0

if [ -n "$1" ];then
/usr/bin/sudo /usr/sbin/tunctl -u `whoami` -t $1
/usr/bin/sudo /sbin/ip link set $1 up
sleep 0.5s
/usr/bin/sudo /usr/sbin/brctl addif $switch $1
exit 0
else
echo "Error: no interface specified"
exit 1
fi

* Generate a MAC address, either manually or using:
#!/bin/bash
# generate a random mac address for the qemu nic
printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

* Run each guest with the following, replacing $macaddress with the value from the previous step
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0

'''Notes:'''
* If you don't want to run as root, the qemu-ifup must be executable by the user you want to use
* You can either create a system-wide qemu-ifup in /etc/qemu-ifup or use another one. In the latter case, run
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0,script=/path/to/qemu-ifup

* Each guest on the private virtual network must have a different MAC address

== public bridge ==

'''WARNING:''' The here shown method, will not work with most(all?) wireless drivers, as these do not support bridging.

'''Use case:'''

* You want to assign an IP address to your virtual machines and make them accessible from your local network
* You also want performance out of your virtual machine.

'''Prerequisites:'''
* You need kvm up and running
* If you don't want to run as root, the user you want to use needs to have rw access to /dev/kvm
* You need the following commands installed on your system, and if you don't want to run as root, the user you want to use needs to be able to sudo the following command:
/sbin/ip
/usr/sbin/brctl
/usr/sbin/tunctl

* Your host system must be able to access the internet or the local network

'''Solution 1: using distro sysconfig script'''
{|border="1"
!RedHat's way
!Debian's way
!SuSE's way
|-
|
* Edit /etc/sysconfig/network-scripts/ifcfg-eth0
** comment out BOOTPROTO
** Add BRIDGE=br0
* Create /etc/sysconfig/network-scripts/ifcfg-br0
** The content should be:
DEVICE=br0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Bridge
|
/etc/network/interfaces

# Replace old eth0 config with br0
auto <strike>eth0</strike> br0

# Use old eth0 config for br0, plus bridge stuff
iface br0 inet dhcp
bridge_ports eth0
bridge_stp off
bridge_maxwait 0
bridge_fd 0

|
* Start YaST
* Go to Network Configuration
* Add new device -> Bridge
* Tick your existing network device
* done
|}

* /etc/init.d/networking restart
* The bridge br0 should get the ip address (either static/dhcp) while the physical eth0 is left without ip address.

'''VLANs'''

Please note that the rtl8139 virtual network interface driver does not support VLANs. If you want to use VLANs with your virtual machine, you must use another virtual network interface like virtio.

When using VLANs on a setup like this and no traffic is getting through to your guest(s), you might want to do:
# cd /proc/sys/net/bridge
# ls
bridge-nf-call-arptables bridge-nf-call-iptables
bridge-nf-call-ip6tables bridge-nf-filter-vlan-tagged
# for f in bridge-nf-*; do echo 0 > $f; done

'''Solution 2: manual'''

* You need to create a bridge, e-g:
sudo /usr/sbin/brctl addbr br0

* Add one of your physical interface to the bridge, e-g for eth0:
sudo /usr/sbin/brctl addif br0 eth0

* You need a qemu-ifup script containing the following:

#!/bin/sh
set -x

switch=br0

if [ -n "$1" ];then
/usr/bin/sudo /usr/sbin/tunctl -u `whoami` -t $1
/usr/bin/sudo /sbin/ip link set $1 up
sleep 0.5s
/usr/bin/sudo /usr/sbin/brctl addif $switch $1
exit 0
else
echo "Error: no interface specified"
exit 1
fi

* Generate a MAC address, either manually or using:

#!/bin/sh
# generate a random mac address for the qemu nic
printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

* Run each guest with the following, replacing $macaddress with the value from the previous step
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0

'''Notes:'''
* If you don't want to run as root, the qemu-ifup must be executable by the user you want to use
* You can either create a system-wide qemu-ifup in /etc/qemu-ifup or use another one. In the latter case, run
qemu-system-x86_64 -hda /path/to/hda.img -device e1000,netdev=net0,mac=$macaddress -netdev tap,id=net0,script=/path/to/qemu-ifup

* Each guest on the network must have a different MAC address

== iptables/routing ==

you can also connect your guest vm to a tap in your host. then setting iptables rules in your host to become a router + firewall for your vm.

Routing would be done simply by creating the default route on the client to the IP of the host (and allowing IP forwarding) and setting a route to the tap? device of the client on the host.

Test the setup beforehand:

*Hostside: Allow IPv4 forwarding and add route to client (could be put in a script - route has to be added after the client has started):

sysctl -w net.ipv4.ip_forward=1 # allow forwarding of IPv4
route add -host <ip-of-client> dev <tap-device> # add route to the client

*Clientside: Default GW of the client is of course then the host (<ip-of-host> has to be in same subnet as <ip-of-client> ...):

route add default gw <ip-of-host>

*Clientside v2: If you host IP is not on the same subnet as <ip-of-client>, then you must manually add the route to host before you create default route:

route add -host <ip-of-host> dev <network-interface>
route add default gw <ip-of-host>

== vde ==

Another option is using vde (virtual distributed ethernet).

== performance ==

Data on benchmarking results should go in here.
There's now a page dedicated to ideas for improving
[[Networking Performance]].

Some 10G NIC performance comparison between VFIO passthrough and virtio [[VFIO V.S. virtio]]

== Compatibility ==

There's another, old and obsolete syntax of specifying network for virtual machines. Above examples uses -netdev..-device model, old way used -net..-net pairs. For example,

-netdev tap,id=net0 -device e1000,netdev=net0,mac=52:54:00:12:34:56

is about the same as old

-net tap,vlan=0 -net nic,vlan=0,model=e1000,macaddr=52:54:00:12:34:56

(note mac => macaddr parameter change as well; vlan=0 is the default).

Old way used the notion of "VLANs" - these are QEMU VLANS, which has nothing to do with 802.1q VLANs. Qemu VLANs are numbered starting with 0, and it's possible to connect one or more devices (either host side, like -net tap, or guest side, like -net nic) to each VLAN, and, in particular, it's possible to connect more than 2 devices to a VLAN. Each device in a VLAN gets all traffic received by every device in it. This model was very confusing for the user (especially when a guest has more than one NIC).

In new model, each host side correspond to just one guest side, forming a pair of devices based on -netdev id= and -device netdev= parameters. It is less confusing, it is faster (because it's always 1:1 pair), and it supports more parameters than old -net..-net way.

However, -net..-net is still supported, used widely, and mentioned in lots of various HOWTOs and guides around the world. It is also a bit shorter and so faster to type.

Tuning KVM

2015-05-08T08:29:38Z

Robert: /* Networking */

== CPU Performance ==

Modern processors come with a wide variety of performance enhancing features such as streaming instructions sets (sse) and other performance-enhancing instructions. These features vary from processor to processor.

QEMU and KVM default to a compatible subset of cpu features, so that if you change your host processor, or perform a live migration, the guest will see its cpu features unchanged. This is great for compatibility but comes at a performance cost.

To pass all available host processor features to the guest, use the command line switch

qemu '''-cpu host'''

if you wish to retain compatibility, you can expose selected features to your guest. If all your hosts have these features, compatibility is retained:

qemu '''-cpu qemu64,+ssse3,+sse4.1,+sse4.2,+x2apic'''

To see the difference between the capabilities of the host CPU versus the guest, just compare the output of the following commands on each system:

cat /proc/cpuinfo | grep flags | uniq

For example, the default setting on a 64 bit host machine is "-cpu qemu64". This includes the following flags:

fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up pni hypervisor

The host itself might support other flags like cx16, mmxext, and so on.

See [http://blog.incase.de/index.php/cpu-feature-flags-and-their-meanings/ CPU Feature Flags And Their Meanings] (and other resources on the web) for more information.

== Networking ==

QEMU defaults to user-mode networking (slirp), which is available without prior setup and without administrative privileges on the host. It is also unfortunately very slow. To get high performance networking, switch to a bridged setup via the -net tap command line switches.

qemu -net nic,model=virtio,mac=... '''-net tap,ifname=...'''

or
qemu -net nic,model=virtio,mac=... '''-net bridge,br=...,helper=/usr/local/libexec/qemu-bridge-helper'''

Besides virtio, approach of vt-d passthrough give high (even higher) performance as well.

qemu '''-net none -device vfio-pci,host=...'''

QEMU also defaults to the RTL8139 network interface card (NIC) model. Again this card is compatible with most guests, but does not offer the best performance. If your guest supports it, switch to the virtio model:

qemu -net nic,'''model=virtio''',mac=... -net tap,ifname=...

== Storage ==

QEMU supports a wide variety for storage formats and back-ends. Easiest to use are the raw and qcow2 formats, but for the best performance it is best to use a raw partition. You can create either a logical volume or a partition and assign it to the guest:

qemu -drive '''file=/dev/mapper/ImagesVolumeGroup-Guest1''',cache=none,if=virtio

QEMU also supports a wide variety of caching modes. If you're using raw volumes or partitions, it is best to avoid the cache completely, which reduces data copies and bus traffic:

qemu -drive file=/dev/mapper/ImagesVolumeGroup-Guest1,'''cache=none''',if=virtio

As with networking, QEMU supports several storage interfaces. The default, IDE, is highly supported by guests but may be slow, especially with disk arrays. If your guest supports it, use the virtio interface:

qemu -drive file=/dev/mapper/ImagesVolumeGroup-Guest1,cache=none,'''if=virtio'''

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.

Tuning KVM

2015-05-08T08:23:43Z

Robert: /* Networking */

== CPU Performance ==

Modern processors come with a wide variety of performance enhancing features such as streaming instructions sets (sse) and other performance-enhancing instructions. These features vary from processor to processor.

QEMU and KVM default to a compatible subset of cpu features, so that if you change your host processor, or perform a live migration, the guest will see its cpu features unchanged. This is great for compatibility but comes at a performance cost.

To pass all available host processor features to the guest, use the command line switch

qemu '''-cpu host'''

if you wish to retain compatibility, you can expose selected features to your guest. If all your hosts have these features, compatibility is retained:

qemu '''-cpu qemu64,+ssse3,+sse4.1,+sse4.2,+x2apic'''

To see the difference between the capabilities of the host CPU versus the guest, just compare the output of the following commands on each system:

cat /proc/cpuinfo | grep flags | uniq

For example, the default setting on a 64 bit host machine is "-cpu qemu64". This includes the following flags:

fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up pni hypervisor

The host itself might support other flags like cx16, mmxext, and so on.

See [http://blog.incase.de/index.php/cpu-feature-flags-and-their-meanings/ CPU Feature Flags And Their Meanings] (and other resources on the web) for more information.

== Networking ==

QEMU defaults to user-mode networking (slirp), which is available without prior setup and without administrative privileges on the host. It is also unfortunately very slow. To get high performance networking, switch to a bridged setup via the -net tap command line switches.

qemu -net nic,model=virtio,mac=... '''-net tap,ifname=...'''

or
qemu -net nic,model=virtio,mac=... '''-net bridge,br=...,helper=/usr/local/libexec/qemu-bridge-helper'''

QEMU also defaults to the RTL8139 network interface card (NIC) model. Again this card is compatible with most guests, but does not offer the best performance. If your guest supports it, switch to the virtio model:

qemu -net nic,'''model=virtio''',mac=... -net tap,ifname=...

== Storage ==

QEMU supports a wide variety for storage formats and back-ends. Easiest to use are the raw and qcow2 formats, but for the best performance it is best to use a raw partition. You can create either a logical volume or a partition and assign it to the guest:

qemu -drive '''file=/dev/mapper/ImagesVolumeGroup-Guest1''',cache=none,if=virtio

QEMU also supports a wide variety of caching modes. If you're using raw volumes or partitions, it is best to avoid the cache completely, which reduces data copies and bus traffic:

qemu -drive file=/dev/mapper/ImagesVolumeGroup-Guest1,'''cache=none''',if=virtio

As with networking, QEMU supports several storage interfaces. The default, IDE, is highly supported by guests but may be slow, especially with disk arrays. If your guest supports it, use the virtio interface:

qemu -drive file=/dev/mapper/ImagesVolumeGroup-Guest1,cache=none,'''if=virtio'''

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.