How to assign devices with VT-d in KVM

2011-05-27T02:03:47Z

Luke-Jr: Graphics card passthrough success

= How to assign devices with VT-d in KVM =

== VT-d support ==
* In order to assign devices in KVM, you'll need a system which supports VT-d. This has nothing to do with the VT-x support of your CPU, VT-d needs to be supported by both your chipset on your motherboard and by your CPU.

* If you are in doubt whether your motherboard or CPU supports VT-d or not, the Xen VT-d wikipage has some pointers of VT-d enabled chipsets, motherboards and CPUs:
http://wiki.xensource.com/xenwiki/VTdHowTo

* If your hardware doesn't have an IOMMU ("Intel VT-d" support in case of Intel - "AMD I/O Virtualization Technology" support in case of AMD), you'll not be able to assign devices in KVM. Some work towards allowing this were done, but the code never made it into KVM, due to various issues with the code. At the moment it doesn't seem like device assignment without hardware support, will ever be integrated into KVM.

* Assignment of graphics cards are not officially supported at the moment, but there has been success passing through a secondary Radeon HD 5850 as a VM's secondary display. It also seems like one person is currently working on writing patches for primary display support in his spare time (February, 2010):
http://www.spinics.net/lists/kvm/msg25977.html

== Assigning device to guest ==
'''1. Modifying kernel config:'''
* make menuconfig
* set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*"
* set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*"
* set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"
* optional setting:
set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*"
* exit/save

'''2. build kernel:'''
* make
* make modules_install
* make install

'''3. reboot and verify that your system has IOMMU support'''
* AMD Machine
** dmesg | grep AMD-Vi
...
AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Lazy IO/TLB flushing enabled
AMD-Vi: Initialized for Passthrough Mode
...
* Intel Machine
** dmesg | grep -e DMAR -e IOMMU
...
DMAR:DRHD base: 0x000000feb03000 flags: 0x0
IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000
...
* If you get no output you'll need to fix this before moving on. Check if your hardware supports VT-d and check that it has been enabled in BIOS.

'''NOTE:''' If you still get an error "No IOMMU found." Check dmesg for errors suggesting your BIOS is broken. Another possible reason:
CONFIG_DMAR_DEFAULT_ON is not set. In that case, pass "intel_iommu=on" as kernel parameter to enable it. AMD uses different kernel parameter than Intel, on AMD you need to pass "iommu=pt iommu=1".

'''4. unbind device from host kernel driver (example PCI device 01:00.0)
'''
* Load the PCI Stub Driver if it is compiled as a module
modprobe pci_stub
* lspci -n
* locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9
...
01:00.0 0200: 8086:10b9 (rev 06)
...

* echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
* echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
* echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

'''5. load KVM modules:'''
* modprobe kvm
* modprobe kvm-intel

'''6. assign device:'''
* /usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -device pci-assign,host=01:00.0

== VT-d device hotplug ==
KVM also supports hotplug devices with VT-d to guest. In guest command interface (you can press Ctrl+Alt+2 to enter it), you can use following command to hot add/remove devices to/from guest:
* hot add:
device_add pci-assign,host=01:00.0,id=mydevice
* hot remove:
device_del mydevice

== Notes ==
* VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
* If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.

Status

2011-05-27T02:01:27Z

Luke-Jr: PCI-Express passthrough

=Status=

KVM is included in the mainline linux kernel since 2.6.20 and is stable and fast for most workloads.

It is also available as a patch for recent Linux kernel versions and as an external module that can be used with your favorite distro- provided kernel going back up to 2.6.16, therefore including all latest versions for Enterprise Linux Distributions.

===Working:===

* Intel-based hosts (requires VT capable processors)
* AMD-based hosts (requires SVM capable processors)
* Windows/Linux/Unix guests (32-bit and 64-bit)
* SMP hosts
* SMP guests (as of kvm-61, max 16 cpu supported)
* Live [[Migration]] of guests from one host to another (32-bit and 64-bit)
* See the [[Guest Support Status]] page for a list of guest operating systems known to work
* See the [[Host Support Status]] page for information on host hardware.
* Guest swapping
* [[Paravirtualized networking]]
* [[Paravirtualized block device]]
* [[How_to_assign_devices_with_VT-d_in_KVM|PCI-Express passthrough]]

===In progress:===

* [[PowerPC|PowerPC port]]
* IA64 port
* xenner (http://kraxel.fedorapeople.org/xenner), a project to run x86 xen guest (domU) kernels
* [https://wiki.ncl.cs.columbia.edu/wiki/index.php/AndroidVirt:MainPage ARM port]

KVM - User contributions [en]

How to assign devices with VT-d in KVM

Status