How to assign devices with VT-d in KVM

VT-d support

In order to assign devices in KVM, you'll need a system which supports VT-d. This has nothing to do with the VT-x support of your CPU, VT-d needs to be supported by your chipset on your motherboard.

If you are in doubt whether your motherboard supports VT-d or not, the Xen VT-d wikipage has some pointers of VT-d enabled chipsets and motherboards:

       http://wiki.xensource.com/xenwiki/VTdHowTo

Assigning device to guest

1. Modifying kernel config:

make menuconfig
set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*"
set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*"
set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"
optional setting:

       set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*"

exit/save

2. build kernel:

make
make modules_install
make install

3. reboot and verify that your system has IOMMU support

dmesg | grep -e DMAR -e IOMMU

...
DMAR:DRHD base: 0x000000feb03000 flags: 0x0
IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000
...

If you get no output you'll need to fix this before moving on. Check if your hardware supports VT-d and check that it has been enabled in BIOS.

NOTE: If you still get an error "No IOMMU found." Check dmesg for errors suggesting your BIOS is broken.

4. unbind device from host kernel driver (example PCI device 01:00.0)

lspci -n
locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9

       ...
       01:00.0 0200: 8086:10b9 (rev 06)
       ...

echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

5. load KVM modules:

modprobe kvm
modprobe kvm-intel

6. assign device:

/usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -pcidevice host=01:00.0

VT-d device hotplug

KVM also supports hotplug devices with VT-d to guest. In guest command interface (you can press Ctrl+Alt+2 to enter it), you can use following command to hot add/remove devices to/from guest:

hot add:

       pci_add pci_addr=auto host host=01:00.0

hot remove (e.g bdf is 00:06.0 in guest):

       pci_del pci_addr=6

Notes

VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.