How to assign devices with VT-d in KVM

Assigning device to guest

1. Modifying kernel config:

make menuconfig
set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*"
set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*"
set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*"
set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"
exit/save

2. build kernel:

make
make modules_install
make install

3. reboot

4. unbind device from host kernel driver (example PCI device 01:00.0)

lspci -n
locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9

       ...
       01:00.0 0200: 8086:10b9 (rev 06)
       ...

echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

5. load KVM modules:

modprobe kvm
modprobe kvm-intel

6. assign device:

/usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -pcidevice host=01:00.0

VT-d device hotplug

KVM also supports hotplug devices with VT-d to guest. In guest command interface (you can press Ctrl+Alt+2 to enter it), you can use following command to hot add/remove devices to/from guest:

hot add:

       pci_add pci_addr=auto host host=01:00.0

hot remove (e.g bdf is 00:06.0 in guest):

       pci_del pci_addr=6

Notes:

VT-d spec specifies that all conventional PCI devices behind a PCIe-to-PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.