Nested Guests

Nested guests are KVM guests run in a KVM guest. As of Feb 2018 this feature is considered working but experimental, and some limitations apply.

When describing nested guests, we will use the following terminology:

"L0" – the bare metal host, running KVM
"L1" – a VM running on L0; also called the "guest hypervisor" — as it itself is capable of running KVM
"L2" – a VM running on L1, also called the "nested guest"

Detailed

Why use it?

An additional layer of virtualization sometimes comes in handy. You might have access to a large virtual machine in a cloud environment that you want to compartmentalize into multiple workloads. You might be running a lab environment in a training session.

How to run

The KVM kernel modules do not enable nesting by default (though your distribution may override this default). To enable nesting, set the nested module parameter to Y or 1. You may set this parameter persistently in a file in /etc/modprobe.d in the L0 host, for example:

# If you have an Intel CPU, use this:
$ cat /etc/modprobe.d/kvm_intel.conf
options kvm-intel nested=Y

# If you have an AMD CPU, then this:
$ cat /etc/modprobe.d/kvm_amd.conf
options kvm-amd nested=1

Once your L0 host is capable of nesting, you should be able to start an L1 guest with the -cpu host option (or for better live migration compatibility, use a named CPU model supported by QEMU, such as: -cpu Haswell-noTSX-IBRS,vmx=on) and the guest will subsequently be capable of running an L2 guest with accelerated KVM.

Limitations

Once an L1 guest has started an L2 guest, it is no longer capable of being migrated, saved, or loaded (see Migration for details on these actions) until the L2 guest shuts down. This is currently an inherent limitation (that is being worked on, as of Feb 2018) of the KVM implementation on all architectures except s390x.

Attempting to migrate or save & load an L1 guest while an L2 guest is running will result in undefined behavior. You might see a kernel BUG! entry in dmesg, a kernel oops, or an outright kernel panic. At any rate, a thus migrated or loaded L1 guest can no longer be considered stable or secure, and must be restarted.

Migrating an L1 guest merely configured to support nesting, while not actually running L2 guests, is expected to function normally. Live-migrating an L2 guest from one L1 guest to another is also expected to succeed.

Links

Discussion on limitations of nested guests (libvirt-users mailing list, February 2018)
Red Hat Bugzilla Bug 1076294
kernel.org Bugzilla Bug 53851
kernel.org Bugzilla Bug 198621
Intel nVMX upstream specification