Nested Guests

Feature functional description

Nested guests are KVM guests run in a KVM guest. As of 2018 this feature is considered working but experimental, and some limitations apply.

When describing nested guests, the following conventions apply:

• "L0" is an unvirtualized host system capable of running KVM.
• "L1" is a virtual system running on L0, which is itself also capable of running KVM.
• "L2" is a virtual system running on L1, which does no further virtualization.

In this context, we refer to L1 as an unnested guest, and L2 as a nested guest.

Detailed

Why use it?

An additional layer of virtualization sometimes comes in handy. You might have access to a large virtual machine in a cloud environment that you want to compartmentalize into multiple workloads. You might be running a lab environment in a training session.

How to run

The KVM kernel modules do not enable nesting by default (though your distribution may override this default). To enable nesting, set the nested module parameter to Y or 1. You may set this parameter persistently in a file in /etc/modprobe.d in the L0 host, for example:

# cat /etc/modprobe.d/kvm_intel.conf
options kvm-intel nested=Y

# cat /etc/modprobe.d/kvm_amd.conf
options kvm-amd nested=1

Once your L0 host is capable of nesting, you should be able to start an L1 guest with the -cpu host option, and the guest will subsequently be capable of running an L2 guest with accelerated KVM.

Limitations

Once an L1 guest has started an L2 guest, it is no longer capable of being migrated, saved, or loaded (see Migration for details on these actions) until the L2 guest shuts down. This is currently an inherent limitation of the KVM implementation on all architectures except s390x.

Attempting to migrate or save & load an L1 guest while an L2 guest is running will result in undefined behavior. You might see a kernel BUG! entry in dmesg, a kernel oops, or an outright kernel panic. At any rate, a thus migrated or loaded L1 guest can no longer be considered stable or secure, and must be restarted.

Migrating an L1 guest merely configured to support nesting, while not actually running L2 guests, is expected to function normally. Live-migrating an L2 guest from one L1 guest to another is also expected to succeed.

Links

• Discussion on limitations of nested guests (libvirt-users mailing list, February 2018)
• Red Hat Bugzilla Bug 1076294
• kernel.org Bugzilla Bug 198621