Projects/auto-ballooning

Automatic Ballooning

Introduction

When a Linux host is running out of memory, the kernel will take action to reclaim memory. This action may be detrimental to KVM guests performace (eg. swapping) or even extreme to the point where the kernel may kill a VM or an important virt stack component.

To help avoiding this scenario, a KVM guest could automatically return memory to the host when the host is facing memory pressure. By doing so the guest may also get into memory pressure so we also need a way to allow the guest to automatically get memory back.

Design

KVM guests have a driver called the balloon driver. This driver allows guests to shrink and increase their memory. The balloon driver supports two operations:

• Inflate: guest memory shrinks and memory is returned to the host
• Deflate: guest memory increases and memory is taken from the host

Today, both operations are manual. The automatic ballooning project is about making them completely automatic, based on host and guest needs.

Automatic Inflate

Automatic inflate is performed by QEMU, that is, the KVM host. There are two ways I'm playing with automatic inflate, in both of them QEMU registers for memory pressure events so that it's notified when the host is under memory pressure:

1. QEMU registers for the VMPRESSURE_MEDIUM event and inflates the balloon by an user specified value when the event is received. That value value could default to 16MB or 32MB. This is what the current patchset does
2. QEMU registers for the three memory pressure events and inflates the balloon accordingly. Say, it could inflate the balloon by 1MB on VMPRESSURE_LOW; 16MB on VMPRESSURE_MEDIUM and 128MB on VMPRESSURE_CRITICAL

Automatic Deflate

Automatic deflate is performed by the virtio-balloon driver within the guest. There's also two ways of implementing this:

1. The virtio-balloon driver registers a callback with the shrinker API. That callback is called when the guest kernel is facing memory pressure. The number of pages to be returned to the kernel is passed to the callback, so that the callback can deflate the balloon by that amount. This is what the current patchset does
2. The virtio-balloon driver registers for an in-kernel memory pressure event (not upstream yet) and deflates the balloon by some fixed amount (maybe the same amount used in automatic inflate)

Testing

You have to do three things to play with automatic-ballooning:

1. Install kernel 3.10 or higher in the host. Make sure to enable CONFIG_CGROUPS and CONFIG_MEMCG
2. Clone QEMU from git://repo.or.cz/qemu/qmp-unstable.git balloon/auto-ballooning/rfc.v2 (or grab the patch here)
3. Install the following kernel on your guest git://repo.or.cz/linux-2.6/luiz-linux-2.6.git virtio-balloon/auto-deflate/rfc (or grab the first two patches from the web interface)

After setting up the above, do the following to experiment with automatic deflate (which is easy to reproduce):

1. Pass -balloon virtio,auto-balloon=true when starting QEMU
2. Wait for the guest to boot, then generate some memory pressure in the guest (say a kernel build with lots of jobs)
3. Switch to QEMU's monitor and shrink guest memory (say from 1G to 200MB)
4. Watch the guest increase its memory by running "info balloon" on QEMU's monitor

To see automatic inflate and automatic deflate in action, you can run several VMs in parallel doing some heavy memory workload. Make sure to over commit host memory. Say your host has 4GB, run 6 VMs with 1GB each.

Another idea is to run one or more VMs in a memory constrained cgroup, although this will require some hacking on the current patches as they register for the root memory cgroup.