Latest revision as of 05:51, 23 August 2018

Automatic Ballooning

NOTE: This page describes an experimental development project from 2013 that was never completed. It is left here as a historical record. The feature described does not exist in any currently shipping version of QEMU

Introduction

The virtio balloon device allows KVM guests to reduce their memory size (thus relinquishing memory to the host) and to increase it back (thus taking memory from the host).

This feature is mainly intended to support over-committing memory on KVM hosts. That is, hosts that are running VMs whose total memory size is greater than what the host has physically available. For example, a 2G host running two VMs each with 2G would be over-committed.

The balloon device is important to support memory over-commitment because it allows for reducing a guest's memory size if needed. Suppose, in the previous example, that one of the guests is using 1G and its other 1G is free. We could use the balloon device to reduce this guest's size from 2G to 1G, this would free 1G in the host allowing the other VM to use it. Of course, if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again.

That's the problem with the current balloon device, it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. This doesn't work well in practice (if doable at all).

The balloon has to be automatic in order to be really useful. It could like this: when the host is under pressure, it asks guests to relinquish some megas if they can. When/if a guest gets into memory pressure, it gets some megas back from the host. That's what the automatic ballooning series is about.

KVM Forum 2013 presentation slides

They can be found here, but note that much has changed since this talk.

Patches and Git trees

Latest RFC version posted upstream

Guest kernel: RFC PATCH 0/4 virtio_balloon: add pressure notification via a new virtqueue

QEMU: RFC PATCH balloon: add automatic ballooning support

Git trees

Guest kernel:

git://repo.or.cz/linux-2.6/luiz-linux-2.6.git virtio-balloon/pressure-notification/rfc/v1

QEMU:

git://repo.or.cz/qemu/qmp-unstable.git balloon-automatic/handle-all-events/rfc/v1

TODO

More performance tests
Make it dynamic to enable/disable automatic ballooning
Do kernel work required in order to have non-cgroups code using in-kernel memory pressure notification

Testing

Setup:

Install kernel 3.10 or higher in your host. Make sure the kernel options CONFIG_CGROUPS and CONFIG_MEMCG are enabled
Build QEMU from the Git trees section
Build and install the guest kernel from the Git trees section

Then start qemu with:

# qemu [...] -device virtio-balloon,automatic=true

To see automatic ballooning in action, do the following:

Generate pressure in your host (like running several VMs in parallel); or connect to QMP and issue the balloon-inject-host-pressure command
You can monitor automatic ballooning activity by issuing info balloon in QEMU's monitor (also, I have debug code enabled that will dump a lot of info to stdout when "info balloon" is issued)
After the guest has shrinked a bit, you can generate pressure in the guest to see it increasing its size again

@@ Line 1: / Line 1: @@
 = Automatic Ballooning =
+'''NOTE: This page describes an experimental development project from 2013 that was never completed. It is left here as a historical record. The feature described does not exist in any currently shipping version of QEMU'''
 == Introduction ==
-When a Linux host is running out of memory, the kernel will take action to ''reclaim'' memory. This action may be detrimental to KVM guests performace (eg. swapping) or even extreme to the point where the kernel may kill a VM or an important virt stack component.
+The virtio balloon device allows KVM guests to reduce their memory size (thus relinquishing memory to the host) and to increase it back (thus taking memory from the host).
-To help avoiding this situation, a KVM guest could automatically return memory to the host when the host is facing memory pressure (automatic inflate). By doing so the guest may also get into memory pressure so we also need a way to allow the guest to automatically get memory back (automatic deflate).
+This feature is mainly intended to support over-committing memory on KVM hosts. That is, hosts that are running VMs whose total memory size is greater than what the host has physically available. For example, a 2G host running two VMs each with 2G would be over-committed.
-That's what the automatic ballooning project is about. The next section will dive into ways of implementing it. The following section will provide instructions on how to try current patches.
+The balloon device is important to support memory over-commitment because it allows for reducing a guest's memory size if needed. Suppose, in the previous example, that one of the guests is using 1G and its other 1G is free. We could use the balloon device to reduce this guest's size from 2G to 1G, this would free 1G in the host allowing the other VM to use it. Of course, if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again.
-== Design ==
+That's the problem with the current balloon device, it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. This doesn't work well in practice (if doable at all).
-=== Automatic Inflate ===
+The balloon has to be automatic in order to be really useful. It could like this: when the host is under pressure, it asks guests to relinquish some megas if they can. When/if a guest gets into memory pressure, it gets some megas back from the host. That's what the automatic ballooning series is about.
-Automatic inflate is performed by QEMU, that is, the KVM host. There are two ways I'm playing with automatic inflate, in both of them QEMU registers for [http://lwn.net/Articles/544652/ memory pressure] events to learn when the host is under memory pressure:
+=== KVM Forum 2013 presentation slides ===
-# QEMU registers for the VMPRESSURE_MEDIUM event and inflates the balloon by an user specified value when the event is received. That value value could default to 16MB or 32MB. This is what the current patchset does
+[https://www.linux-kvm.org/images/5/58/Kvm-forum-2013-automatic-ballooning.pdf They can be found here], but note that much has changed since this talk.
-# QEMU registers for the three memory pressure events and inflates the balloon accordingly. Say, it could inflate the balloon by 1MB on VMPRESSURE_LOW; 16MB on VMPRESSURE_MEDIUM and 128MB on VMPRESSURE_CRITICAL. This is not done yet because there's a small limitation in the memory pressure code which doesn't allow this. I'll fix it
-=== Automatic Deflate ===
+== Patches and Git trees ==
-Automatic deflate is performed by the virtio-balloon driver within the guest. There's also two ways of implementing this:
+=== Latest RFC version posted upstream ===
-# The virtio-balloon driver registers a callback with the shrinker API. That callback is called when the guest kernel is facing memory pressure. The number of pages to be returned to the kernel is passed to the callback, so that the callback can deflate the balloon by that amount. This is what the current patchset does
+'''Guest kernel:''' [http://marc.info/?l=kvm&m=138988948715638&w=2 RFC PATCH 0/4 virtio_balloon: add pressure notification via a new virtqueue]
-# The virtio-balloon driver registers for an in-kernel memory pressure event (not upstream yet) and deflates the balloon by some fixed amount (maybe the same amount used in automatic inflate)
+'''QEMU:''' [http://marc.info/?l=kvm&m=138988966315690&w=2 RFC PATCH balloon: add automatic ballooning support]
+=== Git trees ===
+'''Guest kernel:'''
+[http://repo.or.cz/w/linux-2.6/luiz-linux-2.6.git git://repo.or.cz/linux-2.6/luiz-linux-2.6.git] virtio-balloon/pressure-notification/rfc/v1
+'''QEMU:'''
+[http://repo.or.cz/w/qemu/qmp-unstable.git git://repo.or.cz/qemu/qmp-unstable.git] balloon-automatic/handle-all-events/rfc/v1
+== TODO ==
+* More performance tests
+* Make it dynamic to enable/disable automatic ballooning
+* Do kernel work required in order to have non-cgroups code using in-kernel memory pressure notification
 == Testing ==
-You have to do three things to play with automatic-ballooning:
+Setup:
-# Install kernel 3.10-rc1 or higher in the host. Make sure to enable CONFIG_CGROUPS and CONFIG_MEMCG
+# Install kernel 3.10 or higher in your host. Make sure the kernel options CONFIG_CGROUPS and CONFIG_MEMCG are enabled
-# Clone QEMU from git://repo.or.cz/qemu/qmp-unstable.git balloon/auto-ballooning/rfc.v2 (or grab the patch [http://repo.or.cz/w/qemu/qmp-unstable.git/commit/59fec94e6396d6c32cdce47777440c3a988be63d here])
+# Build QEMU from [[#Git trees| the Git trees section]]
-# Install the following kernel on your guest git://repo.or.cz/linux-2.6/luiz-linux-2.6.git virtio-balloon/auto-deflate/rfc (or grab the patches from the [http://repo.or.cz/w/linux-2.6/luiz-linux-2.6.git/shortlog/refs/heads/virtio-balloon/auto-deflate/rfc web interface])
+# Build and install the guest kernel from [[#Git trees| the Git trees section]]
-After setting up the above, do the following to experiment with automatic deflate (which is easy to reproduce):
+Then start qemu with:
-# Pass -balloon virtio,auto-balloon=true when starting QEMU
+<nowiki>   # qemu [...] -device virtio-balloon,automatic=true</nowiki>
-# Wait for the guest to boot, then generate some memory pressure in the guest (say a kernel build with lots of jobs)
-# Switch to QEMU's monitor and decrease guest memory (say from 1G to 200MB)
-# Watch the guest increase its memory by repeatedly running "info balloon" on QEMU's monitor
-To see automatic inflate and automatic deflate in action, you can run several VMs in parallel doing some heavy memory workload. Make sure to over commit host memory. Say your host has 4GB, run 6 VMs with 1GB each.
+To see automatic ballooning in action, do the following:
-Another idea is to run one VM, but generate pressure in the host some way like a big kernel build or an application that allocates lots of memory.
+# Generate pressure in your host (like running several VMs in parallel); or connect to QMP and issue the ''balloon-inject-host-pressure'' command
+# You can monitor automatic ballooning activity by issuing ''info balloon'' in QEMU's monitor (also, I have debug code enabled that will dump a lot of info to stdout when "info balloon" is issued)
+# After the guest has shrinked a bit, you can generate pressure in the guest to see it increasing its size again

Projects/auto-ballooning: Difference between revisions