Projects/auto-ballooning - Revision history

DanielBerrange: /* Automatic Ballooning */

2018-08-23T10:51:19Z

Automatic Ballooning

← Older revision		Revision as of 06:51, 23 August 2018
Line 1:		Line 1:
	= Automatic Ballooning =		= Automatic Ballooning =

			'''NOTE: This page describes an experimental development project from 2013 that was never completed. It is left here as a historical record. The feature described does not exist in any currently shipping version of QEMU'''
	== Introduction ==		== Introduction ==

D1I1zB: fix broken link

2018-07-16T04:39:00Z

fix broken link

← Older revision		Revision as of 00:39, 16 July 2018
Line 15:		Line 15:
	=== KVM Forum 2013 presentation slides ===		=== KVM Forum 2013 presentation slides ===

	~~They can be found~~ [~~http~~://www.linux-kvm.org~~/wiki~~/images/f/f6/~~Automatic~~-ballooning~~-slides~~.pdf here], but note that much has changed since this talk.		[https://www.linux-kvm.org/images/5/58/Kvm-forum-2013-automatic-ballooning.pdf They can be found here], but note that much has changed since this talk.

	== Patches and Git trees ==		== Patches and Git trees ==

LuizCapitulino at 20:49, 23 January 2014

2014-01-23T20:49:37Z

← Older revision		Revision as of 16:49, 23 January 2014
Line 29:		Line 29:
	'''Guest kernel:'''		'''Guest kernel:'''

	[http://repo.or.cz/w/linux-2.6/luiz-linux-2.6.git git://repo.or.cz/linux-2.6/luiz-linux-2.6.git] virtio-balloon/pressure-notification/rfc.v1		[http://repo.or.cz/w/linux-2.6/luiz-linux-2.6.git git://repo.or.cz/linux-2.6/luiz-linux-2.6.git] virtio-balloon/pressure-notification/rfc/v1

	'''QEMU:'''		'''QEMU:'''

	[http://repo.or.cz/w/qemu/qmp-unstable.git git://repo.or.cz/qemu/qmp-unstable.git] balloon-automatic/handle-all-events/rfc.v1		[http://repo.or.cz/w/qemu/qmp-unstable.git git://repo.or.cz/qemu/qmp-unstable.git] balloon-automatic/handle-all-events/rfc/v1

	== TODO ==		== TODO ==

LuizCapitulino at 18:32, 20 January 2014

2014-01-20T18:32:09Z

← Older revision		Revision as of 14:32, 20 January 2014
Line 34:		Line 34:

	[http://repo.or.cz/w/qemu/qmp-unstable.git git://repo.or.cz/qemu/qmp-unstable.git] balloon-automatic/handle-all-events/rfc.v1		[http://repo.or.cz/w/qemu/qmp-unstable.git git://repo.or.cz/qemu/qmp-unstable.git] balloon-automatic/handle-all-events/rfc.v1

			== TODO ==

			* More performance tests
			* Make it dynamic to enable/disable automatic ballooning
			* Do kernel work required in order to have non-cgroups code using in-kernel memory pressure notification

	== Testing ==		== Testing ==

LuizCapitulino at 18:30, 20 January 2014

2014-01-20T18:30:24Z

← Older revision		Revision as of 14:30, 20 January 2014
Line 3:		Line 3:
	== Introduction ==		== Introduction ==

	The balloon ~~feature~~ allows KVM guests to reduce their memory size (thus relinquishing memory to the host) and to increase back (thus taking memory from the host).		The virtio balloon device allows KVM guests to reduce their memory size (thus relinquishing memory to the host) and to increase it back (thus taking memory from the host).

	This feature is mainly intended to support ~~memory~~ over-~~committed~~ hosts. That is, hosts that are running VMs whose total memory size is greater than what the host has physically available. For example, a 2G host running two VMs each with 2G would be over-committed.		This feature is mainly intended to support over-committing memory on KVM hosts. That is, hosts that are running VMs whose total memory size is greater than what the host has physically available. For example, a 2G host running two VMs each with 2G would be over-committed.

	The balloon ~~feature~~ is important to support memory over-commitment because it allows for reducing a guest's memory size if needed. Suppose, in the example ~~above~~, that one of the guests is using 1G and its other 1G is free. We could use the balloon to reduce this guest's size from 2G to 1G, this would free 1G in the host allowing the other VM to use it. Of course, if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again.		The balloon device is important to support memory over-commitment because it allows for reducing a guest's memory size if needed. Suppose, in the previous example, that one of the guests is using 1G and its other 1G is free. We could use the balloon device to reduce this guest's size from 2G to 1G, this would free 1G in the host allowing the other VM to use it. Of course, if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again.

	That's the problem with the current balloon ~~feature~~, it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. This ~~is just not doable~~ in practice.		That's the problem with the current balloon device, it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. This doesn't work well in practice (if doable at all).

	The balloon has to be automatic in order to be really useful. It could like this: when the host is under pressure, ~~VMs with spare memory automatically reduce their size by~~ some megas. When a ~~VM get~~ into memory pressure ~~(maybe because they relinquished memory to host)~~ it ~~increases its size by~~ some megas. That's ~~exactly~~ what the automatic ballooning ~~patch~~ series ~~does~~.		The balloon has to be automatic in order to be really useful. It could like this: when the host is under pressure, it asks guests to relinquish some megas if they can. When/if a guest gets into memory pressure, it gets some megas back from the host. That's what the automatic ballooning series is about.

	=== KVM Forum 2013 presentation slides ===		=== KVM Forum 2013 presentation slides ===

LuizCapitulino at 20:44, 16 January 2014

2014-01-16T20:44:24Z

← Older revision		Revision as of 16:44, 16 January 2014
Line 15:		Line 15:
	=== KVM Forum 2013 presentation slides ===		=== KVM Forum 2013 presentation slides ===

	They can be found [http://www.linux-kvm.org/wiki/images/f/f6/Automatic-ballooning-slides.pdf here]~~. But~~ note that much has changed since this talk.		They can be found [http://www.linux-kvm.org/wiki/images/f/f6/Automatic-ballooning-slides.pdf here], but note that much has changed since this talk.

	== Patches and Git trees ==		== Patches and Git trees ==

LuizCapitulino at 20:39, 16 January 2014

2014-01-16T20:39:10Z

← Older revision		Revision as of 16:39, 16 January 2014
Line 29:		Line 29:
	'''Guest kernel:'''		'''Guest kernel:'''

	git://repo.or.cz/linux-2.6/luiz-linux-2.6.git virtio-balloon/pressure-notification/rfc.v1		[http://repo.or.cz/w/linux-2.6/luiz-linux-2.6.git git://repo.or.cz/linux-2.6/luiz-linux-2.6.git] virtio-balloon/pressure-notification/rfc.v1

	'''QEMU:'''		'''QEMU:'''

	git://repo.or.cz/qemu/qmp-unstable.git balloon-automatic/handle-all-events/rfc.v1		[http://repo.or.cz/w/qemu/qmp-unstable.git git://repo.or.cz/qemu/qmp-unstable.git] balloon-automatic/handle-all-events/rfc.v1

	== Testing ==		== Testing ==

LuizCapitulino at 20:35, 16 January 2014

2014-01-16T20:35:26Z

← Older revision		Revision as of 16:35, 16 January 2014
Line 21:		Line 21:
	=== Latest RFC version posted upstream ===		=== Latest RFC version posted upstream ===

	'''Guest kernel:''' [http://marc.info/?l=kvm&m=138988948715638&w=2 [RFC PATCH 0/4] virtio_balloon: add pressure notification via a new virtqueue]		'''Guest kernel:''' [http://marc.info/?l=kvm&m=138988948715638&w=2 RFC PATCH 0/4 virtio_balloon: add pressure notification via a new virtqueue]

	'''QEMU:''' [http://marc.info/?l=kvm&m=138988966315690&w=2 RFC PATCH balloon: add automatic ballooning support]		'''QEMU:''' [http://marc.info/?l=kvm&m=138988966315690&w=2 RFC PATCH balloon: add automatic ballooning support]
Line 49:		Line 49:
	To see automatic ballooning in action, do the following:		To see automatic ballooning in action, do the following:

	# ~~Connect~~ to QMP and issue the balloon-inject-host-pressure command~~. Or,~~		# Generate pressure in your host (like running several VMs in parallel); or connect to QMP and issue the ''balloon-inject-host-pressure'' command
	~~generate pressure in your host (like running several VMs in parallel)~~		# You can monitor automatic ballooning activity by issuing ''info balloon'' in QEMU's monitor (also, I have debug code enabled that will dump a lot of info to stdout when "info balloon" is issued)
	# You can monitor automatic ballooning activity by issuing "info balloon" in QEMU's monitor (also, I have debug code enabled that will dump a lot of info to stdout when "info balloon" is issued)		# After the guest has shrinked a bit, you can generate pressure in the guest to see it increasing its size again
	# After guest has shrinked a bit, you can generate pressure in the guest to see it increasing its size again

LuizCapitulino at 20:31, 16 January 2014

2014-01-16T20:31:29Z

← Older revision		Revision as of 16:31, 16 January 2014
Line 37:		Line 37:
	== Testing ==		== Testing ==

	~~You have to setup the following before experimenting with auto-ballooning~~:		Setup:

	# Install kernel 3.10 or higher in your host. Make sure the kernel options CONFIG_CGROUPS and CONFIG_MEMCG are enabled		# Install kernel 3.10 or higher in your host. Make sure the kernel options CONFIG_CGROUPS and CONFIG_MEMCG are enabled
	# Build ~~and install~~ QEMU from [[#Git trees\| the Git trees section]] ~~(suggest the RFC one)~~		# Build QEMU from [[#Git trees\| the Git trees section]]
	# Build and install the guest kernel from [[#Git trees\| the Git trees section]] ~~(suggest the RFC one)~~		# Build and install the guest kernel from [[#Git trees\| the Git trees section]]

	~~After setting up the above, do the following to experiment~~ with ~~automatic deflate~~:		Then start qemu with:

	# ~~Pass ''~~-~~balloon~~ virtio~~,auto~~-balloon=true~~'' when starting QEMU~~		<nowiki> # qemu [...] -device virtio-balloon,automatic=true</nowiki>
	~~# Wait for the guest to boot, then generate some memory pressure in the guest (say a kernel build with lots of jobs)~~
	~~# Switch to QEMU's monitor and shrink the guest (say from 1G to 200MB)~~
	~~# Watch the guest increase its memory by running "free" within the guest (or "info balloon" in QEMU)~~

	~~There are two ways to play with~~ automatic ~~inflate~~:		To see automatic ballooning in action, do the following:

	# ~~The simplest thing is~~ to ~~create a memory constrained cgroup~~, ~~but this will require code changes~~ in QEMU ~~because it~~'s ~~hardcoded to use the root cgroup~~ (~~yes~~, ~~we need~~ a ~~command-line option for that~~)		# Connect to QMP and issue the balloon-inject-host-pressure command. Or,
	# ~~If you don't want to play with cgroups~~, you can ~~overcommit your host by running several VMs~~ in ~~parallel. The VMs have~~ to ~~run a heavy memory workload~~		generate pressure in your host (like running several VMs in parallel)
			# You can monitor automatic ballooning activity by issuing "info balloon" in QEMU's monitor (also, I have debug code enabled that will dump a lot of info to stdout when "info balloon" is issued)
			# After guest has shrinked a bit, you can generate pressure in the guest to see it increasing its size again

LuizCapitulino at 20:23, 16 January 2014

2014-01-16T20:23:30Z

← Older revision		Revision as of 16:23, 16 January 2014
Line 7:		Line 7:
	This feature is mainly intended to support memory over-committed hosts. That is, hosts that are running VMs whose total memory size is greater than what the host has physically available. For example, a 2G host running two VMs each with 2G would be over-committed.		This feature is mainly intended to support memory over-committed hosts. That is, hosts that are running VMs whose total memory size is greater than what the host has physically available. For example, a 2G host running two VMs each with 2G would be over-committed.

	The balloon feature is important to support memory over-commitment because it allows for reducing a guest's memory size if needed. Suppose, in the example above, that one of the guests is using 1G and its other 1G is free. We could use the balloon to reduce this guest's size to 1G, this would free 1G in the host allowing the other VM to use it. Of course, if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again.		The balloon feature is important to support memory over-commitment because it allows for reducing a guest's memory size if needed. Suppose, in the example above, that one of the guests is using 1G and its other 1G is free. We could use the balloon to reduce this guest's size from 2G to 1G, this would free 1G in the host allowing the other VM to use it. Of course, if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again.

	That's the problem with the current balloon feature, it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. This is just not doable in practice.		That's the problem with the current balloon feature, it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. This is just not doable in practice.

	The balloon has to be automatic in order to be really useful. It could like this: when the host is under pressure, VMs with spare memory automatically reduce their size by some megas. When a VM get into pressure (maybe because they relinquished memory to host) it increases its size by some megas. That's exactly what the automatic ballooning ~~feature~~ does.		The balloon has to be automatic in order to be really useful. It could like this: when the host is under pressure, VMs with spare memory automatically reduce their size by some megas. When a VM get into memory pressure (maybe because they relinquished memory to host) it increases its size by some megas. That's exactly what the automatic ballooning patch series does.

	~~== Design ==~~

	~~KVM guests have a driver called the balloon driver. This driver allows guests to shrink and grow their memory. The balloon driver supports two operations:~~

	* Inflate: memory is taken from the guest and given to the host (guest shrinks)
	* Deflate: memory is returned from the host to the guest (guest grows)

	~~Today, both operations are manual. The automatic ballooning project is about making them completely automatic, based on host and guest needs~~.

	=== KVM Forum 2013 presentation slides ===		=== KVM Forum 2013 presentation slides ===

	They can be found [http://www.linux-kvm.org/wiki/images/f/f6/Automatic-ballooning-slides.pdf here]		They can be found [http://www.linux-kvm.org/wiki/images/f/f6/Automatic-ballooning-slides.pdf here]. But note that much has changed since this talk.

	==~~= Automatic Inflate =~~==		== Patches and Git trees ==

	~~Automatic inflate is performed by QEMU (ie. the KVM host). QEMU registers for [http://lwn.net/Articles/544652/ memory pressure] events so that it's notified when the host is under memory pressure.~~		=== Latest RFC version posted upstream ===

	~~Current patches have pre-defined values to be used by QEMU when it receives a memory pressure notification from the host kernel. Those values are:~~

	* 1MB on LOW pressure		'''Guest kernel:''' [http://marc.info/?l=kvm&m=138988948715638&w=2 [RFC PATCH 0/4] virtio_balloon: add pressure notification via a new virtqueue]
	* 2MB on MEDIUM pressure
	* 4MB on CRITICAL pressure

	~~For example, suppose the host is facing MEDIUM pressure and notifies~~ QEMU. ~~When QEMU receives the event, it asks the guest to inflate its~~ balloon ~~by 2MB. The guest in turn will shrink itself by 2MB and give that memory to the host.~~		'''QEMU:''' [http://marc.info/?l=kvm&m=138988966315690&w=2 RFC PATCH balloon: add automatic ballooning support]

	=~~== Automatic Deflate ===~~		=== Git trees ===

	~~Automatic deflate is performed by the guest or, more specifically, by the balloon driver.~~

	The virtio-balloon driver registers a callback with the shrinker API. That callback is called when the guest kernel is facing memory pressure, and the number of pages to be returned to the kernel is passed to the callback. The balloon driver shrink callback deflates the guest's balloon.

	For example, suppose the guest shrunk itself because of pressure in the host. But after some time, the guest is running more applications which causes memory pressure in the guest due its reduced size. The guest kernel then starts reclaiming memory and calls all shrink callbacks. That's when the balloon driver's shrink callback runs, and deflates the balloon by the number of pages specified. This causes the guest to grow again.

	== Git trees ==

	~~=== Latest RFC version posted upstream~~ ===

	~~'''QEMU:'''~~

	~~git://repo.or.cz/qemu/qmp-unstable.git balloon/auto-ballooning/rfc.v2 (or grab the patch [http://repo.or.cz/w/qemu/qmp-unstable.git/commit/59fec94e6396d6c32cdce47777440c3a988be63d here])~~

	'''Guest kernel:'''		'''Guest kernel:'''

	git://repo.or.cz/linux-2.6/luiz-linux-2.6.git virtio-balloon/~~auto~~-~~deflate~~/rfc ~~(or grab the first two patches from the [http://repo.or~~.~~cz/w/linux-2.6/luiz-linux-2.6.git/shortlog/refs/heads/virtio-balloon/auto-deflate/rfc web interface])~~		git://repo.or.cz/linux-2.6/luiz-linux-2.6.git virtio-balloon/pressure-notification/rfc.v1

	~~=== Development branches ===~~

	'''QEMU:'''		'''QEMU:'''

	git://repo.or.cz/qemu/qmp-unstable.git balloon/~~auto~~-~~ballooning/current~~		git://repo.or.cz/qemu/qmp-unstable.git balloon-automatic/handle-all-events/rfc.v1

	~~'''Guest kernel:'''~~

	~~git://repo.or.cz/linux~~-~~2.6~~/~~luiz-linux-2.6~~.~~git virtio-balloon/auto-deflate/current~~

	== Testing ==		== Testing ==