KVM - User contributions [en]

PowerPC Exittimings

2010-07-15T17:28:14Z

HollisBlanchard: kvmppc_timing.py was never imported from the old wiki; it's gone now

== About exit timing ==

PowerPC KVM exports information to debugfs about the exits caused by virtualization and the time consumed by them. This data can typically be found as <code>/sys/kernel/debug/kvm/vm*_vcpu*_timing</code>.

Because the PowerPC hardware currently supported by KVM has no hardware virtualization support, the guest must run in non-privileged mode. When the guest executes a privileged instruction, the KVM host must emulate its behavior, and this emulation time is accounted for as EMULINST. (It can be expected that upcoming hardware extension reduce most of these emulation exits as the guest can then run in privileged mode.)

Another frequent exit reason is caused by the memory/TLB management. Because a guest can not be allowed to program the real TLB (privileged commands and it would be an isolation violation anyway), the host has to track the state of the guest TLB and recover TLB faults caused because the guest is virtualized. Such a kind of TLB interrupts caused by virtualization is called [DI]TLBVIRT in the exit statistics. If the guest TLB tracked by the host does not have a mapping for the address reported by a TLB exception it is delivered to the guest as it is done on bare metal. This appears as [DI]TLBREAL in the exit statistics.

When the guest idles, it will enter "wait mode" until an interrupt is delivered. This time is accounted for by exit type of WAIT.

The other exits are less frequent like MMIO, DCR and SIGNAL which need to exit to kvm-userspace to be handled. The only value not really being an exit is the TIMEINGUEST which is the time in the guest.

The timing statistic itself tracks exit and reenter time as well as the type of the exit. Then the duration exit -> enter is accounted for the specific exit type while enter -> exit is accounted for the TIMEINGUEST values.

'''Note:''' All times are in usec's.

== workloads and performance expectations ==

It can be expected that loads causing a high number of exits have a high overhead while loads that run in guest with only a few interceptions should be fine. Those loads with high exit ratios can be for example a guest booting and initializing all its virtual hardware (EMULINST), as well as load that creates memory pressure and therefore causes a lot of virtual TLB misses ([DI]TLBVIRT).

The following measurements are taken on a 440epx (Sequoia) board. Thins means running an unmodified guests on Hardware without virtualization support. therefore a lot of overhead can be expected. The following statistics give you an idea which exit reasons are frequent dependeing on the workload. And as mentioned above you can think what happens once you run that on virtualization powerpc Hardware coming with Power ISA 2.06 (http://en.wikipedia.org/wiki/Power_Architecture#Power_ISA_v.2.06).

The following sections describe the workloads shown on this page.
[[Image:workload_classification.gif]]

=== boot guest ===

The workload traces a guest from the initialization until you see a boot prompt. Although being simple, this workload is useful to get a load with a high amount of privileged instructions.

=== Hashes ===
This is a custom written small program calculating a lot of hashes. It uses the hash algorithm used in the Berkley db and calculates hash of hash of and so on.
This program represents a workload that has only a few I/O and privileged instructions and therefore has only a low virtualization overhead.
attachment:foohash.c
To execute it just call the binary without options:
<code>
./foohash
</code>

=== Fhourstone ===
The Fhourstone benchmark (http://homepages.cwi.nl/~tromp/c4/fhour.html) uses a 64Mb transposition table to solve positions of the game connect-4. In a small environment as the sequoia board is this is a high amount of memory pressure and therefore it is expected that it causes a lot of TLB exits. After compiling the sources you get a binary called SearchGame and a file called "inputs" which represents the workload. The benchmark is invoked with:
<code>
SearchGame < inputs
</code>

=== Lame ===
Lame (http://lame.sourceforge.net/) is the known mp3 encoder and in the workload testcase used to convert a file in very high quality (option --preset insane). This workload has to do some I/O, as well as a lot of calculations that should not exit to the hypervisor. Therefore it can be expected that lame is a good example of average workload.
The wav file used is from a free sample page in the web (http://bellatrix.ece.mcgill.ca/Documents/AudioFormats/WAVE/Samples.html). We used the M1F1-int32-AFsp.wav from that page using the "insane" preset to get a max quality mp3.
<code>
lame --preset insane M1F1-int32-AFsp.wav M1F1-int32-AFsp.mp3
</code>

=== find ===

At last using find to find a file and search for it all over the disk is a simple workload using only a few CPU calculations but requiring a lot of I/O.
As testcase find is executed on root to search for the wav file we use in the Lame testcase, but with a wildcard. The disk we have is the ELDK ramdisk plus the files for the workloads listed here. To be fair and use the same file system the bare metal test mounts the same as loop device.
<code>
cd /
time find -name "*.wav*"
</code>

== Performance results ==
This Section lists at which % of the native bare metal speed these tests run on the current kvmppc implementation. As described below there are alerady known options for improvement like paravirtualization. The tests are run on the source level on 11. November 2008 which included some new performance improvements in memory management, interrupt delivery as well as several minor improvements.
The tests are run on a Host and Guest with 64k page size. The Host uses a nfs mount as root file system while the guest is using virtio to access a disk image placed on the host root nfs mount.

{|
|'''workload'''
|'''% of bae metal speed'''
|-

| hashes
| 96.49%
|-

| lame
| 84.47%
|-

| boot
| ~80%
|-

| find
| 6.11%
|-

| fhourstone
| 5.96%
|}

'''Note:''' network latency after the virtio indirection might be a big issue for the find testcase so treat the numbers unfinished until we verified that number it on e.g. a local usb stick.

'''Note:''' the time accounted for MMIO is the time a guest exits and KVM prepares the mmio until it returns to the guest. It is not the time until the IO arrives and is ready for the guest. Additional IO performance data may be obtained by running blktrace on the virtio disk inside the guest.

== Timings results ==

The following tables show the results of the exit timing analysis using the 5 different workloads mentioned above.

=== boot guest ===
<code>
sum of time 8309940
type count min max sum avg stddev %
MMIO: 9402 44 1997 1697610 180.5584 155.768 20.43
DCR: 680 41 99 32096 47.2000 7.008 0.39
SIGNAL: 1 98 98 98 98.0000 0.000 0.00
ITLBREAL: 926 8 14 7810 8.4341 0.658 0.09
ITLBVIRT: 3595 18 202 76185 21.1919 4.954 0.92
DTLBREAL: 950 8 16 8891 9.3589 1.427 0.11
DTLBVIRT: 6695 18 282 156727 23.4096 13.781 1.89
SYSCALL: 1801 6 59 11372 6.3143 2.575 0.14
ISI: 116 6 8 764 6.5862 0.588 0.01
DSI: 43 6 7 292 6.7907 0.407 0.00
EMULINST: 65247 7 96 484081 7.4192 1.818 5.83
EMUL_WAIT: 801 659 9200 3721789 4646.4282 1687.218 44.79
EMUL_CORE: 66806 7 86 540053 8.0839 1.895 6.50
EMUL_MTSPR: 13415 8 62 111358 8.3010 2.583 1.34
EMUL_MFSPR: 7635 8 61 62772 8.2216 1.921 0.76
EMUL_MTMSR: 5678 8 59 45704 8.0493 1.434 0.55
EMUL_MFMSR: 32853 7 67 267603 8.1455 1.875 3.22
EMUL_TLBSX: 354 9 60 3745 10.5791 3.919 0.05
EMUL_TLBWE: 6403 9 112 99522 15.5430 7.668 1.20
EMUL_RFI: 9515 7 57 71420 7.5060 2.108 0.86
DEC: 290 49 161 15786 54.4345 9.708 0.19
EXTINT: 7 74 75 522 74.5714 0.495 0.01
TIMEINGUEST: 233213 0 3954 893740 3.8323 65.837 10.76
</code>

=== Hashes ===
<code>
sum of time 21576367
type count min max sum avg stddev %
MMIO: 827 49 6700 224632 271.6227 259.231 1.04
DCR: 161 42 94 7468 46.3851 4.314 0.03
SIGNAL: 2 291 1214 1505 752.5000 461.500 0.01
ITLBREAL: 53 8 12 445 8.3962 0.682 0.00
ITLBVIRT: 216 19 68 4566 21.1389 3.346 0.02
DTLBREAL: 44 9 16 420 9.5455 1.738 0.00
DTLBVIRT: 407 19 73 8706 21.3907 3.687 0.04
SYSCALL: 66 6 7 428 6.4848 0.500 0.00
ISI: 5 6 8 34 6.8000 0.748 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 67009 6 97 508311 7.5857 1.247 2.36
EMUL_WAIT: 231 1254 8902 1074304 4650.6667 1699.150 4.98
EMUL_CORE: 32964 7 59 262866 7.9743 0.622 1.22
EMUL_MTSPR: 9201 8 14 74751 8.1242 0.339 0.35
EMUL_MFSPR: 379 8 60 3134 8.2691 2.686 0.01
EMUL_MTMSR: 4749 8 9 37996 8.0008 0.029 0.18
EMUL_MFMSR: 14257 7 55 114282 8.0159 0.776 0.53
EMUL_TLBSX: 18 9 15 185 10.2778 1.193 0.00
EMUL_TLBWE: 393 9 69 5477 13.9364 6.905 0.03
EMUL_RFI: 9006 7 57 67271 7.4696 0.722 0.31
DEC: 5065 49 269 267567 52.8267 13.544 1.24
EXTINT: 2 77 451 528 264.0000 187.000 0.00
TIMEINGUEST: 145056 0 3954 18911484 130.3737 678.943 87.65
</code>

=== Lame ===
<code>
sum of time 6592939
type count min max sum avg stddev %
MMIO: 1827 48 18883 550073 301.0799 772.936 8.34
DCR: 392 42 1074 22162 56.5357 83.884 0.34
SIGNAL: 1 1812 1812 1812 1812.0000 0.000 0.03
ITLBREAL: 142 8 13 1200 8.4507 0.623 0.02
ITLBVIRT: 1860 18 118 39336 21.1484 4.514 0.60
DTLBREAL: 164 8 66 1707 10.4085 4.885 0.03
DTLBVIRT: 2724 18 1039 63063 23.1509 23.705 0.96
SYSCALL: 255 6 8 1626 6.3765 0.531 0.02
ISI: 17 6 8 114 6.7059 0.570 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 26682 6 161 203151 7.6138 2.885 3.08
EMUL_WAIT: 261 501 7683 1211114 4640.2835 1683.305 18.37
EMUL_CORE: 19247 7 161 158089 8.2137 4.052 2.40
EMUL_MTSPR: 4309 8 161 35697 8.2843 2.371 0.54
EMUL_MFSPR: 1238 8 61 10395 8.3966 3.510 0.16
EMUL_MTMSR: 2098 8 61 17045 8.1244 2.403 0.26
EMUL_MFMSR: 9158 7 163 75613 8.2565 3.392 1.15
EMUL_TLBSX: 36 9 13 364 10.1111 0.698 0.01
EMUL_TLBWE: 1062 9 75 17011 16.0179 7.539 0.26
EMUL_RFI: 3736 7 142 28230 7.5562 2.974 0.43
DEC: 1109 49 263 61117 55.1100 15.174 0.93
EXTINT: 36 52 1377 12085 335.6944 308.306 0.18
TIMEINGUEST: 76355 0 3954 4081928 53.4599 415.345 61.91
</code>

=== Fhourstone ===
<code>
sum of time 7483768
type count min max sum avg stddev %
MMIO: 818 47 9565 221609 270.9156 501.827 2.96
DCR: 301 40 473 14408 47.8671 29.403 0.19
SIGNAL: 1 2521 2521 2521 2521.0000 0.000 0.03
ITLBREAL: 322 8 58 2665 8.2764 2.810 0.04
ITLBVIRT: 5773 18 1360 120111 20.8056 18.569 1.60
DTLBREAL: 16433 8 73 184196 11.2089 3.709 2.46
DTLBVIRT: 19913 18 1845 500349 25.1268 23.006 6.69
SYSCALL: 91 6 7 579 6.3626 0.481 0.01
ISI: 5 6 8 33 6.6000 0.800 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 127113 6 102 949687 7.4712 2.244 12.69
EMUL_WAIT: 76 3526 7578 354928 4670.1053 1679.367 4.74
EMUL_CORE: 16733 7 159 134306 8.0264 1.701 1.79
EMUL_MTSPR: 71886 8 149 594618 8.2717 2.595 7.95
EMUL_MFSPR: 83877 8 93 689016 8.2146 2.580 9.21
EMUL_MTMSR: 3166 8 57 25438 8.0347 1.220 0.34
EMUL_MFMSR: 7392 7 61 59415 8.0377 1.294 0.79
EMUL_TLBSX: 19 9 11 192 10.1053 0.640 0.00
EMUL_TLBWE: 47187 9 2757 1131527 23.9796 13.359 15.12
EMUL_RFI: 21782 7 132 172428 7.9161 2.323 2.30
DEC: 715 50 224 42155 58.9580 15.052 0.56
EXTINT: 2 142 161 303 151.5000 9.500 0.00
TIMEINGUEST: 423606 0 499 2283277 5.3901 25.629 30.51
</code>

=== find ===
<code>
sum of time 3426052
type count min max sum avg stddev %
MMIO: 222 49 413 48228 217.2432 141.096 1.41
DCR: 91 43 93 4239 46.5824 5.285 0.12
SIGNAL: 3 476 5651 7952 2650.6667 2191.871 0.23
ITLBREAL: 77 8 13 665 8.6364 0.754 0.02
ITLBVIRT: 1341 18 120 28340 21.1335 4.968 0.83
DTLBREAL: 59 8 16 573 9.7119 2.042 0.02
DTLBVIRT: 2253 19 214 48630 21.5846 7.083 1.42
SYSCALL: 4590 6 57 29503 6.4277 2.114 0.86
ISI: 11 6 8 72 6.5455 0.656 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 71560 6 77 525976 7.3501 1.945 15.35
EMUL_WAIT: 374 184 9384 1724701 4611.5000 1752.946 50.34
EMUL_CORE: 32646 7 100 262449 8.0392 1.792 7.66
EMUL_MTSPR: 6668 8 78 54829 8.2227 2.213 1.60
EMUL_MFSPR: 538 8 61 4507 8.3773 3.181 0.13
EMUL_MTMSR: 5829 8 58 47036 8.0693 1.765 1.37
EMUL_MFMSR: 15805 7 92 127426 8.0624 1.774 3.72
EMUL_TLBSX: 29 9 14 297 10.2414 0.857 0.01
EMUL_TLBWE: 462 9 27 6717 14.5390 6.606 0.20
EMUL_RFI: 10855 7 57 81776 7.5335 2.132 2.39
DEC: 160 50 403 9244 57.7750 32.861 0.27
EXTINT: 4 427 1410 2991 747.7500 387.275 0.09
TIMEINGUEST: 153578 0 762 409894 2.6690 6.322 11.96
</code>

== Paravirtualization improvement ==

As mentioned above improvements to all these overhead statistics are already known. On one hand the hardware virtualization support specified in the Power ISA 2.06, on the other hand on older hardware paravirtualization can be an option. For KVMPPC we wrote a simple paravirtualization interface to test hypercalls and measure some benefits from such implementations.
In the concept measured here the hypervisor tells the guest that it supports special paravirtualization features if the guest (hyper)calls him passing a guest virtual and guest physical address and an amount of x (4096byte in the example) of ram big. This is a very basic, but also flexible interface as the hypervisor can now use this guest addressable memory to do all kind of things. In the example the hypervisor rewrites guest code to replace privileged instructions mfspr (SPRG1-3, ESR, DEAR) mtspr ((SPRG1-3, ESR, DEAR) and mfmsr. The hypervisor keeps the guest copies of these values updated in the memory area provided by the guest and rewrites the privileged instructions to simple non trapping load/store instructions. That save a lot of EMULINST exits while running the guest and has shown around 35-50% improvement for workloads with a high amount of EMULINST instructions (e.g. the boot workload)

The net benefit of that improvement is high while the "guest invasiveness" is very low (The guest only has to donate a small amount of ram and virtual address space, all the other things can be done by the hypervisor transparently). And remember this is just one simple example of pv extensions, there are a lot other areas e.g. collaborative guest/host TLB management that can improve performance significantly (could be as easy as telling the guest to program more virtual TLB entries in the guest TLB to allow the host fix more misses directly).

The following numbers show the improvement comparing the same workload with/without this paravirtualization feature. The workload used in this test is the boot workload, but using a more complex guest and Host environment (Ubuntu instead of ELDK) and it also uses an older version of our kernel and userspace code (also containing an older version of that exit timing and therefore not being directly comparable with the measurements above).

=== No paravirtualization ===
<code>
sum of time 144837890 => ~2:24 runtime
count min max sum avg stddev %
MMIO 10105 46 1534 2952467 292.17 295.933 2.04
DCR 5428 40 209 246168 45.35 6.758 0.17
SIGNAL 695 65 3755 89767 129.16 314.421 0.06
ITLBREAL 80045 13 108 1063127 13.28 2.338 0.73
ITLBVIRT 1000585 21 264827 24300596 24.28 264.753 16.78
DTLBREAL 91206 13 69 1285214 14.09 2.225 0.89
DTLBVIRT 977434 21 1446 24007008 24.56 4.426 16.58
SYSCALL 10460 11 55 116447 11.13 1.929 0.08
ISI 11724 11 61 130007 11.08 1.929 0.09
DSI 20737 11 57 230009 11.09 1.914 0.16
EMULINST 5683356 11 3778 79339467 13.96 50.275 54.78
DEC 13079 50 826 732712 56.02 22.382 0.51
EXTINT 55 30 1478 10996 199.92 238.150 0.01
FP_UNAVAIL 280 11 53 3163 11.29 3.495 0.00
TIMEINGUEST 7905189 0 3688 10330742 1.30 8.970 7.13
</code>

=== paravirtualization ===
<code>
sum of time 92206510 => ~1:32 runtime (~37% net improvement)
count min max sum avg stddev %
MMIO 12505 46 3087 3693782 295.38 260.788 4.01
DCR 5595 40 706 273578 48.89 31.305 0.30
SIGNAL 654 65 4132 300027 458.75 571.130 0.33
ITLBREAL 71711 13 104 943053 13.15 2.360 1.02
ITLBVIRT 750649 21 1503 18178245 24.21 7.335 19.71
DTLBREAL 83356 13 102 1146242 13.75 2.406 1.24
DTLBPV 30086 20 237 653556 21.72 4.639 0.71
DTLBVIRT 772811 21 713 19079477 24.68 6.593 20.69
SYSCALL 7647 11 57 84821 11.09 1.897 0.09
HCALL 1 19 19 19 19.00 0.000 0.00
ISI 9895 11 73 109667 11.08 1.904 0.12
DSI 17974 10 57 199504 11.09 2.046 0.22
EMULINST 2567245 11 4212 40501314 15.77 65.673 43.92
DEC 7488 51 641 426813 56.99 23.893 0.46
EXTINT 2215 31 1677 297495 134.30 116.219 0.32
FP_UNAVAIL 258 11 11 2838 11.00 0.000 0.00
TIMEINGUEST 4340090 0 3850 6316079 1.45 12.599 6.85
</code>

== More Results ==
This should actually just be an overview and is already huge, some more results can be found in all kind of timing and improvement discussions on kvm-ppc@vger.kernel.org

----
[[Category:PowerPC]]

PowerPC Exittimings

2010-07-15T17:26:57Z

HollisBlanchard: replace MoinMoin "smiley" formatting

== About exit timing ==

PowerPC KVM exports information to debugfs about the exits caused by virtualization and the time consumed by them. This data can typically be found as <code>/sys/kernel/debug/kvm/vm*_vcpu*_timing</code>.

Because the PowerPC hardware currently supported by KVM has no hardware virtualization support, the guest must run in non-privileged mode. When the guest executes a privileged instruction, the KVM host must emulate its behavior, and this emulation time is accounted for as EMULINST. (It can be expected that upcoming hardware extension reduce most of these emulation exits as the guest can then run in privileged mode.)

Another frequent exit reason is caused by the memory/TLB management. Because a guest can not be allowed to program the real TLB (privileged commands and it would be an isolation violation anyway), the host has to track the state of the guest TLB and recover TLB faults caused because the guest is virtualized. Such a kind of TLB interrupts caused by virtualization is called [DI]TLBVIRT in the exit statistics. If the guest TLB tracked by the host does not have a mapping for the address reported by a TLB exception it is delivered to the guest as it is done on bare metal. This appears as [DI]TLBREAL in the exit statistics.

When the guest idles, it will enter "wait mode" until an interrupt is delivered. This time is accounted for by exit type of WAIT.

The other exits are less frequent like MMIO, DCR and SIGNAL which need to exit to kvm-userspace to be handled. The only value not really being an exit is the TIMEINGUEST which is the time in the guest.

The timing statistic itself tracks exit and reenter time as well as the type of the exit. Then the duration exit -> enter is accounted for the specific exit type while enter -> exit is accounted for the TIMEINGUEST values.

'''Note:''' All times are in usec's.

== workloads and performance expectations ==

It can be expected that loads causing a high number of exits have a high overhead while loads that run in guest with only a few interceptions should be fine. Those loads with high exit ratios can be for example a guest booting and initializing all its virtual hardware (EMULINST), as well as load that creates memory pressure and therefore causes a lot of virtual TLB misses ([DI]TLBVIRT).

The following measurements are taken on a 440epx (Sequoia) board. Thins means running an unmodified guests on Hardware without virtualization support. therefore a lot of overhead can be expected. The following statistics give you an idea which exit reasons are frequent dependeing on the workload. And as mentioned above you can think what happens once you run that on virtualization powerpc Hardware coming with Power ISA 2.06 (http://en.wikipedia.org/wiki/Power_Architecture#Power_ISA_v.2.06).

The following sections describe the workloads shown on this page.
[[Image:workload_classification.gif]]

=== boot guest ===

The workload traces a guest from the initialization until you see a boot prompt. Although being simple, this workload is useful to get a load with a high amount of privileged instructions.

=== Hashes ===
This is a custom written small program calculating a lot of hashes. It uses the hash algorithm used in the Berkley db and calculates hash of hash of and so on.
This program represents a workload that has only a few I/O and privileged instructions and therefore has only a low virtualization overhead.
attachment:foohash.c
To execute it just call the binary without options:
<code>
./foohash
</code>

=== Fhourstone ===
The Fhourstone benchmark (http://homepages.cwi.nl/~tromp/c4/fhour.html) uses a 64Mb transposition table to solve positions of the game connect-4. In a small environment as the sequoia board is this is a high amount of memory pressure and therefore it is expected that it causes a lot of TLB exits. After compiling the sources you get a binary called SearchGame and a file called "inputs" which represents the workload. The benchmark is invoked with:
<code>
SearchGame < inputs
</code>

=== Lame ===
Lame (http://lame.sourceforge.net/) is the known mp3 encoder and in the workload testcase used to convert a file in very high quality (option --preset insane). This workload has to do some I/O, as well as a lot of calculations that should not exit to the hypervisor. Therefore it can be expected that lame is a good example of average workload.
The wav file used is from a free sample page in the web (http://bellatrix.ece.mcgill.ca/Documents/AudioFormats/WAVE/Samples.html). We used the M1F1-int32-AFsp.wav from that page using the "insane" preset to get a max quality mp3.
<code>
lame --preset insane M1F1-int32-AFsp.wav M1F1-int32-AFsp.mp3
</code>

=== find ===

At last using find to find a file and search for it all over the disk is a simple workload using only a few CPU calculations but requiring a lot of I/O.
As testcase find is executed on root to search for the wav file we use in the Lame testcase, but with a wildcard. The disk we have is the ELDK ramdisk plus the files for the workloads listed here. To be fair and use the same file system the bare metal test mounts the same as loop device.
<code>
cd /
time find -name "*.wav*"
</code>

== Performance results ==
This Section lists at which % of the native bare metal speed these tests run on the current kvmppc implementation. As described below there are alerady known options for improvement like paravirtualization. The tests are run on the source level on 11. November 2008 which included some new performance improvements in memory management, interrupt delivery as well as several minor improvements.
The tests are run on a Host and Guest with 64k page size. The Host uses a nfs mount as root file system while the guest is using virtio to access a disk image placed on the host root nfs mount.

{|
|'''workload'''
|'''% of bae metal speed'''
|-

| hashes
| 96.49%
|-

| lame
| 84.47%
|-

| boot
| ~80%
|-

| find
| 6.11%
|-

| fhourstone
| 5.96%
|}

'''Note:''' network latency after the virtio indirection might be a big issue for the find testcase so treat the numbers unfinished until we verified that number it on e.g. a local usb stick.

'''Note:''' the time accounted for MMIO is the time a guest exits and KVM prepares the mmio until it returns to the guest. It is not the time until the IO arrives and is ready for the guest. Additional IO performance data may be obtained by running blktrace on the virtio disk inside the guest.

== Timings results ==

The following tables show the results of the exit timing analysis using the 5 different workloads mentioned above.
You can get similar postprocessed reports when using this script (attachment:kvmppc_timing.py) with the data reported by the debugfs interface.

=== boot guest ===
<code>
sum of time 8309940
type count min max sum avg stddev %
MMIO: 9402 44 1997 1697610 180.5584 155.768 20.43
DCR: 680 41 99 32096 47.2000 7.008 0.39
SIGNAL: 1 98 98 98 98.0000 0.000 0.00
ITLBREAL: 926 8 14 7810 8.4341 0.658 0.09
ITLBVIRT: 3595 18 202 76185 21.1919 4.954 0.92
DTLBREAL: 950 8 16 8891 9.3589 1.427 0.11
DTLBVIRT: 6695 18 282 156727 23.4096 13.781 1.89
SYSCALL: 1801 6 59 11372 6.3143 2.575 0.14
ISI: 116 6 8 764 6.5862 0.588 0.01
DSI: 43 6 7 292 6.7907 0.407 0.00
EMULINST: 65247 7 96 484081 7.4192 1.818 5.83
EMUL_WAIT: 801 659 9200 3721789 4646.4282 1687.218 44.79
EMUL_CORE: 66806 7 86 540053 8.0839 1.895 6.50
EMUL_MTSPR: 13415 8 62 111358 8.3010 2.583 1.34
EMUL_MFSPR: 7635 8 61 62772 8.2216 1.921 0.76
EMUL_MTMSR: 5678 8 59 45704 8.0493 1.434 0.55
EMUL_MFMSR: 32853 7 67 267603 8.1455 1.875 3.22
EMUL_TLBSX: 354 9 60 3745 10.5791 3.919 0.05
EMUL_TLBWE: 6403 9 112 99522 15.5430 7.668 1.20
EMUL_RFI: 9515 7 57 71420 7.5060 2.108 0.86
DEC: 290 49 161 15786 54.4345 9.708 0.19
EXTINT: 7 74 75 522 74.5714 0.495 0.01
TIMEINGUEST: 233213 0 3954 893740 3.8323 65.837 10.76
</code>

=== Hashes ===
<code>
sum of time 21576367
type count min max sum avg stddev %
MMIO: 827 49 6700 224632 271.6227 259.231 1.04
DCR: 161 42 94 7468 46.3851 4.314 0.03
SIGNAL: 2 291 1214 1505 752.5000 461.500 0.01
ITLBREAL: 53 8 12 445 8.3962 0.682 0.00
ITLBVIRT: 216 19 68 4566 21.1389 3.346 0.02
DTLBREAL: 44 9 16 420 9.5455 1.738 0.00
DTLBVIRT: 407 19 73 8706 21.3907 3.687 0.04
SYSCALL: 66 6 7 428 6.4848 0.500 0.00
ISI: 5 6 8 34 6.8000 0.748 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 67009 6 97 508311 7.5857 1.247 2.36
EMUL_WAIT: 231 1254 8902 1074304 4650.6667 1699.150 4.98
EMUL_CORE: 32964 7 59 262866 7.9743 0.622 1.22
EMUL_MTSPR: 9201 8 14 74751 8.1242 0.339 0.35
EMUL_MFSPR: 379 8 60 3134 8.2691 2.686 0.01
EMUL_MTMSR: 4749 8 9 37996 8.0008 0.029 0.18
EMUL_MFMSR: 14257 7 55 114282 8.0159 0.776 0.53
EMUL_TLBSX: 18 9 15 185 10.2778 1.193 0.00
EMUL_TLBWE: 393 9 69 5477 13.9364 6.905 0.03
EMUL_RFI: 9006 7 57 67271 7.4696 0.722 0.31
DEC: 5065 49 269 267567 52.8267 13.544 1.24
EXTINT: 2 77 451 528 264.0000 187.000 0.00
TIMEINGUEST: 145056 0 3954 18911484 130.3737 678.943 87.65
</code>

=== Lame ===
<code>
sum of time 6592939
type count min max sum avg stddev %
MMIO: 1827 48 18883 550073 301.0799 772.936 8.34
DCR: 392 42 1074 22162 56.5357 83.884 0.34
SIGNAL: 1 1812 1812 1812 1812.0000 0.000 0.03
ITLBREAL: 142 8 13 1200 8.4507 0.623 0.02
ITLBVIRT: 1860 18 118 39336 21.1484 4.514 0.60
DTLBREAL: 164 8 66 1707 10.4085 4.885 0.03
DTLBVIRT: 2724 18 1039 63063 23.1509 23.705 0.96
SYSCALL: 255 6 8 1626 6.3765 0.531 0.02
ISI: 17 6 8 114 6.7059 0.570 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 26682 6 161 203151 7.6138 2.885 3.08
EMUL_WAIT: 261 501 7683 1211114 4640.2835 1683.305 18.37
EMUL_CORE: 19247 7 161 158089 8.2137 4.052 2.40
EMUL_MTSPR: 4309 8 161 35697 8.2843 2.371 0.54
EMUL_MFSPR: 1238 8 61 10395 8.3966 3.510 0.16
EMUL_MTMSR: 2098 8 61 17045 8.1244 2.403 0.26
EMUL_MFMSR: 9158 7 163 75613 8.2565 3.392 1.15
EMUL_TLBSX: 36 9 13 364 10.1111 0.698 0.01
EMUL_TLBWE: 1062 9 75 17011 16.0179 7.539 0.26
EMUL_RFI: 3736 7 142 28230 7.5562 2.974 0.43
DEC: 1109 49 263 61117 55.1100 15.174 0.93
EXTINT: 36 52 1377 12085 335.6944 308.306 0.18
TIMEINGUEST: 76355 0 3954 4081928 53.4599 415.345 61.91
</code>

=== Fhourstone ===
<code>
sum of time 7483768
type count min max sum avg stddev %
MMIO: 818 47 9565 221609 270.9156 501.827 2.96
DCR: 301 40 473 14408 47.8671 29.403 0.19
SIGNAL: 1 2521 2521 2521 2521.0000 0.000 0.03
ITLBREAL: 322 8 58 2665 8.2764 2.810 0.04
ITLBVIRT: 5773 18 1360 120111 20.8056 18.569 1.60
DTLBREAL: 16433 8 73 184196 11.2089 3.709 2.46
DTLBVIRT: 19913 18 1845 500349 25.1268 23.006 6.69
SYSCALL: 91 6 7 579 6.3626 0.481 0.01
ISI: 5 6 8 33 6.6000 0.800 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 127113 6 102 949687 7.4712 2.244 12.69
EMUL_WAIT: 76 3526 7578 354928 4670.1053 1679.367 4.74
EMUL_CORE: 16733 7 159 134306 8.0264 1.701 1.79
EMUL_MTSPR: 71886 8 149 594618 8.2717 2.595 7.95
EMUL_MFSPR: 83877 8 93 689016 8.2146 2.580 9.21
EMUL_MTMSR: 3166 8 57 25438 8.0347 1.220 0.34
EMUL_MFMSR: 7392 7 61 59415 8.0377 1.294 0.79
EMUL_TLBSX: 19 9 11 192 10.1053 0.640 0.00
EMUL_TLBWE: 47187 9 2757 1131527 23.9796 13.359 15.12
EMUL_RFI: 21782 7 132 172428 7.9161 2.323 2.30
DEC: 715 50 224 42155 58.9580 15.052 0.56
EXTINT: 2 142 161 303 151.5000 9.500 0.00
TIMEINGUEST: 423606 0 499 2283277 5.3901 25.629 30.51
</code>

=== find ===
<code>
sum of time 3426052
type count min max sum avg stddev %
MMIO: 222 49 413 48228 217.2432 141.096 1.41
DCR: 91 43 93 4239 46.5824 5.285 0.12
SIGNAL: 3 476 5651 7952 2650.6667 2191.871 0.23
ITLBREAL: 77 8 13 665 8.6364 0.754 0.02
ITLBVIRT: 1341 18 120 28340 21.1335 4.968 0.83
DTLBREAL: 59 8 16 573 9.7119 2.042 0.02
DTLBVIRT: 2253 19 214 48630 21.5846 7.083 1.42
SYSCALL: 4590 6 57 29503 6.4277 2.114 0.86
ISI: 11 6 8 72 6.5455 0.656 0.00
DSI: 1 7 7 7 7.0000 0.000 0.00
EMULINST: 71560 6 77 525976 7.3501 1.945 15.35
EMUL_WAIT: 374 184 9384 1724701 4611.5000 1752.946 50.34
EMUL_CORE: 32646 7 100 262449 8.0392 1.792 7.66
EMUL_MTSPR: 6668 8 78 54829 8.2227 2.213 1.60
EMUL_MFSPR: 538 8 61 4507 8.3773 3.181 0.13
EMUL_MTMSR: 5829 8 58 47036 8.0693 1.765 1.37
EMUL_MFMSR: 15805 7 92 127426 8.0624 1.774 3.72
EMUL_TLBSX: 29 9 14 297 10.2414 0.857 0.01
EMUL_TLBWE: 462 9 27 6717 14.5390 6.606 0.20
EMUL_RFI: 10855 7 57 81776 7.5335 2.132 2.39
DEC: 160 50 403 9244 57.7750 32.861 0.27
EXTINT: 4 427 1410 2991 747.7500 387.275 0.09
TIMEINGUEST: 153578 0 762 409894 2.6690 6.322 11.96
</code>

== Paravirtualization improvement ==

As mentioned above improvements to all these overhead statistics are already known. On one hand the hardware virtualization support specified in the Power ISA 2.06, on the other hand on older hardware paravirtualization can be an option. For KVMPPC we wrote a simple paravirtualization interface to test hypercalls and measure some benefits from such implementations.
In the concept measured here the hypervisor tells the guest that it supports special paravirtualization features if the guest (hyper)calls him passing a guest virtual and guest physical address and an amount of x (4096byte in the example) of ram big. This is a very basic, but also flexible interface as the hypervisor can now use this guest addressable memory to do all kind of things. In the example the hypervisor rewrites guest code to replace privileged instructions mfspr (SPRG1-3, ESR, DEAR) mtspr ((SPRG1-3, ESR, DEAR) and mfmsr. The hypervisor keeps the guest copies of these values updated in the memory area provided by the guest and rewrites the privileged instructions to simple non trapping load/store instructions. That save a lot of EMULINST exits while running the guest and has shown around 35-50% improvement for workloads with a high amount of EMULINST instructions (e.g. the boot workload)

The net benefit of that improvement is high while the "guest invasiveness" is very low (The guest only has to donate a small amount of ram and virtual address space, all the other things can be done by the hypervisor transparently). And remember this is just one simple example of pv extensions, there are a lot other areas e.g. collaborative guest/host TLB management that can improve performance significantly (could be as easy as telling the guest to program more virtual TLB entries in the guest TLB to allow the host fix more misses directly).

The following numbers show the improvement comparing the same workload with/without this paravirtualization feature. The workload used in this test is the boot workload, but using a more complex guest and Host environment (Ubuntu instead of ELDK) and it also uses an older version of our kernel and userspace code (also containing an older version of that exit timing and therefore not being directly comparable with the measurements above).

=== No paravirtualization ===
<code>
sum of time 144837890 => ~2:24 runtime
count min max sum avg stddev %
MMIO 10105 46 1534 2952467 292.17 295.933 2.04
DCR 5428 40 209 246168 45.35 6.758 0.17
SIGNAL 695 65 3755 89767 129.16 314.421 0.06
ITLBREAL 80045 13 108 1063127 13.28 2.338 0.73
ITLBVIRT 1000585 21 264827 24300596 24.28 264.753 16.78
DTLBREAL 91206 13 69 1285214 14.09 2.225 0.89
DTLBVIRT 977434 21 1446 24007008 24.56 4.426 16.58
SYSCALL 10460 11 55 116447 11.13 1.929 0.08
ISI 11724 11 61 130007 11.08 1.929 0.09
DSI 20737 11 57 230009 11.09 1.914 0.16
EMULINST 5683356 11 3778 79339467 13.96 50.275 54.78
DEC 13079 50 826 732712 56.02 22.382 0.51
EXTINT 55 30 1478 10996 199.92 238.150 0.01
FP_UNAVAIL 280 11 53 3163 11.29 3.495 0.00
TIMEINGUEST 7905189 0 3688 10330742 1.30 8.970 7.13
</code>

=== paravirtualization ===
<code>
sum of time 92206510 => ~1:32 runtime (~37% net improvement)
count min max sum avg stddev %
MMIO 12505 46 3087 3693782 295.38 260.788 4.01
DCR 5595 40 706 273578 48.89 31.305 0.30
SIGNAL 654 65 4132 300027 458.75 571.130 0.33
ITLBREAL 71711 13 104 943053 13.15 2.360 1.02
ITLBVIRT 750649 21 1503 18178245 24.21 7.335 19.71
DTLBREAL 83356 13 102 1146242 13.75 2.406 1.24
DTLBPV 30086 20 237 653556 21.72 4.639 0.71
DTLBVIRT 772811 21 713 19079477 24.68 6.593 20.69
SYSCALL 7647 11 57 84821 11.09 1.897 0.09
HCALL 1 19 19 19 19.00 0.000 0.00
ISI 9895 11 73 109667 11.08 1.904 0.12
DSI 17974 10 57 199504 11.09 2.046 0.22
EMULINST 2567245 11 4212 40501314 15.77 65.673 43.92
DEC 7488 51 641 426813 56.99 23.893 0.46
EXTINT 2215 31 1677 297495 134.30 116.219 0.32
FP_UNAVAIL 258 11 11 2838 11.00 0.000 0.00
TIMEINGUEST 4340090 0 3850 6316079 1.45 12.599 6.85
</code>

== More Results ==
This should actually just be an overview and is already huge, some more results can be found in all kind of timing and improvement discussions on kvm-ppc@vger.kernel.org

----
[[Category:PowerPC]]

PowerPC Run

2009-10-30T16:54:13Z

HollisBlanchard: replace moinmoin smileys

== Run ==

Log in to target and run:

<code>
./qemu-system-ppcemb --enable-kvm \
-nographic \
-m 128 \
-M bamboo \
-kernel uImage.bamboo \
-L . \
-append "your guest kernel command line"
</code>

The location specified with -L must contain the <code>bamboo.dtb</code> device tree file.

{|
|'''Note:''' Consult the [http://bellard.org/qemu/qemu-doc.html QEMU documentation] for a more complete list and explanation of these options.
|}

== virtio ==
In the case that virtio is activated in the guest kernel, you can add virtio devices at the qemu command line.

Examples (nothing special for ppc):

=== network ===

This adds a virtio network device visible to the guest and the external network:
<code>
./qemu-system-ppcemb [...] -net nic,model=virtio -net tap
</code>

{|
|'''Note:''' You must have CONFIG_VIRTIO_NET=y in your guest kernel .config.
|}

{|
|'''Note:''' You must have Qemu networking configured to use a <code>tap</code> device. See [[Networking]] for details.
|}

=== disk ===

This adds a virtio disk device:
<code>
./qemu-system-ppcemb [...] -drive file=pathtoyourfile.img,if=virtio
</code>

The device appears as <code>/dev/vda</code> in the guest.

{|
|'''Note:''' You must have CONFIG_VIRTIO_BLK=y in your guest kernel .config.
|}

== Known Pitfalls ==
=== no guest console ===
Specifying a console in the guest kernel parameters might let the guest output fail. If you think your output is missing try without any "console=" parameter

=== virtio net requirements ===
You will need to enable tap and bridge support in the host kernel.

You may also need to download and install bridge-utils (use <code>configure --host=i386-linux --target=ppc-linux</code> and all other cross build stuff you need).

Usually embedded boards use a nfs root device, this may conflict with the need to ifdown the device you want to bridge, but you might use the second ethernet device in that case.
You need to edit /etc/qemu-ifup to your needs here is an example that creates bridge br0 on demand and uses eth1 as external device on that bridge:
<code>
#!/bin/bash

bridge=br0
netdev=eth1
guestdev=$1

if [ ! -c /dev/net/tun ]
then
echo "network init failed - /dev/net/tun does not exist"
exit 1
fi

echo "test for existing bridge $bridge"
brctl showmacs $bridge

if [ $? -eq 0 ]
then
echo "ok - bridge $bridge already available"
else
echo "need to create bridge $bridge"
ifconfig $netdev down
brctl addbr $bridge
ifconfig $bridge up
ifconfig $netdev up
brctl addif $bridge $netdev
echo "created bridge $bridge"
brctl show
fi

ifconfig $guestdev up
brctl addif $bridge $guestdev
</code>

----
[[Category:PowerPC]]

PowerPC Host Userspace

2009-08-26T16:48:28Z

HollisBlanchard: remove obsolete configure option

== What You Need ==

# A copy of the [http://savannah.nongnu.org/git/?group=qemu qemu development source tree].
# A copy of the Linux kernel source tree. Note: <code>qemu.git</code> will not build with Linux 2.6.28 or earlier.
# An installed copy of [http://git.jdl.com/gitweb/?p=dtc.git;a=summary libfdt], built for PowerPC.

If you are cross-compiling from a non-PowerPC host, you also need:

# A cross-compiler, such as one built by [http://kegel.com/crosstool crosstool].
# An installed copy of the [http://zlib.net/ zlib] library, built for PowerPC.

=== Where to install the libraries ===

Before building qemu, you install the <code>libfdt</code> and <code>zlib</code> libraries where your toolchain will find them. If you are not cross-compiling, skip this section (because your toolchain will find them in the usual /usr/lib).

You can use <code>cpp -v</code> to find your toolchain's built-in prefix. For example, if your toolchain was created by [http://kegel.com/crosstool crosstool], cpp will say something like this:
<code>
% powerpc-440-linux-gnu-cpp -v
...
#include "..." search starts here:
#include <...> search starts here:
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/include
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/../../../../powerpc-440-linux-gnu/sys-include
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/../../../../powerpc-440-linux-gnu/include
End of search list.
</code>

The prefix here is <code>/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/powerpc-440-linux-gnu</code>. (Basically pick the last ".../include" path and go up one directory.) Use that prefix to install <code>zlib</code> and <code>libfdt</code>.

== Building Qemu for PowerPC KVM ==

Here is an example for the configure script at the top level of the kvm-userspace repository:

If cross-compiling, run this first:
<code>
cross=powerpc-440-linux-gnu-
</code>

<code>
./configure \
--disable-sdl \
--target-list=ppcemb-softmmu \
--cross-prefix="$cross" \
--kerneldir="/home/hollisb/source/kvmppc-dev.hg"
</code>

== Build ==

<code>make</code> will produce <code>qemu/ppcemb-softmmu/qemu-system-ppcemb</code>. You must transfer that executable and <code>qemu/pc-bios/bamboo.dtb</code> to the target.

----
[[Category:PowerPC]]

Documents

2009-07-09T00:01:25Z

HollisBlanchard: fix wiki markup (linkify Forum pages)

= Documents =

== User/Admin documentation ==
* [[HOWTO]]
* [http://www.nongnu.org/qemu/qemu-doc.html QEMU user manual]
== Presentations ==
* Presentations on many aspects of KVM were made at [[KvmForum2007]] (Aug 2007).
* Presentations from the [[KvmForum2008]] (June 2008)
* Avi Kivity's presentation from the [http://ols.108.redhat.com/2007/Reprints/kivity-Reprint.pdf Ottawa Linux Symposium 2007] (Jun 2007).
* TPR patching [attachment:kvm-tpr-patching.odp overview] (Avi Kivity, Oct 2008)
* [http://markmc.fedorapeople.org/virtio-code-review/VirtioCodeReview.pdf Virtio code walkthrough], [http://markmc.fedorapeople.org/virtio-code-review/virtio-talk.txt notes], [http://blogs.gnome.org/markmc/2008/05/28/checksums-scatter-gather-io-and-segmentation-offload/ GSO background] (Mark McLoughlin, Oct 2008), and [http://portal.acm.org/citation.cfm?id=1400097.1400108 ACM pdf about virtio by Rusty Russell]
== White papers: ==
* Qumranet's [http://www.qumranet.com/files/white_papers/KVM_Whitepaper.pdf KVM Whitepaper]
== Magazine Articles ==
* [http://www.linux-magazine.com/issues/2008/86/kernel_tricks Linux Magazine]
* [http://www.linuxplanet.com/linuxplanet/reports/6490/4/ KVM for Embedded] at Linux Planet
== Benchmarks ==
* [http://www.phoronix.com/scan.php?page=article&item=ubuntu_virt_benchmarks&num=1 Phoronix - Ubuntu 8.04 KVM Benchmarks]
* [http://www.phoronix.com/scan.php?page=article&item=intel_corei7_virt&num=1 Phoronix - Intel Core i7 Virtualization Performance]
== Documentation ==
* [[small_look_inside|small look inside(kvm-54)]]
* [[buildup|qemu kvm buildup]]
* [[vl_runthrough|qemu-system-x86_64 startup (kvm-57)]]
* [[initialization|initialization (kvm-57)]]
* [[file_layout_in_kernel|file layout in kernel (~kvm-58)]]
== KVM Doxygen Documentation ==
* [http://kvmapi.ath.cx kvm doxygen documentation]
== Tools ==
* [[tools|Tools]]
== Supported cpus ==
* [[processor_support|cpus]]

PowerPC Host Userspace

2009-05-13T21:03:56Z

HollisBlanchard: link text with URL

== What You Need ==

# A copy of the [http://savannah.nongnu.org/git/?group=qemu qemu development source tree].
# A copy of the Linux kernel source tree. Note: <code>qemu.git</code> will not build with Linux 2.6.28 or earlier.
# An installed copy of [http://git.jdl.com/gitweb/?p=dtc.git;a=summary libfdt], built for PowerPC.

If you are cross-compiling from a non-PowerPC host, you also need:

# A cross-compiler, such as one built by [http://kegel.com/crosstool crosstool].
# An installed copy of the [http://zlib.net/ zlib] library, built for PowerPC.

=== Where to install the libraries ===

Before building qemu, you install the <code>libfdt</code> and <code>zlib</code> libraries where your toolchain will find them. If you are not cross-compiling, skip this section (because your toolchain will find them in the usual /usr/lib).

You can use <code>cpp -v</code> to find your toolchain's built-in prefix. For example, if your toolchain was created by [http://kegel.com/crosstool crosstool], cpp will say something like this:
<code>
% powerpc-440-linux-gnu-cpp -v
...
#include "..." search starts here:
#include <...> search starts here:
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/include
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/../../../../powerpc-440-linux-gnu/sys-include
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/../../../../powerpc-440-linux-gnu/include
End of search list.
</code>

The prefix here is <code>/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/powerpc-440-linux-gnu</code>. (Basically pick the last ".../include" path and go up one directory.) Use that prefix to install <code>zlib</code> and <code>libfdt</code>.

== Building Qemu for PowerPC KVM ==

Here is an example for the configure script at the top level of the kvm-userspace repository:

If cross-compiling, run this first:
<code>
cross=powerpc-440-linux-gnu-
</code>

<code>
./configure \
--disable-sdl \
--disable-gfx-check \
--target-list=ppcemb-softmmu \
--cross-prefix="$cross" \
--kerneldir="/home/hollisb/source/kvmppc-dev.hg"
</code>

== Build ==

<code>make</code> will produce <code>qemu/ppcemb-softmmu/qemu-system-ppcemb</code>. You must transfer that executable and <code>qemu/pc-bios/bamboo.dtb</code> to the target.

----
[[Category:PowerPC]]

PowerPC Host Userspace

2009-05-13T20:55:38Z

HollisBlanchard: update qemu.git URL

== What You Need ==

# A copy of the qemu development source tree (http://savannah.nongnu.org/git/?group=qemu).
# A copy of the Linux kernel source tree. Note: <code>qemu.git</code> will not build with Linux 2.6.28 or earlier.
# An installed copy of [http://git.jdl.com/gitweb/?p=dtc.git;a=summary libfdt], built for PowerPC.

If you are cross-compiling from a non-PowerPC host, you also need:

# A cross-compiler, such as one built by [http://kegel.com/crosstool crosstool].
# An installed copy of the [http://zlib.net/ zlib] library, built for PowerPC.

=== Where to install the libraries ===

Before building qemu, you install the <code>libfdt</code> and <code>zlib</code> libraries where your toolchain will find them. If you are not cross-compiling, skip this section (because your toolchain will find them in the usual /usr/lib).

You can use <code>cpp -v</code> to find your toolchain's built-in prefix. For example, if your toolchain was created by [http://kegel.com/crosstool crosstool], cpp will say something like this:
<code>
% powerpc-440-linux-gnu-cpp -v
...
#include "..." search starts here:
#include <...> search starts here:
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/include
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/../../../../powerpc-440-linux-gnu/sys-include
/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/lib/gcc/powerpc-440-linux-gnu/3.4.5/../../../../powerpc-440-linux-gnu/include
End of search list.
</code>

The prefix here is <code>/opt/crosstool/gcc-3.4.5-glibc-2.3.6/powerpc-440-linux-gnu/powerpc-440-linux-gnu</code>. (Basically pick the last ".../include" path and go up one directory.) Use that prefix to install <code>zlib</code> and <code>libfdt</code>.

== Building Qemu for PowerPC KVM ==

Here is an example for the configure script at the top level of the kvm-userspace repository:

If cross-compiling, run this first:
<code>
cross=powerpc-440-linux-gnu-
</code>

<code>
./configure \
--disable-sdl \
--disable-gfx-check \
--target-list=ppcemb-softmmu \
--cross-prefix="$cross" \
--kerneldir="/home/hollisb/source/kvmppc-dev.hg"
</code>

== Build ==

<code>make</code> will produce <code>qemu/ppcemb-softmmu/qemu-system-ppcemb</code>. You must transfer that executable and <code>qemu/pc-bios/bamboo.dtb</code> to the target.

----
[[Category:PowerPC]]

Virtio

2008-03-24T17:14:40Z

HollisBlanchard: typo

= Paravirtualized drivers for kvm/Linux =
* Virtio was chosen to be the main platform for IO virtualization in KVM
* The idea behind it is to have a common framework for hypervisors for IO virtualization
* More information (although not uptodate) can be found in kvm [http://kvm.qumranet.com/kvmwiki/KvmForum2007?action=[[AttachFile]]&do=get&target=kvm_pv_drv.pdf pv driver]
* At the moment network/block/balloon devices are suported for kvm
* The host implementation is in userspace - qemu, so no driver is needed in the host.

= How to use Virtio =
* Get kvm version >= 60
* Get Linux kernel with virtio drivers for the guest
** Get Kernel >= 2.6.25 and activate (modules should also work, but take care of initramdisk)
*** CONFIG_VIRTIO_PCI=y (Virtualization -> PCI driver for virtio devices)
*** CONFIG_VIRTIO_BALLOON=y (Virtualization -> Virtio balloon driver)
*** CONFIG_VIRTIO_BLK=y (Device Drivers -> Block -> Virtio block driver)
*** CONFIG_VIRTIO_NET=y (Device Drivers -> Network device support -> Virtio network driver)
*** CONFIG_VIRTIO=y (automatically selected)
*** CONFIG_VIRTIO_RING=y (automatically selected)
*** CONFIG_HIGH_RES_TIMERS=y (Processor type and features -> High Resolution Timer Support **optional**)
*** you can safely disable SATA/SCSI and also all other nic drivers if you only use VIRTIO (disk/nic)
* Either build it around Rusty's tree [http://ozlabs.org/~rusty/kernel/hg/ repo]
** Or git clone git://kvm.qumranet.com/home/dor/src/linux-2.6-nv use branch rusty
** Soon an official repository will be released
** As an alternative one can use a standard guest kernel for the guest > 2.6.18 and make use sync backward compatibility option
** Backport and instructions can be found in Anthony Liguori's [http://codemonkey.ws/virtio-ext-modules virtio-ext-modules]
** At the moment it's broken since the guest got developed, soon update
* Use model=virtio for the network devices and if=virtio for disk
** Example
<pre><nowiki>
qemu/x86_64-softmmu/qemu-system-x86_64 -boot c -drive file=/images/xpbase.qcow2,if=virtio,boot=on -m 384 -net nic,model=virtio -net tap,script=/etc/kvm/qemu-ifup
</nowiki></pre>
* -hd[ab] for disk won't work, use -drive
* Disk will show up as /dev/vd[a-z][1-9], if you migrate you need to change "root=" in Lilo/GRUB config
* At the moment the kernel modules are automatically loaded in the guest but the interface should be started manually (dhclient/ifconfig)
* Currently performance is much better when using a host kernel configured with CONFIG_HIGH_RES_TIMERS. Another option is use HPET/RTC and -clock= qemu option.
* Expected performance
** Performance varies from host to host, kernel to kernel
** On my laptop I measured 1.1Gbps rx throughput using 2.6.23, 850Mbps tx.
** Ping latency is 300-500 usec
* Enjoy, more to come :)

__NOTOC__

KVM Forum 2007

2007-09-12T16:30:43Z

HollisBlanchard:

[http://www.qumranet.com/images/kvm_logo.gif]

= Information on the recently-concluded KVM Forum 2007 =
== Agenda ==
==== Wednesday August 29th ====
{| border="1" cellpadding="2" cellspacing="0"
| Time
| Session Topic
| Speaker
|
|-
| 8:45 AM - 9:40 AM
| [attachment:kf2007-keynote.pdf KVM, One Year On]
| Avi Kivity, Qumranet
|
|-
| 9:40 AM - 10:30 AM
| The future of virtualization - KVM
| Sunil Saxena, Intel
|
|-
| 10:45 AM - 11:50 AM
| KVM Security
| Hadi Nahari, Montavista
|
|-
| 1:35 PM - 2:30 PM
| KVM & S390
| Carsten Otte, IBM
|
|-
| 2:30 PM - 3:30 PM
| KVM Lite, No Hardware Support, Fewer Calories
| Rusty Russell, IBM
|
|-
| 4:00 PM - 5:00 PM
| VT Roadmap, Hybrid Virtualization, Power Management, Fat vs Thin Hypervisor
| Jun Nakajima, Intel
|
|}
==== Thursday August 30th ====
{| border="1" cellpadding="2" cellspacing="0"
| Time
| Session Topic
| Speaker
|
|-
| 9:00 AM - 9:50 AM
| [attachment:kvm_pv_drv.pdf KVM Para-Virtualized Guest Drivers]
| Dor Laor, Qumranet
|
|-
| 9:50 AM - 10:50 AM
| [attachment:AMD_Extended_Migration-KVMForum-ElsieWahlig.pdf KVM Support for the AMD DEV]
| Elsie Wahlig, AMD
|
|-
| 9:50 AM - 10:50 AM
| [attachment:kvm_iommu_talk.pdf A KVM friendly IOMMU API for Linux]
| Joerg Roedel, AMD
|
|-
| 10:50 AM - 11:45 AM
| [attachment:KVMForum07_Liguori.pdf Automating VM Installation Testing]
| Anthony Liguori, IBM
|
|-
| 1:35 PM - 2:30 PM
| [attachment:CIM4KVM.pdf Standards Based Systems Management Solution for KVM]
| Anthony Liguori, IBM
|
|-
| 2:35 PM - 3:30 PM
| [attachment:KVM-tuning-testing-SMP2.pdf KVM Performance, SMP and in kernel PIC/APIC, KVM Validation]
| Eddie Dong, Yungen Zhao, Xin Lin, Intel
|
|-
| 4:00 PM - 5:35 PM
| Open Session Panel
| Avi Kivity, Qumranet
|
|}
==== Friday August 31st ====
{| border="1" cellpadding="2" cellspacing="0"
| Time
| Session Topic
| Speaker
|
|-
| 9:00 AM - 9:45 AM
| [attachment:Kvm_Live_Migration_Forum_2007.pdf KVM Live Migration]
| Anthony Liguori, IBM & Uri Lubin, Qumranet
|
|-
| 9:50 AM - 10:35 AM
| [attachment:shadowy-depths-of-the-kvm-mmu.pdf The Shadowy Depths of the KVM MMU]
| Avi Kivity, Qumranet
|
|-
| 10:50 AM - 11:45 AM
| [attachment:KVM-IA64_forum_083107.pdf KVM for IPF (ia64)]
| Anthony Xu, Intel
|
|-
| 11:50 AM -12:45 PM
| [attachment:KVM_Forum_-_Embedded_PowerPC.pdf Implementing KVM for Embedded PowerPC]
| Hollis Blanchard, IBM
|
|-
| 12:45 PM - 1.30 PM
| [attachment:KVM_Forum_Concluding_Keynote.pdf Concluding Keynote]
| Benny Schnaider, Qumranet
|
|}
== Presentations ==
== Blogs ==
== Pictures ==
* (from [[AmitShah]]) Pictures from the Pima Air and Space Museum and the Loews Ventana Canyon Resort: http://travel.webshots.com/album/560535651amWFjh

__NOTOC__