Perf events

2010-06-24T16:11:36Z

Steved: Added text for recording and reporting events

This page describes how to count and trace performance events in the KVM kernel module.

There are two tools, <tt>kvm_stat</tt> and <tt>kvm_trace</tt>, which were previously used for these tasks. Now this can be done using standard Linux tracing tools.

== Counting events ==

Often you want event counts after running a benchmark:

<pre>
$ sudo mount -t debugfs none /sys/kernel/debug
$ sudo ./perf stat -e 'kvm:*' -a sleep 1h
^C
Performance counter stats for 'sleep 1h':

8330 kvm:kvm_entry # 0.000 M/sec
0 kvm:kvm_hypercall # 0.000 M/sec
4060 kvm:kvm_pio # 0.000 M/sec
0 kvm:kvm_cpuid # 0.000 M/sec
2681 kvm:kvm_apic # 0.000 M/sec
8343 kvm:kvm_exit # 0.000 M/sec
737 kvm:kvm_inj_virq # 0.000 M/sec
0 kvm:kvm_page_fault # 0.000 M/sec
0 kvm:kvm_msr # 0.000 M/sec
664 kvm:kvm_cr # 0.000 M/sec
872 kvm:kvm_pic_set_irq # 0.000 M/sec
0 kvm:kvm_apic_ipi # 0.000 M/sec
738 kvm:kvm_apic_accept_irq # 0.000 M/sec
874 kvm:kvm_set_irq # 0.000 M/sec
874 kvm:kvm_ioapic_set_irq # 0.000 M/sec
0 kvm:kvm_msi_set_irq # 0.000 M/sec
433 kvm:kvm_ack_irq # 0.000 M/sec
2685 kvm:kvm_mmio # 0.000 M/sec

3.493562100 seconds time elapsed
</pre>

The <tt>perf</tt> tool is part of the Linux kernel tree in <tt>tools/perf</tt>.

== Tracing events ==

Detailed traces can be generated using ftrace:

<pre>
# mount -t debugfs none /sys/kernel/debug
# echo 1 >/sys/kernel/debug/tracing/events/kvm/enable
# cat /sys/kernel/debug/tracing/trace_pipe
[...]
kvm-5664 [000] 11906.220178: kvm_entry: vcpu 0
kvm-5664 [000] 11906.220181: kvm_exit: reason apic_access rip 0xc011518c
kvm-5664 [000] 11906.220183: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0
kvm-5664 [000] 11906.220183: kvm_apic: apic_write APIC_EOI = 0x0
kvm-5664 [000] 11906.220184: kvm_ack_irq: irqchip IOAPIC pin 11
kvm-5664 [000] 11906.220185: kvm_entry: vcpu 0
kvm-5664 [000] 11906.220188: kvm_exit: reason io_instruction rip 0xc01e4473
kvm-5664 [000] 11906.220188: kvm_pio: pio_read at 0xc13e size 2 count 1
kvm-5664 [000] 11906.220193: kvm_entry: vcpu 0
^D
# echo 0 >/sys/kernel/debug/tracing/events/kvm/enable
</pre>

== Recording events ==

Events can be recorded to a file for later reporting and analysis.
You can record events for the host using the <tt>--host</tt> option.
You can record events for a guest using the <tt>--guest</tt> option.
You can use both options at the same time to record events for both the host and a guest.

If you use just the <tt>--host</tt> option the default output file will be <tt>perf.data.host</tt>.
If you use just the <tt>--guest</tt> option the default output file will be <tt>perf.data.guest</tt>.
If you use both options the default output file will be <tt>perf.data.kvm</tt>.
Use the <tt>-o</tt> option after the <tt>record</tt> key word to save the output to a different file name.

<pre>
# perf kvm --host --guest [kvm options] record -a -o my.perf.data
</pre>

=== Recording events for a guest ===

==== Using copies of guest files ====

In order to record events for a guest, the perf tool needs the <tt>/proc/kallsyms</tt> and <tt>/proc/modules</tt> for the guest.
These are passed to <tt>perf</tt> with the <tt>--guestkallsyms</tt> and <tt>--guestmodules</tt> options.
The files will have to be on the host, but you can get them easily using ssh.

<pre>
# ssh guest "cat /proc/kallsyms" > /tmp/guest.kallsyms
# ssh guest "cat /proc/modules" > /tmp/guest.modules
</pre>

It is better to use <tt>ssh</tt> to <tt>cat</tt> the files and redirect the output than to use <tt>scp</tt>.
Experience has shown that <tt>scp guest:/proc/kallsyms /tmp/guest.kallsyms</tt> will return an empty file.

Using these options you can record the events for a host and a guest with the following command
<pre>
# perf kvm --host --guest --guestkallsyms=/tmp/guest.kallsyms --guestmodules=/tmp/guest.modules record -a
</pre>

The order of the arguments is important.
In general, the syntax for using perf to profile kvm is: <tt>perf kvm <perf kvm args> <perf command> <perf command args></tt>
In this case the perf kvm arguments are <tt>--host --guest --guestkallsyms=/tmp/guest.kallsyms --guestmodules=/tmp/guest.modules</tt>, the perf command is <tt>record</tt>, and the perf command argument is <tt>-a</tt> (profile all processes).

<tt>perf</tt> will record events until it is terminated with SIGINT.
'''It must be SIGINT.'''
If perf is terminated with any other signal, such as SIGTERM or SIGQUIT, the <tt>pref kvm report</tt> command (see below) won't correctly process the file generated.
It will just list the headers with no data.

==== Using sshfs ====

<tt>perf kvm</tt> has a way of getting the guest's <tt>kallsyms</tt> and <tt>modules</tt> by itself instead of you hving to provide them.
It makes use of <tt>sshfs</tt> to mount the root file system of the guest so that it can get the files directly from the guest.

<tt>sshfs</tt> depends on <tt>fuse</tt>, the Filesystem in Userspace.
If your Linux distribution does not have <tt>fuse</tt>, see the [http://fuse.sourceforge.net fuse project page on SourceForge].
If your Linux distribution does not have <tt>sshfs</tt>, see the [http://fuse.sourceforge.net/sshfs.html sshfs project page on SourceForge].

Here's how it works.
You create a directory, e.g., <tt>/tmp/guestmount</tt>.
You then create a subdirectory that has for its name the PID of the qemu process for the guest.
Then you use <tt>sshfs</tt> to mount the guest's root file system on that subdirectory.
For example:

<pre>
# mkdir -p /tmp/guestmount
# ps -eo pid,cmd | grep qemu | grep -v grep
24764 /usr/libexec/qemu-kvm -M pc -m 4096 -smp 4 -name guest01 -boot c -drive file=/var/lib/libvirt/images/guest01.img ...
# mkdir /tmp/guestmount/24764
# sshfs -o allow_other,direct_io guest:/ /tmp/guestmount/24764
</pre>

Then, instead of using the <tt>--guestkallsyms</tt> and <tt>--guestmodules</tt> options, you use the <tt>--guestmount</tt> option.
<pre>
# perf kvm --host --guest --guestmount=/tmp/guestmount
</pre>

When you are finished recording, unmount the guest file system using <tt>fusermount</tt>.
<pre>
# fusermount -u /tmp/guestmount/24764
</pre>

== Reporting events ==

Use the <tt>perf kvm report</tt> command to generate a report from a data file created by <tt>perf kvm record</tt>.
To generate a report for the host, use the <tt>--host</tt> argument.

<pre>
# perf kvm --host --guestmount=/tmp/guestmount report
</pre>

To generate a report for the guest, use the <tt>--guest</tt> argument.

<pre>
# perf kvm --guest --guestmount=/tmp/guestmount report
</pre>

By default the perf report command will read <tt>perf.data.host</tt> for a host report and <tt>perf.data.guest</tt> for a guest report.
Use the <tt>-i</tt> option if you want to use a different input file.

<pre>
# perf kvm --host --guestmount=/tmp/guestmount report -i /tmp/perf.data.kvm
</pre>

The <tt>perf report</tt> command sends its output to standard out. If you want the output to go to a file, redirect the output to a file.
Don't use the <tt>-o</tt> option.
Contrary to the help text, it does not work.

== Links ==

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/trace/ftrace.txt;h=03485bfbd7975792f50d54dabf77533938f255c9;hb=HEAD ftrace.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/trace/events.txt;h=02ac6ed38b2d0ba73c44298484b7d7bac540dae2;hb=HEAD events.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/trace/tracepoint-analysis.txt;h=87bee3c129ba71f8c359b5e849bafd177f421136;hb=HEAD tracepoint-analysis.txt]

KVM - User contributions [en]

Perf events