[Qemu-devel] Using the qemu tracepoints with SystemTap

[Stefan Hajnoczi]

On Mon, Sep 12, 2011 at 4:33 PM, William Cohen <wcohen@redhat.com> wrote:
> The RHEL-6 version of qemu-kvm makes the tracepoints available to SystemTap. I have been working on useful examples for the SystemTap tracepoints in qemu. There doesn't seem to be a great number of examples showing the utility of the tracepoints in diagnosing problems. However, I came across the following blog entry that had several examples:
>
> http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html
>
> I reimplemented the VirtqueueRequestTracker example from the blog in SystemTap (the attached virtqueueleaks.stp). I can run it on RHEL-6's qemu-kvm-0.12.1.2-2.160.el6_1.8.x86_64 and get output like the following. It outputs the pid and the address of the elem that leaked when the script is stopped like the following:
>
> $ stap virtqueueleaks.stp
> ^C
>     pid     elem
>   19503  1c4af28
>   19503  1c56f88
>   19503  1c62fe8
>   19503  1c6f048
>   19503  1c7b0a8
>   19503  1c87108
>   19503  1c93168
> ...
>
> I am not that familiar with the internals of qemu. The script seems to indicates qemu is leaking, but is that really the case?  If there are resource leaks, what output would help debug those leaks? What enhancements can be done to this script to provide more useful information?

Leak tracing always has corner cases :).

With virtio-blk this would indicate a leak because it uses a
request-response model where the guest initiates I/O and the host
responds.  A guest that cleanly shuts down before you exit your
SystemTap script should not leak requests for virtio-blk.

With virtio-net the guest actually hands the host receive buffers and
they are held until we can receive packets into them and return them
to the host.  We don't have a virtio_reset trace event, and due to
this we're not accounting for clean shutdown (the guest driver resets
the device to clear all virtqueues).

I am submitting a patch to add virtio_reset() tracing.  This will
allow the script to delete all elements belonging to this virtio
device.

> Are there other examples of qemu probing people would like to see?

The malloc/realloc/memalign/vmalloc/free/vfree trace events can be
used for a few things:
 * Finding memory leaks.
 * Finding malloc/vfree or vmalloc/free mismatches.  The rules are:
malloc/realloc need free, memalign/vmalloc need vfree.  They cannot be
mixed.

Stefan