ARM, SystemTap, userspace probing, and hardware assisted virtualization

[William Cohen <wcohen at redhat dot com>]

Last week Frank mentioned there were some patches for uprobes on ARM (https://blueprints.launchpad.net/linux-linaro/+spec/arm-uprobes).  I have taken a look at the patches and built a local kernel for my Samsung Chromebook that included those patches.  I thought this is cool!  These allow probing user-space applications on ARM. But then the question was "What would be an interesting user-space application to probe?"  Then I thought about another new feature that is available on ARM, kvm hardware acceleration for virtualization.  With some more tweaking of the 3.11 kernel I was able to get a kernel that supports KVM on arm. I also needed very new versions of elfutils, qemu, and systemtap rpms installed on the machine.

With the host set up I was able to set up a guest Fedora 18 vm based on the instructions at https://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express.  With this guest vm running I can run the qemu_count.stp example.  Things are not perfect (some args locations can't be parsed), but at least to the point of getting some interesting information on ARM:

[wcohen@chromebook-f19 testsuite]$ rpm -q qemu elfutils systemtapqemu-1.6.0-5.fc19.armv7hl
elfutils-0.156-5.fc19.armv7hl
systemtap-2.4-1.fc19.armv7hl
[wcohen@chromebook-f19 testsuite]$ uname -a
Linux chromebook-f19 3.11.0kvm-00011-gb393c6f #2 SMP Mon Sep 9 17:25:40 EDT 2013 armv7l armv7l armv7l GNU/Linux
[wcohen@chromebook-f19 testsuite]$ stap -v systemtap.examples/virtualization/qemu_count.stp 
Pass 1: parsed user script and 155 library script(s) using 134036virt/119520res/2556shr/120116data kb, in 5570usr/390sys/7532real ms.
WARNING: Can't parse SDT_V3 operand 'r10': identifier '$arg3' at /usr/share/systemtap/tapset/qemu-system-alpha.stp:310:9
 source:   ret = $arg3;
                 ^
WARNING: Can't parse SDT_V3 operand '[r4]': identifier '$arg4' at :102:17
 source:   format_name = $arg4;
                         ^
WARNING: Can't parse SDT_V3 operand '[r2]': identifier '$arg2' at :4238:10
 source:   name = $arg2;
                  ^
WARNING: Can't parse SDT_V3 operand '[r1]': identifier '$arg2' at :4243:10
 source:   name = $arg2;
                  ^
WARNING: Can't parse SDT_V3 operand '[sp]': identifier '$arg1' at :3638:9
 source:   mon = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[r7]': identifier '$arg1' at :3443:11
 source:   iscsi = $arg1;
                   ^
WARNING: Can't parse SDT_V3 operand '[r6]': identifier '$arg1' at :2714:9
 source:   cmd = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[r8]': identifier '$arg1' at :2746:9
 source:   cmd = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[r5]': identifier '$arg1' at :2781:9
 source:   cmd = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[ip]': identifier '$arg1' at :2763:9
 source:   cmd = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[r0]': identifier '$arg1' at :2704:9
 source:   cmd = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[r10, #196]': identifier '$arg3' at :280:15
 source:   in_flight = $arg3;
                       ^
WARNING: Can't parse SDT_V3 operand '[r3]': identifier '$arg3' at :1033:9
 source:   nxt = $arg3;
                 ^
WARNING: Can't parse SDT_V3 operand '[r10, #56]': identifier '$arg5' at :1753:12
 source:   length = $arg5;
                    ^
WARNING: Can't parse SDT_V3 operand '[r10, #52]': identifier '$arg4' at :1752:12
 source:   status = $arg4;
                    ^
WARNING: Can't parse SDT_V3 operand '[r10, #36]': identifier '$arg6' at :1745:10
 source:   size = $arg6;
                  ^
WARNING: Can't parse SDT_V3 operand '[r10, #60]': identifier '$arg1' at :963:9
 source:   bus = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[r10, #4]': identifier '$arg1' at :3742:9
 source:   tag = $arg1;
                 ^
WARNING: Can't parse SDT_V3 operand '[r10, #6]': identifier '$arg2' at :3743:8
 source:   id = $arg2;
                ^
Pass 2: analyzed script: 14518 probe(s), 1 function(s), 2 embed(s), 1 global(s) using 587232virt/563912res/5128shr/563756data kb, in 40300usr/4690sys/59483real ms.
Pass 3: using cached /home/wcohen/.systemtap/cache/d1/stap_d1c63eb2fc55869b856116eb5f46dfad_2482326.c
Pass 4: using cached /home/wcohen/.systemtap/cache/d1/stap_d1c63eb2fc55869b856116eb5f46dfad_2482326.ko
Pass 5: starting run.
^Cqemu.system.arm.bdrv_co_io_em = 37733
qemu.system.arm.bdrv_co_readv = 13160
qemu.system.arm.bdrv_co_writev = 24573
qemu.system.arm.kvm_run_exit = 5908038
qemu.system.arm.kvm_vcpu_ioctl = 5908037
qemu.system.arm.kvm_vm_ioctl = 499559
qemu.system.arm.memory_region_ops_read = 2481865
qemu.system.arm.memory_region_ops_write = 3426173
qemu.system.arm.object_dynamic_cast_assert = 499559
qemu.system.arm.paio_submit = 37733
qemu.system.arm.qemu_co_queue_run_restart = 113199
qemu.system.arm.qemu_coroutine_enter = 75466
qemu.system.arm.qemu_coroutine_terminate = 37733
qemu.system.arm.qemu_coroutine_yield = 37733
qemu.system.arm.thread_pool_complete = 37733
qemu.system.arm.thread_pool_submit = 37733
Pass 5: run completed in 20usr/140sys/190875real ms.

-Will