Workarounds for very useful developer userspace tracing scripts

[Jim Keniston]

On Fri, 2007-06-08 at 18:07 +0100, argetek senvx wrote:
> Hi guys,
> 
> I know that the userspace tracing is in works, and I keep checking the
> lists to see if any progress has been made.  But I am tired of going
> over to my Solaris VM all the time to execute the these three DTrace
> scripts or variations thereof:
> 
>         -  Which system calls a particular userspace function in a
>            process invokes
> 
>         -  Aggregate user stacks calling a particular function in
>            userspace
> 
>         -  Good ol' userspace function trace with call flow
>            indentation.
> 
> I was wondering if there's any guess as to when the userspace tracing
> will become available.  But in the meantime, are there any workarounds
> for these scripts that I could use on Linux?
> 
> Regards,
> 
>       ~ argetek

The kernel infrastructure that SystemTap is exploiting for user-space
tracing (utrace and uprobes) is in pretty good shape.  For example, a
kernel module that provides a CRUDE solution for your #1 request above
is attached (argetek1.c).

BTW, #3 (also CRUDE) is in our test suite.  #2 would benefit from
SystemTap's ability to use a stack trace as an index to an associative
array.

Sample usage: You want to trace the syscalls called by function
do_some_syscalls() in the attached program tracemyscs.c.

Window #1
$ objdump -d tracemysc | grep '<do_some_syscalls>:'
080485d4 <do_some_syscalls>:
$ ./tracemysc 2
I am process 9084.  Attach and press Enter:

Window #2
# insmod argetek1.ko func=0x080485d4 pid=9084

Window #1
[Enter]
2 interations in 0.023112 sec
$

Window #2
# rmmod argetek1

What's logged to /var/log/messages is something like this:

Jun  8 11:47:06 ibm-ni9dztukfq8 kernel: Registering uprobe on pid 9084,
vaddr 0x80485d4
Jun  8 11:47:06 ibm-ni9dztukfq8 kernel: Registering return probe on pid
9084, vaddr 0x80485d4
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: Function at 0x80485d4 called
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #45 returns
0x804a000
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #45 returns
0x806b000
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #5 returns 0x3
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #197 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #192 returns
0xb7fcb000
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #4 returns 0x12
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #6 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #91 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #10 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: Function at 0x80485d4 returns
0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: Function at 0x80485d4 called
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #5 returns 0x3
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #197 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #192 returns
0xb7fcb000
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #4 returns 0x12
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #6 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #91 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 9084: syscall #10 returns 0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: Function at 0x80485d4 returns
0x0
Jun  8 11:47:12 ibm-ni9dztukfq8 kernel: 
Jun  8 11:47:21 ibm-ni9dztukfq8 kernel: Unregistering probes on pid
9084, vaddr 0x80485d4

A more friendly interface (a la strace or ltrace) is left as an exercise
for SystemTap. :-}

http://sourceware.org/ml/systemtap/2007-q2/msg00399.html has the latest
uprobes patches and a pointer to utrace.

Jim Keniston
-------------- next part --------------
A non-text attachment was scrubbed...
Name: argetek1.c
Type: text/x-csrc
Size: 3910 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/systemtap/attachments/20070608/cb48f23e/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tracemysc.c
Type: text/x-csrc
Size: 1067 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/systemtap/attachments/20070608/cb48f23e/attachment-0001.bin>