Systemtap versus Java hotspot work TODO

[Mark Wielaard <mark at klomp dot org>]

As followup to the java versus systemtap "It Works!" post a little
overview of what actually fully works and what still needs to be done.
Help with any of the TODO items highly appreciated since I don't
immediately have time to work on any of it.

The easiest to get your "hands dirty" with this is to install systemtap
0.9.9, fetch the IcedTea sources:
$ hg clone http://icedtea.classpath.org/hg/icedtea6
And build them: $ ./autogen.sh && ./configure --enable-systemtap && make
All patches needed (discussed below) are contained in
patches/icedtea-systemtap.patch

What do we have now
-------------------

- Static probes in the hotspot runtime that reuse the dtrace markers for
  use with systemtap.

  The hotspot runtime sources have been cleaned up so they can be
  compiled with -DDTRACE_ENABLED when systemtap-sdt-devel sys/sdt.h is
  around. This was mainly removing some #ifdef SOLARIS and some sun
  compiler oddities.
http://icedtea.classpath.org/hg/icedtea6/raw-file/tip/patches/icedtea-systemtap.patch
  
- A nice hotspot.stp tapset that makes the above "raw probes" available
  through nicer aliases and predefined variables.
http://icedtea.classpath.org/hg/icedtea6/raw-file/tip/tapset/hotspot.stp.in

  This mainly follows the documentation from here:
  http://java.sun.com/javase/6/docs/technotes/guides/vm/dtrace.html

The above gives you "meta probing" of the java vm (vm start/stop,
threads, garbage collector, class loading, monitor events, etc.)

stap -e 'probe hotspot.* { log(probestr) }' -c 'java Hello'
gives you output for probes and the variables available when running a
program. Some extra probes become available when running with java
-XX:+ExtendedDTraceProbes.

And since systemtap 0.9.9 also java method entry/exit work.

Both the interpreter and the jit (when given -XX:+ExtendedDTraceProbes)
call into the shared runtime functions which define probe points, which
stap can now see.

IcedTea added the following two hotspot tapset helpers:

/* hotspot.method_entry (extended probe)
   Triggers when a method is entered.
   Sets thread_id to the current java thread id, class to the name of
   the class, method to the name of the method, and sig to the
   signature string of the method.
   Needs -XX:+ExtendedDTraceProbes.
*/

/* hotspot.method_return (extended probe)
   Triggers when a method returns.
   Sets thread_id to the current java thread id, class to the name of
   the class, method to the name of the method, and sig to the
   signature string of the method.
   Needs -XX:+ExtendedDTraceProbes.
*/

And with that we can now trace java method calls/returns:

$ stap -DMAXSTRINGLEN=512 -DSTP_NO_OVERLOAD -e \
  'probe hotspot.method_entry {log(thread_indent(1) . "=> " . class .
  "." . method . sig)} \
  probe hotspot.method_return {log(thread_indent(-1) . "<= " . class .
  "." . method . sig)}' -c java -XX:+ExtendedDTraceProbes Hello'

  0 java(9760):=> java/lang/Thread.<clinit>()V
 13 java(9760): => java/lang/Thread.registerNatives()V
 33 java(9760): <= java/lang/Thread.registerNatives()V
306 java(9760): => java/lang/RuntimePermission.<init>(Ljava/lang/String;)V
318 java(9760):  => java/security/BasicPermission.<init>(Ljava/lang/String;)V
326 java(9760):   => java/security/Permission.<init>(Ljava/lang/String;)V
334 java(9760):    => java/lang/Object.<init>()V
341 java(9760):    <= java/lang/Object.<init>()V
349 java(9760):   <= java/security/Permission.<init>(Ljava/lang/String;)V

We need to look into whether we can increase the MAXSTRINGLEN by
default (PR10486) and why the STP_NO_OVERLOAD is needed. The
STP_NO_OVERLOAD is only needed on larger runs and might be because
method entry/exit is such a frequent thing that logging them all is just
insane.

Although the way this is done isn't so advanced. These probes are only
there with -XX:+ExtendedDTraceProbes and then it needs to
"DeoptimizeTheWorld" to recompile each and every method to compile in
the new call into the shared runtime function that has the probe. This
is because you cannot select individual java methods to probe, it is all
or nothing (which also probably explains the overload issue above).

What needs to be done
---------------------

= Bug fixes/Extensions

- There is one probe not found hotspot.monitor_notify.
  Unknown why, needs investigation. This is a translation time issue.
  hotspot.monitor_wait, hotspot.monitor_waited and
  hotspot.monitor_notifyAll do work. The probe seems to be there, but
  stap doesn't want to see it. It translates it (through the .probes
  section) to a statement probe and then fails.
  Needs more investigation.

- The JNI provider probes aren't compiled at the moment.  This would
  let you trace all native method invocation calls.  The hotspot
  sources do something "really clever" by defining C preprocessor macros
  for getting markers in all the entry and exit sports (using
  generated c++ destructors). Needs some strong cpp-foo. see
  openjdk/hotspot/src/share/vm/prims/jni.cpp
  There is some improvement since 0.9.9 in that the code now compiles,
  but then the final link fails because of missing labels?

make[6]: Entering directory
`/usr/local/build/icedtea6-obj/openjdk-ecj/build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/product'
{ \
            echo Linking launcher...; \
             \
            gcc -m64 -Xlinker -O1 -m64 -export-dynamic  -L `pwd` -o gamma
launcher.o -ljvm -lm -ldl -lpthread; \
             \
        }
Linking launcher...
/usr/local/build/icedtea6-obj/openjdk-ecj/build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/product/libjvm.so: undefined reference to `.L4457'
/usr/local/build/icedtea6-obj/openjdk-ecj/build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/product/libjvm.so: undefined reference to `.L4494'
[...lots more...[
/usr/local/build/icedtea6-obj/openjdk-ecj/build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/product/libjvm.so: undefined reference to `.L4490'
collect2: ld returned 1 exit status
make[6]: *** [gamma] Error 1
make[6]: Leaving directory
`/usr/local/build/icedtea6-obj/openjdk-ecj/build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/product'

So I assume the systemtap sys/sdt/h macros/defines that also generate
asm using labels somehow don't play nice with the jni.cpp macros that
use them. Needs more investigation.

- It would be nice if the -XX:+ExtendedDTraceProbes were always
  available. For systemtap Stan is adding an EnabledProbe macro
  http://sourceware.org/bugzilla/show_bug.cgi?id=10013
  That would make it possible to switch probes on/off during runtime
  so you don't have to change any startup script to include the extra
  java option. Needs investigation of how this can/is compatible with
  something similar to dtrace, so that it benefits everybody.
  Sun engineers at this years Fosdem said they wanted to add something
  like that, but I haven't seen any progress on this front. So it would
  be a nice collaboration between the dtrace and systemtap teams to get
  this forward.

  Documentation for the comparable dtrace mechanism is really hard to
  find, as far as I know it is only documented in this mailinglist post
  and blog entry:
  http://www.opensolaris.org/jive/thread.jspa?messageID=31314
  http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better

  There is also the "jinfo" application that could be used to switch
  all extended probes on "jinfo -flag +ExtendedDTraceProbes", but this
  seems to hang if the process is already being traced by stap atm.
  Needs investigation.

= Java backtraces

This would be a two stage process. First getting the backtrace
addresses, then turning those addresses into user readable java
class/method names.

Systemtap can use the dwarf unwinder to get backtrace addresses, but
hotspot is explicitly compiled with -fno-omit-frame-pointer, so this
should be easy (needs double checking of systemtap runtime backtrace
fallback code).

Then for each frame we need to turn it into a user string that
resembles the java class/method being used (somewhat). The idea here
would be to try to mimic how dtrace peals this out of the running
VM. dtrace uses "helper functions" to decode the backtrace by
iterating over each address and calling the application specific
"decoding function" on each one.

The main datastructures used by the jit and interpreter are turned
into a JvmOffsetsIndex.h file with lots of constants at compile time
through the program generated with
hotspot/src/os/solaris/dtrace/generateJvmOffsets.cpp.

This "JvmOffsetsIndex.h" is then included in a dtrace script (dtrace
scripts run the C preprocessor on themselves first, we need some
mechanism for the stap script to also easily include these
constants).

Then there is an elaborate "dscript"
hotspot/src/os/solaris/dtrace/jhelper.d
It defines a ustack helper function which gets called (in the probe
context, e.g. in the kernel) for each frame with the PC as arg0 and
the frame pointer as arg1 it returns a string describing the frame.

The script deduces the class, methods and type names in the constant
pool (using the JvmOffsets, the pc and the frame pointer) and
"copyin"s those strings, concatenates them and returns the result.
All this should also be possible using a stap script.