This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Can we do Java probing without invoking stapbm from staprun?
- From: Tetsuo Handa <penguin-kernel at I-love dot SAKURA dot ne dot jp>
- To: systemtap at sourceware dot org
- Date: Sat, 23 Sep 2017 20:51:11 +0900
- Subject: Can we do Java probing without invoking stapbm from staprun?
- Authentication-results: sourceware.org; auth=none
Hello.
I'm using modified version of /usr/libexec/systemtap/stapbm in order to obtain more
specific values of an object which cannot be obtained by implicit .toString() conversion.
For example, passing $1.getRequestURI() instead of plain $1 when probing
service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)
method of javax.servlet.http.HttpServlet class. There should be no problem for SystemTap
side as long as number of arguments matches, for all values are passed to SystemTap side
as string because of PR21020.
Now, I'm trying to go further in order to make
"allow probing without giving permission to run staprun to the user who runs
Java application (e.g. tomcat)"
"automatically start probing as soon as target java process starts (or restarts)
as with normal userspace probing"
possible by not invoking stapbm from staprun.
Steps for demonstrating such usage is shown below. Can we legally support such usage?
(1) Write a script which hooks nonexistent application name / class name / method name
and compile the script. Make sure rule name does not get long by specifying module
name via -m option.
[root@localhost ~]# /usr/bin/stap -p4 -m stap_java -e 'probe java("/").class("class").method("method_enter(java.lang.String, java.lang.String, java.lang.String, java.lang.String)") { println("Enter " . arg1 . " " . arg2 . " " . arg3 . " " . arg4) }
probe java("/").class("class").method("method_return(java.lang.String)").return { println("Return " . arg1) }'
stap_java.ko
(2) Create a dummy /usr/libexec/systemtap/stapbm in order to confirm rule names
included in the module compiled at (1).
[root@localhost ~]# cp -p /usr/libexec/systemtap/stapbm /usr/libexec/systemtap/stapbm.orig
[root@localhost ~]# (echo '#!/bin/sh'; echo 'echo "$@" 1>&2'; echo 'exit 0') > /usr/libexec/systemtap/stapbm
(3) Load the module compiled at (1) in order to confirm rule names. In the example
shown below, the rule names are stap_javaprobe_4113 and stap_javaprobe_4114.
[root@localhost ~]# /usr/bin/staprun stap_java.ko
# /usr/bin/staprun stap_java.ko
install31 / stap_javaprobe_4113 class method_enter(java.lang.String, java.lang.String, java.lang.String, java.lang.String) 4 entry 0
install31 / stap_javaprobe_4114 class method_return(java.lang.String) 1 exit 0
^Cuninstall31 / stap_javaprobe_4113 class method_enter(java.lang.String, java.lang.String, java.lang.String, java.lang.String) 4 entry 0
uninstall31 / stap_javaprobe_4114 class method_return(java.lang.String) 1 exit 0
(4) Create /usr/libexec/systemtap/stapbm2 based on stapbm. Specify rule names
confirmed at (3) into echo_bytemanrule().
[root@localhost ~]# cp -p /usr/libexec/systemtap/stapbm.orig /usr/libexec/systemtap/stapbm2
[root@localhost ~]# vi /usr/libexec/systemtap/stapbm2
[root@localhost ~]# cat /usr/libexec/systemtap/stapbm2
#!/bin/bash
# $1 - PID/unique name
exec 1>&2 # redirect byteman/etc. tracing output to stderr, for easier filtering
mode=install
stap="31"
arg_jvmpid=$1
SYSTEMTAP_DIR=${SYSTEMTAP_DIR-$HOME/.systemtap}
BYTEMAN_HOME=${BYTEMAN_HOME-/usr/share/java/byteman}
JAVA_HOME=${JAVA_HOME-/usr/lib/jvm/java}
BYTEMAN_INSTALL_OPTS=${BYTEMAN_INSTALL_OPTS--Dorg.jboss.byteman.transform.all=true}
SYSTEMTAP_VERBOSE=${SYSTEMTAP_VERBOSE-0}
if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
BYTEMAN_INSTALL_OPTS="$BYTEMAN_INSTALL_OPTS -Dorg.jboss.byteman.verbose"
else
exec >/dev/null 2>&1
fi
# the byteman and byteman-submit jars should be in ${BYTEMAN_HOME}/lib
BYTEMAN_JAR=${BYTEMAN_HOME}/byteman.jar
if [ ! -r ${BYTEMAN_JAR} ]; then
echo "Missing $BYTEMAN_JAR"
exit 1
fi
BYTEMAN_SUBMIT_JAR=${BYTEMAN_HOME}/byteman-submit.jar
if [ ! -r ${BYTEMAN_SUBMIT_JAR} ]; then
echo "Missing $BYTEMAN_SUBMIT_JAR"
exit 1
fi
BYTEMAN_INSTALL_JAR=${BYTEMAN_HOME}/byteman-install.jar
if [ ! -r ${BYTEMAN_INSTALL_JAR} ]; then
echo "Missing $BYTEMAN_INSTALL_JAR"
exit 1
fi
TOOLS_JAR=${JAVA_HOME}/lib/tools.jar
if [ ! -f ${TOOLS_JAR} ]; then
echo "Missing $TOOLS_JAR"
exit 1
fi
# resolve $*prefix fully
prefix=/usr
exec_prefix=/usr
pkglibexecdir=/usr/libexec/systemtap
pkglibexecdir=`eval echo $pkglibexecdir`
pkglibexecdir=`eval echo $pkglibexecdir`
HELPERSDT_JAR=${pkglibexecdir}/HelperSDT.jar
if [ ! -f ${HELPERSDT_JAR} ]; then
echo "Missing $HELPERSDT_JAR"
exit 1
fi
# The JVM that invokes byteman will get classpath/etc. settings from
# this shell script to look directly under our $prefix. However, for
# the target JVM, the HelperSDT* stuff needs to be installed under
# that JVM's paths.
num=`ls -1 ${JAVA_HOME}/jre/lib/ext/HelperSDT.jar ${JAVA_HOME}/jre/lib/*/libHelperSDT_*.so 2>/dev/null | wc -l`
if [ $num -lt 2 ]; then
echo "Missing HelperSDT JNI class/shared library"
echo "Install them like this, as root:"
echo ""
echo "for so in ${pkglibexecdir}/libHelperSDT_*.so; do"
echo ' arch=`basename $so | cut -f2 -d_ | cut -f1 -d.`'
echo " ln -sf ${pkglibexecdir}/libHelperSDT_"'${arch}'".so ${JAVA_HOME}/jre/lib/"'${arch}'"/"
echo "done"
echo "ln -sf ${pkglibexecdir}/HelperSDT.jar ${JAVA_HOME}/jre/lib/ext/"
# exit 1
fi
flagdir="$SYSTEMTAP_DIR/java"
mkdir -p $flagdir
# Find our target jvm pid. Due to the possibility of our
# target jvm pid being passed as a string, we need to allow
# for the possiblity that more than one pid may match the
# target jvm pid. If this is the case, we need to have a
# nested call to stapbm with the actual pid of the jvm pid
if ! [[ $arg_jvmpid =~ ^[0-9]+$ ]]; then
target_pid=`jps -l | grep $arg_jvmpid | cut -f1 -d" "`
for target in $target_pid; do
$0 $target
done;
exit 0
else
target_pid=$arg_jvmpid
fi
# Our target jvm may not have the byteman agent installed yet. Let's do
# that first. We use a signal file in $flagdir to show that the
# JVM is ready for further bytemanning without a prior setup step,
# and include in it the designated byteman agent listening-port number.
#
byteman_installed_portfile=$flagdir/`hostname`-${target_pid}-bm
exec 200>>$byteman_installed_portfile # open/create lock file
flock -x 200 # exclusive-lock it
if [ -s $byteman_installed_portfile ]; then
bmport=`cat $byteman_installed_portfile`
if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
echo "Byteman agent reused for java pid $target_pid, port $bmport"
fi
# XXX: liveness-check the port; bmsubmit with no argument just lists current rules
# if fails, delete the _portfile and retry everything
else
# bmport=9091
bmport=`expr 9090 + $RANDOM % 10000`
existing=`netstat -atn | awk '{print $4}' | grep ':'$bmport'$'`
if [ "x$existing" != "x" ]; then
echo "Byteman port $bmport already in use, retrying."
exec "$@"
fi
# There are two ways to invoke and run byteman operations with the jvm's we're interested
# in, we can alter the startup arguments to include a -javaagent parameter, or use
# byteman and its use of VMAttach libraries, for our case it always makes sense to use
# byteman classes directly and avoid -javaagent
if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
echo java -classpath ${BYTEMAN_INSTALL_JAR}:${BYTEMAN_JAR}:${TOOLS_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.install.Install -b -p $bmport $BYTEMAN_INSTALL_OPTS $target_pid
fi
java -classpath ${BYTEMAN_INSTALL_JAR}:${BYTEMAN_JAR}:${TOOLS_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.install.Install -b -p $bmport $BYTEMAN_INSTALL_OPTS $target_pid
if [ $? -ne 0 ]; then
echo "Byteman agent failed to install for java pid $target_pid, port $bmport"
exit 1
fi
echo $bmport > $byteman_installed_portfile
if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
echo "Byteman agent installed for java pid $target_pid, port $bmport"
fi
# XXX: Erase file to keep it from sticking around indefinitely,
# in case process ends, machine reboots, pid gets reused
# XXX: consider explicit notification to stapbm via process("java").begin/end ?
# ... or else: liveness-check below
fi
exec 200>&- # close file & release flock
function echo_bytemanrule()
{
echo 'RULE stap_javaprobe_4113'
echo 'CLASS javax.servlet.http.HttpServlet'
echo 'METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)'
echo 'HELPER org.systemtap.byteman.helper.HelperSDT'
echo 'AT ENTRY'
echo 'IF TRUE'
echo 'DO METHOD_STAP31_PROBE4("stap_javaprobe_4113", $1, $1.getRequestURI(), $2, "")'
echo 'ENDRULE'
echo 'RULE stap_javaprobe_4114'
echo 'CLASS javax.servlet.http.HttpServlet'
echo 'METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)'
echo 'HELPER org.systemtap.byteman.helper.HelperSDT'
echo 'AT RETURN'
echo 'IF TRUE'
echo 'DO METHOD_STAP31_PROBE1("stap_javaprobe_4114", "done")")'
echo 'ENDRULE'
}
# Generate the byteman rule file on-the-fly
btmfile=$flagdir/`hostname`-$$.btm
echo_bytemanrule > $btmfile
trap 'rm -f $btmfile' 0 1 2 3 4 5 9 15
if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
echo "Byteman rule file:"
cat $btmfile
fi
bmcmd=-l
if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
echo java -classpath ${BYTEMAN_SUBMIT_JAR}:${BYTEMAN_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.submit.Submit -p $bmport $bmcmd $btmfile
fi
exec java -classpath ${BYTEMAN_SUBMIT_JAR}:${BYTEMAN_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.submit.Submit -p $bmport $bmcmd $btmfile
(5) Load the module compiled at (1) in order to start tracing.
[root@localhost ~]# /usr/bin/staprun stap_java.ko
(6) Start target Java process. In the example shown below, tomcat is chosen.
[root@localhost ~]# systemctl start tomcat
(7) Run /usr/libexec/systemtap/stapbm2 in order to enable hooks for tracing
target Java process (via runuser as needed for switching uid).
[root@localhost ~]# SYSTEMTAP_VERBOSE=1 runuser -u tomcat /usr/libexec/systemtap/stapbm2 org.apache.catalina.startup.Bootstrap
java -classpath /usr/share/java/byteman/byteman-install.jar:/usr/share/java/byteman/byteman.jar:/usr/lib/jvm/java/lib/tools.jar:/usr/libexec/systemtap/HelperSDT.jar org.jboss.byteman.agent.install.Install -b -p 16175 -Dorg.jboss.byteman.transform.all=true -Dorg.jboss.byteman.verbose 12131
byteman jar is /usr/share/java/byteman/byteman.jar
Byteman agent installed for java pid 12131, port 16175
Byteman rule file:
RULE stap_javaprobe_4113
CLASS javax.servlet.http.HttpServlet
METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)
HELPER org.systemtap.byteman.helper.HelperSDT
AT ENTRY
IF TRUE
DO METHOD_STAP31_PROBE4("stap_javaprobe_4113", $1, $1.getRequestURI(), $2, "")
ENDRULE
RULE stap_javaprobe_4114
CLASS javax.servlet.http.HttpServlet
METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)
HELPER org.systemtap.byteman.helper.HelperSDT
AT RETURN
IF TRUE
DO METHOD_STAP31_PROBE1("stap_javaprobe_4114", "done")")
ENDRULE
java -classpath /usr/share/java/byteman/byteman-submit.jar:/usr/share/java/byteman/byteman.jar:/usr/libexec/systemtap/HelperSDT.jar org.jboss.byteman.agent.submit.Submit -p 16175 -l /usr/share/tomcat/.systemtap/java/localhost.localdomain-12215.btm
install rule stap_javaprobe_4113
install rule stap_javaprobe_4114
(8) Confirm that trace output is sent to staprun side started at (5) when hooks
with application name specified at (7) and class name / method name specified
at (4) are called.
Enter org.apache.catalina.connector.RequestFacade@44009cf2 /tomcat.css org.apache.catalina.connector.ResponseFacade@43ff6bf
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /tomcat.png org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /asf-logo-wide.svg org.apache.catalina.connector.ResponseFacade@1f3c368
Enter org.apache.catalina.connector.RequestFacade@44009cf2 /bg-nav.png org.apache.catalina.connector.ResponseFacade@43ff6bf
Return done
Return done
Enter org.apache.catalina.connector.RequestFacade@7b9aeef0 /bg-button.png org.apache.catalina.connector.ResponseFacade@78bd0df5
Enter org.apache.catalina.connector.RequestFacade@43df8dc0 /bg-upper.png org.apache.catalina.connector.ResponseFacade@9d35b10
Return done
Return done
Enter org.apache.catalina.connector.RequestFacade@12ff7c7a /favicon.ico org.apache.catalina.connector.ResponseFacade@43453754
Return done
Enter org.apache.catalina.connector.RequestFacade@75338e4c /bg-middle.png org.apache.catalina.connector.ResponseFacade@bc35bb1
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/ org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets/ org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@12ff7c7a /examples/servlets/images/code.gif org.apache.catalina.connector.ResponseFacade@43453754
Return done
Enter org.apache.catalina.connector.RequestFacade@44009cf2 /examples/servlets/images/return.gif org.apache.catalina.connector.ResponseFacade@43ff6bf
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets/images/execute.gif org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets/servlet/HelloWorldExample org.apache.catalina.connector.ResponseFacade@1f3c368
Return done