Analysis of broken systemtap example scripts

[William Cohen <wcohen at redhat dot com>]

Hi,

I am going through the systemtap example scripts and see what things
are still broken and what needs to be done to fix them.

The tapset update due to the changes in the kernel changes in syscall
mechanisms has affected a number of scripts.  socket-events.stp used
internal __syscall.send.return and __syscall.sendmsg.return (marking
those with 'I').  Also a number of scripts assume that the dwarf based
kprobe/kretprobe were being used and had @entry() available for target
variables (marking those with 'E').  Target variable availablility is
questionable since the tapsets often fall to using the non-dwarf
variant which don't provide target variables ('T') or guru mode
changing of varialble ('G')

FAIL: systemtap.examples/general/socket-events build (IE)
FAIL: systemtap.examples/io/iotime build (E)
FAIL: systemtap.examples/process/futexes build (E)
FAIL: systemtap.examples/process/futexes2 build (E)
FAIL: systemtap.examples/process/mutex-contention build (E)
FAIL: systemtap.examples/io/eatmydata build (TG)
FAIL: systemtap.examples/process/noptrace build (TG)

Is there some way to make the dwarf-based probes work on newer 4.18
kernel?  If that was the case the tests above would just work.

For scripts that expect the entry target variable it is possible to
rewrite them to eliminate the @entry() uses.  Attached is a patch that
does that for futexes.stp futexes2.stp, and mutex-contention.stp.  The
main downside is there is more contention for the global associative
array and a lot more skipped probes.


The following test failed because a target variable was no longer
available or couldn't be found.  Some of this is because the function
is now being inline and the parameter just doesn't exist.

FAIL: systemtap.examples/general/varwatch build (inlined function)
FAIL: systemtap.examples/io/capture_ssl_master_secrets build
FAIL: systemtap.examples/network/tcp_trace build (inlined function)

Changes in the kernel internals caused the following tests to fail:
FAIL: systemtap.examples/profiling/periodic build
FAIL: systemtap.examples/process/pfiles build

Perf counters not available on guest vm:
FAIL: systemtap.examples/profiling/perf run

Not sure what is happening with the following test:
FAIL: systemtap.examples/general/badname run


-Will

diff --git a/testsuite/systemtap.examples/process/futexes.stp b/testsuite/systemtap.examples/process/futexes.stp
index 84f000e5f..1a2fb93e4 100755
--- a/testsuite/systemtap.examples/process/futexes.stp
+++ b/testsuite/systemtap.examples/process/futexes.stp
@@ -10,11 +10,22 @@ global FUTEX_CLOCK_REALTIME = 256 /* linux 2.6.29+ */
 global lock_waits # long-lived stats on (tid,lock) blockage elapsed time
 global process_names # long-lived pid-to-execname mapping
 
-probe syscall.futex.return {  
-  if (($op & ~(FUTEX_PRIVATE_FLAG|FUTEX_CLOCK_REALTIME)) != FUTEX_WAIT) next
-  process_names[pid()] = execname()
-  elapsed = gettimeofday_us() - @entry(gettimeofday_us())
-  lock_waits[pid(), $uaddr] <<< elapsed
+global entry_times, uaddrs
+
+probe syscall.futex.return {
+  if (!(entry_times[tid()])) next
+  if (!(pid() in process_names))
+    process_names[pid()] = execname()
+  elapsed = gettimeofday_us() - entry_times[tid()]
+  lock_waits[pid(), uaddrs[tid()]] <<< elapsed
+  delete entry_times[tid()]
+  delete uaddrs[tid()]
+}
+
+probe syscall.futex {
+  if ((op & ~(FUTEX_PRIVATE_FLAG|FUTEX_CLOCK_REALTIME)) != FUTEX_WAIT) next
+  entry_times[tid()] = gettimeofday_us()
+  uaddrs[tid()] = futex_uaddr
 }
 
 probe end {
diff --git a/testsuite/systemtap.examples/process/futexes2.stp b/testsuite/systemtap.examples/process/futexes2.stp
index 4c9dd1948..b371139e5 100644
--- a/testsuite/systemtap.examples/process/futexes2.stp
+++ b/testsuite/systemtap.examples/process/futexes2.stp
@@ -35,17 +35,24 @@ probe kernel.function("futex_wait_setup").return {
   }
 }
 
+global entry_times
+
+probe syscall.futex {
+  if ((op & ~(FUTEX_PRIVATE_FLAG|FUTEX_CLOCK_REALTIME)) != FUTEX_WAIT) next
+  entry_times[tid()] = gettimeofday_us()
+}
+
 probe syscall.futex.return {  
   if (!(tid() in wait_keys)) next
   key = wait_keys[tid()]
   delete wait_keys[tid()]
 
-  cmd = @entry($op) & ~(FUTEX_PRIVATE_FLAG|FUTEX_CLOCK_REALTIME);
-  if (cmd != FUTEX_WAIT) next
+  if (!(tid() in entry_times)) next
 
   process_names[pid()] = execname()
-  elapsed = gettimeofday_us() - @entry(gettimeofday_us())
+  elapsed = gettimeofday_us() - entry_times[tid()]
   lock_waits[pid(), key] <<< elapsed
+  delete entry_times[tid()]
 }
 
 probe end {
diff --git a/testsuite/systemtap.examples/process/mutex-contention.stp b/testsuite/systemtap.examples/process/mutex-contention.stp
index 66a67878f..69bd89655 100755
--- a/testsuite/systemtap.examples/process/mutex-contention.stp
+++ b/testsuite/systemtap.examples/process/mutex-contention.stp
@@ -92,14 +92,25 @@ probe process("/lib*/libpthread.so*").function("__pthread_rwlock_init")
     process_mutex_init($rwlock, ppfunc())
 }
 
+global entry_times, uaddrs
+
+probe syscall.futex {
+  if (pid() != target()) next		// skip irrelevant processes
+  if ((op & ~(FUTEX_PRIVATE_FLAG|FUTEX_CLOCK_REALTIME)) != FUTEX_WAIT) next
+  entry_times[tid()] = gettimeofday_us()
+  uaddrs[tid()] = futex_uaddr
+}
+
 probe syscall.futex.return
 {
-  op = @entry($op)
-  uaddr = @entry($uaddr)
-  if ((op & ~(FUTEX_PRIVATE_FLAG|FUTEX_CLOCK_REALTIME)) != FUTEX_WAIT) next
-  if (pid() != target()) next		// skip irrelevant processes
+  if (!(tid() in entry_times)) next
+  entry = entry_times[tid()]
+  delete entry_times[tid()]
+
+  uaddr = uaddrs[tid()]
+  delete uaddrs[tid()]
 
-  elapsed = gettimeofday_us() - @entry(gettimeofday_us())
+  elapsed = gettimeofday_us() - entry
   if (keep_stats) {
     mutex_contention[uaddr] <<< elapsed
     stack = sprint_ubacktrace()