Re: Analysis of broken systemtap example scripts

[William Cohen <wcohen at redhat dot com>]

On 10/10/18 11:50 AM, William Cohen wrote:
> Hi,
> 
> I am going through the systemtap example scripts and see what things
> are still broken and what needs to be done to fix them.
> 
> The tapset update due to the changes in the kernel changes in syscall
> mechanisms has affected a number of scripts.  socket-events.stp used
> internal __syscall.send.return and __syscall.sendmsg.return (marking
> those with 'I').  Also a number of scripts assume that the dwarf based
> kprobe/kretprobe were being used and had @entry() available for target
> variables (marking those with 'E').  Target variable availablility is
> questionable since the tapsets often fall to using the non-dwarf
> variant which don't provide target variables ('T') or guru mode
> changing of varialble ('G')
> 
> FAIL: systemtap.examples/general/socket-events build (IE)
> FAIL: systemtap.examples/io/iotime build (E)
> FAIL: systemtap.examples/process/futexes build (E)
> FAIL: systemtap.examples/process/futexes2 build (E)
> FAIL: systemtap.examples/process/mutex-contention build (E)
> FAIL: systemtap.examples/io/eatmydata build (TG)
> FAIL: systemtap.examples/process/noptrace build (TG)
> 
> Is there some way to make the dwarf-based probes work on newer 4.18
> kernel?  If that was the case the tests above would just work.
> 
> For scripts that expect the entry target variable it is possible to
> rewrite them to eliminate the @entry() uses.  Attached is a patch that
> does that for futexes.stp futexes2.stp, and mutex-contention.stp.  The
> main downside is there is more contention for the global associative
> array and a lot more skipped probes.

I have checked in the change for futexes.stp, futexes2.stp and mutex-contention.stp.  I am looking into a slightly different way to address the issue with iotime.stp.  Trying to use non-dwarf access to arguments in the @entry.  This works on older kernels but not the newer fedora kernel.  I wrote a little script to illustrate the issue.

[wcohen@paketa ~]$ uname -a
Linux paketa 3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[wcohen@paketa ~]$ cat open_args.stp 
function proc:string() { return sprintf("%d (%s)", pid(), execname()) }

probe syscall.openat {
  printf("%s %s (%x) %s\n", proc(), pp(), pointer_arg(2), filename)
}

probe syscall.openat.return {
  filename = @entry(user_string_quoted(pointer_arg(2)))
  printf("%s %s (%x) %s\n", proc(), pp(), @entry(pointer_arg(2)), filename)
}
[wcohen@paketa ~]$ stap open_args.stp  -T 10
989 (systemd-journal) kernel.function("SyS_openat@fs/open.c:1071").call? (558a7fa23530) "/run/log/journal/1415a05ffedc445fa4381c4d5fcadcbf"
989 (systemd-journal) kernel.function("SyS_openat@fs/open.c:1071").return? (558a7fa23530) "/run/log/journal/1415a05ffedc445fa4381c4d5fcadcbf"
11897 (setroubleshootd) kernel.function("SyS_openat@fs/open.c:1071").call? (cd58e0) "/usr/lib64/python2.7/site-packages"
11897 (setroubleshootd) kernel.function("SyS_openat@fs/open.c:1071").return? (cd58e0) "/usr/lib64/python2.7/site-packages"
11897 (setroubleshootd) kernel.function("SyS_openat@fs/open.c:1071").call? (cd58e0) "/usr/lib/python2.7/site-packages"
11897 (setroubleshootd) kernel.function("SyS_openat@fs/open.c:1071").return? (cd58e0) "/usr/lib/python2.7/site-packages"
11897 (setroubleshootd) kernel.function("SyS_openat@fs/open.c:1071").call? (e489f0) "//etc/selinux/targeted/policy"
11897 (setroubleshootd) kernel.function("SyS_openat@fs/open.c:1071").return? (e489f0) "//etc/selinux/targeted/policy"
...


However, running the same script on a newer fedora kernel:


[wcohen@cervelo io]$ uname -a
Linux cervelo 4.18.11-200.fc28.x86_64 #1 SMP Sun Sep 30 15:31:40 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[wcohen@cervelo io]$ stap open_args.stp -T 1
1335 (libvirtd) kprobe.function("__x64_sys_openat")? (7f331f4b66e0) "/sys/devices/system/node/node0/meminfo"
1335 (libvirtd) kprobe.function("__x64_sys_openat").return? (ffffb97f445eff58) 0xffffb97f445eff58
1335 (libvirtd) kprobe.function("__x64_sys_openat")? (7f33371bbb37) "/proc/cpuinfo"
1335 (libvirtd) kprobe.function("__x64_sys_openat").return? (ffffb97f445eff58) 0xffffb97f445eff58
1335 (libvirtd) kprobe.function("__x64_sys_openat")? (7f3318028f10) "/sys/devices/system/cpu/present"
1335 (libvirtd) kprobe.function("__x64_sys_openat").return? (ffffb97f445eff58) 0xffffb97f445eff58
1335 (libvirtd) kprobe.function("__x64_sys_openat")? (7f331801a170) "/sys/devices/system/cpu/online"
1335 (libvirtd) kprobe.function("__x64_sys_openat").return? (ffffb97f445eff58) 0xffffb97f445eff58
...

The pointers and strings look okay on the entry probe, but are corrupted for in the return probe even when they are in an @entry().

-Will