Bug 27975 - could systemtap probe user process running in docker(overlay2 fs)?
Summary: could systemtap probe user process running in docker(overlay2 fs)?
Status: RESOLVED WORKSFORME
Alias: None
Product: systemtap
Classification: Unclassified
Component: uprobes (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Unassigned
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-06-10 07:44 UTC by cong
Modified: 2021-10-08 03:15 UTC (History)
1 user (show)

See Also:
Host:
Target:
Build:
Last reconfirmed: 2021-06-10 00:00:00


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description cong 2021-06-10 07:44:54 UTC
## version info
systemtap version: 4.0/0.176
kernel version: 3.10.0-1160.25.1.el7.x86_64
os: CentOS Linux 7 (Core)

```bash
stap -V
Systemtap translator/driver (version 4.0/0.176, rpm 4.0-13.el7)
enabled features: AVAHI BOOST_STRING_REF DYNINST BPF JAVA PYTHON2 LIBRPM LIBSQLITE3 LIBVIRT LIBXML2 NLS NSS READLINE
```
## what i want 
i'm try to probe nginx(has debuginfo) which running in docker.

## what i have do
since nginx running in docker,so i just find the hostpath of the nginx in docker
and run 

`stap -x 27368  -e 'probe process.syscall {printf("%s %d %s\n", execname(), pid(), pp())} probe process("/var/lib/docker/overlay2/{a-long-long-string}/merged/usr/local/openresty/nginx/sbin/nginx").function("*") {printf("%s",probefunc())}' 

`process.syscall` works find,but nginx probe just nothing happen.

and 
`stap -x 27368 -e 'probe process.syscall {printf("%s %d %s\n", execname(), pid(), pp())} probe process("/usr/local/openresty/nginx/sbin/nginx").function("*") {printf("%s",probefunc())}' --sysroot=/var/lib/docker/overlay2/{a-long-long-string}/merged/
` 
report err like 

`WARNING: probe process("/usr/local/openresty/nginx/sbin/nginx").function("ngx_write_fd@src/os/unix/ngx_files.h:145") inode-offset 0000000000041e55 registration error (rc -22)`

## what i searched
i find some says [systemtap does not support overlays/union filesystems](https://stackoverflow.com/questions/57487292/systemtap-failed-to-probe-the-functions-registration-error)

there is samiler question asked in stackoverflow,but have to meanning answer [smailer question but no answer](https://stackoverflow.com/questions/29314665/systemtapwhile-resolving-probe-point-identifier-process-at-source-probe-pro)

[SystemTap_Beginners_Guide chapter 4 userspace-probing](https://www.sourceware.org/systemtap/SystemTap_Beginners_Guide/userspace-probing.html) just say you to point the exepath of the process.
## what i want to know
is that true? how should i probe user process which run in docker? 
what should i know to fix this question?
how to know what happen when uprobe just echo nothing?

## other info
```bash
 docker top a0bb57
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                20807               20755               0                   11:19               ?                   00:00:00            /sbin/tini -- /run.sh
root                20852               20807               0                   11:19               ?                   00:00:00            sh /run.sh
root                27368               20807               0                   11:20               ?                   00:00:00            nginx: master process /usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf
root                27369               27368               4                   11:20               ?                   00:10:00            nginx: worker process
root                27370               27368               0                   11:20               ?                   00:01:46            nginx: worker process
```

i have copy /lib/modules/3.10.0-1160.25.1.el7.x86_64/ to /var/lib/docker/overlay2/{a-long-long-string}/merged/
Comment 1 Frank Ch. Eigler 2021-06-10 14:45:35 UTC
Consider trying the probe-by-buildid mechanism added recently:

    probe process("deadbeefdeadbeef").FOO { }
instead of
    probe process("/path/to/binary").FOO { }

... where deadbeef is simply the hexadecimal buildid for the target binary, as printed by "file" or "readelf -n".

Your RHEL7 version of stap is much too old for that, but maybe you can get hold of RH DTS builds, or build your own from upstream sources / RPMs, this should work.

Otherwise, yeah, path name based matching is known to be difficult when there are many names for the same binary, and from different containers' points of view.
Comment 2 cong 2021-06-10 16:59:34 UTC
is there any way to distinguish between
1. systemtap does not probe to correct process (which should not happen because i set pid explicitly)
2. systemtap could not find those function which should probe
3. there is no event came out

since the method 1 i use in before code does not report any wran/erorr ,it seems that it probe to correct process and find the debuginfo and find the function i want to probe.  

a correspond conjecture is that: "is just no event came out".

did this logic chain make sense? or i just missing some point?
Comment 3 Frank Ch. Eigler 2021-06-14 17:42:29 UTC
Running a stap module built with super verbosity like

stap -DDEBUG_TASK_FINDER -DDEBUG_TASK_FINDER_VMA -DDEBUG_SYMBOLS -DDEBUG_UPROBES [...]

may let you see the difference.