On some arches (interestingly, this only seems to be a problem on ppc64le and s390x, but not on x86_64 or aarch64) the stap -L regexp match doesn't work for a function name in case the source file isn't specified. If the source file is specified (a base name without any kind of path is good enough) then the match works fine. Details: 9 ppc64le # stap -L 'process("/lib64/libc.so.6").function("malloc_init_state")' 9 ppc64le # stap -L 'process("/lib64/libc.so.6").function("malloc_init_state*")' process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-48.el9.ppc64le/malloc/malloc.c:1932") $av:mstate 9 ppc64le # stap -L 'process("/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-48.el9.ppc64le/malloc/malloc.c")' process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-48.el9.ppc64le/malloc/malloc.c:1932") $av:mstate 9 ppc64le # stap -L 'process("/lib64/libc.so.6").function("malloc_init_state@malloc.c")' process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-48.el9.ppc64le/malloc/malloc.c:1932") $av:mstate 9 ppc64le # stap -L 'process("/lib64/libc.so.6").function("malloc_init_state")' 9 ppc64le # This is with kernel 5.14.0-186.kpq1.el9.ppc64le and stap upstream commit e51e989e33b0f0a0cf26d00755340cef3c2ea81f .
I am able to replicates this on s390x RHEL9 and a build of the current git checkout of systemtap commit 3ecf49eee2ab7e7: [root@s390x-kvm-032 systemtap]# rpm -q kernel kernel-5.14.0-362.8.1.el9_3.s390x [root@s390x-kvm-032 systemtap]# ../install/bin/stap -L 'process("/lib64/libc.so.6").function("malloc_init_state")' [root@s390x-kvm-032 systemtap]# ../install/bin/stap -L 'process("/lib64/libc.so.6").function("malloc_init_state*")' process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/malloc.c:1932") $av:mstate $tcache:tcache_perthread_struct* On a similar s390x RHEL8 system with the same checkout don't see this issue: [root@s390x-kvm-003 systemtap]# rpm -q kernel kernel-4.18.0-372.9.1.el8.s390x [root@s390x-kvm-003 systemtap]# ../install/bin/stap -L 'process("/lib64/libc.so.6").function("malloc_init_state")' process("/usr/lib64/libc-2.28.so").function("malloc_init_state@/usr/src/debug/glibc-2.28-189.1.el8.s390x/malloc/malloc.c:1803") $av:mstate $i:int [root@s390x-kvm-003 systemtap]# ../install/bin/stap -L 'process("/lib64/libc.so.6").function("malloc_init_state*")' process("/usr/lib64/libc-2.28.so").function("malloc_init_state@/usr/src/debug/glibc-2.28-189.1.el8.s390x/malloc/malloc.c:1803") $av:mstate $i:int
Did some more poking around to see what the differences are between the rhel8 and rhel9 libc.so.6 code. It appears that the problem is linked to functions being inlined on rhel9. On rhel9 the particular function being searched for is inlined while on rhel8 it is an actual function. RHEL9: # ../install/bin/stap -v -L 'process("/lib64/libc.so.6").function("malloc_init_state*").*' Pass 1: parsed user script and 489 library scripts using 127816virt/106228res/10964shr/94752data kb, in 240usr/40sys/286real ms. process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/malloc.c:1932").inline /* pc=.dynamic+0xab576 */ $av:mstate $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/malloc.c:1932").inline /* pc=.dynamic+0xab9e0 */ $av:mstate $tcache:tcache_perthread_struct* Pass 2: analyzed script: 2 probes, 0 functions, 0 embeds, 0 globals using 152984virt/132752res/12260shr/119920data kb, in 130usr/10sys/142real ms. 0xab576 is inlined in arena_get2.part.0 0xab9e0 is inlined in ptmalloc_init.part.0 on rhel8 see .call and .return: # ../install/bin/stap -v -L 'process("/lib64/libc.so.6").function("malloc_init_state*").*' Pass 1: parsed user script and 480 library scripts using 122520virt/102760res/12516shr/89836data kb, in 230usr/20sys/248real ms. process("/usr/lib64/libc-2.28.so").function("malloc_init_state@/usr/src/debug/glibc-2.28-189.1.el8.s390x/malloc/malloc.c:1803").call /* pc=.dynamic+0x92870 */ $av:mstate $i:int process("/usr/lib64/libc-2.28.so").function("malloc_init_state@/usr/src/debug/glibc-2.28-189.1.el8.s390x/malloc/malloc.c:1803").return /* pc=.dynamic+0x92870 */ $av:mstate $i:int Pass 2: analyzed script: 2 probes, 0 functions, 0 embeds, 0 globals using 143736virt/125236res/13712shr/111052data kb, in 110usr/10sys/124real ms. When trying "stap -v -L ..." on rhel9 libc.so.6 functions that were regular functions no trailing '*' was required. However, for inlined functions needed the '*' to get them listed out. For example ptmalloc_init. [root@s390x-kvm-032 systemtap]# ../install/bin/stap -v -L 'process("/lib64/libc.so.6").function("ptmalloc_init").inline' Pass 1: parsed user script and 489 library scripts using 127812virt/106224res/10964shr/94748data kb, in 230usr/30sys/309real ms. Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 152848virt/132860res/12452shr/119784data kb, in 110usr/10sys/124real ms. [root@s390x-kvm-032 systemtap]# ../install/bin/stap -v -L 'process("/lib64/libc.so.6").function("ptmalloc_init*").inline' Pass 1: parsed user script and 489 library scripts using 127816virt/106228res/10964shr/94752data kb, in 230usr/30sys/274real ms. process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xb02b4 */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xb0210 */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xaffa0 */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xafdf8 */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xaf950 */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xaf71e */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xaf6aa */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xaf656 */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xaf31c */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xaea7c */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xb04a2 */ $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("ptmalloc_init@/usr/src/debug/glibc-2.34-83.el9_3.7.s390x/malloc/arena.c:281").inline /* pc=.dynamic+0xb0504 */ $tcache:tcache_perthread_struct* Pass 2: analyzed script: 12 probes, 0 functions, 0 embeds, 0 globals using 152984virt/132764res/12260shr/119920data kb, in 120usr/10sys/129real ms.
This looks like an issue with systemtap understanding the debuginfo generated as part of the rpmbuild process. When locally building the glibc-2.34-83.el9_3.7.src.rpm on the machine the following work fine on the unstripped glibc.so.6: [root@s390x-kvm-033 build-s390x-redhat-linux]# ~/systemtap_write/install/bin/stap -v -L 'process("./libc.so.6").function("malloc_init_state")' Pass 1: parsed user script and 489 library scripts using 127764virt/106232res/10972shr/94700data kb, in 230usr/30sys/256real ms. process("/root/rpmbuild/BUILD/glibc-2.34/build-s390x-redhat-linux/libc.so").function("malloc_init_state@/root/rpmbuild/BUILD/glibc-2.34/malloc/malloc.c:1932") /* pc=.dynamic+0xab576 */ $av:mstate process("/root/rpmbuild/BUILD/glibc-2.34/build-s390x-redhat-linux/libc.so").function("malloc_init_state@/root/rpmbuild/BUILD/glibc-2.34/malloc/malloc.c:1932") /* pc=.dynamic+0xab9e0 */ $av:mstate Pass 2: analyzed script: 2 probes, 0 functions, 0 embeds, 0 globals using 137564virt/117272res/12220shr/104500data kb, in 50usr/0sys/57real ms. However, install the rpms created from that same build a wildcard is needed to find malloc_init_state: #cd rpmbuild/RPMS/s390x/ #dnf install -y ./glibc-gconv-extra-2.34-83.el9.7.s390x.rpm ./glibc-langpack-en-2.34-83.el9.7.s390x.rpm ./glibc-common-2.34-83.el9.7.s390x.rpm ./glibc-2.34-83.el9.7.s390x.rpm ./glibc-headers-2.34-83.el9.7.s390x.rpm ./glibc-devel-2.34-83.el9.7.s390x.rpm ./glibc-debugsource-2.34-83.el9.7.s390x.rpm ./glibc-debuginfo-2.34-83.el9.7.s390x.rpm ./glibc-static-2.34-83.el9.7.s390x.rpm # ~/systemtap_write/install/bin/stap -v -L 'process("/lib64/libc.so.6").function("malloc_init_state")' Pass 1: parsed user script and 489 library scripts using 127764virt/106232res/10972shr/94700data kb, in 230usr/30sys/259real ms. Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 151728virt/131664res/12412shr/118664data kb, in 100usr/0sys/113real ms. # ~/systemtap_write/install/bin/stap -v -L 'process("/lib64/libc.so.6").function("malloc_init_state*")' Pass 1: parsed user script and 489 library scripts using 127768virt/106236res/10972shr/94704data kb, in 240usr/30sys/254real ms. process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-83.el9.7.s390x/malloc/malloc.c:1932") /* pc=.dynamic+0xab576 */ $av:mstate $tcache:tcache_perthread_struct* process("/usr/lib64/libc.so.6").function("malloc_init_state@/usr/src/debug/glibc-2.34-83.el9.7.s390x/malloc/malloc.c:1932") /* pc=.dynamic+0xab9e0 */ $av:mstate $tcache:tcache_perthread_struct* Pass 2: analyzed script: 2 probes, 0 functions, 0 embeds, 0 globals using 151820virt/131536res/12220shr/118756data kb, in 110usr/0sys/120real ms.