On archlinux, with kernel 4.18.16-arch1-1-ARCH, systemtap (version 4.0/0.174, non-git sources), elfutils 0.174. If use stap on target process originated from system packages, print_ubacktrace() works fine: $ stap -v -e 'probe process.function("*") { print_ubacktrace(); exit() }' -x 4924 Pass 1: parsed user script and 470 library scripts using 75324virt/51500res/6800shr/44864data kb, in 100usr/10sys/110real ms. Pass 2: analyzed script: 234 probes, 2 functions, 0 embeds, 0 globals using 77040virt/54192res/7668shr/46580data kb, in 10usr/0sys/11real ms. Pass 3: translated to C into "/tmp/stapOV3tv1/stap_1bc1f6af99b98d562e15bc3b6e4ee1a2_52098_src.c" using 77184virt/54628res/7924shr/46724data kb, in 10usr/100sys/111real ms. Pass 4: compiled C into "stap_1bc1f6af99b98d562e15bc3b6e4ee1a2_52098.ko" in 3070usr/460sys/3418real ms. Pass 5: starting run. WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry with -DDEBUG_UNWIND)' 0x55c3c7f8f9c0 : ProcessList_scan+0x0/0x1a0 [/usr/bin/htop] 0x55c3c7f90497 : ScreenManager_run+0x107/0x11e0 [/usr/bin/htop] 0x55c3c7f86c51 : main+0x451/0x660 [/usr/bin/htop] 0x7f067850d223 Pass 5: run completed in 10usr/20sys/1033real ms. However if I compile a program manually, stap can only print an address, without function name: $ stap -v -DSTP_NO_BUILDID_CHECK -e 'probe process.function("*") { print_ubacktrace(); exit() }' -x 5465 Pass 1: parsed user script and 470 library scripts using 75332virt/51852res/7144shr/44872data kb, in 90usr/10sys/106real ms. Pass 2: analyzed script: 341 probes, 2 functions, 0 embeds, 0 globals using 77444virt/55028res/8080shr/46984data kb, in 10usr/0sys/14real ms. Pass 3: using cached /root/.systemtap/cache/3d/stap_3d92b79fdcedbfc63746fbaa7ff985f2_90414.c Pass 4: using cached /root/.systemtap/cache/3d/stap_3d92b79fdcedbfc63746fbaa7ff985f2_90414.ko Pass 5: starting run. WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry with -DDEBUG_UNWIND)' 0x5608a9394100 Pass 5: run completed in 10usr/20sys/1038real ms. Is it the compatible error between new gcc versions and systemtap (or its dependencies like elfutils) ? Thanks.
Sorry forgot to provide gcc version, it's gcc (GCC) 8.2.1 20180831
It depends how the two versions of the target program were compiled. Perhaps the second one was stripped (including symbol info), so stap (and tools like gdb) would have no chance to resolve addresses to names. Perhaps the unwind data was not generated the same way (possibly CFLAGS without -fasynchronous-unwind-tables?). A comparison with "readelf -S" between the two binaries could help clear this up. Also, that advice re. rerunning with -DDEBUG_UNWIND is a good one, the data would help.
(In reply to Frank Ch. Eigler from comment #2) > It depends how the two versions of the target program were compiled. > Perhaps the second one was stripped (including symbol info), so stap (and > tools like gdb) would have no chance to resolve addresses to names. Perhaps > the unwind data was not generated the same way (possibly CFLAGS without > -fasynchronous-unwind-tables?). > > A comparison with "readelf -S" between the two binaries could help clear > this up. > > Also, that advice re. rerunning with -DDEBUG_UNWIND is a good one, the data > would help. Thanks. Actually the first one was strippped (but with right print_ubacktrace()), the second not: $ file /bin/htop /bin/htop: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=f124a5c165c15004fab233235b393e078a52f4f5, stripped $ file ./htop ./htop: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=b6c11a7eda8d30b90a46f940fc9178e3665b399a, not stripped I used CFLAGS and LDFLAGS as below, to make it same as package built by archlinux (-fasynchronous-unwind-tables is added by your advice, but the problem still occurs): CFLAGS="-fasynchronous-unwind-tables -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt " LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" With -DDEBUG_UNWIND: $ stap -v -DSTP_NO_BUILDID_CHECK -DDEBUG_UNWIND -e 'probe process.function("*") { print_ubacktrace(); exit() }' -x 1174 Pass 1: parsed user script and 470 library scripts using 75332virt/51916res/7208shr/44872data kb, in 90usr/20sys/112real ms. Pass 2: analyzed script: 341 probes, 2 functions, 0 embeds, 0 globals using 77444virt/55156res/8208shr/46984data kb, in 10usr/10sys/15real ms. Pass 3: translated to C into "/tmp/stap9fZbm5/stap_c1cd55b8b696f02a065ad588361b3e36_90426_src.c" using 77884virt/55708res/8400shr/47424data kb, in 20usr/100sys/119real ms. Pass 4: compiled C into "stap_c1cd55b8b696f02a065ad588361b3e36_90426.ko" in 3170usr/420sys/3493real ms. Pass 5: starting run. WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry with -DDEBUG_UNWIND)' _stp_stack_unwind_one_user:496: STARTING user unwind 0x56235b12d100 _stp_stack_unwind_one_user:512: CONTINUING user unwind to depth 1 unwind:1483: pc=56235b12d100, 56235b12d100 unwind:1522: No module found for pc=56235b12d100 _stp_stack_unwind_one_user:537: ret=-22 PC=56235b12d100 SP=7ffedbacaaa8 Pass 5: run completed in 10usr/30sys/1238real ms. readelf output: $ LANG=en_US.UTF-8 readelf -S ./htop There are 27 section headers, starting at offset 0x33b18: Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align [ 0] NULL 0000000000000000 00000000 0000000000000000 0000000000000000 0 0 0 [ 1] .interp PROGBITS 00000000000002a8 000002a8 000000000000001c 0000000000000000 A 0 0 1 [ 2] .note.ABI-tag NOTE 00000000000002c4 000002c4 0000000000000020 0000000000000000 A 0 0 4 [ 3] .note.gnu.build-i NOTE 00000000000002e4 000002e4 0000000000000024 0000000000000000 A 0 0 4 [ 4] .gnu.hash GNU_HASH 0000000000000308 00000308 0000000000000fa0 0000000000000000 A 5 0 8 [ 5] .dynsym DYNSYM 00000000000012a8 000012a8 00000000000029d0 0000000000000018 A 6 1 8 [ 6] .dynstr STRTAB 0000000000003c78 00003c78 00000000000019b8 0000000000000000 A 0 0 1 [ 7] .gnu.version VERSYM 0000000000005630 00005630 000000000000037c 0000000000000002 A 5 0 2 [ 8] .gnu.version_r VERNEED 00000000000059b0 000059b0 00000000000000a0 0000000000000000 A 6 2 8 [ 9] .rela.dyn RELA 0000000000005a50 00005a50 00000000000055b0 0000000000000018 A 5 0 8 [10] .init PROGBITS 000000000000b000 0000b000 000000000000001b 0000000000000000 AX 0 0 4 [11] .text PROGBITS 000000000000b020 0000b020 0000000000014df5 0000000000000000 AX 0 0 16 [12] .fini PROGBITS 000000000001fe18 0001fe18 000000000000000d 0000000000000000 AX 0 0 4 [13] .rodata PROGBITS 0000000000020000 00020000 00000000000034a0 0000000000000000 A 0 0 16 [14] .eh_frame_hdr PROGBITS 00000000000234a0 000234a0 0000000000000a84 0000000000000000 A 0 0 4 [15] .eh_frame PROGBITS 0000000000023f28 00023f28 0000000000003df0 0000000000000000 A 0 0 8 [16] .init_array INIT_ARRAY 0000000000029c50 00028c50 0000000000000008 0000000000000008 WA 0 0 8 [17] .fini_array FINI_ARRAY 0000000000029c58 00028c58 0000000000000008 0000000000000008 WA 0 0 8 [18] .data.rel.ro PROGBITS 0000000000029c60 00028c60 0000000000000d90 0000000000000000 WA 0 0 32 [19] .dynamic DYNAMIC 000000000002a9f0 000299f0 00000000000001d0 0000000000000010 WA 6 0 8 [20] .got PROGBITS 000000000002abc0 00029bc0 0000000000000438 0000000000000008 WA 0 0 8 [21] .data PROGBITS 000000000002b000 0002a000 0000000000002a80 0000000000000000 WA 0 0 32 [22] .bss NOBITS 000000000002da80 0002ca80 0000000000000eb0 0000000000000000 WA 0 0 32 [23] .comment PROGBITS 0000000000000000 0002ca80 000000000000001a 0000000000000001 MS 0 0 1 [24] .symtab SYMTAB 0000000000000000 0002caa0 0000000000004218 0000000000000018 25 260 8 [25] .strtab STRTAB 0000000000000000 00030cb8 0000000000002d5f 0000000000000000 0 0 1 [26] .shstrtab STRTAB 0000000000000000 00033a17 00000000000000fd 0000000000000000 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), l (large), p (processor specific)
I find out which really matters is the version of compiler (maybe different version of gcc turn on different default options or enable new default functions) and the executable file it generate. Because if I compile a program on another host with gcc 6.3.0, and copy it to current host and running stap again, print_ubacktrace() works as expected.
There is a recent systemic problem in the way userspace probes are identified, and especially ubacktrace()d-from. It appears to be something that relates to the different way modern executables are linked / loaded.
commit 4ae4592f1106e941023a5768d34c2381cc869631 fixes