Systemtap in dyninst mode fails to hit PLT probes: ===== # rpm -q dyninst systemtap coreutils coreutils-debuginfo dyninst-8.1.2-6.el7.x86_64 systemtap-2.6-1.mcermak.70f3005.el7.x86_64 coreutils-8.22-11.el7.x86_64 coreutils-debuginfo-8.22-11.el7.x86_64 # stap -ve 'probe process.plt {log(pp())}' -c 'sleep 1' Pass 1: parsed user script and 113 library script(s) using 218004virt/35320res/3048shr/32728data kb, in 140usr/10sys/152real ms. Pass 2: analyzed script: 53 probe(s), 2 function(s), 0 embed(s), 0 global(s) using 219060virt/36812res/3600shr/33784data kb, in 10usr/0sys/7real ms. Pass 3: using cached /root/.systemtap/cache/b3/stap_b350e486c9dd3a7be755c587eee68f84_8320.c Pass 4: using cached /root/.systemtap/cache/b3/stap_b350e486c9dd3a7be755c587eee68f84_8320.ko Pass 5: starting run. process("/usr/bin/sleep").statement(0x401360) process("/usr/bin/sleep").statement(0x401310) process("/usr/bin/sleep").statement(0x401450) process("/usr/bin/sleep").statement(0x401280) process("/usr/bin/sleep").statement(0x401250) process("/usr/bin/sleep").statement(0x4014a0) process("/usr/bin/sleep").statement(0x4012d0) process("/usr/bin/sleep").statement(0x4012d0) process("/usr/bin/sleep").statement(0x401200) process("/usr/bin/sleep").statement(0x4012f0) process("/usr/bin/sleep").statement(0x401260) process("/usr/bin/sleep").statement(0x401200) process("/usr/bin/sleep").statement(0x401300) process("/usr/bin/sleep").statement(0x4014b0) process("/usr/bin/sleep").statement(0x401240) process("/usr/bin/sleep").statement(0x4013d0) process("/usr/bin/sleep").statement(0x401420) process("/usr/bin/sleep").statement(0x401420) process("/usr/bin/sleep").statement(0x4013f0) process("/usr/bin/sleep").statement(0x401270) process("/usr/bin/sleep").statement(0x401240) process("/usr/bin/sleep").statement(0x4013d0) process("/usr/bin/sleep").statement(0x401420) process("/usr/bin/sleep").statement(0x401420) process("/usr/bin/sleep").statement(0x4013f0) process("/usr/bin/sleep").statement(0x401270) Pass 5: run completed in 10usr/30sys/1302real ms. # stap --runtime=dyninst -ve 'probe process.plt {log(pp())}' -c 'sleep 1' Pass 1: parsed user script and 46 library script(s) using 191168virt/9052res/3024shr/5892data kb, in 80usr/10sys/87real ms. Pass 2: analyzed script: 53 probe(s), 2 function(s), 0 embed(s), 0 global(s) using 191696virt/10048res/3588shr/6420data kb, in 10usr/0sys/9real ms. Pass 3: using cached /root/.systemtap/cache/00/stap_007ce3f23360c54bb366cc6c6111b57a_8322.c Pass 4: using cached /root/.systemtap/cache/00/stap_007ce3f23360c54bb366cc6c6111b57a_8322.so Pass 5: starting run. stapdyn: WARNING: Couldn't instrument the function containing 0x4011c0, /usr/bin/sleep+0x11c0 stapdyn: WARNING: Couldn't instrument the function containing 0x4011d0, /usr/bin/sleep+0x11d0 stapdyn: WARNING: Couldn't instrument the function containing 0x4011e0, /usr/bin/sleep+0x11e0 stapdyn: WARNING: Couldn't instrument the function containing 0x4011f0, /usr/bin/sleep+0x11f0 stapdyn: WARNING: Couldn't instrument the function containing 0x401200, /usr/bin/sleep+0x1200 stapdyn: WARNING: Couldn't instrument the function containing 0x401210, /usr/bin/sleep+0x1210 stapdyn: WARNING: Couldn't instrument the function containing 0x401220, /usr/bin/sleep+0x1220 stapdyn: WARNING: Couldn't instrument the function containing 0x401230, /usr/bin/sleep+0x1230 stapdyn: WARNING: Couldn't instrument the function containing 0x401240, /usr/bin/sleep+0x1240 stapdyn: WARNING: Couldn't instrument the function containing 0x401250, /usr/bin/sleep+0x1250 stapdyn: WARNING: Couldn't find an instrumentation point at 0x401260, /usr/bin/sleep+0x1260 stapdyn: WARNING: Couldn't instrument the function containing 0x401270, /usr/bin/sleep+0x1270 stapdyn: WARNING: Couldn't instrument the function containing 0x401280, /usr/bin/sleep+0x1280 stapdyn: WARNING: Couldn't instrument the function containing 0x401290, /usr/bin/sleep+0x1290 stapdyn: WARNING: Couldn't instrument the function containing 0x4012a0, /usr/bin/sleep+0x12a0 stapdyn: WARNING: Couldn't instrument the function containing 0x4012b0, /usr/bin/sleep+0x12b0 stapdyn: WARNING: Couldn't instrument the function containing 0x4012c0, /usr/bin/sleep+0x12c0 stapdyn: WARNING: Couldn't instrument the function containing 0x4012d0, /usr/bin/sleep+0x12d0 stapdyn: WARNING: Couldn't find an instrumentation point at 0x4012e0, /usr/bin/sleep+0x12e0 stapdyn: WARNING: Couldn't instrument the function containing 0x4012f0, /usr/bin/sleep+0x12f0 stapdyn: WARNING: Couldn't instrument the function containing 0x401300, /usr/bin/sleep+0x1300 stapdyn: WARNING: Couldn't instrument the function containing 0x401310, /usr/bin/sleep+0x1310 stapdyn: WARNING: Couldn't instrument the function containing 0x401320, /usr/bin/sleep+0x1320 stapdyn: WARNING: Couldn't instrument the function containing 0x401330, /usr/bin/sleep+0x1330 stapdyn: WARNING: Couldn't instrument the function containing 0x401340, /usr/bin/sleep+0x1340 stapdyn: WARNING: Couldn't instrument the function containing 0x401350, /usr/bin/sleep+0x1350 stapdyn: WARNING: Couldn't instrument the function containing 0x401360, /usr/bin/sleep+0x1360 stapdyn: WARNING: Couldn't find an instrumentation point at 0x401370, /usr/bin/sleep+0x1370 stapdyn: WARNING: Couldn't instrument the function containing 0x401380, /usr/bin/sleep+0x1380 stapdyn: WARNING: Couldn't instrument the function containing 0x401390, /usr/bin/sleep+0x1390 stapdyn: WARNING: Couldn't instrument the function containing 0x4013a0, /usr/bin/sleep+0x13a0 stapdyn: WARNING: Couldn't instrument the function containing 0x4013b0, /usr/bin/sleep+0x13b0 stapdyn: WARNING: Couldn't instrument the function containing 0x4013c0, /usr/bin/sleep+0x13c0 stapdyn: WARNING: Couldn't instrument the function containing 0x4013d0, /usr/bin/sleep+0x13d0 stapdyn: WARNING: Couldn't instrument the function containing 0x4013e0, /usr/bin/sleep+0x13e0 stapdyn: WARNING: Couldn't find an instrumentation point at 0x4013f0, /usr/bin/sleep+0x13f0 stapdyn: WARNING: Couldn't instrument the function containing 0x401400, /usr/bin/sleep+0x1400 stapdyn: WARNING: Couldn't instrument the function containing 0x401410, /usr/bin/sleep+0x1410 stapdyn: WARNING: Couldn't instrument the function containing 0x401420, /usr/bin/sleep+0x1420 stapdyn: WARNING: Couldn't instrument the function containing 0x401430, /usr/bin/sleep+0x1430 stapdyn: WARNING: Couldn't instrument the function containing 0x401440, /usr/bin/sleep+0x1440 stapdyn: WARNING: Couldn't instrument the function containing 0x401450, /usr/bin/sleep+0x1450 stapdyn: WARNING: Couldn't instrument the function containing 0x401460, /usr/bin/sleep+0x1460 stapdyn: WARNING: Couldn't instrument the function containing 0x401470, /usr/bin/sleep+0x1470 stapdyn: WARNING: Couldn't instrument the function containing 0x401480, /usr/bin/sleep+0x1480 stapdyn: WARNING: Couldn't find an instrumentation point at 0x401490, /usr/bin/sleep+0x1490 stapdyn: WARNING: Couldn't find an instrumentation point at 0x4014a0, /usr/bin/sleep+0x14a0 stapdyn: WARNING: Couldn't instrument the function containing 0x4014b0, /usr/bin/sleep+0x14b0 stapdyn: WARNING: Couldn't instrument the function containing 0x4014c0, /usr/bin/sleep+0x14c0 stapdyn: WARNING: Couldn't instrument the function containing 0x4014d0, /usr/bin/sleep+0x14d0 stapdyn: WARNING: Couldn't find an instrumentation point at 0x4014e0, /usr/bin/sleep+0x14e0 stapdyn: WARNING: Couldn't find an instrumentation point at 0x4014f0, /usr/bin/sleep+0x14f0 stapdyn: WARNING: Couldn't find an instrumentation point at 0x401500, /usr/bin/sleep+0x1500 Pass 5: run completed in 3190usr/180sys/4382real ms. # ===== This observation originally comes from the process_by_pid testcase, where plt probes are being hit 0 times. Please, review.
Yes, I'm not surprised. Dyninst does instruction-level analysis to determine how to safely relocate code for instrumentation. But indirect jumps are opaque control flow to this analysis, and usually Dyninst just decides it's unsafe to proceed. In theory it would be fine, since we do know this is well-behaved, so perhaps Dyninst could learn a heuristic to identify and allow PLT instrumentation.
blocked on dyninst capabilities for the forseeable future