I'm having the following compilation error compiling a stap script in ARM with SystemTap 3.2: $ sudo stap -v -g -p4 -DSTP_NO_OVERLOAD systemtap-strace.stp -m stap_trace Compiling systemtap module ... ./systemtap-strace.stp -> ./stap_trace.ko Pass 1: parsed user script and 452 library scripts using 35092virt/26404res/2260shr/24360data kb, in 4950usr/190sys/5187real ms. Pass 2: analyzed script: 734 probes, 230 functions, 104 embeds, 63 globals using 68040virt/60300res/3180shr/57308data kb, in 2348300usr/36720sys/2415777real ms. Pass 3: translated to C into "/tmp/stapBqLAih/stap_trace_src.c" using 68040virt/60492res/3412shr/57308data kb, in 121940usr/440sys/124281real ms. /tmp/stapBqLAih/stap_trace_src.c:198942:28: error: field ‘ops’ has incomplete type /tmp/stapBqLAih/stap_trace_src.c: In function ‘systemtap_module_init’: /tmp/stapBqLAih/stap_trace_src.c:199900:9: error: implicit declaration of function ‘stap_utrace_detach_ops’ [-Werror=implicit-function-declaration] cc1: all warnings being treated as errors make[1]: *** [/tmp/stapBqLAih/stap_trace_src.o] Error 1 make: *** [_module_/tmp/stapBqLAih] Error 2 WARNING: kbuild exited with status: 2 Pass 4: compiled C into "stap_trace.ko" in 143290usr/13730sys/166637real ms. Pass 4: compilation failed. [man error::pass4] - The script is the same that is running in the a x86 machine using the same SystemTap version. - Systemtap seems to be working with simple scripts: $ sudo stap -ve 'probe begin { log("hello world") exit () }' [sudo] password for jd: Pass 1: parsed user script and 452 library scripts using 35216virt/26348res/2244shr/24484data kb, in 5990usr/570sys/6758real ms. Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 35612virt/27116res/2600shr/24880data kb, in 90usr/0sys/92real ms. Pass 3: translated to C into "/tmp/stapUh7ouq/stap_3bc557db95fcecfd9b772ad47121f60e_1080_src.c" using 35612virt/27440res/2892shr/24880data kb, in 20usr/0sys/18real ms. Pass 4: compiled C into "stap_3bc557db95fcecfd9b772ad47121f60e_1080.ko" in 123230usr/12460sys/139093real ms. Pass 5: starting run. hello world Pass 5: run completed in 100usr/300sys/746real ms. Versions: - Custom Kernel 3.2.93-versatile $ uname -a Linux alex-14f9b262f2 3.2.93-versatile-gcm #1 Wed Dec 6 11:56:12 EST 2017 armv5tejl GNU/Linux - Gcc 4.7.2 (Yes, I'm aware about the C++11 standard support needed, but please have a look to https://sourceware.org/bugzilla/show_bug.cgi?id=22572). - SystemTap 3.2/0.152, was compiled from the oficial release sources, just patching staptree.h due to the C++11 issue). --- staptree.h_ori 2017-12-10 23:23:25.872864000 -0500 +++ staptree.h 2017-12-10 23:23:36.332864000 -0500 @@ -518,10 +518,10 @@ unsigned base; unsigned width; unsigned precision; - unsigned flags : 8; - width_type widthtype : 8; - precision_type prectype : 8; - conversion_type type : 8; + unsigned flags; + width_type widthtype; + precision_type prectype; + conversion_type type; interned_string literal_string; bool is_empty() const { Any clue? I could attach a full (-vv) output and the temporary files (-k) if it is needed. Cheers, Gus
(In reply to Gustavo Moreira from comment #0) > I'm having the following compilation error compiling a stap script in ARM > with SystemTap 3.2: > > $ sudo stap -v -g -p4 -DSTP_NO_OVERLOAD systemtap-strace.stp -m stap_trace > Compiling systemtap module ... > ./systemtap-strace.stp -> ./stap_trace.ko > Pass 1: parsed user script and 452 library scripts using > 35092virt/26404res/2260shr/24360data kb, in 4950usr/190sys/5187real ms. > Pass 2: analyzed script: 734 probes, 230 functions, 104 embeds, 63 globals > using 68040virt/60300res/3180shr/57308data kb, in > 2348300usr/36720sys/2415777real ms. > Pass 3: translated to C into "/tmp/stapBqLAih/stap_trace_src.c" using > 68040virt/60492res/3412shr/57308data kb, in 121940usr/440sys/124281real ms. > /tmp/stapBqLAih/stap_trace_src.c:198942:28: error: field ‘ops’ has > incomplete type > /tmp/stapBqLAih/stap_trace_src.c: In function ‘systemtap_module_init’: > /tmp/stapBqLAih/stap_trace_src.c:199900:9: error: implicit declaration of > function ‘stap_utrace_detach_ops’ [-Werror=implicit-function-declaration] > cc1: all warnings being treated as errors Interesting. The task_finder functionality (part of the user-space tracing), needs for several kernel tracepoints to exist. If they don't exist, you should have either gotten a compile error or they should have semi-gracefully just refused to work. Is CONFIG_TRACEPOINTS set in your custom kernel?
Created attachment 10680 [details] stap temporary files
(In reply to David Smith from comment #1) > (In reply to Gustavo Moreira from comment #0) > > I'm having the following compilation error compiling a stap script in ARM > > with SystemTap 3.2: > > > > $ sudo stap -v -g -p4 -DSTP_NO_OVERLOAD systemtap-strace.stp -m stap_trace > > Compiling systemtap module ... > > ./systemtap-strace.stp -> ./stap_trace.ko > > Pass 1: parsed user script and 452 library scripts using > > 35092virt/26404res/2260shr/24360data kb, in 4950usr/190sys/5187real ms. > > Pass 2: analyzed script: 734 probes, 230 functions, 104 embeds, 63 globals > > using 68040virt/60300res/3180shr/57308data kb, in > > 2348300usr/36720sys/2415777real ms. > > Pass 3: translated to C into "/tmp/stapBqLAih/stap_trace_src.c" using > > 68040virt/60492res/3412shr/57308data kb, in 121940usr/440sys/124281real ms. > > /tmp/stapBqLAih/stap_trace_src.c:198942:28: error: field ‘ops’ has > > incomplete type > > /tmp/stapBqLAih/stap_trace_src.c: In function ‘systemtap_module_init’: > > /tmp/stapBqLAih/stap_trace_src.c:199900:9: error: implicit declaration of > > function ‘stap_utrace_detach_ops’ [-Werror=implicit-function-declaration] > > cc1: all warnings being treated as errors > > Interesting. The task_finder functionality (part of the user-space tracing), > needs for several kernel tracepoints to exist. If they don't exist, you > should have either gotten a compile error or they should have > semi-gracefully just refused to work. > > Is CONFIG_TRACEPOINTS set in your custom kernel? Hi David, Yes it is # grep CONFIG_TRACEPOINTS ~/linux-3.2.93/.config CONFIG_TRACEPOINTS=y I've just attached the stap temporary directory with the generated source files. Cheers, Gus
Created attachment 10681 [details] stap verbose output
I took a look at your attachments. Your stapconf_1ddada75a347accc585e3d1a06438f1e_713.h file doesn't contain STAPCONF_UTRACE_VIA_TRACEPOINTS. This means that your kernel doesn't support the 5 tracepoints systemtap needs to do userspace tracing. They are: sched_process_fork, sched_process_exit, sched_process_exec, sys_enter, and sys_exit You can check for these tracepoints via the following: === # stap -l 'kernel.trace("sched_process_*")' kernel.trace("sched:sched_process_exec") kernel.trace("sched:sched_process_exit") kernel.trace("sched:sched_process_fork") kernel.trace("sched:sched_process_free") kernel.trace("sched:sched_process_wait") # stap -l 'kernel.trace("sys_*")' kernel.trace("raw_syscalls:sys_enter") kernel.trace("raw_syscalls:sys_exit") === If the sys_enter/sys_exit tracepoints are missing, that would mean your kernel doesn't have CONFIG_HAVE_SYSCALL_TRACEPOINTS.
(In reply to David Smith from comment #5) > I took a look at your attachments. Your > stapconf_1ddada75a347accc585e3d1a06438f1e_713.h file doesn't contain > STAPCONF_UTRACE_VIA_TRACEPOINTS. This means that your kernel doesn't support > the 5 tracepoints systemtap needs to do userspace tracing. They are: > > sched_process_fork, sched_process_exit, sched_process_exec, sys_enter, and > sys_exit > > You can check for these tracepoints via the following: > > === > # stap -l 'kernel.trace("sched_process_*")' > kernel.trace("sched:sched_process_exec") > kernel.trace("sched:sched_process_exit") > kernel.trace("sched:sched_process_fork") > kernel.trace("sched:sched_process_free") > kernel.trace("sched:sched_process_wait") > # stap -l 'kernel.trace("sys_*")' > kernel.trace("raw_syscalls:sys_enter") > kernel.trace("raw_syscalls:sys_exit") > === > > If the sys_enter/sys_exit tracepoints are missing, that would mean your > kernel doesn't have CONFIG_HAVE_SYSCALL_TRACEPOINTS. Yeah, it seems the syscall tracepoints are disabled. # stap -l 'kernel.trace("sched_process_*")' kernel.trace("sched:sched_process_exit") kernel.trace("sched:sched_process_fork") kernel.trace("sched:sched_process_free") kernel.trace("sched:sched_process_wait") # stap -l 'kernel.trace("sys_*")' And unfortunately it seems this kernel version doesn't support syscall tracepoints for ARM. ~/linux-3.2.93# grep -r HAVE_SYSCALL_TRACEPOINTS arch/* arch/powerpc/Kconfig: select HAVE_SYSCALL_TRACEPOINTS arch/s390/Kconfig: select HAVE_SYSCALL_TRACEPOINTS arch/sh/Kconfig: select HAVE_SYSCALL_TRACEPOINTS arch/sparc/Kconfig: select HAVE_SYSCALL_TRACEPOINTS arch/x86/Kconfig: select HAVE_SYSCALL_TRACEPOINTS Wonderful :S So, let's start again. I will install everything from scratch using a fresh Debian Jessie with kernel 3.16.0-4 and GCC 4.9.2. Should those be enough? As per the kernel config features, what exactly does SystemTap need? That's a bit confusing because different pages say different things. For instance, in the official wiki for using systemtap with self-built kernels [https://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel], it doesn't mention neither CONFIG_TRACEPOINTS nor CONFIG_HAVE_SYSCALL_TRACEPOINTS. I know that would depend on what SystemTap features I will use but, let's say, to have full support. Where can I find the most updated documentation about it? I've gathered this list from different sources, should this be enough? Do you recommend to add or remove something? CONFIG_KPROBES=y CONFIG_KPROBES_SANITY_TEST=n CONFIG_KPROBE_EVENT=y CONFIG_NET_DCCPPROBE=m CONFIG_NET_SCTPPROBE=m CONFIG_NET_TCPPROBE=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_REDUCED=n CONFIG_ARM_KPROBES_TEST=n CONFIG_TRACEPOINTS=y CONFIG_HAVE_SYSCALL_TRACEPOINTS=y Cheers, Gus
(In reply to Gustavo Moreira from comment #6) > So, let's start again. I will install everything from scratch using a fresh > Debian Jessie with kernel 3.16.0-4 and GCC 4.9.2. Should those be enough? Unfortunately, I have no idea. You'll just have to try them. > As per the kernel config features, what exactly does SystemTap need? That's > a bit confusing because different pages say different things. For instance, > in the official wiki for using systemtap with self-built kernels > [https://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel], it > doesn't mention neither CONFIG_TRACEPOINTS nor > CONFIG_HAVE_SYSCALL_TRACEPOINTS. I know that would depend on what SystemTap > features I will use but, let's say, to have full support. Where can I find > the most updated documentation about it? > > I've gathered this list from different sources, should this be enough? Do > you recommend to add or remove something? > > CONFIG_KPROBES=y > CONFIG_KPROBES_SANITY_TEST=n > CONFIG_KPROBE_EVENT=y > CONFIG_NET_DCCPPROBE=m > CONFIG_NET_SCTPPROBE=m > CONFIG_NET_TCPPROBE=y > CONFIG_DEBUG_INFO=y > CONFIG_DEBUG_INFO_REDUCED=n > CONFIG_ARM_KPROBES_TEST=n > CONFIG_TRACEPOINTS=y > CONFIG_HAVE_SYSCALL_TRACEPOINTS=y Those wiki pages were created by someone like you who was motivated to figure out exactly what systemtap needed. I normally don't use self-built kernels, but standard Fedora/RHEL distro kernels. I'm sure at the time that wiki page was written that was a definitive list. But, things have changed (both on the kernel side and the systemtap side). As far as kernel config lines goes, it does depend on what kind of probing you want to do. That list above looks fairly reasonable for kernel probing (CONFIG_KPROBES being the main one). The wiki page lists several you are missing, like: CONFIG_RELAY and CONFIG_DEBUG_FS. Past that, it depends on what kinds of probes you want to use. CONFIG_UPROBES is needed for userspace probing. CONFIG_NETFILTER is needed for netfilter.* probes. CONFIG_TRACEPOINTS is needed for kernel.trace probes. CONFIG_HAVE_HW_BREAKPOINT is needed for kernel.data probes. CONFIG_HAVE_PERF_EVENTS and CONFIG_PERF_EVENTS are needed for perf.* probes. CONFIG_PROC_FS is needed for procfs probes.
It appears to be an issue with the locally built kernel configuration rather systemtap. On the recent version of 32-bit raspberry pi has the sched_process_* and sys_* tracepoints: $ uname -a Linux raspberrypi 6.1.21+ #1642 Mon Apr 3 17:19:14 BST 2023 armv6l GNU/Linux $ sudo /home/wcohen/bin/perf list |grep sched_process* sched:sched_process_exec [Tracepoint event] sched:sched_process_exit [Tracepoint event] sched:sched_process_fork [Tracepoint event] sched:sched_process_free [Tracepoint event] sched:sched_process_hang [Tracepoint event] sched:sched_process_wait [Tracepoint event] $ sudo /home/wcohen/bin/perf list |grep sys_ raw_syscalls:sys_enter [Tracepoint event] raw_syscalls:sys_exit [Tracepoint event] There hasn't been any response on this bug for over six year. Going to close this.