Created attachment 6110 [details] stderr output listing problems with RHEL6 run of stap -L "*, *.*, *.*.*" One would like to get an exhaustive list of probe points with: stap -L "*, *.*, *.*.*" However, on x86-64 RHEL 6.2 with systemtap 1.6 get a number of warnings about arguments not being available. Also get similar warnings on x86-64 Fedora rawhide. On RHEL6: kernel-2.6.32-220.el6.x86_64 systemtap-1.6-4.el6.x86_64 On Fedora rawhide: kernel-3.2.0-0.rc5.git1.1.fc17.x86_64 systemtap-1.6-1.fc16.x86_64
Created attachment 6111 [details] Fedora rawhide stderr output from stap -L "*, *.*, *.*.*"
The first error: ==== semantic error: unable to find member 'tid' for struct task_struct (alternatives: state stack usage flags ptrace lock_depth prio static_prio normal_prio rt_priority sched_class se rt preempt_notifiers fpu_counter btrace_seq policy cpus_allowed sched_info tasks pushable_tasks mm active_mm exit_state exit_code exit_signal pdeath_signal personality did_exec in_execve in_iowait sched_reset_on_fork pid tgid stack_canary real_parent parent children sibling group_leader ptraced ptrace_entry pids thread_group vfork_done set_child_tid clear_child_tid utime stime utimescaled stimescaled gtime prev_utime prev_stime nvcsw nivcsw start_time real_start_time min_flt maj_flt cputime_expires cpu_timers real_cred cred cred_guard_mutex replacement_session_keyring comm link_count total_link_count sysvsem last_switch_count thread fs files nsproxy signal sighand blocked real_blocked saved_sigmask pending sas_ss_sp sas_ss_size notifier notifier_data notifier_mask audit_context loginuid sessionid seccomp utrace utrace_flags parent_exec_id self_exec_id alloc_lock irqaction pi_lock pi_waiters pi_blocked_on journal_info bio_list bio_tail reclaim_state backing_dev_info io_context ptrace_message last_siginfo ioac acct_rss_mem1 acct_vm_mem1 acct_timexpd mems_allowed cpuset_mem_spread_rotor cpuset_slab_spread_rotor mems_allowed_change_disable cgroups cg_list robust_list compat_robust_list pi_state_list pi_state_cache __reserved_perf__ perf_event_mutex perf_event_list mempolicy il_next fs_excl rcu splice_pipe delays dirties timer_slack_ns default_timer_slack_ns scm_work_list curr_ret_stack ret_stack ftrace_timestamp trace_overrun tracing_graph_pause trace trace_recursion rh_reserved perf_event_ctxp): operator '->' at /usr/share/systemtap/tapset/kprocess.stp:120:22 source: released_tid = $p->tid; ==== is a result of code introduced by bug #12413.
The second set of RHEL6 errors comes from the the 'netdev.change_rx_flag' probe alias: ==== # stap -L 'netdev.change_rx_flag' semantic error: not accessible at this address (0xffffffff814483dc, dieoffset: 0x3c69de1): identifier '$dev' at /usr/local/share/systemtap/tapset/networking.stp:159:29 source: dev_name = get_netdev_name($dev) ^ semantic error: not accessible at this address (0xffffffff814483dc, dieoffset: 0x3c69dd8): identifier '$flags' at :160:10 source: flags = $flags ^ semantic error: unresolved type : identifier 'flags' at :160:2 source: flags = $flags ^ netdev.change_rx_flag dev_name:string ==== This happens because the underlying kernel function 'dev_change_rx_flags' is inlined on RHEL6 and systemtap can't find the arguments: ==== # stap -L 'kernel.function("dev_change_rx_flags").*' kernel.function("dev_change_rx_flags@net/core/dev.c:3818").inline ==== On f15, systemtap can find the arguments to both the real function call and the inlined version: ==== stap -L 'kernel.function("dev_change_rx_flags").*' kernel.function("dev_change_rx_flags@net/core/dev.c:4360").call $dev:struct net_device* $flags:int kernel.function("dev_change_rx_flags@net/core/dev.c:4360").return $dev:struct net_device* $flags:int ====
The following set of errors are a bit bogus, but I'm not sure how to quiet them. ==== # stap -L '__scheduler.ctxswitch.kp' semantic error: unable to find local 'new' near pc 0xffffffff8150aff7 in context_switch kernel/sched.c ( (alternatives: $next $prev $rq): identifier '$new' at /usr/local/share/systemtap/tapset/scheduler.stp:174:19 source: next_priority = $new->prio ^ semantic error: unable to find local 'new' near pc 0xffffffff8150aff7 in context_switch kernel/sched.c ( (alternatives: $next $prev $rq): identifier '$new' at :175:14 source: next_pid = $new->tgid ^ semantic error: unable to find local 'new' near pc 0xffffffff8150aff7 in context_switch kernel/sched.c ( (alternatives: $next $prev $rq): identifier '$new' at :176:14 source: next_tid = $new->pid ^ semantic error: unable to find local 'new' near pc 0xffffffff8150aff7 in context_switch kernel/sched.c ( (alternatives: $next $prev $rq): identifier '$new' at :177:15 source: next_task = $new ^ semantic error: unable to find local 'new' near pc 0xffffffff8150aff7 in context_switch kernel/sched.c ( (alternatives: $next $prev $rq): identifier '$new' at :178:34 source: next_task_name = task_execname($new) ^ semantic error: unable to find local 'new' near pc 0xffffffff8150aff7 in context_switch kernel/sched.c ( (alternatives: $next $prev $rq): identifier '$new' at :179:19 source: nexttsk_state = $new->state ^ semantic error: unresolved type : identifier 'next_priority' at :174:3 source: next_priority = $new->prio ^ semantic error: unresolved type : identifier 'next_pid' at :175:3 source: next_pid = $new->tgid ^ semantic error: unresolved type : identifier 'next_tid' at :176:3 source: next_tid = $new->pid ^ semantic error: unresolved type : identifier 'next_task' at :177:3 source: next_task = $new ^ semantic error: unresolved type : identifier 'nexttsk_state' at :179:3 source: nexttsk_state = $new->state ^ semantic error: unresolved type : identifier 'next_priority' at :158:3 source: next_priority = $next->prio ^ semantic error: unresolved type : identifier 'next_pid' at :159:3 source: next_pid = $next->tgid ^ semantic error: unresolved type : identifier 'next_tid' at :160:3 source: next_tid = $next->pid ^ semantic error: unresolved type : identifier 'next_task' at :161:3 source: next_task = $next ^ semantic error: unresolved type : identifier 'nexttsk_state' at :163:3 source: nexttsk_state = $next->state ^ __scheduler.ctxswitch.kp prev_priority:long prev_pid:long prev_tid:long prev_task:long prev_task_name:string prevtsk_state:long next_priority:unknown next_pid:unknown next_tid:unknown next_task:unknown next_task_name:string nexttsk_state:unknown $prev:struct task_struct* $rq:struct rq* ==== The above errors occur because on RHEL6 systemtap can't find the 'next' argument of the inlined kernel function 'context_switch': ==== # stap -L 'kernel.function("context_switch").*' kernel.function("context_switch@kernel/sched.c:2917").inline $prev:struct task_struct* $rq:struct rq* ==== But, on RHEL6 the '__scheduler.ctxswitch.kp' probe alias isn't used - the 'scheduler.ctxswitch' prefers the '__scheduler.ctxswitch.tp' (tracepoint) probe alias over the '__scheduler.ctxswitch.kp' (kprobe) probe alias. ==== # stap -L '__scheduler.ctxswitch.tp' __scheduler.ctxswitch.tp __tracepoint_arg_prev:long __tracepoint_arg_next:long next_pid:long next_tid:long next_task:long next_task_name:string nexttsk_state:long next_priority:long prev_priority:long prev_pid:long prev_tid:long prev_task:long prev_task_name:string prevtsk_state:long $rq:struct rq* $prev:struct task_struct* $next:struct task_struct* # stap -L 'scheduler.ctxswitch' scheduler.ctxswitch __tracepoint_arg_prev:long __tracepoint_arg_next:long next_pid:long next_tid:long next_task:long next_task_name:string nexttsk_state:long next_priority:long prev_priority:long prev_pid:long prev_tid:long prev_task:long prev_task_name:string prevtsk_state:long name:string $rq:struct rq* $prev:struct task_struct* $next:struct task_struct* ====
The next set of errors can be seen by the following: ==== # stap -L 'sunrpc.clnt.create_client' semantic error: unresolved type : identifier 'args' at /usr/local/share/systemtap/tapset/rpc.stp:175:3 source: args = __rpc_create_args[tid()] ==== When we can't find the 'args' parameter of kernel.function("rpc_new_client").inline, we used a cached copy that is in the __rpc_create_args global. This error means we can't figure out what type __rpc_create_args is. I'm unsure of how to fix this one.
(In reply to comment #5) > The next set of errors can be seen by the following: > > ==== > # stap -L 'sunrpc.clnt.create_client' > semantic error: unresolved type : identifier 'args' at > /usr/local/share/systemtap/tapset/rpc.stp:175:3 > source: args = __rpc_create_args[tid()] > ==== > > When we can't find the 'args' parameter of > kernel.function("rpc_new_client").inline, we used a cached copy that is in the > __rpc_create_args global. This error means we can't figure out what type > __rpc_create_args is. > > I'm unsure of how to fix this one. Josh Stone helped develop a workaround for this one by initializing the 'args' local variable even in the branch that doesn't use 'args'. See commit d7699f7.
(In reply to comment #4) > The following set of errors are a bit bogus, but I'm not sure how to quiet > them. > > ==== > # stap -L '__scheduler.ctxswitch.kp' Fixed in commit 5d9e97d by merging the scheduler.ctxswitch .kp and .tp probes.
This one happens because one branch doesn't define '__tsk': # stap -L '__scheduler.process_exit.kp' > /dev/null semantic error: unresolved type : identifier '__tsk' at /usr/local/share/systemtap/tapset/scheduler.stp:388:3 source: __tsk = task_current() ^ Fixed in commit df15ab5.
For RHEL6 (2.6.32-131.12.1.el6.x86_64.debug) I now see the following output: ==== # stap -L '**' > /dev/null semantic error: not accessible at this address (0xffffffff814483dc, dieoffset: 0x3c69de1): identifier '$dev' at /usr/local/share/systemtap/tapset/networking.stp:159:29 source: dev_name = get_netdev_name($dev) ^ semantic error: not accessible at this address (0xffffffff814483dc, dieoffset: 0x3c69dd8): identifier '$flags' at :160:10 source: flags = $flags ^ semantic error: unresolved type : identifier 'flags' at :160:2 source: flags = $flags ==== The above set of errors is describe more fully in comment #3 and is caused by systemtap being unable to find inlined function arguments. This is probably as good as we can get for this kernel.
Here is my analysis of the results of 'stap -L "**" > /dev/null' on F16. All the failures have the same problem - systemtap can't find inlined function arguments. ==== semantic error: unable to find local 'task' near pc 0x336e3 in nfs_read_done fs/nfs/proc.c ( (alternatives: $data): identifier '$task' at /usr/local/share/systemtap/tapset/nfs_proc.stp:798:11 source: status = $task->tk_status ==== nfs.proc2.read_done - inline version missing $task # stap -L 'module("nfs").function("nfs_read_done").*' module("nfs").function("nfs_read_done@fs/nfs/proc.c:632").call $task:struct rpc_task* $data:struct nfs_read_data* module("nfs").function("nfs_read_done@fs/nfs/proc.c:632").inline $data:struct nfs_read_data* module("nfs").function("nfs_read_done@fs/nfs/proc.c:632").return $return:int $task:struct rpc_task* $data:struct nfs_read_data* ==== semantic error: unable to find local 'task' near pc 0x33645 in nfs_write_done fs/nfs/proc.c ( (alternatives: $data): identifier '$task' at :1035:11 source: status = $task->tk_status ^ ==== nfs.proc2.write_done - inline version missing $task: # stap -L 'module("nfs").function("nfs_write_done").*' module("nfs").function("nfs_write_done@fs/nfs/proc.c:654").call $task:struct rpc_task* $data:struct nfs_write_data* module("nfs").function("nfs_write_done@fs/nfs/proc.c:654").inline $data:struct nfs_write_data* module("nfs").function("nfs_write_done@fs/nfs/proc.c:654").return $return:int $task:struct rpc_task* $data:struct nfs_write_data* ==== semantic error: unable to find local 'task' near pc 0x3b0e3 in nfs3_write_done fs/nfs/nfs3proc.c ( (alternatives: $data): identifier '$task' at :1066:11 source: status = $task->tk_status ==== ^ nfs.proc3.write_done - inline version missing $task: # stap -L 'module("nfs").function("nfs3_write_done").*' module("nfs").function("nfs3_write_done@fs/nfs/nfs3proc.c:817").call $task:struct rpc_task* $data:struct nfs_write_data* module("nfs").function("nfs3_write_done@fs/nfs/nfs3proc.c:817").inline $data:struct nfs_write_data* module("nfs").function("nfs3_write_done@fs/nfs/nfs3proc.c:817").return $return:int $task:struct rpc_task* $data:struct nfs_write_data* ==== semantic error: unable to find local 'rqstp' near pc 0x25e49 in nfsd_open fs/nfsd/vfs.c ( (alternatives: $filp $access $fhp): identifier '$rqstp' at /usr/local/share/systemtap/tapset/nfsd.stp:996:33 source: client_ip = addr_from_rqst_str($rqstp) ^ semantic error: unable to find local 'type' near pc 0x25e49 in nfsd_open fs/nfsd/vfs.c ( (alternatives: $filp $access $fhp): identifier '$type' at :1000:9 source: type = $type ==== nfsd.open - inline version missing $rqstp, $type # stap -L 'module("nfsd").function("nfsd_open").*'module("nfsd").function("nfsd_open@fs/nfsd/vfs.c:715").call $rqstp:struct svc_rqst* $fhp:struct svc_fh* $type:int $access:int $filp:struct file** $flags:int $host_err:int module("nfsd").function("nfsd_open@fs/nfsd/vfs.c:715").exported $rqstp:struct svc_rqst* $fhp:struct svc_fh* $type:int $access:int $filp:struct file** $flags:int $host_err:int module("nfsd").function("nfsd_open@fs/nfsd/vfs.c:715").inline $filp:struct file** $access:int $fhp:struct svc_fh* module("nfsd").function("nfsd_open@fs/nfsd/vfs.c:715").return $return:__be32 $rqstp:struct svc_rqst* $fhp:struct svc_fh* $type:int $access:int $filp:struct file** $flags:int $host_err:int ==== semantic error: unable to find local 'signr' near pc 0xffffffff81013168 in handle_signal arch/x86/kernel/signal.c ( (alternatives: $regs $ka $info $sig): identifier '$signr' at /usr/local/share/systemtap/tapset/signal.stp:606:15 source: sig = $signr ^ semantic error: unable to find local 'signr' near pc 0xffffffff81013168 in handle_signal arch/x86/kernel/signal.c ( (alternatives: $regs $ka $info $sig): identifier '$signr' at :607:33 source: sig_name = _signal_name($signr) ^ semantic error: not accessible at this address (0xffffffff81013168, dieoffset: 0x127bfc): identifier '$info' at :609:13 source: sinfo = $info ^ semantic error: not accessible at this address (0xffffffff81013168, dieoffset: 0x127bfc): identifier '$info' at :610:16 source: sig_code = $info->si_code ^ semantic error: not accessible at this address (0xffffffff81013168, dieoffset: 0x127bf3): identifier '$ka' at :611:15 source: ka_addr = $ka ^ semantic error: unable to find local 'set' near pc 0xffffffff81013168 in handle_signal arch/x86/kernel/signal.c ( (alternatives: $regs $ka $info $sig): identifier '$set' at :612:50 source: oldset_addr = (@defined($oldset) ? $oldset : $set) ^ semantic error: not accessible at this address (0xffffffff81013168, dieoffset: 0x127bea): identifier '$regs' at :613:12 source: regs = $regs ==== signal.handle - inlined function missing all args # stap -L 'kernel.function("handle_*signal*").*' kernel.function("handle_signal@arch/x86/kernel/signal.c:682").inline ==== semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at /usr/local/share/systemtap/tapset/rpc.stp:437:29 source: servername = kernel_string($clnt->cl_server) ^ semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at :438:27 source: progname = kernel_string($clnt->cl_protname) ^ semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at :439:24 source: prog = prog_from_clnt($clnt) ^ semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at :440:24 source: vers = vers_from_clnt($clnt) ^ semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at :441:24 source: prot = prot_from_clnt($clnt) ^ semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at :442:24 source: port = port_from_clnt($clnt) ^ semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at :443:23 source: xid = xid_from_clnt($clnt) ^ semantic error: failed to retrieve location attribute for local 'clnt' (dieoffset: 0x17132): identifier '$clnt' at :447:22 source: : (atomic_read(&$clnt->cl_count) == 0))) ^ semantic error: failed to retrieve location attribute for local 'msg' (dieoffset: 0x1712c): identifier '$msg' at :449:23 source: proc = proc_from_msg($msg) ^ semantic error: failed to retrieve location attribute for local 'flags' (dieoffset: 0x17126): identifier '$flags' at :454:10 source: flags = $flags ==== sunrpc.clnt.call_sync - inlined function missing all args # stap -L 'module("sunrpc").function("rpc_call_sync").*' module("sunrpc").function("rpc_call_sync@net/sunrpc/clnt.c:667").call module("sunrpc").function("rpc_call_sync@net/sunrpc/clnt.c:667").exported module("sunrpc").function("rpc_call_sync@net/sunrpc/clnt.c:667").return $return:int
debuginfo quality gradually increases, but no stap bug/limitation evident here