Bug 1836

Summary: function("*") probe causes crash
Product: systemtap Reporter: James Dickens <jamesd.wi>
Component: kprobesAssignee: Jim Keniston <jkenisto>
Status: RESOLVED DUPLICATE    
Severity: normal CC: ldimaggi
Priority: P2    
Version: unspecified   
Target Milestone: ---   
Host: Target:
Build: Last reconfirmed:

Description James Dickens 2005-11-10 03:30:33 UTC
global calls

probe kernel.function("*")
        {
                calls++;
        }


this should genereate thousands of probes... yet this is the output of stap -v 

Pass 2: analyzed user script.  1 probe(s), 0 function(s), 1 global(s).
Running grep " [tT] " /proc/kallsyms | sort -k 1,8 -s -o
/tmp/stapFOptqu/symbols.sorted
Pass 3: translated to C into "/tmp/stapFOptqu/stap_2526.c"
Running make -C "/lib/modules/2.6.13-1.1532_FC4/build" M="/tmp/stapFOptqu" modules
make: Entering directory `/usr/src/kernels/2.6.13-1.1532_FC4-i686'
  CC [M]  /tmp/stapFOptqu/stap_2526.o
  Building modules, stage 2.
  MODPOST
  CC      /tmp/stapFOptqu/stap_2526.mod.o
  LD [M]  /tmp/stapFOptqu/stap_2526.ko
make: Leaving directory `/usr/src/kernels/2.6.13-1.1532_FC4-i686'
Pass 4: compiled into "stap_2526.ko"
Running sudo /usr/local/libexec/systemtap/stpd -r /tmp/stapFOptqu/stap_2526.ko

of course it also oopses the box. 

 uname -av
Linux localhost.localdomain 2.6.13-1.1532_FC4 #1 Thu Oct 20 01:30:08 EDT 2005
i686 athlon i386 GNU/Linux

[jamesd@localhost ~]$ stap -V
SystemTap translator/driver (version 0.4.2 built 2005-10-31)
Copyright (C) 2005 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
[jamesd@localhost ~]$

please resply for any other details you need.
Comment 1 Frank Ch. Eigler 2005-11-10 03:35:18 UTC
The wildcard probe pattern is actually expanded to a single probe with thousands
of locations.  The verbose message line does not include that last number.  A
couple of months ago, this expansion used to work differently, so that's
probably what misled you.

The oops is a separate and unfortunate problem.  A screenshot or somesuch would
help.
Comment 2 James Dickens 2005-11-10 03:43:49 UTC
Subject: Re:  translator doesn't generate correct code for this.

i was mistaken it doesn't oopse the system but does trigger a reboot
of the box.

On 10 Nov 2005 03:35:18 -0000, fche at redhat dot com
<sourceware-bugzilla@sourceware.org> wrote:
>
> ------- Additional Comments From fche at redhat dot com  2005-11-10 03:35 -------
> The wildcard probe pattern is actually expanded to a single probe with thousands
> of locations.  The verbose message line does not include that last number.  A
> couple of months ago, this expansion used to work differently, so that's
> probably what misled you.
>
> The oops is a separate and unfortunate problem.  A screenshot or somesuch would
> help.
>
>
> --
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>             Summary|translator doesn't generate |translator doesn't generate
>                    |correct code for this.      |correct code for this.
>
>
> http://sourceware.org/bugzilla/show_bug.cgi?id=1836
>
> ------- You are receiving this mail because: -------
> You reported the bug, or are watching the reporter.
>
Comment 3 Frank Ch. Eigler 2005-11-10 04:14:45 UTC
I guess supporting function name wildcards made these cartoon probes too easy. 
We need to analyze further to see whether translator probe point blacklisting
(bug #1235) is sufficient, or if there is another problem.

Reassigning to kprobes to request that they give the can a first kick.  The list
of functions matched may be inspected from the 'stap -p3' source code or the
'stap -p2' list.
Comment 4 Prasanna S Panchamukhi 2005-11-10 13:33:00 UTC
I see a hang/reboot just after inserting probes on do_fork on 2.6.13 kernel.
[root@wks126728wss ~]# stap -V
SystemTap translator/driver (version 0.4.2 built 2005-11-09)
Copyright (C) 2005 Red Hat, Inc. and others
This is free software; see the source for copying conditions.

[root@wks126728wss ~]# cat /home/prasanna/examp/test3.stp

probe kernel.function("do_fork") { log("e") }

[root@wks126728wss examp]# stap -v -g test3.stp
Created temporary directory "/tmp/stapUAG0nh"
Searched '/usr/local/share/systemtap/tapset/2.6.13-1.1532_FC4/*.stp', match coun t 0
Searched '/usr/local/share/systemtap/tapset/2.6.13/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/*.stp', match count 9
Pass 1: parsed user script and 9 library script(s).
parsed 'do_fork' -> func 'do_fork'
pattern 'kernel' matches module 'kernel'
focused on module 'kernel' = [c0100000-c05358d8, bias 0]
pattern 'do_fork' matches function 'do_fork'
selected function do_fork
querying prologue-end of function 'do_fork'
pattern 'kernel' matches module 'kernel'
Pass 2: analyzed user script.  1 probe(s), 6 function(s), 0 global(s).
Running grep " [tT] " /proc/kallsyms | sort -k 1,8 -s -o /tmp/stapUAG0nh/symbols
.sorted
Pass 3: translated to C into "/tmp/stapUAG0nh/stap_2286.c"
Running make -C "/lib/modules/2.6.13-1.1532_FC4/build" M="/tmp/stapUAG0nh" modul es
make: Entering directory `/usr/src/kernels/2.6.13-1.1532_FC4-i686'
  CC [M]  /tmp/stapUAG0nh/stap_2286.o
  Building modules, stage 2.
  MODPOST
  CC      /tmp/stapUAG0nh/stap_2286.mod.o
  LD [M]  /tmp/stapUAG0nh/stap_2286.ko
make: Leaving directory `/usr/src/kernels/2.6.13-1.1532_FC4-i686'
Pass 4: compiled into "stap_2286.ko"
Running sudo /usr/local/libexec/systemtap/stpd -r /tmp/stapUAG0nh/stap_2286.ko
e
e
the system hangs/reboots just after couple of messages. I see a double fault
message sometimes.

Please see bug #1776 comment #2 for details.
Comment 5 Prasanna S Panchamukhi 2005-11-11 15:00:00 UTC
When a kernel module is loaded to insert probes on do_fork(), it does not cause
any hang/reboot of the system and it works fine.
But when a probes is inserted on do_fork() through systemtap, it causes hang/reboot.
Comment 6 Frank Ch. Eigler 2006-06-01 20:46:30 UTC
*** Bug 2725 has been marked as a duplicate of this bug. ***
Comment 7 Jim Keniston 2006-11-16 00:53:27 UTC
It appears that the specific failure in question is not associated with kprobes.
 However, I'm leaving this one open because:
1) At least one other bug is marked as a dup of this.
2) It's worthwhile to probe kernel.function("*") from time to time and chase
down any new failures that crop up -- as they inevitably will, since the
translator blacklist must lag kernel releases.
3) I'm not sure what failures we'd currently see on each architecture given
current kernel.org bits.

However, when such a failure is seen, it's best to isolate and report the
specific failure -- and not just mark it as a dup of this bug.
Comment 8 Frank Ch. Eigler 2006-11-20 21:47:16 UTC

*** This bug has been marked as a duplicate of 2725 ***