[Bug uprobes/9826] New: gdb/uprobes deathmatch goes to OT

["jkenisto at us dot ibm dot com" <sourceware-bugzilla at sourceware dot org>]

In bugzilla #7082, Comment #27, Marl Wielaard said:

> I ran the stap script without -c:
> $ stap -ve 'probe process("bash").function("*").return {}' -w
> 
> Then in another window:
> (gdb) run
> Starting program: /bin/bash 
> 
> Program received signal SIGTRAP, Trace/breakpoint trap.
> 0x0000000000424b70 in open () at /usr/include/bits/fcntl2.h:54
> 54	      return __open_alias (__path, __oflag, __va_arg_pack ());
> (gdb) bt
> #0  0x0000000000424b70 in open () at /usr/include/bits/fcntl2.h:54
> #1  check_dev_tty () at general.c:381
> Cannot access memory at address 0xffffe518
> 
> > In any case, do you see the same sort of failure if you turn the return probe
> > into an entry probe (a al #7092)?
> 
> That also fails, although slightly differently:
> $ stap -ve 'probe process("bash").function("main") {}' -w
> 
> And in GDB:
> (gdb) run
> Starting program: /bin/bash 
> 
> Program terminated with signal SIGILL, Illegal instruction.
> The program no longer exists.
> You can't do that without a process to debug.
> 
> In both cases 2.6.27.12-170.2.5.fc10.x86_64 under xen.

The basic problem here is that gdb and stap/uprobes are both trying
to set breakpoints at the same spot -- e.g., at the entry to main().

If gdb sets the breakpoint first, uprobes will notice it and refuse
to set a probepoint there (and both uprobes and the stap-generated
module will report this via printk()).

But if uprobes sets the breakpoint first (as in this case), gdb 
doesn't notice, and Bad Things ensue when the breakpoint is hit.
Uprobes tries to single-step the original instruction out of line,
and at the same time, gdb tries to single-step the int3 via opcode
replacement.  Typically, a SIGSEGV ensues... but that's not the worst
of it.

The uprobes code that enables it to persevere when an
unexpected-but-harmless signal such as SIGPROF or SIGALRM arrives in
the middle of SSOL -- see uprobe_delay_signal() -- also causes uprobes
to set aside the SIGSEGVs that gdb (or maybe utrace) keeps sending its
way -- tens of thousands per second.  ("When I'm done single-stepping,
I'll pass that SIGSEGV along...  and that one... and that one...")
Unless/until you kill -9 the probed process, gdb and stap hang,
and uprobes eats up memory with delayed_signal objects.

Repeat-by: Use stap to probe main() in any program -- even hello.c --
and then try to run an instance of that program in gdb.

Workaround: Don't probe the same instructions with both gdb and stap.
If you do, you can get out of it by killing the probed process with
kill -9.  (You can't kill gdb or the stap session while the probed
process is hung like this.)

Solution: (1) Teach gdb to check for an existing breakpoint instruction
before trying to insert one.  (2) Teach uprobes to give up eventually,
when appropriate.  I'm pursuing (2).

-- 
           Summary: gdb/uprobes deathmatch goes to OT
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: uprobes
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: jkenisto at us dot ibm dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=9826

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.