[Bug runtime/7082] uretprobes probes causing kernel problems

["jkenisto at us dot ibm dot com" <sourceware-bugzilla at sourceware dot org>]

------- Additional Comments From jkenisto at us dot ibm dot com  2008-12-11 18:48 -------
I pointed a big, ugly stap script at the problem and came up with the following
(heavily annotated) log.  Bottom line is that uprobes gets notified twice of the
same exec, and thinks it's 2 execs.  Badness ensues.  Possible fixes below.

bug1.stp just does
probe process(".../bash").function("*").return {}
----- log ----
# bz7092/bug1.stp was already running when this script started.
# So was one instance of ./bash (built with -g) -- pid = 4726.
4726: stap-utrace begin
# I type "./bash" in shell 4726.
# copy_process() called...
bash(4726) creates 5112
# uprobes is informed of fork
uprobe_report_clone: 4726 -> 5112
# task_finder is informed of fork, and arranges a quiesce, right?
# 5112 quiesces, and task_finder notifies stap-generated uprobes client,
# which registers the 2170 uprobes.
# On the first register_uprobe for a process, uprobes creates a
# uprobe_process, including a utrace engine for each thread.
5112: uprobe_mk_process(5112)
5112: handle_pending_uprobes(5112) #1
5112: handle_pending_uprobes(5112) #2
5112: handle_pending_uprobes(5112) #4
5112: handle_pending_uprobes(5112) #8
5112: handle_pending_uprobes(5112) #16
5112: handle_pending_uprobes(5112) #32
5112: handle_pending_uprobes(5112) #64
5112: handle_pending_uprobes(5112) #128
5112: handle_pending_uprobes(5112) #256
5112: handle_pending_uprobes(5112) #512
5112: handle_pending_uprobes(5112) #1024
5112: handle_pending_uprobes(5112) #2048
# The probe process(".../bash").begin probe fires...
5112: stap-utrace begin
# do_execve() called...
bash(5112) execs ./bash
# utrace_report_exec starts notifying engines...
5112: utrace_report_exec
2 uprocs: 4726 5112
# The probe process(".../bash").end probe fires...
5112: stap-utrace end
# uprobe_report_exec disposes of the uprobe_process for 5112's old image,
# including 5112's engine.
5112: uprobe_report_exec
2 uprocs: 4726 5112
5112: uprobe_free_process(5112)
5112: uprobe_report_exec returns 0x6	# 6 = UTRACE_DETACH
1 uprocs: 4726
# task_finder is informed of exec, and notifies the stap-generated
# uprobes client, which registers the 2170 uprobes in the new image.
# (All this is in task_finder's report_exec callback.)  Note that
# uprobes creates a new engine for 5112, which utrace presumably
# appends to its list of 5112 engines.
5112: uprobe_mk_process(5112)
5112: handle_pending_uprobes(5112) #1
5112: handle_pending_uprobes(5112) #2
5112: handle_pending_uprobes(5112) #4
5112: handle_pending_uprobes(5112) #8
5112: handle_pending_uprobes(5112) #16
5112: handle_pending_uprobes(5112) #32
5112: handle_pending_uprobes(5112) #64
5112: handle_pending_uprobes(5112) #128
5112: handle_pending_uprobes(5112) #256
5112: handle_pending_uprobes(5112) #512
5112: handle_pending_uprobes(5112) #1024
5112: handle_pending_uprobes(5112) #2048
# The probe process(".../bash").begin probe fires...
5112: stap-utrace begin
# Here's the BUG.  utrace_report_exec is still running.  Uprobes's new
# engine for 5112, created as part of exec processing, gets notified
# of the exec.
5112: uprobe_report_exec
2 uprocs: 4726 5112
5112: uprobe_free_process(5112)
5112: uprobe_report_exec returns 0x6
1 uprocs: 4726
# We remove the uprobe_process for 5112 (bad), don't bother removing the
# breakpoints cuz we "know" 5112's "old" image is gone (wrong).  So...
5112: utrace_report_exec returns
# 5112 dies with Trace/breakpoint trap when a breakpoint is hit.
bash(5112) exits
5112: stap-utrace end
----- end of log ---

Here are some ways we could fix this:
1. Modify utrace not to notify engines that are newly created during the exec.
2. Have task_finder (or stap-generated code) request a quiesce from report_exec,
and register the uprobes while quiesced.
3. Fix uprobes not to discard the uprobe_process (and engine) on exec.  This has
the problem that if you're probing a shell, every process forked from that shell
will have a uprobe_process hanging around until it exits.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|systemtap at sources dot    |jkenisto at us dot ibm dot
                   |redhat dot com              |com
             Status|NEW                         |ASSIGNED


http://sourceware.org/bugzilla/show_bug.cgi?id=7082

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.