This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: User-space probes: Plan B+
On Fri, 2006-08-25 at 08:28, Frank Ch. Eigler wrote:
> Jim Keniston <jkenisto@us.ibm.com> writes:
>
> > [...] I tried an approach based on ptrace, with no kernel
> > enhancements, but it lacked certain necessary features (e.g., #2-5
> > below), probe overhead was 12-15x worse than Prasanna's approach
>
> So 12-15 us per probe hit rather than 1 us per kprobe.
Yes.
>
> > and I couldn't get it to work when probing multiple processes.[...]
>
> Yeah, you'd have to have had several cooperating prober processes.
Yes, I modified the prototype to spawn one tracer thread per tracee
process. That worked no better or worse than the original approach. I
can go into more details on a separate (mail) thread, although I'd like
to concentrate on pursuing Plan B+ for now.
>
>
> > [...] b. The "tracer" process detects events (e.g., probe hits) by
> > polling rather than catching SIGCHLD signals. [...]
>
> Polling -- but surely not in a busy loop!
No CPUs are in spinwaits, if that's what you mean. In the standard poll
loop (discounting kernel handlers), the tracer and tracee take turns
sleeping and waking each other -- one round per uprobe_poll() call. I
expect the performance there to be similar to that of a
signal/PTRACE_CONT loop.
>
> > Here are the requirements we will satisfy with this approach.
> > 0. Per-process (not per-executable) tracing.
>
> Plus we may need a facility to propagate probes to child processes.
Yes. That won't be in Rev 0.1, although the tracer can set up to be
notified when the tracee forks, execs, etc.
> > 1. Instrumentation can be coded entirely as a user-space app...
>
> I am not aware of this being a *requirement*. To my mind, it is an
> unfortunate implementation artifact. Running in kernel space but in
> the traced processes' user context would be better.
We certainly believed last spring that kernel-only instrumentation was
sufficient, feeling sure that SystemTap could be educated to emit the
appropriate kernel modules. Our interpretation of feedback from LKML
and other sources was that uprobes would not be accepted without a
reasonably friendly user-mode API.
A few things, such as stack backtraces with symbol info, cannot be
accomplished entirely in the kernel, so you need user-mode cooperation
at some point anyway.
>
> > [...] 3. A user-mode tracer can invoke a previously registered
> > kernel-mode handler, so we have simple and efficient communication
> > between user- and kernel-mode instrumentation. [...]
>
> How is this registration aspect supposed to work?
Kernel-mode handlers can be registered in a kernel module. Registration
requires the handler address, a unique name, and permission info (e.g.,
only root can call it, or only a process that has permission to run
uprobe_register() on the traced process can call it).
Handlers that we expect to have widespread use (e.g., collect
stack-trace addresses, report register contents, report memory contents)
could even be registered by the kernel.
> Is something
> keeping us from compiling & registering this instrumentation
> dynamically, the same way we do normal kprobes handlers?
No. I envision SystemTap generating both the user-space tracer app and
the module that registers any needed ad hoc kernel handlers (as well as
the usual kernel-probing stuff).
>
> - FChE
Jim