This is the mail archive of the mailing list for the systemtap project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: User-space probes: Plan B+

Frank Ch. Eigler wrote:

jkenisto wrote:


1. Instrumentation can be coded entirely as a user-space app...

I am not aware of this being a *requirement*. To my mind, it is an
unfortunate implementation artifact. Running in kernel space but in
the traced processes' user context would be better.

[...] Our interpretation of feedback from LKML and other sources was
that uprobes would not be accepted without a reasonably friendly
user-mode API.

That may be, but we would not be obligated to use such a thing. To
what extent can this work be layered on top of the kernel-side utrace

I agree we are not obliged to use such a thing in SystemTap and i am not expecting we will use either but as the provider of an API in the kernel we have to consider community feedback as we have to get their buy in to get this accepted into the main line kernel. Jim, is trying to provide a flexible interface so that it meets the community need but it gives the freedom for us to just use the kernel mode probe handler which we can dynamically load like we do it today.

A few things, such as stack backtraces with symbol info, cannot be
accomplished entirely in the kernel, so you need user-mode
cooperation at some point anyway.

That's sort of a separate issue. We're not contemplating running live elfutils-style dwarf searches during a probe handler to evaluate backtraces. It is more likely that information required for a precise backtrace would be encoded into the probing module in the form of an extract of the dwarf unwind tables, and searched in situ.

I am not sure i followed exactly what you mean by searched in situ. What i am thinking we will do for back trace is, probe handlers running in the kernel will use platform specific kernel backtrace feature and dump frame pointers out to userland in the order and then stap in the user space consults the dwarf information and prints out the full stack along with symbol information. BTB the stack backtrace i am imagining could be full stack all the way from kernel to app, or just kernel stack. I thought this method would work for Java as well as we need to do Java stack traces in the userland anyway (as far as i know). Does this seem reasonable or i am making it complicated than needed.

[...] 3. A user-mode tracer can invoke a previously registered
kernel-mode handler, so we have simple and efficient communication
between user- and kernel-mode instrumentation. [...]

How is this registration aspect supposed to work?

Kernel-mode handlers can be registered in a kernel module. Registration
requires the handler address, a unique name, and permission info (e.g.,
only root can call it, or only a process that has permission to run
uprobe_register() on the traced process can call it).

Handlers that we expect to have widespread use (e.g., collect
stack-trace addresses, report register contents, report memory
contents) could even be registered by the kernel.

FWIW, such designated "widespread-use" handlers are not in good match
with systemtap's preference for programmable handlers. It may be
helpful for specialized purposes such as LKET (say, by some sort of
clever handler body matching or new syntax), but not for general
systemtap usage.

I agree with you it is not easy (at least i can't think of one easily) have a predefined parametrized handler with complicated filters and aggregation which most of the systemtap ones are. That said, there are simple yet common cases like back trace , register dump etc that many people may like to do as pre-canned handler that you can activate, this was one of the feedback items we got during the review.

Is something
keeping us from compiling & registering this instrumentation
dynamically, the same way we do normal kprobes handlers?

No. I envision SystemTap generating both the user-space tracer app
and the module that registers any needed ad hoc kernel handlers (as
well as the usual kernel-probing stuff).

Would this approach (compiling user-space probe handlers into kernel-space registered but custom-built utrace handlers) make the user-space tracer almost a content-free stub?

The way i am thinking of how systemtap uses, you are very correct, user space tracer is is a content free stub, i am thinking we can add that to our existing stap, the real meat will be dynamically generated like it is today. There may be others (which i doubt very much) who may want to do it differently where they may write the handlers that run completely in the user space and of course pay huge performance overhead but they feel secure and safe. There may be others who only want to do few little things like stack backtrace (this i see a very much a possibility based on a customer requirement i got) using predefined handlers. As you see Jim is trying to address many reqt. hence forth API may be more complicated than needed.

My guess is ( I am not betting the farm yet) most practical purposes handlers copy the required information from the user space using safecopy routines and run the handlers in the kernel.

Jim feel free to correct me if i am wrong.

- FChE

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]