This is the mail archive of the mailing list for the systemtap project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: User-space probes: Plan B+

On 26 Aug 2006 11:40:05 -0400, Frank Ch. Eigler <> wrote:

Hi -

jamesdi wrote:

> [...]  Probing a function that is called often would be a major
> slowdown, as soon as you fire a probe the entire application stops,
> instrumenting something like malloc creates a huge slow down as your
> process, goes to the kernel, then back userland to run the script,

This speculation contemplates the worst possible implementation
scenarios for user-space probing.  This is not what we have in mind.

then how will you deal with a multithreaded app, that calls malloc, 5,000 times a seconds and I'm interested in calls that are for less than 1k of memory, but only when called from inside function y, since its a multithreaded app, malloc probe fires, goes into the kernel, you either have to let the kernel handle it using uprobes? it will simply record the event, generating lots of excess information. or you have to go to the userland probe handler, and the first application is now halted, and the task is now serialized, hurting performance and hiding races. What happens if another thread causes the same probe to be fired while one thread is in the userland probe? are we going to spawn a userland probe thread for every thread that fires a probe? What happens if the first thread recieves a signal causing the same probe to fire? Are you going to disable signals of the program being traced every time you call the userland probe? what does that do to performance?

> and then back even if the probe wasn't even interested in the
> particular event. [...]

Unlike broad strace/ptrace, the intent is to ensconce and surveil only
interesting probes.

I'm not talking about broad events, even one function call being
probed that causes frequent context switches, is enough to overwhelm
the system and cause tremendous slowdowns, just counting probe hits is
not good enough, you might was well resort to embedded printf
statements in the code.

> [...]
> > But I think it's better to provide a feature for which a need has been
> > identified -- even if the feature requires careful use and a few minutes
> > to understand -- than to withhold the feature [...]
> its better to design the system with safety and security in
> mind. [...]

Those things are not mutually contradictory.

I didn't say they weren't but having a system where the user has to think gee will I ever get more than Y probe fires in a second, will i have more threads than I have processors, I better use method X, but if I'm sure its only going to be 100 probe fires a second, i can do this using method Y, that gives me the information I need quicker without being flooded with data. Of course changing things in middle of a major problem will lead to the system crashing, because you went from accessing variables stored in userland to ones stored inside the kernel, then you end up debugging your debugging script instead your application.

Most of this goes away if you do it all inside the kernel, with no
context switch, only one method no matter how many times the probe is
fired. Sure you can have a userland app that prints out stuff, like
stack traces, but the heart of it has to all happen in the kernel or
most of the flexibility of systemtap is lost.

> [...] They ended up with a solution that works for the expert > programmer and overworked system administrator, as well as the > weekend home user [...]

You are confusing a proposed internal low-level implementation, and
its high-level software client.  Few are imagining having end-users
write raw kprobes, utrace clients, or solaris device drivers for that

no just Linux kernel modules, systemtap scripts.


- FChE

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]