Re: proposed instruction trace support in SystemTap

[Dave Nomura <dcnltc at us dot ibm dot com>]

From a conversation with Paul Mackerras I recently learned that  
__debugger_sstep() is the PPC trap handler pointer used only for kernel 
debugging, and its use is based on only one kernel debugger active at a 
time.  Although I think a kernel API could be created to coordinate use 
of this function pointer, I don't think one exists.  It would probably 
be prudent to put restrictions on instruction tracing of kernel code in 
the SystemTap-itrace feature.

Although ptrace() probably is not suitable for instruction tracing of 
user code, if for no other reason that performance issues, I have been 
looking into some of the documentation of utrace, and am wondering if it 
might be suitable.

I
Dave Nomura wrote:
> I've been looking into the kernel API for handling single stepping and 
> haven't really found anything.  ptrace() is used by gdb but it's usage 
> model might be overly restrictive for what we want:  we would have to 
> have a parent process then uses ptrace() to trace it's children.  
> ptrace() also does not trace into the kernel which is an ITRACE 
> requirement.
>
> I think the requirement of tracing into the kernel is only needed in 
> some scenarios and may only be needed for the ITRACE application of 
> SystemTap instruction tracing.  In its most general form Perfomance 
> Inspector ITRACE allows tracing into the kernel and tracing of a whole 
> range of processes.  Due to these requirements it places usage 
> constraints on the user that requires exclusive access ot a machine.  
> In the more common SystemTap instruction tracing scenario only a 
> single process is being traced, and only user code is traced maybe it 
> would be approriate to use ptrace() to do the single stepping.
>
> It has already been suggested that we have different APIs for ITRACE 
> vs. simpler (non-kernel tracing) instruction tracing modes so it might 
> be a simple matter of telling  the SystemTap translator what kind of 
> trap handler to generate (or referencd from the runtime stap 
> scripts).  In the non-kernel-tracing-single-process scenario just the 
> normal process switch management of registers will handle restoring 
> the single step trap bit, or if ptrace() (or possibly utrace()).
>
> The ITRACE-kernel-tracing scenario might require similar usage 
> restrictions as PI ITRACE, and we simply would require that you aren't 
> using other kernel debuggers (xmon, kgdb,...) while trying to do an 
> ITRACE to avoid conflict over the kernel resources needed for 
> instruction tracing.   Alternatively, a kernel API (if it doesn't 
> already exist) for handling these kernel resources could be created.  
> I have heard that xmon and kgdb both use the __debugger_sstep() trap 
> handler pointer.
> Frank Ch. Eigler wrote:
> > > [...] There is basically one single instruction trap handler that
> > > the stap translator will generate with logic to figure out what
> > > handler code to run [...]
> > >     
> >
> > The "existing kernel API" is the key issue here.  How exactly does one
> > activate single-stepping traps on each of the interesting
> > architectures, and on multiple different kernel generations (RHEL)?
> > How does one hook into the handling system correctly (avoiding
> > interference to other consumers of trap data like gdb, uprobes)?
> >
> > These questions need answers in order for systemtap to generate code
> > to implement this.
> >
> >
> >   - FChE
> >
> >