Re: kernel summit session on systemtap

[Theodore Tso <tytso at mit dot edu>]

On Thu, Sep 18, 2008 at 09:00:15AM +0100, R. J. Moore wrote:
> I don't know whether it is possible with the latest code, but for  
> debugging purposes, I would be happy if SystemTap could operate on  
> external names and relative addresses - i.e. without the need to have  
> any symbolic information. 

You can certainly set tracepoints that way.  I don't know how easy it
would be to fetch out register information or interpret complex data
structures living on the stack or in function parameters without debug
information, though.  Usually if I have to do that level of analysis,
at least at this point it's faster for me to rebuild without debuginfo
information, drop the use of system tap, and retreat to printk
debugging.

> This is the way I used ancestors to systemtap  
> for shooting very difficult kernel problems in the field. Generally I  
> started with a crashdump. 

I spent about a couple of hours trying to get kdump to work on my
laptop, and then gave up.  That's another RAS tool which has mostly
ignored by the kernel development community, and deployability and
usability has been one of its problems...

> I admit that this is extreme debugging,  
> but if system tap won't even operate without a ton of extra junk present  
> then I see its application as being very limited indeed.   Not everyone  
> will want to work at the assembler level, but if System Tap can, then  
> tools can be built that do code analysis to help generate and divine  
> probepoints. Much can be done knowing that nearly 100% of the code we  
> probe is generated by the one tool - gcc. In theory, what is generated  
> is deterministic, or the reverse engineering of it is.

My guess is that right now, the level 3 debugging experts at places
like Red Hat, IBM, Novell, etc., are the only people who find
SystemTap useful.  The quick survey done at the kernel summit pretty
conclusively showed that most kernel developers haven't tried to use
it, and of those who tried to use it, a much smaller percentage
succeeded (although keep in mind it Systemtap didn't even compile out
of the box for Debian/Ubuntu distributions until July, and if you are
kernel developer, you can't depend on an ancient distribution-provided
Systemtap, so that may be responsible for the small numbers of people
trying to use it).  So it's not ready for kernel developers, and I'm
pretty sure it's not ready for system administrators yet, due to the
lack of tapsets, lack of tapset documentation (especially compared to
what Dtrace has), and so on.  (By the way, is testing to make sure all
of the tapsets in the tree to make sure they work still turned off
because so many of them are still broken?  That was few months ago
when someone on the list told me that, but that was another face-palm
moment for me.)   

But based on all of that, I suspect that extreme debugging by Level 3
support folks for enterprise linux distributions are the only folks
who are best suited as SystemTap users at this point in time.

    	     	       		       	  - Ted