Re: Fw: [ltc-interlock] [RFC] draft NFS trace hooks (fwd)

[Tony Reix <tony dot reix at bull dot net>]

Hello Gerrit,

Le mardi 22 août 2006 à 09:09 -0700, Gerrit Huizenga a écrit : 
> Hi Tony - a few quick comments below in response to some of yours.

Thanks !


> > >  And I attached a list of what we want to trace for NFS. But we are
> > > not working on NFS itself and don't have much knowledge about it.
> > 
> > We are working since some years on NFSv4, mainly testing or adding IPv6.
> > So we have some understanding of the protocol, but sure we do not master
> > the protocol nor the code.
> 
> Think about this as well from the point of view of someone using or
> debugging NFSv4.  That is the ultimate goal of systemtap - to aid the
> system administrator or support person to find the *right* information
> related to any subject area.

"*right* information" : Yes. But documentation is required in order to
be able to interpret/analyze the traces.


> Most of Linux and related utilities are not so lucky as to have
> design documents, mostly because Linux evolves as do most open
> source projects.

Sorry. I spent so many years working within projects with a heavy
process (documents for everything important ...) that I cannot forget
these reflexes. Documents help to quickly have a global
understanding ...


> > > None. This is the powerful aspect of SystemTap: you can probe the
> > > Kernel without touching the Kernel source codes.
> > >
> > 
> > Ohh. I did not know.
> > Probably this is explained in the OLS'06 paper.
> > Do you have a link to a technical explanation of how that works ?
>  
> Look up on, for instance, lwn the topic "kprobes".  There should be
> lots of kprobes/dprobes information floating around which describes
> the somewhat debugger like functionality of adding trace points.

I've started to read the OLS'06 paper "Problem Solving With Systemtap"
by Frank Eigler. Chapter 2 is very interesting since it explains that
"Static probing markers" have been recently added.
I'm not sure I perfectly understood the explanations, but it seems to me
that "dynamic" probes do require gcc to provide debugging information
that prevent it to run full optimizations. On the contrary, "Static
probing markers" are "faster and more precise than kprobes", at the cost
of the "waste of a few cycles".
I have to read the end of the paper, but mixing dynamic and static
probes could be useful.


> > Do you know what is the position of OSDL about SystemTap ?
> 
> OSDL loves SystemTap.

Good.
Bryce (Harrington) "attended the systemtap talk at OLS and was very
impressed by it." But it does not seem he has understood about "Static
markers".


> > I'm used to have students giving help on some projects (this year: NPTL
> > Trace Tool, and NFSv4 Administration/Security).
> > Do you think a student (best French Computer Science University, 5-6
> > months of work) could help ?
>  
> Possibly - although another project might be to help write tapsets for
> various areas of the kernel.  It is a good way for students to become
> more familiar with specific areas of the kernel that might be interesting.

About your comments about documentation: A dynamic tool like Systemtap
can help people to understand how the kernel works without looking at
its code or modifying it. Just looking at traces (if a global map and a
guide are provided !).


> > Yes. I've talked with 2 guys here working on ext3 and they have no clear
> > idea of how SystemTap could help them ... One of them (a ext3 and Xen
> > expert) did not know about SystemTap even.
> 
> I'll help educate your local team on SystemTap soon, it's benefits and
> our general goals.

Yes. Sure.
My opinion is that a tool becomes a product once enough documentation
and tracing tools are available, because code must live after its
creator has moved to other business, and because the maintenance of code
in the field is critical for the confidence of customers (who hate
upgrading the operating system).
So SystemTap seems very important. But many Linux developers are facing
so complex development problems or are in "zeinot" that they cannot
manage this important part of code development work or even take time to
look at this new tool.

So, yes ! Come and provide education to us about SystemTap !


> > What about a student working on tapsets for NFSv4 in 2007 ? After we
> > have got the position of NFSv4 developers about SystemTap, sure. I'll
> > ask them about SystemTap.
> 
> Yep - that would be very cool!

I've started discussing with NFSv4 developers and with Bryce.
They seem to be interested by Systemtap, except that:
 - they are not aware about the "Static probing markers"
 - the dynamic probes of Peng Li and Guanglei dealing with performance may be
	a duplicate of recent work of Chuck Lever
 - it is not sure if these dynamic probes are the ones that are requested.
	And NFSv4 developers have so few time left ...
 - there already exist many dprintk()s in NFSv4 code.
	What to do with them ? Translate them to SystemTap static markers ?

About having a student that could work between SystemTap experts and NFSv4
experts, I've just asked them. Wait and See.

Regards,

Tony