This is the mail archive of the
mailing list for the systemtap project.
Re: Fw: [ltc-interlock] [RFC] draft NFS trace hooks (fwd)
- From: Li Guanglei <guanglei at cn dot ibm dot com>
- To: Tony Reix <tony dot reix at bull dot net>
- Cc: Vara Prasad <varap at us dot ibm dot com>, Jose Santos <jrs at us dot ibm dot com>, "systemtap at sourceware dot org" <systemtap at sourceware dot org>, xuepengl at cn dot ibm dot com
- Date: Tue, 22 Aug 2006 23:10:02 +0800
- Subject: Re: Fw: [ltc-interlock] [RFC] draft NFS trace hooks (fwd)
- Organization: IBM CSTL
- References: <OFFF6EEA28.97ADF0D1-ON482571CA.000A44E4-482571CA.0009DB34@cn.ibm.com> <44DFE9FE.firstname.lastname@example.org> <email@example.com>
Tony Reix wrote:
Le lundi 14 août 2006 à 11:11 +0800, Li Guanglei a écrit :
SystemTap provides an infrastructure to do a dynamic probing, which
means no patches to Kernel, no recompile and no reboot is required.
I think such features will be very useful in the field, when customers
encounter critical problems and that stopping the machine or updating
the kernel to a new version is not an option.
Yes. We are now doing a tracing tool named LKET based on SystemTap and
one of its usage scenario is just let the customer run the tracing
tool and give analyst the raw trace data for post-processing to find
out the problem.
Currently we don't have a separate design document but you could
refer to the following links to get an overall picture of what we are
These links do not provide me high level information;
However, this one:
contains very useful document to refresh/increase my knowledge of
SystemTap (I studied the features of DTrace some years ago ...).
And I attached a list of what we want to trace for NFS. But we are
not working on NFS itself and don't have much knowledge about it.
We are working since some years on NFSv4, mainly testing or adding IPv6.
So we have some understanding of the protocol, but sure we do not master
the protocol nor the code.
This is why we sending out the NFS trace hooks for a review to make sure
that we didn't miss some important trace points.
Up to know, I've seen only 2 answers from the nfs mailing list, from
Chuck Lever. But I do not see answers to the 2nd email of Chuck. Maybe
I've missed it.
Is it the one that Jose Santos replied on 07/28 with the subject "Re:
[NFS] [ltc-perf] draft of nfs event hook"? I didn't subscribe nfs
mailinglist and I may miss that mail if I was not in to/cc list.
I put some answers to your questions below.
Please tell me if we missed some important functions that should be
probed for NFSv4. But what Xuepeng sent out is only our first step of
NFS trace hooks. The full list of our plan is in the attachment.
As I said before, we do not master NFSv4 internals, which are VERY
complicated. I think only Trond Myklebust, Bruce Fields and Chuck Lever
could really help you to check that these hooks are appropriate. But I
know that Trond, Bruce and Chuck are very busy ... So they probably have
no time to spend to understand SystemTap goals and basics and to check
that the hooks you designed are appropriate.
Chuck asked for a design document. It seems you do not have one. And
building such a document would require a deep knowledge of NFS/NFSv4
internals. So this design document should be written either 1) by means
of a collaborative work between SystemTap and NFSv4 experts, or 2) by
someone who could spend time to build skills on both technologies, and
then discuss with the SystemTap and NFSv4 experts.
Yes. I don't have a dedicated design document for NFS trace hooks.
LKET that we are doing is mainly for system trace so I choose only
some NFS functions for instrument. Although it could also be used for
detailed diagnosis but that's not its focus.
See the attachment.
> I know very few about SystemTap. However, I have questions:
> - Do you have a design document describing what must be traced in NFS ?
Hum. This looks more as a conclusion than as a design explaining the
rationale of the choices.
Yes. I only listed those functions that I think important, especially
for performance analysis. One consideration of choosing these
functions is that they could be correlated with other trace hooks,
like VFS, IO Syscall available in LKET.
> - How do you plan to submit your code to NFS or Kernel maintainers ?
No. The trace hooks written in tapsets is for dynamic tracing, which
means that no patches to Kernel is needed.
> - How much of your code must be included in NFS code, as patches ?
None. This is the powerful aspect of SystemTap: you can probe the
Kernel without touching the Kernel source codes.
Ohh. I did not know.
Probably this is explained in the OLS'06 paper.
Do you have a link to a technical explanation of how that works ?
kprobe is its fundamental. you can refer to Documents/kprobes.txt
> - Have you worked with them (I saw nothing on mailing list) ?
No. we now work with SystemTap community. NFS is only a part of what
we are doing.
I do not remember someone talking about systemtap in the nfs and nfsv4
mailing list. A quick search in the 3200 messages I've kept in my
folders seems to show that no one talked about systemtap before you did.
Do you know what is the position of OSDL about SystemTap ?
oh, I don't know. Vara, do you know it?
> What are your plans about tapsets for NFSv4 ?
only NFSv4 procedure stub functions for the client and server side.
> Do you have resources to do that in 2006 or in 2007 ?
oh. I am not sure. We are assigned to work for SystemTap and LKET :-)
I'm used to have students giving help on some projects (this year: NPTL
Trace Tool, and NFSv4 Administration/Security).
Do you think a student (best French Computer Science University, 5-6
months of work) could help ?
Of course. Although we did some work about NFS trace hooks but there
are still a lot of placed needed to be instrumented for NFS.
> First step could be to check that NFSv4 developers are already SystemTap
> enthusiastic. If not yet, one should discuss of that with them. It seems
> very important to get their comments and approval for which features to
> trace. The problem is that NFSv4 Server developer is overloaded ... so
> it may take some time.
> Also, there already are some trace code in NFSv4.
It seems to me that a lot of developers still not realized the
existence or how SystemTap could facilitate their
development/debugging of Kernel. We need more advertisement for
Yes. I've talked with 2 guys here working on ext3 and they have no clear
idea of how SystemTap could help them ... One of them (a ext3 and Xen
expert) did not know about SystemTap even.
I know that proposing debugging/tracing tools to expert/guru developers
is difficult: they are often more efficient without using such tools but
they often forget that such tools will greatly help maintenance in the
future and to understand problems in the field.
What do Linux gurus (Linus, ...) think about SystemTap ?
Now, it seems that SystemTap is only an initiative by RedHat, IBM, Intel
and Hitachi. People close to the customers.
In fact I don't know. Vara, could you give us some news about this?
SystemTap wiki is a good place to share the experience:
Yes. I've found useful documents.
Now, I need to find time to read them ...
Feel free to let me know your suggestions and questions. Thanks.
What about a student working on tapsets for NFSv4 in 2007 ? After we
have got the position of NFSv4 developers about SystemTap, sure. I'll
ask them about SystemTap.
of course that will be great. What we have done is only a start and a
lot of additional work is needed for NFS tapsets. We are not dedicated
to work for NFS tapsets so it will help if we can find someone
continue work on it.