Minutes of 6/9/05 meeting

[Jim Keniston <jkenisto at us dot ibm dot com>]

Minutes of SystemTap Conference Call, June 9, 2005

Participants:
IBM:
	Larry Kessler (lkessler@us.ibm.com -- Beaverton, OR)
	Vara Prasad (varap@us.ibm.com -- Beaverton)
	Jim Keniston (jkenisto@us.ibm.com -- Beaverton)
	Ananth Mavinakayanahalli (amavin@redhat.com, ananth@in.ibm.com --
Boston)
	Prasanna Panchamukhi (prasanna@in.ibm.com -- Bangalore)
Red Hat:
	Elena Zannoni (ezannoni@redhat.com -- Boston)
	Will Cohen (wcohen@redhat.com -- Raleigh)
	Martin Hunt (hunt@redhat.com -- Seattle)
	Frank Eigler (fche@redhat.com -- Toronto)
	Andrew Cagney (cagney@redhat.com -- Toronto)
Intel:
	Brad Chen (brad.chen@intel.com -- Santa Clara)
	Rusty Lynch (rusty.lynch@intel.com -- Hillsboro, OR)
	Anil KeshavaMurthy (anil.s.keshavamurthy@intel.com -- Hillsboro)
	Charles Spirakis (charles.spirakis@intel.com)
	Chris Elford

Actions Required (ARs):
>From 5/12/05:
17. Vara: Publish new call-in number via private email.
[Update: We used the Red Hat number again for this morning's call,
but for the 6/16 meeting agreed to use the new IBM number, which Vara
will email everybody.  If you fail to receive this number from Vara
by 6/16, call the Red Hat number.]

>From 5/19/05:
19. Vara: Collect sample kprobes modules and post them to the
SystemTap web site.

20. Rusty: Create a kprobes test for ia64, per his 5/12/05 design,
that tests probes on a large subset of the ia64 instruction set.
Work with Prasanna to create such tests for other architectures.
[Update: See patches/functional_kprobe_test in CVS.]

21. Rusty, Will: Look into adding kprobes tests to LDP and the
Red Hat suite, respectively.

22. Rusty: Look into x86_64 limitation re: temporary disarming of
reentrant probes.

23. Jim, Martin: Ensure that the runtime's stack-trace feature works
appropriately when there are return probes in the call stack.

>From 6/2/05:
24. Vara: Is it OK to post the OLS paper to the SystemTap web page?

>From 6/9/05:
25. Frank: Post matrix of kprobes features vs. architectures for
kprobes authors to fill in.

26. Rusty/Brad: Forward email address of Intel test person to Jay
Turner.

27. Charles: Document (e.g., via a tapset template) the various
approaches (e.g., timestamp, sequence number) to sequencing and
merging relayfs streams from different CPUs.


We got a slow start due to multiple problems trying to dial in.
See AR #17.

Round table:

Will and Charles were at the Red Hat Summit.  They discussed ways of
providing SystemTap with access to performance monitoring hardware.
They discussed abstraction layers such as PAPI, and porting perfmon
features to i386.  Stephane _____ (Eranian?) is working on this.
(Sorry, I didn't catch all this discussion.)

Martin has mostly completed the aggregations feature of the runtime.
He still has some testing and documentation/examples to do.
There's demand at IBM for a ppc64 port of the runtime.  Ananth will
attempt this.

Frank continues work on the translator.

Frank asked that we begin maintaining a matrix that documents which
kprobes features are working on which architectures.  Frank will
post the initial matrix; kprobes authors will maintain it.

Elena said we need more kprobes stress testing.  We will take this
on in the coming weeks.  Jay Turner will assist.  Red Hat may not
have all the systems needed to run these tests.  Intel has a testing
resource; Rusty (Brad?) will forward his email address to Jay.

Ananth has been working on various ppc64 cleanup patches for kprobes,
three of which Andrew M. has pushed to Linus.  He has been working on
SMP scalability again, and hopes to post an update next week.

Jim reviewed several kprobes enhancements, reviewed Vara's latest C
Tapset proposal, and continued consulting with IBM engineers using
kprobes and the SystemTap runtime.

Vara has been doing more SystemTap evangelizing around IBM.  He
hopes to update his C Tapset proposal again soon.

Tom is working on relayfs support in the runtime -- e.g:
- a protocol to determine whether to use netlink or relayfs (?)
- merging the per-CPU data streams created by relayfs
He expects to submit relayfs again for inclusion in the kernel.

Is there a need for a relayfs/netlink performance comparison?  No,
they're intended for use in different situations.  Performance isn't
the only issue.

When merging per-CPU data streams, what will be the merge key?
Sequence number?  Timestamp?  There's no guarantee in a multiprocessor
that the CPUs' clocks (TSCs) will be synchronized.  But atomic
increment of a shared sequence number won't scale to a large number
of CPUs (e.g., 128) due to cache ping-pong.  (Devoting a whole cache
line to the counter won't eliminate the ping-pong, but will minimize
the involvement of other data structures.)  For now, go with a shared
sequence number.  Per Frank's suggestion, Charles (possibly with
Matin's help) will create a tapset-style template that lays out the
possible approaches (timestamps, sequence numbers, etc.).

Rusty has posted a patch for return probes that simplifies things
somewhat and solves at least a couple of architecture-specific
problems.  The fundamental change is that we store the task pointer
rather than the stack pointer in struct kretprobe_instance.
Stack unwind may be trickier with this implementation.

Anil has been working on ia64 kprobes cleanup -- e.g., supporting
(or refusing to probe) more instruction types.  See LKML on 6/6/05.

Brad reports that Victoria Gromova will post a tapset proposal
in the next several days.  It will focus on kernel locks and
on user-level locks that use system calls, but not on other
user-level locks.

Chris is looking at Java probes based on what DTrace does.  We can't do
much here without user-mode probes.

Prasanna lost his connection during the call, but reported the
following status via email:
"Last week most of the time was spent on making a system call list
and corresponding kernel routines.  The list has been posted to
systemtap mailing lists. Also a clean up patch to move serveral kprobes
global routine to static has been submitted to lkml and systemtap.
Currently this patch is in -mm tree. I will be working on tapsets
for I/O subsystems."