Re: Discussion at Linux Foundation Japan Symposium

["Frank Ch. Eigler" <fche at redhat dot com>]

Hi, Ted -


Thanks for commenting.


On Mon, Dec 22, 2008 at 01:19:21PM -0500, Theodore Tso wrote:
> > Some of this has been predicated on the belief that debuginfo quality
> > will improve enough that we do not need to resort to hard-coded
> > per-architecture per-compiler hacks like some in gdb.
> 
> Yes, but how long will people need to wait before debuginfo quality
> will improve?  And when such a hypothetical improved compiler be
> available and trusted to compile the kernel correctly?  [...]

You misinterpret the status quo.  In reality, debuginfo quality with
current gcc has been satisfactory for many purposes.  That is what
enables user-space debuggers to work as well as they do, and indeed
enables systemtap to work as well as it does.  The major current
debuginfo quality shortcoming involves only those intra-function
statement probes.


> More importantly, what will systemtap development do in the meantime?
> There are a couple of choices:

> 1) Do nothing.  Kernel developers will continue to largely give up on
> Systemtap, mostly out of frustration.  [...]

If you're still talking about debuginfo per se, I am led to believe
that only a small number of kernel developers are negatively affected:
those that build large kernel configurations but on such miniscule
hardware that the sheer disk I/O of the -g data is deemed intolerable.

If you're talking about other aspects, please elaborate.


> The process of getting userspace probes in will go slowly (if at
> all).  I haven't seen a new version of the patches submitted in
> months --- and the 2.6.29 merge window will be opening soon.  [...]

Roland?  Please advise.


> 2) Explain to developers how to use Systemtap in a painless way as
> possible.  

The http://sources.redhat.com/systemtap/wiki/SystemTapWithSelfBuiltKernel
page is becoming simpler and simpler.  Please try it.

> This means no modules, since debuginfo is a disaster with modules
> (so helping people create configurations that don't use modules
> would be helpful);

Please elaborate what you mean.  Why systemtap is forced to use
(create) modules at all has been explained often.


> and/or with a focus on markers as a debugging mechanism, even if
> that means that you have to recompile the kernel first before they
> can be used.  If making markers more helpful is the goal, then work
> to make the Modules.marker file consistently installed in the same
> place, or to make Systemtap look to find it in multiple places, is
> going to be important.

Sure, upstream changes in the location of Modules.marker are something
we can easily adapt to.  Are you aware of a current problem with that?
On the other hand, we plan to work on also connecting straight to
tracepoints, and to various slices of ftrace, for which a file such
the troubled-history Modules.markers will not be necessary.


> 3) Actually try to find a way for Systemtap to be more useful
> *without* waiting for [new gcc]

As I said above, the main thing we're expecting from a future gcc is
improvement on statement (arbitraty source line) probing.  Otherwise
quality is fine and we're not having to wait.  (As everyone knows,
quantity (compression) is not great but at least this is only a
usability rather than functionality matter.)

> Consider how ftrace works; it very cleverly uses/abuses the
> profiling hook inserted gcc -pg.

Yes, it's clever (though somewhat fragile).

> Note that it doesn't wait for magic new capabilities in the
> toolchain.

Nor do we.  In fact, we've been as unimposing as possible upon both
kernel and user-space build systems -- perhaps to a fault.

> [...] the only advantage SystemTap has over ftrace is (a) Systemtap
> has access to the function parameters, and (b) Systemtap allows you
> to filter logging information based on conditions, which is more
> efficient than filtering in userspace.

Beyond those, there is really plenty of difference in capability.
Line-by-line function tracing?  User-space?  Concurrent users?  The
list goes on...


> > > [...] Would could build some scripts with the binaries where debug
> > > info is available, interpret that info as relative offsets to public
> > > symbols then dispense with the info on the shipped production
> > > script.  See where I'm going here?
> > 
> > It sounds like an interesting sort of shortcut - one that we have not
> > seriously considered.  It may be far too optimistic, considering the
> > degree of configuration/optimization change possible between two
> > similar kernel builds, and the prevalent hostility toward reusable
> > kernel modules.
> 
> Another solution which I've suggested in the past, 

(I must have missed it.  This one's new to me.)

> [...] is to simply generate the debuginfo information on demand so
> the function parameters on entry can be decoded.  We have the header
> files after all, so it shouldn't be hard to generate a pseudo function:
> 
> extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
>        void *ptr)
> {
>         return 0;
> }

It has some potential.  On first blush this seems to require parsing
all the C header files (accounting for arch/kconfig cpp macros) to
look for a matching function declaration.  Then the parameter/return
types may require listing additional C headers.  As you know, kernel
headers are complicated and rich with #ifdefs and macros that may
expand to inlines inlines, so a naive text processing inspired by the
following may not be worthwhile.  How do you imagine it working?

fgrep -v '[' /proc/kallsyms | while read addr type fn 
do
echo $fn; find /lib/modules/`uname -r`/build/include -type f | xargs egrep " ${fn}[ ]*\("
done


- FChE