Bug 7042

Here is a video of James presentation in a conference in Japan, 

http://www.linuxfoundation.org/events/node/174

From 00:54:00-, he answered a question about off-kernel project like systemtap.
We might have to respond his message and change our way.

> http://www.linuxfoundation.org/events/node/174
> From 00:54:00-, he answered a question about off-kernel project like systemtap.
> We might have to respond his message

http://sources.redhat.com/systemtap/wiki/Myths

> and change our way

OK, but what changes would that particular video inspire?

(In reply to comment #2)
> OK, but what changes would that particular video inspire?

From his presentation, I think he want us to follow "upstream first" policy.
As he said in the video that out-of-kernel project like closed source driver costs
high to follow the upstream kernel.

I definitely agree that it's hard to merge whole of the systemtap into kernel tree.

however, I think most of our runtime code(and parts of code generated by translator)
can be merged into kernel. (ex. safety memory access(which has already been
merged!),
trace framework including probing and buffering)

And if we merge them, we can concentrate on developing translator, and will not take
care about upstream kernel changes any more.

> > OK, but what changes would that particular video inspire?
> From his presentation, I think he want us to follow "upstream first" policy.
He made many claims in that talk, but that one did not seem to be one of them.

I'm not saying that attempting to put some parts of the runtime into
the kernel is a bad thing - just that he was off that topic.

BTW, here is another video of the presentation by Ted Ts'o.

http://www.linuxfoundation.org/events/node/175

From 00:39:40-, I think he explained what was the problem of Systemtap project
more clearly.
- There is no *senior* kernel developers in the project.
 - To develop useful tapset, they(systemtap) need a help from kernel subsystem
maintainers.
 - RAS tools is usually for Enterprise users and not for kernel community, it's
not useful.
- Now kernel community developing their own tracer(ftrace).

(In reply to comment #2)
> http://sources.redhat.com/systemtap/wiki/Myths

By the way, I think that many people claimed that the systemtap always required
kernel-debuginfo which was too big and was hard to find an appropriate package.

Here is the action ideas which I sent to systemtap-ml on
(http://sources.redhat.com/ml/systemtap/2008-q4/msg00639.html)

- Communication issue
 - Notifying releasing on LKML frequently.
 - Preparing more documents for kernel developers.
 - Feeding back kernel developers' opinions.

- Solving debuginfo issue
 - in the short term, notifying how to minimize debuginfo size.
 - in the mid term, making dummy (function-entry) debuginfo.
 - in the long term, Roland's dwarf compressor can solve this issue.

- Upstream first issue
 - Utrace/Uprobe
  - Develop code on LKML and embrace feedbacks.
  - Involving main kernel maintainer to speed up the code merging.
 - Systemtap itself
  - Merging a part of runtime and basic tapset to kernel tree, which
   will provide us a permanent API for systemtap generated code.
  (BTW, I'd like to suggest modifying ftrace plug-able and systemtap
   generating ftrace-plugin tracers. This will be more acceptable for
   upstream because we don't need to add so many code).

Would what you illustrate in comment #7 entail a new git branch for upstream
merge, especially for the runtime and buffer portions?

(In reply to comment #8)
> Would what you illustrate in comment #7 entail a new git branch for upstream
> merge, especially for the runtime and buffer portions?

Yes, because we need to clean up the runtime code for merging, and share it.
After merging, we may still have a thin layer of runtime code.

(In reply to comment #7)
>  - Systemtap itself
>   - Merging a part of runtime and basic tapset to kernel tree, which
>    will provide us a permanent API for systemtap generated code.
>   (BTW, I'd like to suggest modifying ftrace plug-able and systemtap
>    generating ftrace-plugin tracers. This will be more acceptable for
>    upstream because we don't need to add so many code).

What I suggested here is changing runtime code (and translator too).
It will lose some kind of backward compatibility, but since we can't bump ftrace
out from kernel any more, there is no way to merge runtime directly to kernel.
Instead of that, we have to use and improve ftrace interface for systemtap.

So, Here are what we have to do, I think
- Assign someone to runtime merging task, who may talk with kernel developers.
- Modify ftrace to support kernel modules. this might have to involve Steven
Rostedt.
- Modify staprun/stapio to support ftrace interface, and replace transport
functions with ftrace interfaces.
- Get rid of duplicate codes from runtime.
- Modify translator to generate ftrace module.
- Push useful parts of runtime code to upstream kernel(might be "kernel APIs for
tracers").

ftrace and systemtap should be able to coexist on the kenrel.

Here, I'd like to try to figure out technical details of my idea.

> - Modify ftrace to support kernel modules. this might have to involve
> Steven Rostedt.

This basically means exporting register_tracer() for kernel modules and adding
unregister_tracer().

> - Modify staprun/stapio to support ftrace interface, and replace transport
> functions with ftrace interfaces.

staprun may not be changed so much. stapio will be changed, but mainly the
change will be just about the path of debugfs file.

> - Get rid of duplicate codes from runtime.

this needs more investigation. I assume that most of autoconfs will be removed
and some of safety memory/register accessing code should be merged to upstream
kernel for debugging.

> - Modify translator to generate ftrace module.

This doesn't means using ftrace_printk(). Systemtap will generate a kernel
module which has a struct tracer and its own ring buffer file(because several
tracers will run in parallel).

> - Push useful parts of runtime code to upstream kernel(might be "kernel
> APIs for tracers").

As I said above, some of safety code should be merged, and other utility codes
including unwinder, map, statistics, taskfinder and so on.

Hi

if Systemtap people works on upstream merging, Fujitsu might help some works and
activity.
please contact me.

(In reply to comment #11)
> > - Modify ftrace to support kernel modules. this might have to involve
> > Steven Rostedt.
> 
> This basically means exporting register_tracer() for kernel modules and adding
> unregister_tracer().

OK...

> > - Modify staprun/stapio to support ftrace interface, and replace transport
> > functions with ftrace interfaces.
> 
> staprun may not be changed so much. stapio will be changed, but mainly the
> change will be just about the path of debugfs file.

OK, bug #7043.

> > - Get rid of duplicate codes from runtime.
> this needs more investigation. I assume that most of autoconfs will be removed

(But they can't, if we wish to still run on non-bleeding-edge kernels.)

> and some of safety memory/register accessing code should be merged to upstream
> kernel for debugging.

Sure, if there is upstream interest in those small and stable bits.

> > - Modify translator to generate ftrace module.
> 
> This doesn't means using ftrace_printk(). Systemtap will generate a kernel
> module which has a struct tracer and its own ring buffer file(because several
> tracers will run in parallel).


> > - Push useful parts of runtime code to upstream kernel(might be "kernel
> > APIs for tracers").
> 
> As I said above, some of safety code should be merged, and other utility codes
> including unwinder, map, statistics, taskfinder and so on.

The unwinder is blocked on more experience with it.
The task-finder is blocked on utrace.

The map/statistics stuff could go, to see if kernel folks were interested in
having it.  (Since this part has not needed porting effort, the status quo does
not represent a cost to the project.)  Who wants to try proposing it?

> > > - Get rid of duplicate codes from runtime.
> > this needs more investigation. I assume that most of autoconfs will be removed
> 
> (But they can't, if we wish to still run on non-bleeding-edge kernels.)

Hmm, so, how about just branching sources for newer kernels?
Fortunately, we already have -R option.

> > and some of safety memory/register accessing code should be merged to upstream
> > kernel for debugging.
> 
> Sure, if there is upstream interest in those small and stable bits.

Those stuffs will be useful for other developers who want to write a
custom tracer in plain c.

> > > - Push useful parts of runtime code to upstream kernel(might be "kernel
> > > APIs for tracers").
> > 
> > As I said above, some of safety code should be merged, and other utility codes
> > including unwinder, map, statistics, taskfinder and so on.
> 
> The unwinder is blocked on more experience with it.
> The task-finder is blocked on utrace.
> 
> The map/statistics stuff could go, to see if kernel folks were interested in
> having it.  (Since this part has not needed porting effort, the status quo does
> not represent a cost to the project.)  Who wants to try proposing it?

It's a part of runtime merging task. We may need to provide some ftrace-based
examples for these parts as Frank did for utrace.

I still doubt that just using ftrace's ring buffer can clear all of the
criticism from kernel developers. Perhaps, we need more interaction between us
and them. That means we may need to improve kernel itself by using some portion
of systemtap code or our knowledge. That is what they hope.

Therefore, I think systemtap-ftrace integration has big advantage for that purpose.
- Kernel can have only one trace infrastructure, it simplifies code maintenance.
- Kernel developers can easily make prototypes of their tracers by using
systemtap. When they think that tracer is enough stable and useful, they can
push the refined c code into kernel. So, it's good for ftrace too.
- systemtap can show its presence in the kernel community.

This is probably moot, with more lkml attention on perf.