Re: Ftrace and Systemtap

[Masami Hiramatsu <mhiramat at redhat dot com>]

Masami Hiramatsu wrote:
> Hi Steve,
> 
> swb6@aber.ac.uk wrote:
>> Hi,
>> I am currently writing up my
>> dissertation based on dynamic tracing and reducing the learning curve to
>> understanding the UNIX kernel. I used two dynamic tracing technologies
>> (Systemtap and Dtrace) to monitor some virtual machines on a network.
>> Whilst going over my initial review of technologies in the area of dynamic
>> tracing I realised I had underestimated ftrace, and even dismissed it as
>> not being dynamic. However I read here (http://lwn.net/Articles/343766/)
>> that you were as of July last year trying to introduce a patch that would
>> allow ftrace the ability to dynamically trace. Also it would be completely
>> instrumented from the command line without the need to compile a kernel
>> module al la Systemtap.
> 
> Right, dynamic event tracer allows ftrace to add new events online.
> Even though it requires debuginfo, it doesn't need any kernel build
> environment.
> 
>> My question is do you think Systemtap will be replaced by ftrace in the
>> future if so I feel I may need to include this fact in my updated
>> technology review.
> 
> Good question.
> 
> Ftrace dynamic tracer may be enough for 80% of kernel developers,
> but Systemtap has other advantages, e.g. fault injection, pre-recording
> scripting, multiple instances etc. Some of these advantages are just
> not implemented on ftrace yet, and some of them can not be implemented
> on ftrace. Even if ftrace implements it, systemtap team can port stap
> script engine on it.
> So, the *technical* answer is no.
> 
> However, there is another issue between systemtap project and kernel
> development. Since both projects are asynchronously developed,
> I always hear complaints which systemtap can't be used for the latest
> developing kernel. And yes, that's true.

It will happen on a personal development branch. (note that
all developments are started on a personal branch :))
Not(or rarely) on public linus/linux-next tree.

If someone changes some APIs which  systemtap depends on,
systemtap can't work on his kernel.
e.g. just removing EXPORT_SYMBOL_GPL(register_kprobe).

This also shows why the out-of-tree driver is so fragile.

Currently, ftrace is kernel-internal subsystem, so it will
not be effected by those changes, or if some APIs which ftrace
depends on, are changed, he will just fail to build kernel.
(and they need to fix ftrace too)

Of course, someone who wants to use systemtap on his kernel
knows what APIs unchanged, it can be avoided, but it still
needs some experience on the systemtap development. :-(

Maybe, we'd better make a blacklist of APIs which systemtap
depends on.

Thank you,

> 
> And now, ftrace and perf tools are the primary tracing tools for linux
> kernel (at least for kernel developers). New tracing features without
> them will NOT be merged into upstream anymore. That means systemtap
> can't add any new facilities without implementing it on ftrace/perf-tools.
> IOW, systemtap development already depends on ftrace/perf.
> (This is why I'm working on that.)
>  
> So, I don't have any simple answer for your question. What I can
> say is that ftrace never be replaced by systemtap anymore. :-)
> 
> Thank you,
> 

-- 
Masami Hiramatsu
e-mail: mhiramat@redhat.com