This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: [PATCH -tip v6 00/22] kprobes: introduce NOKPROBE_SYMBOL(), cleanup and fixes crash bugs
- From: Masami Hiramatsu <masami dot hiramatsu dot pt at hitachi dot com>
- To: Ingo Molnar <mingo at kernel dot org>
- Cc: "Frank Ch. Eigler" <fche at redhat dot com>, linux-arch at vger dot kernel dot org, Ananth N Mavinakayanahalli <ananth at in dot ibm dot com>, Sandeepa Prabhu <sandeepa dot prabhu at linaro dot org>, x86 at kernel dot org, lkml <linux-kernel at vger dot kernel dot org>, "Steven Rostedt (Red Hat)" <rostedt at goodmis dot org>, systemtap at sourceware dot org, "David S. Miller" <davem at davemloft dot net>
- Date: Mon, 10 Feb 2014 13:02:41 +0900
- Subject: Re: [PATCH -tip v6 00/22] kprobes: introduce NOKPROBE_SYMBOL(), cleanup and fixes crash bugs
- Authentication-results: sourceware.org; auth=none
- References: <20131219090353 dot 14309 dot 15496 dot stgit at kbuild-fedora dot novalocal> <y0m38loefhx dot fsf at fche dot csb> <52B3C5E6 dot 2040802 at hitachi dot com> <20131220082056 dot GA15934 at gmail dot com> <52B40E79 dot 8040701 at hitachi dot com> <20131220104615 dot GA22609 at gmail dot com> <52E8E487 dot 2090007 at hitachi dot com> <20140209143709 dot GA31608 at gmail dot com>
(2014/02/09 23:37), Ingo Molnar wrote:
>
> * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>
>> I guess the second reason is why the stap takes so long time to set
>> probes. stap tries to register kprobes without disabled flag, that
>> means we enables thousands of probes (overheads).
>>
>> So the similar thing happens when we enables events as below;
>>
>> # for i in /sys/kernel/debug/tracing/events/kprobes/* ; do date; echo 1 > $i; done
>> Wed Jan 29 10:44:50 UTC 2014
>> ...
>>
>> I tried it and canceled after 4 min passed. It enabled about 17k
>> events and slowed down my system very much(I almost got hang check
>> timer).
>
> Ok, I guess that's the slowdown bug that Frank reported.
>
>> I think we should have some performance statistics (hit count?) and
>> if it goes over a threshold, we should stop enabling other events.
>
> That really feels like a hack. How about fixing the root cause? Does
> the enabling of all probes have to be so slow?
When I tried to use perf top, most of the time was consumed in
kprobe_ftrace_handler and optimized_callback, both of them
are the handler of kprobes. Since I just tried on a VM guest and it
didn't support NMI nor PMU, thus I have to use a bare metal machine
for deeper investigation (and I'll do).
And yes, it seems that the performance problem comes from probing
and tracing itself.
Thank you,
--
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com