This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: How to get the detailed kernel stack trace if specified fuction takes too long to finish?
- From: fche at redhat dot com (Frank Ch. Eigler)
- To: liang xie <xieliang007 at gmail dot com>
- Cc: systemtap at sourceware dot org
- Date: Fri, 28 Mar 2014 12:56:47 -0400
- Subject: Re: How to get the detailed kernel stack trace if specified fuction takes too long to finish?
- Authentication-results: sourceware.org; auth=none
- References: <CADu=CFo=AtrBJLacPTa_6EfQA7vao=bSuT2Yns45x-hWYfmWZA at mail dot gmail dot com>
Hi -
xieliang007 wrote:
> Probably it's a newbie question, to be honest, i am new to SystemTap,
> currently i want to capture the kernel stack trace if "sys_write" runs
> too long, if that i could know where it sucks.
OK.
> [...] probe kernel.function("sys_write").return [...]
> print_backtrace();
> Returning from: 0xffffffff81176f90 : sys_write+0x0/0x90 [kernel]
> Returning to : 0xffffffff8100b0f2 : system_call_fastpath+0x16/0x1b [kernel]
>
> It doesn't show my "expected" kernel stack trace,e.g. file system or
> block layer related stack trace.
The expectation is mistaken. By the time that sys_write returns, all
that kernel activity is in the past, and will have left no trace on
the stack.
> How to archive that? Any pointers will be highly appreciated!
Not easy.
One way could be to sample the kernel backtrace (from probe
timer.profile) while a sys_write is in effect. Attributing the
backtraces to the particular thread suffering from a slow sys_write is
going to be tricky, but maybe this could work:
global backtraces%
global active
probe syscall.write {
active[tid()]=1
}
probe timer.profile {
if (! (tid() in active)) next;
backtraces[tid(), backtrace()] <<< 1
}
probe syscall.write.return {
delete active[tid()]
elapsed = gettimeofday_ms() - @entry(gettimeofday_ms())
if (elapsed <= 4)
next; // oops, leaves behind backtraces[] records
printf("%s[%d] sleepy for %d ms:\n", execname(), tid(), elapsed)
foreach ([t,bt] in backtraces- limit /* top */ 10)
if (t == tid()) {
printf("\n%d hits:\n", @count(backtraces[t,bt]))
print_stack(bt)
}
}
# stap --all-modules FOO.stp
- FChE