user mode backtrace

[David Boreham]

Stone, Joshua I wrote:

>On Thursday, October 19, 2006 1:47 PM, David Boreham wrote:
>  
>
>>I'd like to get a stack trace for the process that made the
>>system call I'm probing (I'm looking at filesystem access
>>typically, so reads/writes/syncs etc). The systemtap backtrace
>>function appears to only get the kernel mode stack which
>>is not much use to me. I was wondering if anyone had
>>discovered a good solution to this problem already ?
>>I was thinking perhaps I could invoke pstack (gdb)
>>on the current pid/tid. But I'm worried that doing so
>>might deadlock since the process is inside a system
>>call.
>>
>>I'm looking at a very large application that beats up on
>>the filesystem, in case you're wondering why I want to do
>>this. It's so large that nobody is quite sure what code
>>access which files, when and why.
>>
>>Thanks.
>>    
>>
>
>Deadlock issues aside, there's not really a way for you to invoke a
>process (like pstack) from within a SystemTap script.  You could run a
>separate user program or script to do this for you though, and then you
>just need to coordinate with SystemTap.  Such a method might look like
>this:
>  
>
Yeah, that was the kind of hack I was thinking of.

>Of course at the end of the day, this is just a convoluted strace with a
>stack printout.  
>
Not really. The application that I'm battling with is resistent to my 
'traditional'
analysis techniques, which would include strace, gdb/pstack, oprofile etc.
The big show stopper is that it isn't one big process that you can look at.

It fires up numerous short lived processes. There isn't even one master
process that exec's the subprocesses: some of them kick in on timers,
watching the modification dates on files, and via local pipes.
Hence the system-wide approach, hooking kernel activity and then looking
upward to figure out what code in the application, in which process, is 
burning
I/O resources.