user mode backtrace

[David Boreham]

> The disk I/O schedules the reads and writes. It is unlikely that the 
> matching user process is running when the physical read occurs. 

Can I clarify the problem here ? : the user process isn't running 
because it's been put to sleep waiting on I/O that is done
in the context of another thread (or whatever we call that animal in the 
Linux kernel) , or is this when read-ahead is being
done on the file, at the behest of the kernel, with no corresponding 
user process read() call having been issued yet ?
(apologies my knowledge of present-day linux kernel details is rather 
patchy).

> Would it be useful to look at the elapsed time between entry and exit 
> time of the read system call and trigger recording of information if 
> it is over some threshold?

Possibly. I wouldn't want to get too hung up on this specific example -- 
I thought it up off the top of my head.
The point I was trying to make is that probing in userland in glibc may 
not do everything that's required because sometimes
one is interested in events that are only visible inside the kernel. 
Physical I/O was just the first one that I could
think of. I suppose a kernel mutex being busy might be another case.

> Would it make sense for the probing code to mark that userstack is 
> needed and then record when the processor is about to return to 
> userspace? The marking takes a fixed amount of time to do, so should 
> be safe for probes.

Yes, that sounds like a good way to do it.