Re: Script to identify contended mutex locks with a stack trace.

[William Cohen <wcohen at redhat dot com>]

On 09/07/2010 05:21 PM, Alan Conway wrote:
> I'm trying to write a systemtap script that will identify contended mutex locks and print a user stack trace to identify them. Two unsuccessful attempts are attached, I'd appreciate any tips on how to make them work.
> 
> futexes.stp is a modified version of the futexes example provided with stap. I added a pthread_mutex_lock.return trace point and I attempt to print the user stack trace there. However the stack trace is just a hex address. Also for my contention_test.c (attached) test it claims that there are 9 locks contended 1 times each rather than 1 lock contended 9 times.
> 
> mutexes.stp sets probes on pthread_muted_lock and .return. This almost works but the stack trace for contention_test.c doesn't show functions main and foo as I expect, I get:
>  0x0000003073408ca0 : __pthread_mutex_lock+0x0/0x135 [/lib64/libpthread-2.5.so]
>  0x000000000040065e : _end+0x15de06/0x1e97a8 [/home/remote/aconway/install/lib/libqpidbroker.so.2.0.0]
> 
> If I run mutexes.stp on anything non-trivial it prints a bunch of very promising looking stack traces but then falls over with:
> 
> ERROR: Skipped too many probes, check MAXSKIPPED or try again with stap -t for more details.
> WARNING: Skipped due to global 'thread_blocktime' lock timeout: 101
> 
> Any tips on how to make these work would be greatly appreciated!
> 

Hi Alan,

What Linux distribution and version of systemtap are you using? RHEL-5 and Systemtap-1.2?

This is maybe a side effect of print_ubacktrace() taking too long. 

http://sources.redhat.com/bugzilla/show_bug.cgi?id=10665

You might try and see if this problem goes away with systemtap-1.3.

-Will