Unable to retrieve pthread mutex symbols

[David Carter]

Mark,

Although I showed you how to set up the tests using the RPMs, I  
actually ran them with a development build compiled with the '-g'  
option,  so that's not the problem. This was true of the agent, the  
libraries, and the test program.

I think Frank nailed it when he noticed that the futexes I seek are  
actually in the shared memory region, so won't have any symbols  
available.

There are futexes used within the Berkeley DB library as well, so I  
would probably need to add it's debuginfo as well. I probably won't  
get to it this weekend due to other commitments, but I'll try it  
earlier next week. And the uprobe as well.

Thanks for the help guys!

- Dave

On May 29, 2009, at 11:56 AM, Mark Wielaard wrote:

> Hi David,
>
> On Thu, 2009-05-28 at 12:43 -0230, David Carter wrote:
>> Basically, I'm trying to debug mutex use in my program. I use the
>> pthreads version, which in turn uses futexes. Using the attached
>> systemtap script, I get the addresses of the futexes under  
>> contention,
>> but not the symbols associated with them. This makes debugging
>> difficult.
>>
>> There are a few steps involved in setting up the test.
>>
>> 1. Using either FC10 or FC11, install the arm4 and arm4-devel
>> packages. These don't have SELinux support yet, so you'll either have
>> to disable it or make it permissive.
>
> You also want the arm4-debuginfo package and dependencies, which you  
> can
> get through debuginfo-install arm4.
>
>> 2. Add your user to the arm4 group
>>
>>  # usermod -a -G arm4 username
>>
>> You'll have to log out and log in again for this to take effect.
>>
>> 3. Start the arm4 service
>>
>> 4. Compile the arm4 sample program
>>
>>  $ cc -o arm4test_hello_throughput arm4test_hello_throughput.c -larm4
>
> Again add debug symbols by adding -g. Currently all symbolic  
> information
> (backtraces, symbol names, etc) are fetched from the debuginfo.
>
>> 5. Run the systemtap script while running the arm4 sample program. I
>> recommend starting the systemtap script before starting the sample.
>> The sample has an infinite loop so stop it after 30 seconds or so.
>> The output I get shows the hex addresses of the futexes rather than
>> the symbols. This makes it difficult to debug.
>
> Yes, then it still doesn't work :(
> I am not sure how futexes work though, the futex_uaddr given seems to
> point in the middle of the address range that has /var/lib/arm4/__db. 
> 001
> mapped for me (see with /proc/<pid>/maps). Obviously there isn't an  
> user
> symbol there. But the address doesn't really make sense to me, so  
> maybe
> I am misinterpreting something.
>
> BTW. For user space symbol resolving to work properly systemtap  
> needs to
> have the symbol table for the application and needs to know which
> executables to track mmapping of code for (the symbol tables it keeps
> are relative to the start of the address that a code segments gets
> mapped to into the process space). This is automatically done if you
> have a uprobe on the process. So in this case I would add something  
> like
> probe process("arm4_daemon").function("main") { log("daemon  
> started"); }
> just to take care of that.
>
> Cheers,
>
> Mark
>
>

----
Custom Software Solutions
http://www.davidcarter.ca
http://davidcarterca.wordpress.com