libc with ARM unwind tables and frame pointers

Adhemerval Zanella adhemerval.zanella@linaro.org
Sun Jun 28 18:42:00 GMT 2015



On 26-06-2015 19:18, Sundar Dev wrote:
> Hi Adhemerval:
> 
> On Fri, Jun 26, 2015 at 12:57 PM, Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
>> Hi
>>
>> On 26-06-2015 15:26, Sundar Dev wrote:
>>> Hi List Members:
>>>
>>> First things first- we are using GNU libc-2.19 that comes packaged in
>>> Yocto Poky 'Daisy' release for one of our projects. This is
>>> cross-compiled for ARMv7 and we use arm-poky-linux-gnueabi-gcc (GCC)
>>> 4.8.2 toolchain.
>>>
>>> We use the *non-debug* version of libc in our project for obvious size
>>> and performance reasons. With the non-debug version, when a user space
>>> thread is executing a libc function like poll(), read(), etc., and we
>>> attach a remote gdbserver to the process and try to get backtrace, all
>>> we see is the following 4 backtrace frames-
>>>
>>> (gdb) bt
>>> #0  0x759abc00 in poll () from /lib/libc.so.6
>>> #1  0x759abbf4 in poll () from /lib/libc.so.6
>>> #2  0x00182c70 in ?? ()
>>> #3  0x00182c70 in ?? ()
>>> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>>>
>>> This, I can understand, is because the non-debug version is stripped
>>> of all debug symbols and hence GDB cannot backtrace. But here is the
>>> interesting part- we compiled libc (and all of yocto Poky toolchain
>>> components) with ARM unwind table and ARM frame pointers for another
>>> reason[1]. And, I know that the version of gdb that we're using
>>> (7.6.2) has support to backtrace using ARM unwind tables and frame
>>> pointers (see [2] and [3]). But, even then, all we get from GDB
>>> backtrace is the above shown output. Does anybody have any comments on
>>> why enabling ARM unwind tables and ARM frame pointers in libc still
>>> does not give us backtrace? And is there any other way we can get
>>> backtrace to go past libc functions without including debug sections
>>> in the libc elf?
>>
>> In fact your debug session GDB is indeed getting a stacktrace, what
>> it is not being to do is associate the address with meaningful names.
>> This is due not because missing debug information, but because not all
>> symbols in the executable are in dynamic symbol table.
>>
>> Backtrace man pages explains it briefly in 'NOTES' section and you
>> need to add -rdynamic compiler flags (which is will be translated to
>> -export-dynamic linker option). In a armhf environment using the
>> man pages example:
> So, I understand that -rdynamic flag will add entries for all symbols
> in the dynamic symbol table. But, I think my problem here is different
> and not related to symbolicating the addresses. Please allow me to
> explain. The addresses without symbols you see in the GDB backtrace
> (new one pasted below) are not even in my process' stack regions.
> 
> (gdb) bt
> #0  0x758b9190 in poll () from /lib/libc.so.6
> #1  0x758b9184 in poll () from /lib/libc.so.6
> #2  0x013df120 in ?? ()
> #3  0x013df120 in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> 
> And address 0x013df120 is in the heap region in proc/<pid>/maps (shown
> below) of my process-
> root@xyz# cat /proc/621/maps
> ...<snip>...
> 01389000-0154e000 rw-p 00000000 00:00 0          [heap]
> ...<snip>...
> 74941000-74980000 rwxp 00000000 00:00 0          [stack:756]
> 74980000-74981000 ---p 00000000 00:00 0
> 74981000-749c0000 rwxp 00000000 00:00 0          [stack:730]
> 749c0000-749c1000 ---p 00000000 00:00 0
> 749c1000-74a00000 rwxp 00000000 00:00 0          [stack:729]
> 74a00000-74a01000 ---p 00000000 00:00 0
> 74a01000-74a40000 rwxp 00000000 00:00 0          [stack:728]
> 74a40000-74a41000 ---p 00000000 00:00 0
> 74a41000-74a80000 rwxp 00000000 00:00 0          [stack:727]
> 74a80000-74a81000 ---p 00000000 00:00 0
> 74a81000-74ac0000 rwxp 00000000 00:00 0          [stack:726]
> 74ac0000-74ac1000 ---p 00000000 00:00 0
> 74ac1000-74b00000 rwxp 00000000 00:00 0          [stack:725]
> 74b00000-74b01000 ---p 00000000 00:00 0
> 74b01000-74b40000 rwxp 00000000 00:00 0          [stack:724]
> 74b40000-74b41000 ---p 00000000 00:00 0
> 74b41000-74b80000 rwxp 00000000 00:00 0          [stack:722]
> 74b80000-74b81000 ---p 00000000 00:00 0
> 74b81000-74bc0000 rwxp 00000000 00:00 0          [stack:721]
> 74bc0000-74bc1000 ---p 00000000 00:00 0
> 74bc1000-74c00000 rwxp 00000000 00:00 0          [stack:720]
> 74c00000-74c01000 ---p 00000000 00:00 0
> 74c01000-75400000 rwxp 00000000 00:00 0          [stack:669]
> 75400000-75500000 rw-p 00000000 00:00 0
> 75515000-75516000 ---p 00000000 00:00 0
> 75516000-75555000 rwxp 00000000 00:00 0          [stack:718]
> 75555000-755c0000 r--p 00000000 00:0b 1131       /usr/share/SetBitmaps.bitmap
> 755c0000-755c1000 ---p 00000000 00:00 0
> 755c1000-75600000 rwxp 00000000 00:00 0          [stack:659]
> 75600000-75621000 rw-p 00000000 00:00 0
> 75621000-75700000 ---p 00000000 00:00 0
> 75700000-75701000 ---p 00000000 00:00 0
> 75701000-75708000 rwxp 00000000 00:00 0          [stack:723]
> 75708000-75709000 ---p 00000000 00:00 0
> 75709000-75748000 rwxp 00000000 00:00 0          [stack:658]
> 75748000-75749000 ---p 00000000 00:00 0
> 75749000-75788000 rwxp 00000000 00:00 0          [stack:648]
> 75788000-7578c000 rw-p 00000000 00:00 0
> ...<snip>...
> 7592f000-75936000 ---p 0013d000 00:0b 428        /lib/libc-2.19.so
> 75936000-75938000 r--p 0013c000 00:0b 428        /lib/libc-2.19.so
> 75938000-75939000 rw-p 0013e000 00:0b 428        /lib/libc-2.19.so
> ...<snip>...
> 7e9d8000-7e9f9000 rwxp 00000000 00:00 0          [stack]
> 
> So, one of two things could have happened here-
> 1. GDB backtrace is just unable to get the stacktrace without debug
> symbols in libc. Having just ARM unwind tables and ARM frame pointers
> is not helping GDB to walk up the stack. This I'm little less
> suspicious because I've looked at the code in gdb's ARM port and I
> know the code exists there to use unwind tables and frame pointers to
> get backtrace.

I am not really familiar on how exactly gdb is getting and interacting
over the stacktrace, I was referring about GLIBC code. But I think the
issue is related to code generation in fact.

> 2. Or (I'm more suspicious about this one)- even though I build glibc
> with -fasynchronous-unwind-tables, -fno-omit-frame-pointer and
> -mapcs-frame compiler flags, glibc Makefiles may be overriding these
> flags for certain files/parts of the libc build and thus preventing
> gdb from walking the stack for these functions. Is there an easy way
> (I know I can grep and replace in all Makefile and then fix build
> errors which is the harder way) to force glibc to use the above 3
> flags and not allow the individual Makefiles to override them? And I
> want to do all this with a non-debug version of glibc (without .debug
> sections in elf).
> 

I checked with master and 2.19 and all debug/tst-backtrace{1-6} works on
ARM.  Also for the example in man page I think we are getting only 3
stack frames due a code generation issue. Building with lower optimization
flags and no inline I am getting:

$ gcc -Wall test.c -O1 -o test -funwind-tables -rdynamic -fno-inline
$ ./test 5
backtrace() returned 9 addresses
./test(myfunc3+0xb) [0x892c]
./test() [0x8986]
./test(myfunc+0x11) [0x899a]
./test(myfunc+0xb) [0x8994]
./test(myfunc+0xb) [0x8994]
./test(myfunc+0xb) [0x8994]
./test(myfunc+0xb) [0x8994]
./test(main+0x31) [0x89ce]
/lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x99) [0xf7694632]

I think it is due some missing unwind table information in the executable
with -O2 and higher.  Readelf different shows:

--- readelf-O1   2015-06-28 18:35:03.257521000 +0000        
+++ readelf-O2   2015-06-28 18:35:10.767521000 +0000

@@ -132,37 +132,30 @@                 
 00011034  00001616 R_ARM_JUMP_SLOT   0000882c   atoi
 00011038  00001016 R_ARM_JUMP_SLOT   00008838   abort
 
-Unwind table index '.ARM.exidx' at offset 0xa84 contains 6 entries:
+Unwind table index '.ARM.exidx' at offset 0xa64 contains 5 entries:
   
-0x8844 <_start>: 0x1 [cantunwind]
-
-0x8920 <myfunc3>: 0x80243fab
-  Compact model index: 0 
-  0x24      vsp = vsp + 148
-  0x3f      vsp = vsp + 256
-  0xab      pop {r4, r5, r6, r7, r14}
-
-0x8980 <myfunc2>: @0x8a6c
+0x8844 <main>: @0x8a58
   Compact model index: 1
   0xb1 0x08 pop {r3}
   0x84 0x00 pop {r14}
   0xb0      finish
   0xb0      finish
 
-0x8988 <myfunc>: @0x8a78
-  Compact model index: 1
-  0xb1 0x08 pop {r3}
-  0x84 0x00 pop {r14}
-  0xb0      finish
-  0xb0      finish
+0x887c <_start>: 0x1 [cantunwind]
+
+0x8958 <myfunc3>: 0x80243fab
+  Compact model index: 0
+  0x24      vsp = vsp + 148                                                    
+  0x3f      vsp = vsp + 256
+  0xab      pop {r4, r5, r6, r7, r14}
 
-0x899c <main>: 0x80a8b0b0
+0x89b8 <myfunc2>: 0x80b0b0b0
   Compact model index: 0
-  0xa8      pop {r4, r14}
+  0xb0      finish
   0xb0      finish
   0xb0      finish

-0x89d4 <__libc_csu_init>: 0x1 [cantunwind]
+0x89c0 <__libc_csu_init>: 0x1 [cantunwind]


 Symbol table '.dynsym' contains 37 entries:


As you can see the -O2 contains no unwind information for myfunc. I tried
with GCC 4.9.3 and saw the same behaviour and I am not sure if it is
being tracked or fixed in newer releases. 

>>
>> $ ./test 3
>> backtrace() returned 0 addresses
>> $ gcc -Wall test.c -O3 -o test -funwind-tables
>> $ ./test 3
>> backtrace() returned 3 addresses
>> ./test() [0x8710]
>> ./test() [0x8620]
>> /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x99) [0xf76aa632]
>> $ gcc -Wall test.c -O3 -o test -funwind-tables -rdynamic
>> $ ./test 3
>> backtrace() returned 3 addresses
>> ./test(myfunc3+0xb) [0x8968]
>> ./test(main+0x33) [0x8878]
>> /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x99) [0xf70c6632]
>>
>>
>>>
>>> Thanks,
>>> Sundar Dev
>>>
>>> [1] In the release version of our code, we don't want to have our
>>> applications and libraries built with debug symbols (for reasons of
>>> code bloat, performance, etc). But we would still like to get a stack
>>> trace in the event of a crash by using the glibc backtrace() and
>>> backtrace_symbols_fd() APIs. The libgcc implementation of these APIs
>>> works with ARM unwind tables and frame pointers. And hence we build
>>> our application and libraries with -fasynchronous-unwind-tables,
>>> -fno-omit-frame-pointer and -mapcs-frame flags and it works.
>>> [2] https://sourceware.org/ml/gdb-cvs/2011-02/msg00011.html
>>> [3] https://sourceware.org/ml/gdb-cvs/2011-02/msg00012.html
>>>



More information about the Libc-help mailing list