Contents

  1. Debugging the Loader
    1. Debugging With An Alternate Loader

Debugging the Loader

This page describes different methods for debugging the GLIBC dynamic linker/loader ld[64].so to achieve certain goals.

There are several loader environment scenarios:

  1. Loader is in the system toolchain, e.g. /lib[64]/ld[64].so.1
  2. Loader is in a self-contained toolchain, e.g. /opt/toolchain/lib[64]/ld[64].so.1

  3. Loader is in an install_root installed GLIBC directory, e.g. /home/user/glibc/install_root/lib[64]/ld[64].so.1 (This is an intermediary install step of GLIBC performed using make install_root=path install and is used for packaging purposes).

  4. Loader is in a non-installed GLIBC build directory, e.g. /home/user/glibc/build/glibc[32|64]/elf/ld[64].so.1

The first two scenarios are relatively similar and debugging shouldn't provide any great challenges other than the following:

The third scenario and fourth scenarios are challenging because you have to break the toolchain packages' assumptions about library paths. This is described by the following section:

Debugging With An Alternate Loader

Note: The examples here were done on the Power Architecture. The symbols that are used to break into the loader may vary by platform.

Note: For this example assume that GLIBC was configured and built into /home/user/glibc/build/glibc[32|64]. It doesn't need to be installed in order to test it. Our test application resides in /home/user/glibc/build/test.

Often times you may need to test an application against a GLIBC build that is not installed into the system root. You can use the following method to debug against a non-installed GLIBC build that was configured with any --prefix (but not installed).

In order to runtime test against the GLIBC build one must invoke the newly built loader, ld[64].so, from the GLIBC build directly passing ld[64].so the application name and using the --library-path directive to identify the location of the libraries built by the GLIBC build, e.g.

We'll create a BASH script which can direct the loader to launch the application and load the appropriate libraries:

If this GLIBC installation or the application that runs against it need to be debugged one must direct GDB to first debug the newly installed loader and the loader is directed to run the application to be debugged. We'll trap in the loader and then step into the application or libraries. This can be accomplished with a .gdb script.

In order to be able to step into the application or the libraries that are loaded we'll want to immediately trap in the loader. So we'll set a breakpoint on the following loader symbol:

In order to direct the loader to execute the test application and load the dependent libraries we'll use the --library-path directive.

We can accomplished all of this easily using a GDB script: Note: if the application you're debugging is a GLIBC test-suite application you must append the --direct flag to the application invocation as you'll see in the following examples.

Now we'll create a script to execute GDB and direct it to use this gdb script. Note: the -d directive tells GDB where to find the library sources.

NOTE: In order for gdb to be able to fully debug the application and standalone GLIBC you need to invoke it through the new standalone loader.

Otherwise GDB will indicate that it's version of libthread_db is different from the libpthread you're attempting to debug by giving the following error: 

versions of libpthread and libthread_db do not match

So we need GDB to be able to find all of the libraries created by the standalone GLIBC build.

You should see output like the following:

You'll notice that using the GDB script we've developed we're able to find the breakpoint for [_dl_main_dispatch\_dl_start_user] because GDB queried ld[64].so for debugging information. If we try to set a breakpoint on main GDB doesn't know where this is at (since libc the application hasn't been loaded yet) as indicated by the following output: 

Function "main" not defined.

If we were to execute the GDB directive continue at this point the debugger would not trap on main because it has no knowledge of how the addresses it encounters map to the symbols associated with the executable and libraries.

The root of the problem is that when GDB is directed to debug the loader, ld[64].so it simply queries the loader for symbol information since the loader is the target executable. The loader is then directed to load and launch the actual application test and the libraries, e.g. libc.so, using the GDB run directive: 

run --library-path <path to libraries> ./test

When the executable and libraries are loaded into the process address space GDB doesn't have any symbol or location information on these libraries and therefore it can't associate and symbols with addresses.

We can tell GDB how to find the symbol information about the executable and libraries by using the GDB add-symbol-file directive. But first we need to do a bit of homework to find out where the executable and libraries are mapped into the process's virtual address space.

We can use GDB's info proc mapping directive to find out where the libraries and application are mapped into the process address space: 

(gdb) info proc mapping
process 17755
cmdline =
'/home/user/glibc/build/glibc32/elf/ld.so.1'
cwd = '/home/user/glibc/build/test'
exe = '/home/user/glibc/build/glibc32/elf/ld.so'
Mapped address spaces:

Start Addr   End Addr     Size     Offset objfile
  0x100000   0x103000   0x3000   0x100000
 0x8000000  0x8020000  0x20000          0 /home/user/glibc/build/glibc32/elf/ld.so
 0x802f000  0x8030000   0x1000    0x1f000 /home/user/glibc/build/glibc32/elf/ld.so
 0x8030000  0x8031000   0x1000    0x20000 /home/user/glibc/build/glibc32/elf/ld.so
 0xfc95000  0xfd3c000  0xa7000          0 /home/user/glibc/build/glibc32/math/libm.so
 0xfd3c000  0xfd4c000  0x10000    0xa7000 /home/user/glibc/build/glibc32/math/libm.so
 0xfd4c000  0xfd4f000   0x3000    0xa7000 /home/user/glibc/build/glibc32/math/libm.so
 0xfd4f000  0xfd50000   0x1000    0xaa000 /home/user/glibc/build/glibc32/math/libm.so
 0xfd93000  0xff32000 0x19f000          0 /home/user/glibc/build/glibc32/libc.so
 0xff32000  0xff42000  0x10000   0x19f000 /home/user/glibc/build/glibc32/libc.so
 0xff42000  0xff46000   0x4000   0x19f000 /home/user/glibc/build/glibc32/libc.so
 0xff46000  0xff47000   0x1000   0x1a3000 /home/user/glibc/build/glibc32/libc.so
 0xff47000  0xff4a000   0x3000  0xff47000
0x10000000 0x10001000   0x1000          0 /home/user/glibc/build/test/test
0x10010000 0x10011000   0x1000          0 /home/user/glibc/build/test/test
0xf7ffc000 0xf7fff000   0x3000 0xf7ffc000
0xff857000 0xff86d000  0x16000 0xff857000
[stack]

So which Start Addr do we use for the libraries we're interested in? GDB doesn't tell us which section is associated with each address range.

We can check /proc/<pid>/maps to find out information about the sections: 

> ps aux | grep test | grep -v "grep" | awk -F' ' '{print $2}'
18171
> cat /proc/18171/maps
00100000-00103000 r-xp 00100000 00:00 0
08000000-08020000 r-xp 00000000 09:00 80040685 /home/user/glibc/build/glibc32/elf/ld.so
0802f000-08030000 r--p 0001f000 09:00 80040685 /home/user/glibc/build/glibc32/elf/ld.so
08030000-08031000 rw-p 00020000 09:00 80040685 /home/user/glibc/build/glibc32/elf/ld.so
0fc95000-0fd3c000 r-xp 00000000 09:00 78987266 /home/user/glibc/build/glibc32/math/libm.so
0fd3c000-0fd4c000 ---p 000a7000 09:00 78987266 /home/user/glibc/build/glibc32/math/libm.so
0fd4c000-0fd4f000 r--p 000a7000 09:00 78987266 /home/user/glibc/build/glibc32/math/libm.so
0fd4f000-0fd50000 rw-p 000aa000 09:00 78987266 /home/user/glibc/build/glibc32/math/libm.so
0fd93000-0ff32000 r-xp 00000000 09:00 78615158 /home/user/glibc/build/glibc32/libc.so
0ff32000-0ff42000 ---p 0019f000 09:00 78615158 /home/user/glibc/build/glibc32/libc.so
0ff42000-0ff46000 r--p 0019f000 09:00 78615158 /home/user/glibc/build/glibc32/libc.so
0ff46000-0ff47000 rw-p 001a3000 09:00 78615158 /home/user/glibc/build/glibc32/libc.so
0ff47000-0ff4a000 rw-p 0ff47000 00:00 0
10000000-10001000 r-xp 00000000 09:00 80561416 /home/user/glibc/build/test/test
10010000-10011000 rw-p 00000000 09:00 80561416 /home/user/glibc/build/test/test
f7ffc000-f7fff000 rw-p f7ffc000 00:00 0
ff94e000-ff964000 rw-p ff94e000 00:00 0        [stack]

We're interested in the Start Addr of the .text section of the application and libraries, i.e. the section marked by r-xp. 

> cat /proc/18171/maps | grep "r-xp" | grep test | awk -F'-' '{printf $1}'
10000000
> cat /proc/18171/maps | grep "r-xp" | grep "libc\.so" | awk -F'-' '{printf $1}'
0fd93000

These addresses indicate where in the process's virtual memory the application and C library are mapped, respectively. This information isn't enough but we'll use them later.

We need to look at the actual executable and library and determine at what offset the .text section actually begins. This is important because GDB needs to know where the .text information is at when it tries to gather symbol information about a library or executable. If we simply used the virtual address mapping of the executable and libraries then the symbol addresses would be wrong.

We can look at the actual executable and library ELF information for the .text information: 

> objdump -s --section=".text" /home/user/glibc/build/glibc32/libc.so | grep Contents -A 1 | tail -n 1 | awk -F' ' '{printf $1}'
01f760

Executables are a bit different. If the process is built without PIE (position independent executable) then it will reference the absolute address. 

> objdump -s --section=".text" /home/user/glibc/build/test/test | grep Contents -A 1 | tail -n 1 | awk -F' ' '{printf $1}'
100003a0

We're only interested in the offset portion. 

0x3a0

At this point we want to tell GDB, via the command add-symbol-file, the address ranges of the application and libraries in virtual memory so that it can query the symbol information for those libraries.

In the following modification to the GDB script the variables $test and $libc hold the virtual address mapping of the test application and libc library, respectively. To these we add the start address offsets we found in the objdump output which indicates where the .text section starts: 

> cat test.gdb

Let's re-run GDB with this modified script: 

> gdb -x test.gdb -d /home/user/glibc/libc /home/user/glibc/build/glibc32/elf/ld.so.1
GNU gdb (GDB) 6.8.50.20080707-cvs
...
Breakpoint 1 at 0x16bb8

Breakpoint 1, 0x08016bb8 in _dl_main_dispatch ()
add symbol table from file "./printf_dfp_test" at
        .text_addr = 0x100003a0
$1 = 0x1000050c
Breakpoint 2 at 0x10000520: file /home/ryanarn/glibc/stages/stage_printf_scanf/build/test/test.c, line 6.
add symbol table from file "../glibc32_power5/libc.so" at
        .text_addr = 0xfdb2760
$2 = 0xfde9930
Breakpoint 3 at 0xfde996c: file printf-prs.c, line 117.

If we execute the GDB directive continue the debugger will then trap on the main and register_printf_function function addresses as we desired.

None: Debugging/Loader_Debugging (last edited 2011-04-27 16:07:39 by RyanScottArnold)