• Debugging
• Development_Debugging

General GLIBC Debugging Techniques

Debugging the Loader

There are several loader debugging scenarios:

1. The application crashes very early. - There is a bug in the libc init code which is prior to application main being executed. You must debug the loader in order to investigate the init code.

2. The loader itself crashes. - This could be due to a runtime error in the loader. You need to debug the loader directly.

3. The loader has demonstrated fault in its symbol resolver code. - You can step into the loader from the application main procedure.

4. An alternate loader is being used and an application or library needs to be debugged. - An application or library that uses an alternate loader must be debugged by debugging the loader and stepping into the application or library.

Debugging the Loader Before Application Main

Direct GDB to debug the loader and then direct the loader under debugging to load the application.

For example:

bash-4.2$ gdb /home/carlos/build/glibc/elf/ld.so 
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/carlos/build/glibc/elf/ld.so...done.
(gdb) set exec-wrapper env LD_LIBRARY_PATH=/home/carlos/build/glibc:/home/carlos/build/glibc/elf:/home/carlos/build/glibc/math
(gdb) break _start
Breakpoint 1 at 0x11e0
(gdb) run /home/carlos/test
Starting program: /home/carlos/build/glibc/elf/ld.so /home/carlos/test

Breakpoint 1, 0x00005555555551e0 in _start ()
(gdb) 

One important thing to note is that you must not use set env to set environment variables that affect the dynamic loader. The reason because gdb by default runs the inferior in a sub-shell, this is done for several reasons the most important is argument expansion. If the environment variables are set for the dyanmic loader they will also impact the sub-shell and may cause the sub-shell to fail to start the inferior.

Stepping Into the Loader

Some parts of the loader can be stepped into (like the runtime resolver). These parts are easier to debug because you can simply step into them from your application.

For example:

bash-4.2$ gdb /home/carlos/test
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/carlos/test...done.
(gdb) break main
Breakpoint 1 at 0x400534: file test.c, line 8.
(gdb) r
Starting program: /home/carlos/test 

Breakpoint 1, main () at test.c:8
8         keys[512] = 42;
(gdb) n
9         printf ("%d\n", keys[512]);
(gdb) si
0x0000000000400544      9         printf ("%d\n", keys[512]);
(gdb) 
0x0000000000400546      9         printf ("%d\n", keys[512]);
(gdb) 
0x000000000040054b      9         printf ("%d\n", keys[512]);
(gdb) 
0x0000000000400550      9         printf ("%d\n", keys[512]);
(gdb) 
0x0000000000400410 in printf@plt ()
(gdb) 
0x0000000000400416 in printf@plt ()
(gdb) 
0x000000000040041b in printf@plt ()
(gdb) 
0x0000000000400400 in ?? ()
(gdb) 
0x0000000000400406 in ?? ()
(gdb) 
_dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:34
34              subq $56,%rsp
(gdb) 

In the example you can see the debugger on x86_64 stepping through the procedure linkage table (3 instructions) followed by another 3 instructions (PLT entry that always jumps into the resolver and to which all lazy PLT entries are bound to at startup) that jumps to the runtime resolver routine. The purpose of this routine in glibc is to locate the symbol you need, bind the PLT entry to that symbol, and call the symbol with your arguments. This only happens if your have lazy symbol resolution, if LD_BIND_NOW is enabled you will see a direct jump to prtinf from the PLT.

Debugging With an Alternate Loader

Often, during development, GLIBC will be installed into a standalone environment. In order to test an application in this environment the application will be launched using the loader installed into the standalone environment, i.e. not the system loader. This creates a unique debugging environment where in order to debug the exact execution environment of an application or library the user must initiate debugging of the loader and then step into the application or library.

This presents some challenges for the GNU debugger gdb in finding symbol information about the application and libraries. When GDB debugs the loader it simply loads the symbol information for the loader alone. You need to perform some special operations to encourage GDB to find the symbol and addresses for the application and libraries.

Follow the techniques described in the Loader Debugging page under the section entitled "Debugging With an Alternate Loader".

Injecting Code From A Bad Build Into A Testcase From A Good Build

Often times one will be working on a bit of code (e.g. memcpy) that'll be picked up and used by a fundamental component of GLIBC, like the loader. If this code is buggy then the first time the loader is used it may crash the loader (i.e. segmentation violation) which will prevent the make check from running any tests.

Scenario Preconditions

We'll investigate a scenario where this has happened and introduce some methods for debugging the segmentation violation.

Scenario Debugging Options

The following debugging options offer several techniques for debugging this problem. We'll be most interested in the third option.

1. Try to debug the loader: It is probably premature to debug the loader at this point since you don't even know if the code you inserted for memcpy would pass the memcpy testcase. If you know it is necessary to debug the loader at this point then see the above section on `Debugging the Loader'.

2. Statically link a standalone test case against the problematic .o file. This method can detect some bugs but it can't tell you why the loader might be bombing on the code.

3. Statically link a GLIBC test case from a successful build against the problematic .o file. This is the better method but more complicated.

Identifying the Point of Failure

The GLIBC builds in these example were built with cross-compiling=yes therefore the loader isn't run during the build.

When this is the case the GLIBC make stage will complete successfully as long as there are no syntax errors. Generally a fundamental runtime error will not appear until the newly built GLIBC is first used.

The first fragment of GLIBC to be executed is the loader being run in GLIBC's make check.

If cross-compiling=no (which is the default) then the loader will be run during the build and you'll probably get a build failure during the make stage.

It is good form to save make and make check stage output for later grepping, e.g.

> make -j4 2>&1 | tee _make64_1
> make check 2>&1 | tee _check64_1

The place to look for the failure is to start in the _check64_1 output.

There's a good chance that the segmentation fault happened in the loader. Let's check:

> /home/user/glibc/bad-build/glibc64/elf/ld64.so.1
Segmentation fault

There is definitely a problem in the loader most likely due to our version of memcpy being faulty. We'll use the technique in debugging option three to approach the debug.

Extracting the Problematic .o File From the Bad Build

Search through the make output, _make64_1, of the bad build and identify where memcpy.o was generated.

At this point we could simply save off the problematic memcpy.o file for later use. Instead we'll want to use this fragment to create a script so that we can regenerate the problematic memcpy.o at will and have it place the output into a stage directory.

Create a stage directory

> mkdir /home/user/stage/

From the point in _make64_1 where memcpy.S was identified search backward for the term 'Entering'. Each GLIBC compilation fragment uses a local path to the source so you need to know which source directory the shell was in when this particular file was built.

make[2]: Entering directory `/home/user/glibc/bad-build/libc/string'

Copy this and the previously identified compilation fragment for memcpy.o into a file called memcpy.sh in your stage directory. There are a few things you'll need to search and replace to make this useful

Execute this script from the stage directory and you should have a generated file /home/user/stage/memcpy.o:

> cd /home/user/stage/
> sh memcpy.sh

In the future if you need to rework memcpy.S and recompile it into memcpy.o simply:

> vim /home/user/glibc/bad-build/libc/sysdeps/powerpc/powerpc64/cell/memcpy.S
  <make modifications>
> sh /home/user/stage/memcpy.sh

Generate Script To Link Problematic .o File to Clean Build test-memcpy

The next steps will use the clean build's make check log, _check64_1 to find and extract a script fragment for re-linking the clean build's test-memcpy with the problematic memcpy.o so that a new test-memcpy can use the clean build's loader but use the bad memcpy function. This should help us identify where the bug in memcpy is located.

Search for the compilation of the test program test-memcpy.c

Copy test-memcpy.o to your stage directory

> cp /home/user/glibc/good-build/glibc64/string/test-memcpy.o /home/user/stage

Identify the fragment in _check_64_1 which links test-memcpy.o with the rest of the libraries and generated a test-memcpy executable.

Save this fragment into your stage directory as test-memcpy.sh.

At this point we want to turn this dynamically linked example we've saved as test-memcpy.sh into a script which statically links in our problematic test case. We need to do a few things:

This should have created the file /home/user/stage/test-memcpy.

At this point you can verify that test-memcpy is using the loader from the good build:

> readelf -l /home/user/stage/test-memcpy | grep INTERP -A 2
  INTERP         0x0000000000000238 0x0000000010000238 0x0000000010000238
                 0x0000000000000045 0x0000000000000045  R      1
      [Requesting program interpreter: /home/user/glibc/good-build/glibc64/elf/ld64]

If you execute test-memcpy you should now see the segmentation violation:

> ./test-memcpy 
                        simple_memcpy   builtin_memcpy  memcpy
Length    1, alignment  0/ 0:   1       3       1
Length    1, alignment  0/ 0:   1       2       2
Length    1, alignment  0/ 0:   1       3       1
Length    1, alignment  0/ 0:   1       3       1
...
Length  496, alignment 31/ 0:   249     48      47
Length  496, alignment  0/31:   249     51      49
Length  496, alignment 31/31:   249     29      29
Length 4096, alignment  0/ 0:   2052    134     134
Didn't expect signal from child: got `Segmentation fault'

Now run the debugger and make sure to run with the --direct flag. The debugger will pass the --direct flag to the GLIBC test-skeleton framework which takes the flag as direction to run the test directly and not fork the function immediately.

> gdb64 test-memcpy
...
(gdb) run --direct
Starting program: /home/user/stage/test-memcpy --direct
warning: Breakpoint address adjusted from 0x100a1dd0 to 0x10000200.
                        simple_memcpy   builtin_memcpy  memcpy
Length    1, alignment  0/ 0:   2       4       2
Length    1, alignment  0/ 0:   2       4       2
Length    1, alignment  0/ 0:   2       4       2
...
Length 4096, alignment  0/ 0:   2052    135     137

Program received signal SIGSEGV, Segmentation fault.
0x000000001000188c in .memcpy ()
(gdb) bt
#0  0x000000001000188c in .memcpy ()
#1  0x0000000010000ce8 in test_main () at test-memcpy.c:193
#2  0x0000000010001490 in main (argc=2, argv=0xffffff04fb8) at ../test-skeleton.c:274

Identify Problematic Code, Regenerate Problematic mempcy.o and Relink

Now that you're able to debug the problematic code simply make modifications to /home/user/stage/bad-build/libc/sysdeps/powerpc/powerpc64/cell/memcpy.S and re-run memcpy.sh as well as test-memcpy.sh. Then simply re-run test-memcpy; rinse; repeat.

Corner Cases Where Dyanamic Linking the Final Executable Doesn't Work

If you can't get the dynamic-linker to correctly link memcpy.o and test-memcpy.o you may need to statically link the .o files though this is a hackish solution. The -rpath and -dynamic-linker method is preferred.

Note: In the preceding invocation the entire -rpath-link is on one line with no line breaks.

GLIBC Search Order

The GLIBC source and header file search order is saved in the configure output. This is determined first by OS and platform, secondly by depth, third by Implies files, and fourth by Subdirs files.

Compel GCC to Generate Intermediary Output For Examining Macro Expansion

Use -dD -E or --save-temps <expand on this>