• TestingGDB

This page collects useful notes on testing GDB, including running the GDB testsuite.

Caveat emptor

Jim Blandy had the following insight at the gdb-patches@ mailing list about analyzing testsuite results:

One unfortunate characteristic of the GDB testsuite is that, even if GDB is horribly broken, many tests will still pass.  You can't have much confidence that things are basically working with 356 failures. You should look through gdb.log and work through a bunch of them to get a feel for the state of things.

Be aware with tools or weird configurations that redirect logs and corefiles. This may affect some tests results. Rogerio Alves find out that Apport - a tool that intercept program crashes and collect debugging information aka: corefile - may affects some test results like gdb.base/corefile-buildid.exp. GDB testsuite expects that corefile will be generated on the same folder of the binary used on the test, when Apport is enabled, it can result in a FAIL/UNTESTED since Apport collects the corefile generated and corefind function in GDB testsuite is unable to find it:

Running ../binutils-gdb/build/gdb/testsuite/../../../gdb/testsuite/gdb.base/corefile-buildid.exp ...
WARNING: can't generate a core file - core tests suppressed - check ulimit -c
UNTESTED: gdb.base/corefile-buildid.exp: could not generate core file
WARNING: can't generate a core file - core tests suppressed - check ulimit -c
UNTESTED: gdb.base/corefile-buildid.exp: could not generate core file
DUPLICATE: gdb.base/corefile-buildid.exp: could not generate core file

How?

For native gdb testing, just run:

$ make check

in the @gdb/@ build directory to execute all test files. The testsuite supports some parallel execution, which speeds up testing on most modern machines. E.g., run the testsuite with:

$ make check -j8

Running the unit tests

The unit tests are a small subset of the overall GDB test suite, they test individual components of some of GDB's functions. They are not a substitute for the full test suite.

$ make -k run GDBFLAGS='-batch -ex "maintenance selftest"'

Running specific tests

You can run a single test file with (still in the @gdb/@ build directory):

make check TESTS="gdb.base/break.exp"

You can use wildcards to run multiple test files. For example, to run all tests in a test suite directory:

make check TESTS="gdb.base/*.exp"

Making sense of test results

The summary of the test results is recorded in the testsuite/gdb.sum file. The complete transcript of the test run (automated interaction with GDB, and more) is recorded in testsuite/gdb.log.

Unfortunately, the GDB test suite does not pass cleanly out-of-the-box. This makes hard to spot new failures when testing a change. The recommended way to look at test results is to run the testsuite both without and with the change, and compare (using diff or a diff tool) the gdb.sum of the two test runs.

Running GDB under Valgrind in the testsuite

$ cd testsuite
$ runtest --tool gdb GDB="valgrind ../gdb"

or

$ make check RUNTESTFLAGS='GDB=\"valgrind ../gdb\"'

Suitably augment the Valgrind command line as desired. You may want to increase the testsuite's timeout threshold to avoid failures due to GDB running slower. Edit the file <build-dir>/gdb/testsuite/site.exp and add the line:

set gdb_test_timeout 120

RUNTESTFLAGS and parallel (-j) testing

RUNTESTFLAGS disables parallel testing. To re-enable it, set FORCE_PARALLEL as well. E.g.,

make check RUNTESTFLAGS="--target_board=native-gdbserver" FORCE_PARALLEL="1" -j8

That runs the testsuite against a gdbserver running locally on your machine (the testsuite spawns it for you), in parallel mode.

Passing an option to GDB / Running the whole test suite in a non-default mode

Sometimes it is useful to test running GDB in a specific non-default mode or option. For example, you might want to run the whole test suite forcing breakpoints always-inserted on mode. There's usually a CLI command to toggle that mode on, but no GDB specific command line option or specific test suite knob. Instead, we can make use of the "-ex" GDB command line option, which allows running any command when GDB starts, and couple that with the test suite's GDBFLAGS global, that is available/reserved for the user.

From the command line:

make check RUNTESTFLAGS="GDBFLAGS='-ex set\ breakpoint\ always-inserted\ on'"

Or, from a board file:

set GDBFLAGS "-ex \"set break always-inserted on\""

Running GDB with a larger remote serial protocol timeout

If running the test suite against a remote target over a high latency link, you may need to increase the time GDB waits for target responses. The default is 2 seconds. GDB has a "-l" command line option for tweaking that timeout (and an equivalent "set remotetimeout" command too).

Similarly to the use case above, you can use GDBFLAGS to specify a larger remote timeout.

From the command line:

make check RUNTESTFLAGS="GDBFLAGS='-l 9999'"

Or, from a board file:

set GDBFLAGS "-l 9999"

Passing an option to GCC

Sometimes it is useful to pass a specific option to GCC. You can do so like this:

make check RUNTESTFLAGS="CFLAGS_FOR_TARGET='-mthumb' CXXFLAGS_FOR_TARGET='-mthumb' gdb.base/test.exp"

If you want to use specific flags for debug info generation (for example, you want to make the compiler generate DWARF 5), you can pass it like this:

make check RUNTESTFLAGS="--target_board unix/gdb:debug_flags=-gdwarf-5"

Changing the compiler used to build the testcases

make check RUNTESTFLAGS='--target_board "unix/-m32" CC_FOR_TARGET="/path/to/some/other/gcc"'

You can also specify other compilers using parameters such as CXX_FOR_TARGET, F77_FOR_TARGET and AS_FOR_TARGET.

Running massif

If you run massif (the valgrind memory profiler) on GDB, you will quickly see that GDB uses a number of wrapper functions when calling memory allocators. This list of massif options has proven useful to see what is really happening:

--alloc-fn=PyObject_Malloc
--alloc-fn=xmalloc
--alloc-fn=xrealloc
--alloc-fn=xcalloc
--alloc-fn=_objalloc_alloc
--alloc-fn=objalloc_create
--alloc-fn=_obstack_newchunk
--alloc-fn=htab_expand
--alloc-fn=xstrdup
--alloc-fn=xzalloc
--alloc-fn=_obstack_begin
--alloc-fn=bfd_alloc
--alloc-fn=bfd_zalloc

The Awk Regression Index (ARI)

The ARI stands for Awk Regression index and is a script (named gdb_ari.sh) that is run once a day by the machine on which the website is. You can also run it locally. See the ARI page.

Testing gdbserver in a native configuration

You can use the boards/native-gdbserver.exp board file to configure the testsuite to run natively using gdbserver. To do so, follow the instructions provided in introductory comment of that file.

Testing gdbserver in a remote cross-target configuration

To run gdbserver on a remote machine with a different architecture, first you might need a cross-compiler properly configured in your local machine. Also, you need to build gdb and gdbserver for the specific target architecture. In this case, the GDB you're using for tests should be build with the option "--with-sysroot=/path/to/sysroot" pointing to the sysroot directory where the cross-target libraries are located or "--with-sysroot=remote:" (don't forget the colon at the end) if the libraries can be retrieved from the target machine.

Remember to check if gdb was built with expat support (--with-expat option, enabled by default), otherwise it will not be able to communicate properly with gdbserver. Moreover, you might need also dejagnu version 1.5 (or newer), since GDB testsuite cannot run due to a bug existing on dejagnu's previous versions.

To transfer binary files and run gdbserver remotely, there are alternatives such as scp/ssh or rsh/ftp. The example below uses scp/ssh. In this case, it is highly recommendable to use public keys for ssh authentication and ssh session multiplexing (setting up the directives ControlPath and ControlMaster in $HOME/.ssh/config file) in order to speed up the tests.

The remaining steps follow a similar setup from when you are testing gdbserver locally:

1. Create an empty file named site.exp.
2. Create a directory named boards in the same location as site.exp.
3. Create a file named remote-gdbserver.exp in the boards directory (see below).
4. Set the DEJAGNU environment variable to point to the empty site.exp.
5. Run the testsuite with make check-gdb RUNTESTFLAGS="--target_board remote-gdbserver".

Remote Board File

# gdbserver running remotely over ssh

load_generic_config "gdbserver"

process_multilib_options ""

# The default compiler for this target.
#set_board_info compiler  "[find_gcc]"
# If gdbserver runs in a cross-target arch, the testsuite should use a cross-compiler
set_board_info compiler  "/opt/at4.0/bin/powerpc64-linux-gcc"
set_board_info c++compiler  "/opt/at4.0/bin/powerpc64-linux-g++"

set_board_info rsh_prog /usr/bin/ssh
set_board_info rcp_prog /usr/bin/scp
set_board_info protocol standard
set_board_info username user
set_board_info hostname remote.example.com

# Path to the gdbserver executable on target board.
set_board_info gdb_server_prog /home/user/gdbserver-devel/gdbserver

# We will be using the standard GDB remote protocol
set_board_info gdb_protocol "remote"

# Name of the computer whose socket will be used, if required.
set_board_info sockethost "remote.example.com:"

# Port ID to use for socket connection
#set_board_info gdb,socketport "4004"

# Use techniques appropriate to a stub
set_board_info use_gdb_stub 1

# This gdbserver can only run a process once per session.
set_board_info gdb,do_reload_on_run 1

# There's no support for argument-passing (yet).
set_board_info noargs 1

# Can't do input (or output) in the current gdbserver.
set_board_info gdb,noinferiorio 1

# Can't do hardware watchpoints, in general (it depends on gdbserver support for your target arch)
set_board_info gdb,no_hardware_watchpoints 1

Coverage Testing

The simplest way to look at test coverage of gdb is to rebuild it with CFLAGS='-fprofile-arcs -ftest-coverage', then run the test suite. You can visualize the results easily with lcov.

For example:

make mostlyclean
make CFLAGS='-g -fprofile-arcs -ftest-coverage'
make check
lcov -c -d .  > lcov.out
genhtml --legend -o coverage lcov.out  # see also the -p option

Interacting with a test case

It can be useful, when working on a test case, to stop it at a certain point to inspect or change the state of the debugger. This can be done by adding gdb_interact at the desired location. Once interrupted, the script will output the following:

+------------------------------------------+
| Script interrupted, you can now interact |
| with by gdb. Type >>> to continue.       |
+------------------------------------------+

All keyboard input is now forwarded to gdb, so you should be able to use it as usual. To continue the execution of the test case, type >>> (they won't appear on the screen, as they are intercepted by expect). For the same reason, if you type a single >, it won't appear until you input a different character.

Installing GDB testsuite prerequisites

BuildBot#Fedora-specific_instructions