This page collects useful notes on testing GDB, including running the GDB testsuite.
- Caveat emptor
- Running specific tests
- Making sense of test results
- Running GDB under Valgrind in the testsuite
- RUNTESTFLAGS and parallel (-j) testing
- Passing an option to GDB / Running the whole test suite in a non-default mode
- Running GDB with a larger remote serial protocol timeout
- Passing an option to GCC
- Changing the compiler used to build the testcases
- Running massif
- The Awk Regression Index (ARI)
- Testing gdbserver in a native configuration
- Testing gdbserver in a remote cross-target configuration
- Coverage Testing
- Interacting with a test case
- Installing GDB testsuite prerequisites
Jim Blandy had the following insight at the gdb-patches@ mailing list about analyzing testsuite results:
One unfortunate characteristic of the GDB testsuite is that, even if GDB is horribly broken, many tests will still pass. You can't have much confidence that things are basically working with 356 failures. You should look through gdb.log and work through a bunch of them to get a feel for the state of things.
Be aware with tools or weird configurations that redirect logs and corefiles. This may affect some tests results. Rogerio Alves find out that Apport - a tool that intercept program crashes and collect debugging information aka: corefile - may affects some test results like gdb.base/corefile-buildid.exp. GDB testsuite expects that corefile will be generated on the same folder of the binary used on the test, when Apport is enabled, it can result in a FAIL/UNTESTED since Apport collects the corefile generated and corefind function in GDB testsuite is unable to find it:
Running ../binutils-gdb/build/gdb/testsuite/../../../gdb/testsuite/gdb.base/corefile-buildid.exp ... WARNING: can't generate a core file - core tests suppressed - check ulimit -c UNTESTED: gdb.base/corefile-buildid.exp: could not generate core file WARNING: can't generate a core file - core tests suppressed - check ulimit -c UNTESTED: gdb.base/corefile-buildid.exp: could not generate core file DUPLICATE: gdb.base/corefile-buildid.exp: could not generate core file
For native gdb testing, just run:
$ make check
in the gdb/ build directory to execute all test files. The testsuite supports some parallel execution, which speeds up testing on most modern machines. E.g., run the testsuite with:
$ make check -j8
Running specific tests
You can run a single test file with (replace XXX and testfilename with the actual names):
make check TESTS="gdb.XXX/testfilename.exp"
You can use wildcards to run multiple test files. For example, to run all tests in a test suite directory:
make check TESTS="gdb.XXX/*.exp"
Making sense of test results
The summary of the test results is recorded in the testsuite/gdb.sum file. The complete transcript of the test run (automated interaction with GDB, and more) is recorded in testsuite/gdb.log.
Unfortunately, the GDB test suite does not pass cleanly out-of-the-box. This makes hard to spot new failures when testing a change. The recommended way to look at test results is to run the testsuite both without and with the change, and compare (using diff or a diff tool) the gdb.sum of the two test runs.
Running GDB under Valgrind in the testsuite
$ cd testsuite $ runtest --tool gdb GDB="valgrind ../gdb"
$ make check RUNTESTFLAGS='GDB=\"valgrind ../gdb\"'
Suitably augment the Valgrind command line as desired. You may want to increase the testsuite's timeout threshold to avoid failures due to GDB running slower. Edit the file <build-dir>/gdb/testsuite/site.exp and add the line:
set gdb_test_timeout 120
RUNTESTFLAGS and parallel (-j) testing
RUNTESTFLAGS disables parallel testing. To re-enable it, set FORCE_PARALLEL as well. E.g.,
make check RUNTESTFLAGS="--target_board=native-gdbserver" FORCE_PARALLEL="1" -j8
That runs the testsuite against a gdbserver running locally on your machine (the testsuite spawns it for you), in parallel mode.
Passing an option to GDB / Running the whole test suite in a non-default mode
Sometimes it is useful to test running GDB in a specific non-default mode or option. For example, you might want to run the whole test suite forcing breakpoints always-inserted on mode. There's usually a CLI command to toggle that mode on, but no GDB specific command line option or specific test suite knob. Instead, we can make use of the "-ex" GDB command line option, which allows running any command when GDB starts, and couple that with the test suite's GDBFLAGS global, that is available/reserved for the user.
From the command line:
make check RUNTESTFLAGS="GDBFLAGS='-ex set\ breakpoint\ always-inserted\ on'"
Or, from a board file:
set GDBFLAGS "-ex \"set break always-inserted on\""
Running GDB with a larger remote serial protocol timeout
If running the test suite against a remote target over a high latency link, you may need to increase the time GDB waits for target responses. The default is 2 seconds. GDB has a "-l" command line option for tweaking that timeout (and an equivalent "set remotetimeout" command too).
Similarly to the use case above, you can use GDBFLAGS to specify a larger remote timeout.
From the command line:
make check RUNTESTFLAGS="GDBFLAGS='-l 9999'"
Or, from a board file:
set GDBFLAGS "-l 9999"
Passing an option to GCC
Sometimes it is useful to pass a specific option to GCC. You can do so like this:
make check RUNTESTFLAGS="CFLAGS_FOR_TARGET='-mthumb' CXXFLAGS_FOR_TARGET='-mthumb' gdb.base/test.exp"
If you want to use specific flags for debug info generation (for example, you want to make the compiler generate DWARF 5), you can pass it like this:
make check RUNTESTFLAGS="--target_board unix/gdb:debug_flags=-gdwarf-5"
Changing the compiler used to build the testcases
make check RUNTESTFLAGS='--target_board "unix/-m32" CC_FOR_TARGET="/path/to/some/other/gcc"'
You can also specify other compilers using parameters such as CXX_FOR_TARGET, F77_FOR_TARGET and AS_FOR_TARGET.
If you run massif (the valgrind memory profiler) on GDB, you will quickly see that GDB uses a number of wrapper functions when calling memory allocators. This list of massif options has proven useful to see what is really happening:
--alloc-fn=PyObject_Malloc --alloc-fn=xmalloc --alloc-fn=xrealloc --alloc-fn=xcalloc --alloc-fn=_objalloc_alloc --alloc-fn=objalloc_create --alloc-fn=_obstack_newchunk --alloc-fn=htab_expand --alloc-fn=xstrdup --alloc-fn=xzalloc --alloc-fn=_obstack_begin --alloc-fn=bfd_alloc --alloc-fn=bfd_zalloc
The Awk Regression Index (ARI)
The ARI stands for Awk Regression index and is a script (named gdb_ari.sh) that is run once a day by the machine on which the website is. You can also run it locally. See the ARI page.
Testing gdbserver in a native configuration
You can use the boards/native-gdbserver.exp board file to configure the testsuite to run natively using gdbserver. To do so, follow the instructions provided in introductory comment of that file.
Testing gdbserver in a remote cross-target configuration
To run gdbserver on a remote machine with a different architecture, first you might need a cross-compiler properly configured in your local machine. Also, you need to build gdb and gdbserver for the specific target architecture. In this case, the GDB you're using for tests should be build with the option "--with-sysroot=/path/to/sysroot" pointing to the sysroot directory where the cross-target libraries are located or "--with-sysroot=remote:" (don't forget the colon at the end) if the libraries can be retrieved from the target machine.
Remember to check if gdb was built with expat support (--with-expat option, enabled by default), otherwise it will not be able to communicate properly with gdbserver. Moreover, you might need also dejagnu version 1.5 (or newer), since GDB testsuite cannot run due to a bug existing on dejagnu's previous versions.
To transfer binary files and run gdbserver remotely, there are alternatives such as scp/ssh or rsh/ftp. The example below uses scp/ssh. In this case, it is highly recommendable to use public keys for ssh authentication and ssh session multiplexing (setting up the directives ControlPath and ControlMaster in $HOME/.ssh/config file) in order to speed up the tests.
The remaining steps follow a similar setup from when you are testing gdbserver locally:
- Create an empty file named site.exp.
- Create a directory named boards in the same location as site.exp.
- Create a file named remote-gdbserver.exp in the boards directory (see below).
- Set the DEJAGNU environment variable to point to the empty site.exp.
- Run the testsuite with make check-gdb RUNTESTFLAGS="--target_board remote-gdbserver".
Remote Board File
# gdbserver running remotely over ssh load_generic_config "gdbserver" process_multilib_options "" # The default compiler for this target. #set_board_info compiler "[find_gcc]" # If gdbserver runs in a cross-target arch, the testsuite should use a cross-compiler set_board_info compiler "/opt/at4.0/bin/powerpc64-linux-gcc" set_board_info c++compiler "/opt/at4.0/bin/powerpc64-linux-g++" set_board_info rsh_prog /usr/bin/ssh set_board_info rcp_prog /usr/bin/scp set_board_info protocol standard set_board_info username user set_board_info hostname remote.example.com # Path to the gdbserver executable on target board. set_board_info gdb_server_prog /home/user/gdbserver-devel/gdbserver # We will be using the standard GDB remote protocol set_board_info gdb_protocol "remote" # Name of the computer whose socket will be used, if required. set_board_info sockethost "remote.example.com:" # Port ID to use for socket connection #set_board_info gdb,socketport "4004" # Use techniques appropriate to a stub set_board_info use_gdb_stub 1 # This gdbserver can only run a process once per session. set_board_info gdb,do_reload_on_run 1 # There's no support for argument-passing (yet). set_board_info noargs 1 # Can't do input (or output) in the current gdbserver. set_board_info gdb,noinferiorio 1 # Can't do hardware watchpoints, in general (it depends on gdbserver support for your target arch) set_board_info gdb,no_hardware_watchpoints 1
The simplest way to look at test coverage of gdb is to rebuild it with CFLAGS='-fprofile-arcs -ftest-coverage', then run the test suite. You can visualize the results easily with lcov.
make mostlyclean make CFLAGS='-g -fprofile-arcs -ftest-coverage' make check lcov -c -d . > lcov.out genhtml --legend -o coverage lcov.out # see also the -p option
Interacting with a test case
It can be useful, when working on a test case, to stop it at a certain point to inspect or change the state of the debugger. This can be done by adding gdb_interact at the desired location. Once interrupted, the script will output the following:
+------------------------------------------+ | Script interrupted, you can now interact | | with by gdb. Type >>> to continue. | +------------------------------------------+
All keyboard input is now forwarded to gdb, so you should be able to use it as usual. To continue the execution of the test case, type >>> (they won't appear on the screen, as they are intercepted by expect). For the same reason, if you type a single >, it won't appear until you input a different character.