Differences between revisions 28 and 29
Revision 28 as of 2013-08-28 16:47:29
Size: 12155
Editor: TomTromey
Comment: document making test names not depend on environment
Revision 29 as of 2013-09-17 13:27:11
Size: 12750
Editor: brobecke
Comment:
Deletions are marked like this. Additions are marked like this.
Line 72: Line 72:
Testing a command in GDB should be done using either {{{gdb_test}}}, {{{gdb_test_no_output}}} or {{{gdb_test_multiple}}}. Please do not use {{{gdb_expect}}} as the latter does not automatically handle all known error situations (internal errors, timeouts, etc). Known failing new testcases must produce KFAIL (GDB problem) or XFAIL (environment problem). Testing a command in GDB should be done using either {{{gdb_test}}}, {{{gdb_test_no_output}}}, {{{gdb_test_multiple}}} or {{{gdb_test_sequence}}}. Please do not use {{{gdb_expect}}} as the latter does not automatically handle all known error situations (internal errors, timeouts, etc). Known failing new testcases must produce KFAIL (GDB problem) or XFAIL (environment problem).
Line 100: Line 100:

Below is an example where {{{gdb_test_sequence}}} is used. Note that, at the moment, the function treats the sequence as raw output, not as a list of lines, and thus does not implicitly add new-line markers in between each element.

{{{
    gdb_test_sequence "where" "where in corefile" {
        "\[\r\n\]+#0 .* terminal_func \\(\\) at "
        "\[\r\n\]+#1 .* array_func \\(\\) at "
        "\[\r\n\]+#2 .* factorial_func \\(value=1\\) at "
        "\[\r\n\]+#3 .* factorial_func \\(value=2\\) at "
        "\[\r\n\]+#4 .* main \\(.*\\) at "
    }
}}}

This page collects notes on how to write testcases in the GDB testsuite, traps to avoid, etc.

When adding a feature or fixing a bug which is not covered by the testsuite, you should write a new testcase or extend an existing one.

Building the Example Program

The typical template to start with development of a new testcase can be copied from:

gdb/testsuite/gdb.base/template.exp

# Copyright 2013 Free Software Foundation, Inc.

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

standard_testfile

if { [prepare_for_testing ${testfile}.exp ${testfile} ${srcfile}] } {
    return -1
}

if ![runto_main] {
    untested "could not run to main"
    return -1
}

gdb_test "advance [gdb_get_line_number "next-line"]" "next-line.*" "advance to next-line"

gdb/testsuite/gdb.base/template.c

/* This testcase is part of GDB, the GNU debugger.

   Copyright 2013 Free Software Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3 of the License, or
   (at your option) any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */

int
main (void)
{
  int i = 0;

  return i; /* next-line */
}

See gdb/testsuite/lib/gdb.exp for the list of routines available and their documentation.

Performing a Test

Testing a command in GDB should be done using either gdb_test, gdb_test_no_output, gdb_test_multiple or gdb_test_sequence. Please do not use gdb_expect as the latter does not automatically handle all known error situations (internal errors, timeouts, etc). Known failing new testcases must produce KFAIL (GDB problem) or XFAIL (environment problem).

When matching GDB output in a test, you have to wait for the prompt explicitly. If you ever leave expect in a state where two prompts should be arriving, it won't know which is which; if it sees them separately, you can get out of sync and all tests will fail with unknown output. This means your testcase will have a race condition and may pass or fail intermittently. gdb_test waits for the GDB prompt, gdb_test_multiple doesn't. When using the latter, you have to explicitly put the GDB prompt in your regular expressions.

Also, gdb_test puts an End-of-Line marker at the end of your regular expression.

    gdb_test "break $srcfile:factorial" \
        "Breakpoint.*at.* file .*$srcfile, line.*" \
        "breakpoint function in file"

    gdb_test_no_output "set \$foo=$bp_location11" \
        "set convenience variable \$foo to bp_location11"

    set test "print/x $var"
    gdb_test_multiple "$test" $test {
        -re "\\$\[0-9\]+ = [string_to_regexp $val].*\r\n$gdb_prompt $" {
            pass $test
        }
        -re "\\$\[0-9\]+ = $addr.*\r\n$gdb_prompt $" {
            fail "$test (prints just address)"
        }
        -re "\\$\[0-9\]+ = 0x\[a-f0-9\]+.*\r\n$gdb_prompt $" {
            fail "$test (prints unexpected address)"
        }

Below is an example where gdb_test_sequence is used. Note that, at the moment, the function treats the sequence as raw output, not as a list of lines, and thus does not implicitly add new-line markers in between each element.

    gdb_test_sequence "where" "where in corefile" {
        "\[\r\n\]+#0 .* terminal_func \\(\\) at "
        "\[\r\n\]+#1 .* array_func \\(\\) at "
        "\[\r\n\]+#2 .* factorial_func \\(value=1\\) at "
        "\[\r\n\]+#3 .* factorial_func \\(value=2\\) at "
        "\[\r\n\]+#4 .* main \\(.*\\) at "
    }

If you want to send a command to GDB without increasing the test count, use gdb_test or gdb_test_no_output with an empty message (i.e., pass an empty string as the third argument). Do not use send_gdb since it won't wait for the prompt and thus will make the testcase get out of sync with the GDB output.

Convenient variables defined for use in output matching

Dejagnu defines the following convenient variables (see <dejagnu-prefix>/share/dejagnu/runtest.exp):

  • hex: Matches any hexadecimal number, including the 0x prefix.

  • decimal: Matches a decimal number. It does not handle + or - signs, however, and thus cannot be used to match negative numbers.

The GDB testsuite framework also defines the following convenient variables:

  • octal: Matches an octal number. The number does not need to start with the digit zero.

  • gdb_prompt: Matches the GDB prompt, and the GDB prompt alone. Beware that id does not match the space that usually follows it.

  • inferior_exited_re: Matches the notification printed by GDB when the inferior exited, either normally or not.

  • fullname_syntax: Matches what GDB considers an absolute path. The regexp is not platform-specific, and matches all known forms of absolute paths, including the Windows one, even when testing a Unix debugger on a Unix platform.

  • EXEEXT: The extension used for executables. It looks like this variable is set using the environment variable EXEEXT, or else to the empty string. (FIXME: Is this correct?)

Running the Example Program in GDB

Use either gdb_run_cmd, runto, or runto_main. Do not use gdb_test with either "run" or "start" as the command being sent. This would not work when debugging with gdbserver, for instance. The same applies to gdb_start_cmd.

Note that gdb_run_cmd only sends the command needed to start the execution. It should be followed by a test that verifies the output. For example:

gdb_run_cmd
gdb_test "" ".*Breakpoint.*1.*callee.*14.*" "run"

Similarly to when building the program fails, if you want the testcase to abort, please use return.

Inserting a Breakpoint

It is fine to use gdb_test to insert the breakpoint and match the output in the usual way. In fact, the vast majority of testcase do it this way. However, the gdb.exp library provides a set of routines that can be convenient when a breakpoint needs to be inserted:

  • gdb_get_line_number: This routine allows us to avoid hard-coding the line number where to insert the breakpoint. This is useful when the line number changes because of some adjustment in the code, or even some comment update as we do every year to add the new year in the copyright notices.

  • gdb_breakpoint: This routines simplifies the insertion of the breakpoint and the matching of the output.

gdb_breakpoint ${srcfile}:[gdb_get_line_number "watchpoint-here"]

You should specify the source file, unless you are specifically testing the "break LINE" command. For example, if GDB finds debug information for _start, the default source file may not be what you expect!

Continuing the Execution until Reaching a Breakpoint

It is fine to use gdb_test to continue the execution and match the output in the usual way. However, there is a function provided by gdb.exp which might be useful in simplifying the writing of the test: gdb_continue_to_breakpoint.

gdb_continue_to_breakpoint "Place to set the watchpoint" ".*pattern that should be seen.*"

Implementation note: We should rewrite this routine using gdb_test_multiple.

Restarting GDB

We sometimes need to restart GDB in a testcase. Rather than using the "gdb_exit; gdb_start; gdb_reinitialize_dir; gdb_load" sequence, you can use the clean_restart function.

clean_restart $executable

Dos and Don'ts

Make sure test messages are unique

We sometimes need to test the same operation multiple times in a testcase, so the test results would be the same and duplicated, which should be avoided. After running your new or modified testcase, please check that test messages are unique

$ make check RUNTESTFLAGS="mytest.exp"
$ cat testsuite/gdb.sum | grep "PASS" | sort | uniq -c | sort -n

If you see something like this below in the output, it means messages are not unique. The first column indicates how many times a single line appears, so you want to only see 1 there.

2 PASS: gdb.trace/trace-break.exp: 5 trace ftrace ftrace@0: ftrace after_set_point
2 PASS: gdb.trace/trace-break.exp: 5 trace ftrace trace@1: trace set_point

The with_test_prefix function is one way to address this -- it allows easily prefixing a set of test messages with a unique pass identifier.

Make sure test names are environment-independent

The names of test cases must be environment-independent. That is, a given test should have the same name, regardless of factors such as the host, the target, or the directory in which the test was run. This rule makes it simpler to compare different test runs; and in particular makes it easy to run a baseline test in one directory and a patched test somewhere else.

A typical error here is to use a directory in a test name accidentally, by not specifying a test name explicitly:

gdb_test_no_output "set debug-file-directory $somewhere"

This may yield output like:

PASS: gdb.base/whatever.exp: set debug-file-directory /tmp/gdb/build/testsuite/

In this case the fix is to specify a test name:

gdb_test_no_output "set debug-file-directory $somewhere" "set debug-file-directory"

Make sure test executables are unique

Each test compilation should produce a separate executable.

It is good practice for each test to have its own source code, but if for some reason a new test reuses the sources of another existing test, the new test shall compile to its own executable. E.g., if newtest.exp wants to reuse oldtest.c, then the new test should make sure to not overwrite the old tests' executable. The simplest way to do this is to use standard_testfile, which sets various globals based on the current .exp file's name.

Similarly, if a single .exp test file builds more than one executable, perhaps to exercise several variants of the same test, then each of the compilations should produce a separate executable. This makes it easier to reproduce test problems by hand. Each such executable, or shared library, should include the .exp file's base name; if the file uses standard_testfile then this is in the global testfile.

Avoid "gdb_suppress_entire_file"

Please avoid using gdb_suppress_entire_file. This function is now deprecated. Use untested/return instead.

"untested" calls

In untested calls, please spell out the reason the test ends up untested, instead of just writing the test name, as with the latter we just end up with the test name duplicated in the gdb.sum output. For example:

untested mytest.exp

results in the not very helpful:

UNTESTED: gdb.base/mytest.exp: mytest.exp

A better untested call would look like:

untested "could not compile test program"

which results in the helpful:

UNTESTED: gdb.base/mytest.exp: could not compile test program

There are many bad examples in the testsuite, but we don't want to add more.

Do not refer directly to "$objdir"

Do not refer directly to $objdir. Instead, use the standard_output_file proc to compute a file name from a base name.

Do not write files into "."

Do not write files into . This is not safe when tests are run in parallel. Again, use standard_output_file to control where output is directed.

None: GDBTestcaseCookbook (last edited 2014-10-27 19:32:02 by DougEvans)

All content (C) 2008 Free Software Foundation. For terms of use, redistribution, and modification, please see the WikiLicense page.