This is the mail archive of the
gdb-patches@sourceware.org
mailing list for the GDB project.
Re: How to abort a test?
- From: Pedro Alves <palves at redhat dot com>
- To: Simon Marchi <simon dot marchi at ericsson dot com>, dejagnu at gnu dot org, gdb-patches <gdb-patches at sourceware dot org>
- Date: Thu, 14 Jan 2016 17:28:26 +0000
- Subject: Re: How to abort a test?
- Authentication-results: sourceware.org; auth=none
- References: <56958359 dot 8070708 at ericsson dot com> <5697CC09 dot 8010306 at redhat dot com> <5697D721 dot 1000305 at ericsson dot com>
On 01/14/2016 05:13 PM, Simon Marchi wrote:
> On 16-01-14 11:25 AM, Pedro Alves wrote:
>>> The test will be aborted, runtest will output a detailed error, but the test will still
>>> pass. Intuitively, I would think that a test that throws an error should automatically
>>> be failed or unresolved, since something unexpected happened.
>>
>> IIUC, you wouldn't want pass/fail to even be reached, so I don't understand
>> what test would you want to fail.
>
> I think unresolved would be appropriate in that case, since it's not the program under test
> failed, but the test, or its setup/environment.
OK. I'd say "testcase" to disambiguate with a particular pass/fail invocation.
>
> According to the Dejagnu doc, unresolved: "... usually means the test did not execute as
> expected, and a human being must go over results to determine if it passed or failed (and
> to improve the test case). ". I think that applies here.
>
> So when there is an uncaught exception coming from the test, Dejagnu could issue a:
>
> unresolved "Uncaught error from test " # needs a better message
>
> The important part is that the runtest command returns a failure return code, so that
> automated builds or scripts consider the run as failed. To me, a test that ends with
> an exception is not a test that ran successfully.
>
>>>
>>> The only option I see right now would be to fix the whole return chain and add proper
>>> error handling everywhere, to exit early when an error happens. However, that means
>>> changing tens (hundreds?) of callsites through the testsuite, which is why I'm
>>> looking for alternative solutions first.
>>
>> Test usually return early if running to main fails (if ![runto_main]), so maybe it
>> won't be that many places. Maybe some judiciously placed "return -code return"
>> hacks saves you a good chunk. I don't have any other idea.
>
> Many tests use gdb_run_cmd directly without checking the result:
>
> gdb/testsuite $ grep -rin '^gdb_run_cmd$' gdb.* | wc -l
> 73
>
> So they would need to be changed:
>
> -gdb_run_cmd
> +if ![gdb_run_cmd] {
> + fail "Failed to run"
> +}
>
That's where the "return -code return" hack would come in. You'd
do that inside gdb_run_cmd so that would return directly to gdb_run_cmd's
caller. But it's an ugly hack.
>> For the particular case of gdbserver not being present in the target,
>> probably the easiest would be to check that earlier, likely before
>> runtest, even. Not ideal, since the testsuite can mix native and gdbserver
>> tests, for instance, but...
>
> And it's a bit hard to check in the case of a remote target, given that it's runtest
> that knows how to "compute" the remote hostname, username, expected gdbserver path,
> etc, from the --target_board.
I meant, before runtest, the dejagnu procedure, not before invoking the
runtest binary. E.g., straight from the target board file, say.
E.g., with:
--- c/gdb/testsuite/boards/gdbserver-base.exp
+++ w/gdb/testsuite/boards/gdbserver-base.exp
@@ -33,6 +33,8 @@ set_board_info gdb,predefined_tsv "\\\$trace_timestamp"
set GDBFLAGS "${GDBFLAGS} -ex \"set auto-connect-native-target off\""
+error "gdbserver not present on target!"
+
proc ${board}_file { dest op args } {
if { $op == "delete" } {
return 0
then I get:
$ time make check RUNTESTFLAGS="--target_board=native-gdbserver"
make[1]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
Nothing to be done for all...
make check-single
make[2]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
rootme=`pwd`; export rootme; srcdir=/home/pedro/gdb/mygit/src/gdb/testsuite ; export srcdir ; EXPECT=`if [ "${READ1}" != "" ] ; then echo ${rootme}/expect-read1; elif [ -f ${rootme}/../../expect/expect ] ; then echo ${rootme}/../../expect/expect ; else echo expect ; fi` ; export EXPECT ; EXEEXT= ; export EXEEXT ; LD_LIBRARY_PATH=$rootme/../../expect:$rootme/../../libstdc++:$rootme/../../tk/unix:$rootme/../../tcl/unix:$rootme/../../bfd:$rootme/../../opcodes:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; if [ -f ${rootme}/../../expect/expect ] ; then TCL_LIBRARY=${srcdir}/../../tcl/library ; export TCL_LIBRARY ; fi ; runtest --target_board=native-gdbserver
Test Run By pedro on Thu Jan 14 17:24:37 2016
Native configuration is x86_64-pc-linux-gnu
=== gdb tests ===
Schedule of variations:
native-gdbserver
Running target native-gdbserver
Using /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/native-gdbserver.exp as board description file for target.
Using /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp as board description file for target.
ERROR: tcl error sourcing board description file for target /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp.
gdbserver not present on target!
gdbserver not present on target!
while executing
"error "gdbserver not present on target!""
(file "/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp" line 36)
invoked from within
"source /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp"
("uplevel" body line 1)
invoked from within
"uplevel #0 source /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp"
invoked from within
"catch "uplevel #0 source ${dir}/${initfile}" error"
make[2]: *** [check-single] Error 1
make[2]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
make: *** [check] Error 2
real 0m0.404s
user 0m0.336s
sys 0m0.060s
Obviously not ideal; I'm just pointing out the likely easiest.
Thanks,
Pedro Alves