Re: [PATCH] gdb/testsuite: Don't attempt tests if they fail to compile

[Andrew Burgess <andrew dot burgess at embecosm dot com>]

* Simon Marchi <simon.marchi@ericsson.com> [2018-01-11 17:03:51 -0500]:

> On 2018-01-11 02:00 PM, Andrew Burgess wrote:
> > In the gdb.base/whatis-ptype-typedefs.exp test, if the test program
> > fails to compile, don't run the tests.
> 
> Hi Andrew,
> 
> That would make the test similar to other test, in that if we fail to
> build the test program it's not a failure (it shows as UNTESTED, doesn't
> make the test run fail).  I find it's a strange behavior though.  If a
> test program starts not building for some reason, I'd certainly like to
> know (e.g. it could be UNRESOLVED), instead of it silently failing.
> 
> Any other opinion?

If the test fails to compile we don't get a silent failure, as you
mention, we get the UNTESTED.  Changing this to something stronger,
like UNRESOLVED, would I fear make cases where we legitimately can't
compile a test program seem worse than they really are.

The concern about missing the case where a test program goes from
compiling to not compiling is fair, however, I don't think that it's
something we need to worry about.  My understanding of the "normal"
testing flow for GDB is to compare against a baseline set of results,
a few hundred tests disappearing should raise a red flag, and once the
developer has realised that this particular test script has something
weird going on, the extra UNTESTED should guide them to the cause of
the problem.

The failed to prepare leading to skipping the tests seems like the
"standard" pattern within the GDB testsuite, so, if you agree, I think
having this test fall in line with that is probably a good thing.
That doesn't mean we can't change the standard pattern in the future
if we can come up with a better model (though I don't have any good
suggestions).

Thanks,
Andrew