This is the mail archive of the mailing list for the glibc project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Do not terminate default test runs on test failure

On Tue, 1 Apr 2014, Andreas Schwab wrote:

> "Joseph S. Myers" <> writes:
> > Use cases that involve identifying a list of failures from a build log are 
> > expected to involve changes to the system you use to list failures - that 
> > is, this is working as designed.
> It is unacceptable that there is no summary just because one of the test
> cases fails to build, since there is no other way to see which test
> cases have failed if the only source of information is the build log.
> This is never a problem with dejagnu, for example.

It is the case with DejaGnu.  The equivalent in DejaGnu is that a problem 
in a .exp file cause an ERROR that terminates the test run early or 
otherwise prevents some tests from being run (ERRORs in DejaGnu can't be 
reliably associated with individual tests).  I've seen that plenty of 

The output of "make" is not a stable interface (the cases in which the 
exit status is nonzero are such an interface).  If you have automation to 
list failures from the output of "make", it needs changing.  If you want 
to use -k (and I discourage using -k with "make check" after my changes) 
and allow build failures, either use stop-on-test-failure and otherwise 
keep your old automation, or combine failures from the make logs with a 
concatentation of all .test-result files from the tree where you ran "make 
-k check".

I don't object to counting the builds of testcases as tests themselves and 
using $(evaluate-test) in the makefile rules (obviously you need to avoid 
conflicts in the PASS/FAIL lines between the entry for the build and the 
entry for the execution), but that would be a further incremental 
improvement to remove more cases for use of -k or external automation to 
identify failures.

Joseph S. Myers

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]