[PATCH 0/N] test-suite improvement - PASS/FAIL: initial patch

[Joseph S. Myers]

On Mon, 17 Sep 2012, Tomas Dohnalek wrote:

> Ok, but then the behaviour of running simple `make check' (no -k) will be
> changed, is that ok? I guess that it is probably desired to run the whole
> test-suite even when error occurs, but maybe someone has some objections?

Good point - I suppose you need to check MAKEFLAGS in the test rule and 
emulate -k handling there, rather than just unconditionally ignoring 
errors from $(MAKE) tests, so that errors from the inner make are properly 
ignored or not depending on whether -k is passed.

Ideally I think the whole testsuite should run in any case (meaning 
passing down -k whether or not the user specified it) and "make check" 
should exit with a nonzero status if any test had an *unexpected* failure 
(including failure of any of the commands involved in building an test, or 
the various separate makefile rules that do bits of setup for running 
tests) - printing on stdout/stderr a summary of the unexpected failures.  
But that would be for a later stage, since you'd need to develop a way to 
track results of test compilation / setup as well as the tests themselves, 
and a way for expectedness of failures to be more directly visible to the 
test harness.  (As is, expected failures simply use "-" in their makefile 
rules directly, e.g. in posix/Makefile for $(objpfx)annexc.out.)

-- 
Joseph S. Myers
joseph@codesourcery.com