Re: [PATCH 0/N] test-suite improvement - PASS/FAIL: initial patch

["Joseph S. Myers" <joseph at codesourcery dot com>]

On Mon, 17 Sep 2012, Siddhesh Poyarekar wrote:

> If you want a predictable way to find out if a test passed or failed,
> then you can simply parse the stderr of the `make -k check` result.
> There will only be three types of content in it:
> 
> 1) Warnings, either from compilation or from the test case itself
> 2) Compiler errors
> 3) The Error line signifying which test failed.

This doesn't readily give a list of passing tests, though you can get that 
by using "make --debug=b -k check".  But really the point is so that not 
everyone needs to reinvent the wheel in such ways - so that you can simply 
use standard glibc testsuite machinery to give a list of test results.

Note that eventually you don't just want PASS and FAIL; you should have 
other statuses such as UNRESOLVED and UNSUPPORTED:

* UNRESOLVED: a test detected a condition (such as an out-of-memory error) 
meaning it couldn't test what it was meant to test.  Also, ideally you'd 
get PASS or FAIL for compiling/linking a test as well as running it, and 
if those failed then the run of the test would be UNRESOLVED.

* UNSUPPORTED: the test depends on some feature not supported in this 
configuration of glibc or the system on which the test was run.  (Some 
tests aren't built at all in particular configurations; it would be best 
to show those as UNSUPPORTED.  Some may just exit early if something isn't 
supported, e.g. tst-setcontext if getcontext returns with an ENOSYS 
error.)

Actually implementing those may require reviewing how all the existing 
tests use exit statuses so that suitable statuses can be reserved to 
signal UNRESOLVED / UNSUPPORTED.

> * For the standards check, it may make sense to save expected or
>   thresholds for the conformance so that a PASS/FAIL can be determined.
>   I am not very well versed with this part of the tests, so someone
>   else may need to confirm/deny this.

See what I said in 
<http://sourceware.org/ml/libc-alpha/2012-09/msg00299.html> about 
splitting up this into smaller tests run by separate makefile commands, so 
you can get finer-grained PASS/FAIL information, maybe at the level of 
(standard, header) pairs as well as allowing more parallelism.  
(Splitting up tests with lots of separate assertions is generally a good 
thing to help finer-grained tracking of what passes and fails in a given 
configuration.)

Regarding expected failure states, one might hope they could be 
represented in some way that has *both* the following effects:

* "make check" (no -k) doesn't stop on a failure of the test.

* A failure appears as XFAIL rather than FAIL in the summary, and a pass 
as XPASS rather than PASS.


The more of these extra features you want, the harder it is to get them 
simply by parsing existing "make" output without new infrastructure such 
as that proposed in this thread.

-- 
Joseph S. Myers
joseph@codesourcery.com