This is the mail archive of the
mailing list for the glibc project.
Re: [PATCH 0/N] test-suite improvement - PASS/FAIL: initial patch
On Mon, 17 Sep 2012, Siddhesh Poyarekar wrote:
> If you want a predictable way to find out if a test passed or failed,
> then you can simply parse the stderr of the `make -k check` result.
> There will only be three types of content in it:
> 1) Warnings, either from compilation or from the test case itself
> 2) Compiler errors
> 3) The Error line signifying which test failed.
This doesn't readily give a list of passing tests, though you can get that
by using "make --debug=b -k check". But really the point is so that not
everyone needs to reinvent the wheel in such ways - so that you can simply
use standard glibc testsuite machinery to give a list of test results.
Note that eventually you don't just want PASS and FAIL; you should have
other statuses such as UNRESOLVED and UNSUPPORTED:
* UNRESOLVED: a test detected a condition (such as an out-of-memory error)
meaning it couldn't test what it was meant to test. Also, ideally you'd
get PASS or FAIL for compiling/linking a test as well as running it, and
if those failed then the run of the test would be UNRESOLVED.
* UNSUPPORTED: the test depends on some feature not supported in this
configuration of glibc or the system on which the test was run. (Some
tests aren't built at all in particular configurations; it would be best
to show those as UNSUPPORTED. Some may just exit early if something isn't
supported, e.g. tst-setcontext if getcontext returns with an ENOSYS
Actually implementing those may require reviewing how all the existing
tests use exit statuses so that suitable statuses can be reserved to
signal UNRESOLVED / UNSUPPORTED.
> * For the standards check, it may make sense to save expected or
> thresholds for the conformance so that a PASS/FAIL can be determined.
> I am not very well versed with this part of the tests, so someone
> else may need to confirm/deny this.
See what I said in
splitting up this into smaller tests run by separate makefile commands, so
you can get finer-grained PASS/FAIL information, maybe at the level of
(standard, header) pairs as well as allowing more parallelism.
(Splitting up tests with lots of separate assertions is generally a good
thing to help finer-grained tracking of what passes and fails in a given
Regarding expected failure states, one might hope they could be
represented in some way that has *both* the following effects:
* "make check" (no -k) doesn't stop on a failure of the test.
* A failure appears as XFAIL rather than FAIL in the summary, and a pass
as XPASS rather than PASS.
The more of these extra features you want, the harder it is to get them
simply by parsing existing "make" output without new infrastructure such
as that proposed in this thread.
Joseph S. Myers