This is the mail archive of the mailing list for the glibc project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [PATCH roland/test-unsupported] Let tests result in UNSUPPORTED; use that for unbuildable C++ cases

On Fri, 6 Mar 2015, Roland McGrath wrote:

> now follows the Autotest convention of exit code 77
> meaning a test that cannot be built or cannot be usefully run,

I wonder if any existing tests return a count of errors (incrementing a 
variable on error rather than setting it to 1, then returning that 
variable) and could potentially return 77 by accident.  But such tests 
would already be broken if they returned 256, so any issues can be 
considered pre-existing.

> We have many existing cases of tests that do compile but are turned into
> no-ops that always pass either at compile time or at run time.  Once this
> lands, we should start converting those such that they report UNSUPPORTED
> rather that PASS.  But that is a later step.

And if cross-compiling without test-wrapper (so run-built-tests = no), in 
principle all affected tests should be UNSUPPORTED.  And at some point 
there's the issue of allowing tests to report status for their individual 
assertions (so a test might have some assertions that pass and some that 
are unsupported).

> Notably, I've changed the summarizing stuff so that it prints out all
> results that are neither PASS nor XFAIL, so that catches all ERROR and
> XPASS as well as the new UNSUPPORTED.  Similarly, I've changed the logic
> that decides whether 'make check' as a whole succeeded or not so that it
> rejects anything that is not PASS, XFAIL, or UNSUPPORTED, rather than
> rejecting only things that are exactly FAIL or ERROR.  This means that any
> existing XPASS cases now cause 'make check' to fail.  I think that's the
> right thing, since XPASS means something funny is going on and either the
> test needs to be changed or the XFAIL expectation needs to be dropped.  I
> also think it's definitely the right thing to both display and barf on any
> statuses that are not in the expected set.

I don't think XPASSes should cause "make check" to fail - or at least, 
that's contrary to how we use XFAILs at present.  XPASSes appear right now 
for cases where the expected failure depends on conditions there is 
otherwise no need to test for in configure or makefiles.  In conform/, 
that includes (a) tests requiring GCC 4.7 or later, (b) tests that may 
depend on whether some non-glibc package has installed ndbm.h, (c) tests 
that are expected to fail on at least one of x86_64 and x86 but where the 
failure is for an architecture-specific (library or header) issue and so 
the tests pass on some other platforms.  Only (c) is a case where the 
XFAILs could simply be moved to sysdeps if desired and the relevant 
sysdeps condition is worked out.  It's not clear to me that it's useful to 
condition these XFAILs more precisely.  And XFAILs might well be useful 
for a few nptl/ and rt/ tests that are known to fail intermittently, 
possibly because of Linux kernel issues (where such a test issue is known, 
it seems better to me for the test to be variably XPASS or XFAIL, than 
variably PASS or FAIL).

Joseph S. Myers

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]