[2.20] [3/6] Support expected failures in .test-result files

[Joseph S. Myers]

On Fri, 14 Feb 2014, Carlos O'Donell wrote:

> I agree with that. However, my worry is that this will lead to an overly
> complex meta-language to describe when XFAILs are expected. If instead a

The meta-language is GNU make, controlling makefile variable settings 
(together with arbitrary C code when it's actually the test itself 
disabling something).  It may not be ideal, but it's what we have at 
present to control lots of things about what tests are run and how they 
are run.

> It's the "whatever the relevant conditions are" that worries me. The result
> of such language naturally leads me to believe we'll have lots and lots of
> XFAILs with complex conditions because we support many versions of gcc,
> many versions of the kernel, and other related tools. Worse the version
> numbers don't mean the same to all distributions which apply custom patches.

In the kernel case, it will probably be conditionals within the tests, 
like the tst-setgetname.c case you dealt with.  We also use those for 
other cases, e.g. math-tests.h.

Ideally tests would be able to give more fine-grained information in such 
cases - as well as an overall result from the test (a dummy PASS from the 
exit status here) there would be results from individual assertions within 
the test (all UNRESOLVED because the old kernel prevents them from being 
tested).  But in the absence of that, we can still incrementally stop 
tests giving a spurious FAIL when actually they are indicating an issue 
external to glibc, or some other known issue.  (And incrementally making 
more tests use test-skeleton.c would likely help with any future move to 
fine-grained test assertion status, in that test-skeleton.c is the natural 
place for functions to say "record this assertion as having this result".)

> In this case might we remove old XFAILS if say the kernel version condition
> they depended upon is no longer supported?

Yes.

-- 
Joseph S. Myers
joseph@codesourcery.com