Bug 14233

On Thu, 14 Jun 2012, bugdal at aerifal dot cx wrote:

> Browsing some of the tst-*.c files in glibc, I noticed that many lack any
> comments about what requirement they're testing for. I'm not sure if this is a
> "bug" strictly speaking, but it seems problematic in that:

As with bug 13959, I think Bugzilla is not the right place for this.  
Anything involving "many" without a clear objective way to assess whether 
the source tree has been fixed for an issue is probably better suited to 
policy discussions on libc-alpha and tracking open projects on wiki pages.  
If you create your own project page then you can track for each test 
whether it meets your own standards and other people can independently 
track, if they wish, whether each test meets their standards.

> 1. It's difficult to audit/debug them and determine if they're actually testing
> for what they were intended to test. This is especially problematic for tests
> that involve (either as the subject of the test or as a side-difficulty of
> testing) race conditions, where it may not be at all obvious where the races
> are.

It's quite likely that the function of a test changes over time as the 
test itself changes.  If you wish to propose a policy against changing 
existing tests to test new things - a policy that new tests should be 
added in preference to changing an existing test - then libc-alpha is the 
right place to do so, with the wiki being used to document any resulting 
consensus.

> 2. If a change introduces a bug, it may be hard to determine the cause without
> reverse engineering the test to see what it was testing.

Patches explaining obscure aspects of a test are always welcome, but 
there's no need to repeat in English what the test already says in C or 
what is plain from the documented semantics of the functions, and it's 
inherently subjective what is an appropriate level of commenting for a 
test.

If you want better explanations in new tests, you should do *both* of the 
following: raise the issue on libc-alpha of what appropriate levels of 
explanation are (with examples of specific well-explained and 
badly-explained tests), with a view to documenting consensus on the wiki; 
review testcases in submitted patches and ask for improvements if you 
think they are inadequately explained.  Since it's subjective what's 
enough explanation, only you can review new tests for whether they meet 
your own standards of explanation, although to some extent a standard may 
be documented and illustrated by examples.

> 3. It would be nice to know if tests are asserting behavior required by the
> standards, behavior required as part of the documented or unofficial
> specification of glibc, undocumented behavior that known applications expect,
> and/or quality of implementation issues.

Again, this is open-ended and in general a test is simply helping to 
ensure that unexpected changes to glibc's behavior are detected.

As discussed on libc-alpha and now documented at <http://sourceware.org/glibc/wiki/Bugzilla%20Procedures>, please take general policy questions and open-ended projects to libc-alpha and the wiki.

If there is an objective problem with a test - for example, it has a race condition that may cause it to fail sometimes - then a bug in Bugzilla for the specific issue with the specific test is appropriate.  If in doubt about whether the testsuite should be ensuring some particular behavior beyond the relevant standards (where not documented in the glibc manual) then libc-alpha will be the right place to raise such a question about what the glibc API should be (likely resulting in a change to either the manual or the test, or an agreement about which is wrong which could then result in a Bugzilla bug pointing to that agreement).