Browsing some of the tst-*.c files in glibc, I noticed that many lack any comments about what requirement they're testing for. I'm not sure if this is a "bug" strictly speaking, but it seems problematic in that: 1. It's difficult to audit/debug them and determine if they're actually testing for what they were intended to test. This is especially problematic for tests that involve (either as the subject of the test or as a side-difficulty of testing) race conditions, where it may not be at all obvious where the races are. 2. If a change introduces a bug, it may be hard to determine the cause without reverse engineering the test to see what it was testing. 3. It would be nice to know if tests are asserting behavior required by the standards, behavior required as part of the documented or unofficial specification of glibc, undocumented behavior that known applications expect, and/or quality of implementation issues.
On Thu, 14 Jun 2012, bugdal at aerifal dot cx wrote: > Browsing some of the tst-*.c files in glibc, I noticed that many lack any > comments about what requirement they're testing for. I'm not sure if this is a > "bug" strictly speaking, but it seems problematic in that: As with bug 13959, I think Bugzilla is not the right place for this. Anything involving "many" without a clear objective way to assess whether the source tree has been fixed for an issue is probably better suited to policy discussions on libc-alpha and tracking open projects on wiki pages. If you create your own project page then you can track for each test whether it meets your own standards and other people can independently track, if they wish, whether each test meets their standards. > 1. It's difficult to audit/debug them and determine if they're actually testing > for what they were intended to test. This is especially problematic for tests > that involve (either as the subject of the test or as a side-difficulty of > testing) race conditions, where it may not be at all obvious where the races > are. It's quite likely that the function of a test changes over time as the test itself changes. If you wish to propose a policy against changing existing tests to test new things - a policy that new tests should be added in preference to changing an existing test - then libc-alpha is the right place to do so, with the wiki being used to document any resulting consensus. > 2. If a change introduces a bug, it may be hard to determine the cause without > reverse engineering the test to see what it was testing. Patches explaining obscure aspects of a test are always welcome, but there's no need to repeat in English what the test already says in C or what is plain from the documented semantics of the functions, and it's inherently subjective what is an appropriate level of commenting for a test. If you want better explanations in new tests, you should do *both* of the following: raise the issue on libc-alpha of what appropriate levels of explanation are (with examples of specific well-explained and badly-explained tests), with a view to documenting consensus on the wiki; review testcases in submitted patches and ask for improvements if you think they are inadequately explained. Since it's subjective what's enough explanation, only you can review new tests for whether they meet your own standards of explanation, although to some extent a standard may be documented and illustrated by examples. > 3. It would be nice to know if tests are asserting behavior required by the > standards, behavior required as part of the documented or unofficial > specification of glibc, undocumented behavior that known applications expect, > and/or quality of implementation issues. Again, this is open-ended and in general a test is simply helping to ensure that unexpected changes to glibc's behavior are detected.
As discussed on libc-alpha and now documented at <http://sourceware.org/glibc/wiki/Bugzilla%20Procedures>, please take general policy questions and open-ended projects to libc-alpha and the wiki. If there is an objective problem with a test - for example, it has a race condition that may cause it to fail sometimes - then a bug in Bugzilla for the specific issue with the specific test is appropriate. If in doubt about whether the testsuite should be ensuring some particular behavior beyond the relevant standards (where not documented in the glibc manual) then libc-alpha will be the right place to raise such a question about what the glibc API should be (likely resulting in a change to either the manual or the test, or an agreement about which is wrong which could then result in a Bugzilla bug pointing to that agreement).