[2.20] [4/6] Enumerate tests with special rules in tests-special variable

[Joseph S. Myers]

On Mon, 10 Feb 2014, Brooks Moses wrote:

> How would this interact with the idea you mentioned in an earlier patch in the
> series of breaking up tests that have an "examine the output" step into
> multiple tests?  Do we end up having multiple lines in the enumeration for
> them?

Yes.  (Sometimes the initial output generation might not need any special 
makefile rules at all.)

> > * Should tests that don't generate output files be changed (in a
> >    separate patch or patches) to do so?
> 
> What would the output files contain?

Whatever, possibly empty, goes on standard output at present.  For 
example, for the check-abi tests, the diffs compared to the ABI baselines 
(nonempty only if the test fails).  That would have the advantage that 
"make check", in a directory where some tests have already been run, 
doesn't rerun those tests as it does at present, unless the dependencies 
indicate they need to be rerun.

> If there are compile errors, do they go in the output file, or...?

Compile errors aren't handled by this series; they'd continue to cause 
"make check" to fail, without being recorded in the PASS / FAIL results 
(but given patch 6, failure of test execution would not halt "make 
check").

> If there are runtime errors like a segfault, will they get captured in the
> output file?  If so, I think that's a strong argument for always having an
> output file.

Segfaults will get reported by the shell used by make, so not captured.  
The present practice is that only stdout not stderr gets redirected (this 
is questionable, as is tests using both anyway without a clear 
distinction) - but the reporting of a segfault happens after the 
segfauling process has exited, meaning output redirection for that process 
has no effect on the report of the segfault.

-- 
Joseph S. Myers
joseph@codesourcery.com