This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH 0/N] test-suite improvement - PASS/FAIL: initial patch
On Sun, 16 Sep 2012 11:46:57 -0400 (EDT), Tomas wrote:
> I have reworked the patch, so now it creates .test-result files to
> each test, where is stores result of single test-case in text form
> and than collects it. What do you think?
If you want a predictable way to find out if a test passed or failed,
then you can simply parse the stderr of the `make -k check` result.
There will only be three types of content in it:
1) Warnings, either from compilation or from the test case itself
2) Compiler errors
3) The Error line signifying which test failed.
AFAICT, the only exceptions to the above are:
1) The abi-check result does not *quite* follow the above rule. It
prints out the error message on stdout directly instead of a .out
file.
2) The various standards tests don't have a direct PASS/FAIL
equivalent.
You can use this information to get the kind of summary you want:
* All 'Error' lines translate to FAIL tests. You can parse that line to
find out which test failed, possibly even printing out the content of
the relevant .out file.
* For abi-check, you need to make a modification to make it follow the
same flow as other tests and then use the same logic for it.
* For the standards check, it may make sense to save expected or
thresholds for the conformance so that a PASS/FAIL can be determined.
I am not very well versed with this part of the tests, so someone
else may need to confirm/deny this.
Regards,
Siddhesh