Re: RFC: Unique testnames

[Jeff Law <law at redhat dot com>]

On 9/13/18 5:15 AM, Nick Clifton wrote:
> Hi Guys,
> 
>   I am planning on committing the attached patch, but I thought that it
>   would be a good idea to check with the community first to see if there
>   are any comments.
> 
>   The patch fixes an issue with the binutils testsuites where some tests
>   have identical names.  This is a problem when in comes to reviewing
>   failures and trying to identify exactly which test has failed, and it
>   is an issue for automatic testing frameworks which need unique names
>   for each test.
> 
>   In most cases I have just added a number or string to the end of tests
>   which have identical names, but in one case I made a bigger change:
>   The "objcopy zlib-gnu compress debug sections 3" test in the binutils'
>   compress.exp file appears twice, and the two tests appear to be
>   completely identical.  I have assumed that the intent was that the
>   second version of the test would check the copying of an object file
>   containing sections compressed using the zlib-gabi method.  So I have
>   tweaked the test to do this.
> 
>   One other change that might also affect people who run regression
>   tests is that there was a bug in ld-size/size.exp where the size-3e
>   test was not properly escaped, meaning that it was not run...  So this
>   patch fixes that too.
> 
>   Any comments before I check this patch in ?
So to give wider background to the community.  It's the .sum file
comparison tool provided by GCC (contrib/compare_tests) that doesn't
handle this case gracefully.

If the tests have different PASS/FAIL states, the tool will report that
there were regressions/errors in the new testrun.  Uniqueness of the
testnames avoids this problem.

The irony is that the Codesourcery guys were raising the uniqueness of
testnames issues for GCC eons ago (in the QMtest discussions).  I
largely ignored it at the time since I've always done my comparisons by
hand (thankfully others took on the task of addressing this issue for
GCC).  Of course this stuff matters with automatic regression testers :-)

jeff