[PATCH v2] Add regression tests for ctf reading

[Guillermo Martinez]

On Tuesday, November 23, 2021 9:48:35 AM CST Jose E. Marchesi wrote:

Hello Jose/everybody

Thanks for your comments!, answers below:

The following items summarise steps currently implemented in
the testsuite for DWARF and CTF readers:

1) Create corpus using the ELF input file, e.g:
      *src*/libabigail/tests/data/test-read-ctf/test0
     - In this step the SUTs are: *the front-end and readers*.

2) Serialize the corpus object to XML ABI description in output
    directory:
     *build*/libabigail_x86_64/tests/output/test-read-ctf/test0.abi
     - SUT: *writter* using write_corpus function (it doesn't
     use libxml (IMHO something to change if we want to use in the
     future properties, name space, etc).

3) Spawn *abidw* tool with ELF input file using --abidiff argument, e.g:
     abidw --abidiff  --ctf *src*/libabigail/tests/data/test-read-ctf/test0

    Internally abiw works as follow:
     
   3.1) Create corpus from ELF input file, e.g:
            *src*/libabigail/tests/data/test-read-ctf/test0
    - In this step SUTs: *DWARF/CTF frond-end readers*

   3.2) Serialize the (*first*) corpus object to XML ABI description in 
          *temp* directory:
             /tmp/libbigail-xyz
    - SUTs: ABI writter,

   3.3) Build a (*second*) corpus from this temporary file (*same file!*):
        - SUTs: *XML reader*.

   3.4) Compute the corpus differences
              compute_diff(corp, corp2 ..)
     - SUTs: *comparison algorithm* (diff_context/corpus diff)

       if there are differences return 1 otherwise return 0.

4) The return value is read by test-read-ctf and if it's 1 test
    is marked as *FAILED*

5) Use a external *diff command* to compare the XML abi input file with
    the file outputted in the step 2. The return code of the diff command
    is used to mark the test as SUCCESS or FAILED

So, the abidw doesn't use the expected XML file used in the testsuite, so
if there are changes on the input ELF or in the libabigail subsystems (readers,
writer, corpus, etc) we could get false negatives, because it is working with
itself result as an incoming file, instead of use an expected file.

I think that we should use the ABI XML reader from the expected file (e.g:
tests/data/test-read-ctf/test0.abi) to build the corpus to be compared
with the corpus built with ELF input file (e.g: tests/data/test-read-ctf/test0)
and in this way replace the external abidw and diff command calls, this also
helps to avoid false positives when XML ABIs files has the same nodes but not
in the same order.

What do you think?

So, my answer below: 

> I have a couple of general comments/questions regarding libabigail
> testing and DWARF/CTF.
> 
> 1) The usage of diff in the testsuite puzzles me.  I would have expected
>    to find at least two kinds of tests in the testsuite:
> 
>    a) Tests testing the frontend/readers, which can assume the XML
>       reader and comparison algorithm works properly.
       Done in steps 1, 3.1.

>    b) Tests testing the comparison algorithm, which can assume the
>       frontend/readers work properly.
       Done by abidw in step: 3.4  
> 
>    This way no external comparison (like diffing) would be necessary.
>    Is this not the case?
       Yes, agree.
> 
> 2) I am surprised by the fact the CTF reader seems to be working as good
>    as the DWARF reader re. the testsuite.  Not buying it.  I think that
>    the testsuite skips .cc tests when testing CTF support.
       Correct, cc files are skipped, just I collected the C test used in dwarf test reader,
       and the test used by libctf, validating the expected result, so the functionality
       tested in libctf is also tested in libabigail. (except aliases symbols).

>    So I wonder... to what extent are C features (not C++) tested using
>    .cc files?  If we want good coverage for both DWARF and CTF (and
>    eventually BTF or other formats not supporting C++) then we would
>    need to turn these .cc files into .c files...
      sure, I can "translate" C++ test's data and add it to CTF testsuite.
> 
> > Hello libabigail team,
> >
> > This patch includes regression tests v2 for CTF.
> >
> > The change from v1 is disable CTF test alias, because
> > currently is is not supported.
> >
> > Dependencies/limitations:
> >
> > * It was worked on the top of the following patches:
> >  https://sourceware.org/pipermail/libabigail/2021q4/003853.html
> >
> > * Some CTF tests were *disabled* because it generates the XML ABI
> > corpus with *same information* but the XML nodes *are not* always
> > in the *same order*, so using diff command fails. Details here:
> > https://sourceware.org/pipermail/libabigail/2021q4/003824.html
> >
[...]

Sincerely,
Guillermo