Running gas testsuite

[Andrew Burgess]

* Stephen Casner <casner@acm.org> [2020-05-17 17:48:55 -0700]:

> On Sun, 17 May 2020, Andrew Burgess wrote:
> 
> > So maybe double check that your build tree is correctly configured to
> > your source tree, and that your source tree doesn't have any missing
> > files under gas/testsuite ?
> 
> I figured out the true nature of the problem, and it may have to do
> with how "correctly configured" is defined.
> 
> The problem I observed occurs if configure is invoked with a relative
> pathname rather than an absolute pathname.  That becomes a problem
> through this chain of steps in the build-dir/gas/Makefile:
> 
>   - srcdir is relative, e.g. ../../gas
> 
>   - The site.exp rule makes build-dir/gas/site.exp including the line
> 
>     set srcdir ../../gas
> 
>   - The check-DEJAGNU rule includes these commands:
> 
>     cp site.exp testsuite/site.exp
>     cd testsuite; ... runtest ...
> 
> That cp is not valid because then the relative path for srcdir is no
> longer correct when in the testsuite directory.  That's why runtest
> reported the error I mentioned when it does not $srcdir/testsuite and
> the files necessary to configure the tests.
> 
> I had found that removing the "cd testsuite" allowed the test to run
> because then the first copy of site.exp is used from the directory
> where its definition of srcdir is still valid.
> 
> Is it declared that configure should never be invoked with a relative
> pathname?  If so, configure should check and refuse to run if invoked
> with a relative pathname.

I think there's more to it that what you outline above.

So, for me at least, this absolutely works:

  git clone ssh://sourceware.org/git/binutils-gdb.git
  cd binutils-gdb
  mkdir build
  cd build
  ../configure
  make all-gas
  cd gas
  make check

I suspect the reason this works is that the check-DEJAGNU rule
contains:

        rm -f testsuite/site.exp
        cp site.exp testsuite/site.exp
        rootme=`pwd`; export rootme; \
        srcdir=`cd ${srcdir}; pwd` ; export srcdir ; \
        LC_ALL=C; export LC_ALL; \
        EXPECT=${EXPECT} ; export EXPECT ; \
        runtest=$(RUNTEST); \
        cd testsuite; \
        if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
          $$runtest --tool $(DEJATOOL) --srcdir $${srcdir}/testsuite \
                $(RUNTESTFLAGS); \
        else echo "WARNING: could not find \`runtest'" 1>&2; :;\
        fi

Notice that runtest is invoked with `--srcdir'.  I don't know exactly
what role the $srcdir variable in the site.exp file plays, but given
it is wrong for me just like it is wrong for you then I assume it's
actually playing no part in anything.

> 
> Otherwise there are a few ways the Makefile could be made more robust:
> 
>   - The tests for gas could be run directly in build-dir/gas rather
>     than in a subdirectory "testsuite".  That's how the tests for ld
>     and the binutils tools are run.
> 
>   - Any relative pathnames in site.exp could be adjusted when copying
>     to subdirectory "testsuite".
> 
>   - The site.exp srcdir line could be constructed from abs_srcdir
>     instead.

I don't claim to be an expert, but I think the problem here is moving
the site.exp into the testsuite directory.

The site.exp rule is created automatically by automake, and the srcdir
reflects the directory in which the site.exp creation is done.  So, if
the rules that triggered the creation of site.exp were is
gas/testsuite, srcdir would be correct.

So, I'd like to offer two other possible solutions to your list of 3
above, these are:

  - Create a Makefile.in inside gas/testsuite, and have this create
    the site.exp file, we'd probably need to add some extra rules to
    gas/Makefile.in so that the check rules get passed through
    correctly.... I think.

  - Use EXTRA_DEJAGNU_SITE_CONFIG, this allows us to add extra
    site.exp content, this extra content could include a "correct"
    absolute path setting for srcdir that would override the incorrect
    setting.

> In my opinion this should be considered a bug.

Agreed, but I'm still curious why testing fails for you at all, when
it works for me - unless I misunderstood how your building / testing?

Thanks,
Andrew