Running gas testsuite

[Stephen Casner]

On Mon, 18 May 2020, Andrew Burgess wrote:

> I think there's more to it that what you outline above.
>
> So, for me at least, this absolutely works:
>
>   git clone ssh://sourceware.org/git/binutils-gdb.git
>   cd binutils-gdb
>   mkdir build
>   cd build
>   ../configure
>   make all-gas
>   cd gas
>   make check

I'm configuring a specific target and just making all implicitly, but
I agree your test looks like a reasonable attempt to reproduce.
Overnight I came up with the idea that it might be gmake vs. the Mac's
system make, but that didn't make a difference for me.

> I suspect the reason this works is that the check-DEJAGNU rule
> contains:
>
>         rm -f testsuite/site.exp
>         cp site.exp testsuite/site.exp
>         rootme=`pwd`; export rootme; \
>         srcdir=`cd ${srcdir}; pwd` ; export srcdir ; \
>         LC_ALL=C; export LC_ALL; \
>         EXPECT=${EXPECT} ; export EXPECT ; \
>         runtest=$(RUNTEST); \
>         cd testsuite; \
>         if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
>           $$runtest --tool $(DEJATOOL) --srcdir $${srcdir}/testsuite \
>                 $(RUNTESTFLAGS); \
>         else echo "WARNING: could not find \`runtest'" 1>&2; :;\
>         fi
>
> Notice that runtest is invoked with `--srcdir'.  I don't know exactly
> what role the $srcdir variable in the site.exp file plays, but given
> it is wrong for me just like it is wrong for you then I assume it's
> actually playing no part in anything.

I see that the relative pathname in srcdir in site.exp does appear to
be used.  In the failure case, the test output includes the following:

    Running target unix
    Using /usr/local/share/dejagnu/baseboards/unix.exp as board description file for target.
    Using /usr/local/share/dejagnu/config/unix.exp as generic interface file for target.
    WARNING: Couldn't find tool config file for unix, using default.

If I remove the "cd testsuite" line to allow the test to run
successfully, then I see:

    Running target unix
    Using /usr/local/share/dejagnu/baseboards/unix.exp as board description file for target.
    Using /usr/local/share/dejagnu/config/unix.exp as generic interface file for target.
    Using ../../gas/testsuite/config/default.exp as tool-and-target-specific interface file.

Maybe there is a difference in the Dejagnu code between our two
installations.  The code in /usr/local/share/dejagnu/runtest.exp that
reads site.exp is the following:

...
    set local_init_file	site.exp	;# testsuite-local init file name
...
    set base_dir [pwd]
...
    # Begin sourcing the config files.
    # All are sourced in order.
    #
    # Search order:
    #    (local)	$base_dir/$local_init_file -> $objdir/$local_init_file ->
    #    (global)	installed($global_init_file) -> $DEJAGNU -> $HOME/.dejagnurc
    #
    # For the normal case, we expect $base_dir/$local_init_file to set
    # host_triplet and target_triplet.
    #

    load_file [file join $base_dir $local_init_file]

    # From this point until the command line is parsed for the second time,
    # some variables are overridden by the local init file.  Most notably,
    # $srcdir is *not* what was given on the command line if Automake is used.
    # Instead, $srcdir is Automake's @srcdir@ for now.

... [code to set $objdir, $testsuitedir, $tool_root_dir]

    verbose "Using test sources in $srcdir"
    verbose "Using test binaries in $objdir"
    verbose "Testsuite root is $testsuitedir"
    verbose "Tool root directory is $tool_root_dir"

So it is loading .../binutils-gdb/pdp11-aout/gas/testsuite/site.exp.

I don't know how to interpret this comment about Automake.  Is that
referring to srcdir in site.exp which ultimately comes from @srcdir@
in gas/Makefile.am?

To see the output of the verbose statements, I ran the test manually:

    runtest --verbose --tool gas --srcdir .../binutils-gdb/gas/testsuite

Near the beginning is this output:

    Found /Users/casner/epos/binutils/binutils-gdb/pdp11-aout/gas/testsuite/site.exp
    Found /Users/casner/epos/binutils/binutils-gdb/pdp11-aout/gas/site.exp
    Using test sources in ../../gas
    Using test binaries in /Users/casner/epos/binutils/binutils-gdb/pdp11-aout/gas
    Testsuite root is ../../gas/testsuite
    Tool root directory is /Users/casner/epos/binutils/binutils-gdb/pdp11-aout

So it is trying to find the test sources and testsuite using the
relative pathnames that are incorrect for the program running in
the .../binutils-gdb/pdp11-aout/gas/testsuite directory.

What do you see if you run the test manually with --verbose?

> I don't claim to be an expert, but I think the problem here is moving
> the site.exp into the testsuite directory.

That's the same conclusion that I stated.

> The site.exp rule is created automatically by automake, and the srcdir
> reflects the directory in which the site.exp creation is done.  So, if
> the rules that triggered the creation of site.exp were is
> gas/testsuite, srcdir would be correct.
>
> So, I'd like to offer two other possible solutions to your list of 3
> above, these are:
>
>   - Create a Makefile.in inside gas/testsuite, and have this create
>     the site.exp file, we'd probably need to add some extra rules to
>     gas/Makefile.in so that the check rules get passed through
>     correctly.... I think.

That seems like a bunch of extra work for not much gain.

>   - Use EXTRA_DEJAGNU_SITE_CONFIG, this allows us to add extra
>     site.exp content, this extra content could include a "correct"
>     absolute path setting for srcdir that would override the incorrect
>     setting.

That seems more like a hack.

                                                        -- Steve