This is the mail archive of the
mailing list for the binutils project.
Re: New ELF linker code added to GNU binutils
On Tue, 25 Mar 2008, Ian Lance Taylor wrote:
> I used DejaGNU for years, and I really don't like it. It is difficult
> to work with and I think it encourages bad test writing. I've written
> the gold testsuite to focus on what I think matters, which is that you
> can use the linker with the compiler to produce programs that do the
> right thing when they are run.
(The way in which I consider DejaGnu encourages bad test writing is that
it's easy to write a testsuite without a well-defined enumeration of the
test assertions (the names after PASS or FAIL in the output: in bad
testsuites the set of names depends on the results of the tests, or some
names are duplicated); I prefer the QMTest approach where it must be
possible to statically enumerate the test assertions separately from
running the tests.)
There's more than programs doing the right thing when run to testing
linker correctness. The following examples apply *even if you assume you
can always run execution tests on the target*.
* The linker is implementing external ABI specifications, which enable
interoperation with third-party compilers and assemblers and debuggers and
other tools and operating systems that execute the resulting binaries, and
assertions "this relocation in the input file causes a GOT entry to be
generated in the output file against this symbol with this relocation" are
relevant as well.
(The present gold testsuite looks like it only has target-independent C++
tests; target-dependent tests of each input relocation are in my view
desirable, whether they are written as execution tests or not. The linker
is not used just with GCC-generated or binutils-generated objects;
certainly not just with those generated by a particular GCC version.)
* Also relevant are tests that the linker implements a particular
optimization, where more than one output is correct but some outputs may
not be optimized.
* Tests that the linker implements a CPU erratum fix, if done with dump
files, don't rely on having a CPU with the erratum in order to test the
fix. (See <http://sourceware.org/ml/binutils/2007-01/msg00233.html> for
example with rationale for such a fix present in binutils. The dump tests
therein did subsequently show up a bug elsewhere in the linker with a bad
qsort comparison function when tested on Windows host. If you don't like
erratum fixes in the linker, see the more recent --fix-v4bx and
--fix-v4bx-interworking options, where no CPU bug is involved but
execution tests would still only be effective on certain CPUs.)
* And the output should of course run on all implementations conforming to
a given specification, not just one; this can never be tested perfectly,
but various aspects of the output can be examined.
* The ABI may specify aspects of the output that cannot be determined
simply by running the program; for example, the ARM EABI rules on how to
merge object attributes in input files to produce attributes on the
output, and on what cases should be diagnosed as incompatible. (Yes, this
is not very well tested in the ld testsuite at present; the test coverage
for the attributes I added for Power and MIPS ABI tagging is better.)
> Of course, a consequence is that the testsuite only works for native
> development (apart from the unittests, of which there are currently
> only two). I recognize that this is a real deficiency for cross
> development. I would prefer to address that by using some sort of
> script to run the program in the remote environment. It would be fine
> with me if that script uses DejaGNU.
You need this for running programs on the host (the linker, compiler etc.
themselves), which may not be the build system, as well as for running
programs on the target.
> > * Testing an installed toolchain through creating a site.exp file and
> > running runtest with appropriate options, rather than testing in the build
> > directory, so you can test the (relocatable) tools in exactly the
> > arrangements in which they will be distributed and used, or compare two
> > different tool versions on the same testsuite version.
> It would be reasonably easy to modify the existing gold testsuite to
> work this way. We could simply change the uses of gcctestdir/ld and
> -Bgcctestdir/ to use make variables instead. Then you could pass the
> appropriate make variables to run the testsuite.
Another variable you need is the compiler's multilib options.
> Basically I really think DejaGNU has only one thing going for it,
> which I admit is rather important: it has a lot of information about
> running programs on different target boards. I'm happy to use that
> part of it, but I would prefer to stay away from it for the testsuite
> in general.
- And about running programs on hosts, not just targets.
- And about how to link programs for different boards. (Remember all the
"generic" ELF targets where by design $target-gcc will not link by
default, without a linker script selecting particular board support code.
DejaGnu board files contain the information about how to link.)
Then there's the DejaGnu .sum and .log output format and scripts set up to
process it. I think it's a good idea even for non-DejaGnu testsuites to
produce output files in this format with the results of each test
assertion. (It's possible of course to process the output of "make" - I
do so even for the glibc testsuite, which is much worse in this regard -
but testsuites can be helpful and create output in the usual form as
Yes, arbitrary testsuites can be wrapped so they use DejaGnu support for
running programs on hosts and targets, and the output postprocessed into
DejaGnu format. But every testsuite being randomly and arbitrarily
different from every other is a pain to deal with, which is why I prefer
testsuites to use the same standard infrastructure as each other; and in
practical terms DejaGnu is that standard infrastructure for toolchain
testing, even if QMTest (or your-preferred-test-harness-here) is in some
ways more theoretically elegant.
> That is, I think a linker testsuite should test functionality. It
> shouldn't test that the output is unchanged from previous versions. I
It should however test output where that output is part of the
functionality (e.g. correspondence of input relocations to output
specified in the ABI). And it's an important feature (although not tested
as such in the present ld testsuite, and as with other features there will
be bugs) that the output for a given version and target does not depend on
the host system.
Tests of unchanged output can be a pain (as I've observed when changing
anything related to MIPS GOT output), but they are also good at showing up
output changing when it shouldn't, or different on different hosts when it
shouldn't be (the qsort comparison function mentioned above).
> > The --version output is a bit strange:
> > GNU gold (GNU binutils (GNU Binutils) 126.96.36.19980325) version 1.4
> > (note the "GNU binutils (GNU Binutils)"). BFD_VERSION_STRING includes
> > both the package version ("(GNU Binutils)" by default, possibly overridden
> > by --with-pkgversion) and a version number - it's not expected for
> > binutils components to have their own independent version numbers such as
> > "1.4". The word "version" isn't part of the standard --version output.
> Yikes. This was better when I wrote it--something must have changed
> since the binutils snapshot I was using for development. I'll clean
> it up.
Yes. The output for all binutils programs was made to follow the GNU
Coding Standards more closely, with --with-pkgversion and --with-bugurl
support added to support distributors putting their own "package name and
package version" in place of "GNU Binutils" and their own bug-reporting
URLs in place of the FSF ones.
Joseph S. Myers