[PATCH] ld: Add more tests for --as-needed

[Alan Modra]

On Wed, Sep 09, 2020 at 09:16:40AM -0700, H.J. Lu via Binutils wrote:
> commit fb7331ae2b551247a33e8da5387ebc910c4a7308
> Author: Alan Modra <amodra@gmail.com>
> Date:   Wed Sep 9 12:16:23 2020 +0930
> 
>     lto-18 test
> 
> added tests to show the reason to use IR symbols when deciding an
> --as-needed library should be loaded.

Exactly.  The testcase compiles and links fine without LTO, and
compiles and links with LTO if references from IR symbols are used to
decide whether a library is needed.  Reverting 1e3b96fd6cf0 results in

FAIL: lto-18 (1)
FAIL: lto-18 (2)

You kept asking for a testcase in the thread where I committed
1e3b96fd6cf0, and now you have one.  I think the testcase is valid,
but like many testcases, is nasty.

That testcase is not the only reason why I think we should be using IR
symbol refs to decide that an as-needed library is needed.

Here are my reasons:

1) --as-needed is supposed to load a shared library if there is a
non-weak undefined symbol remaining from previously loaded objects,
defined by the library.  By previously loaded objects we mean those
loaded on the command line or in scripts prior to the library, similar
to the logic used by the linker to extract objects from archives.
Archive searches must extract objects to satisfy IR references,
because the object in the archive might be an LTO object.  Since
--as-needed was intended to be somewhat modelled on the way ld
extracts objects from archives, it makes sense to do the same for
--as-needed libraries.

2) Trying to only emit DT_NEEDED for --as-needed libraries based on
the final LTO set of symbols is complicated, and has led to a number
of bug reports in the past.  Those bug reports have meant that we keep
some symbol state from an as-needed library that isn't marked as
needed at that point in linking, to feed to the LTO recompilation.
While doing that for references generally won't break anything if
the as-needed library is also found to be not needed after LTO, I
worry that doing the same for symbol definitions will lead to
breakage.  If a symbol being defined might affect LTO output, but the
LTO output not have a reference to the symbol in question, then that
could result in an as-needed library not being marked as needed on the
second pass through libraries after LTO.  The result would be that ld
tells LTO a symbol is defined but it ends up undefined.

3) Further patches complicating the linker like the one you posted at
https://sourceware.org/pipermail/binutils/2020-September/113227.html
are likely to be necessary, if we go back to IR symbols not marking
as-needed libraries as needed.

>  But the requirement of the
> --as-needed library doesn't come from the IR inputs since the IR symbol
> reference of the --as-needed library is removed by LTO.  The requirement
> comes from another --as-needed library which is referenced by the LTO
> output as shown by PR ld/26590.

PR26590 merely shows --as-needed working as designed.

>  Not use IR symbols when deciding an
> --as-needed library should be loaded exposes the same issue with input
> --as-needed libraries as non-LTO inputs.  The current ld behavior is
> desirable since it sures that both lazy and non-lazy bindings work the
> same way.
> 
> Add more tests for --as-needed and add tmpdir/liblto-18b.so DT_NEEDED
> to liblto-18c.so.

No, you may not commit the lto.exp patch.  Shared libraries are not
required to have DT_NEEDED entries.  Imagine for moment that
liblto-18b.so is a library from author B that has some undefined
functions that normally would be provided in the executable.  Author B
doesn't even know about liblto-18c.so, thus no DT_NEEDED for it.
Author C has a similar liblto-18c.so library with some undefined
functions normally provided in the executables.  Author A writing the
app wants to use both liblto-18b.so and liblto-18c.so and realizes
that they satisfy each other's undefined symbols.  So he doesn't need
to write those functions in the executable.  He gets everything
working without LTO, then turns on LTO and it fails mysteriously.

The other tests are fine to commit.

-- 
Alan Modra
Australia Development Lab, IBM