Evolution of ELF symbol management

[Florian Weimer]

On 11/19/2016 06:25 PM, Zack Weinberg wrote:
> On 11/18/2016 10:48 AM, Florian Weimer wrote:
>> On 11/16/2016 04:55 PM, Zack Weinberg wrote:
>>
>>> First, I don't have a problem with adding __libc_* aliases as
>>> non-default but public alternative names, so that third-party libraries
>>> can avoid namespace pollution in static linkage and/or choose to refer
>>> to symbols that are more likely to get resolved to the C library's
>>> export than to a colliding symbol in the main executable.  I'd reserve
>>> that for cases where there is already a demonstrated need, though, such
>>> as libstdc++.  I also don't have a problem with a hypothetical policy
>>> that all new symbols that aren't in the current revisions of C+POSIX
>>> should be given an impl-namespace name as their primary definition, with
>>> the user-namespace name being a weak alias (but still the only name used
>>> in the public headers).
>>
>> If we don't declare the library-safe names in headers, how can libraries
>> call them?
>
> We could _declare_ the library-safe names in the headers, just not as
> the primaries.  Like how string.h currently declares both bzero and __bzero.

I have no idea at all why __bzero was introduced.  It doesn't look like 
a particularly compelling example (similar to __secure_getenv).  It may 
have been a SunOS thing.

> Incidentally, it occurrs to me that the user-namespace name must exist,
> for the sake of people using dlsym(RTLD_DEFAULT, "whatever") to access
> symbols that they anticipate existing in future revisions of libc
> (relative to the one they used at link time).

There also configure scripts which do

extern char whatever();
main () { whatever(); }

for link testing, without including the appropriate header (perhaps 
deliberately).  That could be considered broken.  I'm not sure.

>> I also do not want to encourage application or library code to reference
>> the implementation namespace at the source code level.  It's ugly, and I
>> suspect it encourages implementation namespace pollution once
>> programmers are used to it.
>
> I don't like it either, but how else could a library's headers opt into
> these special names on a per-symbol, per-use basis?

In general, headers should avoid using libc types, particularly off_t, 
time_t, struct timeval, struct stat, and so on.  But there might be 
exceptions.

For C, it's not clear at all to me whether we need any kind of opt-in 
besides compiling with a particular _*_SOURCE variant (which introduces 
the definition).  We accept that switching to a new glibc version 
requires adjusting source code across the system:

   <https://sourceware.org/ml/libc-alpha/2016-10/msg00307.html>

We can't provide complete source level compatibility on updates, and for 
things compiled with _GNU_SOURCE, it seems to be an explicit non-goal. 
At least that's how I read the previous discussion.

For C++, we might use something based on namespaces to get a clear 
separation.  However, the problem there is that type names and struct 
tags end up in C++ mangled identifiers and thus impact application ABI. 
I have no good idea what to do there.

But to be honest, I was mainly concentrating on function symbols.  I 
understand that other identifiers also present problems.  But with 
those, you can work around issues by reducing the size of translation 
units, at least for C.

> Come to think of it, to actually avoid polluting the user namespace, any
> library that wants to use these will need a secondary set of libc
> headers that declare _only_ the private names.  (This is especially
> relevant for C++ with so much code in headers.)  If we don't do that,
> the user-namespace libc prototype (which still exists under your plan)
> might conflict with an unrelated application definition.

Yes, that's what using a namespace for C++ would achieve.

I'm less convinced we should do this for C.  There is precedent, 
Microsoft did exactly this for the POSIX-inspired interfaces in their 
libc (functions like _open, _close and so on).

> I don't like _that_, because now we have to maintain multiple copies of
> the same set of prototypes with different names *in different headers*.

See Josephs' idea about auto-generating public headers.  It's definitely 
a requirement for this.

> But again, I don't see an alternative in C.

Reducing the size of translation units can be used to sidestep the 
issue, and you can write bridge functions to glue together parts which 
are not directly compatible, providing that there are no symbol clashes 
(which is why I'm particularly interested in symbol collisions).

>>> I _do_ have problems with causing these symbols to be used by default.
>>> My main concern is that I think it's tackling the problem in the wrong
>>> place.  If we want to back away from the original principle that the
>>> executable always wins, isn't an expanded version of -Bsymbolic mode
>>> what's _really_ wanted?
>>
>> We could cover a lot of ground if we had a new flag on versioned symbol
>> definitions which tells the static linker to set a flag on the versioned
>> symbol reference, and the dynamic linker would then use this flag to
>> ignore unversioned symbols for binding symbols.
>
> I was imagining a new annotation on _all_ undefined symbols in a shared
> object, giving the soname of the object that they were satisfied by at
> link time.  At load time, 'getrandom!libc.so.6' resolves to the
> 'getrandom' definition in libc.so.6, ignoring all other definitions of
> the same name.  If there are symbol versions involved, only the versions
> exported by libc.so.6 are considered.  For instance,
> 'getrandom!libc.so.6@GLIBC_2.25' cannot be satisfied by
> 'getrandom@GLIBC_2.25' exported by libmissing-syscalls.so.1.

We still need to support LD_PRELOAD and interposition of arbitrary 
symbols, and not just malloc-related ones, for the benefit of Address 
Sanitizer, fakeroot, cwrap, memstomp and other tools.

This is why hard-coding the DSO name does not seem advisable.

>> (The static linker currently does not add the version of the interposed
>> symbol when interposition happens at static link time.)
>
> I don't understand this statement, and that makes me worry that you are
> trying to solve a different problem from the one I thought you were
> talking about -- a problem that I might not even know exists.  Can you
> elaborate, please?  How can a shared object be interposed upon at static
> link time?  Its own static link has already happened!

When an application is linked against a shared object, if it interposes 
any symbols in it, the symbols becomes exported, so that interposition 
works at run time (otherwise, it could not happen).  You can see an 
example here:

$ nm -Dg malloc/tst-interpose-nothread  | grep ' T '

The application is *not* compiled with -Bdynamic or something like that, 
it happens automatically.

But the symbol version from libc.so.6 is not attached to this symbol 
(“nm” would not show it, but you can check with eu-readelf, for example).

> You keep talking about symbol versioning but, again, I don't understand
> how symbol versioning is relevant,

It's a namespace mechanism for ELF, so it's tempting to use it to 
address this problem as well.

> and that makes me worry that you're
> trying to solve a different problem that I don't even know about.  I
> thought the issue here was controlling *which library* provides a
> symbol, independent of whether the symbol has versions.

I'm not convinced this desirable because of the exceptions I listed 
above.  Both manual name mangling (which I currently prefer) and symbol 
versioning with the no-interpose flag (the one I sketched earlier) would 
support them.  But only name mangling addresses collisions before the 
first static link.

> As for what appears in the headers -- again, see above: to solve the
> libstdc++/_GNU_SOURCE problem, which I believe I *do* understand, there
> needs to be a set of alternative names that libstdc++ headers can
> *explicitly* refer to; otherwise we have still polluted the user
> namespace.  REDIRECTed declarations do not solve that problem.  In fact,
> with my proposed which-library annotations, we could perfectly well
> redirect the mangled names to the normal names:
>
> extern ssize_t __libc_getrandom (void *, size_t, unsigned int)
>   __asm ("getrandom") __attribute__ ((__bind_to_library ("libc.so.6")));
>
> #ifdef __USE_GNU
> extern typeof (__libc_getrandom) getrandom;
> #endif

Yes, I this would work (subject to the existing shortcomings).

> All this aside, this discussion is still very brainstormy and that makes
> me think that we should *not* yet be supplying mangled names for public
> use.  Once we start doing that we are stuck with it forever, after all.
> Contrariwise, we *can* always retrofit __libc_* aliases or whatever once
> we know what we ought to be doing.

I think getrandom is special because the name is rather generic, just 
like some of the new libm names (where we still might have to introduce 
mangling based on feedback from distribution rebuilds; I just don't know 
yet).

Thanks,
Florian