Re: ELF linking question related to symbol collisions

["Carlos O'Donell" <carlos at redhat dot com>]

On 12/18/2013 09:47 AM, Florian Weimer wrote:
>> The problem is mainly that they fall afoul of the ELF rules for 
>> interposition.
> 
> Right.  But apart from the occasional LD_PRELOAD and the language
> interpreter optimization (which Fedora doesn't use—you compile your
> main Python/Perl/etc interpreter binary with a statically linked
> non-PIC copy of the actual interpreter implementation, exported
> dynamically, and your extension modules link against that instead of
> the interpreter DSO), I don't think there is actual use of that.
> 
> Or more to the point, I'm not sure if the current linking algorithm
> is what we need.  Something like -B direct might be a better fit for
> our current needs.

I assume we're talking about some of the features present in Solaris?

e.g.
"Direct Bindings"
http://docs.oracle.com/cd/E23824_01/html/819-0690/aehzq.html#scrolltoc
and
"Direct Bindings and Interposition"
http://docs.oracle.com/cd/E23824_01/html/819-0690/gejgf.html
 
>> Symbol collisions are only bad if both symbols do not implement
>> the same ABI and API. If they do implement the same ABI and API
>> then it's a replacement function that is safe to interpose.
> 
> Unless that function references static symbols, then it may or may
> not be safe.  If everything ends up being interposed, it is okay, but
> if not, code might hit different static symbols.

That is correct. If the API has state then it would not work to use
a mixed implementation as you point out.

>> Symbol versioning does not solve the problem in general either
>> since then you need a global version name management, and you need
>> to fix all applications to use versioning which is a huge amount of
>> work. Even then you can still have problems if the projects lack
>> the rigour required to update their version maps.
> 
> I had hoped it would be possible to use a static map with a single
> version-like string, like JSON-C, JSON-GLIB, and JANSSON.  Maybe we
> could even use the soname for this by default.
> 
> This is easier than renaming everything under a shared prefix, and
> would not affect backward compatibility.  Perhaps I'm a bit naïve,
> but I suspect this could be used to implement part of the linking
> semantics of -B direct and reap some of the performance benefits
> because it is easier to locate the correct hash table.

I'd have to think about this more to comment meaningfully.

>> A different linking algorithm isn't helpful either because at
>> static link time you don't need the devel libraries for all
>> dependent libraries, and requiring it would make compiling anything
>> much more complicated.
> 
> Sure, but I think static linking has no future.  If it reappears, it
> will be in the form of LTO object files.

Static has use cases that will never go away.

I plan to support it in glibc forever or until some new method is
as easy as static linking to solve the same use cases.

>> The only robust solution I see is a post-build tool that looks for 
>> global namespace collisions and rejects the build if they exist.
> 
> That's difficult to do because anything that needs to consider more
> than one piece of software in isolation has a high overhead, both
> performance-wise and administratively.
> 
> If we decide to implement namespace management outside of the
> toolchain, I think we should have a list of symbols and symbol
> prefixes that map to library (soname, and then OS package) that
> defines them.  If a library (or dynamic executable) uses a symbol
> outside that list, we'd either have to fix the list or address the
> unintentional out-of-namespace symbol leakage.  Both measures can be
> taken even before any actual collision materializes, and it's only

I agree that toolchain features for managing namespaces are
important.

However, no amount of tooling will fix people from breaking
the namespace. The tooling only helps you resolve the breakage.

There must be a higher level analysis and policy at the distribution
level.

Do you disagree with that last statement?

>> The workaround might be to register your allowed symbol
>> interpositions in the spec file such that the post-build tool can
>> use those to resolve such allowances. Note that just stating that
>> symbol X may be interposed is not sufficient to make this system
>> safe, you must say symbol X from SONAME Y may interpose.
> 
> I think it should be outside the SPEC file because of the syntax
> issues (no good parsers, parsing requires arbitrary code execution
> etc.).

Sounds fine. The Solaris functionality exposes some interesting features.

>> What's your next step?
> 
> If there's a strong desire to implement -B direct, that's the way to
> go, but it's a bit out of my area of expertise.  If there isn't, I'll
> look into writing an out-of-band namespace model for the upcoming
> Fedora Base package set.

I think both are required.

Cheers,
Carlos.