Enable preloading in dlopen-ed shared libraries?

[Adhemerval Zanella]

On 28/07/2021 03:17, Fengkai Sun wrote:
> Hi Adhemerval,
> 
> This seems very specific to add such complexity and extra internal state
>> on loader and I want to understand better the issue you are trying to solve
>> and why you can't use the current tools instead.
>>
>> Thanks for your reply! I now realize that changing the interface of the
> dynamic linker needs reason and user cases that are solid and general
> enough. But originally I planned to ask if this is a good idea and if there
> is an existing interface to achieve this. Thank you for the information
> provided!
> 
> I will explain the original problem I want to solve here:
> 
> We have an acadamic project which proposes a framework, and it should load
> and isolate some external libraries completely, making them self-contained
> and does not need to refer to anything outside their own namespace.
> Ideally, dlmopen would suffice, but I found that dlmopen can only support
> up to 16 namespaces, but we might use way more libraries than that.

Do you need each library to be completely isolated from each other? Can't you
load all of them in a single namespace or at least work around specific sets
up to the 16 namespace limits?

> 
> So I came up with another solution to simulate multi-namespace inside one:
> use dlopen and load the libraries all into the default namespace. To
> achieve complete isolation, I recursively renamed the dependencies of the
> external libraries to bypass the limitation of allowing only one instance
> inside a namespace. This is done by ELF patching tools, and I used patchelf.
> For example, let's say there is a dependency graph like this:
> libexternal.so -> lib1.so; libexternal.so -> libc.so
> lib1.so -> libc.so
> libc.so -> ld-linux.so
> 
> After the renaming, the filename and dependency names of each library is
> changed, so that ld.so will load another instance even if some libraries
> already exist in the default namespace.
> The dependency graph after renaming is like this:
> libexternal.so -> lib1-1.so; libexternal.so -> libc-1.so
> lib1-1.so -> libc-1.so
> libc-1.so -> ld-linux-1.so
> 
> This is not a good solution because my understanding of glibc is quite
> shallow. dlopen seems to have problems in initializing libc and libpthread
> when the filename of them are changed. So I tried not to rename libc or
> libpthread and made them shared(just like the RTLD_UNIQUE flag proposed by
> Vivek).

You seems to want unlimited namespace, which is different than what Vivek 
is trying to fix with RTLD_SHARED / RTLD_UNIQUE.  This should be doable by 
adding an extra list on _rtld_global and handle all places that access the 
_dl_ns link_namespaces (dlopen, dlclose, etc.). This would require some work
on how we will handle static TLS (since this take in consideration the maximum
number of namespaces).

And you are correct, glibc internally check if DT_SONAME is the one expected
by the hard-coded LIBC_SO.  Trying to mess with it might broke in some places
and that's exactly that dlmopen intends to provide.

> 
> And we also want to interpose the symbols inside the external libraries. So
> I asked if there is a way to provide a different definition for symbols in
> local scope, just like what LD_PRELOAD does to global scope.
> Currently I use patchelf to prepend a dependency, let's call it
> libpreload.so, in the dynamic section of libexternal.so, so that ld.so will
> search the symbol for libexternal.so and its dependencies first in
> libpreload.so. Things are going well so far, and I will be very grateful if
> you and other people on the list have a look at this solution.
> It seems that our use case is really rare, and a full-functioned rtld-audit
> interface can solve the problem for it.

The dlmopen should work to provide the symbol resolution for a specific set of
libraries: you first load on the namespace and then load the set of libraries.

But without dlmopen I don't have a easy solution for you. If you are not bounded
to the system glibc, you might change the DL_NNS internally to expected number
of namespace.  It has some implication, specially in the static TLS requirement,
but I think it should work.

I am not really against the local scope preload environment variable, but this 
will be another *very* specific feature that we will need to support indefinitely
for an specific usercase that might not even used extensively (as you put this is
a research project). This will also have some semantic issues we will need to sort
out also, like how to represent an specific namespace (since glibc can return any
value and Lmid_t does not need to map to an specific number).