RFC: Treat RTLD_GLOBAL as unique to namespace when used with dlmopen

[Carlos O'Donell]

Michael Kerrisk and I are working on a man page for dlmopen.

I have a question, and a proposal for the community.

We do not allow dlmopen to use RTLD_GLOBAL. Was this really
intended or simply a QoI issue?

Without RTLD_GLOBAL support in dlmopen it means that
the newly loaded DSO in the given namespace is always RTLD_LOCAL.
This seems wrong since it means no DSO loaded via dlmopen can
be used to provide symbols to subsequently dlmopen'd DSOs in the
same namespace?

Therefore dlmopen at present serves only as a limited way to
load one library in an isolated namespace along with all of
the dependent (DT_NEEDED) libraries. It would seem to me that
RTLD_LOCAL already provides this functionality with the exception
that such a DSO may get promoted to RTLD_GLOBAL if future dlopen
calls load a DSO RTLD_GLOBAL that has an implicit dependency
on the RTLD_LOCAL DSO (DT_NEEDED). In this case the DSO loaded
RTLD_LOCAL is promoted to RTLD_GLOBAL to resolve the dependencies.
This breaks the RTLD_LOCAL isolation, and is one of the benefits
of loading a DSO with dlmopen since at least *that* copy will
never be promoted to RTLD_GLOBAL.

The clever developer says "No problem, I will dlmopen a stub
that dlopen's my library with RTLD_GLOBAL" under the impression
that global search list is unique per namespace. On expects
this allows the dlmopen'd stub to load several conjoined plugin
DSOs into the new namspace, having them to resolve their symbols
against eachother in an isolated way. This fails immediately
with a sigsegv (see Bug 18684[1]).

This trick fails for the same reason that calling dlmopen
with RTLD_GLOBAL would fail if you removed the check in dlfcn/dmlopen.c
(dlmopen_doit). When you go to add the DSO to the global
search list you find there is no search list setup. In the case of
the application we have rtld setup the global search list.

Which begs the question? What should the global search list
be for a new namespace? I propose that the global search
list for a new namespace should be a copy of the symbol search
list (scope) of the first DSO loaded into the namespace with
RTLD_GLOBAL, and subsequent RTLD_GLOBAL loads into the namespace
add to that list.

The Solaris documentation is silent on exactly what should happen
in this case. Since an alternate interpretation could be: All objects,
regardless of namespace (link map list) loaded with RTLD_GLOBAL are
available for symbol resolution for any objects. In which case
dlmopen with RTLD_GLOBAL makes no sense, other than perhaps symmetry
with dlopen, because the namespace isolation is lost. This still doesn't
solve the most compelling use case of an isolated set of dlmopen/dlopen
plugins with their own global search list.

The proposed interpretation of RTLD_GLOBAL for dlmopen would allow:

* Use dlmopen with RTLD_GLOBAL, making the symbols of the first
  object loaded into the namespace immediately available to
  subsequent DSOs loaded in constructors or other dlopen implicitly
  into the namespace.

* Use dlopen RTLD_GLOBAL to make symbols available for resolution
  only within the namespace the caller was in.

* Allows complete isolation of a group of dependent DSOs, either
  via DT_NEEDED dependencies or via dlopen or subsequent dlmopen.
  This isolation allows plugin virtualization via dlmopen.

Attached is a patch that fixes this for master. I still need to write
something like a dozen tests to show that this works as expected in
all the cases, but so far every test I've written works and doesn't
regress anything.

Obviously not for 2.22, but 2.23 material, along with Michael's
new dlmopen/dlinfo man pages we should be ready to help developers
use such a feature more extensively. At present I find almost no
code using dlmopen in userspace because it has languished as an
unsupported undocumented feature (Bug 15971, Bug 15271, and Bug 15134
all need fixing).

Thoughts?

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=18684
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix-dlmopen.diff
Type: text/x-patch
Size: 2744 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20150716/284cb903/attachment.bin>