RFC: Treat RTLD_GLOBAL as unique to namespace when used with dlmopen

["Carlos O'Donell" <carlos at redhat dot com>]

Michael Kerrisk and I are working on a man page for dlmopen.

I have a question, and a proposal for the community.

We do not allow dlmopen to use RTLD_GLOBAL. Was this really
intended or simply a QoI issue?

Without RTLD_GLOBAL support in dlmopen it means that
the newly loaded DSO in the given namespace is always RTLD_LOCAL.
This seems wrong since it means no DSO loaded via dlmopen can
be used to provide symbols to subsequently dlmopen'd DSOs in the
same namespace?

Therefore dlmopen at present serves only as a limited way to
load one library in an isolated namespace along with all of
the dependent (DT_NEEDED) libraries. It would seem to me that
RTLD_LOCAL already provides this functionality with the exception
that such a DSO may get promoted to RTLD_GLOBAL if future dlopen
calls load a DSO RTLD_GLOBAL that has an implicit dependency
on the RTLD_LOCAL DSO (DT_NEEDED). In this case the DSO loaded
RTLD_LOCAL is promoted to RTLD_GLOBAL to resolve the dependencies.
This breaks the RTLD_LOCAL isolation, and is one of the benefits
of loading a DSO with dlmopen since at least *that* copy will
never be promoted to RTLD_GLOBAL.

The clever developer says "No problem, I will dlmopen a stub
that dlopen's my library with RTLD_GLOBAL" under the impression
that global search list is unique per namespace. On expects
this allows the dlmopen'd stub to load several conjoined plugin
DSOs into the new namspace, having them to resolve their symbols
against eachother in an isolated way. This fails immediately
with a sigsegv (see Bug 18684[1]).

This trick fails for the same reason that calling dlmopen
with RTLD_GLOBAL would fail if you removed the check in dlfcn/dmlopen.c
(dlmopen_doit). When you go to add the DSO to the global
search list you find there is no search list setup. In the case of
the application we have rtld setup the global search list.

Which begs the question? What should the global search list
be for a new namespace? I propose that the global search
list for a new namespace should be a copy of the symbol search
list (scope) of the first DSO loaded into the namespace with
RTLD_GLOBAL, and subsequent RTLD_GLOBAL loads into the namespace
add to that list.

The Solaris documentation is silent on exactly what should happen
in this case. Since an alternate interpretation could be: All objects,
regardless of namespace (link map list) loaded with RTLD_GLOBAL are
available for symbol resolution for any objects. In which case
dlmopen with RTLD_GLOBAL makes no sense, other than perhaps symmetry
with dlopen, because the namespace isolation is lost. This still doesn't
solve the most compelling use case of an isolated set of dlmopen/dlopen
plugins with their own global search list.

The proposed interpretation of RTLD_GLOBAL for dlmopen would allow:

* Use dlmopen with RTLD_GLOBAL, making the symbols of the first
  object loaded into the namespace immediately available to
  subsequent DSOs loaded in constructors or other dlopen implicitly
  into the namespace.

* Use dlopen RTLD_GLOBAL to make symbols available for resolution
  only within the namespace the caller was in.

* Allows complete isolation of a group of dependent DSOs, either
  via DT_NEEDED dependencies or via dlopen or subsequent dlmopen.
  This isolation allows plugin virtualization via dlmopen.

Attached is a patch that fixes this for master. I still need to write
something like a dozen tests to show that this works as expected in
all the cases, but so far every test I've written works and doesn't
regress anything.

Obviously not for 2.22, but 2.23 material, along with Michael's
new dlmopen/dlinfo man pages we should be ready to help developers
use such a feature more extensively. At present I find almost no
code using dlmopen in userspace because it has languished as an
unsupported undocumented feature (Bug 15971, Bug 15271, and Bug 15134
all need fixing).

Thoughts?

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=18684

diff --git a/dlfcn/dlmopen.c b/dlfcn/dlmopen.c
index 38dca7a..ba468d2 100644
--- a/dlfcn/dlmopen.c
+++ b/dlfcn/dlmopen.c
@@ -61,11 +61,6 @@ dlmopen_doit (void *a)
       if (args->file == NULL)
 # endif
 	GLRO(dl_signal_error) (EINVAL, NULL, NULL, N_("invalid namespace"));
-
-      /* It makes no sense to use RTLD_GLOBAL when loading a DSO into
-	 a namespace other than the base namespace.  */
-      if (__glibc_unlikely (args->mode & RTLD_GLOBAL))
-	GLRO(dl_signal_error) (EINVAL, NULL, NULL, N_("invalid mode"));
     }
 
   args->new = GLRO(dl_open) (args->file ?: "", args->mode | __RTLD_DLOPEN,
diff --git a/elf/dl-open.c b/elf/dl-open.c
index 027c1e0..175ef16 100644
--- a/elf/dl-open.c
+++ b/elf/dl-open.c
@@ -72,6 +72,31 @@ add_to_global (struct link_map *new)
     if (new->l_searchlist.r_list[cnt]->l_global == 0)
       ++to_add;
 
+  struct link_namespaces *ns = &GL(dl_ns)[new->l_ns];
+
+  if (__glibc_unlikely (new->l_ns != LM_ID_BASE
+			&& ns->_ns_main_searchlist == NULL))
+    {
+      /* An initial object was loaded with dlmopen into a distinct namespace
+	 that has no global searchlist (RTLD_GLOBAL) and RTLD_GLOBAL was used.
+	 Or that object then dlopened another object into the global
+	 searchlist.  We find ourselves with no global searchlist initialized.
+	 We have two choices, either we forbid this scenario and return an
+	 error or treat the first RTLD_GLOBAL DSOs searchlist as the global
+	 searchlist of the namespace.  We do the latter since it's the most
+	 sensible course of action since you may dlmopen other libraries which
+	 have no idea they have been isolated.  Thus RTLD_GLOBAL dlopen calls
+	 within the new namespace are restricted to the new namespace and may
+	 reference the symbols of the initial RTLD_GLOBAL dlmopen'd
+	 libraries.  */
+      ns->_ns_main_searchlist = &new->l_searchlist;
+      /* Treat this list like it is read-only.  A value of zero forces a copy
+	 later if we need to extend this list.  The list itself is already
+	 being used as the primary scope for the first loaded RTLD_GLOBAL
+	 object into the new namespace, thus we don't want to free it.  */
+      ns->_ns_global_scope_alloc = 0;
+    }
+
   /* The symbols of the new objects and its dependencies are to be
      introduced into the global scope that will be used to resolve
      references from other dynamically-loaded objects.
@@ -86,7 +111,6 @@ add_to_global (struct link_map *new)
      in an realloc() call.  Therefore we allocate a completely new
      array the first time we have to add something to the locale scope.  */
 
-  struct link_namespaces *ns = &GL(dl_ns)[new->l_ns];
   if (ns->_ns_global_scope_alloc == 0)
     {
       /* This is the first dynamic object given global scope.  */