Linker Namespaces
The dlmopen API was first introduced in Sun OS. The purpose of the API is to create an entirely new link namespace where libraries can be isolated from the rest of the application. The default link namespace is called the "base" namespace and is identified via the constant LM_ID_BASE. The API can be used in conjunction with the LM_ID_NEWLM constant to create new link namespaces, and libraries can be loaded into these namespaces that do not impact the running application's own loaded libraries.
Initial use case
In glibc the primary use case for dlmopen and multiple link namespaces was to implement the LD_AUDIT interface, also a Sun OS design, to allow users to write shared libraries which can inspect and modify the behaviour of the dynamic loader. The name "library auditing interface" is a bit of a misnomer since the API allows the loaded shared objects to manipulate much of the search path and symbol binding behaviours of the loader. In order to isolate these loaded libraries and hide them from the running application they were loaded into a new linker namespace. The new linker namespace gives them their own unique copy of libc.so.6 and that ensures that any references to things like errno are only coming from the namespace copy of libc.so.6.
The OpenGL use case
The existing dlmopen implementation was designed to isolate relatively simple LD_AUDIT libraries. It has been the generally accepted consensus that a more fully fleshed out dlmopen design could help other developers and meet the needs of their new use cases.
The next most common use case that is raised for dlmopen is related to OpenGL. Assume an application written in C++ tries to use an OpenGL stack that also at some point is using C++. Both the C++ used to write the application and the C++ in the OpenGL stack have nothing to do with eachother and in fact never share any information since the OpenGL API is entirely a C-based API. However because both are running in the context of one dynamic loader, there can only be one referenced libstdc++ library for all libraries that are part of the in-memory image. One way to resolve this is to create a OpenGL shim library that internally uses dlmopen to load the real OpenGL library in a new link namespace. This shim library allows the dlmopen OpenGL to have a different libstdc++ from the application using OpenGL.
Generalizing the shim-library use case
The thread which calls dlmopen is the thread under which the entire dlmopen functionality will execute under, and that has some significant consequences. Firstly the thread has a thread control block head (tcbhead_t), thread control block (TCB), and thread descriptor (struct pthread_t), and all of this data has for a long time been considered internal details of the implementation. However, when you can dlmopen an arbitrary libc.so.6, these internals become part of an implicit ABI which you must share with the new link namespace libc.so6. The sharing is implicit, consider a thread calls dlmopen, the thread descriptor status does not change, and the thread may subsequently use dlsym to attain and call a function in the new link namespace (if we allow this). Such a thread calling into the alternate libc.so.6 functions may, for example, have a need to compare a stack guard for a function fortified with SSP, but the stack_guard is a value in the thread's tcbhead_t, and thus the offset, although internal, is an ABI now. The offset is already an ABI in reality because there is GCC code which is emitted that reads it also, but this is just an example.
The problem gets more complicated when a second libpthread.so.0 is loaded. In this case the implementation of seteuid is not atomic and there is no kernel function for this, instead a signal must be delivered to each thread (SIGSETXID) and each thread must change their own effective UID. The new libpthread.so.0 in the link namespace is not aware of any threads, and so any calls to seteuid from this link namespace will not transition the entire process to a new effective user ID. The new libpthread.so.0 is unaware of threads that were created before it was loaded and this has an impact on the implementation of standards conforming APIs like seteuid.
As you can see generalizing the shim-library use case, at least in the case of glibc, breaks the ability for the implementation to provide conforming behaviour. In the case of a second libpthread.so.0, it's probably OK since one might argue that in Linux where UID is a per-thread attribute, it is conceivably acceptable to have the threads created by each libpthread.so.0 have different UIDs and that is simply a consequence of the isolation. However, note that a thread not created in the namespace will have it's effective UID altered by calling into the namespace's seteuid, but not any other threads, this is because the calling thread always adjusts itself last. This might be fixed by sending signals to all threads on the known list instead of optimizing the thread self case (local delivery of a seteuid change). This is just the start of the problems you have with collusion between the implementation libraries.
The reality is that any global state that the implementation has is part of the problem. When a thread passes the boundary into the new link namespace it carries with it a certain amount of shared state e.g. per-thread locales. This state must be usable if we allow the thread to call a function in the link namespace, accepting that such a function can impact the state of the thread. A logical consequence of this is that it becomes simpler to consider that the implementation is shared in all namespaces and prevent any ABI issues.
The case for strong and partial isolation
The case for strong isolation remains LD_AUDIT.
In fact we already have some problems with strong isolation because the thread which triggers the audit event e.g. PLT symbol binding, is the same thread which calls into the dynamic loader and runs the audit code, and that audit code may inspect tcbhead_t. In practice the existing audit modules have been so simple that they do not break the threads that call into them, but it can happen without stronger isolation.
The case for partial isolation is made by arguing that shim-libraries are a useful use case, and coordinating the same glibc implementation for all the shims solves all the shared information problems. If the underlying libc.so.6, libpthread.so.0, etc, are the same in all namespaces, then the implementation can continue to provide a consistent view of the required APIs, though some leakage can now happen with non-reentrant functions like strtok.
A case is made for RTLD_UNIQUE, but this has several problems, firstly is that all the dependencies of the RTLD_UNIQUE object would also need mapping in, otherwise you risk having a malloc/free boundary problem. The already loaded object pulled in via RTLD_UNIQUE will use the malloc/free from the namespace it was originally loaded into, but now it's also visible from another namespace, and this will cause problems. It may not cause problems if we are using partial isolation, but consider more than just malloc/free, all of the dependencies of the RTLD_UNIQUE loaded object may have APIs that behave like malloc/free, that is to say they require a register/deregister model. In this case the local namespace user of the RTLD_UNIQUE library cannot call deregister properly becaues it can't see the already initialized dependent libraries for the RTLD_UNIQUE library. Similarly the RTLD_UNIQUE library may need to write to global data for a dependency and the namespace into which it is loaded will have it's own copy of the global data, and so will be out of sync and not see any updates. All of this means that implementing RTLD_UNIQUE is quite a complex process involving bringing in all the dependencies of that library into the namespace and that can mean there could be collisions with already loaded libraries.
The design should be kept simple. Either we have total isolation starting at the C runtime implemetnation, or we have partial isolation with collusion. Today both have clear use cases. Above the C runtime the user is reponsible for data sharing across link namespaces. While it is clear that the same problems the C library faces can be seen at higher levels, where a developer may want library X the same in multiple namespaces, it is a feature which should be provided after glibc is enhanced to provide these two levels of isolation.