Re: RFC: Treat RTLD_GLOBAL as unique to namespace when used with dlmopen

["Michael Kerrisk (man-pages)" <mtk dot manpages at gmail dot com>]

Hi Carlos,

On 07/24/2015 08:37 PM, Carlos O'Donell wrote:
> On 07/20/2015 02:56 PM, Michael Kerrisk (man-pages) wrote:

[...]

>>> The clever developer says "No problem, I will dlmopen a stub
>>> that dlopen's my library with RTLD_GLOBAL" under the impression
>>> that global search list is unique per namespace. On expects
>>> this allows the dlmopen'd stub to load several conjoined plugin
>>> DSOs into the new namspace, having them to resolve their symbols
>>> against eachother in an isolated way. This fails immediately
>>> with a sigsegv (see Bug 18684[1]).
>>
>> This is precisely the use case the Solaris dlmopen() does support:
>> isolation of load namespaces, while allowing DSOs inside a namespace
>> to share symbols via RTLD_GLOBAL.
> 
> I have seen some academic projects that used dlmopen in Solaris to
> implement a form  of virtualization via the isolation of library
> loading. The use of dlmopen solves quite a number of interesting
> security and isolation problems within an application.

Yes,I came across one paper also about this.

[...]

(Yes, I supposed you were just looking for completeness in the discussion.)

>>> Obviously not for 2.22, but 2.23 material, along with Michael's
>>> new dlmopen/dlinfo man pages we should be ready to help developers
>>> use such a feature more extensively. At present I find almost no
>>> code using dlmopen in userspace because it has languished as an
>>> unsupported undocumented feature (Bug 15971, Bug 15271, and Bug 15134
>>> all need fixing).
>>
>> I would said "... because it currently serves no useful purpose".
>> The dlmopen() seems to have been added to Solaris to support
>> precisely the use cases that Carlos describes, and the glibc
>> implementation doesn't support those cases at all.
>>
>> The attached tarball contains a short build script that creates a few
>> shared libraries from (mostly) simple (and commented) source files.
>>
>> The overall structure is as follows:
>>
>>     main():
>>
>>         1. Loads libabc.so with either dlmopen() or dlopen() and 
>>            with either RTLD_GLOBAL or RTLD_LOCAL, depending on the 
>>            command-line arguments. If no arguments are provided, the 
>>            default is dlmopen(..., RTLD_GLOBAL);
>>
>>         2. Invokes abc_start() in libabc.so
>>
>>     abc_start():
>>         1. Loads some other shared libraries using different
>>            combinations of dlmopen() and RTLD_GLOBAL vs RTLD_LOCAL.
>>
>>         2. Invokes a function qrs_start() in the libqrs.so
>>            library.
>>
>>     qrs_start():
>>         Looks up (dlsym()) various symbols in the other shared
>>         libraries and reports on success or failure of the lookups.
>>
>>     main():
>>         Control eventually returns to main(), and it then looks up
>>         some of the same symbols as qrs_start() and reports on
>>         success or failure of the lookups.    
>>
>> The program produces log messages that should make the results 
>> reasonably easy to interpret. Annotated output from a sample
>> run follows.
>>
>> ---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---
>> $ uname -a
>> SunOS login 5.10 Generic_150400-17 sun4v sparc SUNW,SPARC-Enterprise-T5220
>> $ sh build.sh && ./main
>> main(): lmid from dlopen(NULL) is 0 (handle = 0xff3634d8)
>> main(): dlmopen LM_ID_NEWLM ./libabc.so   RTLD_GLOBAL
>> main(): lmid from dlopen("libabc.so") is -13222656 (handle = 0xff371560)
>> main(): invoking abc_start()
>>     Called abc_start()
>> # Note in next line that dlopen(NULL) gave us back a handle for something
>> # other than initial NS. Linux differs on this point.
>>     abc_start(): lmid from dlopen(NULL) is -13222656 (handle = 0xff173690)
>>     abc_start(): dlmopen LM_ID_BASE  ./libdef.so   RTLD_GLOBAL
>>     abc_start(): dlopen              ./libjkl.so   RTLD_GLOBAL
>>     abc_start(): dlopen              ./libmno.so   RTLD_LOCAL
>>     abc_start(): dlopen              ./libqrs.so   RTLD_LOCAL
>>     abc_start(): invoking qrs_start()
>>         Called qrs_start()
>>         qrs_start(): lmid from dlopen(NULL) is -13222656 (handle = 0xff173690)
>>         qrs_start(): lookup of "abc" succeeded   # In this NS, with 
>>         qrs_start(): lookup of "def" failed      # Was loaded into initial NS
>>         qrs_start(): lookup of "jkl" succeeded
>>         qrs_start(): lookup of "mno" failed      # Was loaded with RTLD_LOCAL
>>         qrs_start(): lookup of "main" failed     # Is in initial NS
>> # Now do some lookups from initial NS
>> main(): lookup of "abc" failed                   # In another NS
>> main(): lookup of "def" succeeded                # Was loaded into initial NS
>> main(): lookup of "jkl" failed                   # In another NS
>> main(): lookup of "mno" failed                   # In another NS (+ RTLD_LOCAL)
>> ---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---8x---
> 
> With a few more patches I get *almost* all the way there:
> 
> [carlos@athas dlmopen_expt]$ ./main
> main(): lmid from dlopen(NULL) is 0 (handle = 0x0x7fa3e58ec188)
> main(): dlmopen LM_ID_NEWLM ./libabc.so   RTLD_GLOBAL
> main(): lmid from dlopen("libabc.so") is 1 (handle = 0x0x2267030)
> main(): invoking abc_start()
>     Called abc_start()
>     abc_start(): lmid from dlopen(NULL) is 1 (handle = 0x0x2267030)
>     abc_start(): dlmopen LM_ID_BASE  ./libdef.so   RTLD_GLOBAL
>     abc_start(): dlopen              ./libjkl.so   RTLD_GLOBAL
>     abc_start(): dlopen              ./libmno.so   RTLD_LOCAL
>     abc_start(): dlopen              ./libqrs.so   RTLD_LOCAL
>     abc_start(): invoking qrs_start()
>         Called qrs_start()
>         qrs_start(): lmid from dlopen(NULL) is 1 (handle = 0x0x2267030)
>         qrs_start(): lookup of "abc" succeeded
>         qrs_start(): lookup of "def" failed
> Segmentation fault (core dumped)
> 
> There is more work to be done. This failure is from calling free() in
> the non-LM_ID_BASE namespace for the first time.

Thanks for working on this, and running my test.

> My opinion is that this should all just work, but may require some special
> cases in libc.so.6 and ld.so to make sure everything is initialized in the
> new namespace and has it's own distinct TLS blocks (doesn't use the base
> namespace TLS blocks).
> 
> The bummer is that gdb stops working to debug anything after the dlmopen.
> We're going to need their help to continue debugging this after we get
> the basic patches in place for 2.23.

Okay.

Cheers,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/