I was surprised recently to discover that if you have the following situation: Files: a.out A/A.so A/libncurses.so.5 B/B.so A/A.so has RUNPATH=$ORIGIN A/A.so is linked against (DT_NEEDED) libncurses.so.5 B/B.so has no RUNPATH set B/B.so is linked against (DT_NEEDED) libncurses.so.5 Execution flow: run program a.out, which 1) dlopen A/A.so with RTLD_LOCAL 2) dlopen B/B.so with RTLD_LOCAL Then: B/B.so will get loaded linked against A/libncurses.so.5, rather than the system libncurses. OTOH, if a.out is modified to swap the order, so that it instead does: 1) dlopen B/B.so with RTLD_LOCAL 2) dlopen A/A.so with RTLD_LOCAL then both A.so and B.so will end up linked against the system version of libncurses.so.5, and A's RUNPATH will be ignored. Sample code to demonstrate this behavior: https://github.com/njsmith/rpath-weirdness I find this behavior very surprising -- I expected that A.so's RUNPATH would be respected, so that A.so always linked to A/libncurses.so, while B.so's lack of RUNPATH would also be respected, so that B.so always linked to the system libncurses.so.5. The reason I expected this is that (1) AFAICT, all available documentation says that what I expected to happen is what should have happened (e.g. ld.so(8) clearly documents the library search order, and doesn't say anything about this; Drepper's dsohowto.pdf AFAICT also seems to say that what I expected to happen is what should have happened), and (2) the actual behavior is very weird and undesireable (IMO). In every other way, A.so and B.so are isolated from each other by being loaded with RTLD_LOCAL -- they get independent ELF scopes, and in particular LD_DEBUG=scopes seems to indicate that we actually end up with two different instances of libncurses.so.5 -- they're both loaded from the same file, but because they're loaded into different ELF scopes they might act differently. (E.g., if A.so interposes some symbol in libncurses, then when A.so calls into its copy of libncurses then libncurses might end up calling back into A.so; but when B.so calls into its copy of libncurses then will never call back into A.so.) OTOH, looking at elf/dl-load.c:_dl_map_object it seems like the current behavior might be intentional. At the very least this is a documentation bug -- Windows has somewhat similar behavior (if some DLL with a given basename has been loaded, then attempting to load another DLL with the same basename will short-circuit all the normal library searching and simply return the previously loaded DLL, even if it's no longer on the search path), but on Windows at least this is well documented. (The actual situation that led to the discovery of this issue is that we're trying to package up Python extensions into self-contained bundles that can run on many different linux systems, which involves vendoring libraries like libgfortran. And to our surprise we found that two independently distribute Python extensions that each had their own vendored copy of libgfortran can interfere with each other, or with locally-compiled Python extensions that expect to use the system libgfortran. See: https://mail.python.org/pipermail/wheel-builders/2016-March/000069.html)
The larger problem (more precise namespace control for dlopen) is related to bug 16881, except that concerns glibc-internal dlopen use. The general is case is very difficult because some libraries need process-global variables to reflect the process-global state. POSIX file locking is the prime example: a library may want to implement per-thread locks on top of POSIX file locks, and this requires that the auxiliary lock table exists only once per process.
*If* the same library is loaded twice, then having the two copies share state makes perfect sense. The problem here is that according to all the library lookup rules, we're actually loading two different libraries that live in two different files at two different locations, yet they still end up sharing state... Yes, the two different libraries have the same SONAME, but there's no guarantee that all libraries with a given SONAME will be interchangeable. "interchangeable" means fully backwards *and* forward compatible, and that's not even true for basic libraries like glibc or libgcc. (Actually, I'm not even sure how much state the two copies share -- since they're in different scopes they get different relocation processing, right? I guess those are in a different segment or something, so the two copies share .data but not relocations or something?)
If you have two libraries with the same SONAME then I think you are entering undefined territory.
(In reply to Andreas Schwab from comment #3) > If you have two libraries with the same SONAME then I think you are entering > undefined territory. This is probably correct now, but I view this bug more as an enhancement request for the dynamic linker to provide a level of encapsulation that allows them to do what they are trying to do.
> If you have two libraries with the same SONAME then I think you are entering undefined territory In what sense is it undefined? Right now the documentation defines very clearly what will happen in this case -- it just turns out that the code does something different :-). So it's not undefined in the sense of the C specification's explicit "invoking undefined behavior". If it should be undefined, then at the least the docs should be updated to say that. Or maybe you mean "the behavior in this case *should* be undefined" -- usually I guess the rationale for this would be that there is no good answer to what should happen, so there's not a lot of value in pinning down the behavior, and there is some value in allowing different implementations to treat it differently. But I don't see how that argument applies here. The whole ELF scopes mechanism goes to great lengths to make it possible to provide "plugin-style" shared libraries that can be loaded at runtime into a host and are isolated from each other so that the authors of different plugins don't need to coordinate. (For those who haven't just spent a few hours poring over dsohowto: the way ELF scopes work is that for libraries loaded via dlopen(..., RTLD_LOCAL), some elaborate work is done to make sure that each library gets its own tree of dependent libraries and symbol resolution scope, so that you can't get accidental symbol collisions between two libraries loaded this way.) So IMO it's a shame that after going to all this trouble to provide isolated symbol scopes, we allow two plugins to crash into each other and break everything if they happen to link to different versions of the same library that have the same SONAME. And this isn't hypothetical at all -- we actually hit this problem, using an actual library that has different non-interchangeable versions that share the same SONAME. We hit it with libgfortran, but it applies equally well to other language support runtime libraries, like, say, libstdc++. If I have the very common situation of a plugin that uses C++ internally but presents a C API to the external world, then I should be able to ship a private copy of libstdc++ with my plugin without worrying that loading my plugin into some host process will cause every other piece of C++ code loaded into that process to blow up.
Nathaniel, it occurred to me that your use case is not helped at all if we fixed this in glibc *now*. We'd need a time machine to fix this in the EL5 time frame, so that you get binary compatibility with libcs from that era. From a Red Hat perspective, changing search path resolution in Red Hat Enterprise Linux 5 and CentOS 5 is not a good idea because it is precisely the kind of change users do *not* expect, even if it is technically a bug fix. It would have to be gated by a dlopen flag. (This also applies to Red Hat Enterprise Linux 6 and CentOS 6.) Therefore, I suggest we move the discussion to libc-help and find a way you can build DSOs for Python modules so that they do not exhibit this problem, largely with tools available on an EL5 distribution.
> Nathaniel, it occurred to me that your use case is not helped at all if we fixed this in glibc *now*. We'd need a time machine to fix this in the EL5 time frame, so that you get binary compatibility with libcs from that era. Right, this bug report is more by way of being a good citizen -- I don't expect any response to this bug report to directly improve my life anytime in the next 5 years, but at least if I report it now then there's some chance that it *will* be fixed and in 5 years it might do me some good :-). > Therefore, I suggest we move the discussion to libc-help and find a way you can build DSOs for Python modules so that they do not exhibit this problem In fact we've already worked around it. The workaround is rather complicated -- it involves a script that renames all our $(NAME).so files to $(NAME)-$(CONTENTHASH).so, and then uses patchelf[1] to fix up the embedded sonames and all DT_NEEDED entries to refer to the new names. (Also, to make this work you need some fixes to patchelf that aren't even fully merged upstream yet...[2][3].) But it works well for our use case, and I don't see any other approach. I am aware though of at least one other use case that this bug is impacting, and where there is no workaround, and where a fix now would actually do some good within a reasonable time frame. The conda package manager [4], which is extremely popular in the scientific programming world, ships its own version of runtime libraries like libgfortran and libstdc++. This causes problems in the following situation: (1) you are using some python or R or whatever extension modules from conda, which are linked against the conda libgfortran, (2) you build a python or R extension module on your own machine, using your system version of GCC, so this extension module requires libgfortran >= your system's version, (3) you try to use this extension module in the same process as the extension modules that ship with conda, and so -- because of this bug -- your extension module ends up linked against conda's libgfortran, (4) your system has a newer version of libgfortran than conda does, so this blows up. Currently the conda maintainers "workaround" to this problem is just to be really quick about updating the libgfortran that they ship whenever a new version is released, in the hopes of making sure that no user ever ends up in a situation where their compiler is newer than their conda install. But this is a bandaid at best. And, it's a case where fixing this bug would actually make a difference in the short term, because the case that breaks right now is systems that are aggressive about upgrading their gcc; so it doesn't matter if RH5's glibc will never be updated, because its gcc will never be updated either, and system who do keep aggressively up to date on gcc probably also keep aggressively up to date on glibc. [1] https://github.com/NixOS/patchelf [2] https://github.com/NixOS/patchelf/pull/85 [3] https://github.com/NixOS/patchelf/pull/86 [4] http://conda.pydata.org/docs/
An additional wrinkle: We also need to be able to at *runtime* say "ah, here's a new shared library libfoo.so.1 that we just found in some random directory, let's add it to the library search path so that we can load another DSO that has DT_NEEDED=libfoo.so.1". But, it turns out that ld.so doesn't respect changes to LD_LIBRARY_PATH within the same process, and AFAICT there isn't any API to add a directory to the loader's search path. So it looks like our workaround for *that* will be to exploit *this* bug, by preloading libfoo.so.1 when we find it, so that it'll be on the global library search path for any future DSO loads. So I still think that this is a bug that should be fixed, but please add a real API for adjusting the loader's search path first before fixing this :-)
(In reply to Nathaniel J. Smith from comment #0) > I was surprised recently to discover that if you have the following > situation: > > Files: > > a.out > A/A.so > A/libncurses.so.5 > B/B.so > > A/A.so has RUNPATH=$ORIGIN > A/A.so is linked against (DT_NEEDED) libncurses.so.5 > > B/B.so has no RUNPATH set > B/B.so is linked against (DT_NEEDED) libncurses.so.5 > > Execution flow: > > run program a.out, which > 1) dlopen A/A.so with RTLD_LOCAL > 2) dlopen B/B.so with RTLD_LOCAL > > Then: B/B.so will get loaded linked against A/libncurses.so.5, rather than > the system libncurses. > > OTOH, if a.out is modified to swap the order, so that it instead does: > 1) dlopen B/B.so with RTLD_LOCAL > 2) dlopen A/A.so with RTLD_LOCAL > > then both A.so and B.so will end up linked against the system version of > libncurses.so.5, and A's RUNPATH will be ignored. This is as expected. You may only have 1 copy of a SONAME library in the in-process memory image in a given namespace at a time. The first loaded copy of SONAME libncurses.so.5 will be used for all other DT_NEEDED resolutions. Even though the first loaded libncurses.so.5 won't be used for relocation and symbol references (RTLD_LOCAL), the DT_NEEDED from the library itself will mean that it has libncurses.so.5 added to it's own search scope. > I find this behavior very surprising -- I expected that A.so's RUNPATH would > be respected, so that A.so always linked to A/libncurses.so, while B.so's > lack of RUNPATH would also be respected, so that B.so always linked to the > system libncurses.so.5. The golden rule for an ELF link namespace: The first loaded library wins. This is precisely the reason why you can LD_PRELOAD a new malloc, otherwise your suggested "fix" would break using tcmalloc, jemalloc and other alternate allocators. > The reason I expected this is that (1) AFAICT, all > available documentation says that what I expected to happen is what should > have happened (e.g. ld.so(8) clearly documents the library search order, and > doesn't say anything about this; Drepper's dsohowto.pdf AFAICT also seems to > say that what I expected to happen is what should have happened), and (2) > the actual behavior is very weird and undesireable (IMO). If you want B.so to use a distinct libncurses.so.5 that means you want _two_ copies of the same potentially conflicting library in the same in-memory process image, and that's dangerous. It's dangerous because it means you can't share ncurses data between A and B, and if you do, they will operate on different ncureses instances of the library. The only way to do what you want with more isolation is to use dlmopen, which was designed for this purpose. However, today, dlmopen is not yet fully supported in glibc, and will take a while before it is. With dlmopen you create a new link namespace and loading B.so with dlmopen will search all over again for libncureses.so.5 without using the on already present and pulled in by A.so. > In every other > way, A.so and B.so are isolated from each other by being loaded with > RTLD_LOCAL That is not isolation. And be careful that RTLD_LOCAL may be promoted to RTLD_GLOBAL if another dlopen references A.so with RTLD_GLOBAL. The only way to get isolation is via dlmopen. > -- they get independent ELF scopes, and in particular > LD_DEBUG=scopes seems to indicate that we actually end up with two different > instances of libncurses.so.5 -- they're both loaded from the same file, but > because they're loaded into different ELF scopes they might act differently. This is not true. The scopes are just used for symbol resolution and relocation information lookup. There is only one instance of the library loaded. > (E.g., if A.so interposes some symbol in libncurses, then when A.so calls > into its copy of libncurses then libncurses might end up calling back into > A.so; but when B.so calls into its copy of libncurses then will never call > back into A.so.) Again this is not true. You are in the single global link namespace. If A.so interposes symbols during the relocation processing of libncurses.so.5, then calls from B.so into libncureses.so.5 may eventually call functions in A.so that were interposed. The only solution you have is dlmopen (when we get it finished). > OTOH, looking at elf/dl-load.c:_dl_map_object it seems like the current > behavior might be intentional. At the very least this is a documentation bug > -- Windows has somewhat similar behavior (if some DLL with a given basename > has been loaded, then attempting to load another DLL with the same basename > will short-circuit all the normal library searching and simply return the > previously loaded DLL, even if it's no longer on the search path), but on > Windows at least this is well documented. I agree we need better documentation. Patches welcome for the glibc manual or Linux kernel man pages. > (The actual situation that led to the discovery of this issue is that we're > trying to package up Python extensions into self-contained bundles that can > run on many different linux systems, which involves vendoring libraries like > libgfortran. And to our surprise we found that two independently distribute > Python extensions that each had their own vendored copy of libgfortran can > interfere with each other, or with locally-compiled Python extensions that > expect to use the system libgfortran. See: > https://mail.python.org/pipermail/wheel-builders/2016-March/000069.html) Correct. You need dlmopen.
I'm going to reuse this bug for Ben Woodards new chapter on libdl functions. https://sourceware.org/ml/libc-alpha/2015-06/msg00110.html https://sourceware.org/ml/libc-alpha/2015-06/msg00110/0001-Add-a-new-dynamic-chapter-to-the-manual.patch
(In reply to Nathaniel J. Smith from comment #2) > *If* the same library is loaded twice, then having the two copies share > state makes perfect sense. The problem here is that according to all the > library lookup rules, we're actually loading two different libraries that > live in two different files at two different locations, yet they still end > up sharing state... > > Yes, the two different libraries have the same SONAME, but there's no > guarantee that all libraries with a given SONAME will be interchangeable. That is wrong. That's exactly what having the same SONAME means. > "interchangeable" means fully backwards *and* forward compatible, and that's > not even true for basic libraries like glibc or libgcc. It is true, but the story is more complicated. With the advent of symbol versioning the state became a tuple { SONAME, <Versioned symbols> }, where symbol versions extended the SONAME described interface. > (Actually, I'm not even sure how much state the two copies share -- since > they're in different scopes they get different relocation processing, right? No. The DSO is mapped only once and has relocations processed only once. The first loaded library wins. > I guess those are in a different segment or something, so the two copies > share .data but not relocations or something?) No.
(In reply to Nathaniel J. Smith from comment #8) > So I still think that this is a bug that should be fixed, but please add a > real API for adjusting the loader's search path first before fixing this :-) Please file a distinct bug for this.