This is the mail archive of the libc-help@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Building glibc 2.19 for OS/ABI UNIX - System V


On 24 July 2014 13:10, Carlos O'Donell <carlos@systemhalted.org> wrote:
> On Thu, Jul 24, 2014 at 3:44 PM, Shaun Jackman <sjackman@gmail.com> wrote:
>> I'd like to run a binary executable that was compiled on a newer
>> system (Ubuntu 14.04) on an older system (CentOS 5.10). To that end,
>> I'm compiling glibc 2.19 on the older system (CentOS 5.10). I realize
>> that the short answer is probably "Don't do that". All the same, call
>> it a learning experience for me. If it doesn't work and can't work,
>> I'd like to learn first hand why it doesn't work.
>
> It is not my place to judge your requirements or needs, your question
> is a perfectly valid one, and I have some answers for you.
>
> To run a newer application on an older system requires *all* of
> runtimes that are needed by the newer application.
>
> It does work, and I've done things like run Chrome 30 on RHEL6 but it
> requires 35+ newer libraries, changing the scripts to call the new
> ld.so, and because I don't want to modify the chrome binaries
> (licensing reasons) I preload a DSO that interposes all of the exec*
> family of functions to run any helper programs under the right
> runtimes.

I'm okay with upgrading dynamic libraries. That I can accomplish with
RPATH, RUNPATH and LD_LIBRARY_PATH. The two things that I cannot
upgrade easily are the Linux kernel 2.6.18 and the system's default
loader /lib64/ld-2.5.so.

>> The current problem I'm facing is the error message `ELF file OS ABI
>> invalid` because the `libc-2.19.so` that I've compiled on CentOS 5.10
>> uses an ABI unsupported by the system's loader `/lib64/ld-2.5.so`. Is
>> it possible to get past this without using a newer loader?
>
> Yes it is possible to revert the OS ABI changes, but that doesn't
> solve your problem.

I'd like to explore reverting the OS ABI as one possible solution. I
understand that running a newer glibc with an older ld.so is not
supported, and if it goes up in smoke, I only have myself to blame.
Could you please elaborate on how to compile a glibc that uses the
older OS ABI?

> Let me reiterate why.
>
> All of the core runtime, loader, libc, libm, libpthread, libdl, etc,
> which are built by glibc, must be upgraded *together* or downgraded
> *together*. They are coordinating between each other to provide the
> implementation and it is unsupported to mix and match.

I can upgrade all these components of glibc except the system default loader.

>> Using a newer loader, although possible, would make running the new
>> executable(s) pretty cumbersome. I could either wrap all the new
>> executables in a shell script that runs the new loader, pretty ugly,
>> or I could patch the ELF header to specify the new dynamic linker (the
>> effect of `ld --dynamic-linker`). I'd prefer to use the old loader, if
>> possible.
>
> In order of least isolation to most isolation you have the following options:
>
> (a) Copy new versions of all the new libraries you need to your
> system. Rewriting the program header e.g. INTERP and adding DT_RPATH
> to force the application to search in a special place first. This
> option is great since you use the dyanmic loader to your advantage to
> ensure the right libraries are used.

This worked really well! I used PatchELF to modify the dynamic loader
and the RPATH of the executable. It'll have to be done on deployment
to each system, because the location of the new glibc is in the user's
home directory, but it's possible. I don't suppose there's support to
expand environment variables like ${HOME} in the ELF header, is there?

patchelf --set-interpreter ${libc}/lib/ld-linux-x86-64.so.2
--set-rpath ${prefix}/lib foo

[patchelf]: https://nixos.org/patchelf.html

> (b) Copy new versions of all the new libraries you need to your
> system. Wrap all executables in shell scripts that invoke them via the
> new loader and the right `--library-path` options, possibly preloading
> a shared library that interposes exec*. To give you an idea of what an
> interposer library would look like see the attached exec_wrapper.c (it
> may make your eyes bleed) that I've used to run chrome in a contained
> runtime. This option is harder but allows you to avoid changing any
> binaries if licensing is a problem.

If the patchelf solution above works, I won't explore this option.

> (c) Setup a Ubuntu chroot, chroot into it, and run your application.

chroot requires root permission, and the end user won't have root
access. The target machine is a high performance computing cluster.

> (d) Setup a linux container (cgroup) with a Ubuntu chroot, and run
> your application in there.

Docker is a possible solution. It require root access to install, but
not to run. It requires a newer Linux kernel though.

> (e) Setup a Ubuntu VM and run your application there.

I'd rather not. Let's call that plan F.

> Lastly, the core runtimes are backwards compatible, and we work very
> very hard to keep it that way (at least on x86*). You could upgrade
> glibc to a newer glibc from say Fedora, but at that point you might
> want to stop compiling applications on that system. Any newly compiled
> applications might possibly use newer symbols from the newer glibc
> (contaminated with a newer ABI) and you could never run those
> applications on other Cent OS 5.10 boxes because they would not have
> the newer glibc installed. However, nothing stops you from installing
> a new core runtime... except that we don't guarantee bug backwards
> compatibility or undefined behaviour compatibility, so some other old
> applications may stop working (despite the fact that glibc is
> correct).

I can't unfortunately upgrade the system's default libc/loader.

> The question of running newer applications on older distributions is a
> very old question, and one that has subtle tradeoffs.
>
> The industry believes the ultimate solution is going to be containers,
> but that's just a synonym for "static linking" and has it's own
> problems e.g. all your containers, not just all of your systems, need
> updating if there is a security bug in a shipped component. So it's
> much more like a light-weight VM.

I agree with you here.

> In full disclosure, I work for Red Hat and I am the team lead for
> glibc within the base OS tools team. I could talk all day about these
> problems if you're interested, but I figure you've got a specific
> problem to solve :-)

Thanks for your help, Carlos. This discussion has already been fruitful.

Cheers,
Shaun


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]