Bug 27924

It is probably well-known that position independent executables have many relative relocations.

If the Distribution adopts -Bsymbolic-non-weak-functions (safer alternative -Bsymbolic-functions which respects pointer equality for C++ vague linkage definitions), shared objects can benefit a lot as well https://sourceware.org/bugzilla/show_bug.cgi?id=27871

Here is an example of libLLVM-13git.so

# -Bsymbolic-functions (-Bsymbolic-non-weak-functions is similar since -fvisibility-inlines-hidden is already used)
% llvm-readelf -WS symbolic.so | egrep 'Nr|\.rel[ar]'
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 8] .rela.dyn         RELA            00000000003c8d60 3c8d60 27d368 18   A  1   0  8
  [ 9] .rela.plt         RELA            00000000006460c8 6460c8 0020a0 18  AI  1  25  8

# -Bsymbolic-functions --pack-dyn-relocs=relr
% llvm-readelf -WS relr.so | egrep 'Nr|\.rel[ar]'    
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 8] .rela.dyn         RELA            00000000003c8d60 3c8d60 016bd8 18   A  1   0  8
  [ 9] .relr.dyn         RELR            00000000003df938 3df938 007ba8 08   A  0   0  8
  [10] .rela.plt         RELA            00000000003e74e0 3e74e0 0020a0 18  AI  1  26  8

I can find a Chrome OS patch against the old glibc 2.23. Hope a glibc maintainer can refresh it and apply it in the upstream.

https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/sys-libs/glibc/files/local/glibc-2.23-sht_relr.patch

The proposal to the ABI seems stalled and probably needs to be pushed again to the gABI group with the updated wording for review again? The alternative is not to extend the gABI but consider this a GNU ABI extension. If Google has interest in moving this forward someone needs to be a change champion here and work with upstream.

Objectively I have no problem with SHT_RELR, but it should be documented in our ABI docs *somewhere* so we have a concrete place to discuss future changes and accept those changes.

Today the gABI list is a "consensus" group with Cary Coutant documenting that consensus. The GNU ABI is implemented by the GNU Toolchain, so there is probably some consensus there that needs forming, but we can do that too.

GNU gABI discussion:
https://sourceware.org/legacy-ml/gnu-gabi/2017-q4/msg00005.html

(In reply to Carlos O'Donell from comment #3)
> The proposal to the ABI seems stalled and probably needs to be pushed again
> to the gABI group with the updated wording for review again? The alternative
> is not to extend the gABI but consider this a GNU ABI extension. If Google
> has interest in moving this forward someone needs to be a change champion
> here and work with upstream.
> 
> Objectively I have no problem with SHT_RELR, but it should be documented in
> our ABI docs *somewhere* so we have a concrete place to discuss future
> changes and accept those changes.
> 
> Today the gABI list is a "consensus" group with Cary Coutant documenting
> that consensus. The GNU ABI is implemented by the GNU Toolchain, so there is
> probably some consensus there that needs forming, but we can do that too.

http://www.sco.com/developers/gabi/latest/contents.html is unmaintained and receives no update since circa 2016. So SHT_RELR is wording adopted by many groups, including Solaris which has many differences with the GNU ABI.

If https://groups.google.com/g/generic-abi/c/9OO5vhxb00Y "Ongoing Maintenance of the gABI" will take some time, perhaps the wording can be added to System V Application Binary Interface Linux Extensions temporarily?

(In reply to Fangrui Song from comment #5)
> (In reply to Carlos O'Donell from comment #3)
> > The proposal to the ABI seems stalled and probably needs to be pushed again
> > to the gABI group with the updated wording for review again? The alternative
> > is not to extend the gABI but consider this a GNU ABI extension. If Google
> > has interest in moving this forward someone needs to be a change champion
> > here and work with upstream.
> > 
> > Objectively I have no problem with SHT_RELR, but it should be documented in
> > our ABI docs *somewhere* so we have a concrete place to discuss future
> > changes and accept those changes.
> > 
> > Today the gABI list is a "consensus" group with Cary Coutant documenting
> > that consensus. The GNU ABI is implemented by the GNU Toolchain, so there is
> > probably some consensus there that needs forming, but we can do that too.
> 
> http://www.sco.com/developers/gabi/latest/contents.html is unmaintained and
> receives no update since circa 2016. So SHT_RELR is wording adopted by many
> groups, including Solaris which has many differences with the GNU ABI.

No update since 2016 does not mean unmaintained. Please speak with Cary Coutant to determine the current status.

If Solaris has adopted SHT_RELR I assume they put it into their OS-specific ABI?

> If https://groups.google.com/g/generic-abi/c/9OO5vhxb00Y "Ongoing
> Maintenance of the gABI" will take some time, perhaps the wording can be
> added to System V Application Binary Interface Linux Extensions temporarily?

It could, but why temporary? If there are objections at the ELF gABI level, and there might be, then we can just look at a Linux extension [1].

We would submit the constants in the scope of the OS-specific extensions, and then binaries with the old ABI would not match, but I don't know if that is an issue you want to address?

[1] Just for the sake of others reading, the generic Linux ABI (GNU ABI) is maintained by HJ with input from various toolchain developers: https://gitlab.com/x86-psABIs/Linux-ABI

(In reply to Carlos O'Donell from comment #6)
> (In reply to Fangrui Song from comment #5)
> > (In reply to Carlos O'Donell from comment #3)
> > > The proposal to the ABI seems stalled and probably needs to be pushed again
> > > to the gABI group with the updated wording for review again? The alternative
> > > is not to extend the gABI but consider this a GNU ABI extension. If Google
> > > has interest in moving this forward someone needs to be a change champion
> > > here and work with upstream.
> > > 
> > > Objectively I have no problem with SHT_RELR, but it should be documented in
> > > our ABI docs *somewhere* so we have a concrete place to discuss future
> > > changes and accept those changes.
> > > 
> > > Today the gABI list is a "consensus" group with Cary Coutant documenting
> > > that consensus. The GNU ABI is implemented by the GNU Toolchain, so there is
> > > probably some consensus there that needs forming, but we can do that too.
> > 
> > http://www.sco.com/developers/gabi/latest/contents.html is unmaintained and
> > receives no update since circa 2016. So SHT_RELR is wording adopted by many
> > groups, including Solaris which has many differences with the GNU ABI.
> 
> No update since 2016 does not mean unmaintained. Please speak with Cary
> Coutant to determine the current status.

I am now under contract with Xinuos to convert the documentation
to a more maintainable form and maintain it from there. Current plans
are to translate it to ReStructured Text and place it on github.

SHT_RELR is the first of the updates I plan to apply once the
conversion is done.

Thanks for forwarding!

-cary

On Thu, Jun 24, 2021 at 11:15 AM carlos at redhat dot com
<sourceware-bugzilla@sourceware.org> wrote:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=27924
>
> --- Comment #6 from Carlos O'Donell <carlos at redhat dot com> ---
> (In reply to Fangrui Song from comment #5)
> > (In reply to Carlos O'Donell from comment #3)
> > > The proposal to the ABI seems stalled and probably needs to be pushed again
> > > to the gABI group with the updated wording for review again? The alternative
> > > is not to extend the gABI but consider this a GNU ABI extension. If Google
> > > has interest in moving this forward someone needs to be a change champion
> > > here and work with upstream.
> > >
> > > Objectively I have no problem with SHT_RELR, but it should be documented in
> > > our ABI docs *somewhere* so we have a concrete place to discuss future
> > > changes and accept those changes.
> > >
> > > Today the gABI list is a "consensus" group with Cary Coutant documenting
> > > that consensus. The GNU ABI is implemented by the GNU Toolchain, so there is
> > > probably some consensus there that needs forming, but we can do that too.
> >
> > http://www.sco.com/developers/gabi/latest/contents.html is unmaintained and
> > receives no update since circa 2016. So SHT_RELR is wording adopted by many
> > groups, including Solaris which has many differences with the GNU ABI.
>
> No update since 2016 does not mean unmaintained. Please speak with Cary Coutant
> to determine the current status.
>
> If Solaris has adopted SHT_RELR I assume they put it into their OS-specific
> ABI?
>
> > If https://groups.google.com/g/generic-abi/c/9OO5vhxb00Y "Ongoing
> > Maintenance of the gABI" will take some time, perhaps the wording can be
> > added to System V Application Binary Interface Linux Extensions temporarily?
>
> It could, but why temporary? If there are objections at the ELF gABI level, and
> there might be, then we can just look at a Linux extension [1].
>
> We would submit the constants in the scope of the OS-specific extensions, and
> then binaries with the old ABI would not match, but I don't know if that is an
> issue you want to address?
>
> [1] Just for the sake of others reading, the generic Linux ABI (GNU ABI) is
> maintained by HJ with input from various toolchain developers:
> https://gitlab.com/x86-psABIs/Linux-ABI
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.

Patch: https://sourceware.org/pipermail/libc-alpha/2021-October/131768.html

in-reply-to: https://bugzilla.redhat.com/show_bug.cgi?id=2014699#c1

> * [v2] elf: Support DT_RELR relative relocation format [BZ #27924]
>  * Need official gABI assignment (Carry Coutant status?)
>   * Needs to be reserved verbally by Carry (Action item)
>   * Required for patch acceptance.

s/Carry/Cary.

https://groups.google.com/g/generic-abi/c/bX460iggiKg/m/6psrcLHLAgAJ (2018-02)
https://sourceware.org/bugzilla/show_bug.cgi?id=27924#c7 (2021-06) and
https://sourceware.org/pipermail/libc-alpha/2021-October/131781.html
are verbal by Cary.

> * Need BFD port or a released llvm/lld that supports DT_RELR to review patches.
>  * Minimally a 32-bit and 64-bit port that can build glibc and passes regression testing.
>  * Required for patch review.

LLD since 7.0.0 supports --pack-dyn-relocs=relr.
Both arm and aarch64 ports have users.

ld.lld --pack-dyn-relocs=relr produces working executables on Fuchsia, Linux kernel's arm64 port, FreeBSD-CURRENT, ChromeOS's (patched glibc) today.
Just that the produced executable segfaults on glibc due to lack of support.

> * Official glibc relese with feature?
>  * Need BFD port to support build-many-glibcs to provide testing confidence.
> * HJ: Would like to see a BFD port before a glibc release with the features.
>  * Sufficient to have a released binutils with DT_RELR also.

Wish that a GNU ld maintainer can work on this...

> * Release date of released static linkers with support should be close to glibc release with feature.
>  * When glibc is released users can pick a static linker to test with (binutils or lld).

LLD since 7.0.0 supports --pack-dyn-relocs=relr, so the release date (2018) cannot be close to the glibc version now...
Even in the absence of GNU ld support,  with DT_RELR support, users can use ld.lld --pack-dyn-relocs=relr today.

The code doesn't work when DT_RELR is enabled in ld.so:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7639766 in _dlerror_run (operate=operate@entry=0xf7639ba0 <dlopen_doit>, args=args@entry=0xffffddb0) at dlerror.c:138
138	  int errcode = GLRO (dl_catch_error) (&objname, &errstring, &malloced,
(gdb) p _rtld_local_ro._dl_catch_error
$1 = (int (*)(const char **, const char **, _Bool *, void (*)(void *), 
    void *)) 0xeffae7f0
(gdb) watch _rtld_local_ro._dl_catch_error
Hardware watchpoint 1: _rtld_local_ro._dl_catch_error
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /export/build/gnu/tools-build/glibc-gitlab-test/build-x86_64-linux/elf/global --direct
BFD: /export/build/gnu/tools-build/glibc-gitlab-test/build-x86_64-linux/elf/ld.so: unknown type [0x13] section `.relr.dyn'
warning: `/export/build/gnu/tools-build/glibc-gitlab-test/build-x86_64-linux/elf/ld.so': Shared library architecture unknown is not compatible with target architecture i386:x86-64.
warning: `/export/build/gnu/tools-build/glibc-gitlab-test/build-x86_64-linux/elf/ld.so': Shared library architecture unknown is not compatible with target architecture i386:x86-64.

Hardware watchpoint 1: _rtld_local_ro._dl_catch_error

Old value = (int (*)(const char **, const char **, _Bool *, void (*)(void *), void *)) 0x2a7f0
New value = (int (*)(const char **, const char **, _Bool *, void (*)(void *), void *)) 0xf7fec7f0 <_rtld_catch_error>
0x00007ffff7fcffb7 in _dl_start (arg=<error reading variable: Cannot access memory at address 0xffffde60>) at rtld.c:572
572	      ELF_DYNAMIC_RELOCATE (&bootstrap_map, NULL, 0, 0, 0);
(gdb) c
Continuing.
BFD: /export/build/gnu/tools-build/glibc-gitlab-test/build-x86_64-linux/elf/ld.so: unknown type [0x13] section `.relr.dyn'
warning: `/export/build/gnu/tools-build/glibc-gitlab-test/build-x86_64-linux/elf/ld.so': Shared library architecture unknown is not compatible with target architecture i386:x86-64.

Hardware watchpoint 1: _rtld_local_ro._dl_catch_error

Old value = (int (*)(const char **, const char **, _Bool *, void (*)(void *), void *)) 0xf7fec7f0 <_rtld_catch_error>
New value = (int (*)(const char **, const char **, _Bool *, void (*)(void *), void *)) 0xeffae7f0
0x00007ffff7fdae3f in _dl_relocate_object (l=l@entry=0xf7ff9250 <_rtld_local+2736>, scope=<optimized out>, reloc_mode=reloc_mode@entry=0, consider_profiling=<optimized out>, consider_profiling@entry=0) at dl-reloc.c:288
288	    ELF_DYNAMIC_RELOCATE (l, scope, lazy, consider_profiling, skip_ifunc);
(gdb) 

The problem is that ELF_DYNAMIC_DO_RELR is down twice on ld.so.  The first
time is for RTLD_BOOTSTRAP.  It needs to be fixed.

(In reply to H.J. Lu from comment #11)
> The code doesn't work when DT_RELR is enabled in ld.so:
> 
> [...]
>
> The problem is that ELF_DYNAMIC_DO_RELR is down twice on ld.so.  The first
> time is for RTLD_BOOTSTRAP.  It needs to be fixed.

I noticed this (building ld.so itself with DT_RELR) but did not investigate it. Thanks for investigation! How to fix it?

Created attachment 13891 [details]
A patch

(In reply to Fangrui Song from comment #12)
> (In reply to H.J. Lu from comment #11)
> > The code doesn't work when DT_RELR is enabled in ld.so:
> > 
> > [...]
> >
> > The problem is that ELF_DYNAMIC_DO_RELR is down twice on ld.so.  The first
> > time is for RTLD_BOOTSTRAP.  It needs to be fixed.
> 
> I noticed this (building ld.so itself with DT_RELR) but did not investigate
> it. Thanks for investigation! How to fix it?

See my patch.

(In reply to H.J. Lu from comment #14)
> (In reply to Fangrui Song from comment #12)
> > (In reply to H.J. Lu from comment #11)
> > > The code doesn't work when DT_RELR is enabled in ld.so:
> > > 
> > > [...]
> > >
> > > The problem is that ELF_DYNAMIC_DO_RELR is down twice on ld.so.  The first
> > > time is for RTLD_BOOTSTRAP.  It needs to be fixed.
> > 
> > I noticed this (building ld.so itself with DT_RELR) but did not investigate
> > it. Thanks for investigation! How to fix it?
> 
> See my patch.

Thanks. Folded into https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/maskray/relr

There is another problem.  A static PIE with DT_RELR can have
0 DT_RELA entry when all relative relocations are in DT_RELR:

[hjl@gnu-tgl-3 DT_RELR-3]$ readelf -d x-static

Dynamic section at offset 0xbbd88 contains 24 entries:
  Tag        Type                         Name/Value
 0x000000000000000c (INIT)               0x1000
 0x000000000000000d (FINI)               0x8d630
 0x0000000000000019 (INIT_ARRAY)         0xb9750
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0xb9758
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x368
 0x0000000000000005 (STRTAB)             0x3a0
 0x0000000000000006 (SYMTAB)             0x388
 0x000000000000000a (STRSZ)              1 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0xbd000
 0x0000000000000002 (PLTRELSZ)           576 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x3a8
 0x0000000000000007 (RELA)               0x0
 0x0000000000000008 (RELASZ)             0 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffb (FLAGS_1)            Flags: PIE
 0x0000000000000024 (RELR)               0x5e8
 0x0000000000000023 (RELRSZ)             312 (bytes)
 0x0000000000000025 (RELRENT)            8 (bytes)
 0x0000000000000000 (NULL)               0x0
[hjl@gnu-tgl-3 DT_RELR-3]$ 

ld.so doesn't work with 0 DT_RELA entry.

Created attachment 13892 [details]
A patch to properly handle zero DT_RELA/DT_REL values

On Fedora 35/x86-64, I saw

FAIL: nptl/test-cond-printers
FAIL: nptl/test-condattr-printers
FAIL: nptl/test-mutex-printers
FAIL: nptl/test-mutexattr-printers
FAIL: nptl/test-rwlock-printers
FAIL: nptl/test-rwlockattr-printers
FAIL: nptl/tst-pthread-gdb-attach-static

when DT_RELR is enabled on glibc itself:

$ cat nptl/test-cond-printers.out 
Error: Response does not match the expected pattern.
Command: print *condvar
Expected pattern: pthread_cond_t
Response:  Cannot access memory at address 0xffffd858
(gdb) 
$ 

Is GDB DT_RELR change needed for these tests?

It is too late for 2.35.  I'd like to support it in 2.36 with binutils 2.38
after 2.35 is branched.

(In reply to H.J. Lu from comment #18)
> On Fedora 35/x86-64, I saw
> 
> FAIL: nptl/test-cond-printers
> FAIL: nptl/test-condattr-printers
> FAIL: nptl/test-mutex-printers
> FAIL: nptl/test-mutexattr-printers
> FAIL: nptl/test-rwlock-printers
> FAIL: nptl/test-rwlockattr-printers

This is caused by PR 28757.

Resolved by some changes pushed on 2022-04-26: https://sourceware.org/pipermail/libc-alpha/2022-April/138085.html ([PATCH v11 0/7] Support DT_RELR relative relocation format)