lld status with powerpc64

Sun Nov 7 14:24:39 GMT 2021

Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.

Thanks!
Bill

On 11/5/21 8:58 AM, Adhemerval Zanella via Libc-alpha wrote:
>
> On 05/11/2021 04:23, Fangrui Song wrote:
>> On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>>> Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:
>>>
>>>> Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>>>>
>>>>> On 2021-10-26, Adhemerval Zanella wrote:
>>>>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>>>>     from
>>>>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>>>>  #3  0x0000000000000000 in ?? ()
>>>>>>  (gdb) disas
>>>>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>>>> This is a pla, but this GDB isn't able to disassemble it.  This instruction
>>>> shouldn't be used unless when configuring glibc using --with-cpu=power10.
>>> I can reproduce this issue even when configuring glibc with
>>> --with-cpu=power9 --disable-multi-arch, which means the build should not have
>>> any Power10 instructions.
>>>
>>> -- 
>>> Tulio Magno
>> __gep_setup___vmx__sigjmp_save means a r12 setup stub in ld.lld and is
>> used with R_PPC64_REL24_NOTOC for a non-PLT branch.
>>
>> Clang only emits `bl foo@notoc` with -mcpu=power10.
>> However, sysdeps/powerpc/powerpc64/configure.ac enables USE_PPC64_NOTOC
>> when the assembler (gas) supports it.
>>
>>
>> Adhemerval noticed that ld.lld has a behavior difference from GNU ld:
>> ld.lld defaults to PC-relative paddi stub while GNU ld doesn't (like
>> --power10-stubs=no).
>> Is USE_PPC64_NOTOC supposed to be used when targeting POWER9 and below?
>> If yes, the ld.lld default should be fixed.
> I figure out the issue and both bfd, gold, and lld align themselves on the
> power10-stubs options handling.  Using --power10-stubs without an arg is 
> equivalent to --power10-stubs=yes, but not specifying --power10-stubs at
> all should be equivalent to --power10-stubs=auto.  It somewhat confusing,
> but I think it is to allow linker and compiler to be independently
> regarding power10 stub generation.
>
> The issue is bfd enables power10 relocs generation on stubs iff it sees
> the new pc-relative relocations (for instance R_PPC64_D34), otherwise 
> it generates default stubs (ppc64_elf_check_relocs:4700).
>
> So we have two options here:
>
>   1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
>      aims to support such behavior).
>
>   2. Define USE_PPC64_NOTOC iff linker supports such optimization.
>      It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
>      link a simple binary without any pcrel and check if the stub has 
>      power10 instruction.
>
>   3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
>      and on the syscall definition to call the __syscall_error.
>
> I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC 
> manually on config.h when configuring with lld I can build glibc.
>
>>
>> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
>> USE_PPC64_NOTOC is undefined.
>>
>> % gcc --version
>> gcc (Debian 8.3.0-6) 8.3.0
>> ...
>> % ld.bfd --version
>> GNU ld (GNU Binutils for Debian) 2.31.1
>> ...
>>
>> I only see 8 more FAILs with ld.lld than ld.bfd
> I am still having trouble to *finish* make check with lld release 13
> my environment (some tests stuck on infinite loop).
>
>>
>> % diff -u 0 1
>> --- 0   2021-11-05 00:11:43.218731302 -0700
>> +++ 1   2021-11-05 00:11:37.659286448 -0700
>> @@ -9,6 +9,14 @@
>>  FAIL: debug/tst-lfschk6
>>  FAIL: dlfcn/bug-atexit3
>>  FAIL: elf/check-abi-libc
>> +FAIL: elf/ifuncmain1pic
>> +FAIL: elf/ifuncmain1pie
>> +FAIL: elf/ifuncmain1vis
>> +FAIL: elf/ifuncmain1vispic
>> +FAIL: elf/ifuncmain1vispie
>> +FAIL: elf/ifuncmain3
>> +FAIL: elf/ifuncmain6pie
>> +FAIL: elf/tst-tlsopt-powerpc
>>  FAIL: nptl/tst-cancel24
>>  FAIL: nptl/tst-minstack-throw
>>  FAIL: nptl/tst-once5
>>
>>
>> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
>> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
>> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
>> But perhaps Adhemerval can look a look at it.
> I still need to understand why armhf also fails with the ifunc using
> protected symbols, although I am not if they are related to IRELATIVE
> ordering.
>
>>
>> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
>> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
>> Let me send a configure patch to disable it...
>>
>> Actually I do not know how to disable tst-tlsopt-powerpc properly.
>>
>> Perhaps add sysdeps/powerpc/configure.ac and move the
>> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
>> to sysdeps/powerpc/configure.ac?
>> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
>>
>> The patch requires some non-trivial configure.ac change, so I'd hope
>> that an expert can do it 
> Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
> by the linker?
>
>
>>
>> Hey, so lld linked glibc powerpc64 is in a pretty good status!
> I am still not sure about it, I did could run some tests but I am still
> struggling to get a make check to finish.