Created attachment 9249 [details] testcase to reproduce LLVMgold.so Unexpected resolution failure on ld I'm enabling clang LTO feature to to improve code size of Uefi standard (http://www.uefi.org/) firmware (https://github.com/tianocore/edk2). My project is in https://github.com/shijunjing/edk2 branch llvm : https://github.com/shijunjing/edk2/tree/llvm. I find ld in binutils 2.26 cannot co-work correctly with LLVM gold plugin (LLVMgold.so), and the LLVM gold plugin will report unexpected resolution failure as below when ld do the optimization. But this failure will not happen on gold. Unexpected resolution UNREACHABLE executed at /home/jshi19/llvm-3.8.0.src/tools/gold/gold-plugin.cpp:679! Below is the steps on how to setup clang compiler and reproduce the clang LTO link failure: 1. Download and extract the llvm 3.8.0 Pre-Built Binaries from http://www.llvm.org/releases/ (e.g. http://www.llvm.org/releases/3.8.0/clang+llvm-3.8.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz and extract it as ~/clang38). 2. Decompress and rename the debug version LLVM gold plugin in https://github.com/shijunjing/edk2/blob/llvm/BaseTools/Bin/LLVMgold-debug.tar.gz as LLVMgold.so. Copy it to above clang lib folder (e.g. ~/clang38/lib/LLVMgold.so) 3. Copy GNU Binutils 2.26 linker ld to /usr/bin/ld 4. run below clang LTO link command: ~/clang38/bin/clang -o Hello.dll -flto -nostdlib -Wl,-n -Wl,-q -Wl,--gc-sections -Wl,-z,common-page-size=0x40 -Wl,--entry,_ModuleEntryPoint -Wl,-u,_ModuleEntryPoint -Wl,-Map,Hello.map -Wl,-melf_x86_64 -Wl,--oformat=elf64-x86-64 -Wl,--start-group,,@static_library_files.lst -Wl,--end-group ld 2.26 fail in gold plugin optimization with below output: steven: getModuleForFile pass Unexpected resolution UNREACHABLE executed at /home/jshi19/llvm-3.8.0.src/tools/gold/gold-plugin.cpp:679! clang-3.8: error: unable to execute command: Aborted (core dumped) clang-3.8: error: linker command failed due to signal (use -v to see invocation) 5. Copy GNU Binutils 2.26 gold linker ld-new to /usr/bin/ld 6. re-run step 4 clang LTO link command again: ~/clang38/bin/clang -o Hello.dll -flto -nostdlib -Wl,-n -Wl,-q -Wl,--gc-sections -Wl,-z,common-page-size=0x40 -Wl,--entry,_ModuleEntryPoint -Wl,-u,_ModuleEntryPoint -Wl,-Map,Hello.map -Wl,-melf_x86_64 -Wl,--oformat=elf64-x86-64 -Wl,--start-group,,@static_library_files.lst -Wl,--end-group gold pass gold plugin optimization with below output: steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass steven: getModuleForFile pass /usr/bin/ld: internal error in do_layout, at ../../binutils-2.26/gold/object.cc:1819 clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation) Above gold do_layout issue is a known bug which is not related LLVM LTO, and has been tracked in Bug 20062 https://sourceware.org/bugzilla/show_bug.cgi?id=20062 The "steven: getModuleForFile pass" is debug message I especially added in the debug version LLVMgold.so, which is line 905 in llvm-3.8.0.src\tools\gold\gold-plugin.cpp. I have a gold-plugin.cpp copy in this folder. This folder example libraries are abstracted from Uefi Hello module. You can build them by yourself from https://github.com/shijunjing/edk2/tree/llvm 0. Download and extract the llvm 3.8.0 Pre-Built Binaries from http://www.llvm.org/releases/ (e.g. http://www.llvm.org/releases/3.8.0/clang+llvm-3.8.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz and extract it as ~/clang38). 0. Decompress and rename the debug version LLVM gold plugin in https://github.com/shijunjing/edk2/blob/llvm/BaseTools/Bin/LLVMgold-debug.tar.gz as LLVMgold.so. Copy it to above clang lib folder (e.g. ~/clang38/lib/LLVMgold.so) 1. Setup EDK2 build environment as steps in the link: https://github.com/tianocore/tianocore.github.io/wiki/Using-EDK-II-with-Native-GCC 2. git clone https://github.com/shijunjing/edk2 (e.g. ~/edk2) 3. $ cd edk2 4. $ git checkout llvm 5. $ export CLANG38_BIN=path/to/your/clang38/ (e.g. export CLANG38_BIN=~/clang38/bin/) 6. $ source edksetup.sh 7. $ make -C BaseTools/Source/C 8. Comment out the line 33, 34 in ~/edk2/AppPkg/Applications/Hello/Hello.c, which will trigger significant LTO optimization. I have a updated Hello.c copy in this folder. 9. $ build -t CLANGLTO38 -a X64 -p AppPkg/AppPkg.dsc -m AppPkg/Applications/Hello/Hello.inf After build, you can find all intermediate files in below folder ~/edk2/Build/AppPkg/DEBUG_CLANGLTO38/X64/AppPkg/Applications/Hello/Hello
Created attachment 9253 [details] A patch Please try this.
Hi H.J. Your patch works for me. Thank you!
The master branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=3355cb3b643bd50aafae768e7cf990d4bec40fe1 commit 3355cb3b643bd50aafae768e7cf990d4bec40fe1 Author: H.J. Lu <hjl.tools@gmail.com> Date: Thu May 12 16:50:34 2016 -0700 Handle symbols defined/referenced only within IR The plugin is called to claim symbols in an archive element from plugin_object_p. But those symbols aren't needed to create output. They are defined and referenced only within IR. get_symbols should return resolution based on IR symbol kinds. PR ld/20070 * Makefile.am (noinst_LTLIBRARIES): Add libldtestplug4.la. (libldtestplug4_la_SOURCES): New. (libldtestplug4_la_CFLAGS): Likewise. (libldtestplug4_la_LDFLAGS): Likewise. * Makefile.in: Regenerated. * plugin.c (get_symbols): Return resolution based on IR symbol kinds for symbols defined/referenced only within IR. * testplug4.c: New file. * ld/testsuite/ld-plugin/pr20070.d: Likewise. * ld/testsuite/ld-plugin/pr20070a.c: Likewise. * ld/testsuite/ld-plugin/pr20070b.c: Likewise. * testsuite/ld-plugin/plugin.exp (plugin4_name): New. (plugin4_path): Likewise. Add a test for ld/20070.
Fixed for 2.27.
*** Bug 20090 has been marked as a duplicate of this bug. ***
Hello, You know I’m debugging and I hope to disable all optimizations in the LTO firstly, then enable them one by one later. But when I try to disable the optimization by enforcing –O0 in the LTO build, I find the ld fails to recognize some clang bitcode library, and fail to link. e.g. use the Clang_LTO_Fails_On_LD example in below bug attachment https://sourceware.org/bugzilla/show_bug.cgi?id=20070 If I enforce the –O0 to disable the optimization in LTO, the ld fail to link: ~/clang38/bin/clang -o Hello.dll -flto -O0 -nostdlib -Wl,-n -Wl,-q -Wl,--gc-sections -Wl,-z,common-page-size=0x40 -Wl,--entry,_ModuleEntryPoint -Wl,-u,_ModuleEntryPoint -Wl,-Map,Hello.map -Wl,-melf_x86_64 -Wl,--oformat=elf64-x86-64 -Wl,--start-group,,@static_library_files.lst -Wl,--end-group BaseLib.lib: error adding symbols: File format not recognized clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation) But if I enable the –O1, the ld link pass: ~/clang38/bin/clang -o Hello.dll -flto –O1 -nostdlib -Wl,-n -Wl,-q -Wl,--gc-sections -Wl,-z,common-page-size=0x40 -Wl,--entry,_ModuleEntryPoint -Wl,-u,_ModuleEntryPoint -Wl,-Map,Hello.map -Wl,-melf_x86_64 -Wl,--oformat=elf64-x86-64 -Wl,--start-group,,@static_library_files.lst -Wl,--end-group So, I cannot correctly disable the the optimization in ld LTO. Please help to fix it. Thanks!
Created attachment 9264 [details] A patch Try this. But this doesn't fix llvm LTO bug. I got /tmp/lto-llvm-2a2442.o: In function `UefiDevicePathLibDuplicateDevicePath': /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:394: undefined reference to `GetDevicePathSize' /tmp/lto-llvm-2a2442.o: In function `UefiDevicePathLibAppendDevicePath': /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:447: undefined reference to `DuplicateDevicePath' /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:451: undefined reference to `DuplicateDevicePath' /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:462: undefined reference to `GetDevicePathSize' /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:463: undefined reference to `GetDevicePathSize' /tmp/lto-llvm-2a2442.o: In function `UefiDevicePathLibAppendDevicePathNode': /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:522: undefined reference to `DuplicateDevicePath' /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:542: undefined reference to `AppendDevicePath' /tmp/lto-llvm-2a2442.o: In function `UefiDevicePathLibAppendDevicePathInstance': /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:585: undefined reference to `DuplicateDevicePath' /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:596: undefined reference to `GetDevicePathSize' /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:597: undefined reference to `GetDevicePathSize' /tmp/lto-llvm-2a2442.o: In function `UefiDevicePathLibGetNextDevicePathInstance': /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:684: undefined reference to `DuplicateDevicePath' /tmp/lto-llvm-2a2442.o: In function `FileDevicePath': /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:872: undefined reference to `AppendDevicePath' /tmp/lto-llvm-2a2442.o: In function `CatVSPrint': /home/jshi19/edk2-fork/MdePkg/Library/UefiDevicePathLib/DevicePathUtilities.c:876: undefined reference to `StrCpyS' BasePrintLib.lib(PrintLib.obj): In function `UnicodeVSPrint': /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:71: undefined reference to `DebugAssertEnabled' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:71: undefined reference to `DebugAssert' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:72: undefined reference to `DebugAssert' BasePrintLib.lib(PrintLib.obj): In function `UnicodeVSPrintAsciiFormat': /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:218: undefined reference to `DebugAssertEnabled' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:218: undefined reference to `DebugAssert' BasePrintLib.lib(PrintLib.obj): In function `SPrintLength': /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:730: undefined reference to `DebugAssertEnabled' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:730: undefined reference to `DebugAssert' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLib.c:731: undefined reference to `DebugAssert' BasePrintLib.lib(PrintLibInternal.obj): In function `BasePrintLibSPrintMarker': /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:366: undefined reference to `DebugAssertEnabled' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:366: undefined reference to `DebugAssert' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:397: undefined reference to `DebugAssertEnabled' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:405: undefined reference to `AsciiStrSize' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:405: undefined reference to `DebugAssert' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:397: undefined reference to `StrSize' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:397: undefined reference to `DebugAssert' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:701: undefined reference to `ReadUnaligned32' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:702: undefined reference to `ReadUnaligned16' BasePrintLib.lib(PrintLibInternal.obj): In function `BasePrintLibValueToString': /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:129: undefined reference to `DivU64x32Remainder' BasePrintLib.lib(PrintLibInternal.obj): In function `BasePrintLibSPrintMarker': /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:397: undefined reference to `DebugAssertEnabled' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:973: undefined reference to `DebugAssert' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:982: undefined reference to `StrSize' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:982: undefined reference to `DebugAssert' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:987: undefined reference to `AsciiStrSize' /home/jshi19/edk2-fork/MdePkg/Library/BasePrintLib/PrintLibInternal.c:987: undefined reference to `DebugAssert' clang-3.9: error: linker command failed with exit code 1 (use -v to see invocation)
Hi H.J. Thank you for the quick fix. How can I know what different optimization used by the ld between -O0 and -O1? Is the compiler domain optimization (e.g. clang/llvm) or the linker (e.g. ld) domain optimization make the difference? How can I enable or disable these specific optimization besides using -O0 or -O1?
(In reply to Steven Shi from comment #8) > Hi H.J. > Thank you for the quick fix. How can I know what different optimization used > by the ld between -O0 and -O1? Is the compiler domain optimization (e.g. > clang/llvm) or the linker (e.g. ld) domain optimization make the difference? > How can I enable or disable these specific optimization besides using -O0 or > -O1? -O0 and -O1 are for llvm, not for ld. This LTO bug is in llvm.
LLVM guys say "LTO is linker specific, clang is only forwarding the option to the linker here." see below discussion about "How to debug if LTO generate wrong code?": http://lists.llvm.org/pipermail/cfe-dev/2016-May/048906.html So it looks, if I compile my llvm bitcode without any optimization level (-O0) and pass them to linker to finish the LTO build, all the optimizations are done only by the linker, not llvm. Is it right?
(In reply to Steven Shi from comment #10) > LLVM guys say "LTO is linker specific, clang is only forwarding the option > to the linker here." see below discussion about "How to debug if LTO > generate wrong code?": > http://lists.llvm.org/pipermail/cfe-dev/2016-May/048906.html > > So it looks, if I compile my llvm bitcode without any optimization level > (-O0) and pass them to linker to finish the LTO build, all the optimizations > are done only by the linker, not llvm. > > Is it right? LTO stands for link-time optimization. -On is passed back to llvm LTO plugin.
The master branch has been updated by Rainer Orth <ro@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=3e97ba7d583055bdd5439dd300c59a2f5bc02476 commit 3e97ba7d583055bdd5439dd300c59a2f5bc02476 Author: Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> Date: Tue Apr 7 16:52:03 2020 +0200 ld: Fix several 32-bit SPARC plugin tests Several ld plugin tests currently FAIL on 32-bit Solaris/SPARC: FAIL: load plugin with source FAIL: plugin claimfile lost symbol with source FAIL: plugin claimfile replace symbol with source FAIL: plugin claimfile resolve symbol with source FAIL: plugin claimfile replace file with source FAIL: plugin set symbol visibility with source FAIL: plugin ignore lib with source FAIL: plugin claimfile replace lib with source FAIL: plugin 2 with source lib FAIL: load plugin 2 with source FAIL: load plugin 2 with source and -r FAIL: plugin 3 with source lib FAIL: load plugin 3 with source FAIL: load plugin 3 with source and -r FAIL: PR ld/20070 all of them in the same way: ./ld-new: BFD (GNU Binutils) 2.34.50.20200328 internal error, aborting at /vol/src/gnu/binutils/hg/master/git/bfd/elf32-sparc.c:154 in sparc_final_write_processing This happens when bfd_get_mach returns 0 when abfd refers to a source file: $11 = { filename = 0x28c358 "/vol/src/gnu/binutils/hg/master/local/ld/testsuite/ld-plugin/func.c (symbol from plugin)", xvec = 0x24ed6c <sparc_elf32_sol2_vec>, [...] While I could find no specification what abfd's are allowed/expected in *_final_write_processing, I could find no other target that behaved the same. And indeed ignoring the 0 case fixes the failures. The code now errors for other values. 64-bit SPARC is not affected because it doesn't have a specific implementation of elf_backend_final_write_processing. Tested on sparc-sun-solaris2.11. 2020-04-07 Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> Nick Clifton <nickc@redhat.com> * elf32-sparc.c (sparc_final_write_processing): Fix whitespace. <0>: Ignore. <default>: Error rather than abort.
As per comment #4. Comment #7 patch was committed too.