Initially noticed as a nodejs-14.17.3 link failure against gcc-HEAD+binutils-2.37 on x86_64-linux. I did not manage to extract minimal example yet, but was able to bisect binutils down to seemingly relevant commit 7a30ac441a4e776e68399cb916c4731b5192e931 Author: H.J. Lu <hjl.tools@gmail.com> Date: Tue Jul 6 06:21:54 2021 -0700 ld: Check archive only for archive member Since plugin_maybe_claim calls bfd_close on the original input BFD if it isn't an archive member, pass NULL to bfd_plugin_close_file_descriptor to indicate that the BFD isn't an archive member. bfd/ PR ld/18028 * plugin.c (bfd_plugin_close_file_descriptor): Check archive only of abfd != NULL. (try_claim): Pass NULL to bfd_plugin_close_file_descriptor if it isn't an archive member. ld/ PR ld/18028 * plugin.c (plugin_input_file): Add comments for abfd and ibfd. (plugin_object_p): Set input->ibfd to NULL if it isn't an archive member. bfd/plugin.c | 25 +++++++++++++------------ ld/plugin.c | 4 +++- 2 files changed, 16 insertions(+), 13 deletions(-) Looks relevant. Build failure is visible as: x86_64-pc-linux-gnu-g++ -o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/mksnapshot -pthread -rdynamic -m64 -m64 -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -Wl,--start-group /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/embedded-empty.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/embedded-file-writer.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-aix.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-base.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-generic.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-mac.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-win.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/mksnapshot.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/snapshot-empty.o /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_base_without_compiler.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_init.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_libbase.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_libplatform.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/icu/libicui18n.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/icu/libicuucx.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/icu/libicudata.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_zlib.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_compiler.a /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_initializers.a -lz -luv -lbrotlidec -lbrotlienc -lcares -lnghttp2 -lcrypto -lssl -ldl -lrt -Wl,--end-group /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_compiler.a: error adding symbols: malformed archive collect2: error: ld returned 1 exit status The special thing here is lib*.a thin archives, built as: x86_64-pc-linux-gnu-ar crsT /dev/shm/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/tools/v8_gypfiles/libv8_compiler.a /dev/shm/portage/... I'll attach full build.log if it's of any help to craft artificial example.
Created attachment 13566 [details] net-libs:nodejs-16.5.0:20210725-155525.log.xz
I used the following reproducer linker script for bisection. Fun fact: If I comment out all the -plugin/-plugin-opt= flags the link will succeed. $ cat repro.bash #!/bin/bash cmd=( # bisect /home/slyfox/dev/git/binutils-gdb-bisect/ld/ld-new #/usr/x86_64-pc-linux-gnu/binutils-bin/2.37/ld #works: #/usr/x86_64-pc-linux-gnu/binutils-bin/2.36.1/ld # somehow these are needed to trigger a bug: -plugin /usr/libexec/gcc/x86_64-pc-linux-gnu/12.0.0/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/x86_64-pc-linux-gnu/12.0.0/lto-wrapper -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -L /usr/lib/gcc/x86_64-pc-linux-gnu/12.0.0/ -L /usr/lib64 --eh-frame-hdr -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -o mksnapshot /usr/lib64/Scrt1.o /usr/lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/12.0.0/crtbeginS.o -O1 --hash-style=gnu --defsym=__gentoo_check_ldflags__=0 --start-group /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_initializers.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/embedded-empty.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/embedded-file-writer.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-aix.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-base.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-generic.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-mac.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-win.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/mksnapshot.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/mksnapshot/deps/v8/src/snapshot/snapshot-empty.o /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_base_without_compiler.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_init.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_libbase.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_libplatform.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/icu/libicui18n.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/icu/libicuucx.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/icu/libicudata.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_libsampler.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_zlib.a /tmp/portage/net-libs/nodejs-14.17.3/work/node-v14.17.3/out/Release/obj.target/tools/v8_gypfiles/libv8_compiler.a -lz -luv -lbrotlidec -lbrotlienc -lcares -lnghttp2 -lcrypto -lssl -ldl -lrt --end-group -lstdc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-pc-linux-gnu/12.0.0/crtendS.o /usr/lib64/crtn.o ) "${cmd[@]}" "$@"
Does it fail with GCC 11?
Yes, the same failure happens on gcc-11.1.0 + binutils-2.37. I hoped that the failure is immediately obvious. I'll spend some time in attempt to extract minimal example from nodejs for easier exploration.
I can't reproduce it with GCC 12.
(In reply to Sergei Trofimovich from comment #4) > Yes, the same failure happens on gcc-11.1.0 + binutils-2.37. I hoped that > the failure is immediately obvious. > GCC 11 also works for me on node commit: commit eaca13ddc9092e831b9b45180c6e0ee1f3292e5e (HEAD, tag: v16.5.0) Author: Michaël Zasso <targos@protonmail.com> Date: Tue Jul 13 08:33:13 2021 +0200 2021-07-14, Version 16.5.0 (Current)
I think I got closer to the problem: it's a bad error handling related to -EMFILE (too many open files). nodejs is probably just very close to open file limit (default 1023): $ strace -f -olog -etrace=execve,openat ./mk.bash ... 684205 execve("/usr/lib/gcc/x86_64-pc-linux-gnu/12.0.0/../../../../x86_64-pc-linux-gnu/bin/ld", ... 684205 openat(AT_FDCWD, "/tmp/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/v8_compiler/deps/v8/src/compiler/simd-scalar-lowering.o", O_RDONLY) = 10 23 684205 openat(AT_FDCWD, "/tmp/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/v8_compiler/deps/v8/src/compiler/simd-scalar-lowering.o", O_RDONLY) = 89 8 684205 openat(AT_FDCWD, "/tmp/portage/net-libs/nodejs-16.5.0/work/node-v16.5.0/out/Release/obj.target/v8_compiler/deps/v8/src/compiler/backend/instruction-scheduler.o", O_RDO NLY) = -1 EMFILE (Too many open files) 684205 +++ exited with 1 +++ Don't yet know why EMFILE handling did not kick in.
Here is the full reproducer: 1200 interdependent files required for a thin archive. gcc-11.1.0 + binutils-2.36.1: $ ./mk.bash thin archive for 1200 176 gcc-11.1.0 + binutils-2.37: $ ./mk.bash thin archive for 1200 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: //libthin.a: error adding symbols: malformed archive Full script: $ cat mk.bash #!/bin/bash rm -rf -- libthin.a main.o m s mkdir -p s cc=x86_64-pc-linux-gnu-gcc ar=x86_64-pc-linux-gnu-ar nfiles=${1-1200} for i in `seq 1 ${nfiles}`; do prev_i=$((i - 1)) printf "extern int a${prev_i}(void); int a${i}(void) { return 1 + a${prev_i}(); }" > s/a${i}.c $cc -c s/a${i}.c -o s/a${i}.o done printf "int a0(void) { return 0; }; extern int a${nfiles}(void); int main() { return a${nfiles}(); }" > main.c $cc -c main.c -o main.o # thin archive echo "thin archive for $nfiles" >&2 $ar crsT libthin.a s/a*.o $cc -pthread -o m -rdynamic -m64 main.o ${PWD}/libthin.a ./m; echo $?
A patch is posted at https://sourceware.org/pipermail/binutils/2021-July/117502.html
(In reply to H.J. Lu from comment #9) > A patch is posted at > > https://sourceware.org/pipermail/binutils/2021-July/117502.html The patch works for nodejs. Thank you!
The master branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=5b37a5ca1d2e90ea1fc29593046428f7ed116a7f commit 5b37a5ca1d2e90ea1fc29593046428f7ed116a7f Author: H.J. Lu <hjl.tools@gmail.com> Date: Mon Jul 26 05:37:57 2021 -0700 bfd: Set error to bfd_error_malformed_archive only if unset When reading an archive member, set error to bfd_error_malformed_archive on open_nested_file failure only if the error is unset. PR ld/28138 * archive.c (_bfd_get_elt_at_filepos): Don't set error to bfd_error_malformed_archive if it has been set.
The master branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=5a98fb7513b559e20dfebdbaa2a471afda3b4742 commit 5a98fb7513b559e20dfebdbaa2a471afda3b4742 Author: H.J. Lu <hjl.tools@gmail.com> Date: Mon Jul 26 05:59:55 2021 -0700 bfd: Close the file descriptor if there is no archive fd Close the file descriptor if there is no archive plugin file descriptor to avoid running out of file descriptors on thin archives with many archive members. bfd/ PR ld/28138 * plugin.c (bfd_plugin_close_file_descriptor): Close the file descriptor there is no archive plugin file descriptor. ld/ PR ld/28138 * testsuite/ld-plugin/lto.exp: Run ld/28138 tests. * testsuite/ld-plugin/pr28138.c: New file. * testsuite/ld-plugin/pr28138-1.c: Likewise. * testsuite/ld-plugin/pr28138-2.c: Likewise. * testsuite/ld-plugin/pr28138-3.c: Likewise. * testsuite/ld-plugin/pr28138-4.c: Likewise. * testsuite/ld-plugin/pr28138-5.c: Likewise. * testsuite/ld-plugin/pr28138-6.c: Likewise. * testsuite/ld-plugin/pr28138-7.c: Likewise.
Fixed on master branch so far.
Wrong version committed, I think. aarch64-linux +FAIL: PR ld/28138 alpha-linux +FAIL: PR ld/28138 arm-linuxeabi +FAIL: PR ld/28138 hppa-linux +FAIL: PR ld/28138 ia64-linux +FAIL: PR ld/28138 m68k-linux +FAIL: PR ld/28138 microblaze-linux +FAIL: PR ld/28138 mips64-linux +FAIL: PR ld/28138 mips-linux +FAIL: PR ld/28138 powerpc64le-linux +FAIL: PR ld/28138 powerpc64-linux +FAIL: PR ld/28138 powerpc-linux +FAIL: PR ld/28138 riscv64-linux +FAIL: PR ld/28138 s390-linux +FAIL: PR ld/28138 s390x-linux +FAIL: PR ld/28138 sparc64-linux +FAIL: PR ld/28138 tilepro-linux +FAIL: PR ld/28138
The master branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=7dc37e1e1209c80e0bab784df6b6bac335e836f2 commit 7dc37e1e1209c80e0bab784df6b6bac335e836f2 Author: H.J. Lu <hjl.tools@gmail.com> Date: Wed Jul 28 07:32:40 2021 -0700 ld: Run tmpdir/pr28138 only for native build * PR ld/28138 * testsuite/ld-plugin/lto.exp: Run tmpdir/pr28138 only for native build.
The staging-2.37 branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=1c611b40e6bfc8029bff7696814330b5bc0ee5c0 commit 1c611b40e6bfc8029bff7696814330b5bc0ee5c0 Author: H.J. Lu <hjl.tools@gmail.com> Date: Mon Jul 26 05:59:55 2021 -0700 bfd: Close the file descriptor if there is no archive fd Close the file descriptor if there is no archive plugin file descriptor to avoid running out of file descriptors on thin archives with many archive members. bfd/ PR ld/28138 * plugin.c (bfd_plugin_close_file_descriptor): Close the file descriptor there is no archive plugin file descriptor. ld/ PR ld/28138 * testsuite/ld-plugin/lto.exp: Run tmpdir/pr28138 only for native build. PR ld/28138 * testsuite/ld-plugin/lto.exp: Run ld/28138 tests. * testsuite/ld-plugin/pr28138.c: New file. * testsuite/ld-plugin/pr28138-1.c: Likewise. * testsuite/ld-plugin/pr28138-2.c: Likewise. * testsuite/ld-plugin/pr28138-3.c: Likewise. * testsuite/ld-plugin/pr28138-4.c: Likewise. * testsuite/ld-plugin/pr28138-5.c: Likewise. * testsuite/ld-plugin/pr28138-6.c: Likewise. * testsuite/ld-plugin/pr28138-7.c: Likewise. (cherry picked from commit 5a98fb7513b559e20dfebdbaa2a471afda3b4742) (cherry picked from commit 7dc37e1e1209c80e0bab784df6b6bac335e836f2)
The binutils-2_37-branch branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=1c611b40e6bfc8029bff7696814330b5bc0ee5c0 commit 1c611b40e6bfc8029bff7696814330b5bc0ee5c0 Author: H.J. Lu <hjl.tools@gmail.com> Date: Mon Jul 26 05:59:55 2021 -0700 bfd: Close the file descriptor if there is no archive fd Close the file descriptor if there is no archive plugin file descriptor to avoid running out of file descriptors on thin archives with many archive members. bfd/ PR ld/28138 * plugin.c (bfd_plugin_close_file_descriptor): Close the file descriptor there is no archive plugin file descriptor. ld/ PR ld/28138 * testsuite/ld-plugin/lto.exp: Run tmpdir/pr28138 only for native build. PR ld/28138 * testsuite/ld-plugin/lto.exp: Run ld/28138 tests. * testsuite/ld-plugin/pr28138.c: New file. * testsuite/ld-plugin/pr28138-1.c: Likewise. * testsuite/ld-plugin/pr28138-2.c: Likewise. * testsuite/ld-plugin/pr28138-3.c: Likewise. * testsuite/ld-plugin/pr28138-4.c: Likewise. * testsuite/ld-plugin/pr28138-5.c: Likewise. * testsuite/ld-plugin/pr28138-6.c: Likewise. * testsuite/ld-plugin/pr28138-7.c: Likewise. (cherry picked from commit 5a98fb7513b559e20dfebdbaa2a471afda3b4742) (cherry picked from commit 7dc37e1e1209c80e0bab784df6b6bac335e836f2)
Fixed for 2.38 and 2.37 branch.
The master branch has been updated by H.J. Lu <hjl@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=0d81afad67091936f75457fac19f6c136669c2b8 commit 0d81afad67091936f75457fac19f6c136669c2b8 Author: H.J. Lu <hjl.tools@gmail.com> Date: Wed Aug 11 05:53:56 2021 -0700 ld: Update the pass and fail strings of PR ld/28138 test PR ld/28138 * testsuite/ld-plugin/lto.exp: Update the pass and fail strings of PR ld/28138 test to indicate which part of the test passed and failed.
*** Bug 24723 has been marked as a duplicate of this bug. ***