See this build log: https://663690.bugs.gentoo.org/attachment.cgi?id=543520 That is, the failing command is: "i686-pc-linux-gnu-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m32" "-L" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build0-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build1-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build10-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build11-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build12-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build13-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build2-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build3-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build4-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build5-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build6-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build7-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build8-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.build_script_build9-77d5a85e30ae1e45c94134e00546b38.rs.rcgu.o" "-o" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/build/failure_derive-162cd16feb0dec2f/build_script_build-162cd16feb0dec2f.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/var/tmp/portage/dev-util/cargo-0.29.0/work/cargo-0.29.0/target/release/deps" "-L" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libstd-b064191cfffad400.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libpanic_unwind-bff7f3c6aca0eab3.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liballoc_jemalloc-e8490267764d2751.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libunwind-506dc971f8656a43.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liballoc_system-bc637fdb4bbf1bad.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liblibc-2d557e68cf8813b1.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liballoc-60a04a28d006d26c.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libcore-4d67520eec7d6412.rlib" "-Wl,--end-group" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libcompiler_builtins-879d39e0292e5929.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util" This could be triggered by something in the Rust code generation. I have already talked to the Rust compiler folks, but we couldn't understand the assertion context enough to be able to debug this further. This triggers somewhat reproducibly for one particular user on 32-bit x86 -- I have not heard of other users being affected. They are on a machine with 3 GB of memory, which I suppose might be less than most users.
If the assertion failure is this one ASSERT (entry->the_bfd->link.next == NULL); then a failure means you have the linked list of input files broken somehow. A memory bit flip would do that, so if the machine isn't rock solid otherwise I would suspect the hardware.
Yes, that appears to be the one. This is reproducible (happens every time), and the user has seen it on two separate machines -- one running on bare metal, the other a VM running on a machine with ECC memory. So it seems unlikely this is a hardware problem.
I've just had exactly the same error. Building Firefox 60.2.0 in a 32 bit systemd-nspawn Gentoo (again) container on Linux. A second attempt failed in exactly the same way. The previous build of firefox-52.9.0 succeeded on 2018-06-30. Everything works fine in 64 bit. 8 4:59.15 Compiling cose v0.1.4 8 4:59.95 error: linking with `/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/build/cargo-linker` failed: exit code: 1 8 4:59.95 | 8 4:59.96 = note: "/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/build/cargo-linker" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m32" "-L" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib" "/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/ff/toolkit/library/release/build/cose-07f74104284ed8a2/build_script_build-07f74104284ed8a2.build_script_build0.rcgu.o" "-o" "/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/ff/toolkit/library/release/build/cose-07f74104284ed8a2/build_script_build-07f74104284ed8a2" "/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/ff/toolkit/library/release/build/cose-07f74104284ed8a2/build_script_build-07f74104284ed8a2.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/ff/toolkit/library/release/deps" "-L" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libstd-3a04e5487357b9dc.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libpanic_unwind-75baa38f22cc081b.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liballoc_jemalloc-cc6cad7f4861198e.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libunwind-0df808026f9bff72.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liballoc_system-5885ff2e63807fd7.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liblibc-4b46c18a44d67980.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/liballoc-22af9bc8b767b147.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libstd_unicode-46e7ab27ec583312.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libcore-b4b9c53c65e9ed4f.rlib" "/usr/lib/rustlib/i686-unknown-linux-gnu/lib/libcompiler_builtins-0bcc823f82eb498d.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util" 8 4:59.96 = note: /usr/lib/gcc/i686-pc-linux-gnu/6.4.0/../../../../i686-pc-linux-gnu/bin/ld: internal error /var/tmp/portage/sys-devel/binutils-2.30-r2/work/binutils-2.30/ld/ldlang.c 6635 8 4:59.96 collect2: error: ld returned 1 exit status 8 4:59.96 8 4:59.96 8 4:59.96 error: aborting due to previous error 8 4:59.96 8 5:00.16 error: Could not compile `cose`. 8 5:00.16 8 5:00.16 To learn more, run the command again with --verbose. 8 5:00.17 gmake[4]: *** [/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/config/rules.mk:972: force-cargo-library-build] Error 101 8 5:00.17 gmake[4]: Leaving directory '/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/ff/toolkit/library/rust' 8 5:00.17 gmake[3]: *** [/var/tmp/portage/www-client/firefox-60.2.0/work/firefox-60.2.0/config/recurse.mk:73: toolkit/library/rust/target] Error 2
One of you who can manage to reproduce this bug will need to spend a little time using gdb to find out what is going on. If I had a reproducible test case, I'd a) Run the failing link with "-save-temps -v -Wl,-v" added to the command line. b) Run the ld invocation so found to check that the error still occurs. c) Run that under gdb, putting a breakpoint on the assertion. d) Find the address of entry->the_bfd->link.next when the assertion triggers. e) Put a watchpoint on writes to that address, and run it again. This should reveal the wild write to memory that destroys the linked list. Once that is known it should be easy to fix.
One of the bug reporters is now saying the problem went away after upgrading to binutils-2.31.1. Is that halfway plausible given the changes between those releases?
Upgraded binutils from 2.30-r2 to 2.31.1 Firefox now builds
I had a look at it in https://bugs.gentoo.org/663690#c19 TL;DR: Gentoo has two libbfd-2.30.0.so libraries with incompatible ABIs: /usr/lib/binutils/i686-pc-linux-gnu/2.30/libbfd-2.30.0.so and /usr/lib/libbfd-2.30.0.so One of them is built --enable-64-bit-bfd, then other is not. Unfortunately the have the same SONAME and the error is subtle. Normally it's not al issue as 'ld' uses private library at /usr/lib/binutils/i686-pc-linux-gnu/2.30/libbfd-2.30.0.so Unfortunately cargo(rust?) uses LD_LIBRARY_PATH=/usr/lib environment when calls gcc/ld and tricks ld into loading wrong (ABI-incompatible) library.
> Gentoo has two libbfd-2.30.0.so libraries with incompatible ABIs: Ouch, that would explain this bug. All sorts of mayhem would ensue. It's a wonder that ld didn't segfault immediately.
(In reply to Alan Modra from comment #8) > > Gentoo has two libbfd-2.30.0.so libraries with incompatible ABIs: > > One of them is built --enable-64-bit-bfd, then other is not > > Ouch, that would explain this bug. All sorts of mayhem would ensue. It's a > wonder that ld didn't segfault immediately. Would it be sensible to request binutils libraries to expand SONAME to include the bits that influence ABI? For example to change libbfd.so -> libbfd-2.30.0.so to something like libbfd.so -> libbfd-2.30.0-64-bit-bfd.so That way when user rebuilds binutils with different flags they would get the errors about missing library instead of obscure SIGSEGV.
The master branch has been updated by Alan Modra <amodra@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=bf2dd8d7cf4114b8a60dbb83b340f76b9b2474d1 commit bf2dd8d7cf4114b8a60dbb83b340f76b9b2474d1 Author: Alan Modra <amodra@gmail.com> Date: Mon Oct 15 16:10:27 2018 +1030 BFD_INIT_MAGIC This patch performs a run-time test that a shared libbfd.so has been compiled with the same size bfd_vma as that of apps using the library. On a 32-bit host it is easily possible to have one libbfd.so compiled to support 64-bit targets (or configured with --enable-64-bit-bfd) while another only supports 32-bit targets. The two libraries will have differently sized bfd_vma types, and if the wrong one is loaded all sorts of weird behaviour might be seen. bfd/ PR 23534 * init.c (BFD_INIT_MAGIC): Define. (bfd_init): Return BFD_INIT_MAGIC. bfd-in2.h: Regenerate. binutils/ PR 23534 * addr2line.c (main): Exit with fatal error if bfd_init returns an unexpected value. * ar.c (main): Likewise. * dlltool.c (identify_dll_for_implib): Likewise. * nm.c (main): Likewise. * objcopy.c (main): Likewise. * objdump.c (main): Likewise. * size.c (main): Likewise. * strings.c (main): Likewise. * windmc.c (main): Likewise. * windres.c (main): Likewise. gas/ PR 23534 * as.c (main): Exit with fatal error if bfd_init returns an unexpected value. ld/ PR 23534 * ldmain.c (main): Exit with fatal error if bfd_init returns an unexpected value.
I've applied a patch that should make it obvious when an --enable-64-bit-bfd incompatibility is encountered.
That sounds like a nice improvement, thanks for implementing!
The master branch has been updated by Alan Modra <amodra@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=0e0dd7f1e8b64efff9a981d0421ffeaeff70cd20 commit 0e0dd7f1e8b64efff9a981d0421ffeaeff70cd20 Author: Alan Modra <amodra@gmail.com> Date: Tue Oct 16 09:01:55 2018 +1030 Re: BFD_INIT_MAGIC I should know better than to introduce the first use of size_t in bfd.h. PR 23534 * init.c (bfd_init): Return an unsigned int. bfd-in2.h: Regenerate.