Bug 28545 - cross compile incorrectly using host libraries in install relink
Summary: cross compile incorrectly using host libraries in install relink
Status: WAITING
Alias: None
Product: binutils
Classification: Unclassified
Component: libctf (show other bugs)
Version: 2.37
: P2 normal
Target Milestone: ---
Assignee: Nick Alcock
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-11-04 23:52 UTC by John
Modified: 2021-11-18 20:55 UTC (History)
3 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed: 2021-11-09 00:00:00


Attachments
Full compile log for OpenWrt ipq40xx binutils 2.37 package (45.79 KB, text/plain)
2021-11-06 03:04 UTC, John
Details
Full compile log for OpenWrt ath79 binutils 2.37 package (47.54 KB, text/plain)
2021-11-06 03:05 UTC, John
Details
Full compile log for OpenWrt (toolchain only) ipq40xx binutils 2.37 (32.11 KB, text/plain)
2021-11-11 20:02 UTC, John
Details
build log detailing error (56.34 KB, text/plain)
2021-11-14 09:13 UTC, da_audiophile
Details
Full compile log for bootlin toolchain armv7-eabihf (31.13 KB, text/plain)
2021-11-18 20:55 UTC, John
Details

Note You need to log in before you can comment on or make changes to this bug.
Description John 2021-11-04 23:52:03 UTC
I am trying to cross compile an arm OpenWrt target on an x86_64 Arch Linux host (host Arch Linux binutils package 2.36.1-3).
This worked without issue before OpenWrt switched to binutils 2.37

The issue is seen in install, relink, (cross) compiling openwrt's package/devel/binutils:
/usr/lib/libiberty.a: error adding symbols: file format not recognized

The issue I am seeing looked very similar to https://sourceware.org/bugzilla/show_bug.cgi?id=27360
With that fix applied to both the OpenWrt toolchain, and package binutils, 2.37 moves past the above error, and gives this one:

/usr/lib/libz.so: file not recognized: file format not recognized

Is it possible that zlib needs the same kind of fix?

OpenWrt does make some slight patches to binutils:
https://git.openwrt.org/?p=openwrt/openwrt.git;a=tree;f=toolchain/binutils/patches/2.37;h=0b79f8ca1a95eb53cf16d70e4229c90639cb2448;hb=HEAD

Cheers

… trimmed to error
before applying libctf: link against libiberty before linking in libbfd or libctf-nobfd
 /bin/sh ./libtool   --mode=install /mnt/pool_ssd/code/openwrt/staging_dir/host/bin/install -c   libctf.la libctf-nobfd.la '/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/ipkg-install/usr/lib'
libtool: install: warning: relinking `libctf.la'
libtool: install: (cd /mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf; /bin/sh /mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf/libtool  --tag CC --mode=relink arm-openwrt-linux-muslgnueabi-gcc -std=gnu99 -Wall -W -Wall -Wno-narrowing -Wwrite-strings -Wmissing-format-attribute -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -pedantic -Wno-long-long -I./../zlib -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=hard -I/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/include -I/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/include -fmacro-prefix-map=/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37=binutils-2.37 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -DPIC -fpic -Wno-unused-value -version-info 0:0:0 -Wl,--version-script=./libctf.ver -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib -znow -zrelro -o libctf.la -rpath /usr/lib libctf_la-ctf-archive.lo libctf_la-ctf-dump.lo libctf_la-ctf-create.lo libctf_la-ctf-decl.lo libctf_la-ctf-error.lo libctf_la-ctf-hash.lo libctf_la-ctf-labels.lo libctf_la-ctf-dedup.lo libctf_la-ctf-link.lo libctf_la-ctf-lookup.lo libctf_la-ctf-open.lo libctf_la-ctf-serialize.lo libctf_la-ctf-sha1.lo libctf_la-ctf-string.lo libctf_la-ctf-subr.lo libctf_la-ctf-types.lo libctf_la-ctf-util.lo libctf_la-ctf-qsort_r.lo libctf_la-ctf-open-bfd.lo ../bfd/libbfd.la -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf/../libiberty/pic -liberty -L./../zlib -lz -inst-prefix-dir /mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/ipkg-install)
libtool: relink: arm-openwrt-linux-muslgnueabi-gcc -shared  -fPIC -DPIC  .libs/libctf_la-ctf-archive.o .libs/libctf_la-ctf-dump.o .libs/libctf_la-ctf-create.o .libs/libctf_la-ctf-decl.o .libs/libctf_la-ctf-error.o .libs/libctf_la-ctf-hash.o .libs/libctf_la-ctf-labels.o .libs/libctf_la-ctf-dedup.o .libs/libctf_la-ctf-link.o .libs/libctf_la-ctf-lookup.o .libs/libctf_la-ctf-open.o .libs/libctf_la-ctf-serialize.o .libs/libctf_la-ctf-sha1.o .libs/libctf_la-ctf-string.o .libs/libctf_la-ctf-subr.o .libs/libctf_la-ctf-types.o .libs/libctf_la-ctf-util.o .libs/libctf_la-ctf-qsort_r.o .libs/libctf_la-ctf-open-bfd.o   -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/ipkg-install/usr/lib -L/usr/lib -lbfd -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/bfd/../libiberty/pic -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/zlib -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf/../libiberty/pic -liberty -lz  -mfloat-abi=hard -Wl,-z -Wl,now -Wl,-z -Wl,relro -Wl,--version-script=./libctf.ver -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib   -Wl,-soname -Wl,libctf.so.0 -o .libs/libctf.so.0.0.0
/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib/gcc/arm-openwrt-linux-muslgnueabi/11.2.0/../../../../arm-openwrt-linux-muslgnueabi/bin/ld: /usr/lib/libiberty.a: error adding symbols: file format not recognized
collect2: error: ld returned 1 exit status
libtool: install: error: relink `libctf.la' with the above command before installing it
make[6]: *** [Makefile:561: install-libLTLIBRARIES] Error 1
make[6]: Leaving directory '/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf'
make[5]: *** [Makefile:1245: install-am] Error 2
make[5]: Leaving directory '/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf'
make[4]: *** [Makefile:10547: install-libctf] Error 2
make[4]: Leaving directory '/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37'
make[3]: *** [Makefile:2339: install] Error 2
make[3]: Leaving directory '/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37'
make[2]: *** [Makefile:143: /mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/.built] Error 2
make[2]: Leaving directory '/mnt/pool_ssd/code/openwrt/package/devel/binutils'
time: package/devel/binutils/compile#2.31#0.62#2.80
    ERROR: package/devel/binutils failed to build.
make[1]: *** [package/Makefile:116: package/devel/binutils/compile] Error 1
make[1]: Leaving directory '/mnt/pool_ssd/code/openwrt'
make: *** [/mnt/pool_ssd/code/openwrt/include/toplevel.mk:230: package/devel/binutils/compile] Error 2


After applying fix to 2.37 https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=7d53105d6ed984aec255fa0eacd0405f3c1bb874

libtool: install: (cd /mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf; /bin/sh /mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf/libtool  --tag CC --mode=relink arm-openwrt-linux-muslgnueabi-gcc -std=gnu99 -Wall -W -Wall -Wno-narrowing -Wwrite-strings -Wmissing-format-attribute -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -pedantic -Wno-long-long -I./../zlib -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=hard -I/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/include -I/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/include -fmacro-prefix-map=/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37=binutils-2.37 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -DPIC -fpic -Wno-unused-value -version-info 0:0:0 -Wl,--version-script=./libctf.ver -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib -znow -zrelro -o libctf.la -rpath /usr/lib libctf_la-ctf-archive.lo libctf_la-ctf-dump.lo libctf_la-ctf-create.lo libctf_la-ctf-decl.lo libctf_la-ctf-error.lo libctf_la-ctf-hash.lo libctf_la-ctf-labels.lo libctf_la-ctf-dedup.lo libctf_la-ctf-link.lo libctf_la-ctf-lookup.lo libctf_la-ctf-open.lo libctf_la-ctf-serialize.lo libctf_la-ctf-sha1.lo libctf_la-ctf-string.lo libctf_la-ctf-subr.lo libctf_la-ctf-types.lo libctf_la-ctf-util.lo libctf_la-ctf-qsort_r.lo libctf_la-ctf-open-bfd.lo -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf/../libiberty/pic -liberty ../bfd/libbfd.la -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf/../libiberty/pic -liberty -L./../zlib -lz -inst-prefix-dir /mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/ipkg-install)
libtool: relink: arm-openwrt-linux-muslgnueabi-gcc -shared  -fPIC -DPIC  .libs/libctf_la-ctf-archive.o .libs/libctf_la-ctf-dump.o .libs/libctf_la-ctf-create.o .libs/libctf_la-ctf-decl.o .libs/libctf_la-ctf-error.o .libs/libctf_la-ctf-hash.o .libs/libctf_la-ctf-labels.o .libs/libctf_la-ctf-dedup.o .libs/libctf_la-ctf-link.o .libs/libctf_la-ctf-lookup.o .libs/libctf_la-ctf-open.o .libs/libctf_la-ctf-serialize.o .libs/libctf_la-ctf-sha1.o .libs/libctf_la-ctf-string.o .libs/libctf_la-ctf-subr.o .libs/libctf_la-ctf-types.o .libs/libctf_la-ctf-util.o .libs/libctf_la-ctf-qsort_r.o .libs/libctf_la-ctf-open-bfd.o   -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf/../libiberty/pic -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/ipkg-install/usr/lib -L/usr/lib -lbfd -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/bfd/../libiberty/pic -L/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/zlib -liberty -lz  -mfloat-abi=hard -Wl,-z -Wl,now -Wl,-z -Wl,relro -Wl,--version-script=./libctf.ver -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib   -Wl,-soname -Wl,libctf.so.0 -o .libs/libctf.so.0.0.0
/usr/lib/libz.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
libtool: install: error: relink `libctf.la' with the above command before installing it
make[6]: *** [Makefile:566: install-libLTLIBRARIES] Error 1
make[6]: Leaving directory '/mnt/pool_ssd/code/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/binutils-2.37/libctf'
Comment 1 John 2021-11-06 03:03:00 UTC
Minimum OpenWrt config example:

cat <<EOF>.config
CONFIG_TARGET_ipq40xx=y
CONFIG_TARGET_ipq40xx_mikrotik=y
CONFIG_TARGET_ipq40xx_mikrotik_DEVICE_mikrotik_hap-ac2=y

CONFIG_DEVEL=y
CONFIG_EXPERIMENTAL=y
CONFIG_TESTING_KERNEL=y

CONFIG_PACKAGE_ip-full=y
EOF

make defconfig
make clean
make package/devel/binutils/clean
make package/devel/binutils/compile

# /usr/lib/libiberty.a: error adding symbols: file format not recognized
sudo mv /usr/lib/libiberty.a{,_moved}
# /usr/lib/libz.so: file not recognized: file format not recognized               
sudo mv /usr/lib/libz.so{,_moved}
# /usr/lib/libz.a: error adding symbols: file format not recognized
sudo mv /usr/lib/libz.a{,_moved}
# relink okay


Not seeing this on either an ath79 or ramips mt7621 device build

The ath79 device build still tries these external libraries, but skips for incompatible:
skipping incompatible /usr/lib/libiberty.a when searching for -liberty
/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib/gcc/mips-openwrt-linux-musl/11.2.0/../../../../mips-openwrt-linux-musl/bin/ld: skipping incompatible /usr/lib/libz.so when searching for -lz
/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib/gcc/mips-openwrt-linux-musl/11.2.0/../../../../mips-openwrt-linux-musl/bin/ld: skipping incompatible /usr/lib/libz.a when searching for -lz


The previous OpenWrt package binutils version was 2.35.2, and it compiled without this issue.
Comment 2 John 2021-11-06 03:04:20 UTC
Created attachment 13762 [details]
Full compile log for OpenWrt ipq40xx binutils 2.37 package
Comment 3 John 2021-11-06 03:05:25 UTC
Created attachment 13763 [details]
Full compile log for OpenWrt ath79 binutils 2.37 package
Comment 4 John 2021-11-06 07:14:56 UTC
this libtool relink libctf.so is the only place in the whole build log where -L/usr/lib is used
Comment 5 Andreas Schwab 2021-11-06 08:22:38 UTC
How did you configure?
Comment 6 John 2021-11-06 10:56:19 UTC
I used the OpenWrt package defaults.
These look like:
  80 TARGET_CFLAGS += $(FPIC) -Wno-unused-value
  81 
  82 TARGET_LDFLAGS += $(if $(INTL_FULL),-lintl)
  83 
  84 CONFIGURE_ARGS += \
  85         --host=$(REAL_GNU_TARGET_NAME) \
  86         --target=$(REAL_GNU_TARGET_NAME) \
  87         --enable-shared \
  88         --enable-install-libiberty \
  89         --enable-install-libbfd \
  90         --enable-install-libctf
  91 
  92 define Build/Install
  93         $(call Build/Install/Default)
  94         $(MAKE) -C $(PKG_BUILD_DIR)/libiberty \
  95                 target_header_dir=libiberty \
  96                 DESTDIR="$(PKG_INSTALL_DIR)" \
  97                 MULTIOSDIR="" \
  98                 install


https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/devel/binutils/Makefile;h=5fdb83036ea394e7114f0c83362fb1992d96cff5;hb=HEAD
Comment 7 Andreas Schwab 2021-11-06 12:35:27 UTC
Please show the full configure command line.
Comment 8 John 2021-11-06 21:10:16 UTC
./configure --target=arm-openwrt-linux --host=arm-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls  --host=arm-openwrt-linux-muslgnueabi --target=arm-openwrt-linux-muslgnueabi --enable-shared --enable-install-libiberty --enable-install-libbfd --enable-install-libctf

There are many FLAGS set as well.
On line 330 (or search binutils-2.37.tar.xz) here:
https://sourceware.org/bugzilla/attachment.cgi?id=13762

I do see the OpenWrt package is:
- trying to do an autoreconf with some custom options and failing before configure
- replaces config.guess and config.sub
removing both of these results in the same
/usr/lib/libiberty.a: error adding symbols: file format not recognized

Minimum example:
curl -O https://downloads.openwrt.org/snapshots/targets/ipq40xx/mikrotik/openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz

tar -xf openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz
cd openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64

echo 'src-git base https://git.openwrt.org/openwrt/openwrt.git' > feeds.conf.default
./scripts/feeds update -a
./scripts/feeds install binutils

cat <<EOF>.config
# CONFIG_ALL is not set
# CONFIG_ALL_KMODS is not set
# CONFIG_ALL_NONSHARED is not set
# CONFIG_SIGNED_PACKAGES is not set
CONFIG_PACKAGE_binutils=m
EOF

make defconfig
make -j1 V=s package/binutils/compile 2>&1 | tee binutils.log
Comment 9 Hauke Mehrtens 2021-11-07 16:05:37 UTC
I see a similar problem with the gdb compile in OpenWrt on Arch Linux, on Debian stable it works.

gdb configure is called like this:
-------------------------
./configure --target=mips-openwrt-linux --host=mips-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix= --program-suffix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls --with-system-readline --with-system-zlib --without-expat --without-lzma --disable-unit-tests --disable-ubsan --disable-sim --disable-werror --disable-source-highlight --without-mpc --without-mpfr --without-isl --with-gmp=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr
-------------------------


The configure in the gdb folder is called like this:
-------------------------
/home/hauke/openwrt/openwrt/build_dir/target-mips_24kc_musl/gdb-11.1/gdb/configure --srcdir=.././gdb --cache-file=./config.cache --with-system-zlib --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls --with-system-readline --with-system-zlib --without-expat --without-lzma --disable-unit-tests --disable-ubsan --disable-sim --disable-werror --disable-source-highlight --without-mpc --without-mpfr --without-isl --with-gmp=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr --program-transform-name=s&$&&;s&^&& --disable-option-checking --build=x86_64-pc-linux-gnu --host=mips-openwrt-linux --target=mips-openwrt-linux
-------------------------


It then fails in gdb/configure like this:
-------------------------
configure:10433: checking for libgmp
configure:10453: mips-openwrt-linux-musl-gcc -o conftest -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -fmacro-prefix-map=/home/hauke/openwrt/openwrt/build_dir/target-mips_24kc_musl/gdb-11.1=gdb-11.1 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include      -I/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include  -L/home/hauke/openwrt/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/usr/lib -L/home/hauke/openwrt/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib -znow -zrelro -L/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/lib -L/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib -Wl,-rpath-link=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib -L/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib -Wl,-rpath-link=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib  -static-libstdc++ -Wl,--gc-sections  conftest.c -lncursesw -lm  /usr/lib/libgmp.so >&5
/home/hauke/openwrt/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib/gcc/mips-openwrt-linux-musl/11.2.0/../../../../mips-openwrt-linux-musl/bin/ld: /usr/lib/libgmp.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
configure:10453: $? = 1
configure: failed program was:
....
-------------------------


The gdb/configure script does not take a --with-gmp option, but --with-libgmp-prefix.
-------------------------
[hauke@hauke-arch gdb]$ ./configure --help|fgrep  gmp
  --with-libgmp-prefix[=DIR]  search for libgmp in DIR/include and DIR/lib
  --without-libgmp-prefix     don't search for libgmp in includedir and libdir
  --with-libgmp-type=TYPE     type of library to search for (auto/static/shared)
[hauke@hauke-arch gdb]$ 
-------------------------

When I compile it with --with-libgmp-prefix on Arch Linux it works.
Comment 10 Nick Alcock 2021-11-09 21:25:56 UTC
These libraries all have no_install set: these must indeed be treated like libiberty is in 27360 and have their full path in the build tree listed before any libctf deps that transitively include them, to ensure that the right copy gets linked in, since it won't ever be installed and using an installed copy is always wrong. I'll look at it shortly.

(It never occurred to me that this applied to libs other than libiberty, but it's dead obvious in hindsight.)
Comment 11 Nick Alcock 2021-11-11 14:25:24 UTC
Hm. This is not easy to replicate. A straight cross build, even one with --prefix=/usr and --disable-fast-install to force relinking, does not go wrong.

Looking at your compile command, what's breaking it is that something is forcing "-L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib" in front of the -o in the libtool command.

Looking at the compile log, the openwrt build scripts are passing *this* in:

LDFLAGS="-L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -znow -zrelro "

This is a disaster waiting to happen (and now it has happened). You can't set up sysroots by just passing in -L arguments in LDFLAGS like this! The problem is that the resulting library search order is wrong: LDFLAGS is injected early in link lines, and libraries are searched for in order of -L arguments, so any libraries specified in -L will take precedence over any -L arguments specified by binutils's actual makefiles to find its own libraries in the build tree, and there's nothing we can do about it. -L arguments should only ever appear in LIBS, accompanying specific libraries found in nonstandard paths.

The compiler should have been configured to find libraries suitably without needing a massive spray of -L arguments like this: GCC has a --sysroot argument for a reason (and --sysroot does not mess up the library search order). You should probably raise this with OpenWRT (and link to this bug).

I am tempted to mark this INVALID right away, but I'll leave it as WAITING until we see what the OpenWRT people say (since they probably do a lot more cross-compiling than me, to be honest).

(But I clearly need to backport bug 27360 to the 2.37 branch. I'm on that.)
Comment 12 John 2021-11-11 20:00:30 UTC
Thank you Nick,
I will take this up again with OpenWrt
Sorry I had not done an out-of-openwrt cross compile

Just to make sure, I am seeing this error when I have both configure args:
--enable-shared --prefix=/usr
The error then shows during make install

I do not yet have a non-openwrt cross compile toolchain setup. I will try to get this done.

In my recreate using the openwrt toolchain, but not build scripts, I (still) see this error, without those unwanted OpenWrt LDFLAGS: will attach log.

curl -O https://downloads.openwrt.org/snapshots/targets/ipq40xx/mikrotik/openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz

tar -xf openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz
cd openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64

PATH="$PATH:$(realpath staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/bin)"
export PATH

STAGING_DIR="$(realpath staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi)"
export STAGING_DIR

cd dl
curl -LO 'http://mirror.downloadvn.com/gnu/binutils/binutils-2.37.tar.xz'
cd ../

mkdir temp
cd temp

curl -L -o libctf-link-against-libibierty-first.patch 'https://sourceware.org/git?p=binutils-gdb.git;a=patch;h=7d53105d6ed984aec255fa0eacd0405f3c1bb874'

tar -xf ../dl/binutils-2.37.tar.xz
cd binutils-2.37

patch -Np1 -i ../libctf-link-against-libibierty-first.patch

mkdir tmp


./configure --target=arm-openwrt-linux --host=arm-openwrt-linux --build=x86_64-pc-linux-gnu --enable-shared --prefix=/usr 2>&1 | tee log
make 2>&1 | tee -a log
DESTDIR=$(realpath tmp) make install 2>&1 | tee -a log
Comment 13 John 2021-11-11 20:02:10 UTC
Created attachment 13778 [details]
Full compile log for OpenWrt (toolchain only) ipq40xx binutils 2.37
Comment 14 da_audiophile 2021-11-14 09:12:00 UTC
On Arch Linux x86_64 I am also hit by this bug.  Interestingly, I can build the bcm27xx target for RPi4 just fine but I cannot build ipq806x or ramips targets.
Comment 15 da_audiophile 2021-11-14 09:13:30 UTC
Created attachment 13786 [details]
build log detailing error
Comment 16 John 2021-11-18 20:55:22 UTC
Created attachment 13798 [details]
Full compile log for bootlin toolchain armv7-eabihf

I did try this away from the OpenWrt toolchain, and see the same error using the Bootlin armv7-eabihf toolchain.

This is the build process I used for the attached log:

curl -OL 'https://toolchains.bootlin.com/downloads/releases/toolchains/armv7-eabihf/tarballs/armv7-eabihf--glibc--bleeding-edge-2020.08-1.tar.bz2'

tar -xf armv7-eabihf--glibc--bleeding-edge-2020.08-1.tar.bz2
cd armv7-eabihf--glibc--bleeding-edge-2020.08-1
sh relocate-sdk.sh
PATH="$PATH:$(realpath bin)"

cd ../

curl -LO 'http://mirror.downloadvn.com/gnu/binutils/binutils-2.37.tar.xz'
curl -L -o libctf-link-against-libibierty-first.patch 'https://sourceware.org/git?p=binutils-gdb.git;a=patch;h=7d53105d6ed984aec255fa0eacd0405f3c1bb874'

tar -xf binutils-2.37.tar.xz
cd binutils-2.37

patch -Np1 -i ../libctf-link-against-libibierty-first.patch

mkdir tmp

./configure --target=arm-buildroot-linux-gnueabihf --host=arm-buildroot-linux-gnueabihf --build=x86_64-pc-linux-gnu --enable-shared --prefix=/usr 2>&1 | tee log
make 2>&1 | tee -a log
DESTDIR=$(realpath tmp) make install 2>&1 | tee -a log