Bug 28545

Minimum OpenWrt config example:

cat <<EOF>.config
CONFIG_TARGET_ipq40xx=y
CONFIG_TARGET_ipq40xx_mikrotik=y
CONFIG_TARGET_ipq40xx_mikrotik_DEVICE_mikrotik_hap-ac2=y

CONFIG_DEVEL=y
CONFIG_EXPERIMENTAL=y
CONFIG_TESTING_KERNEL=y

CONFIG_PACKAGE_ip-full=y
EOF

make defconfig
make clean
make package/devel/binutils/clean
make package/devel/binutils/compile

# /usr/lib/libiberty.a: error adding symbols: file format not recognized
sudo mv /usr/lib/libiberty.a{,_moved}
# /usr/lib/libz.so: file not recognized: file format not recognized               
sudo mv /usr/lib/libz.so{,_moved}
# /usr/lib/libz.a: error adding symbols: file format not recognized
sudo mv /usr/lib/libz.a{,_moved}
# relink okay


Not seeing this on either an ath79 or ramips mt7621 device build

The ath79 device build still tries these external libraries, but skips for incompatible:
skipping incompatible /usr/lib/libiberty.a when searching for -liberty
/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib/gcc/mips-openwrt-linux-musl/11.2.0/../../../../mips-openwrt-linux-musl/bin/ld: skipping incompatible /usr/lib/libz.so when searching for -lz
/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib/gcc/mips-openwrt-linux-musl/11.2.0/../../../../mips-openwrt-linux-musl/bin/ld: skipping incompatible /usr/lib/libz.a when searching for -lz


The previous OpenWrt package binutils version was 2.35.2, and it compiled without this issue.

Created attachment 13762 [details]
Full compile log for OpenWrt ipq40xx binutils 2.37 package

Created attachment 13763 [details]
Full compile log for OpenWrt ath79 binutils 2.37 package

this libtool relink libctf.so is the only place in the whole build log where -L/usr/lib is used

How did you configure?

I used the OpenWrt package defaults.
These look like:
  80 TARGET_CFLAGS += $(FPIC) -Wno-unused-value
  81 
  82 TARGET_LDFLAGS += $(if $(INTL_FULL),-lintl)
  83 
  84 CONFIGURE_ARGS += \
  85         --host=$(REAL_GNU_TARGET_NAME) \
  86         --target=$(REAL_GNU_TARGET_NAME) \
  87         --enable-shared \
  88         --enable-install-libiberty \
  89         --enable-install-libbfd \
  90         --enable-install-libctf
  91 
  92 define Build/Install
  93         $(call Build/Install/Default)
  94         $(MAKE) -C $(PKG_BUILD_DIR)/libiberty \
  95                 target_header_dir=libiberty \
  96                 DESTDIR="$(PKG_INSTALL_DIR)" \
  97                 MULTIOSDIR="" \
  98                 install


https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/devel/binutils/Makefile;h=5fdb83036ea394e7114f0c83362fb1992d96cff5;hb=HEAD

Please show the full configure command line.

./configure --target=arm-openwrt-linux --host=arm-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls  --host=arm-openwrt-linux-muslgnueabi --target=arm-openwrt-linux-muslgnueabi --enable-shared --enable-install-libiberty --enable-install-libbfd --enable-install-libctf

There are many FLAGS set as well.
On line 330 (or search binutils-2.37.tar.xz) here:
https://sourceware.org/bugzilla/attachment.cgi?id=13762

I do see the OpenWrt package is:
- trying to do an autoreconf with some custom options and failing before configure
- replaces config.guess and config.sub
removing both of these results in the same
/usr/lib/libiberty.a: error adding symbols: file format not recognized

Minimum example:
curl -O https://downloads.openwrt.org/snapshots/targets/ipq40xx/mikrotik/openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz

tar -xf openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz
cd openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64

echo 'src-git base https://git.openwrt.org/openwrt/openwrt.git' > feeds.conf.default
./scripts/feeds update -a
./scripts/feeds install binutils

cat <<EOF>.config
# CONFIG_ALL is not set
# CONFIG_ALL_KMODS is not set
# CONFIG_ALL_NONSHARED is not set
# CONFIG_SIGNED_PACKAGES is not set
CONFIG_PACKAGE_binutils=m
EOF

make defconfig
make -j1 V=s package/binutils/compile 2>&1 | tee binutils.log

I see a similar problem with the gdb compile in OpenWrt on Arch Linux, on Debian stable it works.

gdb configure is called like this:
-------------------------
./configure --target=mips-openwrt-linux --host=mips-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix= --program-suffix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls --with-system-readline --with-system-zlib --without-expat --without-lzma --disable-unit-tests --disable-ubsan --disable-sim --disable-werror --disable-source-highlight --without-mpc --without-mpfr --without-isl --with-gmp=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr
-------------------------


The configure in the gdb folder is called like this:
-------------------------
/home/hauke/openwrt/openwrt/build_dir/target-mips_24kc_musl/gdb-11.1/gdb/configure --srcdir=.././gdb --cache-file=./config.cache --with-system-zlib --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls --with-system-readline --with-system-zlib --without-expat --without-lzma --disable-unit-tests --disable-ubsan --disable-sim --disable-werror --disable-source-highlight --without-mpc --without-mpfr --without-isl --with-gmp=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr --program-transform-name=s&$&&;s&^&& --disable-option-checking --build=x86_64-pc-linux-gnu --host=mips-openwrt-linux --target=mips-openwrt-linux
-------------------------


It then fails in gdb/configure like this:
-------------------------
configure:10433: checking for libgmp
configure:10453: mips-openwrt-linux-musl-gcc -o conftest -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -fmacro-prefix-map=/home/hauke/openwrt/openwrt/build_dir/target-mips_24kc_musl/gdb-11.1=gdb-11.1 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include      -I/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include  -L/home/hauke/openwrt/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/usr/lib -L/home/hauke/openwrt/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib -znow -zrelro -L/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/lib -L/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib -Wl,-rpath-link=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib -L/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib -Wl,-rpath-link=/home/hauke/openwrt/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/lib  -static-libstdc++ -Wl,--gc-sections  conftest.c -lncursesw -lm  /usr/lib/libgmp.so >&5
/home/hauke/openwrt/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/lib/gcc/mips-openwrt-linux-musl/11.2.0/../../../../mips-openwrt-linux-musl/bin/ld: /usr/lib/libgmp.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
configure:10453: $? = 1
configure: failed program was:
....
-------------------------


The gdb/configure script does not take a --with-gmp option, but --with-libgmp-prefix.
-------------------------
[hauke@hauke-arch gdb]$ ./configure --help|fgrep  gmp
  --with-libgmp-prefix[=DIR]  search for libgmp in DIR/include and DIR/lib
  --without-libgmp-prefix     don't search for libgmp in includedir and libdir
  --with-libgmp-type=TYPE     type of library to search for (auto/static/shared)
[hauke@hauke-arch gdb]$ 
-------------------------

When I compile it with --with-libgmp-prefix on Arch Linux it works.

These libraries all have no_install set: these must indeed be treated like libiberty is in 27360 and have their full path in the build tree listed before any libctf deps that transitively include them, to ensure that the right copy gets linked in, since it won't ever be installed and using an installed copy is always wrong. I'll look at it shortly.

(It never occurred to me that this applied to libs other than libiberty, but it's dead obvious in hindsight.)

Hm. This is not easy to replicate. A straight cross build, even one with --prefix=/usr and --disable-fast-install to force relinking, does not go wrong.

Looking at your compile command, what's breaking it is that something is forcing "-L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -Wl,-rpath-link=/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libiconv-stub/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/usr/lib/libintl-stub/lib" in front of the -o in the libtool command.

Looking at the compile log, the openwrt build scripts are passing *this* in:

LDFLAGS="-L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/usr/lib -L/mnt/pool_ssd/code/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/lib -znow -zrelro "

This is a disaster waiting to happen (and now it has happened). You can't set up sysroots by just passing in -L arguments in LDFLAGS like this! The problem is that the resulting library search order is wrong: LDFLAGS is injected early in link lines, and libraries are searched for in order of -L arguments, so any libraries specified in -L will take precedence over any -L arguments specified by binutils's actual makefiles to find its own libraries in the build tree, and there's nothing we can do about it. -L arguments should only ever appear in LIBS, accompanying specific libraries found in nonstandard paths.

The compiler should have been configured to find libraries suitably without needing a massive spray of -L arguments like this: GCC has a --sysroot argument for a reason (and --sysroot does not mess up the library search order). You should probably raise this with OpenWRT (and link to this bug).

I am tempted to mark this INVALID right away, but I'll leave it as WAITING until we see what the OpenWRT people say (since they probably do a lot more cross-compiling than me, to be honest).

(But I clearly need to backport bug 27360 to the 2.37 branch. I'm on that.)

Thank you Nick,
I will take this up again with OpenWrt
Sorry I had not done an out-of-openwrt cross compile

Just to make sure, I am seeing this error when I have both configure args:
--enable-shared --prefix=/usr
The error then shows during make install

I do not yet have a non-openwrt cross compile toolchain setup. I will try to get this done.

In my recreate using the openwrt toolchain, but not build scripts, I (still) see this error, without those unwanted OpenWrt LDFLAGS: will attach log.

curl -O https://downloads.openwrt.org/snapshots/targets/ipq40xx/mikrotik/openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz

tar -xf openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64.tar.xz
cd openwrt-sdk-ipq40xx-mikrotik_gcc-11.2.0_musl_eabi.Linux-x86_64

PATH="$PATH:$(realpath staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi/bin)"
export PATH

STAGING_DIR="$(realpath staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.2.0_musl_eabi)"
export STAGING_DIR

cd dl
curl -LO 'http://mirror.downloadvn.com/gnu/binutils/binutils-2.37.tar.xz'
cd ../

mkdir temp
cd temp

curl -L -o libctf-link-against-libibierty-first.patch 'https://sourceware.org/git?p=binutils-gdb.git;a=patch;h=7d53105d6ed984aec255fa0eacd0405f3c1bb874'

tar -xf ../dl/binutils-2.37.tar.xz
cd binutils-2.37

patch -Np1 -i ../libctf-link-against-libibierty-first.patch

mkdir tmp


./configure --target=arm-openwrt-linux --host=arm-openwrt-linux --build=x86_64-pc-linux-gnu --enable-shared --prefix=/usr 2>&1 | tee log
make 2>&1 | tee -a log
DESTDIR=$(realpath tmp) make install 2>&1 | tee -a log

Created attachment 13778 [details]
Full compile log for OpenWrt (toolchain only) ipq40xx binutils 2.37

On Arch Linux x86_64 I am also hit by this bug.  Interestingly, I can build the bcm27xx target for RPi4 just fine but I cannot build ipq806x or ramips targets.

Created attachment 13786 [details]
build log detailing error

Created attachment 13798 [details]
Full compile log for bootlin toolchain armv7-eabihf

I did try this away from the OpenWrt toolchain, and see the same error using the Bootlin armv7-eabihf toolchain.

This is the build process I used for the attached log:

curl -OL 'https://toolchains.bootlin.com/downloads/releases/toolchains/armv7-eabihf/tarballs/armv7-eabihf--glibc--bleeding-edge-2020.08-1.tar.bz2'

tar -xf armv7-eabihf--glibc--bleeding-edge-2020.08-1.tar.bz2
cd armv7-eabihf--glibc--bleeding-edge-2020.08-1
sh relocate-sdk.sh
PATH="$PATH:$(realpath bin)"

cd ../

curl -LO 'http://mirror.downloadvn.com/gnu/binutils/binutils-2.37.tar.xz'
curl -L -o libctf-link-against-libibierty-first.patch 'https://sourceware.org/git?p=binutils-gdb.git;a=patch;h=7d53105d6ed984aec255fa0eacd0405f3c1bb874'

tar -xf binutils-2.37.tar.xz
cd binutils-2.37

patch -Np1 -i ../libctf-link-against-libibierty-first.patch

mkdir tmp

./configure --target=arm-buildroot-linux-gnueabihf --host=arm-buildroot-linux-gnueabihf --build=x86_64-pc-linux-gnu --enable-shared --prefix=/usr 2>&1 | tee log
make 2>&1 | tee -a log
DESTDIR=$(realpath tmp) make install 2>&1 | tee -a log

i am running into a similar issue to this while trying to port Alpine Linux to m68k. the second pass building binutils 2.41 (first time, to run on my build machine; this time, to run on the target) fails during the install stage because it tries to relink against the build system's libraries (in this case, ppc64le-alpine-linux-gnu, in /usr/lib) rather than the host/target libraries (m68k-alpine-linux-gnu). the other compile logs in this issue seem to be the same scenario, just different architectures (`--build` is passed one target triple, `--host` and `--target` are passed another)

what stands out to me is the mention that --sysroot should be used instead of -L options. this is correct, but i don't think it fixes the problem. the copy of libtool included in the binutils and gcc source distributions does not appear to have the necessary support for resolving sysroots. this support was added in october of 2010. when is the last time binutils updated libtool?

LFS also notes a workaround for the issue, and outdated libtool as the culprit, in their second pass of binutils: https://www.linuxfromscratch.org/lfs/view/development/chapter06/binutils-pass2.html