Running Debian Bookworm armhf user space on 64-bit kernels broke after the update from binutils-2.39 to binutils-2.40. Booting a 32-bit root image using a 16KB or 64KB page kernel now fails as soon as it loads a shared library that was built after the binutils update: udevadm: error while loading shared libraries: libacl.so.1: ELF load command address/offset not page-aligned 10:32 AM /sbin/init: error while loading shared libraries: libapparmor.so.1: ELF load command address/offset not page-aligned Reverting commit 1a26a53a0de ("ARM: Fix ld bloat introduced between binutils-2.38 and 2.39") solves the problem again.
https://inbox.sourceware.org/binutils/Y7OGX08FCfvucZFW@squeak.grove.modra.org/T/#t I think this is by design ...
An "armhf" userspace on a "arm64" kernel with non-4KiB pagesize is an odd beast, odd enough that it's hidden behind EXPERT settings when building the kernel and warnings about having to rebuild tools. Quoting linux/arch/arm64/Kconfig as of c8451c141e07: menuconfig COMPAT bool "Kernel support for 32-bit EL0" depends on ARM64_4K_PAGES || EXPERT select HAVE_UID16 select OLD_SIGSUSPEND3 select COMPAT_OLD_SIGACTION help This option enables support for a 32-bit EL0 running under a 64-bit kernel at EL1. AArch32-specific components such as system calls, the user helper functions, VFP support and the ptrace interface are handled appropriately by the kernel. If you use a page size other than 4KB (i.e, 16KB or 64KB), please be aware that you will only be able to execute AArch32 binaries that were compiled with page size aligned segments. If you want to execute 32-bit userspace applications, say Y. Is this non-4KiB page-size + armhf userspace part of an easily accessible distribution? Please tell more. It's a judgement call; you can't have it all with the default build. I mentioned that this setup would break, but my argument was and is also, that such a setup is odd enough, that people with it have to build tools specially, or use special options like ld -z max-page-size=0x10000, rather than bloating for everyone else. See the mailing list discussion at the time. Are there reasons to reconsider this setup any more common than as portrayed; an odd corner-case where users are developers ready to patch and build tools themselves?
The case I ran into was trying to create a bootable disk image for a Debian armhf or armel system using the 'mmdebstrap' tool. The system does support running 'compat' userspace on the CPU itself, so this uses the 'qemu-user-arm' tool to run the installation process, but the problem is the same as when running compat userspace. mmdebstrap is useful for me to quickly test a 32-bit kernel image with a working userspace, and it works fine on the other architectures supported by Debian, as well as on stable Debian arm releases that were produced with binutils versions 2.25 (which added support for 64KB page alignment) through 2.39. My workstation is an Apple M1 based system that only works with 16KB pages, the only other alternative I have would be to test on a much slower Chromebook, or an cloud instance. Regarding the kernel option, support for running 32-bit binaries with larger pages was added experimentally at the same time as binutils support, but at the time no distributions were built with new enough binutils yet so it was left under CONFIG_EXPERT, see https://lore.kernel.org/linux-arm-kernel/20141204182049.GB7749@arm.com/. Since all distributions that are still supported now are built with binutils-2.25 or higher, this is probably something that could be revisited, especially as more users are going to want 16KB page support in their kernels, which has a better tradeoff between performance and space overhead than using 64KB pages.
pr28824 has some interesting comments from someone familiar with the mold linker.