Bug 27318 - glibc fails to load binaries when built with -march=sandybridge: CPU ISA level is lower than required
Summary: glibc fails to load binaries when built with -march=sandybridge: CPU ISA lev...
Status: RESOLVED FIXED
Alias: None
Product: glibc
Classification: Unclassified
Component: dynamic-link (show other bugs)
Version: 2.33
: P2 normal
Target Milestone: 2.34
Assignee: H.J. Lu
URL:
Keywords:
: 27446 (view as bug list)
Depends on:
Blocks:
 
Reported: 2021-02-02 19:48 UTC by Sergei Trofimovich
Modified: 2023-08-17 04:18 UTC (History)
6 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed: 2021-02-02 00:00:00
fweimer: security-


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sergei Trofimovich 2021-02-02 19:48:46 UTC
The system uses binutils-2.36 and gcc-11 (master). Configure glibc as:

$ ../glibc/configure --host=x86_64-pc-linux-gnu CC="x86_64-pc-linux-gnu-gcc -march=sandybridge" --prefix=/usr --disable-werror

$ ./elf/ld.so ./libc.so
./libc.so: (p) CPU ISA level is lower than required: needed: 7; got: 3

The CPU is `Intel(R) Core(TM) i7-2700K CPU @ 3.50GHz` (sandbybridge + AES).

After adding a bit of debug glibc contains ISA_1_V3 level (too high) while CPU contains ISA_1_V2 level (ok).

I patched glibc slightly to get ISA levels:

--- a/sysdeps/x86/dl-prop.h
+++ b/sysdeps/x86/dl-prop.h
@@ -54,8 +54,8 @@ dl_isa_level_check (struct link_map *m, const char *program)
          != l->l_x86_isa_1_needed)
        {
          if (program)
-           _dl_fatal_printf ("%s: CPU ISA level is lower than required\n",
-                             *l->l_name != '\0' ? l->l_name : program);
+           _dl_fatal_printf ("%s: (p) CPU ISA level is lower than required: needed: %d; got: %d\n",
+                             *l->l_name != '\0' ? l->l_name : program, l->l_x86_isa_1_needed, cpu_features->isa_1);
          else
            _dl_signal_error (0, l->l_name, "dlopen",
                              N_("CPU ISA level is lower than required"));

I suspect that binutils somehow managed to infer too high ISA level for final binary. 

$ readelf -a elf/ld.so | tail -n 5
Displaying notes found in: .note.gnu.property
  Owner                Data size        Description
  GNU                  0x00000020       NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature used: x86, x87, XMM, YMM, ZMM, FXSR, XSAVE, XSAVEC
        x86 ISA used: x86-64-baseline, x86-64-v3, x86-64-v4

$ LANG=C h readelf -a libc.so | tail -n 5

Displaying notes found in: .note.ABI-tag
  Owner                Data size        Description
  GNU                  0x00000010       NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0
Comment 1 Florian Weimer 2021-02-02 20:19:42 UTC
Something similar has been reported on libc-help: https://sourceware.org/pipermail/libc-help/2021-January/005648.html

If your GCC hasn't been configured with something like --with-arch=cascadelake, then I really do believe this is a binutils issue. It really looks like it's not computing the ISA used property correctly.
Comment 2 Sergei Trofimovich 2021-02-02 20:54:52 UTC
Aha, that makes sense. I'll try to find where v3 leaks out. So far it looks like some rtld files need newer ISA:

$ LANG=C h readelf -a $(find -name 'rtld-*.o*') |& egrep 'File|v3' | fgrep -B1 v3
File: ./string/rtld-strcspn-c.os
      Properties: x86 ISA used: x86-64-baseline, x86-64-v3
--
File: ./gmon/rtld-profil.os
      Properties: x86 ISA used: x86-64-baseline, x86-64-v3
--
File: ./signal/rtld-sigaction.os
      Properties: x86 ISA used: x86-64-baseline, x86-64-v3
Comment 3 H.J. Lu 2021-02-02 21:19:51 UTC
sysdeps/x86/isa-level.c has

# if defined __AVX__ || defined __AVX2__ || defined __F16C__ \
     || defined __FMA__ || defined __LZCNT__ || defined __MOVBE__ \
     || defined __XSAVE__
#  define ISA_V3	GNU_PROPERTY_X86_ISA_1_V3
# else
#  define ISA_V3	0
# endif

Since -march=sandybridge enables parts of v3 ISAs, x86-64-v3 is set.
We couldn't set x86 ISA needed to x86-64-v2 since this libc won't run on
v2 machines.  Technically, the v3 marker is correct.  We could issue
an error in this case.
Comment 4 H.J. Lu 2021-02-02 21:52:34 UTC
A patch is posted at

https://sourceware.org/pipermail/libc-alpha/2021-February/122268.html
Comment 5 Xi Ruoyao 2021-02-08 01:46:06 UTC
Should we move this to binutils, add a binutils configure option to disable ISA markers completely?  If CFLAGS contains -march=something we'll have "too high" ISA markers all over the system.  It severly affects Gentoo or LFS users using their custom -march= settings.
Comment 6 Xi Ruoyao 2021-02-08 02:27:25 UTC
Sorry, I misunderstood the problem.  Please ignore my previous message.
Comment 7 Sergei Trofimovich 2021-02-08 07:18:27 UTC
(In reply to Xi Ruoyao from comment #5)
> Should we move this to binutils, add a binutils configure option to disable
> ISA markers completely?  If CFLAGS contains -march=something we'll have "too
> high" ISA markers all over the system.  It severly affects Gentoo or LFS
> users using their custom -march= settings.

Gentoo should not be affected after https://gitweb.gentoo.org/repo/gentoo.git/commit/sys-libs/glibc?id=5dbd6a821ff753e3b41324c4fb7c58cf65eeea33 workaround.
Comment 8 Andreas Schwab 2021-03-01 14:54:05 UTC
*** Bug 27446 has been marked as a duplicate of this bug. ***
Comment 9 H.J. Lu 2021-03-06 16:49:47 UTC
Fixed on master and 2.33 branch.
Comment 10 Matt Whitlock 2021-04-27 17:16:35 UTC
The fix is deficient. The arguments to 'test' are missing needed quoting:

 121 if test $libc_cv_have_x86_lahf_sahf = yes; then
 122   AC_DEFINE(HAVE_X86_LAHF_SAHF)
 123 fi
 124 if test $libc_cv_have_x86_movbe = yes; then
 125   AC_DEFINE(HAVE_X86_MOVBE)
 126 fi

Those variables can be unset, resulting in 'test' throwing errors, as reported at https://bugs.gentoo.org/785091.

/var/tmp/portage/sys-libs/glibc-2.33/work/glibc-2.33/configure: 168: test: =: unexpected operator
/var/tmp/portage/sys-libs/glibc-2.33/work/glibc-2.33/configure: 172: test: =: unexpected operator

A fix is to put double-quotation marks around the shell variable references.

Do I need to open a new bug to report this, or can it be handled in the context of this bug?
Comment 11 Florian Weimer 2021-04-27 17:22:56 UTC
Please file a new bug for this. Thanks.
Comment 12 Matt Whitlock 2021-04-27 17:47:55 UTC
Never mind. The problem on Gentoo is that the Gentoo ebuild is forcing libc_cv_include_x86_isa_level=no on the configure command line (as a vestigial workaround to this bug), and that's causing the configure test to be skipped, thereby skipping over the initialization of libc_cv_have_x86_lahf_sahf and libc_cv_have_x86_movbe. I've reported that problem to Gentoo. Apologies for the noise here.