Bug 13007 - have_avx in sysdeps/x68_64/dl-trampoline.S does not check hard enough
Summary: have_avx in sysdeps/x68_64/dl-trampoline.S does not check hard enough
Alias: None
Product: glibc
Classification: Unclassified
Component: libc (show other bugs)
Version: 2.14
: P2 normal
Target Milestone: ---
Assignee: Ulrich Drepper
Depends on:
Reported: 2011-07-20 13:53 UTC by Andy Lutomirski
Modified: 2014-06-27 12:54 UTC (History)
0 users

See Also:
Last reconfirmed:
fweimer: security-


Note You need to log in before you can comment on or make changes to this bug.
Description Andy Lutomirski 2011-07-20 13:53:53 UTC
have_avx returns 1 if CPUID.01H:ECX.AVX is set.  This is wrong: the AVX bit indicates that the CPU understands AVX instructions, but it can be set even if the kernel hasn't enabled YMM state.

The correct check is to also read CPUID.01H:ECX.OSXSAVE to check that xgetbv is enabled and then to use xgetbv to check XCR0.SSE and XCR0.AVX.

This comes from an Intel blog post here:

and is borne out by the Exceptions Type 6 section of the SDM, which indicates that VEX-coded instructions will #UD if XCR0 does not have those two bits set.

This bug presumably caused this crash:
Comment 1 Ulrich Drepper 2011-07-20 16:18:26 UTC
That's the generic model but why should anyone care on Linux?
Comment 2 Andy Lutomirski 2011-07-20 16:26:07 UTC
See the gmane link.  There is apparently at least one person running a legacy kernel on Sandy Bridge hardware, and glibc gets SIGILL on that box.

I'm not affected myself.  I suspect that only servers will ever see this, because desktop machines will probably have other problems (like no graphics).
Comment 3 Ulrich Drepper 2011-07-21 01:47:06 UTC
I checked in a patch.
Comment 4 Andreas Schwab 2011-07-21 12:48:31 UTC
Which is of course wrong.
Comment 5 Ulrich Drepper 2011-07-23 02:10:57 UTC
Don't reopen bugs like this.
Comment 6 Andy Lutomirski 2011-07-23 02:19:18 UTC
Should it be a new bug, then?  From my admittedly imperfect recollection of x86 assembly, this:

+       testl   $((1 << 28) | (1 << 27)), %ecx
+       je      2f

is checking whether one of the bits is set, not whether both are set.  I think you'll get SIGILL when you do xgetbv if OSXSAVE isn't set.  (And you'll get a different failure if YMM is enabled but AVX isn't present, in the unlikely event that such a CPU exists.)