The behavior of isnan/__builtin_isnan on bit patterns that do not correspond to something that the CPU would produce from valid inputs is currently under-defined in the toolchain. (The GCC built-in and glibc disagree.) The isnan check in PRINTF_FP_FETCH in stdio-common/printf_fp.c assumes the GCC behavior that returns true for non-normal numbers which are not specified as NaN. (The glibc implementation returns false for such numbers.) At present, passing non-normal numbers to __mppn_extract_long_double causes this function to produce irregularly shaped multi-precision integers, triggering undefined behavior in __printf_fp_l. With GCC 10 and glibc 2.32, this behavior is not visible because __builtin_isnan is used, which avoids calling __mppn_extract_long_double in this case.
Patch posted: https://sourceware.org/pipermail/libc-alpha/2020-September/117779.html
Fixed for glibc 2.33 via: commit 681900d29683722b1cb0a8e565a0585846ec5a61 Author: Florian Weimer <fweimer@redhat.com> Date: Tue Sep 22 19:07:48 2020 +0200 x86: Harden printf against non-normal long double values (bug 26649) The behavior of isnan/__builtin_isnan on bit patterns that do not correspond to something that the CPU would produce from valid inputs is currently under-defined in the toolchain. (The GCC built-in and glibc disagree.) The isnan check in PRINTF_FP_FETCH in stdio-common/printf_fp.c assumes the GCC behavior that returns true for non-normal numbers which are not specified as NaN. (The glibc implementation returns false for such numbers.) At present, passing non-normal numbers to __mpn_extract_long_double causes this function to produce irregularly shaped multi-precision integers, triggering undefined behavior in __printf_fp_l. With GCC 10 and glibc 2.32, this behavior is not visible because __builtin_isnan is used, which avoids calling __mpn_extract_long_double in this case. This commit updates the implementation of __mpn_extract_long_double so that regularly shaped multi-precision integers are produced in this case, avoiding undefined behavior in __printf_fp_l.
*** Bug 4586 has been marked as a duplicate of this bug. ***