Bug 13246 - RFE: strtold: do not include uninitialized bytes when converting "NaN"
Summary: RFE: strtold: do not include uninitialized bytes when converting "NaN"
Alias: None
Product: glibc
Classification: Unclassified
Component: libc (show other bugs)
Version: 2.14
: P2 normal
Target Milestone: ---
Assignee: Ulrich Drepper
Depends on:
Reported: 2011-10-01 17:38 UTC by jim@meyering.net
Modified: 2014-06-27 11:59 UTC (History)
2 users (show)

See Also:
Last reconfirmed:
fweimer: security-


Note You need to log in before you can comment on or make changes to this bug.
Description jim@meyering.net 2011-10-01 17:38:55 UTC
strtold ("nan", NULL) appears to return a result that depends on the prior state of the stack.  That non-determinism caused GNU sort to infloop, but we've worked around this glibc-specific issue: http://debbugs.gnu.org/9612

Here's a little program to demonstrate the problem.
Note how the results differ with compilation options:

  #include <stdlib.h>
  #include <stdio.h>

  static char *
  fmt_nan (long double x)
    unsigned int i;
    static char buf[33];
    unsigned char const *p = (unsigned char const *) &x;
    for (i = 0; i < sizeof x; i++)
      sprintf (buf + 2*i, "%02x", *p++);
    return buf;

  main ()
    const char *q = "nan";
    long double x = strtold (q, NULL);
    printf ("%s\n", fmt_nan (x));

    x = 0;
    x = strtold (q, NULL);
    printf ("%s\n", fmt_nan (x));

    return 0;

  $ gcc -O0 -Wall -Wextra -W /t/strtold-bogosity.c && ./a.out
  $ gcc -O1 -Wall -Wextra -W /t/strtold-bogosity.c && ./a.out
Comment 1 Andreas Schwab 2011-10-05 14:52:26 UTC
That's a compiler bug.
Comment 2 Rich Felker 2011-10-05 22:09:50 UTC
Please explain how this is a compiler bug. This should not be able to happen unless glibc's implementation of strtold is invoking undefined behavior.
Comment 3 Andreas Schwab 2011-10-06 07:56:35 UTC
glibc does not use any undefined behaviour, thus it is a compiler bug.
Comment 4 Jakub Jelinek 2011-10-06 08:15:00 UTC
I think it is neither.  On x86_64/i686, the hw long double is 80 bit, but sizeof (long double) is 16 (on x86_64) resp. 12 (on i686).  If you store a long double from a hw register into memory, it only stores there the 80 bits, the remaining bits are left unmodified, it is padding that is ignored, thus e.g. if it is stored
onto the stack, whatever bits were in the padding before are kept there.
It is a bug to look for something meaningful among the padding bits.
That is similar to looking at padding in structures, undefined behavior in C.
Comment 5 jim@meyering.net 2011-10-06 08:32:36 UTC
Hi Jakub,

Thanks for commenting.
I realize that this is not officially a bug.  That's why I wrote "RFE: ...".
However, is there some other way to avoid valgrind "warnings" about those stack bytes being used uninitialized?

Alternatively, can you recommend a portable way to compare the valid bits of two NaN values?
Comment 6 Jakub Jelinek 2011-10-06 08:45:17 UTC
First of all, it is impossible to do anything about this on the glibc side.
Just look at the generated code, it is:
  call strtold@plt
  fstpt  (%rsp)          # resp. fstpt (%esp)
For x86_64/i686 it would be sufficient if you used memset to clear the whole var, then store the value into it after making sure the compiler doesn't optimize it away.  Perhaps something like:
  union U { long double l; unsigned char c[sizeof (long double); };
  volatile union U u;
  size_t i;
  for (i = 0; i < sizeof (long double); i++)
    u.c[i] = 0;
  u.l = strtold (...);
would do the job portably?
Comment 7 jim@meyering.net 2011-10-06 09:30:49 UTC
Thanks for the suggestion.
The fix I applied to sort.c was to use a bare memset, since that happened to solve the problem at least with gcc-4.7.x.  Your approach is better if an optimizer makes the false assumption that the bytes returned by strtold are always defined, in which case it might eliminate my seemingly redundant memset, and we'd have to report the optimizer bug.

However, do you really think strtold should continue to return undefined bytes? That implies that any application that must sort (stably) user-supplied floating point numbers including NaNs must jump through these hoops.
Comment 8 Andreas Schwab 2011-10-06 09:40:57 UTC
The return value of strtold does not contain any undefined bytes.
Comment 9 jim@meyering.net 2011-10-06 09:47:47 UTC
"uninitialized" bytes.
Comment 10 Jakub Jelinek 2011-10-06 09:56:33 UTC
Andreas is right, the value is returned in %st(0), which is an 80-bit hw register.
So, as I said earlier, there is really nothing wrong on the strtold side.
The problem is when the compiler stores that %st(0) into memory.  And you really don't want to penalize all long double using code that on each memory store
instead of using fstp (%rsp) there is fstp %(rsp); movw $0, 10(%rsp); movl $0, 12(%rsp);
Comment 11 Rich Felker 2011-10-06 12:39:06 UTC
OK, it's the test program and GNU sort that are wrong (reading padding bits). Unfortunately I don't see a portable way to "compare" NaNs like this. But if they're being read from strings, why not just compare the string form when the value is NaN?