[Bug string/24024] New: strerror() might set errno to ENOMEM due to -fno-math-error

["aurelien at aurel32 dot net" <sourceware-bugzilla at sourceware dot org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=24024

            Bug ID: 24024
           Summary: strerror() might set errno to ENOMEM due to
                    -fno-math-error
           Product: glibc
           Version: 2.28
            Status: NEW
          Severity: normal
          Priority: P2
         Component: string
          Assignee: unassigned at sourceware dot org
          Reporter: aurelien at aurel32 dot net
  Target Milestone: ---

Commit 1294b1892e ("Add support for sqrt asm redirects") added the
-fno-math-errno flag to build most of the glibc in order to enable GCC to
inline math function.  This allows GCC to optimize-out saving and restoring
errno around function calls. In turn this causes strerror to set the errno
value to ENOMEM in case of an invalid error number and memory allocation error,
which is not allowed by POSIX.

This happens for example when running the following example under qemu-arm:

#include <stdio.h>
#include <string.h>
#include <errno.h>

int main(void)
{
        errno=0;
        char *msg=strerror(-3);
        printf("errno=%d, msg=%s\n", errno, msg);

        return 0;
}

The strerror function in glibc is the following:

char *
strerror (int errnum)
{
  char *ret = __strerror_r (errnum, NULL, 0);
  int saved_errno;

  if (__glibc_likely (ret != NULL))
    return ret;
  saved_errno = errno;
  if (buf == NULL)
    buf = malloc (1024);
  __set_errno (saved_errno);
  if (buf == NULL)
    return _("Unknown error");
  return __strerror_r (errnum, buf, 1024);
}

For unrelated/unknown reason, malloc (1024) succeed under qemu-arm but sets
errno to ENOMEM. When this code is built with -fno-math-errno, the code
corresponding to save_errno = errno and __set_errno (saved_errno) is
optimized-out, so strerror ends up setting errno to ENOMEM.

I have checked the generated code to confirm that saving and restoring errno is
optimized-out with the -fno-math-errno option, but not without -f-math-errno.
This can also be checked on the code generated by x86_64 GCC (version 5 to 8
and snapshot from 2018-12-17), although I haven't found a way to exercise this
code path.

qsort() also saves and restores errno around a malloc call and might be
affected.

I think we should only build libm with -fno-math-errno and build all other
parts with -fmath-errno. This should be safe as libm doesn't contain any code
saving and restoring errno.

-- 
You are receiving this mail because:
You are on the CC list for the bug.