[PATCH] malloc: skip allocation retry when encountering faulty requests (bug 29709)

[David Milo]

Steps to reproduce the (bug 29709):

    1. build glibc without tcache (--disable-experimental-malloc)
    2. create a multithreaded process
    3. make an invalid request (> PTRDIFF_MAX) via malloc/memalign

Step 3 then results in a second allocation attempt with a different arena. This
behavior does not make much sense, since retrying with another arena wont fix
the real issue - the invalid request size. Read (bug xx) for more information.

This patch should fix the issue. I see two possible solutions for this problem:

    1. let _int_malloc/_int_memalign set errno to EINVAL instead of ENOMEM
       when encountering a faulty request size. This way, we can check
       errno before starting a second allocation attempt. If errno is
       set to EINVAL, we know that there is an issue with the argument
       itself and a second attempt would not behave differently, even
       if we would switch arenas.

    2. add an extra condition for the second allocation attempt. Only
       start a second attempt if following conditions are met

         - returned pointer is NULL
         - requested size is less or equals PTRDIFF_MAX
         - current arena pointer is not NULL

In my opinion solution 1 is the better choice. It would make more sense
if we set errno to EINVAL instead of ENOMEM whenever we deal with
faulty requests, since EINVAL indicates that invalid arguments were
passed to library functions.

Nevertheless, I decided to go with solution 2 in order to avoid
bigger changes in exposed library functions (malloc/memalign
would then also be able to set EINVAL instead of only ENOMEM).

Signed-off-by: David Milosevic <milod2048@gmail.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bug29709.patch
Type: application/octet-stream
Size: 3027 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/libc-alpha/attachments/20221020/9e208a7f/attachment.obj>