[PATCH v2 2/2] malloc: make malloc fail with requests larger than PTRDIFF_MAX

[Paul Eggert]

Adhemerval Zanella wrote:
> Changes from previous version:

>    - Remove padsize check, requested size after aligning and padding
>      are checked against PTRDIFF_MAX.  This does not guarantee that
>      sysmalloc will eventually issue an request larger than PTRDIFF_MAX.
>      One option to enforce it is to either add the PTRDIFF_MAX check on
>      mmap or create an internal mmap implementation to enforce it.

Sorry, I'm having trouble parsing that, starting with the word "guarantee"; it 
sounds like you're saying sysmalloc must issue a request larger than 
PTRDIFF_MAX, which surely is the opposite of what is intended.

More important, I thought that the idea was to allow malloc (s) if s <= 
PTRDIFF_MAX, even if s > PTRDIFF_MAX - DELTA for some nonzero but small DELTA. 
See <https://www.sourceware.org/ml/libc-alpha/2018-12/msg00908.html>, where I 
don't understand the point of your reply 
<https://www.sourceware.org/ml/libc-alpha/2018-12/msg00953.html>, as the concern 
about the compiler is the compiler's assumption about the object that malloc 
returns, not about how glibc behaves internally (I am assuming that glibc is 
careful to never subtract pointers whose integer representations differ by more 
than PTRDIFF_MAX).

> +/* Check if REQ overflows when padded and aligned and if the resulting value
> +   is less than PTRDIFF_T.  Returns TRUE and the requested size or MINSIZE in
> +   case the value is less than MINSIZE on SZ or false if any of the previous
> +   check fail.  */
> +static inline bool
> +checked_request2size (size_t req, size_t *sz) __nonnull (1)
> +{
> +  size_t ret;
> +  if (__glibc_unlikely (__builtin_add_overflow (req,
> +						SIZE_SZ + MALLOC_ALIGN_MASK,
> +						&ret)))
> +    return false;
> +
> +  ret = ret < MINSIZE ? MINSIZE : ret & ~MALLOC_ALIGN_MASK;
> +  if (__glibc_unlikely (ret > PTRDIFF_MAX))
> +    return false;
> +
> +  *sz = ret;
> +  return true;
> +}

This function typically does 3 comparisons. Since we're assuming PTRDIFF_MAX <= 
SIZE_MAX / 2, we can shrink it to at most 2 comparisons. Something like the 
following, say, where on x86-64 the typical case is one conditional branch and 
one conditional move.

static inline bool
checked_request2size (size_t req, size_t *sz)
{
   if (__glibc_likely (req <= PTRDIFF_MAX))
     {
       int min_nontiny_size = SIZE_SZ < MINSIZE ? MINSIZE - SIZE_SZ : 0;
       size_t areq = MAX (min_nontiny_size, req);
       *sz = (areq + SIZE_SZ + MALLOC_ALIGN_MASK) & ~MALLOC_ALIGN_MASK;
       return true;
     }
   return false;
}