vfprintf typing problem

[Carlos O'Donell]

On Sun, Apr 1, 2012 at 12:20 AM, David Miller <davem@davemloft.net> wrote:
> From: "Carlos O'Donell" <carlos@systemhalted.org>
> Date: Sat, 31 Mar 2012 22:10:44 -0400
>
>> Does that make sense?
>
> Yep.
>
>> The check probably has to be:
>>
>> if (retval > INT_MAX/10-digit)
>>   return -1;
>> retval *= 10;
>> retal += digit;
>>
>> That way you *know* the next two operations won't overflow, and either
>> you exit or get another chance at looking at the next operation.
>
> Ok, I'll let this patch cook in my build/testsuite runs overnight:
>
> diff --git a/stdio-common/printf-parse.h b/stdio-common/printf-parse.h
> index bcf914b..2038541 100644
> --- a/stdio-common/printf-parse.h
> +++ b/stdio-common/printf-parse.h
> @@ -75,10 +75,12 @@ read_int (const UCHAR_T * *pstr)
>
>   while (ISDIGIT (*++(*pstr)))
>     {
> -      retval *= 10;
> -      if (retval > INT_MAX)
> +      unsigned int digit = **pstr - L_('0');
> +
> +      if (retval > INT_MAX/10 - digit)
>        return -1;
> -      retval += **pstr - L_('0');
> +      retval *= 10;
> +      retval += digit;
>     }
>
>   return (int) retval;

Shucks... if you look closely I *did* do the math wrong.

It should be (retval > (INT_MAX - digit)/10)

That way when you rearrange it becomes:

retval*10+digit > INT_MAX

But because of the rearrangement you're assured you check before any
possible unsigned int or signed int overflow.

Cheers,
Carlos.