[PATCH] xdr_array and calloc security fix

[Jakub Jelinek]

On Fri, Aug 02, 2002 at 01:07:53PM +0200, Andreas Schwab wrote:
> Jakub Jelinek <jakub@redhat.com> writes:
> 
> |> On Fri, Aug 02, 2002 at 02:50:40AM -0700, Ulrich Drepper wrote:
> |> > Ulrich Drepper wrote:
> |> > 
> |> > > It should be possible to have something like
> |> > > 
> |> > >   ((a | b) > (a * b))
> |> > > 
> |> > > for unsigned values.  I'm not 100% sure, though.
> |> > 
> |> > I mean, this is an approximation which lets us avoid the division in 
> |> > many (most?) cases.
> |> 
> |> Many. a=1 b=2 -> is this overflow?
> |> a=0x6000000 b=64 -> this would signal no overflow, while in fact
> 
> But (a > a * b || b > a * b) should work, shouldn't it?

No. For a=1 b=2 this will give the correct answer (no overflow), but
for a=0x6000000 b=64 it will give incorrect one (no overflow, while
0x180000000LL certainly doesn't fit into 32-bits (but 0x80000000 is
still bigger than any of the operands).

	Jakub