[PATCH] xdr_array and calloc security fix

[Jakub Jelinek]

On Fri, Aug 02, 2002 at 11:57:29AM -0000, Wolfram Gloger wrote:
> > > But (a > a * b || b > a * b) should work, shouldn't it?
> > 
> > No. For a=1 b=2 this will give the correct answer (no overflow), but
> > for a=0x6000000 b=64 it will give incorrect one (no overflow, while
> > 0x180000000LL certainly doesn't fit into 32-bits (but 0x80000000 is
> > still bigger than any of the operands).
> 
> Ok, if we're going to have two comparisions anyway, I'd suggest we
> assume at least 32bits and use
> 
> a >= 46340 || b >= 46340
> 
> (46340 <= sqrt(2^31), if I did my math correctly)
> Of course this will detect some cases as overflow which actually
> aren't, but that is harmless.

Why not 2^32? size_t is unsigned.

So you mean something like:
  bytes = n * elem_size;
  if (__builtin_expect ((a | b) >= 65536, 0)) {
    if (bytes / elem_size != n) {
      MALLOC_FAILURE_ACTION;
      return 0;
    }
  }

(ie. do the division only in the unlikely case)?

	Jakub