[BZ #14090] Fix md5/sha512 with large block sizes

[Andreas Jaeger <aj at suse dot com>]

Paul attached the following patches to bugzilla.

Bugreport:

  "crypt/md5.c: md5_process_block() (lines 319..321) contains this code
  for total length handling. With large block sizes (ctx->total[0] +
  len) can exceed 8 GB, thus the total[1] needs to be incremented more
  than by one:

  ------------------------------------------------------------------------------
  /* First increment the byte count.  RFC 1321 specifies the possible
     length of the file up to 2^64 bits.  Here we only compute the
     number of bytes.  Do a double word increment.  */
  ctx->total[0] += len;
  if (ctx->total[0] < len)
    ++ctx->total[1];"

I've tested the patches on x86-64 and also run the test program for md5
manually. Due to the nature of the program (it links against openssl
crypto lib) and uses quite some memory, we cannot add this to glibc,

Ok?
Andreas

2012-07-26 Paul Eggert <eggert@cs.ucla.edu>

        [BZ #14090]
        * crypt/md5.c (md5_process_block): Don't assume the buffer
        length is less than 2**32.
	* crypt/sha512.c (sha512_process_block): Don't assume the buffer
        length is less than 2**64.

diff --git a/crypt/md5.c b/crypt/md5.c
index 292bee1..3d2e79b 100644
--- a/crypt/md5.c
+++ b/crypt/md5.c
@@ -1,7 +1,6 @@
 /* Functions to compute MD5 message digest of files or memory blocks.
    according to the definition of MD5 in RFC 1321 from April 1992.
-   Copyright (C) 1995,1996,1997,1999,2000,2001,2005,2011
-	Free Software Foundation, Inc.
+   Copyright (C) 1995-2012 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
 
    The GNU C Library is free software; you can redistribute it and/or
@@ -312,13 +311,13 @@ md5_process_block (buffer, len, ctx)
   md5_uint32 B = ctx->B;
   md5_uint32 C = ctx->C;
   md5_uint32 D = ctx->D;
+  md5_uint32 lolen = len;
 
   /* First increment the byte count.  RFC 1321 specifies the possible
      length of the file up to 2^64 bits.  Here we only compute the
      number of bytes.  Do a double word increment.  */
-  ctx->total[0] += len;
-  if (ctx->total[0] < len)
-    ++ctx->total[1];
+  ctx->total[0] += lolen;
+  ctx->total[1] += (len >> 31 >> 1) + (ctx->total[0] < lolen);
 
   /* Process all bytes in the buffer with 64 bytes in each round of
      the loop.  */
diff --git a/crypt/sha512.c b/crypt/sha512.c
index 6e531c5..bec7bb3 100644
--- a/crypt/sha512.c
+++ b/crypt/sha512.c
@@ -1,6 +1,6 @@
 /* Functions to compute SHA512 message digest of files or memory blocks.
    according to the definition of SHA512 in FIPS 180-2.
-   Copyright (C) 2007, 2011 Free Software Foundation, Inc.
+   Copyright (C) 2007-2012 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
 
    The GNU C Library is free software; you can redistribute it and/or
@@ -123,9 +123,10 @@ sha512_process_block (const void *buffer, size_t len, struct sha512_ctx *ctx)
 #ifdef USE_TOTAL128
   ctx->total128 += len;
 #else
-  ctx->total[TOTAL128_low] += len;
-  if (ctx->total[TOTAL128_low] < len)
-    ++ctx->total[TOTAL128_high];
+  uint64_t lolen = len;
+  ctx->total[TOTAL128_low] += lolen;
+  ctx->total[TOTAL128_high] += ((len >> 63 >> 1)
+				+ (ctx->total[TOTAL128_low] < lolen));
 #endif
 
   /* Process all bytes in the buffer with 128 bytes in each round of

-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
  SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 NÃrnberg, Germany
   GF: Jeff Hawn,Jennifer Guild,Felix ImendÃrffer,HRB16746 (AG NÃrnberg)
    GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126