soft-fp: Fix _FP_TO_INT latent bug in overflow handling

[Joseph S. Myers]

This patch, relative to a tree with
<https://sourceware.org/ml/libc-alpha/2014-09/msg00461.html> and
<https://sourceware.org/ml/libc-alpha/2014-09/msg00463.html> (pending
review) applied, fixes a latent bug in _FP_TO_INT regarding handling
of arguments with maximum exponent (infinities and NaNs).  If the
maximum exponent is below that calculated as an overflow threshold,
such values would incorrectly be treated as normal values for the
purposes of the conversion.  This could not occur for any of the
conversions actually occurring in glibc, libgcc or the Linux kernel
(the maximum exponent for float is, just, big enough to ensure
overflow for unsigned __int128), but would apply if soft-fp were used
for IEEE binary16.  Appropriate checks are inserted to ensure that the
maximum exponent is always treated as an overflowing exponent, and
never as a normal one.

Tested for powerpc-nofpu that the disassembly of installed shared
libraries is unchanged by this patch.

2014-09-19  Joseph Myers  <joseph@codesourcery.com>

	* soft-fp/op-common.h (_FP_TO_INT): Ensure maximum exponent is
	treated as invalid conversion, not as normal exponent.

diff --git a/soft-fp/op-common.h b/soft-fp/op-common.h
index 833658d..8c928af 100644
--- a/soft-fp/op-common.h
+++ b/soft-fp/op-common.h
@@ -1381,7 +1381,9 @@
 	  else								\
 	    FP_SET_EXCEPTION (FP_EX_INEXACT);				\
 	}								\
-      else if (X##_e >= _FP_EXPBIAS_##fs + rsize - (rsigned > 0 || X##_s) \
+      else if (X##_e >= (_FP_EXPMAX_##fs < _FP_EXPBIAS_##fs + rsize	\
+			 ? _FP_EXPMAX_##fs				\
+			 : _FP_EXPBIAS_##fs + rsize - (rsigned > 0 || X##_s)) \
 	       || (!rsigned && X##_s))					\
 	{								\
 	  /* Overflow or converting to the most negative integer.  */	\
@@ -1398,7 +1400,10 @@
 		r = ~r;							\
 	    }								\
 									\
-	  if (rsigned && X##_s && X##_e == _FP_EXPBIAS_##fs + rsize - 1) \
+	  if (_FP_EXPBIAS_##fs + rsize - 1 < _FP_EXPMAX_##fs		\
+	      && rsigned						\
+	      && X##_s							\
+	      && X##_e == _FP_EXPBIAS_##fs + rsize - 1)			\
 	    {								\
 	      /* Possibly converting to most negative integer; check the \
 		 mantissa.  */						\

-- 
Joseph S. Myers
joseph@codesourcery.com