Improve overflow detection in gdbserver

[ekurzinger@nvidia.com]

Hi GDB Team,

Currently, the function used by gdbserver to parse integers from
received commands will detect overflow and fail for any value over
0xfffffff. Among other things, this has the effect of limiting the
file offsets for reading or writing to about 268MB which can be
insufficient for particularly large libraries.

This change allows the parsing of integers up to the true maximum
positive value of 0x7fffffff, increasing the file size limit to
about 2GB.

Note that I don't currently have a copyright assignment form on file,
but your contributor guidlines state that one is not required for
minor changes, so I was hoping this would qualify.

Also, just wanted to say I really appreciate the work you folks do on
this awesome tool, and am glad to be able to make a contribution
(however small it may be)!

Cheers,
Erik

diff --git a/gdb/gdbserver/ChangeLog b/gdb/gdbserver/ChangeLog
index 5e7ea108b5..58a5f2c30c 100644
--- a/gdb/gdbserver/ChangeLog
+++ b/gdb/gdbserver/ChangeLog
@@ -1,3 +1,7 @@
+2018-05-18  Erik Kurzinger  <ekurzinger@nvidia.com>
+	* hostio.c (require_int): do not report overflow
+	for integers between 0xfffffff and 0x7fffffff
+
 2018-05-10  Joel Brobecker  <brobecker@adacore.com>
 
 	* lynx-i386-low.c (LYNXOS_178): New macro.
diff --git a/gdb/gdbserver/hostio.c b/gdb/gdbserver/hostio.c
index d2b5a71bad..c621edfef5 100644
--- a/gdb/gdbserver/hostio.c
+++ b/gdb/gdbserver/hostio.c
@@ -96,22 +96,27 @@ static int
 require_int (char **pp, int *value)
 {
   char *p;
-  int count;
+  int count, firstdigit;
 
   p = *pp;
   *value = 0;
   count = 0;
+  firstdigit = -1;
 
   while (*p && *p != ',')
     {
       int nib;
 
-      /* Don't allow overflow.  */
-      if (count >= 7)
+      if (safe_fromhex (p[0], &nib))
 	return -1;
 
-      if (safe_fromhex (p[0], &nib))
+      if (firstdigit == -1)
+	firstdigit = nib;
+
+      /* Don't allow overflow.  */
+      if (count >= 8 || (count == 7 && firstdigit >= 0x8))
 	return -1;
+
       *value = *value * 16 + nib;
       p++;
       count++;