[binutils-gdb] Fix HP/PA GNU/Linux "long double" format

[Pedro Alves <palves at sourceware dot org>]

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=aacca8a7a9c7f93955fa9dbf796b030ffce1b956

commit aacca8a7a9c7f93955fa9dbf796b030ffce1b956
Author: Pedro Alves <palves@redhat.com>
Date:   Wed Mar 9 01:50:02 2016 +0000

    Fix HP/PA GNU/Linux "long double" format
    
    This:
    
     $ ./gdb -ex "set architecture hppa1.0" -ex "set osabi GNU/Linux" -ex "ptype 1.0L"
    
    Shows that HPPA/Linux support for long doubles is broken.  It causes
    GDB to access memory out of bounds.  With Valgrind, we see:
    
     The target architecture is assumed to be hppa1.0
     ==4371== Invalid write of size 8
     ==4371==    at 0x4C2F21F: memset (vg_replace_strmem.c:1224)
     ==4371==    by 0x8451C4: convert_doublest_to_floatformat (doublest.c:362)
     ==4371==    by 0x845F86: floatformat_from_doublest (doublest.c:769)
     ==4371==    by 0x84628E: store_typed_floating (doublest.c:873)
     ==4371==    by 0x6A7C3D: value_from_double (value.c:3662)
     ==4371==    by 0x6AA211: evaluate_subexp_standard (eval.c:745)
     ==4371==    by 0x7F306D: evaluate_subexp_c (c-lang.c:716)
     ==4371==    by 0x6A8C6A: evaluate_subexp (eval.c:79)
     ==4371==    by 0x6A8E87: evaluate_type (eval.c:174)
     ==4371==    by 0x817B8D: whatis_exp (typeprint.c:456)
     ==4371==    by 0x817D68: ptype_command (typeprint.c:508)
     ==4371==    by 0x5F2977: do_cfunc (cli-decode.c:105)
     ==4371==  Address 0x8998d18 is 0 bytes after a block of size 8 alloc'd
     ==4371==    at 0x4C2AA98: calloc (vg_replace_malloc.c:711)
     ==4371==    by 0x8732B6: xcalloc (common-utils.c:83)
     ==4371==    by 0x8732F5: xzalloc (common-utils.c:93)
     ==4371==    by 0x6A37AF: allocate_value_contents (value.c:1036)
     ==4371==    by 0x6A37E5: allocate_value (value.c:1047)
     ==4371==    by 0x6A7BEE: value_from_double (value.c:3656)
     ==4371==    by 0x6AA211: evaluate_subexp_standard (eval.c:745)
     ==4371==    by 0x7F306D: evaluate_subexp_c (c-lang.c:716)
     ==4371==    by 0x6A8C6A: evaluate_subexp (eval.c:79)
     ==4371==    by 0x6A8E87: evaluate_type (eval.c:174)
     ==4371==    by 0x817B8D: whatis_exp (typeprint.c:456)
     ==4371==    by 0x817D68: ptype_command (typeprint.c:508)
    
    The trouble is that hppa_linux_init_abi overrides the default
    long_double_bit set by the generic hppa-tdep.c:
    
      set_gdbarch_long_double_bit (gdbarch, 128);
      set_gdbarch_long_double_format (gdbarch, floatformats_ia64_quad);
    
    with:
    
      /* On hppa-linux, currently, sizeof(long double) == 8.  There has been
         some discussions to support 128-bit long double, but it requires some
         more work in gcc and glibc first.  */
      set_gdbarch_long_double_bit (gdbarch, 64);
    
    which misses overriding the long_double_format, so we end with a weird
    combination of:
    
      set_gdbarch_long_double_bit (gdbarch, 64);
      set_gdbarch_long_double_format (gdbarch, floatformats_ia64_quad);
    
    Weird because floatformats_ia64_quad's totalsize is longer than 64-bits.
    
    The floatformat conversion routines use the struct floatformat's
    totalsize (in bits) to know how much to copy/convert, thus the buffer
    overruns.
    
    gdb/ChangeLog:
    2016-03-09  Pedro Alves  <palves@redhat.com>
    
    	* hppa-linux-tdep.c (hppa_linux_init_abi): Set the long double
    	format to floatformats_ieee_double.

Diff:
---
 gdb/ChangeLog         | 5 +++++
 gdb/hppa-linux-tdep.c | 1 +
 2 files changed, 6 insertions(+)

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 3f020ff..7303c32 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,8 @@
+2016-03-09  Pedro Alves  <palves@redhat.com>
+
+	* hppa-linux-tdep.c (hppa_linux_init_abi): Set the long double
+	format to floatformats_ieee_double.
+
 2016-03-07  Pedro Alves  <palves@redhat.com>
 
 	* mips-tdep.c (mips_gdbarch_init): Check whether info.abfd is NULL
diff --git a/gdb/hppa-linux-tdep.c b/gdb/hppa-linux-tdep.c
index 4013dfd..86b8d14 100644
--- a/gdb/hppa-linux-tdep.c
+++ b/gdb/hppa-linux-tdep.c
@@ -518,6 +518,7 @@ hppa_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
      some discussions to support 128-bit long double, but it requires some
      more work in gcc and glibc first.  */
   set_gdbarch_long_double_bit (gdbarch, 64);
+  set_gdbarch_long_double_format (gdbarch, floatformats_ieee_double);
 
   set_gdbarch_iterate_over_regset_sections
     (gdbarch, hppa_linux_iterate_over_regset_sections);