Incorrect data detected in the nested float struct with x86/libffi on Linux/64bit

[Jakub Jelinek]

On Wed, Jun 09, 2021 at 12:50:08PM -0400, Anthony Green wrote:
> Thank you, Cheng.  Are you able to submit this test case as a github pull
> request?  The resulting CI testing will give us a broader picture of where
> we have problems.

Comparing gcc/config/i386/ classify_argument and libffi classify_argument,
I found two important differences.

The first one seems the most important one, even GCC 3.2 included the bit
offset (byte offset in libffi) in the calculation of number of words.
And the other change is https://gcc.gnu.org/PR38781.

With this patch the posted testcase works and the testsuite on x86_64-linux
still passes, but haven't done more testing than that.

Haven't tried yet to adapt one of the
libffi/testsuite/libffi.call/nested_struct*.c tests to cover this though.

2021-06-15  Jakub Jelinek  <jakub@redhat.com>

	* src/x86/ffi64.c (classify_argument): For FFI_TYPE_STRUCT set words
	to number of words needed for type->size + byte_offset bytes rather
	than just type->size bytes.  Compute pos before the loop and check
	total size of the structure.

--- libffi/src/x86/ffi64.c.jj	2020-01-14 20:02:48.557583260 +0100
+++ libffi/src/x86/ffi64.c	2021-06-15 19:50:06.059108230 +0200
@@ -217,7 +217,8 @@ classify_argument (ffi_type *type, enum
     case FFI_TYPE_STRUCT:
       {
 	const size_t UNITS_PER_WORD = 8;
-	size_t words = (type->size + UNITS_PER_WORD - 1) / UNITS_PER_WORD;
+	size_t words = (type->size + byte_offset + UNITS_PER_WORD - 1)
+		       / UNITS_PER_WORD;
 	ffi_type **ptr;
 	int i;
 	enum x86_64_reg_class subclasses[MAX_CLASSES];
@@ -241,16 +242,16 @@ classify_argument (ffi_type *type, enum
 	/* Merge the fields of structure.  */
 	for (ptr = type->elements; *ptr != NULL; ptr++)
 	  {
-	    size_t num;
+	    size_t num, pos;
 
 	    byte_offset = ALIGN (byte_offset, (*ptr)->alignment);
 
 	    num = classify_argument (*ptr, subclasses, byte_offset % 8);
 	    if (num == 0)
 	      return 0;
-	    for (i = 0; i < num; i++)
+	    pos = byte_offset / 8;
+	    for (i = 0; i < num && (i + pos) < words; i++)
 	      {
-		size_t pos = byte_offset / 8;
 		classes[i + pos] =
 		  merge_classes (subclasses[i], classes[i + pos]);
 	      }


	Jakub