Incorrect data detected in the nested float struct with x86/libffi on Linux/64bit

[Kaz Kylheku (libffi)]

On 2021-06-09 09:41, Cheng Jin via Libffi-discuss wrote:
> Hi,
> 
> We found that the nested struct [[float, float], float] works good with
> x86/libffi (at https://github.com/libffi/libffi/tree/master/src/x86)
> on Linux/64bit (verified on Ubuntu 19) while a variant of it like
> [float,[float,float]] was messed up at the 3rd element when invoking
> ffi_call() in the following test:

In the libffi that comes with Ubuntu 18.04 Server, x84-64, your test
case aborts in ffi_call.

I built a similar test case in which I put your function into a shared
lib, and used a libffi-based programming language to call it. Same
thing: abort in ffi_call.

There is some problem in that version of libffi; it doesn't like small
structures. If I add padding to the structure, then there is no abort.

My version of the function looks like this:

float_t testNestedFloatStruct(float_t arg1, stru_Nested_F arg2)
{
   float_t floatSum = arg1 + arg2.elem1 + arg2.elem2.elem1 + 
arg2.elem2.elem2;
   puts("called");
   printf("arg1 = %f, arg2 = { %f, { %f, %f } }\n", arg1, arg2.elem1,
          arg2.elem2.elem1, arg2.elem2.elem2);
   return floatSum;
}

On the FFI side, if I add a uint64's worth of padding to the structure,
the ffi_call proceeds without abort, and the result is like this:

arg1 = 1.000000, arg2 = { 0.000000, { 0.000000, 
-52784298695107543040.000000 } }
-5.27842986951075e19

The second line is the caller printing the return value. As you can
see, complete rubbish.

If I take out the padding and change everything to double (my float_f
typedef lets me do that easily), it looks like this:

arg1 = 1.000000, arg2 = { 2.000000, { 3.000000, 4.000000 } }
10.0

That's the expected behavior; the all-float case should look the same.

Basically with the padding, although the call works, it's using the 
wrong
calling conventions. I think this small structure is supposed to be 
passed
in registers.

If I step over the ffi_prep_cif call to see what args were passed to it
and how the sizes and alignment were set, all looks good:

ffi_make_call_desc (ntotal=<optimized out>, nfixed=<optimized out>,
     rettype=0x7ffff7f97780, argtypes=<optimized out>, name_in=<optimized 
out>)
     at ffi.c:4831
4831      if (ffis != FFI_OK)
(gdb) p args[0]
$5 = (ffi_type *) 0x758c80 <ffi_type_float>
(gdb) p args[1]
$6 = (ffi_type *) 0xc38220
(gdb) p (*args[1])
$7 = {size = 12, alignment = 4, type = 13, elements = 0xc38240}
(gdb) p (*args[1]).elements[0]
$8 = (struct _ffi_type *) 0x758c80 <ffi_type_float>
(gdb) p (*args[1]).elements[1]
$9 = (struct _ffi_type *) 0xb92160
(gdb) p (*(*args[1]).elements[1])
$10 = {size = 8, alignment = 4, type = 13, elements = 0xb92180}
(gdb) p (*(*args[1]).elements[1]).elements[0]
$11 = (struct _ffi_type *) 0x758c80 <ffi_type_float>
(gdb) p (*(*args[1]).elements[1]).elements[1]
$12 = (struct _ffi_type *) 0x758c80 <ffi_type_float>
(gdb) p (*(*args[1]).elements[1]).elements[2]
$13 = (struct _ffi_type *) 0x0
(gdb) p (*args[1]).elements[2]
$14 = (struct _ffi_type *) 0x0


Both elements arrays have two elements and a null terminator. Two floats
at the leaves, and a float + struct at the first level.