[PATCH RESEND 01/14] stdio-common: Add tests for formatted scanf input specifiers

[Maciej W. Rozycki]

On Sat, 15 Feb 2025, Joseph Myers wrote:

> > +   We need to be careful with parsing reference data in the handling of
> > +   the sign as with the IBM long double binary format unary negation
> > +   of a positive value whose lower-magnitude part is zero produces a
> > +   bit pattern that is different from one produced by initialization or
> > +   conversions made by functions such as 'scanf' or 'strtod' from the
> > +   complementing negative value.  The difference is in the sign of zero
> > +   held in the lower-magnitude part and therefore the bit patterns are
> > +   arithmetically equivalent, but they do not match byte-wise.
> > +
> > +   Therefore we set the sign with the initial value of the significand,
> > +   either -0.0 or 0.0, which sets the sign of both parts according to
> > +   what 'scanf' does, and never negate it afterwards.
> > +
> > +   Additionally the Intel long double binary format uses only 80 bits
> > +   out of 96 that the data type occupies.  Therefore preinitialize the
> > +   reference value with the same 0xa5 bit pattern that the value under
> > +   test has been so that the byte-wise comparison matches as well for
> > +   the unused parts of the two data pieces.
> 
> I don't think it's safe to make any assumptions about whether padding bits 
> of x86 (or m68k?) long double are or are not modified by scanf; that may 
> depend on the details of what instructions the compiler happens to 
> generate in scanf to store the long double value that's been read.  So I 
> don't think the strategy of comparing with memcmp is safe in that case, 
> unless you first copy just the value bits (and not padding bits) to a 
> character array (or other type without room for the padding bits) before 
> comparing.

 Speaking of x86 only I fail to see why the compiler should access the 
unused 16-bits beyond the underlying hardware data type.  None of the FPU 
instructions does, based on my past experience and I have double-checked 
with Intel SDM too.  NB this format is x87-only and therefore very old, 
going back to 1980, so I expect hardware implementations to be very 
conservative lest they break software compatibility.

 So any access there would be by an explicit store of some data only to 
the sign/exponent component, presumably by an integer instruction, though 
32-bit FIST/FISTP could be thought of too even if barely.  But then why do 
a 32-bit store when 16-bit ones are available and can do exactly the same 
without clobbering bits outside actual data?  Then it's not the kind of 
operation I'd expect a compiler to produce for an FP operation, and if it 
were produced from our code, then obviously it would be completely under 
our control.

 This testset indicates no such store happens and while I agree the 
contents of the padding bits are unspecified, I think it is going to be 
worthwhile to keep it verified that the current situation does not change 
without raising awareness.  With no such test it may happen inadvertently.  
With it in place we can decide at the time whether to back out the change 
or to update the test accordingly.

 I can't speak of m68k offhand, but given that likewise it comes from the 
time when memory was expensive I'd expect m68k hardware not to poke at 
bits beyond the FP datum itself.  I could double-check I suppose, though I 
don't have m68k hardware to hand, so that would be by documentation only.

 The verifier handler can be easily adjusted if we decide it's indeed 
needed with a sysdep-specific variant of tst-scanf-format-real.h.  I'm not 
convinced it is at this point though.

> For IBM long double, the cases where the bit pattern is not fully defined 
> are (a) the low 64 bits of a NaN are entirely unspecified; (b) if the low 
> part is zero, its sign does not matter (regardless of the value of the 
> high part).  You're less likely to have things varying with the details of 
> compiler code generation there, however.

 My understanding has been (and please correct me if I'm wrong) that the 
IBM long double format is software-assisted only; I have double-checked 
with documentation I have to hand and at least "Power ISA, Version 3.1" 
dated May 1, 2020 only mentions 32-bit single and 64-bit double binary 
hardware FP formats.

 So actually we have less control here with actual FP operations made by 
libgcc or whatever equivalent other compilers may use.  As I noted for the 
time being I have carefully avoided the ambiguities in the results at 
least for the libgcc implementation, and maybe it would be worth it if the 
implementation was updated such as to make the sign of low-part zero agree 
with the sign of the high part.

 Again a sysdep-specific variant of tst-scanf-format-real.h could be 
provided instead, even if checking for the special cases of NaN data and 
zero low-part, but I'm not convinced at this point.

> For NaNs in general, if a payload is *not* specified in the scanf string 
> then you'll get whatever default NaN scanf returns, while computing some 
> default NaN in the test code.  Maybe these are in fact reliably the same, 
> but it's beyond the API I'd consider to be guaranteed.

 With ISO C being deliberately vague here I think implementation-defined 
does not preclude from being consistent, under the principle of least 
surprise.  I'd expect the canonical NaN encoding to be always produced 
where a specific payload has not been requested (which we ignore anyway), 
and we can certainly verify for output to stay consistent.

> In short: for x86/m68k long double I think the comparison approach used is 
> unreliable, depending on the compiler version, because it relies on 
> padding bits not changing.  For IBM long double and for NaNs without 
> payload specified, I think comparing using memcmp means you're at least 
> testing internal implementation details (whether intentionally or 
> otherwise) that go beyond what I think is properly part of the specified 
> API for glibc.

 I think there is value in verifying the consistency of the implementation 
even if it goes beyond the specified API.

 NB I note that tst-scanf-format-c-c.input data marked BZ12701 cannot be 
made to work with this testset for the lack of a way (except for `sscanf', 
`vsscanf') in the current implementation to force EOF at the separator 
colon.  An independent testset will be needed, but I'm going to keep these 
markers for the time being for documentation purposes, until the part of 
BZ #12701 concerned has been resolved.

  Maciej