unnormal Intel 80-bit long doubles and isnanl

[Siddhesh Poyarekar]

Hi,

The Intel 80-bit long double format has a concept of "unnormal" numbers 
that have a non-zero exponent and zero integer bit (i.e. bit 63) in the 
mantissa; all valid long double numbers have their integer bit set to 1. 
  Unnormal numbers are mentioned in "8.2.2 Unsupported Double 
Extended-Precision Floating-Point Encodings and Pseudo-Denormals" and 
listed in Table 8-3 in the Intel 64 and IA-32 Architectures Software 
Developer’s Manual Volume 1:Basic Architecture.

As per the manual, these numbers are considered unsupported and generate 
an invalid-operation exception if they are used as operands to any 
floating point instructions.  The question of this email is how the 
toolchain (including glibc) should treat these numbers since as things 
stand today, glibc and gcc disagree when it comes to isnanl.

glibc evaluates the bit pattern of the 80-bit long double and in the 
process, ignores the integer bit, i.e. bit 63.  As a result, it 
considers the unnormal number as a valid long double and isnanl returns 0.

gcc on the other hand, simply uses the number in a floating point 
comparison and uses the parity flag (which indicates an unordered 
compare, signalling a NaN) to decide if the number is a NaN.  The 
unnormal numbers behave like NaNs in this respect, in that they set the 
parity flag and with -fsignalling-nans, would result in an 
invalid-operation exception.  As a result, __builtin_isnanl returns 1 
for an unnormal number.

So the question is, which behaviour should be considered correct? 
Strictly speaking, unnormal numbers are listed separately from NaNs in 
the document and as such are distinct from NaNs.  So on the question of 
"is nan?" the answer ought to be "No".

On the flip side, the behaviour described (and experienced through code) 
is exactly the same as a NaN, i.e. a floating point operation sets the 
parity flag and generates an invalid-operation exception.  So if it 
looks like a NaN, behaves like a NaN, then even if the document hints 
(and it is just a hint right, since it doesn't specifically state it?) 
that it's different, it likely is a NaN.  What's more, one of the fixes 
to glibc[1] assumes that __builtin_isnanl will do the right thing.

The third alternative (which seems like a step back to me, but will 
concede that it is a valid resolution) is to state that unnormal input 
to isnanl would result in undefined behaviour and hence it is the 
responsibility of the application to ensure that inputs to isnanl are 
never unnormal.

Thoughts?

Siddhesh

[1] 
https://sourceware.org/git/?p=glibc.git;h=0474cd5de60448f31d7b872805257092faa626e4