New GNU C Library (glibc) security flaw reported on 30 Jan 2024

[Vincent Lefevre]

On 2024-02-01 14:41:04 +0800, Xi Ruoyao wrote:
> On Thu, 2024-02-01 at 01:51 +0100, Vincent Lefevre wrote:
> > On 2024-02-01 02:47:18 +0800, Xi Ruoyao wrote:
> > > On Wed, 2024-01-31 at 12:52 -0300, Adhemerval Zanella Netto wrote:
> > > 
> > > /* snip */
> > > 
> > > > 
> > > > I see this is an manual issue rather than a GNU 'extension' to qsort semantic.
> > > > And I think we should fix BZ#31322 by using a transitive comparison instead of
> > > > trying to support such cases.
> > > 
> > > To me the documentation is correct (though arguably in a very subtle
> > > way):
> > > 
> > >    Here is an example of a comparison function which works with an array
> > > of numbers of type ‘double’:
> > > 
> > >      int
> > >      compare_doubles (const void *a, const void *b)
> > >      {
> > >        const double *da = (const double *) a;
> > >        const double *db = (const double *) b;
> > > 
> > >        return (*da > *db) - (*da < *db);
> > >      }
> > > 
> > > It says "numbers."  But NaN literally means, "Not a Number."
> > 
> > Yes, the point is to sort numbers. But since NaN may occur, the code
> > must not yield undefined behavior in such a case. This is the goal
> > of NaN: avoid undefined behavior for operations that do not make any
> > sense, and be able to detect errors at the end.
> 
> When we sort *numbers* NaN cannot be passed to the comparator.

What I mean is that the intent is to sort numbers. But in any case,
the code needs to consider that NaN may occur; the result would be
an array in an indeterminate order, but the code must not produce
undefined behavior with consequences like memory corruption. If the
code is designed considering that NaN cannot occur, e.g. because
the user is required to ensure that before calling qsort, then
this must explicitly be documented with a non-ambiguous vocabulary
(typically using "assume").

> And I doubt if silently producing a NaN is really good for error
> detection. Simply crashing when an invalid operation happens is
> easier for debugging, IMO. And it's possible with "feenableexcept
> (FE_INVALID)" (where FP exceptions are supported).

Silently producing a NaN on "invalid" inputs is what happens in
practice, following the spec of the IEEE 754 standard. For instance,
sqrt(-1.) silently returns NaN (a flag is also set). But in general,
the user will not check for NaN (by testing the value or the flag)
after every operation/function, even when it is known that they can
generate a NaN. He will let NaN propagate (the flag can also be
checked later as it is sticky).

Note also that getting a NaN does not necessarily mean that the
program is buggy: after a sequence of computations, there may be
code to decide what to do when a NaN is obtained. So enabling
traps for FE_INVALID is not necessarily correct.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)