undefined references since newlib-3.2.0

[Keith Packard]

Josef Wolf <jw@raven.inka.de> writes:

>>         atof
>>         atoff
>> [ ... ]
>>         strtod
>>         strtof
>>         strtold
>>         wcstod
>>         wcstold
>>         strtodg
>
> Uh! Why on earth would those functions need to allocate memory?

Because they are performing string to float conversions using code
written in 1991 by David Gay, based on research done by Will Clinger
which shows that exact conversion from arbitrary strings of decimal
digits to fixed precision binary requires arbitrary precision
arithmetic.

        https://dl.acm.org/doi/10.1145/93548.93557

>> These now return infinity and set errno to ERANGE on allocation
>> failure. (not ideal, but the options are limited)
>> 
>> Here are some which do return a pointer, but do not document any errors:
>> 
>>         ecvt
>>         fcvt
>
> Maybe the documentation can be fixed?

The documentation is based on a standard, and fixing that standard
involves a bit of process...

>>         gcvt
>>         ecvtbuf
>>         fcvtbuf
>>         gcvtbuf
>
> Those get a pointer passed. No need to allocate memory.

These functions are using code also written by David Gay to perform
float to string conversion, based on research done by Guy Steele and Jon
White in how to print floating point numbers accurately (which happened
to be presented at the same conference as the work above!). In this
work, they showed that exact conversion could be done using 1050 bit
arithmetic to generate a 64-bit double result:

        https://dl.acm.org/doi/10.1145/93548.93559

David Gay's code in newlib for both directions uses arbitrary precision
arithmetic code found in newlib/libc/stdlib/mprec.c. This code allocates
variable sized arrays of integers on the heap to hold all of the values.
Before the eBalloc patch, none of these allocations were checked,
leading to a rather long list of CVEs as the code could end up storing
through a NULL pointer, which can cause security problems on some
architectures.

>> And here's a list of functions which I feel reasonable applications
>> should not expect an allocation error from:
>
> I don't think any application should expect those functions to call exit()
> and/or abort() either.

I'm in complete agreement here. It's better to return an error that an
application *might* check than to not give it any chance to recover at
all.

>>         sprintf
>>         snprintf
>
> Those should return -1 on failure.
>
>>         sscanf
>
> For this, ENOMEM is documented.

Yes, but as I suggested, applications probably aren't expecting a call
to sscanf to return EOF and set errno to ENOMEM.

The real answer to your concerns is to replace the old arbitrary
precision based float/string conversion code with code that uses results
from new research by Ulf Adams.

That research improves on Steel & White by reducing the precision
required for exact 64-bit float to string conversion from 1050 bits to
128 bits. Adams also presents an algorithm using a similar technique to
perform (a slightly weaker form of) exact string to float conversion in
the same precision:

        https://dl.acm.org/citation.cfm?doid=3296979.3192369

This reasonably small fixed precision can be statically allocated in
memory, or allocated on the stack. Either of these solutions eliminates
the use of the dynamic heap through malloc, and eliminates the need to
change the specification of all of these functions to account for the
heap usage in the existing newlib code.

Ulf Adams also published code to implement this algorithm on github:

        https://github.com/ulfjack/ryu

I've ported this code to picolibc, a fork of newlib designed for
embedded systems. That library has an alternate stdio implementation
that doesn't need to use malloc, and it made sense to add this
malloc-free float/string conversion code to that (the previous
float/string conversion code in this implementation was not exact). When
compiled using that code, picolibc will not return errors from malloc
failures in the above cases because it does not call malloc in those
code paths.

The picolibc source repository also includes the stdio code from newlib
which can be used in place of the default picolibc stdio code by setting
a build option. That code has been modified to catch allocation
failures and return the failures above. I did that in case someone
wanted to use the original stdio code as I felt even this non-default
code should not expose applications to arbitrary calls to abort from
inside the library. I believe this code should be ported back to newlib
so that at least newlib wouldn't call abort. Even better would be to
have someone take a look at the Ryu paper and code and make that work in
newlib.

(The definition of 'exact' used in Ulf Adams work offers the guarantee
that you can print any floating point value, and then re-read that
string to exactly reproduce the original floating point value in
memory. This is weaker than what Clinger's research used; in that work,
the goal was to generate the floating point value closest to an
arbitrary string of decimal digits.)

-- 
-keith
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/newlib/attachments/20200614/241b8fac/attachment.sig>