UB status of snprintf on invalid ptr+size combination?

[Vincent Lefevre]

On 2023-03-20 17:15:49 +0100, Alejandro Colomar wrote:
> On 3/20/23 16:09, Vincent Lefevre wrote:
> > Here's another example where the support of snprintf with a large n
> > argument (larger than the buffer size) may be used:
> > 
> > In GNU MPFR, for our function mpfr_snprintf, we have assumed a
> > behavior analogue to the ISO C behavior. In particular, we use
> > that in our tests in order to check that large values of n are
> > correctly handled. This allowed us to trigger/detect a bug in
> > the choice of the integer types in our implementation:
> > 
> > The test:
> > 
> > https://gitlab.inria.fr/mpfr/mpfr/-/commit/67a75bfe41d3a7f95367ee9e62bd7dfc73e5b395
> > 
> > The bug fix (replacing an int by a size_t in a variable declaration):
> > 
> > https://gitlab.inria.fr/mpfr/mpfr/-/commit/6b8cf3e2bdc285027627281cac230ed932c1b73f
> 
> I don't understand how snprintf(3) helped catch the bug.

It doesn't. *Currently*, MPFR does not use snprintf. With the buggy
version, on a typical 64-bit machine (where int = 32 bits), the size
given to mpfr_snprintf became the value modulo 2^32, so if n is 2^32
(possible as size_t has 64 bits), the implementation of mpfr_snprintf
assumed that the size was 0 (instead of 2^32). Hence the incorrect
behavior.

The test helped to catch this bug because it checks mpfr_snprintf
on this value n = (size_t) UINT_MAX + 1, which is 2^32 here. But in
order not to use much memory, the test is done with a small buffer.
Using a buffer of size n would need 4 GB, and this amount of memory
is not available everywhere.

As MPFR does not currently use snprintf, there would not be any issue
with any snprintf behavior. But in the future, we should use snprintf
(actually gmp_snprintf or similar, but this may call snprintf at the
end) to implement mpfr_snprintf. In such a case, the above test will
no longer work if snprintf aborts just because n is larger than the
buffer size.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)