Bug in vfprintf?

[Charles Wilson]

It appears that there is a bug in _vfprintf_r in stdio/vfprintf.c (or 
cvt() in that same file, or _ldtoa_r in stdlib/ltdoa.c, depending on 
your point of view).

I stumbled on this bug while running the selftests for glib.  In glib's 
string-test, it calls vfprintf() with a fmt string:
   "%s|%0100d|%s|%s|%0*d|%*.*f|%10000.10000f"

The culprit here is the "%10000.10000f" format specifier.  vfprintf() 
calls cvt() for floating point, which calls _ldtoa_r().  However, 
_ldtoa_r overrides ndigits (e.g. 'prec') if it is greater than NDEC (42 
on cygwin).  However, cvt() and vfprintf() don't know that _ldtoa_r used 
"42" instead of "10000" -- and worse, they don't realize that _ldtoa_r 
allocated only slightly more than 42 bytes (52bytes on cygwin) for the 
digits string.

Thus, this code in fprintf.c cvt() generates a segfault:

1168        digits = _ldtoa_r(data, value, mode, ndigits, decpt, &dsgn, 
&rve);
1169    #endif /* !_NO_LONGDBL */
1170
1171        if ((ch != 'g' && ch != 'G') || flags & ALT)
1172            bp = digits + ndigits;

we set bp to something WAY past the end of digits.  Since rve points to 
the actual end of digits (where the '\0' is)...

1173            if (ch == 'f') {
1174                if (*digits == '0' && value)
1175                    *decpt = -ndigits + 1;
1176                bp += *decpt;
1177            }
1178            if (value == 0)
1179                rve = bp;
1180            while (rve < bp)
1181                *rve++ = '0';

this while loop (lines 1180-1181) attempts to put 0's into 
digits[43...10000].  But, there were only 24 +  (2^3-1)*4 == 52 bytes 
allocated for digits.

I'm not sure what the correct fix is...should ndigits/prec be passed by 
reference to _ldtoa_r, so that cvt "knows" that it got changed?  (Ditto 
pass-by-reference to cvt so that vfprintf_r knows about the override as 
well).  This changes the signature of these two (admittedly internal) 
routines, but I'm not sure of the "ripple effects" such a change might 
cause.

Or is this a case of "doctor, it hurts when I do this?" "Don't call 
printf with prec specifiers greater than 42, then."  That can't be good; 
memory corruption just because an internal newlib routine doesn't like 
the a given format spec?

This has been discussed somewhat in the following thread on the cygwin 
mailing list:
http://cygwin.com/ml/cygwin/2003-01/msg00023.html

--Chuck