[PATCH] stdio-common: Add tests for formatted printf output specifiers

[Maciej W. Rozycki]

On Fri, 9 Aug 2024, Florian Weimer wrote:

> > Each test completes within single seconds except for the long double 
> > one.  There the F/f formats produce a large number of digits, which 
> > appears to be computationally intensive and CPU-bound.  Standalone 
> > execution time for `tst-printf-format-ldouble --direct f' is in the 
> > range of 00m36s for POWER9@2.166GHz and 09m52s for FU740@1.2GHz and 
> > output redirected locally to /dev/null, and 10m11s for FU740 and output 
> > redirected over 100Mbps network via SSH to /dev/null, so the throughput 
> > of the network adds very little (~3.2% in this case) to the processing 
> > time.  This is with IEEE 754 quad.
> >
> > Consequently have I scaled the timeout for tst-printf-format-ldouble 
> > accordingly.  Regardless, following recent practice the test has been 
> > added to the standard rather than extended set.
> 
> This seems to add approximately one minute to total testing time on a
> Core i9-9900K CPU.  That's a few generations old by now, but it's still
> not what I consider slow.  Is there a way to make the whole thing go
> faster?  I suspect it's due to serial execution within the ldouble test.

 I did some benchmarking to verify your hypothesis.  For this experiment I 
chose to use my 4-way SMP RISC-V target as it is suitably slow and still 
state-of-the-art both at a time, so it can't be considered obsolete yet.

 Without the new tests the whole testsuite takes ~2h57m and ~1h30m to 
complete in a serial run and at -j4 respectively.  Then the long double 
test itself takes ~29m to run and the testsuite with the new tests 
included takes respectively ~3h26m and ~1h57m, so it wasn't at all clear 
to me whether splitting the long double test would affect the execution 
time and if so, then how much.

 Even in the presence of a long-running test there are still 3 CPUs 
available to schedule other tests on.  In an ideal arrangement a -j4 run 
would take a quarter of time required while in reality the factor is only 
~0.5 (the inverse of ~2) without the new tests and ~0.57 (the inverse of 
~1.75) with the new tests included.

 The regression indicates an improvement should be possible and indeed 
splitting the test reduced the time elapsed to ~1h38m, bringing the factor 
back to ~0.5.

> Maybe it's possible to split this up at the Makefile level, so that the
> subtests can run in parallel?

 Done in v2 now posted; <https://inbox.sourceware.org/libc-alpha/32b978d4-c6e0-6eec-49cb-e581b3f15214@redhat.com/>.

> Can we avoid the REF_FMT approach if we embed the C source expression in
> the output (using macro stringization), instead of relying on the printf
> implementatiion to produce at least some correct outputs?  I think this
> test would still have passed with the “can't print 0” bug present
> because it failed consistently across all formats.

 I tried that, but found it infeasible and gave up, because several macros 
used with the data set expand to expressions that are not plain literals.  
Taking the first example UCHAR_MAX ultimately expands to "(0x7f * 2 + 1)", 
so stringification won't turn it into a string representation of the 
intended integer literal.

 An assumption has been made that plain unmodified conversions have been 
already been covered elsewhere, such as by stdio-common/tst-printf.c.  If 
there are cases missing, then perhaps we can identify them and add a test 
case to cover them separately.

 Alternatively we could try evaluating the expressions produced by the C 
preprocessor in GAWK, but then it would rely on getting the result correct 
there instead.  Especially in the floating-point case this could turn out 
problematic.

 Thank you for your review.

  Maciej