awk and stdio-common/tst-printf-format-as-double* trouble (mpfr support needed)

[Sam James]

Sam James <sam@gentoo.org> writes:

> "Maciej W. Rozycki" <macro@redhat.com> writes:
>
>> On Wed, 18 Dec 2024, Adhemerval Zanella Netto wrote:
>>
>>> > A quick look at stdio-common/tst-printf-format-double.sh made me suspect
>>> > the arbitrary precision (mpfr) support. Indeed, recompiling Gnu awk with
>>> > that and re-testing, the testsuite passes just fine (zero failures).
>>> > 
>>> > 1) Seems like the FAILs are a bug in the testsuite, since in this case each
>>> > second test should be unsupported too?
>>> > 
>>> > 2) Gnu awk upstream states that the arbitrary precision support is "on parole", 
>>> > and may go away in a future version. It's probably not a good idea to rely 
>>> > on it then?
>>> > https://www.gnu.org/software/gawk/manual/html_node/MPFR-On-Parole.html
>>> 
>>> Afaik awk was used as reference printf implementation that does not call
>>> or rely on libc.  I tend to prefer to use pre-generated results, in the
>>> same way we check the math results with the math/gen-auto-libm-tests.c,
>>> but we will ended with a similar issue that how to autogenerate the results
>>> (for the math case we use mpfr/gmp/mpc as reference).
>>> 
>>> Maciej, would be feasible to use pre-generated results instead of checking
>>> against on awk on every make check (this can improve the testing time 
>>> as well)? 
>>
>>  I thought I had included the rationale in the change description, but it 
>> turns out it was only discussed internally.  Sorry about it.
>
> That does remind me that I don't think there's really been much
> discussion of the general test push (apologies if I've missed it) --
> it's very welcome, just clear that at least one team at RH is focusing
> on it all of a sudden ;)
>
>>
>>  The short answer is it is *not* feasible to use pre-generated results, 
>> because I figured out they would require at least a gigabyte of storage, 
>> i.e. possibly an order of magnitude more than our current whole source 
>> tree takes now.
>>
>>  You can determine it quite easily by running the individual test programs 
>> and storing data produced to stdout somewhere.  The structure of this data 
>> is documented in stdio-common/tst-printf-format-skeleton.c and at the very 
>> least the <VAL> field would have to be permanently recorded.  A quick test 
>> shows e.g.:
>>
>> $ .../stdio-common/tst-printf-format-p-ldouble --direct e | sed -n '/^%/s/^.*:\([^:]\+\):$/\1/p' | wc -c
>> 143501792
>> $ .../stdio-common/tst-printf-format-p-ldouble --direct f | sed -n '/^%/s/^.*:\([^:]\+\):$/\1/p' | wc -c
>> 227613378
>> $ .../stdio-common/tst-printf-format-p-ldouble --direct g | sed -n '/^%/s/^.*:\([^:]\+\):$/\1/p' | wc -c
>> 137682196
>>
>> -- with the 128-bit IBM format, and then data for other formats would have 
>> to be included as well (even if we cheated and handled `%e' vs `%E', etc. 
>> via postprocessing rather than having individual data records for each 
>> case variant of the output conversions).
>>
>>> I am not sure if there is really a requirement for a reference implementation,
>>> specially one that might generate wrong results in the future.  Maybe we can
>>> use mpfr printf to get the expected values for format strings.
>>
>>  Is MPFR printf widely available?  It's the first time I hear of such a 
>> tool and I've used Linux for almost 30 years now.  Conversely there's GNU 
>> AWK installed across most glibc-based systems.
>>
>>  I don't think the old note as to MPFR in GNU AWK documentation should be 
>> an issue for us: GNU AWK has a maintainer now, Arnold D. Robbins
>>  (CC'd),
>
> I believe you're mistaken on this point: while GNU AWK certainly has a
> wonderful maintainer who has always been excellent in our interactions,
> Arnold is the one who *added* such a deprecation notice _just for the
> MPFR functionality_ 2 years ago:
> https://git.savannah.gnu.org/cgit/gawk.git/commit/?id=d2b7dbb165707778ef960d59c4b2c9798fd4723b.

I should say it indeed got promoted to "on parole" later but that is
already reflected in Andreas' link:
https://git.savannah.gnu.org/cgit/gawk.git/commit/?id=03f79dcb6aff98574c32609c80ecdd6e52730244.

>
> I think there was some ML discussion about it too but I can't find it
> right now.
>
>> who I worked with in the course of making this patchset as I discovered 
>> numerous bugs in GNU AWK, including in the MPFR part, as documented in the 
>> change description for commit 7ec4d7e3d1c0 ("stdio-common: Add tests for 
>> formatted printf output specifiers"), and I found Arnold very positive 
>> towards it, so I think it would be unfair to his effort to refrain from 
>> using his work.
>>
>
> See above. 
>
>>  NB I've looked into getting rid of the spurious mtrace failures in the 
>> case where the corresponding test hasn't been run and a fix might be quite 
>> easy, but we currently have a mess across Makefiles in the handling of 
>> these mtrace tests and I'd like to fix that first rather than making 
>> things even messier.  Since this only triggers for installations with a 
>> crippled compilation of GNU AWK I think it's not a high priority item and 
>
> Please note that we're following the upstream recommendation not to rely
> on it given it's deprecated and has no maintainer.
>
> If this were just some users who happened to flip a random configure
> option toggle, I'd agree.
>
> If the state of affairs has changed since:
> a) that notice was added;
> b) the ML discussions I recall,
> then of course, it's far less of an issue.
>
>> therefore I chose to defer it until early next year once I'm done with the 
>> current stuff on my plate now.  I do hope it's acceptable.
>>
>
> I think early next year is certainly acceptable either way, of course.
>
>>   Maciej
>
> thanks,
> sam