[PATCH 1/7] stdio-common: Verify printf format tests with Python rather than AWK
Adhemerval Zanella Netto
adhemerval.zanella@linaro.org
Wed Sep 2 16:20:48 GMT 2026
On 24/08/26 19:16, Matt Turner wrote:
> The formatted printf output tests verify their records against GNU AWK,
> relying on it to provide an implementation of format processing that is
> independent from ours.
>
> AWK has to run in the bignum mode for the floating-point conversions,
> because otherwise it uses the system sprintf(3) internally and we end up
> verifying our code against itself. That in turn makes gawk compiled
> with MPFR support a requirement for testing the library at all. Beyond
> that gawk mishandles a number of cases which the AWK script then has to
> undo by hand: the extraneous leading 0 produced for the alternative form
> with the octal conversion, the 0 produced where no characters are
> expected for the hexadecimal conversions, the missing + and space
> characters for a zero value with the precision of zero, and a collection
> of sign, flag and field width anomalies for Inf and NaN values. Each
> such workaround suppresses whatever we might get wrong in the same
> place. The a, A, b, and B conversions cannot be verified at all,
> because gawk either does not handle them or produces different output.
>
> Replace the AWK script with an equivalent one written in Python, which
> is already a requirement for building the library. Rather than calling
> into any formatting routine it computes the reference output directly,
> using exact integer and rational arithmetic. Working exactly means the
> result does not depend on the range or precision of any host
> floating-point type, so the wider types are handled without
> arbitrary-precision arithmetic having to be built into the interpreter,
> and none of the workarounds listed above are needed: the corner cases
> they cover are computed correctly. The same property removes the reason
> the a, A, b, and B conversions had to be left out; adding them is left
> for the commits that follow. Rendered digits are memoized per value,
> without which the exact arithmetic makes the long double conversions
> slower than AWK.
>
> As the capability probes only ever detected gawk build options, they go
> away along with the unsupported status they could produce, so the f and
> F conversions are now always verified rather than silently skipped where
> gawk was built without them. Drop the corresponding note on MPFR from
> the installation instructions.
>
> Tested on x86_64-linux-gnu, where all 576 results continue to pass.
LGTM, thanks.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> INSTALL | 4 -
> manual/install.texi | 4 -
> stdio-common/Makefile | 4 +-
> stdio-common/tst-printf-format-c.sh | 4 +-
> stdio-common/tst-printf-format-char.sh | 4 +-
> stdio-common/tst-printf-format-double.sh | 54 +--
> stdio-common/tst-printf-format-int.sh | 19 +-
> stdio-common/tst-printf-format-ldouble.sh | 54 +--
> stdio-common/tst-printf-format-llong.sh | 19 +-
> stdio-common/tst-printf-format-long.sh | 19 +-
> stdio-common/tst-printf-format-s.sh | 4 +-
> stdio-common/tst-printf-format-short.sh | 4 +-
> stdio-common/tst-printf-format-uchar.sh | 4 +-
> stdio-common/tst-printf-format-uint.sh | 19 +-
> stdio-common/tst-printf-format-ullong.sh | 19 +-
> stdio-common/tst-printf-format-ulong.sh | 19 +-
> stdio-common/tst-printf-format-ushort.sh | 4 +-
> stdio-common/tst-printf-format.awk | 130 -------
> stdio-common/tst-printf-format.py | 440 ++++++++++++++++++++++
> 19 files changed, 478 insertions(+), 350 deletions(-)
> delete mode 100644 stdio-common/tst-printf-format.awk
> create mode 100644 stdio-common/tst-printf-format.py
>
> diff --git ./INSTALL ./INSTALL
> index bafae84f75..0f3745d3b2 100644
> --- ./INSTALL
> +++ ./INSTALL
> @@ -522,10 +522,6 @@ build the GNU C Library:
> version 5.4.1 is the newest verified to work to build the GNU C
> Library.
>
> - Testing the GNU C Library requires 'gawk' to be compiled with
> - support for high precision arithmetic via the 'MPFR'
> - multiple-precision floating-point computation library.
> -
> * GNU 'bison' 2.7 or later
>
> 'bison' is used to generate the 'yacc' parser code in the 'intl'
Ok.
> diff --git ./manual/install.texi ./manual/install.texi
> index fe4a368385..e2f45b8e6f 100644
> --- ./manual/install.texi
> +++ ./manual/install.texi
> @@ -567,10 +567,6 @@ function, which was introduced in version 3.1.2 of @code{gawk}.
> As of release time, @code{gawk} version 5.4.1 is the newest verified
> to work to build @theglibc{}.
>
> -Testing the GNU C Library requires @code{gawk} to be compiled with
> -support for high precision arithmetic via the @code{MPFR}
> -multiple-precision floating-point computation library.
> -
> @item
> GNU @code{bison} 2.7 or later
>
Ok.
> diff --git ./stdio-common/Makefile ./stdio-common/Makefile
> index a948bfde8f..537962ad86 100644
> --- ./stdio-common/Makefile
> +++ ./stdio-common/Makefile
> @@ -633,7 +633,7 @@ $(objpfx)tst-printf.out: tst-printf.sh $(objpfx)tst-printf
>
> # We can't split a quoted recipe line, so do it via an auxiliary variable.
> make-tst-printf-format-out = \
> - AWK='$(AWK)' BASH='$(BASH)' \
> + PYTHON='$(PYTHON)' BASH='$(BASH)' \
> $(BASH) $< $@ $(common-objpfx) \
> '$(run-program-prefix-before-env) \
> $(run-program-env) \
> @@ -642,7 +642,7 @@ make-tst-printf-format-out = \
> $(run-program-prefix-after-env)'
> $(objpfx)tst-printf-format-%.out: \
> tst-printf-format.sh $(foreach c,$(convs),tst-printf-format-$(c).sh) \
> - $(foreach f,$(xprintf-srcs),$(objpfx)$(f)) tst-printf-format.awk
> + $(foreach f,$(xprintf-srcs),$(objpfx)$(f)) tst-printf-format.py
> $(make-tst-printf-format-out) > $@; \
> $(evaluate-test)
>
Ok.
> diff --git ./stdio-common/tst-printf-format-c.sh ./stdio-common/tst-printf-format-c.sh
> index 73d28c5607..70a8f59204 100644
> --- ./stdio-common/tst-printf-format-c.sh
> +++ ./stdio-common/tst-printf-format-c.sh
> @@ -23,12 +23,10 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> echo Verifying c
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-c c |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion c output error, first line:\n/") 2>&1 ||
> exit 1
Ok.
> diff --git ./stdio-common/tst-printf-format-char.sh ./stdio-common/tst-printf-format-char.sh
> index aa4d211126..ac0f273039 100644
> --- ./stdio-common/tst-printf-format-char.sh
> +++ ./stdio-common/tst-printf-format-char.sh
> @@ -23,8 +23,6 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> status=0
>
> for f in d i; do
> @@ -32,7 +30,7 @@ for f in d i; do
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-char $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format-double.sh ./stdio-common/tst-printf-format-double.sh
> index 051e4716c5..3201a419ab 100644
> --- ./stdio-common/tst-printf-format-double.sh
> +++ ./stdio-common/tst-printf-format-double.sh
> @@ -24,51 +24,15 @@ format=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -# For floating-point formats we need to use the bignum mode even if the
> -# regular mode would do, because GAWK in the latter mode uses sprintf(3)
> -# internally to process the conversion requested, so any bug in our code
> -# would then be verified against itself, defeating the objective of doing
> -# the verification against an independent implementation.
> -AWK="env LC_ALL=C ${AWK:-awk} -M"
> +status=0
>
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="-1.79769313486231570814527423731704357e+308"
> -val=$(echo "$ref" | $AWK '{ printf "%.35e\n", $1 }' 2>&1) &&
> - test "$val" = "$ref" && status=0
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> -
> -# Check for any additional conversions that AWK handles conditionally
> -# according to its version and/or the environment it has been built in.
> -# The 'A' and 'a' conversions are not suitable to use at this point, as
> -# output produced by AWK is different apparently due to a subtlety in
> -# rounding, so do not try them.
> -declare -A conversion_disabled
> -ref="-inf"
> -for f in f F; do
> - conversion_disabled[$f]=true
> - val=$(echo "$ref" | $AWK '{ printf "%'$f'\n", $1 }' 2>&1) &&
> - test "${val^^}" = "${ref^^}" && unset conversion_disabled[$f]
> -done
> -
> -if test "${conversion_disabled[$format]+set}" = set; then
> - echo Unsupported $format
> - status=77
> -else
> - echo Verifying $format
> - (set -o pipefail
> - ${test_program_prefix} \
> - ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-double $format |
> - $AWK -f tst-printf-format.awk 2>&1 |
> - head -n 1 |
> - sed "s/^/Conversion $format output error, first line:\n/") 2>&1 ||
> - status=1
> -fi
> +echo Verifying $format
> +(set -o pipefail
> + ${test_program_prefix} \
> + ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-double $format |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> + head -n 1 |
> + sed "s/^/Conversion $format output error, first line:\n/") 2>&1 ||
> + status=1
>
> exit $status
Ok.
> diff --git ./stdio-common/tst-printf-format-int.sh ./stdio-common/tst-printf-format-int.sh
> index e9dcbedc04..983cd9013f 100644
> --- ./stdio-common/tst-printf-format-int.sh
> +++ ./stdio-common/tst-printf-format-int.sh
> @@ -23,29 +23,14 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="-2147483648"
> -for AWK in "$AWK -M" "$AWK"; do
> - val=$(echo "$ref" | $AWK '{ printf "%d\n", $1 }' 2>&1) || continue
> - test "$val" = "$ref" && status=0 && break
> -done
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> +status=0
>
> for f in d i; do
> echo Verifying $f
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-int $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format-ldouble.sh ./stdio-common/tst-printf-format-ldouble.sh
> index 7ee097ac0a..2be2a592e0 100644
> --- ./stdio-common/tst-printf-format-ldouble.sh
> +++ ./stdio-common/tst-printf-format-ldouble.sh
> @@ -24,51 +24,15 @@ format=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -# For floating-point formats we need to use the bignum mode even if the
> -# regular mode would do, because GAWK in the latter mode uses sprintf(3)
> -# internally to process the conversion requested, so any bug in our code
> -# would then be verified against itself, defeating the objective of doing
> -# the verification against an independent implementation.
> -AWK="env LC_ALL=C ${AWK:-awk} -M"
> +status=0
>
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="-1.18973149535723176508575932662800702e+4932"
> -val=$(echo "$ref" | $AWK '{ PREC=113; printf "%.35e\n", $1 }' 2>&1) &&
> - test "$val" = "$ref" && status=0
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> -
> -# Check for any additional conversions that AWK handles conditionally
> -# according to its version and/or the environment it has been built in.
> -# The 'A' and 'a' conversions are not suitable to use at this point, as
> -# output produced by AWK is different apparently due to a subtlety in
> -# rounding, so do not try them.
> -declare -A conversion_disabled
> -ref="-inf"
> -for f in f F; do
> - conversion_disabled[$f]=true
> - val=$(echo "$ref" | $AWK '{ printf "%'$f'\n", $1 }' 2>&1) &&
> - test "${val^^}" = "${ref^^}" && unset conversion_disabled[$f]
> -done
> -
> -if test "${conversion_disabled[$format]+set}" = set; then
> - echo Unsupported $format
> - status=77
> -else
> - echo Verifying $format
> - (set -o pipefail
> - ${test_program_prefix} \
> - ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-ldouble $format |
> - $AWK -f tst-printf-format.awk 2>&1 |
> - head -n 1 |
> - sed "s/^/Conversion $format output error, first line:\n/") 2>&1 ||
> - status=1
> -fi
> +echo Verifying $format
> +(set -o pipefail
> + ${test_program_prefix} \
> + ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-ldouble $format |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> + head -n 1 |
> + sed "s/^/Conversion $format output error, first line:\n/") 2>&1 ||
> + status=1
>
> exit $status
Ok.
> diff --git ./stdio-common/tst-printf-format-llong.sh ./stdio-common/tst-printf-format-llong.sh
> index 98a79660cc..d90af6f881 100644
> --- ./stdio-common/tst-printf-format-llong.sh
> +++ ./stdio-common/tst-printf-format-llong.sh
> @@ -23,29 +23,14 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="9223372036854775807"
> -for AWK in "$AWK -M" "$AWK"; do
> - val=$(echo "$ref" | $AWK '{ printf "%d\n", $1 }' 2>&1) || continue
> - test "$val" = "$ref" && status=0 && break
> -done
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> +status=0
>
> for f in d i; do
> echo Verifying $f
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-llong $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
> diff --git ./stdio-common/tst-printf-format-long.sh ./stdio-common/tst-printf-format-long.sh
> index 89ac3302b8..c00f899aa4 100644
> --- ./stdio-common/tst-printf-format-long.sh
> +++ ./stdio-common/tst-printf-format-long.sh
> @@ -23,29 +23,14 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="9223372036854775807"
> -for AWK in "$AWK -M" "$AWK"; do
> - val=$(echo "$ref" | $AWK '{ printf "%d\n", $1 }' 2>&1) || continue
> - test "$val" = "$ref" && status=0 && break
> -done
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> +status=0
>
> for f in d i; do
> echo Verifying $f
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-long $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format-s.sh ./stdio-common/tst-printf-format-s.sh
> index 015c730609..3be2406c6e 100644
> --- ./stdio-common/tst-printf-format-s.sh
> +++ ./stdio-common/tst-printf-format-s.sh
> @@ -23,12 +23,10 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> echo Verifying s
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-s s |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion s output error, first line:\n/") 2>&1 ||
> exit 1
> diff --git ./stdio-common/tst-printf-format-short.sh ./stdio-common/tst-printf-format-short.sh
> index a7df8b8e6d..b474f65b96 100644
> --- ./stdio-common/tst-printf-format-short.sh
> +++ ./stdio-common/tst-printf-format-short.sh
> @@ -23,8 +23,6 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> status=0
>
> for f in d i; do
> @@ -32,7 +30,7 @@ for f in d i; do
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-short $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format-uchar.sh ./stdio-common/tst-printf-format-uchar.sh
> index 356de4217d..d0d3627e8d 100644
> --- ./stdio-common/tst-printf-format-uchar.sh
> +++ ./stdio-common/tst-printf-format-uchar.sh
> @@ -23,8 +23,6 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> status=0
>
> for f in o u x X; do
> @@ -32,7 +30,7 @@ for f in o u x X; do
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-uchar $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
> diff --git ./stdio-common/tst-printf-format-uint.sh ./stdio-common/tst-printf-format-uint.sh
> index b496047a49..2c32133212 100644
> --- ./stdio-common/tst-printf-format-uint.sh
> +++ ./stdio-common/tst-printf-format-uint.sh
> @@ -23,29 +23,14 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="4294967295"
> -for AWK in "$AWK -M" "$AWK"; do
> - val=$(echo "$ref" | $AWK '{ printf "%d\n", $1 }' 2>&1) || continue
> - test "$val" = "$ref" && status=0 && break
> -done
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> +status=0
>
> for f in o u x X; do
> echo Verifying $f
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-uint $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format-ullong.sh ./stdio-common/tst-printf-format-ullong.sh
> index f030f66a24..1136f61bf9 100644
> --- ./stdio-common/tst-printf-format-ullong.sh
> +++ ./stdio-common/tst-printf-format-ullong.sh
> @@ -23,29 +23,14 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="18446744073709551615"
> -for AWK in "$AWK -M" "$AWK"; do
> - val=$(echo "$ref" | $AWK '{ printf "%d\n", $1 }' 2>&1) || continue
> - test "$val" = "$ref" && status=0 && break
> -done
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> +status=0
>
> for f in o u x X; do
> echo Verifying $f
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-ullong $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format-ulong.sh ./stdio-common/tst-printf-format-ulong.sh
> index 7102575ed2..15802cec63 100644
> --- ./stdio-common/tst-printf-format-ulong.sh
> +++ ./stdio-common/tst-printf-format-ulong.sh
> @@ -23,29 +23,14 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> -status=77
> -
> -# Verify that AWK can handle the range required. It also catches:
> -# "gawk: warning: -M ignored: MPFR/GMP support not compiled in"
> -# message produced where bignum support is not there, which is the
> -# only indication as the use of '-M' does not affect the exit status
> -# in this case.
> -ref="18446744073709551615"
> -for AWK in "$AWK -M" "$AWK"; do
> - val=$(echo "$ref" | $AWK '{ printf "%d\n", $1 }' 2>&1) || continue
> - test "$val" = "$ref" && status=0 && break
> -done
> -
> -test $status -eq 0 || { echo "No working AWK found" && exit $status; }
> +status=0
>
> for f in o u x X; do
> echo Verifying $f
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-ulong $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format-ushort.sh ./stdio-common/tst-printf-format-ushort.sh
> index 5f612b5398..b08ae2906c 100644
> --- ./stdio-common/tst-printf-format-ushort.sh
> +++ ./stdio-common/tst-printf-format-ushort.sh
> @@ -23,8 +23,6 @@ xprintf=$1; shift
> common_objpfx=$1; shift
> test_program_prefix=$1; shift
>
> -AWK="env LC_ALL=C ${AWK:-awk}"
> -
> status=0
>
> for f in o u x X; do
> @@ -32,7 +30,7 @@ for f in o u x X; do
> (set -o pipefail
> ${test_program_prefix} \
> ${common_objpfx}stdio-common/tst-printf-format-${xprintf}-ushort $f |
> - $AWK -f tst-printf-format.awk 2>&1 |
> + ${PYTHON:-python3} tst-printf-format.py 2>&1 |
> head -n 1 | sed "s/^/Conversion $f output error, first line:\n/") 2>&1 ||
> status=1
> done
Ok.
> diff --git ./stdio-common/tst-printf-format.awk ./stdio-common/tst-printf-format.awk
> deleted file mode 100644
> index 57bea12621..0000000000
> --- ./stdio-common/tst-printf-format.awk
> +++ /dev/null
> @@ -1,130 +0,0 @@
> -# Testing of printf conversions.
> -# Copyright (C) 2024-2026 Free Software Foundation, Inc.
> -# This file is part of the GNU C Library.
> -
> -# The GNU C Library is free software; you can redistribute it and/or
> -# modify it under the terms of the GNU Lesser General Public
> -# License as published by the Free Software Foundation; either
> -# version 2.1 of the License, or (at your option) any later version.
> -
> -# The GNU C Library is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> -# Lesser General Public License for more details.
> -
> -# You should have received a copy of the GNU Lesser General Public
> -# License along with the GNU C Library; if not, see
> -# <https://www.gnu.org/licenses/>.
> -
> -BEGIN {
> - FS = ":"
> -}
> -
> -/^prec:/ {
> - PREC = $2
> - next
> -}
> -
> -/^val:/ {
> - val = $2
> - # Prepend "+" for +Inf or +NaN value lacking a sign, because gawk
> - # interpretes them as strings rather than numeric values in the
> - # non-bignum mode unless a sign has been explicitly given. Keep
> - # original 'val' for reporting.
> - value = gensub(/^(INF|NAN|inf|nan)/, "+\\1", 1, val)
> - # Neither changes between the conversions applied to this value.
> - value_infnan = value ~ /(INF|NAN|inf|nan)/
> - value_zero = value == 0
> - next
> -}
> -
> -/^%/ {
> - # Discard the trailing empty field, used to improve legibility of data.
> - input = $--NF
> - format = $1
> - width = $2
> - precision = "." $(NF - 1)
> - # Discard any negative precision, which is to be taken as if omitted.
> - sub(/\.-.*/, "", precision)
> - # Simplify handling and paste the precision and width specified as
> - # arguments to '*' directly into the format.
> - sub(/\.\*/, precision, format)
> - sub(/\*/, width, format)
> - # Discard length modifiers. They are only relevant to C data types.
> - sub(/([DHLjhltz]|wf?[1-9][0-9]*)/, "", format)
> - # Discard the '#' flag with the octal conversion if output starts with
> - # 0 in the absence of this flag. In that case no extra 0 is supposed
> - # to be produced, but gawk prepends it anyway.
> - if (index(format, "#") && format ~ /#.*o/)
> - {
> - tmpfmt = gensub(/#/, "", "g", format)
> - tmpout = sprintf(tmpfmt, value)
> - if (tmpout ~ /^ *0/)
> - format = tmpfmt
> - }
> - # Likewise with the hexadecimal conversion where zero value with the
> - # precision of zero is supposed to produce no characters, but gawk
> - # outputs 0 instead.
> - else if (index(format, "#") && format ~ /#.*[Xx]/)
> - {
> - tmpfmt = gensub(/#/, "", "g", format)
> - tmpout = sprintf(tmpfmt, value)
> - if (tmpout ~ /^ *$/)
> - format = tmpfmt
> - }
> - # AWK interpretes input opportunistically as a number, which interferes
> - # with how the 'c' conversion works: "a" input will result in "a" output
> - # however "0" input will result in "^@" output rather than "0". Force
> - # the value to be interpreted as a string then, by appending "".
> - output = sprintf(format, value "")
> - # Make up for various anomalies with the handling of +/-Inf and +/-NaN
> - # values and reprint the output produced using the string conversion,
> - # with the field width carried over and the relevant flags handled by
> - # hand.
> - if (value_infnan && format ~ /[EFGefg]/)
> - {
> - minus = format ~ /-/ ? "-" : ""
> - sign = value ~ /-/ ? "-" : format ~ /\+/ ? "+" : format ~ / / ? " " : ""
> - if (format ~ /^%[^\.1-9]*[1-9][0-9]*/)
> - width = gensub(/^%[^\.1-9]*([1-9][0-9]*).*$/, "\\1", 1, format)
> - else
> - width = ""
> - output = gensub(/[-+ ]/, "", "g", output)
> - output = sprintf("%" minus width "s", sign output)
> - }
> - # Produce "+" where the '+' flag has been used with a signed integer
> - # conversion for zero value, observing any field width in effect.
> - # In that case "+" is always supposed to be produced, but with the
> - # precision of zero gawk in the non-bignum mode produces any padding
> - # requested only.
> - else if (value_zero && format ~ /\+.*[di]/)
> - {
> - output = gensub(/^( *) $/, format ~ /-/ ? "+\\1" : "\\1+", 1, output)
> - output = gensub(/^$/, "+", 1, output)
> - }
> - # Produce " " where the space flag has been used with a signed integer
> - # conversion for zero value. In that case at least one " " is
> - # supposed to be produced, but with the precision of zero gawk in the
> - # non-bignum mode produces nothing.
> - else if (value_zero && format ~ / .*[di]/)
> - {
> - output = gensub(/^$/, " ", 1, output)
> - }
> - if (output != input)
> - {
> - printf "(\"%s\"%s%s, %s) => \"%s\", expected \"%s\"\n", \
> - $1, (NF > 2 ? ", " $2 : ""), (NF > 3 ? ", " $3 : ""), val, \
> - input, output > "/dev/stderr"
> - status = 1
> - }
> - next
> -}
> -
> -{
> - printf "unrecognized input: \"%s\"\n", $0 > "/dev/stderr"
> - status = 1
> -}
> -
> -END {
> - exit status
> -}
Ok.
> diff --git ./stdio-common/tst-printf-format.py ./stdio-common/tst-printf-format.py
> new file mode 100644
> index 0000000000..d10580ae8a
> --- /dev/null
> +++ ./stdio-common/tst-printf-format.py
> @@ -0,0 +1,440 @@
> +#!/usr/bin/python3
> +# Verification of formatted printf output.
> +# Copyright (C) 2024-2026 Free Software Foundation, Inc.
s/2024/2026.
> +# This file is part of the GNU C Library.
> +#
> +# The GNU C Library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +#
> +# The GNU C Library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# Lesser General Public License for more details.
> +#
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with the GNU C Library; if not, see
> +# <https://www.gnu.org/licenses/>.
> +
> +"""Verify records of formatted printf output.
> +
> +The record stream produced by tst-printf-format-skeleton.c is read from
> +standard input and each record's output is checked against a reference
> +value computed here. Diagnostics for records that do not match are written
> +to standard error and a nonzero exit status is returned.
> +
> +This is deliberately an independent implementation of the ISO C format
> +processing rules: every conversion is computed with exact integer and
> +rational arithmetic and nothing defers to the C library, so a bug in printf
> +cannot verify itself. Working exactly also means no dependence on the
> +range or precision of any host floating-point type, so the wider types are
> +handled without arbitrary-precision support having to be built into the
> +interpreter.
> +"""
> +
> +import io
> +import os
> +import sys
> +from fractions import Fraction
OK, afaik this lives in the core python stdlib so there should require no
extra support in distro setup.
> +
> +# Conversions grouped by the C type of the corresponding argument.
> +FLOAT_CONVS = frozenset("eEfFgG")
> +INT_CONVS = frozenset("diouxX")
> +
> +# The conversion specifier selects the base an integer is written in.
> +INT_FORMATS = {"o": "%o", "u": "%d", "x": "%x", "X": "%X"}
> +
> +# Flags and length modifiers accepted in a conversion specification.
> +FLAG_CHARS = "-+ #0"
> +LENGTH_CHARS = "hlLqjzt"
Ok although the AWK regex also stripped D, H and C23 wN/wfN. But there is also
no usage current for this.
> +
> +# log10(2), used only to seed the decimal exponent search.
> +LOG10_2 = 0.3010299956639812
> +
> +
> +def diagnose(text):
> + """Report TEXT on standard error.
> +
> + Records hold arbitrary bytes, taken as Latin-1 on the way in, so put
> + them back the same way rather than through whatever encoding the
> + locale would otherwise select."""
> + sys.stderr.buffer.write(text.encode("latin-1") + b"\n")
> + sys.stderr.buffer.flush()
> +
> +
> +def round_ratio(num, den):
> + """Round the non-negative ratio NUM/DEN to the nearest integer, with
> + ties going to even, matching the default rounding mode."""
> + quot, rem = divmod(num, den)
> + rem *= 2
> + if rem > den or (rem == den and quot & 1):
> + quot += 1
> + return quot
> +
> +
> +def scale_ratio(num, den, power):
> + """Return NUM/DEN multiplied by 10**POWER, as a pair of integers."""
> + if power >= 0:
> + return num * 10 ** power, den
> + return num, den * 10 ** -power
> +
> +
> +def decimal_to_binary(text, mant_bits):
> + """Convert the decimal string TEXT to the nearest value having MANT_BITS
> + of significand, returned as an exact Fraction.
> +
> + The generator prints reference values with enough digits to reproduce
> + the original, so this recovers the exact value converted."""
> + fr = Fraction(text)
> + if fr == 0:
> + return fr
> + neg = fr < 0
> + num, den = (-fr.numerator if neg else fr.numerator), fr.denominator
> + # Scale so that 2**(mant_bits-1) <= num/den < 2**mant_bits.
> + exp = num.bit_length() - den.bit_length() - mant_bits
> + if exp >= 0:
> + den <<= exp
> + else:
> + num <<= -exp
> + while num >= den << mant_bits:
> + den <<= 1
> + exp += 1
> + while num < den << (mant_bits - 1):
> + num <<= 1
> + exp -= 1
> + mant = round_ratio(num, den)
> + if mant >> mant_bits:
> + mant >>= 1
> + exp += 1
> + fr = Fraction(mant << exp) if exp >= 0 else Fraction(mant, 1 << -exp)
> + return -fr if neg else fr
> +
> +
> +def decimal_digits(fr, ndigits):
> + """Round the positive Fraction FR to NDIGITS significant decimal digits.
> +
> + Return the digit string and the decimal exponent X such that the value
> + is D[0].D[1:] * 10**X, i.e. X is the exponent an 'e' conversion prints."""
> + num, den = fr.numerator, fr.denominator
> + # Seed from the binary magnitude; the estimate is within one decade.
> + x = int((num.bit_length() - den.bit_length()) * LOG10_2)
> + while True:
> + hi_num, hi_den = scale_ratio(num, den, -(x + 1))
> + if hi_num < hi_den:
> + break
> + x += 1
> + while True:
> + lo_num, lo_den = scale_ratio(num, den, -x)
> + if lo_num >= lo_den:
> + break
> + x -= 1
> + digits = round_ratio(*scale_ratio(num, den, ndigits - 1 - x))
> + if digits >= 10 ** ndigits:
> + # Rounding carried into the next decade, e.g. 9.99 -> 10.0.
> + digits //= 10
> + x += 1
> + return str(digits).zfill(ndigits), x
> +
> +
> +class Spec:
> + """A parsed conversion specification."""
> +
> + __slots__ = ("minus", "plus", "space", "alt", "zero", "width", "prec",
> + "conv", "neg_zero")
> +
> + def __init__(self, fmt, args):
> + """Parse FMT, taking values for any '*' from ARGS in order."""
> + self.minus = self.plus = self.space = self.alt = self.zero = False
> + self.neg_zero = False
> + i = 1 # skip '%'
> + while i < len(fmt) and fmt[i] in FLAG_CHARS:
> + char = fmt[i]
> + self.minus |= char == "-"
> + self.plus |= char == "+"
> + self.space |= char == " "
> + self.alt |= char == "#"
> + self.zero |= char == "0"
> + i += 1
> +
> + self.width = 0
> + if i < len(fmt) and fmt[i] == "*":
> + self.width = int(args.pop(0))
> + i += 1
> + # A negative field width is a '-' flag with a positive width.
> + if self.width < 0:
> + self.minus = True
> + self.width = -self.width
> + else:
> + start = i
> + while i < len(fmt) and fmt[i].isdigit():
> + i += 1
> + if i > start:
> + self.width = int(fmt[start:i])
> +
> + self.prec = None
> + if i < len(fmt) and fmt[i] == ".":
> + i += 1
> + if i < len(fmt) and fmt[i] == "*":
> + self.prec = int(args.pop(0))
> + i += 1
> + # A negative precision is taken as if it were omitted.
> + if self.prec < 0:
> + self.prec = None
> + else:
> + start = i
> + while i < len(fmt) and fmt[i].isdigit():
> + i += 1
> + self.prec = int(fmt[start:i]) if i > start else 0
> +
> + # Length modifiers only select the C type of the argument, which the
> + # generator has already applied; they do not affect the output.
> + while i < len(fmt) and fmt[i] in LENGTH_CHARS:
> + i += 1
> + self.conv = fmt[i]
> +
> + def pad(self, body, zero_ok=True, sign=""):
> + """Apply the field width to SIGN followed by BODY."""
> + count = self.width - len(sign) - len(body)
> + if count <= 0:
> + return sign + body
> + if self.minus:
> + return sign + body + " " * count
> + if self.zero and zero_ok:
> + return sign + "0" * count + body
> + return " " * count + sign + body
> +
> + def sign_of(self, neg):
> + """The character introducing a signed conversion of a value whose
> + sign is NEG, honoring the '+' and space flags."""
> + if neg:
> + return "-"
> + if self.plus:
> + return "+"
> + if self.space:
> + return " "
> + return ""
> +
> +
> +def convert_int(value, spec):
> + conv = spec.conv
> + if conv in "di":
> + neg = value < 0
> + digits = str(-value if neg else value)
> + sign = spec.sign_of(neg)
> + else:
> + digits = INT_FORMATS[conv] % value
> + sign = ""
> + if spec.prec is not None:
> + # A zero value converted with a precision of zero produces no
> + # characters at all.
> + if value == 0 and spec.prec == 0:
> + digits = ""
> + digits = digits.rjust(spec.prec, "0")
> + if spec.alt:
> + if conv == "o" and not digits.startswith("0"):
> + digits = "0" + digits
> + elif conv in "xX" and value != 0:
> + # The base prefix precedes any '0' flag padding, so it pads
> + # along with the sign rather than with the digits.
> + sign = "0x" if conv == "x" else "0X"
> + # An explicit precision defeats the '0' flag.
> + return spec.pad(digits, zero_ok=spec.prec is None, sign=sign)
> +
> +
> +def convert_string(value, spec):
> + if spec.prec is not None:
> + value = value[:spec.prec]
> + return spec.pad(value, zero_ok=False)
> +
> +
> +def convert_char(value, spec):
> + return spec.pad(value[:1], zero_ok=False)
> +
> +
> +def convert_special(kind, neg, spec):
> + """Convert an infinity or a NaN. The field is padded with spaces
> + whether or not the '0' flag was given, and the alternative form has no
> + effect."""
> + body = kind.upper() if spec.conv in "EFG" else kind
> + return spec.pad(body, zero_ok=False, sign=spec.sign_of(neg))
> +
> +
> +def render_f(fr, prec, alt, strip):
> + """Render the non-negative Fraction FR in the style of 'f'."""
> + scaled = round_ratio(*scale_ratio(fr.numerator, fr.denominator, prec))
> + text = str(scaled).zfill(prec + 1)
> + if prec:
> + whole, frac = text[:-prec], text[-prec:]
> + else:
> + whole, frac = text, ""
> + if strip:
> + frac = frac.rstrip("0")
> + if frac or alt:
> + return whole + "." + frac
> + return whole
> +
> +
> +def render_e(fr, prec, alt, upper, strip):
> + """Render the non-negative Fraction FR in the style of 'e'."""
> + if fr == 0:
> + digits, exp = "0" * (prec + 1), 0
> + else:
> + digits, exp = decimal_digits(fr, prec + 1)
> + whole, frac = digits[0], digits[1:]
> + if strip:
> + frac = frac.rstrip("0")
> + mant = whole + "." + frac if frac or alt else whole
> + return "%s%s%s%02d" % (mant, "E" if upper else "e",
> + "-" if exp < 0 else "+", abs(exp))
> +
> +
> +def convert_float(value, spec, cache):
> + """Convert VALUE, an exact Fraction. CACHE memoizes rendered digits for
> + the value currently being converted."""
> + conv = spec.conv
> + neg = value < 0 or (value == 0 and spec.neg_zero)
> + fr = -value if value < 0 else value
> + prec = 6 if spec.prec is None else spec.prec
> + upper = conv in "EFG"
> +
> + # The generator iterates a large number of flag and field width
> + # combinations over each value, so the same digits are called for over
> + # and over. They depend only on the magnitude, the precision and the
> + # alternative form, which is what keeps the wider types inexpensive
> + # despite the arithmetic being exact.
> + key = (conv, prec, spec.alt)
> + body = cache.get(key)
> + if body is None:
> + if conv in "gG":
> + sig = 1 if prec == 0 else prec
> + _, exp = decimal_digits(fr, sig) if fr != 0 else ("", 0)
> + if -4 <= exp < sig:
> + body = render_f(fr, sig - 1 - exp, spec.alt,
> + strip=not spec.alt)
> + else:
> + body = render_e(fr, sig - 1, spec.alt, upper,
> + strip=not spec.alt)
> + elif conv in "eE":
> + body = render_e(fr, prec, spec.alt, upper, strip=False)
> + else:
> + body = render_f(fr, prec, spec.alt, strip=False)
> + cache[key] = body
> +
> + return spec.pad(body, zero_ok=True, sign=spec.sign_of(neg))
> +
> +
> +class Block:
> + """The state a run of records shares: the C type they exercise and the
> + single value they all convert."""
> +
> + __slots__ = ("mant_bits", "val_text", "value", "special",
> + "neg_zero", "cache")
> +
> + def __init__(self):
> + self.mant_bits = 0
> + self.set_value("")
> +
> + def set_value(self, text):
> + """Begin a run of records converting the value written as TEXT."""
> + self.val_text = text
> + self.value = None
> + self.special = None
> + self.neg_zero = False
> + self.cache = {} # digits memoized by convert_float
> +
> + def interpret(self, conv):
> + """Interpret the value text as the C type conversion CONV takes."""
> + if conv in FLOAT_CONVS:
> + lowered = self.val_text.lower()
> + if "inf" in lowered or "nan" in lowered:
> + self.special = "nan" if "nan" in lowered else "inf"
> + else:
> + self.value = decimal_to_binary(self.val_text, self.mant_bits)
> + self.neg_zero = self.val_text.lstrip().startswith("-")
> + elif conv in INT_CONVS:
> + self.value = int(self.val_text)
> + else:
> + self.value = self.val_text
> +
> + def expect(self, spec):
> + """The output SPEC is required to produce for this block's value."""
> + conv = spec.conv
> + # Which C type the value has follows from the conversion specifier,
> + # which is not known until the first record of the block, so the
> + # text is interpreted here rather than where it was read.
> + if self.value is None and self.special is None:
> + self.interpret(conv)
> + spec.neg_zero = self.neg_zero
> + if conv in FLOAT_CONVS:
> + if self.special is not None:
> + return convert_special(self.special, self.neg_zero, spec)
> + return convert_float(self.value, spec, self.cache)
> + if conv in INT_CONVS:
> + return convert_int(self.value, spec)
> + if conv == "c":
> + return convert_char(self.value, spec)
> + return convert_string(self.value, spec)
> +
> +
> +def check(raw, block):
> + """Check one record against the value BLOCK holds, reporting a mismatch
> + on standard error. Return whether the record was in order."""
> + # Records are colon separated and end with an empty field, included to
> + # make the data easier to read. What precedes the output are the format
> + # and then any width and precision supplied as arguments.
> + fields = raw.split(":")
> + fields.pop()
> + fmt, actual, args = fields[0], fields[-1], fields[1:-1]
> +
> + expect = block.expect(Spec(fmt, list(args)))
> + if expect == actual:
> + return True
> + diagnose('("%s"%s, %s) => "%s", expected "%s"'
> + % (fmt, "".join(", " + arg for arg in args), block.val_text,
> + actual, expect))
> + return False
> +
> +
> +def main():
> + # The wider types at the precisions exercised produce integers well past
> + # the default limit on conversion to a string. Versions that predate
> + # the limit have no such call and need no lifting.
> + if hasattr(sys, "set_int_max_str_digits"):
> + sys.set_int_max_str_digits(0)
> +
> + # Records hold arbitrary bytes, so read them as Latin-1 rather than
> + # through whatever encoding the locale would otherwise select.
> + stdin = io.TextIOWrapper(sys.stdin.buffer, encoding="latin-1",
> + newline="\n")
> +
> + block = Block()
> + status = 0
> +
> + for raw in stdin:
> + raw = raw.rstrip("\n")
> +
> + if raw.startswith("prec:"):
> + block.mant_bits = int(raw[5:])
> + elif raw.startswith("val:"):
> + block.set_value(raw[4:])
> + elif raw.startswith("%"):
> + if not check(raw, block):
> + status = 1
> + else:
> + diagnose('unrecognized input: "%s"' % raw)
> + status = 1
> +
> + return status
> +
> +
> +if __name__ == "__main__":
> + try:
> + sys.exit(main())
> + except BrokenPipeError:
> + # The wrappers pipe into 'head', which stops reading once it has
> + # the first diagnostic. Point what is left of standard error at
> + # the null device, so that flushing at exit does not run into the
> + # same failure and report it a second time.
> + os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stderr.fileno())
> + sys.exit(1)
Ok.
More information about the Libc-alpha
mailing list