[PATCH v2] Format test results closer to what DejaGnu does

[Adhemerval Zanella Netto]

On 01/11/23 07:28, Maxim Kuvyrkov wrote:
> The only change from v1 is removal of "=== glibc failures ==="
> line.
> 
> The years of dealing with Binutils, GCC and GDB test results
> made the community create good tools for comparison and analysis
> of DejaGnu test results.  This change allows to use those tools
> for Glibc's test results as well.
> 
> The motivation for this change is Linaro's pre-commit testers,
> which use a modified version of GCC's validate_failures.py
> to create test xfail lists with baseline failures and known
> flaky tests.  See below links for an example xfails file (only
> one link is supposed to work at any given time):
> - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/
> - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/
> 
> Specifacally, this patch changes format of glibc's .sum files from ...
> <cut>
> FAIL: elf/test1
> PASS: string/test2
> </cut>
> ... to ...
> <cut>
>              === glibc tests ===
> 
> Running elf ...
> FAIL: elf/test1
> 
> Running string ...
> PASS: string/test2
> </cut>.

I like the idea of splitting tests by subfolder.

> 
> And output of "make check" from ...
> <cut>
> FAIL: elf/test1
> </cut>
> ... to ...
> <cut>
> FAIL: elf/test1
> 		=== Summary of results ===
>       1 FAIL
>       1 PASS
> </cut>.
> 
> Signed-off-by: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>

LGTM, thanks.

Reviewed-by: Adhemerval Zanella  <adhemerval.zanella@linaro.org>

> ---
>  Makefile                      | 11 +++++++----
>  scripts/merge-test-results.sh |  3 +++
>  2 files changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index b938721166..a4f3378e21 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -577,11 +577,14 @@ $(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh
>  	$(SHELL) $< "$(PYTHON)" `pwd` > $@ ; \
>  	$(evaluate-test)
>  
> +# Print test summary for tests in $1 .sum file;
> +# $2 is optional test identifier.
> +# Fail if there are unexpected failures in the test results.
>  define summarize-tests
> -@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
> -@echo "Summary of test results$2:"
> -@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c
> -@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1
> +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
> +@echo "		=== Summary of results$2 ==="
> +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
> +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
>  endef
>  
>  # The intention here is to do ONE install of our build into the
> diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
> index e4dcc2520a..8f9d81f6eb 100755
> --- a/scripts/merge-test-results.sh
> +++ b/scripts/merge-test-results.sh
> @@ -50,7 +50,10 @@ case $type in
>    -t)
>      subdir_file_name=$1
>      shift
> +    echo "		=== glibc tests ==="
>      for d in "$@"; do
> +      echo
> +      echo "Running $d ..."
>        if [ -f "$objpfx$d/$subdir_file_name" ]; then
>  	cat "$objpfx$d/$subdir_file_name"
>        else