[PATCH] Makefile: add allow-list for failures

[Maxim Kuvyrkov]

Hi Florian,

Thanks for the pointer!

Hi Martin,

Where is allowed-failures.txt is supposed to be stored?  Is it going to be located in the glibc git repo or locally?  If we are going to store it in the glibc repo, then it needs to be per-target, and, sometimes, with additional variants -- e.g., many targets will have different expected FAILs depending on running on real hardware vs emulation, the kernel version and many other nuances.

Also, did you consider using validate_failures.py script from GCC [1]?  It is already compatible with glibc's .sum files, and provides good support for expected FAILs and flaky tests.

[1] https://gcc.gnu.org/git/?p=gcc.git;a=tree;f=contrib/testsuite-management

Thanks,

--
Maxim Kuvyrkov
https://www.linaro.org

> On Dec 16, 2025, at 02:53, Florian Weimer <fweimer@redhat.com> wrote:
> 
> * Martin Coufal:
> 
>> Enable adding known failures to allowed-failures.txt and ignore failures
>> in case they are in the list.  In case the allowed-failures.txt does not
>> exist, all failures lead to a failed status as before.
>> 
>> When the file is present, failures of listed tests are ignored and reported
>> on stdout. If tests not in the allowed list fail, summarize-tests exits with
>> status 1 and reports the failing tests.
>> 
>> The expected format of allowed-failures.txt file is:
>> <test_name> # <comment>
>> ---
>> Makefile | 29 ++++++++++++++++++++++++++++-
>> 1 file changed, 28 insertions(+), 1 deletion(-)
>> 
>> diff --git a/Makefile b/Makefile
>> index cb927fb9f9..99518ead64 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -594,7 +594,34 @@ define summarize-tests
>> @grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
>> @echo " === Summary of results$2 ==="
>> @sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
>> -@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
>> +@{ \
>> + grep -E '^[A-Z]+:' $(objpfx)$1 | \
>> + grep -E -v '^(X?PASS|XFAIL|UNSUPPORTED):' | \
>> + ( \
>> +   if ! test -f $(..)allowed-failures.txt; then \
>> +     read -r _; exit $$(( $$? == 0 )); \
>> +   fi; \
>> +   status=0; \
>> +   while IFS= read -r line; do \
>> +     case "$$line" in \
>> +       FAIL:*) \
>> +         name=$${line#FAIL: }; \
>> +         escaped_name=`printf '%s' "$$name" | sed 's/[.+]/\\&/g'`; \
>> +         if grep -Eq -- "^$${escaped_name}[[:space:]]*#" $(..)allowed-failures.txt; then \
>> +           echo "Ignoring allowed FAIL: $${name}"; \
>> +           continue; \
>> +         fi; \
>> +         echo "Unallowed FAIL: $${name}"; \
>> +         status=1; \
>> +         ;; \
>> +       *) \
>> +         status=1; \
>> +         ;; \
>> +     esac; \
>> +   done; \
>> +   exit $$status; \
>> + ); \
>> +      }
>> endef
>> 
>> # The intention here is to do ONE install of our build into the
> 
> Maxim, you've previously updated the code.  Would you be able to review
> Martin's patch, or at least comment on the general direction?
> 
> Thanks,
> Florian