[PATCH v2 6/6] Makerules: add 'make check-parallel' to run tests without serialization

[Sam James]

Adhemerval Zanella <adhemerval.zanella@linaro.org> writes:

> The default 'make check' serializes the timing-sensitive test runs: the
> threading (nptl, or htl on Hurd) and realtime (rt) subdirectories run
> with .NOTPARALLEL and are ordered after the rest of the test run, so they
> are not perturbed by competing machine load.
>
> Add a check-parallel (and xcheck-parallel) variant for when that is not
> wanted -- an idle machine, or a run where the extra throughput is worth
> the risk of flakiness in the timing-sensitive tests.
>

Perhaps link to
https://inbox.sourceware.org/libc-alpha/lhutsr3khz7.fsf@oldenburg.str.redhat.com/
for context in the commit message. I think ultimately we want to try do
something better for these, like adding some classifier for them as CPU
intensive or timing sensitive.

It also reminds me a bit of some of the xcheck discussion:
https://inbox.sourceware.org/libc-alpha/5816153.DvuYhMxLoT@pinacolada/.

> A new serialize-tests flag (default yes, defined in Makeconfig) gates both
> the per-subdirectory .NOTPARALLEL and the top-level run-time ordering;
> check-parallel just runs the suite with serialize-tests=no, so every test
> program builds and runs at full concurrency in a single pass.
>
> 'make check' and its default behavior are unchanged.

Can you make sure you document this on the wiki as well at
https://sourceware.org/glibc/wiki/Testing/Testsuite once in?

Reviewed-by: Sam James <sam@gentoo.org>

> ---
>  Makeconfig    |  8 ++++++++
>  Makefile      |  3 +++
>  Makerules     | 13 +++++++++++++
>  htl/Makefile  |  5 +++--
>  nptl/Makefile |  5 +++--
>  rt/Makefile   |  5 +++--
>  6 files changed, 33 insertions(+), 6 deletions(-)
>
> diff --git a/Makeconfig b/Makeconfig
> index 8fe7217dd7c..6cd590cf216 100644
> --- a/Makeconfig
> +++ b/Makeconfig
> @@ -782,6 +782,14 @@ run-built-tests = yes
>  endif
>  endif
>  
> +# Whether the timing-sensitive test runs are serialized: each is run with
> +# .NOTPARALLEL and, at the top level, ordered after the rest of the test run.
> +# This is the default; 'make check-parallel' clears it to run every test
> +# concurrently.
> +ifndef serialize-tests
> +serialize-tests = yes
> +endif
> +
>  # Whether to build the static math tests
>  ifndef build-math-static-tests
>  build-math-static-tests = no
> diff --git a/Makefile b/Makefile
> index 5f2293dfd7d..a38f11facf6 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -594,7 +594,9 @@ $(objpfx)testroot.pristine/install.stamp: | others
>  # 'make subdir/tests' is left alone.  And it only orders the test run
>  # (run-built-tests=yes); the "build the tests" pass (run-built-tests=no)
>  # is left fully parallel, so every test program still builds concurrently.
> +# serialize-tests=no ('make check-parallel') drops the ordering entirely.
>  ifeq ($(run-built-tests),yes)
> +ifeq (yes,$(serialize-tests))
>  ifneq (,$(filter tests xtests check xcheck,$(MAKECMDGOALS)))
>  +late-test-subdirs := $(filter nptl htl,$(subdirs)) $(filter rt,$(subdirs))
>  +test-run-prev := \
> @@ -604,6 +606,7 @@ $(foreach d,$(+late-test-subdirs),\
>    $(eval +test-run-prev += $(d)/tests))
>  endif
>  endif
> +endif
>  
>  ifeq (yes,$(build-shared))
>  # The top-level libc.so and linkobj/libc_pic.a rules list these
> diff --git a/Makerules b/Makerules
> index 318a916eef0..d7ab610b294 100644
> --- a/Makerules
> +++ b/Makerules
> @@ -1214,6 +1214,19 @@ check: tests
>  xcheck: xtests
>  endif
>  
> +# 'make check-parallel' runs the whole suite with maximum concurrency:
> +# serialize-tests=no drops the per-subdirectory .NOTPARALLEL and the run-time
> +# ordering, so every test builds and runs in parallel.  A single pass suffices
> +# (there is no .NOTPARALLEL to work around, so the test programs already build
> +# concurrently).
> +# This is faster on an idle machine, at the cost of possible flakiness in the
> +# timing-sensitive tests under the heavier load.
> +.PHONY: check-parallel xcheck-parallel
> +check-parallel:
> +	$(MAKE) serialize-tests=no tests
> +xcheck-parallel:
> +	$(MAKE) serialize-tests=no xtests
> +
>  # Also handle test inputs in sysdeps.
>  vpath %.input $(sysdirs)
>  
> diff --git a/htl/Makefile b/htl/Makefile
> index 0e5a8b8f689..1035b075649 100644
> --- a/htl/Makefile
> +++ b/htl/Makefile
> @@ -258,8 +258,9 @@ else
>  $(addprefix $(objpfx),$(tests) $(test-srcs)): $(srcdir)/libpthread_syms.a $(objpfx)libpthread.a
>  endif
>  
> -# The tests here better do not run in parallel.
> -ifeq ($(run-built-tests),yes)
> +# The tests here better do not run in parallel, unless serialize-tests is
> +# cleared (make check-parallel).
> +ifeq (yes-yes,$(run-built-tests)-$(serialize-tests))
>  ifneq ($(filter %tests,$(MAKECMDGOALS)),)
>  .NOTPARALLEL:
>  endif
> diff --git a/nptl/Makefile b/nptl/Makefile
> index 02862d1c04b..97a0fb7f658 100644
> --- a/nptl/Makefile
> +++ b/nptl/Makefile
> @@ -738,8 +738,9 @@ tst-audit-threads-ENV = LD_AUDIT=$(objpfx)tst-audit-threads-mod1.so
>  tst-setuid1-static-ENV = \
>    LD_LIBRARY_PATH=$(ld-library-path):$(common-objpfx)elf:$(common-objpfx)nss
>  
> -# The tests here better do not run in parallel.
> -ifeq ($(run-built-tests),yes)
> +# The tests here better do not run in parallel, unless serialize-tests is
> +# cleared (make check-parallel).
> +ifeq (yes-yes,$(run-built-tests)-$(serialize-tests))
>  ifneq ($(filter %tests,$(MAKECMDGOALS)),)
>  .NOTPARALLEL:
>  endif
> diff --git a/rt/Makefile b/rt/Makefile
> index 24e43a87599..f39e9674e58 100644
> --- a/rt/Makefile
> +++ b/rt/Makefile
> @@ -111,8 +111,9 @@ $(objpfx)librt.so: $(shared-thread-library)
>  tst-mqueue7-ARGS = -- $(host-test-program-cmd)
>  
>  # The timer and message-queue tests here are timing-sensitive and better
> -# do not run in parallel.
> -ifeq ($(run-built-tests),yes)
> +# do not run in parallel, unless serialize-tests is cleared (make
> +# check-parallel).
> +ifeq (yes-yes,$(run-built-tests)-$(serialize-tests))
>  ifneq ($(filter %tests,$(MAKECMDGOALS)),)
>  .NOTPARALLEL:
>  endif
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 418 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/libc-alpha/attachments/20260703/8ac55a45/attachment-0001.sig>