[PATCH 2/5] Makefile: Run the subdirectory recursion in parallel
Adhemerval Zanella
adhemerval.zanella@linaro.org
Fri Jun 12 16:34:05 GMT 2026
The top-level makefile was marked .NOTPARALLEL and ran the
per-subdirectory sub-makes strictly one at a time in the topological
order computed by scripts/gen-sorted.awk. Only the compilations inside
a single subdirectory could run in parallel, so on wide machines every
subdirectory's compile tail and link steps left most cores idle, once
per subdirectory per pass.
Drop .NOTPARALLEL and encode the ordering the serial recursion relied
on as explicit dependencies between the per-subdirectory targets:
* The subdirectories that generate shared files in $(common-objpfx)
consumed by the rest of the build without explicit dependencies run
serially, in their sorted order, before the rest fan out: csu
provides the tree-wide gen-as-const headers, and on Hurd the mach
and hurd directories generate the MiG RPC headers (every other
subdirectory otherwise runs a nested make in hurd/ to create them,
racing under parallel recursion; see sysdeps/mach/hurd/Makefile).
The first of them also materializes the other shared generated files
(abi-versions.h, sysd-syscalls, before-compile headers).
* The edges requested by the Depend files (now emitted by
gen-sorted.awk as subdir-deps-*) are preserved. Edges pointing to
elf are dropped, as the sorted list already overrides them by
forcing elf last.
* elf stays last: its rtld link consumes $(common-objpfx)libc_pic.a,
which aggregates every other subdirectory's objects, and its
rtld-Rules recursion compiles into the other subdirectories' object
directories.
* Pass barriers replace the implicit pass ordering: others after lib
(a subdirectory others sub-make would otherwise race to link
libc.so itself), tests/xtests after others, and the testroot
install behind others.
* The subdirectory-built files that the top-level libc.so and
linkobj/libc_pic.a rules list as prerequisites (elf/ld.so,
interp.os, sofini.os, sunrpc/librpc_compat_pic.a, and on Hurd
mach/libmachuser_pic.a and hurd/libhurduser_pic.a, from which the
lib*user-link.so inputs of libc.so are built) get order-only edges
on the corresponding sub-make with an explicit empty recipe. A
prerequisite-only rule would trigger an implicit rule search and
this level would compile them itself in the wrong context.
* The install, clean, abi, and stubs target classes keep the
previous total order via a serial dependency chain.
* The elf DSO sorting test recipes, run when make remakes the
included generated makefiles at parse time, create the elf object
directory before writing into it; the serial recursion no longer
guarantees another rule created it first.
Results on a x86_64 machine [1] with default configuration [3]: a
from-scratch build improves from 78.728s to 61s, and check with
run-built-tests=no from 374s to 190s.
On a 80-core aarch64 machine [2] with default configuration [3]: a
from-scratch build improves from 105.251s to 56.703s, and check with
run-built-tests=no from 886.183s to 298.726s.
Build results are unchanged: all 8919 built objects, archives, and
shared objects are bit-identical to the serial build across 7 clean
parallel builds, the installed tree layout is identical, and the
tests.sum failure sets are identical. i686-gnu was verified with
repeated from-scratch builds.
[1] Ryzen 5900x, 12c/24t, gcc 16.1.1, binutils 2.26, and GNU make 4.3
[2] N1, 80c, gcc 15.1.1, binutils 2.25, GNU make 4.3
[3] --enable-stack-protector=all --enable-bind-now=yes --enable-profile=yes
--enable-fortify-source=2 --enable-hardcoded-path-in-tests
---
Makefile | 109 +++++++++++++++++++++++++++++++++++++++++++++++++--
elf/Makefile | 5 +++
2 files changed, 111 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile
index 6b0e0555189..4e8c29dffb3 100644
--- a/Makefile
+++ b/Makefile
@@ -54,9 +54,6 @@ configure: configure.ac aclocal.m4; $(autoconf-it)
endif # $(AUTOCONF) = no
-# We don't want to run anything here in parallel.
-.NOTPARALLEL:
-
# These are the targets that are made by making them in each subdirectory.
+subdir_targets := subdir_lib objects objs others subdir_mostlyclean \
subdir_clean subdir_distclean subdir_realclean \
@@ -129,6 +126,13 @@ lib-noranlib: subdir_lib
ifeq (yes,$(build-shared))
# Build the shared object from the PIC object library.
lib: $(common-objpfx)libc.so $(common-objpfx)linkobj/libc.so
+ifdef libc.so-version
+# Every program linked in the others pass lists the versioned name
+# (through link-libc-between-gnulib) as a prerequisite, and the rule
+# creating the symbolic link is visible in every sub-make. Build it
+# here once so the concurrent sub-makes do not race to create it.
+lib: $(common-objpfx)libc.so$(libc.so-version)
+endif
endif # $(build-shared)
# Used to build testrun.sh.
@@ -490,6 +494,105 @@ subdir=$(@D)$(if $($(@D)-srcdir),\
endef
.PHONY: $(+subdir_targets) $(all-subdirs-targets)
+
+# Encode the topological ordering computed by scripts/gen-sorted.awk as
+# explicit dependencies between the per-subdirectory targets, so that
+# independent subdirectories build concurrently:
+#
+# * Every subdirectory depends on the first sorted one (csu, or mach on
+# Hurd): its sub-make also materializes the shared generated files in
+# $(common-objpfx) (abi-versions.h, sysd-syscalls, before-compile
+# headers, ...) that concurrent sub-makes would otherwise race to
+# create.
+#
+# * The edges requested by the Depend files (emitted by gen-sorted.awk
+# as subdir-deps-*) are preserved.
+#
+# * elf stays last, as in the sorted list. Its rtld build recurses into
+# the other subdirectories' object directories via elf/rtld-Rules.
+#
+# * Only target classes without cross-directory file conflicts use this
+# sparse ordering; everything else (install, clean, abi, stubs) keeps
+# the previous total order via a serial chain.
+
++parallel_subdir_targets := \
+ subdir_lib \
+ objects \
+ objs \
+ others \
+ tests \
+ xtests \
+ subdir_objs \
+ # +parallel_subdir_targets
++serial_subdir_targets := $(filter-out $(+parallel_subdir_targets),\
+ $(+subdir_targets))
+
+# The subdirectories that generate shared files in $(common-objpfx)
+# consumed by the rest of the build without explicit dependencies: csu
+# provides the gen-as-const headers, and on Hurd the mach and hurd
+# directories generate the MiG RPC headers (every other subdirectory
+# otherwise runs a nested make in hurd/ to create them, racing under
+# parallel recursion; see sysdeps/mach/hurd/Makefile). Run them serially,
+# in their sorted order (mach, hurd, csu).
++subdir-pregen := $(filter mach hurd csu,$(subdirs))
++subdir-rest := $(filter-out $(+subdir-pregen),$(subdirs))
+
+$(foreach t,$(+parallel_subdir_targets),$(eval \
+ $(addsuffix /$(t),$(+subdir-rest)): $(addsuffix /$(t),$(+subdir-pregen))))
++subdir-pregen-prev :=
+$(foreach d,$(+subdir-pregen),$(foreach t,$(+parallel_subdir_targets),$(eval \
+ $(d)/$(t): $(addsuffix /$(t),$(+subdir-pregen-prev))))\
+ $(eval +subdir-pregen-prev := $(d)))
+# Edges pointing to elf are dropped; the sorted list always forces elf
+# last, overriding any Depend request, and the elf-last edges below would
+# otherwise create a cycle.
+$(foreach t,$(+parallel_subdir_targets),$(foreach d,$(+subdir-rest),$(eval \
+ $(d)/$(t): $(addsuffix /$(t),\
+ $(filter-out elf,$(filter $(subdirs),$(subdir-deps-$(d))))))))
+ifneq (,$(filter elf,$(subdirs)))
+$(foreach t,$(+parallel_subdir_targets),$(eval \
+ elf/$(t): $(addsuffix /$(t),$(filter-out elf,$(subdirs)))))
+endif
+
+# Pass barriers: a subdirectory 'others' build links programs against
+# the libraries, so the 'lib' pass (including the top-level libc.so
+# link) must have completed.
+# 'tests' and 'xtests' additionally require the 'others' pass. The
+# testroot used by the container tests performs a full installation in
+# its recipe, which must not run concurrently with the build passes.
+$(addsuffix /others,$(subdirs)): lib
+$(addsuffix /tests,$(subdirs)) $(addsuffix /xtests,$(subdirs)): others
+$(objpfx)testroot.pristine/install.stamp: | others
+
+ifeq (yes,$(build-shared))
+# The top-level libc.so and linkobj/libc_pic.a rules list these
+# subdirectory-built files as prerequisites, but no rule at this level
+# builds them. The explicit empty recipe (';') is required, a
+# prerequisite-only rule would send make on an implicitrule search and
+# have this level compile them itself with the wrong context.
+$(elf-objpfx)ld.so $(elf-objpfx)sofini.os $(elf-objpfx)interp.os: \
+ | elf/subdir_lib ;
+ifneq (,$(filter sunrpc,$(subdirs)))
+# Makerules explicit adds librpc_compat_pic.a as a dependency of
+# libc_pic.a.
+$(common-objpfx)sunrpc/librpc_compat_pic.a: | sunrpc/subdir_lib ;
+endif
+# Hurd sysdedp Makeilfe links libc.so against the lib*user-link.so
+# objects, built by the %-link.so: %_pic.a pattern rule from archives
+# that only the mach and hurd sub-makes create.
+ifneq (,$(filter mach,$(subdirs)))
+$(common-objpfx)mach/libmachuser_pic.a: | mach/subdir_lib ;
+endif
+ifneq (,$(filter hurd,$(subdirs)))
+$(common-objpfx)hurd/libhurduser_pic.a: | hurd/subdir_lib ;
+endif
+endif
+
+# The remaining target classes keep the old total order.
++subdir-chain-prev :=
+$(foreach d,$(subdirs),$(foreach t,$(+serial_subdir_targets),$(eval \
+ $(d)/$(t): $(addsuffix /$(t),$(+subdir-chain-prev))))\
+ $(eval +subdir-chain-prev := $(d)))
# Targets to clean things up to various degrees.
diff --git a/elf/Makefile b/elf/Makefile
index 5ede78c5902..5598bafba76 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -1436,6 +1436,7 @@ ifndef avoid-generated
# Makefile fragment to be included.
define include_dsosort_tests
$(objpfx)$(1).generated-makefile: $(1)
+ $$(make-target-directory)
$(PYTHON) $(..)scripts/dso-ordering-test.py \
--description-file $$< --objpfx $(objpfx) --output-makefile $$@T
mv $$@T $$@
@@ -1444,6 +1445,7 @@ endef
# Likewise, where the .def file itself is generated.
define include_dsosort_tests_objpfx
$(objpfx)$(1).generated-makefile: $(objpfx)$(1)
+ $$(make-target-directory)
$(PYTHON) $(..)scripts/dso-ordering-test.py \
--description-file $$< --objpfx $(objpfx) --output-makefile $$@T
mv $$@T $$@
@@ -1462,12 +1464,15 @@ $(eval $(call include_dsosort_tests,dso-sort-tests-1.def))
$(eval $(call include_dsosort_tests,dso-sort-tests-2.def))
$(objpfx)dso-sort-tests-all2.def: dso-sort-tests-all.py
+ $(make-target-directory)
$(PYTHON) $< 2 > $@
$(objpfx)dso-sort-tests-all3.def: dso-sort-tests-all.py
+ $(make-target-directory)
$(PYTHON) $< 3 > $@
$(objpfx)dso-sort-tests-all4.def: dso-sort-tests-all.py
+ $(make-target-directory)
$(PYTHON) $< 4 > $@
$(eval $(call include_dsosort_tests_objpfx,dso-sort-tests-all2.def))
--
2.43.0
More information about the Libc-alpha
mailing list