[PATCH v2 0/6] Improve make and make check parallelism

[Adhemerval Zanella Netto]

On 16/06/26 17:06, DJ Delorie wrote:
> Adhemerval Zanella <adhemerval.zanella@linaro.org> writes:
>> Every subdirectory's compile tail and link step therefore leaves most
>> cores idle,
> 
> One trick I use is to sort the things to build by size, largest first.
> That tends to pack into available cores better.  Currently we sort by
> name.

That interesting, do you mean to sort by the number of files within
the subfolder, or use the file size as a proxy? 

I might try to check if this pays out, and most likely create a better
observation tools besides measuring time. At least with a very unscientific
measurement (staring at btop output), I noticed that for 'make' the
serialization points seems minimal (csu, then rtld mainly). 

> 
>> This series makes the subdirectory recursion run in parallel while
>> encoding the ordering the serial recursion relied on as explicit
>> dependencies:
> 
> A few weeks ago I asked an AI to try to fix the problems that stopped us
> from doing parallel toplevel.  I won't share the results as it would
> taint your patch, but it had a few suggestions (and claimed way more of
> a speedup than you report, but I never had a chance to validate it):
> 
> * There are some toplevel tasks that need to be run serially, mostly
>   relating to generating the various versions files.  It used flock(1)
>   to serialize them.

I used a different strategy:

546 # The subdirectories that generate shared files in $(common-objpfx)
547 # consumed by the rest of the build without explicit dependencies: csu
548 # provides the gen-as-const headers, and on Hurd the mach and hurd
549 # directories generate the MiG RPC headers (every other subdirectory
550 # otherwise runs a nested make in hurd/ to create them, racing under
551 # parallel recursion; see sysdeps/mach/hurd/Makefile).  Run them serially,
552 # in their sorted order (mach, hurd, csu).
553 +subdir-pregen := $(filter mach hurd csu,$(subdirs))
554 +subdir-rest := $(filter-out $(+subdir-pregen),$(subdirs))
555
556 $(foreach t,$(+ordered_parallel_subdir_targets),$(eval \
557   $(addsuffix /$(t),$(+subdir-rest)): $(addsuffix /$(t),$(+subdir-pregen))))
558 +subdir-pregen-prev :=
559 $(foreach d,$(+subdir-pregen),$(foreach t,$(+ordered_parallel_subdir_targets),$(eval \
560   $(d)/$(t): $(addsuffix /$(t),$(+subdir-pregen-prev))))\
561   $(eval +subdir-pregen-prev := $(d)))

The idea is 'csu' generates the tree-wide gen-as-const headers that every
subfolder requires, and then make prevents any parallel execution unable to 
start until 'csu' has completely finished.

The rules does:

  string/subdir_lib: csu/subdir_lib
  math/subdir_lib:   csu/subdir_lib
  ... (all ~49) ...

So when you do make -jN lib (the subdir_lib pass):

  * csu/subdir_lib is the only subdir_lib target with no subdir prerequisite, 
    so it's the only one eligible at the start.
  * string/subdir_lib, math/subdir_lib, etc. all list csu/subdir_lib as a 
    prerequisite, so they are not eligible while csu is still building.

So it is essentially:

  [ csu/subdir_lib sub-make ]
   * generates abi-versions.h, gen-as-const headers, etc.
   * compiles csu's own objects
   |___ exits ___
                 |
                 [ string/subdir_lib ] [ math/subdir_lib ] [ ... 49 in parallel ]
                          each sub-make checks "is abi-versions.h up to date?"
                          -> yes (csu made it) -> does NOT regenerate it


The flock *might* be a interesting approach because during trying to parallelize
the build I hit two issues where shared file is produced lazily by a non-csu
subdir (csu gen-as-const headers and the catgets locales, handled in the third
patch - and I am continuously running multiple parallel builds to if I missed 
something).  

It would allow parallelize the 'csu' subfolder itself, and start the subfolders 
in parallel.

But I think flock has two main extra complications: it is not POSIX (so Linux
specific, not sure if this is properly support by Hurd), and it leaves the make 
graph (the dependencies is defined by writing access). I this the later might
be source of potential issues, I never saw Makefile actively using it.

> 
> * it added various cross-directory build dependencies, mostly things
>   depending on support/ but elf->csu and support->elf.  I don't know if
>   gen-sorted handles this.

The 'csu' is special, but the extra dependencies is expressed by the extra
subdir-deps* from the first patch. On x86_64-linux-gnu, the sysd-sorted
is now create with there extra definitions:

subdir-deps-assert += iconvdata
subdir-deps-assert += localedata
subdir-deps-catgets += intl
subdir-deps-debug += localedata
subdir-deps-iconvdata += iconv
subdir-deps-iconvdata += localedata
subdir-deps-intl += iconvdata
subdir-deps-intl += localedata
subdir-deps-libio += localedata
subdir-deps-localedata += locale
subdir-deps-malloc += dlfcn
subdir-deps-malloc += nptl
subdir-deps-malloc += htl
subdir-deps-malloc += rt
subdir-deps-mathvec += math
subdir-deps-posix += localedata
subdir-deps-rt += nptl
subdir-deps-rt += htl
subdir-deps-stdio-common += localedata
subdir-deps-stdlib += localedata
subdir-deps-string += localedata
subdir-deps-support += elf
subdir-deps-time += timezone
subdir-deps-wcsmbs += localedata
subdir-deps-resolv += nptl
subdir-deps-resolv += htl
subdir-deps-hesiod += nss
subdir-deps-hesiod += resolv
subdir-deps-nis += nss
subdir-deps-nscd += nptl
subdir-deps-nscd += htl
subdir-deps-nss += dlfcn
subdir-deps-nss += resolv

> 
>>   5. The container testroot, which performs a full DESTDIR
>>      installation, is no longer created when run-built-tests=no, since
>>      no container test will run.
> 
> The time needed to build the testroot was a concern back when I added
> it, and the compromise is that we only build it once - it lacks
> dependencies that would cause lots of spurious rebuilds.  Making this
> happen even less often makes sense.
> 
> It can, however, be built in parallel with any non-container test.
> It's disk-heavy, not cpu or ram.

I think we can add some extra parallelization for run-built-tests=yes,
similar on how csu is done. I will take a look.