[PATCH v2 0/6] Improve make and make check parallelism

[DJ Delorie]

Adhemerval Zanella <adhemerval.zanella@linaro.org> writes:
> Every subdirectory's compile tail and link step therefore leaves most
> cores idle,

One trick I use is to sort the things to build by size, largest first.
That tends to pack into available cores better.  Currently we sort by
name.

> This series makes the subdirectory recursion run in parallel while
> encoding the ordering the serial recursion relied on as explicit
> dependencies:

A few weeks ago I asked an AI to try to fix the problems that stopped us
from doing parallel toplevel.  I won't share the results as it would
taint your patch, but it had a few suggestions (and claimed way more of
a speedup than you report, but I never had a chance to validate it):

* There are some toplevel tasks that need to be run serially, mostly
  relating to generating the various versions files.  It used flock(1)
  to serialize them.

* it added various cross-directory build dependencies, mostly things
  depending on support/ but elf->csu and support->elf.  I don't know if
  gen-sorted handles this.

>   5. The container testroot, which performs a full DESTDIR
>      installation, is no longer created when run-built-tests=no, since
>      no container test will run.

The time needed to build the testroot was a concern back when I added
it, and the compromise is that we only build it once - it lacks
dependencies that would cause lots of spurious rebuilds.  Making this
happen even less often makes sense.

It can, however, be built in parallel with any non-container test.
It's disk-heavy, not cpu or ram.