RFC: script to build many glibc configurations

[Joseph Myers]

Attached is a first version of a Python (3.5 or later) script to build
many different configurations of glibc, including building the
required cross compilers first.  The idea is that when it's more
mature it would go in the glibc source tree as a rough sort of
equivalent to GCC's contrib/config-list.mk.  It's not intended to
change any patch testing requirements, although some people may wish
to use it for high-risk patches such as adding warning options (and it
can also be used to test building, including compiling tests, for an
individual configuration, if e.g. you wish to do such a compilation
test of a patch for an architecture it touches).

The present version only tests a limited number of glibc
configurations, to illustrate its features.  I intend to expand it to
cover most or all of the ABI variants in
<https://sourceware.org/glibc/wiki/ABIList> and it would be desirable
to cover enough other variants e.g. for CPUs using different sysdeps
directories to test building each piece of code in glibc at least
once.

You call the script as

build-many-glibcs.py /some/where thing-to-do

where /some/where is a working directory for the script.  It will
create and use subdirectories build, install, logs therein.  It
expects a subdirectory src to already exist; it should contain
directories binutils, gcc, glibc, gmp, linux, mpc, mpfr with the
sources of those components (all except glibc can be symlinks to
sources elsewhere).  thing-to-do is one of host-libraries, compilers,
glibcs.  So you run, in that order:

build-many-glibcs.py /some/where host-libraries
build-many-glibcs.py /some/where compilers
build-many-glibcs.py /some/where glibcs

host-libraries is run once and then those libraries are used for all
the compilers.  compilers can be run once and then used many times for
testing different glibc versions (so a bot only needs to update glibc
and rerun the glibcs task, if using stable GCC / binutils; if testing
the latest versions of the whole toolchain together including mainline
GCC, it would probably want to update everything and rerun both
compilers and glibcs).  You can also name particular variants after
"compilers" or "glibcs" to build just those variants (the possible
variants are hardcoded in the script).

Any comments?  Possible improvements I have in mind include: covering
more variants, hopefully all or nearly all ABIs; support for creating
/ updating the src directory (with a choice of component versions) so
that doesn't need to be done manually; optionally leaving build
directories around, or leaving them around for failed builds (right
now they are always cleaned up); saving content of .out files for
failed glibc tests, and the full content of tests.sum, before cleaning
up; allowing the set of configurations to depend on the GCC version;
optionally looping over architecture-independent glibc variants of
CFLAGS and configure options as well, for every glibc configuration
listed (e.g. -Os).

Testing with this version depends on my patches
<https://sourceware.org/ml/libc-alpha/2016-11/msg00092.html> and
<https://sourceware.org/ml/libc-alpha/2016-11/msg00093.html> to avoid
problems running glibc tests.  GCC versions before 4.9 are not
expected to work (the code uses --with-glibc-version to get the
bootstrap GCC appropriately configured).  With GCC 5 branch, the
ColdFire build falls over with an ICE building libgcc while the other
builds successfully build the compilers (some of them have glibc test
failures I'll investigate once I've set up saving of logs of failed
tests).

On a 2.6 GHz Haswell-based Xeon (16 cores, 32 with HT) the build took
about 17 minutes for the compilers (note that building the compilers
includes building a subset of glibcs so that shared libgcc and
libstdc++ can then be built) and 12 for the glibcs after the
compilers.  However, there probably aren't yet enough variants for
parallelism to be optimally exploited; I expect that with more
variants more parallelism will also be used.

-- 
Joseph S. Myers
joseph@codesourcery.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build-many-glibcs.py
Type: text/x-python
Size: 31295 bytes
Desc: 
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20161103/f4fc5634/attachment.py>