Question about autoreconf to regenerate configuration files

[R. Diez]

Hallo Mike:

Newlib's build system has been a major pain and I would like to take this opportunity to learn more about it, and if that also helps with the other toolchain components, it's a plus point.


> [...]
> autotools (autoreconf really) doesn't run in parallel, so every subdir
> with a configure script needs a separate serialized run of all the tools.
> newlib has many many of these (arguably, too many).
> 
> on my quad core 4.2GHz AMD that is otherwise idle ...
> 
> $ time (cd newlib && autoreconf)
> real    5m22.170s
> user    3m13.709s
> sys     0m12.332s

My system is not really a very fast machine for development purposes:
- A laptop with an Intel Core i5-8265U CPU @ 1.60GHz (4 cores + hyperthreading)
- 1TB consumer-grade SSD Crucial MX500
- Ubuntu 20.04
My result is:

$ ( cd newlib && time autoreconf --force --verbose )
real    3m47,685s
user    1m48,533s
sys     0m10,338s

That's actually a rather long time. But if I understand correctly, your target is to have just 2 'configure' scripts for newlib and libgloss, which means that the Autoconf regeneration time is going to get cut drastically. Is that right?

Is it actually a problem that regenerating the Autoconf files in Newlib takes 5 minutes?

Is the Newlib project doing Continuous Integration? If so, it should actually be doing the Autoconf regeneration anyway, in order to test it as well. In any case, would a 5-minute delay there be an issue?

This Autoconf-related delay would only apply to sources directly obtained from the Git repository. The Newlib tarball releases would already include the Autoconf files, so no delay there.

If somebody, like RTEMS, is tracking Newlib's Git repository directly, that's their choice. But even then, they have options. If they distribute a release with patches etc, they can regenerate the Autoconf files before packaging Newlib for their own release. If the problem lies in the Continuous Integration system, you can cache the Autoconf files: if last time you checked out the same Git commit hash, then you do not need to regenerate the Autotools files again.


> again, this isn't "just newlib".  newlib is part of the historically combined
> toolchain tree/ecosystem.  that means you can take binutils, gdb, gcc, newlib,
> libgloss, cgen, sim, zlib, etc... and have a single monolithic source tree and
> build them all at once.  the projects have separated a little bit in that they
> have diff git repos, but the top-level dir and a few subdirs are still shared,
> and some folks still hand merge them.  newlib is part of that ecosystem and as
> such, follows its conventions.  changing newlib behavior would have a ripple
> effect and is why consensus across all of them is desirable.  although usually
> if you can convince gcc to change, the rest will follow to keep things simple.

OK, let me try to understand the details here.

I have been using the following makefile for years to build a cross-compiler GCC toolchain:

https://github.com/rdiez/JtagDue/blob/master/Toolchain/Makefile

That makefile builds GCC in 2 phases: a first, temporary GCC without a C library, and the final GCC with Newlib.

With that makefile, Newlib is configured, built and installed separately from GCC, so Newlib's build system does not matter at all in this scenario.

I think that is the same strategy as the OpenWrt toolchain makefile:
   https://git.openwrt.org/
   toolchain/Makefile
This way, you can use another libc. I believe that OpenWrt tends to use Musl instead of glibc.


The other way to build such a cross-compiler toolchain is to merge Newlib's sources with GCC's etc. That is called a "combined tree" and is mentioned here:

https://gcc.gnu.org/wiki/Building_Cross_Toolchains_with_gcc

The instructions on that page appear outdated and do not seem complete.

If you are keeping the Newlib build system compatible with GCC's etc, you must be testing with some commands, or maybe a script, that merges the trees. Could you share those steps? Alternatively, do you know of a web page that describes the steps accurately?

The instructions on the page I mentioned above talk about GCC's files overwriting any files in Newlib etc. I wonder at what level that is supposed to happen. For example, gcc-10.3.0 has a top-level 'configure' script. Is that supposed to overwrite Newlib's top-level 'configure' script?

In the combined tree, GCC must know somehow that it should include the 'newlib' subdirectory. Is that what GCC's 'configure' option '--with-newlib' is supposed to do? I could not find any information about "--with-newlib" in the GCC documentation.


To what extent is build system compatibility in this GCC ecosystem a problem? Newlib has had very outdated Autoconf files for quite some time and I presume that it has been working with many different GCC versions over the years. I do not suppose that we maintain a table of build system version compatibility between Newlib and GCC releases, do we?

My guess is that GCC's build system is calling 'configure', 'make' etc. inside newlib/, and it does not really matter if the versions are a little different. Or is a lot of the build system really shared?

-- 
rdiez