[PATCH/RFC] newlib: update build system generation documentation

Mike Frysinger vapier@gentoo.org
Thu Feb 17 05:52:29 GMT 2022

Replace all of the individual autotool steps with a single autoreconf.
This simplifies the documentation greatly, and in the current system,
only takes ~10 seconds to regenerate everything.

Update the developer documentation to cover all the major components
of the current build system.  Hopefully this is a fairly complete road
map to everything.  I tried to include everything that I wish I knew
when I started hacking on this :P.
NB: this requires a few more patches i have pending/waiting review, so it
isn't ready to merge just yet, but assuming the libm.a up-merging is OK
(and it sounds like it is), i think this is ready for review.

 newlib/HOWTO  | 161 ++++++++++++++++++++++++++++++++++++++++++++++++++
 newlib/NEWS   |   1 +
 newlib/README |  94 ++++++++---------------------
 3 files changed, 186 insertions(+), 70 deletions(-)

diff --git a/newlib/HOWTO b/newlib/HOWTO
index 668766096039..5a566456f8eb 100644
--- a/newlib/HOWTO
+++ b/newlib/HOWTO
@@ -110,3 +110,164 @@ Level 4
      Full POSIX or Linux compliance.  Essentially these are functions that are
 present in a standard Linux kernel but are irrelevant to an embedded system.
 These functions do not form part of the EL/IX API.
+     Newlib utilizes GNU Autotools (GNU Autoconf & GNU Automake) for its build
+system.  It can be broken down into two phases: configure & compile (make).
+NB: GNU Libtool is not used.  Newlib only produces static archives, and GNU
+Automake is capable of handling that.
+     Tool versions are tightly controlled.  GNU Autoconf 2.69 exactly must be
+used.  Attempts to use any other version, newer or older, will be rejected.  If
+the current system does not include that version, it can be downloaded from the
+GNU Autoconf website and installed into a local directory tree (like ~/.local).
+GNU Automake 1.15.1 is required as a minimum.  Newer versions are allowed when
+running locally, but developers submitting patches must use 1.15.1 exactly.
+     The configure phase naturally starts with the configure script.  It merges
+multiple inputs to produce the final script.
+* aclocal.m4: Generated by aclocal; do not edit.  Copies all the various m4
+  macros used by configure from the system install into the local copy.  This
+  allows regenerating configure without needing all the same system packages.
+* autom4te.cache/: Generated by aclocal to cache system search results.  Safe
+  to delete at any time, but never edit anything in this directory.
+* iconv.m4: Generated by libc/iconv/ces/mkdeps.pl; do not edit.  Keeps the
+  set of dynamic iconv synced between the source files and configure.
+* acinclude.m4: Historically, a way of sharing configure logic with subdirs.
+  At some point, it should all be merged into configure.ac directly.
+* configure.ac: The main configure script for supporting all Newlib options.
+* configure.host: An index of every support Newlib host and the settings to
+  use when compiling for it.
+* **/acinclude.m4: Subdirectories may include custom tests at configure time
+  without having to clutter the common configure.ac or acinclude.m4 file.  When
+  adding a new one, it must be manually included -- there is no mechanism to
+  automatically include every such file in the tree.  Look for m4_include lines
+  in configure.ac or existing acinclude.m4 files as examples.  Care must be
+  taken to not run excessive tests for incompatible targets, or to declare
+  generic names or symbols that can collide with other targets.  Any such file
+  should try to match $host (or similar) settings before running.  Settings
+  from configure.host (e.g. machine_dir or sys_dir) are guaranteed to be
+  available as well.  When creating names (Makefile variables, defines,
+  etc...), try to include the respective library and machine or sys name.
+  Do not confuse these with the historical top-level acinclude.m4 file!
+     The configure script has to take special consideration to the fact that it
+usually runs with a toolchain that lacks a C library (since Newlib will provide
+it once it's compiled & installed).  Newlib uses the uncommon AC_NO_EXECUTABLES
+macro to tell GNU Autoconf to not automatically require a toolchain that can
+link executables.  As a result, Newlib itself cannot perform any link tests.
+It shouldn't really need to, but configure authors must keep this in mind and
+stick to the appropriate source-level test -- AC_PREPROC for testing behavior
+of the preprocessor and AC_COMPILE for testing compiler behavior.
+     The newlib.h header is created from newlib.hin during the configure phase.
+The newlib.hin file is managed by autoheader, so do not edit it directly.  Any
+symbol declared in configure via AC_DEFINE is automatically included.  The file
+declares all the Newlib-specific configure settings and will be installed as
+part of the standard set of headers.  Critically, all symbols are namespaced
+with a leading _ prefix to avoid collisions with user code, and any symbols not
+namespaced are automatically removed.  This is necessary due to some system m4
+macros automatically adding their own defines that we cannot disable.  It also
+means that, when using AC_DEFINE, the _ prefix must be manually added.
+     A single Makefile is created during the configure phase.  This will build
+all of Newlib using non-recursive make (i.e., there are no subdirectories with
+their own Makefiles).  The single Makefile.am file includes Makefile.inc files
+in its immediate subdirectories, and those further include Makefile.inc files
+in their subdirectories.  This means all variables and targets exist in a
+single global flat namespace, so care must be taken to not use generic names
+like "SOURCES" in any Makefile.inc file.  Instead, use %C%_ on all variables to
+get a unique prefix, and GNU Automake will expand it.  In order to use one of
+the standard GNU Automake variables (e.g., CLEANFILES), first declare it at the
+top of the Makefile.am by assigning it (e.g., CLEANFILES =), and then always
+append to it in Makefile.inc (e.g. CLEANFILES += ...).  Again, GNU Automake
+will take care of combining all of this logic together to produce a portable
+     Developers used to changing to a subdirectory and running make commands
+to iterate quickly on a smaller section of Newlib might be surprised that this
+does not work -- there is no Makefile in any subdirectory.  Instead, make has
+to be run from the top-level, and the full path to the target is specified.
+So instead of running `cd subdir && make foo.o`, run `make subdir/foo.o`.  This
+applies to any target, not just objects.  Some common targets:
+make libm.a
+make libc.a
+     Conditionals are allowed in Makefile.inc files, but using them to elide
+an entire subdir Makefile.inc should be avoided.  Instead, GNU Automake
+conditionals should be used only within a single Makefile.inc file to control
+settings specific to that directory.  See the documentation section below for
+more information.
+     Per-directory compiler settings are allowed.  Similar to the syntax used
+above, the _%C% suffix can be used.  See the Makefile.am file for the current
+set of flags (e.g. CFLAGS, CPPFLAGS, etc...) that are supported.
+libm_a_CFLAGS_%C% = ...
+libc_a_CFLAGS_%C% = ...
+     Per-file compiler settings are allowed.  Similar to the syntax used
+above, the _%C% suffix can be used.  See the Makefile.am file for the current
+set of flags (e.g. CFLAGS, CPPFLAGS, etc...) that are supported.
+libm_a_CFLAGS_%C%_foo.c = ...
+libc_a_CFLAGS_%C%_bar.c = ...
+     An important aspect of the build system is allowing machine-specific
+directories to override common implementations.  For example, Newlib provides
+generic C implementations of string & memory functions in libc/string/ that
+perform adequately for any target, but many architectures include hand written
+assembly files for smaller & faster execution.  Those files live under the
+corresponding libc/machine/<ARCHITECTURE>/ directory.  Both versions are
+compiled when building Newlib, but when generating the libc.a archive, the
+common object is added first, followed by the machine-specific one.  This works
+because the object names follow a consistent convention: libm_a-<filename>.o
+for the math library and libc_a-<filename>.o for the C library.  If the common
+code stores a function foo in foo.c, then it can be overridden by another foo.c
+or foo.S or foo.s file since they all produce foo.o.  For this reason, all
+machine Makefile.inc files must be included last after all common Makefile.inc
+     Note that the Makefiles do not use VPATH to search source files across
+multiple source directories.  VPATH is used to match a single build directory
+to the corresponding source directory, but that is it.  For more details on
+how machine directories override functions in common directories, see the
+previous section.
+     The documentation, both the manual and man pages and header references,
+should be consistent regardless of what configure options are used (including
+the target system).  This makes it easy to get a complete documentation build
+at all times.  If inclusion of chapters depended specific configure options, it
+would be difficult to find the set of options necessary to produce a complete
+manual, if it were even possible in the first place (as some configure options
+are incompatible).  Since documentation chapters and APIs are declared in the
+respective directories and not in a centralized location, it is important to
+not use any GNU Automake conditional to control whether Makefile.inc files are
+included.  Documentation that cover optional APIs should note that it is not
+guaranteed to be included in every build of Newlib for every target.  The level
+of detail is left up to the author's discretion.
+     Newlib automatically builds itself for all multilib configurations that
+the active toolchain supports.  This logic is provided by common code from the
+combined toolchain source tree, but bolts itself into the GNU Autoconf and
+Automake files.  The critical parts to keep in mind:
+* ../config/multi.m4: AM_ENABLE_MULTILIB adds the --{en,dis}able-multilib
+  options to configure for users, and takes care of calling ../config-ml.in
+  for the Makefile in the current directory.  While not applicable to Newlib
+  anymore (since Newlib only produces a single Makefile now), this does not
+  automatically run for any subdir Makefiles.  Instead, it must be run manually
+  in AC_CONFIG_FILES's commands argument.
+* ../config-ml.in: Rewrites the specified Makefile to include the various
+  multilib variables and the multi-do command.  Notably, this only makes the
+  multi-do rule available, it does not cause it to be used anywhere.
+* ../multilib.am: Manually included in Newlib's Makefile.am.  This adds calls
+  to multi-do for the common important rules: all (i.e. compiling), install,
+  clean, etc...
+* FLAGS_TO_PASS: Newlib's Makefile.am declares all the variables to pass to
+  each multilib configuration via the multi-do rule.  While the multi-do rule
+  includes many common flags automatically, Newlib passes along a few extra
+  that are unique to it.
+    The testsuite is not integrated at all currently.  It is not really well
+tested in general.  The README file has more information.  Hopefully someone
+someday will step up to integrate it into the existing framework.
diff --git a/newlib/NEWS b/newlib/NEWS
index aaed47aaebea..c8e5a7cd3674 100644
--- a/newlib/NEWS
+++ b/newlib/NEWS
@@ -2,6 +2,7 @@
 - remove i?86-pc-linux-gnu support
 - remove decstation & sunos support
+- build system internals heavily rewritten & updated
 *** Major changes in newlib version 4.2.0:
diff --git a/newlib/README b/newlib/README
index aa48b246f968..a6645504cba9 100644
--- a/newlib/README
+++ b/newlib/README
@@ -23,13 +23,18 @@ Unpacking and Installation -- quick overview
 When you unpack the newlib-4.2.0.tar.gz file, you'll find a directory
 called `newlib-4.2.0', which contains:
-COPYING          config/          install-sh*      mpw-configure
-COPYING.LIB      config-ml.in     libgloss/        mpw-install
-COPYING.NEWLIB   config.guess*    mkinstalldirs*   newlib/
-CYGNUS           config.sub*      move-if-change*  symlink-tree*
-ChangeLog        configure*       mpw-README       texinfo/
-Makefile.in      configure.in     mpw-build.in
-README           etc/             mpw-config.in
+COPYING           README-maintainer-mode  include/        mkdep
+COPYING.LIB       compile                 install-sh      mkinstalldirs
+COPYING.LIBGLOSS  config/                 libgloss/       move-if-change
+COPYING.NEWLIB    config-ml.in            libtool.m4      newlib/
+COPYING3          config.guess            ltgcc.m4        setup.com
+COPYING3.LIB      config.rpath            ltmain.sh       src-release
+ChangeLog         config.sub              ltoptions.m4    symlink-tree
+MAINTAINERS       configure               ltsugar.m4      texinfo/
+Makefile.def      configure.ac            ltversion.m4    ylwrap
+Makefile.in       depcomp                 lt~obsolete.m4
+Makefile.tpl      djunpack.bat            makefile.vms
+README            etc/                    missing
 To build NEWLIB, you must follow the instructions in the section entitled
 "Compiling NEWLIB".
@@ -46,7 +51,7 @@ More Documentation
    Newlib documentation is available on the net via:
-   http://sourceware.org/newlib/docs.html
+   https://sourceware.org/newlib/docs.html
    All the documentation for NEWLIB comes as part of the machine-readable
 distribution.  The documentation is written in Texinfo format, which is
@@ -77,7 +82,6 @@ format.  On its own, TeX cannot read, much less typeset a Texinfo file.
 `newlib-VERSION-NUMBER/texinfo' directory.
 Compiling NEWLIB
@@ -511,70 +515,20 @@ run the testsuite.
 Regenerating Configuration Files
-At times you will need to make changes to configure.ac and Makefile.am files.
-This will mean that configure and Makefile.in files will need to be
-At the top level of newlib is the file: acinclude.m4.  This file contains
-the definition of the NEWLIB_CONFIGURE macro which is used by all configure.ac
-files in newlib.  You will notice that each directory in newlib containing
-a configure.ac file also contains an aclocal.m4 file.  This file is
-generated by issuing: aclocal -I${relative_path_to_toplevel_newlib_dir}
-The first relative directory is to access acinclude.m4.  The second relative
-directory is to access libtool information in the top-level src directory.
-For example, to regenerate aclocal.m4 in newlib/libc/machine/arm:
-  aclocal -I ../../.. -I ../../../..
-Note that if the top level acinclude.m4 is altered, every aclocal.m4 file 
-in newlib should be regenerated.
-If the aclocal.m4 file is regenerated due to a change in acinclude.m4 or
-if a configure.ac file is modified, the corresponding configure file in the 
-directory must be regenerated using autoconf.  No parameters are necessary.
-In the previous example, we would issue:
-  autoconf
-from the newlib/libc/machine/arm directory.
-If you have regenerated a configure file or if you have modified a Makefile.am
-file, you will need to regenerate the appropriate Makefile.in file(s).
-For newlib, automake is a bit trickier.
-Makefile.in files are generated from the nearest directory up the chain
-which contains a configure.ac file.  In most cases, this is the same
-directory containing configure.ac, but there are exceptions.
-For example, the newlib/libc directory has a number of
-subdirectories that do not contain their own configure.ac files (e.g. stdio).
-For these directories, you must issue the automake command from newlib/libc
-which is the nearest parent directory that contains a configure.ac.
-When you issue the automake command, you specify the subdirectory for
-the Makefile.in you are regenerating.  For example:
-   automake stdio/Makefile stdlib/Makefile
-Note how multiple Makefile.in files can be created in the same step.  You
-would not specify machine/Makefile or sys/Makefile in the previous example
-because both of these subdirectories contain their own configure.ac files.
-One would change to each of these subdirectories and in turn issue:
-   automake Makefile
+At times you will need to make changes to configure.ac, Makefile.am and
+Makefile.inc files.  This will mean that configure and Makefile.in files will
+need to be regenerated.  The easiest way to do so is by using the autoreconf
+tool in the newlib directory.
-Let's say you create a new machine directory XXXX off of newlib/libc/machine.
-After creating a new configure.ac and Makefile.am file, you would issue:
+  autoreconf
-   aclocal -I ../../..
-   autoconf
-   automake Makefile
+This will run a number of autotool programs for you.  To see the individual
+steps, add the -v option.
-from newlib/libc/machine/XXXX
+  autoreconf -v
-It is strongly advised that you use an adequate version of autotools.
-For this latest release, the following were used: autoconf 2.69, aclocal 1.15.1,
-and automake 1.15.1.
+It is strongly advised that you use an adequate version of autotools.  For this
+latest release, the following were used: autoconf 2.69 and automake 1.15.1.
 Reporting Bugs
@@ -587,4 +541,4 @@ Since NEWLIB supports many different configurations, it is important
 that you be precise about this.
 Archives of the newlib mailing list are on-line, see
-	http://sourceware.org/ml/newlib/
+	https://sourceware.org/ml/newlib/

More information about the Newlib mailing list