This is the mail archive of the mailing list for the crossgcc project.

See the CrossGCC FAQ for lots more information.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[crosstool-NG] Design discussion

Hello all!

Recently, I've been challenged about the design of crosstool-NG.

This post is to present the overall design of crosstool-NG, how and why
I came up with it, and to eventually serve as a base for an open discussion
on the matter.

Hopefully some will jump in to offer their views on the subject, and offer
sugestions as what should be done to improve the situation, should the need

The mail is structured that way:
1) Genesis of crosstool-NG
2) The way to fullfill the requirements
  2.a) Ease maintenance
  2.b) Ease configuration of the toolchain
  2.c) Support newer versions of components
  2.d) Add new features
  2.e) Add alternatives where available
3) crosstool-NG installation
  3.a) Setting up crosstool-NG: why using a ./configure?
  3.b) Installing crosstool-NG: why is it required?
  3.c) Runing crosstool-NG: why can't I run make menuconfig?
4) crosstool-NG internals
  4.a) Programming languages used in crosstool-NG
  4.b) Internal API
5) Conclusion

In advance, I do apology for the really, really long post, and for the
limited subset of the English language I use.


1) Genesis of crosstool-NG

First, a little introduction to put things straight.

About four years ago, I needed to generate cross-compilers for ARM and
MIPS. One of the requirements was to be able to use various versions of
the components (gcc, glibc, binutils...), and a second was to be able to
switch between glibc and uClibc. Of the different tools I tested, crosstool
was the one most closely matching the requirements, so I ended up using
that for the following 1.5 years.

But crosstool was not easy to configure, and the available versions of the
components were most of the time lagging behind. It was glibc-centric, and
I had to add uClibc support, which was not accepted mainstream.

In the end, maintaining my own tree became problematic, and I decided to
give a try at enhancing crosstool with the following main goals in mind,
in this approximative order of importance:

 a- ease overall maintenance
 b- ease configuration of the toolchain
 c- support newer versions of components
 d- add new features
 e- add alternatives where it was available

I mostly saw my changes as a experimental branch of crosstool, which
would ultimately pick interesting features as they mature, while dumping
the uninteresting ones. So crosstool would be the stable branch, while
my work would serve as a kind of testbed. Hence the name: crosstool-NG,
"NG" for "Next Generation".

Never, at any one time, did I intend this stuff to replace crosstool.
What happened is that, around the time I was working on this, Dan KEGEL
became less and less responsive, and changes sent to the list (by any
one, not just me) took ages to get applied, if they even get applied at

So that was how crosstool-NG was born to the world...


2) The way to fullfill the requirements

The first move I made was to first start from scratch. That way, it
sounded to me it would be easier to come up with a good layout of

2.a) Ease maintenance

At the heart of crosstool was a single script. In there was all the build
procedures for all the components, from the installing the kernel headers
up to building gdb.

The first step was to split up this script into smaller ones, each
dedicated to building a single component. This way, I hoped that it would
be easier to maintain each build procedure on its own.

2.b) Ease configuration of the toolchain

In the state, configuring crosstool required editing a file containing
shell variables assignements. There was no proper documentation at what
variables were used, and no clear explanations about each variables

The need for a proper way to configure a toolchain arose, and I quite
instinctively turned to the configuration scheme used by the Linux
kernel. This kconfig language is easy to write. The frontends that
then present the resulting menuconfig have limitations in some corner
cases, but they are maintained by the kernel folks.

Again, of with the build scripts, above, I decided to split each components
configuration into separate files, with an almost 1-to-1 mapping.

Of course, there are configuration sections that do not apply to a
specific component, but to the overall toolchain: the place to install
it, the target and its options (BE/LE, CPU variants...). And some options
tell crosstool-NG how to behave: the place to find source tarballs, log
verbosity, and so on...

2.c) Support newer versions of components

Adding support for newer versions merely required building an appropriate
patchset for that version. As I didn't have time to dig in every individual
mailing lists, I shamelessly vampirised patches from different sources:

 - the original crosstool patchset
 - the Gentoo patchset
 - maybe even the LFS and CLFS patchests
 - the buildroot patchset (later, when integrating uClibc)
 - others, floating around, or made up by myself

Of course I kept appropriate attribution (or at least I tried to, I may
have forgotten some, apologies) on every patches I so vampirised.

2.d) Add new features

Ah! Big sentence that the one reading "new feature"!

So, by new features, I mean, for example, new configuration options for:
 - NPTL-enabled glibc
 - using Linux exported headers
 - ...

That implied adding aconfig kno in a config file, and updating the code in
the corresponding build script, to set the appropriate ./configure option(s),
to create the appropriate config file...

2.e) Add alternatives where available

The most obvious example of an alternative component is the C library to
use: glibc vs uClibc. Then came eglibc.

It was almost as simple as adding a new config file, a new build script
and a choice in the menuconfig. Sometimes, it required tweaking other
components so they recognised the new alternative; gcc was one that needed
a few tricks and patches to recognise uClibc. But all in all, it went
quite smoothly.


3) crosstool-NG installation

3.a) Setting up crosstool-NG: why using a ./configure?

crosstool-NG itself requires a few tools to be present so it can run.
Usualy, packages do that via a script called ./configure. So it goes
for crosstool-NG.

crosstool-NG, for its own usage, requires the following:
 - grep that accepts -E
 - sed that accepts -i and -e
 - bash-3 (or bash-4)
 - cut, xargs, install, awk
 - gcc, to build the kconfig parsers
 - curl (or wget), tar, gzip, bzip2, patch, to fetch, extract and patch
   the tarballs
 - make

Alongside those tools, there are a few that are absolutely required to build
the components:
 - bison
 - flex, bison
 - makeinfo (yeah, we migh not want the doc, but some components build in
 - automake
 - libtool

crosstool-NG does not use them, but better check them now, than fail later.

Now, once crosstool-NG has been configured, it's time to "build" it.
Merely, this is a matter of putting the right paths in a file that the
build scripts do include, and to replace the #!@@bash@@ occurences with
the bash we found (or were told to use) in a few scripts.

3.b) Installing crosstool-NG: why is it required?

OK, now why would it be needed to _install_ crosstool-NG? Why can't it
simply run from the source dir?

The first answer is: yes, you can! Just pass --local to ./configure and
crosstool-NG will be able to run from its source directory. But then, why
on earth is it needed to run make install at all? Well, the install step
sets the execute bit on the frontend, ct-ng, and I find it more convenient,
in the point of view of the prgorammer I am, to have a single code path in
the Makefile. That maybe is not what you would expect, but that's the way
I did it. I agree that it can seem ugly, but that's a minor issue, IMHO.

Now, why is crosstool-NG intended to be installed at all?

Think of crosstool-NG as a package like any other: you configure it, you
install it, then you use it. For example, would it make sense that you
build your own program in the gcc source directory? I don't think you
do that, but that you really install gcc in some place, add that to the
PATH, and then cd into your programs source dir and run gcc from there.

To parallel between crosstool-NG to gcc:
gcc:                   crosstool-NG:
  vi foo.c               ct-ng menuconfig
  gcc -o foo foo.c       ct-ng build
  ./foo                  target-tuple-gcc

Of course, this is overly simplistic, but you get the idea...

3.c) Runing crosstool-NG: why can't I run make menuconfig?

As said below, the frontend, ct-ng, is written as a Makefile script. That
does *not* mean that is is a _Makefile_: it is an executable that happens
to be written in the same language that Makefiles are.

Now, what if it were written in Python? Would you expect to run it with
"python ct-ng", or would you simply expect to run it with "ct-ng"? The
same applies if it were written as shell script: you don't call shell
scripts with "sh my_script", do you?. And worse, what if it were written
in C ? Surely you'd expect to run it with "ct-ng", no?

Well, the fact that ct-ng is written in the Makefile language might not be
the wisest decision I made on crosstool-NG, but that does *not* change the
fact that you don't want to run make (or any other interpreter) manually
to run it, right?


4) crosstool-NG internals

In this section, we'll discuss the internals of crosstool-NG. Most notably,
we'll speak about the API that acts as a frontier between the various parts
of crosstool-NG: the config entries, the main script, the specific build

Note: although probably the most important section, it is also the shortest.
I can't and don't want to simply quote what is already available in the
docs/ directory in the sources.

4.a) Programming languages used in crosstool-NG

The frontend, called ct-ng out of laziness, is written as a GNU Makefile
script. Yes. It's a Makefile that can get run. It uses the sha-bang
#!/where/ever/is/make -rf

The core of crosstool-NG is written as a set of bash-3 shell scripts. I do
insist on bash-3, because I use bashisms, and thus the scripts are *not*
POSIX compliant. So, they do not use the traditional sha-bang #!/bin/sh,
but they do explictly use the sha-bang #!/where/ever/is/bash

The config files are written with the kconfig syntax. The parsers were
initially copied and updated from the one in toybox, then updated with the
one from the Linux kernel 2.6.28. It has had a few changes:
  - the CONFIG_ prefix has been replaced with CT_
  - a leading | in prompts is skipped, and subsequent leading spaces are
    not trimmed
  - otherwise-leading spaces in prompts are silently trimmed

4.b) Internal API

That API is 2-fold:
 - variables set by the main script, and available to build scripts
 - entry-points that build scripts must provide, and called by the main

The file in docs/overview.txt tries to document this API. It is almost
complete wrt the "arch" and "kernel" API. Others are inexistant. This is
due to a lack of time to properly document the API, because it has been
much fluctuating until not so long ago.

In this respect, I did not do much better than the original crosstool. :-(

I won't discuss it here, you are encouraged to read that file for more
on the subject.


5) Conclusion

So far, I hope that I've explained enough so people can jump in and comment.
All critiscism, whether positive or negative, is very welcome. I don't claim
to be either omniscient or omnipotent. I will gladly look at your coments!
Of course, should you find a point on which you disagree, I'd appreciate an
alternate solution to the problem.

Again, I do apologise for the long mail. But that seemed to be really

Yann E. MORIN.

PS. My partner is currently badly sick, and I need to take care of her.
In advance, sorry for the delay that situation may impose to my answers.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +0/33 662376056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| --==< ^_^ >==-- `------------.-------:  X  AGAINST      |  \e/  There is no  |
| | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

For unsubscribe information see

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]