Re: [crosstool-NG] Design discussion

[Rob Landley <rob at landley dot net>]

On Monday 06 April 2009 17:12:35 Yann E. MORIN wrote:
> Rob,
> All,
>
> On Monday 06 April 2009 07:45:49 Rob Landley wrote:
> > Ok, I think I left off around here:
>
> OK, I think I'm coping with the back-log ;-)
>
> > > 2.b) Ease configuration of the toolchain
> > >
> > > In the state, configuring crosstool required editing a file containing
> > > shell variables assignements. There was no proper documentation at what
> > > variables were used, and no clear explanations about each variables
> > > meaning.
> >
> > My response to this problem was to write documentation.
>
> Sure. But documentation is not all. Fixing and enhancing both the code
> and the configuration scheme is also a way to achieve a better end-user
> experience (Ahaha! I've wandered too much with the marketting dept lately).
>
> > While I've used kconfig myself, there's an old saying: "If all you have
> > is a hammer, everything looks like a nail".
>
> Hehe! Show me a better and simpler way.

My attempt to do so was why I brought my build system up.  Not going there 
now.

> > The failure mode of kconfig is having so much granularity
>
> So what? Should I restrict what the end-user is allowed to do, based solely
> on my own experience?

First of all, I never assume I'm going to think of everything the end user is 
going to want to do.  I'm not that smart, or that crazy.

Secondly, I point out that when it comes down to it, crosstool-ng is more or 
less a series of shell scripts.

I chose a simple shell script implementation so that if somebody really wanted 
to tweak what it did, they could just go in and edit said shell script.  "If 
all else fails, it's just a shell script."  (In fact, reading through my 
shell scripts probably takes considerably less time than reading through my 
documentation.)  Obviously that's not a first resort, but it gave me a 
threshold for "at what point is it collectively less work for the 0.1% of 
people who will ever care about this to go in and edit the dumb script than 
it is for the other 99.9% to figure out what this option _means_ and that 
they don't need to do it."

Making crosstool-ng's scripts simple, easy to read, well-documented, and 
generally easy to edit means that menuconfig isn't the ONLY way people can 
get the behavior they want out of them, either.  (Whether you intended that 
or not. :)  You are giving your users the complete source code to your build 
system.

Also, on "-pipe" specifically, why not just give them a $CFLAGS that's fed in 
to every build, which they can set in menuconfig?  What I mean is, why limit 
it to -pipe, or treat that option specially?

> If the possibility exists in the tools, then why 
> prevent the user from using (or not using) that? Configuring a compiler
> has soooo many options, it drives you mad.

As the lawyers would say, "asked and answered".

> Quite a lot can be inferred  
> from higher-level options, such as the arch name and the CPU variant, or
> the ABI...

Assuming you've got a year's experience with cross compiling, sure.

Ok, standard usability exercise, ala Alan Cooper's "The Inmates are Running 
the Asylum" or Don Norman's "The Design of Everyday Things".

Imagine you need to build a toolchain for an end user, but you'll have to 
configure it for them.  You've sat them down in a chair and get to ask them 
questions, in person, and you're going to write down the answers on a piece 
of paper.

The first questions that come to mind are, probably things like "what target 
processor is it for?", "linux or raw elf", "gcc or uClibc", right?

Look at your menuconfig.  What's the first questions it asks in the first 
menu?  "Would you like to use obsolete or experimental features?  What 
directory should I save the tarballs I download in?"

A reasonable amount of newbie anguish might be spared just by moving that menu 
down to just above "debug facilities".  The next menus three are a good 
start, although why "GMP and MPFT" is at the top level, I have no idea.  
(These are gcc 4.3 implementation details, aren't they?  Presumably they have 
help entries now, but shouldn't they be under the gcc menu or something?)

It might make sense to have a "packages" menu, containing binutils, C 
compiler, C library, and tools menus.  Although the tools menu with libelf 
and sstrip in it is kind of inexplicable; when do you need that and why?

Also, a newbie should probably start with a preconfigured sample toolchain 
their first time through, but the README never mentions those, and when you 
run ct-ng with no arguments in an 80x25 screen it scrolls off, if you pipe it 
to less and start reading you see "menuconfig" and start investigating that 
without reading through the rest.  Once you're in menuconfig, there's no hint 
of the 

I read the README all the way through and looked at the 
initial ./configure --help fairly closely, but by the time I got to ct-ng and 
saw it did menuconfig I skipped straight to reading the menuconfig help and 
thus setting up a toolchain build the slow way.  (I did at one point 
find . -name "*_defconfig" and it hit all the stuff in 
targets/src/linux-*...)

Oh, and under "target optimizations", a little judicious renaming might be 
nice.

"Generate code for which ABI"
"Emit assembly for which CPU"

(When I saw "Emit assembly for CPU" I went "um, yes please?" and had to read 
the help to have a clue what it was on about.)

It might help to indent "Tune for which CPU" under "Emit assembly for which 
CPU", since it is a sub-option.

> > Ironically, kconfig is only really worth using when you have enough
> > config options to bother with it.
>
> But there *are* a lot of options! There isn't even all that could be
> available!

I agree there are a lot of options and could be more.

> > When you have small numbers of config options
> > that are usually going to be off, I prefer environment variables (with a
> > config file in which you can set those in a persistent manner) or command
> > line options.  Since you can set an environment variable on the command
> > line, ala:
> >
> >   FORK=1 ./buildall.sh
> >
> > I lean towards those.  Possibly a matter of personal taste...
>
> I think so. To configure stuff, I prefer having a GUI that is not vi, but
> that is still simple. and the kconfig language and its mconf interpreter
> seems quite fitted, even if it is not the best. But here is no alternative
> that I'm aware of. Plus, it is quite well known thanks to the Linux kernel
> using it.

Sure.  If you're going to use something like kconfig, kconfig is a good 
choice. :)

> > Similarly, the "experimental" one seems useless because when you enable
> > it the experimental versions already say "EXPERIMENTAL" in their
> > descriptions (wandered around until I found the binutils version choice
> > menu and looked at it to be sure).  They're marked anyway, so why is an
> > option to hide them an improvement?
>
> EXPERIMENTAL in the prompt is just a string. A config knob makes the user
> really aware that he/she's trying something that might break.

They're always trying something that might break.  I've broken the "cat" 
and "echo" commands before.  (And my friend Piggy broke "true" in a way that 
took down his system!  I bow before him, I can't top that one.)

> Plus, it marks the resulting config file as containing EXPERIMENTAL
> features/versions/... and is easier to process.

Sounds like it's there to discourage them from doing it, which I suppose makes 
sense.

> > As for the third, wasn't there a debug menu?  Why is "Debug crostool-NG"
> > in the paths menu?  (Rummage, rummage... Ah, I see, the debug menu is a
> > list of packages you might want to build and add to the toolchain.  Ok,
> > sort of makes sense.  Still, the third thing a newbie sees going through
> > in order as a "very very expert" option.  Moving on...)
>
> That's why the former in titled "Debug crosstol-NG", while the latter
> is titled "debug facilities". Again, maybe the wording is wrong.

I'm trying to figure out a coherent organization.

Under "operating system" you have "Check installed headers", which is another 
debug option.  Similarly the kernel menu has kernel verbosity but the paths 
and misc options menu has general verbosity, and not only are they nowhere 
near each other but I dunno what the difference is.  (There isn't a "uClibc 
verbosity" and it has a similar V=1 logic, if that's what the other option 
means which is just a guess.  No help available for this kernel option...)

> >   ()  Local tarballs directory (NEW)
> >   (${CT_TOP_DIR}/targets) Working directory (NEW)
> >   (${HOME}/x-tools/${CT_TARGET}) Prefix directory (NEW)
> >
> > Most users aren't going to care where the local tarballs directory is, or
> > the working directory.
>
> Most. Not all. And the help entries are here to tel the user whether
> it wise to change.
>
> > The "prefix directory" is presumably different from where
> > we just installed with --prefix.
>
> The help roughly says: the path were the toolchain is expected to run from.
> Unfortunately, there is yet no support for DESTDIR, the place where the
> toolchain will be installed, to allow installing out-of-tree. For the time
> being, the DESTDIR is plainly / that is, the toolchain is expected to run
> on the system it is built on. But that should eventually be fixed.

I used a wrapper.  Not going there.

> > I suppose it's nice that you can override
> > the defaults, but having it be one of the first questions a user's posed
> > with when going through the options in order trying to configure the
> > thing isn't really very helpful.  It's not my problem, just _work_.
>
> Where should I install the toolchain? In the user's home directory?
> This is indeed the default, but you are pestering against it!

/me is totally spoiled by automatically relocatable toolchains, pleads the 
5th.

> If not in ${HOME}, where should I install the toolchain? In /opt ?
> In /usr/local ? Bah, most useres don't have right access there.
> Except root. But building as root is asking for problems!

I forgot how big an issue this is for unwrapped gcc's.

> > (I also don't know
> > what CT_TOP_DIR and CT_TARGET are, I'd have to go look them up.)
>
> docs/overview.txt is advertised in the top-level README.

I read through the introduction and history parts of that file fairly 
dilligently, which consumed most of the attention span I had devoted to it.  
Then I got to the point where it told me to install and I went and did that, 
spending half an hour or so fighting with ./configure for and installing the 
various prerequisite packages.  Then I came back to the overview and skimmed 
a bit more until it told me about menuconfig, at which point I went off and 
read menuconfig help for a long time, started reading the project's source 
code while I was doing that to see what some of the config symbols actually 
affected, and never really came back to the overview file.  (Skimmed over it 
a couple times, but didn't read it closely past the first couple hundred 
lines.)

(I didn't say I didn't believe I _could_ look up those symbols, just that I 
didn't know what they did off the top of my head.)

Part of what I'm reacting to here is the experience the project presents to 
new users.  Once you've been using something for six months you can get used 
to almost anything, but learning it the first time can be really frustrating.

> > For comparison, my system creates a tarball from the resulting cross
> > compiler, and leaves an extracted copy as "build/cross-compiler-$ARCH". 
> > You can put them wherever you like, it's not my problem.  They're fully
> > relocatable.
>
> Toolchains built with crosstool-NG are also fully relocatable.
> Having the user tell before hand where to install the stuff is also
> another good option.

So why were you talking about where to install it being a big deal earlier?

I'm confused.

> >   [*] Remove documentation (NEW)
> > Nice, and possibly the first question someone who _isn't_ a cross
> > compiler toolchain developer (but just wants to build and use the thing)
> > might actually be interested in.
> >
> :-)
> :
> > Your ./configure still requires you to install makeinfo no matter what
> > this is set to.  You have to install the package so this can delete its
> > output?
>
> Unfortunately, gcc/glibc/... build and install their documentation by
> default. I haven't seen any ./configure option that would prevent them
> from doing so... :-(

You already mentioned the breakage in some version of some package that didn't 
have makeinfo.

> > Wouldn't it be better to group this with a "strip the resulting binaries"
> > option, and any other space saving switches?  (I'm just assuming you
> > _have_ them, somewhere...)
>
> Nope. But that's a good idea. :-)
>
> >   [*] Render the toolchain read-only (NEW)
> > This is something the end user can do fairly easily for themselves, and
> > I'm not quite sure what the advantage of doing it is supposed to be
> > anyway.  In any case it's an install option, and should probably go with
> > other install options, but I personally wouldn't have bothered having
> > this option at all.
>
> In fact, it's here and ON by default, and why this is so is clearly
> explained both in docs/overview.txt and in the help of this option. Well,
> that might not be so obvious, after all. :-(

The complete help entry in 1.3.2:

  â CT_INSTALL_DIR_RO:
  â
  â Render the directory of the toolchain (and its sub-directories)
  â read-only.
  â
  â Usefull for toolchains destined for production.

Presumably the "so clearly explained" went into 1.3.3 or svn.  (If you install 
it as root in /usr or some such, normal users shouldn't be able to write to 
it because they don't own it.  That would be "production", no?)

Is some portion of the toolchain by default world writeable?

The overview file suggests the user will accidentally install stuff into it 
otherwise, apparently because users are accident prone like that.

I've known to intentionally install stuff like zlib headers and libraries into 
their cross compiler because they built and installed zlib into their target 
root filesystem, and they then wanted to cross compile more stuff that used 
zlib, and needed to link against its headers and libraries at compile time.  
But they did this quite intentionally...

> >   (10) connection timeout (NEW)
> > This is an implementation detail.  Users should hardly ever care.
>
> No. I have a case were the network is sooo slow that connections are
> established well after the 10s default timeout (17s if I remember well).

That's why I used a default of 20, with one retry.  Still tolerable for 
waiting humans, but enough of a gap that systems that don't respond in time 
usually are down.

> > As a higher level design issue, It would have been easier for me to
> > implement my build system in python than in bash, but the point of doing
> > it in bash is it's the exact same set of commands you'd run on the
> > command line, in the order you'd run them, to do this yourself by hand. 
> > So to an extent the shell scripts act as documentation and a tutorial on
> > how to build cross compilers. (And I added a lot of #comments to help out
> > there, because I _expect_ people to read the scripts if they care about
> > much more than just grabbing prebuilt binary tarballs and using them to
> > cross compile stuff.)
>
> That paragraph also applies to crosstool-NG. Word for word, except for
> the python stuff, that I don't grok.

So my earlier point about not needing to expose sufficiently obscure config 
entries via menuconfig when they can just tweak the shell scripts should 
apply to crosstool-ng as well?

> >   [ ] Stop after downloading tarballs (NEW)
> > This seems like it should be a command line option.
>
> Granted, same answer as for "Force downloads"
>
> >   [ ] Force extractions (NEW)
> > Ah, you cache the results of tarball extraction too.  I hadn't noticed. 
> > (I hadn't bothered to mention that mine's doing it because it's just an
> > implementation detail.)
> >
> > This is one of the things my setupfor function does: it extracts source
> > into build/sources, in a subdirectory with the same name as the package.
>
> Unfortunately not all package are good boys. Some have a hyphen in the
> package name and a dash in the corresponding directory.

Yup, I noticed that.  Hence the "extract into empty directory and mv *" trick.

> > Again, I detect "good/stale" cached data via sha1sums.
>
> I'm missing this. It's on my TODO as well, but low priority...

It bit me personally often enough that I did something about it.

I did already write generic code to do this, you know.  Available under GPLv2 
if any of it's of interest to you.  (Yes, I did read the whole of LICENSES.)

> >   (1) Number of parallel jobs (NEW)
> > My sources/includes.sh autodetects the number of processors and sets
> > CPUS. You can override it by setting CPUS on the command line.  (I often
> > do "CPUS=1 ./build.sh x86_64" when something breaks so I get more
> > understandable error messages.)
> >
> > In general, I try never to ask the user for information I can autodetect
> > sane defaults for, I just let them override the defaults if they want to.
>
> At work, we have a build farm whose purpose is to build the firmwares for
> our targets. The machines are quad-CPUs. Deploying a new toolchain needs
> not be done in the snap of fingers, and building the firmwares have the
> priority. So I use that to restrain the number of jobs to run in // .

I usually run the builds as "nice 20" when they're not a priority, but the 
ability to easily override the autodetected CPU value is also there.  
Autodetecting a default doesn't mean you can't specify...

> >   (0) Maximum allowed load (NEW)
> > Ooh, that's nice, and something mine doesn't have.
>
> Yeah! One good point! :-)
>
> > Personally I've never had
> > a clear enough idea of what loadavg's units were to figure out how it
> > equated to slowing down my desktop, and I've actually found that my
> > laptop's interactivity going down the drain is almost never due to
> > loadavg, it's due to running out of memory and the thing going swap happy
> > with the disk pegged as constantaly active.  (The CPU scheduler is way
> > the heck better than the I/O scheduler, and virtual memory is
> > conceptually horrible and quite possibly _never_ going to be properly
> > fixed at the theoretical level.  You have to accurately predict the
> > future in order to do it right, that's slightly _worse_ than solving the
> > halting problem...)
> >
> >   (0) Nice level (NEW)
> > Again, trivial to do from the command line:
>
> I already have # of // jobs, and loadavg. Why not having nice as well?...

See, when that kind of question comes up, I go the other way. :)

(I've had many evenings devoted to "This is too complicated, how can it be 
simplified?  Can I automate something away, or avoid needing it entirely, or 
group things and have something generic pop out, or...?"  You've seen the 
kind of projects this attracts me to...)

> >   [*] Use -pipe (NEW)
> > Why would you ever bother the user with this?  It's a gcc implementation
> > detail, and these days with a modern 2.6 kernel dentry and page caches
> > you probably can't even tell the difference in benchmarks because the
> > data never actually hits the disk anyway.
> > Have you actually benchmarked the difference?
>
> 10% gain on my machine (Dual AMD64, 1GiB RAM) at the time of testing.
> Much less now I have a Quad-core with 4GiB RAM.

Cool.  (And possibly a good thing to put in the help text.)

So what's the downside of just having it on all the time?

> >   [ ] Use 'ash' as CONFIG_SHELL (NEW)
> > A) I haven't got /bin/ash installed.  Presumablly you need to install it
> > since the help says it's calling it from an absolute path?
>
> crosstool-NG does not build and install it. If the user wants that,
> he/she's responsible for installing it, yes. Maybe I should build my own...
>
> > B) If your scripts are so slow that you need a faster shell to run them,
> > possibly the problem is with the scripts rather than with the shell?
>
> My scripts are not so slow. ./configure scripts and Makefiles are.
> Again, using dash, the build went 10%/15% faster on my quad-core.

Configure is enormously slow, yes.  Is there a downside to autodetecting 
that /bin/ash or /bin/dash is installed and using it if it is?

Also, why is this a yes/no instead of letting them enter a path to 
CONFIG_SHELL (so they can use /bin/dash or /bin/ash if they want)?

> > I admit that one of the potential weaknesses of my current system is that
> > it calls #!/bin/bash instead of #!/bin/sh.  I agonized over that one a
> > bit.  But I stayed with bash because A) dash is seriously broken, B) bash
> > has been the default shell of Linux since before the 0.0.1 release.
>
> I do explicitly call bash as well, because I use bashisms in my scripts.
> ./configure and Makefiles should be POSIX compliant. I am not.
>
> >   Maximum log level to see: (INFO)  --->
> > I don't have a decent idea of what you get with each of these.  (Yes,
> > I've read the help.)
>
> OK, that may require a litle bit more explanations in the help.
> I don't care about the components build log. I just want to know whether
> the build was successful, or failed. In crosstool-NG the messages are
> sorted in order of importance:
> ERROR   : crosstool-NG detected an error: failed download/extract/patch/...
>           incorrect settings, internal error... ERRORs are fatal
> WARNING : non-fatal condition that crosstool-NG knows how to work around,
>           but if you better give it correct input, rather than letting it
>           guess.

Typo, "but it's better if you give it".  (If you're going to use that as new 
help text...)

> INFO    : informs the user of the overall process going on. Very terse.
>           Tells the current high-level step being done: doanloading,
>           extracting, building a component...
> EXTRA   : informs the user with a finer level of what's going on. For
>           each steps listed above, prints sub-sequences: package being
>           downloaded/exxtracted/patched, sub-step in building a component:
>           ./configure-ing, make-ing, installing...
> DEBUG   : messages aimed at debugging crosstool-NG's behavior.
> ALL     : print everything: ./configure output, make output...

Is this a simple increasing scale of detail like the kernel's log levels, or 
does "DEBUG" output things that "ALL" doesn't?

>
> >   [*] Progress bar (NEW)
> > I have the "dotprogress" function I use for extracting tarballs, prints a
> > period every 25 lines of input.
>
> Mine rotates the bar every tenlines. Which is the better? ;-)

Possibly yours, I just didn't care that much. :)

> > I used to change the color of the output so you could see at a glance
> > what stage it was, but people complained
>
> Right. crosstool-NG's output used to be colored, but that just plainly
> sucks. And, believe it or not, some people are still using
> non-color-capable terminals...

Somebody actually asked me how to get it to stop changing the title bar (even 
though bash changes the title bar right back when your prompt comes up).  I 
showed them what line to comment out.  This would be an example of me not 
adding a config entry even though one real person did want to configure 
it. :)

I'd like to clarify: these granularity decisions are HARD.  Yes I'm 
questioning all your menuconfig entries, because I question everything.  
Several of the things you're doing you've already come up with an excellent 
reason for, it just wasn't obvious from the menuconfig help or a really quick 
scan of the source code.

> >   [*] Log to a file (NEW)
> > Again, "./build.sh 2>&1 | tee out.txt".  Pretty much programming 101
> > these days, if you haven't learned that for building all the other source
> > packages out there, cross compiling probably isn't something you're ready
> > for.
>
> Don't believe that. I have seen many a newbie asked to build embedded stuff
> on an exotic board that barely had support for in Linux, said newby just
> getting out of school, totaly amazed at the fact that you could actually
> run something else than Windows on a PC, let alone that something else
> than PCs even existed...
>
> (Well that newbie has grown since, and is now quite a capable Linux guru).

And I'm happy to explain to them about shell redirection if they ask. :)

There are many, many people who have asked me questions about system building 
and got pointed at Linux From Scratch or 
http://tldp.org/HOWTO/Bootdisk-HOWTO/ (which is alas getting a bit long in 
the tooth now but still a pragmatic introduction to the basics) or 
http://tldp.org/HOWTO/From-PowerUp-To-Bash-Prompt-HOWTO.html

In this case,
http://www.gnu.org/software/bash/manual/bashref.html
and
http://www.opengroup.org/onlinepubs/9699919799/idx/shell.html

Might be good references to foist 'em off on... :)

Yet another tangent, of course.

> > And I'm at the end of this menu, so I'll pause here for now.  (And you
> > were apologizing for writing a long message... :)
>
> I knew you would surpass me in this respect! ;-P
>
> ( Woohoo! I caught up with your mails! :-) )

Not anymore!

Rob
-- 
GPLv3 is to GPLv2 what Attack of the Clones is to The Empire Strikes Back.

--
For unsubscribe information see http://sourceware.org/lists.html#faq