This is the mail archive of the
newlib@sourceware.org
mailing list for the newlib project.
Re: How do I point out OS includes? -- Or how to Build My Own SDK using Newlib
- From: Duane Ellis <duane at duaneellis dot com>
- To: Anders Lindgren <ali at df dot lth dot se>
- Cc: newlib at sourceware dot org, shap at eros-os dot com
- Date: Fri, 13 Apr 2007 10:25:58 -0400
- Subject: Re: How do I point out OS includes? -- Or how to Build My Own SDK using Newlib
- References: <Pine.GSO.4.60.0704131150430.4637@igloo.df.lth.se>
Anders Lindgren wrote:
>> How do I point out OS includes?
Jonathan S. Shapiro wrote:
In any case, I would be interested in the answer to this as well...
anders>> The reason is that I don't know how to tell the gcc build where
my OS include files are.
You did not say _which_ OS you are using... I'll assume you have your
own Custom OS
(I see Jonathan was involved with Eros-Os..)
If _YOUR_ OS is linux, go seach for "crosstool", it is a great solution.
perhaps I should make a "Non-Linux" Cross tool solution.
anders>> the OS includes are typically not even in a fixed place -- it
depends entirely on where the user has the OS source checked out to.
The same goes for NEWLIB header files too..
Generally, just about everything in the world makes an assumption that
there is an official
installation directory. You cannot fight that, you have to live with it,
and roll with the punches.
You have hit one of the mine fields. You need a road map through the
mine field....
You need to control your user, it's not hard, see below.
I'll offer this bit of advice. (I have cross-built numerous operating
SDKs for numerous platforms for years upon years)
What I do to build a _complete_ sdk is described below, and this
addresses your questions
and describes how to integrate newlib into that environment.
It also probably describes a few more things you are about to come across
(I've been doing this for a while)
-- Duane Ellis.
===============================
0) You want to create a "tool kit build script" and try to get everyone
to use it.
It's possible....
1) Create a uniquely named directory, I name it based upon the current
date and time.
ie: /toolkit/sdk/BUILD_YYYYMMDD_HHMMSS
Some would call this stupid, or silly, but there is a very good
reason to do this, see below.
2) I create a shell environment variable
TOOLKIT=/toolkit/sdk/BUILD_YYYYMMDD_HHMMSS
Everything is based upon that variable, period.
Every script, as part of the SDK - makes use of ${TOOLKIT}
Every application Makefile - makes use of ${TOOLKIT} too.
If anybody hard codes something, shoot them.
Good developers understand it is bad to "hard code something"
EXAMPLES:
Take a look at Adam Dunkel's "CONTIKI" - operating system.
a key thing in everything is the CONTIKI variable.
Look at UBOOT, toplevel Makefile -
"BUILD_DIR" and "SRCTREE"
Look at "FREERTOS" -
You'll find: "RTOS_SOURCE_DIR"
and "DEMO_SOURCE_DIR"
The GCC folk, like the idea of "with-root" type ./configure options.
That's nice - if your target is cross building linux and has a file
system
Often mine is not.
3) When I ./configure and build tools, ie: binutils, gcc, and all of
those things
I specify the following "./configure" option
./configure --prefix=${TOOLKIT}/install
FYI, my sdk build script checks things out of CVS and SVN, and
places them
under the directory ${TOOLKIT}/build/<name>, hence I have a
self-contained build.
i *NEVER* and i mean *NEVER*EVER* install things into
/usr/local/<somename>
See below for reasons.
4) There is a build order one must follow, for example: binutils before
gcc, before, newlib, ... etc.
For my own things, example: My custom operating system - like in
your case.
This could also be various IP components - Audio codecs, Video
Libraries, hardware libraries etc
I specifically create a script to "install" the library into that
BUILD_YYYYMMDD_HHMMSS directory.
In my stuff, I create a "Makefile" that responds to:
TOOLKIT=/somedir
export TOOLKIT
cd ${TOOLKIT}/build/<modulename>
make sdk-build-install-headers
Sometimes, it is nothing more then a shell script that does this:
"cp -r ${fromdir} ${TOOLKIT}/some/directory"
5) All is not well with the above build order... I see you have hit
this problem.
(the "--disable this and --disable that options")
One must also deal with "chicken or the egg" problem, GCC is full
of mine fields
We Americans often use the term: "Catch-22" - it refers to a
funny movie name.
You need newlib built and installed to build gcc,
You need gcc to build and install newlib
You need your operating system to build newlib
there are many eggs, and many chickens....
As I said above, I learned a lesson long ago - I create a make
target like this:
make sdk-build-install-headers
My stuff does *NOT* require me to "./configure" anything, thus fewer
scrambled eggs.
Sigh, GCC has come a long way but not everything has...
The "crosstool" package is helpful for this, but it is some what
target=linux centric.
6) Often, one must do 2, sometimes 3, distinct GCC builds, generally in
this order:
(0) Assuming BINUTILS are in place
(1) Unpack, build and install gcc, with only --enable-languages=c
(2) Unpack, build, and install NEWLIB
(3) install your custom operating system header files - and
maybe libraries
(ie: Copy your newlib fixes over the top of the newlib
installed files)
(4) erase your "gcc-build" - unpack it again
(5) Build and install GCC, this time with --enable--languages=c,c++
(6) Build other things for your target - your operating system...
(7) Build GCC again, this time using the C++ features of your
new Operating System
7) Here's the problem, GCC _hardcodes_ the include path, and the
library path into it self when you build it.
GCC is not alone in this, other tools do the same (some variations
of GDB do this too)
It's not _wrong_ for GCC to do this, it's valid for 99.9% of GCC's
targets.
In my world, (cross-compiling) I want my targets version of
*stdio.h* - not /usr/include/stdio.h which
is for my linux host.... Often times, newbies trying to build
their own version of GCC hit this wall.
It's not just GCC, it's every other package on the face of the
planet. They all thing they are going to be
be installed in an offical place.
*** THE KEY HERE IS THIS ***
By installing everything in that *COMMON* place - GCC can now find
all of your header files.
So can every other package you wish to use.
8) NEXT - "-lc" (which is linking NEWLIB as the standard C library) -
does not fix everything.
I disable the newlib syscalls support,
(--disable-newlib-supplied-syscalls)
Later, after _MY_ os is built, which has its own syscalls, I have a
"fixit" script that:
a) Unpacks libc.a found in: ${TOOLKIT}/install/<somedirectory>/libc.a
b) Copies my operating system 'syscal files' (plus a few more) into
the same directory
c) Re-creates 'libc.a' and copies the new-and-improved "libc.a" into
the ${TOOLKIT}
yea, i could integrate my operating system syscalls into NEWLIB....
See below: One ring to bind them, one ring to find them....
9) Thankfully, most GNU tools understand the "--prefix" and honor it
very well, and keep them selfs out of
and do not mess up your "/usr/include" directories.
10) An alternative to the above - is this:
If you have 10 different libraries your product uses, you always
specify 10 different "-I" include paths.
On every single command line...
In your stated environment, ie:
"it depends entirely on where the user has the OS source
checked out to"
You may also need 10 different "-L" library search paths....
It becomes ugly, ugly, ugly.
Imagine if you had to deal with Windows network drive mappings too...
ie: Developer (A) has drive "S" -(source code drive) pointing
HERE...
Developer (B) uses "S" for the "shared drive"
It is alot like a windows machine: "Which drive letter is your CDROM?"
Well... That depends..... it could be D, or E, or something else....
Now go fix all of your build scripts to adapt to this new developers
drive letter mapping....
Please - Just shoot me - put me out of my misery ....
===============================================================
Ok time to answer some detractors statements that I know are going to
come up:
------------------
(A) The ONE TREE Build... Also known as the "cygnus build tree" -
It makes for a great movie - I'm sure you know the quote
One ring to rule them all, One ring to find them, One ring to
bring them all, and in the darkness bind them.
The general idea is, you can copy everything, GCC, GDB, NEWLIB,
BINUTILS, and the kitchen sink
under a single directory structure and build everything at once,
all of these problems go away.
Movies are the land of make believe.
I live in the real world.
If I do not ship product on a timely basis, I have no money.
IF - and it is a *HUGE* IF - I choose to 100% integrate my
operating system into Newlib
And 100% integrate it into GCC, and every other package - Yes, "One
ring will bind them"
If I was doing this once, I might go this route.
If I never needed to 're-integrate' _my_ solution into a new
version of GCC
If I never needed to 're-integrate' _my_ solution into a new
version of BINUTILS
If I never needed to 're-integrate' _my_ solution into a new
version of NEWLIB
If I never needed to 're-integrate' _my_ solution into a new
version of _something_else_
"And in the darkness bind them" -- In my world, the simpler light of
day is to hack it other ways.
------------------
(B) The directory /toolkit/sdk/BUILD_YYYYMMDD_HHMMSS is stupid, it
belongs in a standard place.
Or: You are install it it wrong, it should be put in:
"/usr/local/<SOMETHING>"
Or: No, it belongs in: /opt/<yourname>
I'll agree with something like: /opt/toolkit/BUILD_YYYYMMDD_HHMMSS
But Most importantly, HHMMSS (or an equal) is vitally important!
Reason #0
I have fixed a bug, only to fix it again in 20 minutes...
I call these "A fix a minute day..."
Nah... Never happens does it - We are all Professionals!
Ok, use HHMM, not HHMMSS... or YYMMDD_morning YYMMDD_afternoon
Reason #1
I could use "build numbers" - ie: Think RELEASE (version) NUMBERS
/opt/toolkit/BUILD_1234
/opt/toolkit/BUILD_1235
/opt/toolkit/BUILD_1236
I find that HUMANs understand YYYMMDD_HHMMSS better.
Developer: I found a bug...
Me: Which SDK are you using?
Developer: Version X.
Me: I fixed that Tuesday afternoon
Yes, I've lived in a very _fluid_ development environment.
BTW - learn to live and love *scorched*earth* builds..
They suck - they take time to build.
But they are totally reliable!
If done right, and scripted well -
people complain - but headaches like what you describe go away.
Reason #2
Have you ever needed *THREE* incompatible versions of a tool
installed at the same time?
1 - for that new experiment you are working on
2 - for product you are releasing today
3 - for that "quick fix (oops panic) revision" that is a year old
Actually, GCC supports this, see the -V and -B options.
But sadly, the 16 other components of my tool kit have different
ideas and thoughts *ONE*
or *TWO* of those are written by competent "autoconfigure" gurus,
most are not.
Often times, I have supported developers who always ask: "Why can't
I use Visual Studio..."
It's part of that "And in darkness bind them" problem, or "The
lessor of two evils..."
Solution:
bash$ export TOOLKIT=/opt/toolkit/BUILD_<oldversion>
bash$ ... enjoy ...
This is even more true on a _shared_ development server.
Allen - installs the new SDK
Bob - says "Thank you Allen, you just broke my project"
Charlie ... is clueless...
Reason #3 :-) QA and Release management people like this:
I can create a CVS tag across the entire SDK, and recover it at some
later point.
For example:
for $module in $ALL_SDK_MODULES
do
cvs rtag $module FOOBLASTER_2007_REV_3_00_RELEASE
done
If I setup my "sdk build script" right, then _I_ do not have to
recover the SDK
for the poor sucker who has to revise that 4 year old product....
I tell them to type build_sdk.sh CVS_REV=FOOBLASTER_2001_REV_2
I've been there, I've done that.
This is also helpful with full release management...
If done right, you should be able to - on an isolated machine
(a) Scorched Earth - Empty directory, only connected is to the
CVS server.
(b) Checkout and build the tool kit from CVS.
(c) Check out the product from CVS - and build it using that tool kit
(d) build your product release image and hand it to QA....
Repeat the above with a "-D <date>" or -r <REVISION> CVS option...
That is the Holy Grail.... and it starts with the way *YOU* make the
tool kit.
------------------
(C) All is not without problems... The SDK is not in your $PATH
People like having: "target-gcc" in your path,
For example installed in: /usr/local/bin
(1) In your Makefiles, you kind of have to do this:
PATH := ${TOOKIT}/install/bin:${PATH}
export PATH
(2) To simplifiy some developers lives, do this:
Create a SYM LINK, change as needed
/home/duane/my_sdk -> /toolkit/sdk/BUILD_YYYMMDD_HHMMSS
Then, export TOOLKIT=/home/yourname/my_sdk
In your ".bashrc" add /home/yourname/my_sdk/install/bin to
your PATH
But - shoot them, when they approach you with problems.
------------------
(D) Some will tell you that you can "edit GCC's SPECS" file to tell it
about your target.
Yea, "and in darkness bind them" -
Will every one who understands the GCC SPECS file syntax please
raise their hand so we can count you..
The short cut around it is this:
By creating your own _INSTALL_ directory (build_yyyymmdd_hhmmss)
You know exactly where GCC looks for its header files.
You know that NEWLIB will put its header files there too!
Same thing happens with the libraries NEWLIB and GCC create
SOLUTION:
Copy (install) your headers in the exact same place.
Copy (install) your library files in the same place.
Need to "hack" one of those files? (ie: Add your operating
system features)
Step 1: great, HACK it -
Step 2: As part of _your_install_ copy your hack over the
top of it
If you build a new SDK.. ie: YYMMDD(tomorrow)_HHMMSS then
install your stuff again, to the NEW sdk location.
------------------
(E) But - GCC cannot link my program.... ie: I want to do this:
Execute: target-gcc hello_world.c
Result: "a.out" for my target.
If you _really_ need this, you have to integrate it more with
GCC... Ugh... Sigh....
If not, you can do this:
1) Specify it all on the GCC command line - Let GCC link to ELF format
target-gcc -Wl,-T,my_linker_script --no-start-files ... O
files Lib Files .....
2) If needed, write a tool that reads the ELF and creates your
custom exe format.
I've done that before...
UBOOT - for example has a "mkimage" tool that does exactly that.
It's sort of another form of "objcopy" -
OR - (and in darkness bind them)
Update binutils (the linker) to create your own custom output
format.