This is the mail archive of the cygwin mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

RE: Cygwin build system SOOOO SLOOOWWWW ???


Steve,

we've been struggling with speeding up our build process, too, so I'd
like to share some ideas - yet I'm not so certain whether you'll learn
from me or I'll learn from you: Hopefully both.

You wrote on Wednesday, September 14, 2005 3:45 AM:

> Greetings To All,
> 
> We've been using Cygwin for many years to host our software build
> system on Win32.  The system is built on top of gmake.

Same here.

>  We use the
> same gmake system on MacOSX, Linux, and Solaris.  Building under
> Cygwin has always been slower than on true *nix platforms.

We don't even do that. (Unfortunately,) our products are all-Windoze.
We switched to gmake because no other make tool was able to handle
our configurations the way we want. (Don't even ask me about
Virual Studio.)

> In the last year or so, however, the Cygwin port has gotten slower
> and slower, to the point that building the PC version of our software
> is PAINFUL.

I haven't noticed that, at least not to such extremes.

>  We've been changing the system, so maybe it's due to
> something we've added, but the other platforms remain speedy.  This
> leads me to two flavors of questions for the group...

Let's see ...

> 1) How can I tell what Cygwin is doing?  Is there a tool that will
> tell me what tool is actually running at any given time?  Is there
> any way to tell what Cygwin is doing down in its guts?  Does anyone
> have any other suggestions as to how I might get to the
> bottom of this?

Below, I'll tell about some suspicions I have about what cygwin might
actually be doing. To your question, I can offer two Ideas:

- "top" or any Windoze Process Explorer more sophisticated than
  the task manager
- "strace" - though I haven't ever used it, but from what I know
  this will definitely give you an answer - maybe two much of it ;-)

> 2) Has anyone else experienced speed problems with Cygwin?  Has
> anybody else felt that Cygwin has gotten slower over the last
> year or so?  Are there any guidelines or "tricks" for getting
> Cygwin to run faster? 

a) Forking is more expensive in Windoze.
On Unix, especially in make environments, you'll often start new
processes as you're going - and often you'll not even notice. Google
for "bash tricks" on how to fork less often.
Hint: Don't use "sed" in `backticks` just for simple string
replacements.
Much of this can be done in make or bash directly.
Look at the changes you made - maybe you thought it's more elegant?

b) This is especially true for shells.
I'm not really sure on when and where this hits, but under certain
circumstances, bash needs to parse /etc/passwd when it starts. Do
you create /etc/passwd from an LDAP directory using mkpasswd?
Maybe you have hired some more people last year and it got longer?
Hint: Try whether it makes a difference if you replace /etc/passwd
with one that contains only the local users (look at the options for
mkpasswd).

c) /bin/sh is now bash, which is now dynamically linked.
Up until a few months ago, /bin/sh has been "ash", a smaller, but
less powerfull shell. This has been replaced by bash, to reduce the
traffic of repeated questions along the lines of "why does my shell
act different than on linux" (where /bin/sh is bash on most
distributions).
If I understood the traffic on this list correctly, bash is now
dynamically
linked, which might have an impact on starting it - I can't tell.
Hint: Don't start bash so often. Create fewer processes, but if you
must, see if you gain by using ash explicitely instead of bash.

To the gurus - is the following correct?
`echo blub` starts one process, `echo blub | sed -e 's/b/x/g'`
starts three: "echo", "sed", and "bash" to implement the pipe.

d) Beware of lazy evaluation.
Look at this construct:
CFLAGS=$(shell find . -type d -name include)
Read "info make" on setting variables and find out about the
difference between "=" and ":=". The above will run the find
again for every single call to the compiler. Along with the
issues about forking and reading directories and small files,
this can make a difference of *ages*.
Hint: See whether you can use less variables, use ":=" more often,
etc. - and don't use "$(shell ...)" anyway, as stated in a).
Rather, pre-compute makefiles with all the data hardcoded, using
":=".

e) Reading lots of small files seems more expensive on Windoze.
I don't know about your Makefiles, but traditionally, makefiles are
spread across project directories (for build hierarchies), and
makedepend
creates even more of that. For one of our applications, I roughly
calculated that make needs to open, read, and parse well over a
thousand files (not counting the source or objects or any such thing,
just the makefiles), just for telling you that all the targets are
up to date.
Hint: Phew ...

You see, for our configurations, running make to tell me that *nothing*
has changed could take up to half an hour. Therefore we introduced some
magic using Python to generate and split up makefiles two years ago,
and were down below five minutes again.

This is nothing compared to the link time of well over 15 minutes, so
we started to convert to DLLs for development (released applications
are still supposed to be linked statically, as they only run on
dedicated machines). We're currently trying to replace the whole build
chain by
a single daemon written in a decent language - hoping (i) that we need
only one process for the actual rule system etc., and will only start
additional processes for the compiler and linker; and hoping (ii) that
the actual rule set will be much easier to debug. (You know, developers
come to me and say "but I've only touched this little cpp and now
everything's getting compiled again and ..." - how do I know what
really happened?)

> Thanks in advance for any feedback that might help me speed up my
> builds. 

Let's see whether my hints are any good, but you're welcome anyway :-)

So long,
	Jan.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]