Re: Final(?) patch to update libtool in GCC and src trees

[Charles Wilson <cwilson at fastmail dot fm>]

Charles Wilson wrote:
> Steve Ellcey wrote:
> > >   I appreciate the problem, but Cygwin can be a bit 'special' when it 
> > > comes to
> > > libtool; if it hasn't been done yet, can you possibly bear to hold 
> > > off a bit
> > > while I give it some hasty testing?  (By 'a bit', I mean no more than 
> > > say
> > > 'overnight').  Sorry to be a pain :-/
> >
> > I can wait, especially since no one has given me the go ahead yet.  :-)
> > I know of no cygwin testing that has been done.
>
> I'll rip that failed experiment out of my tree and try again with just 
> Steve's patches; however I'd imagine Dave Korn will have results before 
> I do.

Well, I ran into some trouble, related to the new libtool.  I'm building 
in a combined tree, with binutils and gcc, but not winsup or newlib. 
(configuration args at end of post).  All of the binutils stuff built 
fine (non-shared, natch).  The error is this, when I get into building 
libgcc (stage1):

configure: error: cannot compute suffix of object files: cannot compile
See `config.log' for more details.
make[2]: *** [configure-stage1-target-libgcc] Error 1
make[1]: *** [stage1-bubble] Error 2
make: *** [bootstrap] Error 2

In _build/i686-pc-cygwin/libgcc, config.log shows:

configure:2587: /usr/local/src/gcc/_build/./gcc/xgcc 
-B/usr/local/src/gcc/_build
/./gcc/ -B/opt/i686-pc-cygwin/bin/ -B/opt/i686-pc-cygwin/lib/ -isystem 
/opt/i686
-pc-cygwin/include -isystem /opt/i686-pc-cygwin/sys-include 
-L/usr/local/src/gcc
/_build/./ld -c -O2 -g -O2   conftest.c >&5
/bin/sh: /usr/local/src/gcc/_build/./gcc/as: No such file or directory

Taking a look in _build/./gcc, ls -l (for selected files) shows:

-rwxr-xr-x 2 cwilson None     5632 Apr 11 03:57 as.exe*
-rwxr-xr-x 2 cwilson None     5632 Apr 11 03:58 collect-ld.exe*
-rwxr-xr-x 2 cwilson None     5632 Apr 11 03:55 nm.exe*

These .exe files are all hardlinks (I'm using cygwin on NTFS, where 
hardlinks work) to *wrapper* executables (here's the relevant section 
from stdout during the build):

creating as
ln ../gas/as-new.exe as.exe
echo timestamp > stamp-as
creating collect-ld
ln ../ld/ld-new.exe collect-ld.exe
echo timestamp > stamp-collect-ld
creating nm
ln ../binutils/nm-new.exe nm.exe
echo timestamp > stamp-nm

This is wrong.  libtool-ToT (and libtool-1.5) on cygwin and mingw, when 
shared libraries are "in the mix", use the following scheme:

(1) a wrapper script (./foo) is used in the "normal" libtool way to set 
$PATH so that just-built .dll's can be located by the wrapped 
executable.  It will invoke the actual executable (./.libs/foo.exe)

(2) a wrapper executable (./foo.exe) whose sole purpose is to ensure 
that 'make' is happy.  New automake-derived Makefile.in's (and thus, 
Makefile's) have program targets like 'foo$(EXEEXT):'.  If only 'foo' 
exists, then 'make' thinks that the target wasn't built, and continually 
relinks executables...over and over and over.  So, we arrange that 
libtool produces a wrapper executable in . with the correct name, so 
that 'make' will know that the target has been satisfied.  The wrapper 
executable then runs the wrapper script via 'execv("$SHELL",newargz);' 
where newargz is a copy of all args to the wrapper script, with [0] set 
to the path ${thisdir}/foo  (e.g. the wrapper /script/).

(3) ./.libs/foo.exe is the "actual" executable.

Thus, on mingw/cygwin, the "top-level" wrapper is ./foo.exe which calls 
./foo which calls ./.libs/foo.exe

On other platforms, the "top-level" wrapper is ./foo (a script), which 
calls .libs/foo (an executable).

But there's still a problem (and this may affect other platforms; 
*should* do so, I think -- so I'm confused by Steve's success here; 
Steve: are you building the src/ stuff separately from the gcc stuff, or 
are you using a combined tree like I am?  I think the issue I am seeing 
will affect all platforms, but only if doing a combined build AND when 
--disable-shared is NOT explicitly passed during configure).

Current code in gcc/Makefile.in is

stamp-as: $(ORIGINAL_AS_FOR_TARGET)
        @echo creating as; \
        case "$(ORIGINAL_AS_FOR_TARGET)" in \
          ./as) ;; \
          ../*) \
             rm -f as$(exeext); \
             echo $(LN) $< as$(exeext); \
             $(LN) $< as$(exeext) || cp $< as$(exeext) ;; \
          *) \
             rm -f as; \
             echo '#!$(SHELL)' > as; \
             echo 'exec $(ORIGINAL_AS_FOR_TARGET) "$$@"' >> as ; \
             chmod +x as ;; \
        esac
        echo timestamp > $@

The source code for the wrapper executable (if you're curious, it is 
embedded in ltmain.sh), says:

  This wrapper executable should never be moved out of the build
  directory. If it is, it will not operate correctly.

Similarly, wrapper scripts say the following:

# This wrapper script should never be moved out of the build directory.
# If it is, it will not operate correctly.

They can be exec'ed via a relative PATH, but you can't just copy (or 
hardlink) the wrapper to a different directory, and invoke that 
copy/hardlink -- because it will then look in the wrong place for 
.libs/<actual-executable> (non-cygwin) or ./<wrapper-script> 
(cygwin/mingw).  You *can* create a symlink to the wrapper /script/, and 
it will work properly (it walks its own symlink path back to the "real" 
script, before computing the directory in which to look for the target 
executable).  However, the wrapper /executable/ does not have this 
ability -- so symlinking to the wrapper executable is a no-go.

However, we can't use symlinks anyway, because (according to 
gcc/Makefile.in):

# We need hard links so that directories can be shuffled
# during toplevel bootstrap.


So, I'm not really sure how this worked, even on "normal" 
platforms...unless Steve wasn't doing a top-level (combined-tree) bootstrap.


=======

It's a little unorthodox (and assumes a lot about libtool's 
implementation) but we COULD:

(1) hardlink/copy ${ORIGINAL_AS_FOR_TARGET--without-$exeext} .
(2) hardlink/copy ${ORIGINAL_AS_FOR_TARGET--with_$exeext} .
(3) mkdir .libs
(4) hardlink/copy ${ORIGINAL_AS_FOR_TARGET--with-/.libs/-inserted} .libs/

all with appropriate existence checks. And since (e.g) ld-new gets 
renamed to collect-ld, you'd have to rename (both) wrappers, similarly 
-- but NOT rename the .libs/ inhabitant.  (This is because the 
executable wrapper inspects argv[0] to figure out how to invoke the 
script wrapper.  The script wrapper doesn't care what its own name is, 
it just "knows" what its target in .libs/ *should* be named.)

But what a mess.  I've attached a preliminary patch for gcc/Makefile.in. 
It works for native/cygwin -- that is, it gets me past THAT problem -- 
but some more testing for other platforms and crosses is definitely in 
order...

Obviously this patch depends on Steve's new libtool stuff.

2007-04-11  Charles Wilson  <...>

	* gcc/Makefile.in [stamp-as]: handle libtool wrappers when
	copying binutils applications during combined-tree build.
	[stamp-collect-ld]: ditto
	[stamp-nm]: ditto

--
Chuck


configure options:
${RELSRCDIR}/configure \
  --prefix=/opt \
  --exec-prefix=/opt \
  --sysconfdir=/opt/etc \
  --libdir=/opt/lib \
  --libexecdir=/opt/lib \
  --mandir=/opt/share/man \
  --infodir=/opt/share/info \
  --with-datarootdir=/opt/share \
  --enable-languages=c,c++,objc,fortran \
  --with-gcc \
  --enable-nls \
  --without-included-gettext \
  --enable-version-specific-runtime-libs \
  --without-x \
  --with-system-zlib \
  --enable-threads=posix \
  --disable-win32-registry \
  --disable-sjlj-exceptions \ [*]
  --enable-libstdcxx-debug \
  --enable-cxx-flags='-fno-function-sections -fno-data-sections' \
  --enable-libgomp \
  --with-arch=i486 \
  --with-tune=i686 \
  --disable-werror

[*] oh yeah, I've also got Danny Smith's dwarf2 exceptions patch 
(20061203) in my tree.

And that's my next roadblock: the dwarf2 patch adds crtbegin.o/crtend.o 
to cygwin's build, and I get:
../../combined_src/gcc/crtstuff.c:445:2: error: #error "What are you 
doing with crtstuff.c, then?"

Sigh.

Index: Makefile.in
===================================================================
--- Makefile.in	(revision 123710)
+++ Makefile.in	(working copy)
@@ -1478,8 +1478,31 @@
 	  ./as) ;; \
 	  ../*) \
 	     rm -f as$(exeext); \
+	     if [ -e as ]; then \
+	       rm -f as ;\
+	     fi ;\
+	     if [ -e .libs/as$(exeext) ]; then \
+	       rm -f .libs/as$(exeext) ;\
+	     fi ;\
 	     echo $(LN) $< as$(exeext); \
-	     $(LN) $< as$(exeext) || cp $< as$(exeext) ;; \
+	     $(LN) $< as$(exeext) || cp $< as$(exeext) ;\
+	     if [ -n "$(exeext)" ]; then \
+	       if [ -e $(<:%$(exeext)=%) ]; then \
+	         echo $(LN) $(<:%$(exeext)=%) as ;\
+	         $(LN) $(<:%$(exeext)=%) as ||\
+	           cp $(<:%$(exeext)=%) as ;\
+	       fi ;\
+	     fi ;\
+	     if [ -e $(<D)/.libs ]; then \
+	       if [ ! -e .libs ]; then \
+	         mkdir .libs ;\
+	       fi ;\
+	       if [ -e $(<D)/.libs/$(<F) ]; then \
+	         echo $(LN) $(<D)/.libs/$(<F) .libs/$(<F);\
+	         $(LN) $(<D)/.libs/$(<F) .libs/$(<F) ||\
+	           cp $(<D)/.libs/$(<F) .libs/$(<F) ;\
+	       fi ;\
+	     fi ;; \
 	  *) \
 	     rm -f as; \
 	     echo '#!$(SHELL)' > as; \
@@ -1494,8 +1517,31 @@
 	  ./collect-ld) ;; \
 	  ../*) \
 	     rm -f collect-ld$(exeext); \
+	     if [ -e collect-ld ]; then \
+	       rm -f collect-ld ;\
+	     fi ;\
+	     if [ -e .libs/collect-ld$(exeext) ]; then \
+	       rm -f .libs/collect-ld$(exeext) ;\
+	     fi ;\
 	     echo $(LN) $< collect-ld$(exeext); \
-	     $(LN) $< collect-ld$(exeext) || cp $< collect-ld$(exeext) ;; \
+	     $(LN) $< collect-ld$(exeext) || cp $< collect-ld$(exeext) ;\
+	     if [ -n "$(exeext)" ]; then \
+	       if [ -e $(<:%$(exeext)=%) ]; then \
+	         echo $(LN) $(<:%$(exeext)=%) collect-ld ;\
+	         $(LN) $(<:%$(exeext)=%) collect-ld ||\
+	           cp $(<:%$(exeext)=%) collect-ld ;\
+	       fi ;\
+	     fi ;\
+	     if [ -e $(<D)/.libs ]; then \
+	       if [ ! -e .libs ]; then \
+	         mkdir .libs ;\
+	       fi ;\
+	       if [ -e $(<D)/.libs/$(<F) ]; then \
+	         echo $(LN) $(<D)/.libs/$(<F) .libs/$(<F);\
+	         $(LN) $(<D)/.libs/$(<F) .libs/$(<F) ||\
+	           cp $(<D)/.libs/$(<F) .libs/$(<F) ;\
+	       fi ;\
+	     fi ;; \
 	  *) \
 	     rm -f collect-ld$(exeext); \
 	     echo '#!$(SHELL)' > collect-ld; \
@@ -1510,8 +1556,31 @@
 	  ./nm) ;; \
 	  ../*) \
 	     rm -f nm$(exeext); \
+	     if [ -e nm ]; then \
+	       rm -f nm ;\
+	     fi ;\
+	     if [ -e .libs/nm$(exeext) ]; then \
+	       rm -f .libs/nm$(exeext) ;\
+	     fi ;\
 	     echo $(LN) $< nm$(exeext); \
-	     $(LN) $< nm$(exeext) || cp $< nm$(exeext) ;; \
+	     $(LN) $< nm$(exeext) || cp $< nm$(exeext) ;\
+	     if [ -n "$(exeext)" ]; then \
+	       if [ -e $(<:%$(exeext)=%) ]; then \
+	         echo $(LN) $(<:%$(exeext)=%) nm ;\
+	         $(LN) $(<:%$(exeext)=%) nm ||\
+	           cp $(<:%$(exeext)=%) nm ;\
+	       fi ;\
+	     fi ;\
+	     if [ -e $(<D)/.libs ]; then \
+	       if [ ! -e .libs ]; then \
+	         mkdir .libs ;\
+	       fi ;\
+	       if [ -e $(<D)/.libs/$(<F) ]; then \
+	         echo $(LN) $(<D)/.libs/$(<F) .libs/$(<F);\
+	         $(LN) $(<D)/.libs/$(<F) .libs/$(<F) ||\
+	           cp $(<D)/.libs/$(<F) .libs/$(<F) ;\
+	       fi ;\
+	     fi ;; \
 	  *) \
 	     rm -f nm$(exeext); \
 	     echo '#!$(SHELL)' > nm; \