Re: RFC: cygport cross-compiling APIs

[Charles Wilson <cygwin at cwilson dot fastmail dot fm>]

On 7/12/2010 6:06 PM, Yaakov (Cygwin/X) wrote:
> On Mon, 2010-07-12 at 14:41 -0400, Charles Wilson wrote:
>> Are we sure about this, given mingw64's express desire to avoid this 
>> $prefix -- specifically because "the other mingw" has historically used 
>> it, and "the other mingw" has had its preferences enshrined in the 
>> upstream gcc source code?
> 
> The (proposed?) Fedora mingw64 packages use /mingw as well.

Proposed by whom? Some member of the mingw64 project, or outsiders like
us?  Again, what do the actual #mingw64 folks say about this?

>> What about $prefix?  Is this handled automatically by cygport (e.g. 
>> always /usr, unless $host=mingw* then /mingw?) and can't be overridden?
> 
> Correct.
> 
>> I'd really like to be able to override $prefix in general, because it 
>> would make the gcc-tools-* packages a lot easier.
> 
> Are those still needed, with gcc now supporting recent autotools?

JonY just mentioned that he thought a specific *libtool* epoch might be
required -- but I think that's because he was trying to use autoreconf,
which IMO isn't (yet) appropriate for gcc/binutils...heck, it's not
appropriate for anything in the sourceware src/ repository.

gcc-4.6 development apparently still requires autoconf-2.64, while the
"normal" cygwin autoconf is at 2.65 (and should soon be moving to 2.66).
 gcc requires automake-1.11.1, and our "regular" automake1.11 is 1.11.1
so at least that matches.

But the autoconf difference means we still need gcc-tools-epoch2. And
gcc-tools-epoch1 will never go away, but since it's frozen anyway we
never need to rebuild IT.

>> As far as the package .cygport calling dodoc...I think we'll have to 
>> take that as it goes. The NOCROSS idea below might help here.
> 
> Right now I'm thinking that dodoc should ignore CROSS_HOST.

I don't really care either way about that one.  What about things
associated with $sysconfdir and $localstatedir?  (e.g. /etc and /var are
usually "outside" of $prefix).  For cross (clients), suppress
prep_etc_defaults, and assume $sysconfdir and $localstatedir are
underneath $prefix?

>> Well, there's the mingw-runtime stuff, which is still used/needed by 
>> gcc3's -mno-cygwin.
> 
> IIUC Corinna correctly, w32api would stay and we would add a
> mingw-w32api.
>
>> Plus there's the particular version of w32api that 
>> is used by cygwin's own gcc (and gcc3's -mno-cygwin) and is installed 
>> into /usr/include/w32api + /usr/lib/w32api.
> 
> More importantly, mingw-runtime would move from /usr/{include,lib}/mingw
> to /usr/i686-pc-cygwin/sysroot/mingw/{include,lib}, 

All of this agrees with my understanding.

> which would break
> gcc3 -mno-cygwin, unless we provide symlinks.  If we build a
> i686-pc-mingw32-gcc-4.5, can we just get rid of gcc3 -mno-cygwin?
> Pleeeease?

No argument from me.  There will be weeping, wailing, and gnashing of
teeth on the list, tho.

>> I assume this version would NOT be compiled using the i686-pc-mingw32 
>> toolchain, but instead by the cygwin one (e.g. its cygport -- if it is 
>> actually built USING cygport -- wouldn't inherit cross), so that's 
>> probably not an issue.
> 
> Right, the w32api going into /usr/{include,lib}/w32api would be built
> native.

Ack.

>>    the mingw-{zlib,bzip2,xz,libgcrypt,libgpg-error} stuff
>>       -- should be handled by the new cross cygclass with no problems
> 
> Is compiling setup.exe with mingw64-i686 an option?  If so, we'll need
> mingw64-i686 ports of those as well.  Either way, those should all be
> cygport-able.

Hm. A 64bit version of setup.exe; interesting. I don't mind providing
the additional packages for that; the list of setup's dependencies isn't
THAT long.

>>    the tools in the cygwin package, like strace.exe and cygcheck.exe
>>       -- no changes; handled by cygwin gcc with -nostdlib? and not
>>          built using cygport anyway.
>>
>> So...there aren't that many cross-compiled packages that are part of the 
>> cygwin distribution -- and few would use any additional cygclasses, 
> 
> No, I don't think we should be accepting every library *in the distro*
> compiled for every toolchain on the planet.  But by covering the basics
> properly, cygport should still be usable for that purpose by individual
> users.

Right.  Setup's dependencies, IMO, serve as a sufficient "proof of
concept" that the cross compiler toolchains work.

>> The best of all worlds would be to have a local override, that can be 
>> set per-command:
> 
> I have already thought of a couple of cases where it might make sense to
> install outside of sysroot, although that should still be a fringe case.

True. This is not the sort of thing you'd want to do automatically, but
the ability to override might be important in some cases. None spring to
mind right now, but I'm sure they exist.

> If we agree that do* should work under the sysroot, then whatever we
> decide for handling these can come later w/o breaking API.

OK.

>> This scheme can't address the issues related to OTHER inherited 
>> cygclasses which use the do* commands.  But then, we probably don't care 
>> about that very much.  .cygclasses are supposed to be black boxes that 
>> 'just work'(tm); their internal operation shouldn't be second guessed.
> 
> No, they shouldn't, otherwise it's impossible to keep any semblance of
> API stability.

That was my point.

>> Remember that in mingw64-speak, the "mingw64-crt-headers" is a 
>> completely separate package than "mingw64-crt-lib" -- there's no 
>> "top-level" configure typing them together, AFAIR; so it'd be rather 
>> difficult to munge them together into a single package named 
>> "mingw64-*-runtime".
> 
> The upstream "mingw-w64-snapshot" source packages actually contain both
> with a top-level configure.  The only reason to do one without the other
> is when bootstrapping; I see no need to artificially split them for the
> distro.

I wasn't aware of that "extra" level configury -- they've only had that
for a few months, and JonY's original packages didn't use it.  However,
"-snapshot" includes also mingw-w64-libraries (which we probably don't
want, and which is NOT to be confused with mingw-w64-crt which we DO
want).  And other similar issues, like -tools.

Looking at the configure.ac, it appears those "extra" items are NOT
built by default. So, sure, since mingw64 has so helpfully provided the
new top-level stuff, why not use it and avoid splitting -headers and -lib.

>> Or, for that matter, splitting mingw64-$CPU-gcc into separate languages. 
>> I *really* don't want to have to download Fortran, ObjC, (and perhaps 
>> Ada and java?) when all I really want is C++.
> 
> I would have no objections to splitting out gcc into core/g
> ++/objc/fortran subpackages (provided that the DLLs go together with
> their respective compiler), I'm just not bothering with that right
> now.

Well, since JonY already had the cygport machinery to do that, I'm sure
he'll be able to exploit the new cross-toolchain.cygclass in cygport(1),
and merge his cygport(5) and your cygport(5) stuff to handle it correctly.

> As for Ada and Java, JonY didn't enable those either. 

Oh, sure: I imagine Ada/Java would be slow in coming if ever. That's why
I put them in parens.

> Apparently Java
> is "extremely slow" with SJLJ, although I don't know if that means GIJ
> (which we don't need in a cross-compiler) or GCJ-compiled code.

IIUC, it's the gcj compiled code: Java is very exception heavy (well,
the standard java libraries like java.io, java.net, etc).  It's not just
the "throwing" of lots of exceptions, but under the hood every scope is
guarded with handling code to ensure the correct java context is created.

Since sjlj is slow -- it isn't zero-cost even when you're NOT actually
throwing any exceptions -- the whole system is slow.  With zero-cost dw2
exceptions...much better.

Or so I've been told.

>> Finally, you appear to be deliberately precluding any POSSIBILITY of 
>> providing a multilib gcc. I know cygport can't yet support it, but we 
>> might find it necessary or desirable down the road to add that support. 
> 
> I've heard multilib called a lot of things on #mingw-w64, but
> "necessary", "practical", or "stable" were not among them.

Well, if the mingw64 guys are actually opposed (or even, not strongly in
favor of) multilib, then who are we to argue?

> It might be
> nice for hand-compiling small stuff, but from the packaging perspective,
> once you have a separate i686 cross-compiler, it just doesn't make
> sense.  It's not as if most (any?) packages make it any easier to build
> multilib then building i686 and x86-64 separately.

That's certainly true; especially when you throw (current) libtool in
the mix.

>> Which is why I say, *defer* this until later -- don't try to solve these 
>> multilib problems right now.  But also, don't deliberately preclude 
>> various possible solutions, by package nomenclature whose that isn't 
>> expressive enough, or poorly chosen sys-root $prefix values.
> 
> On any native multilib system, you still have only one $prefix, period.
> You just have multiple lib* directories.  Same here.

Right. JonY was playing some games with a few symlinks, but I think that
might have been just experimentation.

>> If you do that, then mingw64-$CPU-$PKG would work fine (but might be a 
>> little confusing, since it could contain $OTHER-CPU stuff).
> 
> Exactly, why do they need to be separated at all?  If they're both meant
> to be used by one toolchain and installed in one sysroot, just package
> them together.

OK, I'm convinced.

> IOW mingw64-x64-zlib doesn't necessary mean just "a Win64 zlib" but "a
> zlib that can be used with mingw64-x64-binutils/gcc", compiled against
> whichever ABI(s) mingw64-x64 supports and installed in the mingw64-x64
> sysroot.  If the latter are multilib, then so can the former be without
> dicing up each package into a million pieces.

OK. (reserving agreement in the case of splitting DLL and EXE components
from the rest of individual cross-compiled client packages)

>> Right, well, I'll obviously handle the stuff I currently maintain: the 
>> mingw-{zlib,bzip2,xz,libgcrypt,libgpg-error} stuff. And I assume DaveK 
>> would do the mingw-{binutils,gcc} ones, since he's already volunteered 
>> for that (unless he's changed his mind?).
> 
> BTW, weren't you going to check mingw.org gcc for patches and options so
> I could cook up .cygport drafts for those as well?

See attached.  Short version: no patches to the source code, but some
odd build techniques.

One thing we probably don't care about: mingw.org uses a "forward.c" app
to emulate hardlinks (and to avoid MSYS's fallback of "make a separate
copy" for symlinks) of very large .exe's.

configure options:

  cd $builddir &&
  $srcdir/configure \
    --enable-languages=c,c++,ada,fortran,objc,obj-c++  \
    --disable-sjlj-exceptions    \
    --with-dwarf2                \
    --enable-shared              \
    --enable-libgomp             \
    --disable-win32-registry     \
    --enable-libstdcxx-debug     \
    --enable-version-specific-runtime-libs \
    --disable-werror \
    --build=mingw32  \
    --prefix=/mingw

Java is not yet ready, but is promised for later. Werror is disabled
because there are some unaviodable warnings in the Ada build.

Oddly, I would have expected --enable-fully-dynamic-string. I'm
surprised it is missing.

Technically, the mingw.org folks say the only cross compilers they
support are ones built using this tool:

http://sourceforge.net/projects/mingw/files/Cross-Hosted%20MinGW%20Build%20Tool/x86-mingw32-build-1.0.1-rc1/x86-mingw32-build-1.0.1-sh.tar.bz2/download

but...we probably would provide support for "our" cross compiler right
here, anyway. And we can probably coordinate "our" build closely enough
with the mingw.org folks that they would likely "bless" our version, too.

>> I just don't like deliberately hardcoding a possibly-subject-to-change 
>> policy discussion into a difficult-to-reverse nomenclature.
> 
> If we don't sub-split the multilib ABIs then there's no problem.

I'm convinced.

>> No, I disagree with this.  mingw-bzip2 provides, in toto, the DLL, 
>> headers, static/link libs, and utility apps.
>>
>> Nobody who ONLY wants to build setup.exe using the new cross compiler 
>> cares about the DLLs (because setup.exe uses -static) nor do they care 
>> about the apps.  OTOH, those elements ARE part of the cross compiled 
>> "sys-root" that, if someone were to 'repackage' for independent 
>> deployment selected contents of sysroot/$prefix/bin, they would need. 
>> Since we ship a mingw-bzip2 complex of packages, why should (a) the 
>> first group have to download and install everything, if they only need 
>> the headers and static libs but everything were bundled into one 
>> package, or (b) the second group have to recompile mingw-bzip2 
>> themselves if we strip the "unnecessary" parts from a single mingw-bzip2 
>> package?
> 
> Remember that I'm looking at *all* cross-compiling scenarios here.  On
> most other platforms, a -devel package absolutely requires its
> corresponding lib package, because the unversioned .so is just a symlink
> to the real lib; without the real lib, the symlink is worthless.  Even
> on PE platforms where we have import libs, the DLLs aren't required to
> link, but once you have linked with them it's nice to have them around
> already so you can use your new program.  Hence, I make my -devel
> packages depend on their libs on Cygwin as well.

OK, I'm with you so far.

> Since, from the distro perspective, the purpose of including
> cross-compiled libraries is so that you can build against them, not run
> them (since in most combinations execution is not possible).  IOW,
> unlike native packages, the "-devel" is of primary importance, and if
> once you install the -devel you need the lib as well, then you already
> have (almost) everything, so what point is there to splitting them?

For the people in my team, who have cygwin installed and I -- being the
fabulous and special fellow I am -- build some *mingw* tools using our
cross environment.  These may depend on liblzma DLL but probably also on
libgcc-1-sjlj.dll or maybe libstdc++-6-sjlj.dll.  I want to be able to
set up a setup.exe-compatible mirror site, with packages to install
these *tools* in /usr/$target/sys-root/$prefix/bin, along with the
necessary DLLs, so that my team can run them.

E.g. I want to have a package:

chucks-tools-VER-REL that has
requires: mingw64-i686-libstdc++6

Why is this verboten, when all it takes is a little cygport magic no
different that any of our other 500 native cygwin packages, or your
other 2000 cygwin ports packages?

> So yes, we install everything in one package.  Will there perhaps be a
> little extra in the package?  Possibly, but we shouldn't be talking
> about so many cross-compiled packages *in the distro* that it should
> make a real difference.

But...do we really want to make it so difficult for me to set up my
unofficial setup.exe repo, such that I'm tempted to create my OWN
mingw64-i686-libstdc++6 package *that conflicts with the official
mingw64-i686-gcc-g++ package*?

Let's not create conditions that cause even somebody like me to become
an evil 3PP.

> FWIW, the Fedora mingw64 packages don't have split -devel packages
> either.

Right, because there is no possibility (short of WINE) that you'd ever
try to distribute .exe's and .dll's for use within the cross compiler's
sys-root.

Again, this boils right back down to the fact that here, our Host
platform can simultaneously execute applications (and libraries) from
both the $host and the $target.

cygwin-hosted mingw*-target cross compilers are special, in a way that
other host/target combos are not.  Even if nobody else in the world
recognizes that, surely WE should?

>> The same argument holds with regards to the gcc language runtimes. 
>> Maybe somebody is interested not in cross compiling anything themselves, 
>> but using (from a private, or corporate) setup.exe repo some native 
>> win32 tools that are meant to be installed into a cygwin-hosted mingw 
>> sysroot.  Shouldn't they be able to get our libgcc_s-1-sjlj.dll without 
>> installing the entire cross compiler?
> 
> Ditto.  The purpose of a cross-gcc isn't to get the libs -- you would
> get those from the distro on which you're running the cross-compiled
> executable -- but to build against them.  The DLLs are just along for
> the ride.

Well, it does appear that we are at loggerheads.  You insist that
cygwin/mingw is EXACTLY the same as linux/mingw or solaris/irix or any
other combination; I say there is a significant difference between
cygwin/mingw and any other host/target combination, and that difference
makes it wise for use to make certain, minor, accommodations.

Like, for instance, simply splitting runtime DLLs -- even i686-mingw32
or x86_64-mingw32 ones -- into separate installable packages JUST LIKE
we do for native cygwin DLLs.  I simply can't believe you're digging
your heels in over two lines of cygport(5) code and an extra .hint file
-- simply for the sake of treating a cygwin-hosted mingw compiler
EXACTLY like a linux-hosted one.

cygwin != linux -- *especially* when it comes to runtime interactions
with win32.

Fortunately, I don't have to convince you of this; I only have to
convince JonY of the utility of splitting the language runtime DLLs into
separate packages.  That's purely the decision of the maintainer, and
how he writes the final cygport(5) and has little to do with the
cygport(1) script.

--
Chuck

Attachment: gcc-4.5.0-1-mingw.org-packaging.tar.bz2
Description: Binary data