[general] some ideas & request for comments (LONG)

Charles S. Wilson cwilson@ece.gatech.edu
Mon May 15 21:09:00 GMT 2000


Over the next several months I'm planning to rebuild many of the
packages that are offered on CygUtils so that they will fit into the new
sourceware distribution. At minimum, this means:

--prefix=/usr
--host=i686-pc-cygwin

But, as I go, I'm also trying to clean up the ports somewhat so that
they build more cleanly, without as much manual intervention. At best,
configure/make/make -install. At worst, patch/configure/[ edit result
:-( ]/make/make-install. In addition, I hope that all the library
packages (like zlib, libtiff, ncurses, etc) will build as a static
library and as a dll+import lib[* see below] in a single pass. The idea
is that dll's will be used by default; static libs provided as a
convenience only.

This leads to a couple of questions:

1) naming & location convention for libs/implibs/dll's.
2) location of these 'extra' packages on Cygnus' sourceware ftp site.
3) how setup.exe handles 'extra' packages
4) what should be included in the package tarballs (especially
-src.tar.gz) on Cygnus' sourceware ftp site. Was in the thread
[jorg.schaible@db.com: Setup and zlib, libpng] on cygwin-apps.

I think it's important that we address these issues soon, while
cygnus-apps is just getting starting, to establish the conventions and
precedents for future additions to the "official" distribution.

This message is very long, so hunt for a row of '====' to find the
beginning of each of the four topics.

I've got my asbestos suit zipped up, so flame ON!

=========================================
1) naming and location convention for libs/implibs/dll's

First I'll present a few options, and then my comments. I'll use zlib as
the example. I'll also assume that you need slightly different .h files
to use the static lib or the dll/importlib. This is not true in the case
of zlib, but other libraries may have restrictions like that, so let's
assume the worst.

a) /usr/lib/libz.a-static (static library)
   /usr/lib/libz.a-dll    (import library for dll)
   /usr/lib/libz.a -> libz.a-dll (symlink; dll by default)
   /usr/bin/libz.dll
   /usr/include/zlib.h        (headers for dll)
   /usr/include/zconf.h
   /usr/include/static/zlib.h (headers for static library)
   /usr/include/static/zconf.h

packages can link with dll OOB
linking with static lib requires -I/usr/include/static and changing the
symlink

b) /usr/lib/libz.a        (import library)
   /usr/lib/static/libz.a (static library)
   /usr/bin/libz.dll
   /usr/include/zlib.h, zconf.h
   /usr/include/static/zlib.h, zconf.h

packages can link with dll OOB
linking with static lib requires -I/usr/include/static and
-L/usr/lib/static, but no filesystem (renames or symlink) changes

c) /usr/lib/libz.a      (import library)
   /usr/lib/libz_s.a    (static library)
   /usr/bin/libz.dll
   /usr/include/zlib.h, zconf.h
   /usr/include/static/zlib.h, zconf.h

packages can link with dll OOB
linking with static lib requires -I/usr/include/static and changing the
target package's makefile -- replace '-lz' with '-lz_s'

d) /usr/lib/libz.a     (static library)
   /usr/lib/libz.dll -> /usr/bin/libz.dll
   /usr/bin/libz.dll
   /usr/include/zlib.h, zconf.h
   /usr/include/static/zlib.h, zconf.h

gcc/binutils can link directly to a dll without an import lib. (Try it
-- put the full pathname of a dll on the link line: 
   'gcc -o foo.exe foo.o /usr/local/bin/libz.dll'
and it works just fine. The only problem is binutils & gcc don't know to
search for "libz.dll" when they see '-lz' on the link line.

I've got a few very minor patches I'm submitting to the binutils and gcc
lists tonight that allow the following behavior:

gcc -o foo.exe foo.o -lbar 
  ---> will link foo.o with bar.dll or libbar.dll or libbar.a (in that
order of preference) without using an import lib (unless, of course,
neither bar.dll nor libbar.dll are found in the library search path, and
libbar.a *is* an import lib)

gcc -static -o foo.exe foo.o -lbar
  ---> ignores bar.dll & libbar.dll, will only link to libbar.a (this
doesn't *guarantee* a statically linked foo.exe, since libbar.a may be
an importlib)

So, assumming these patches get in (at least to the cygwin distro of
those tools) then where should libfoo.dll go? It needs to be in the
windows PATH so that the windows subsystem can do the runtime dynamic
linking. It needs to be in gcc's library search path so that gcc can
find it during the compile-time linking. If libfoo.dll goes in /usr/lib,
then you have to add /usr/lib to the PATH (actually, /lib since windows
doesn't grok the cygwin mount points); symlinks won't work because
windows doesn't understand them. OR, put libfoo.dll in /usr/bin ( ==
/bin which is already in the windows PATH), and symlink from
/usr/lib/libfoo.dll to /usr/bin/libfoo.dll ---> /usr/lib is already in
the gcc library search path, and cygwin-gcc groks symlinks.

Sometimes, the import library provides information that can't be
generated automatically on-the-fly by ld/gcc during the link step. For
instance, libcygwin.a renames some of the functions in cygwin1.dll. In
those rare cases where the import lib provides extra information, then
funky things can be done like:
  /usr/lib/libfoo.a (import lib)
  /usr/lib/libfoo_s.a (static lib)
  /usr/bin/libfoo.dll
  (NO libfoo.dll symlink in /usr/lib )

packages can link with dll OOB
packages can link with static lib, but it depends on whether you NEED to
use an explicit import library (I expect this to be very rare). 
  (1) implicit import lib: change the link command by adding '-static'
option, and put -I/usr/include/static on the compile line. No filesystem
changes are needed.
  (2) explicit import lib: change the link command by using '-lz_s'
instead of '-lz', and put -I/usr/include/static on the compile lines. No
filesystem changes are needed.

+++++++++++++++
The following drawback is shared by (a), (b), and (d): no granularity.
Your target program is either completely staticly linked (other than the
windows dlls and cygwin1.dll, of course), or totally dynamically linked.
With option (c) you can explicitly link libfoo (static), libbar
(dynamic) and libbaz (static): '-lfoo_s -lbar -lbaz_s'

I contend that most users want EITHER (a) smallest executable/easist
upgradeability == completely dynamic, or (b) completely selfcontained
executables == completely static. The third option will be very rare,
and nothing prevents a developer from renaming the static libraries on
her system and linking '-lfoo_s -lbar -lbaz_s' while building that one
package. If she know enough to want that granularity when linking, then
she know how to fiddle with filenames and link commands to get it done.

Assuming binutils/gcc accept my patches (or at least the maintainers of
the cygwin ports of those packages), then I vote for option (d).

=========================================
2) location of these 'extra' packages on Cygnus' sourceware ftp site.

Not everybody wants to download rpm or perl or even libz as part of the
initial setup of cygwin1.1.1. Yet, it would be nice to consolidate many
of the Cygwin ports to sourceware, so that folks can find them easier. 

Now, DJ & I've already stirred up the hornets' nest a bit by putting
zlib and libpng into the main 'latest' directory. By default, setup will
download that to all new users. To work around this problem, Chris
Faylor offered to create a 'cygutils' directory on sourceware for "my"
ports. However, I think that's too specific; perhaps 'extras' or
'add-ons' or something would be better. Wouldn't it be great if
Corinna's OpenSSL and OpenSSH ports were in there? So, let's call it the
'extras' directory for the purposes of this discussion.

Should 'extras' be a child of 'latest' or a sibling?

I think most folks would not see it, if it was a sibling, and as a child
of 'latest' it will be obvious to folks who are just browsing the
downloads. If 'latest/extras' contains only directories, and no
.tar.gz's, then setup.exe will by default ignore it. 

So, how about:

  /pub/cygwin/latest/extras
  /pub/cygwin/latest/extras/zlib/
  /pub/cygwin/latest/extras/OpenSSL/

etc.?

=========================================
3) how setup.exe handles 'extra' packages

I would like for folks to be *able* to use setup.exe to download/upgrade
these 'extras'. However, setup.exe will only access the 'latest'
directory, at present. Whether 'extras' is a child or sibling of
'latest', I propose a new option to setup.exe that tells setup to
download extra stuff.
   setup --extras (and setup -u --extras) ?
   setup --optional-packages ?
Each of this options would tell setup.exe to not only search within
'latest' for dirs ONE level down that contain .tar.gz's, but also to
look in the 'latest/extras' directory one MORE level down for additional
.tar.gz's.

Or perhaps 'setup --search-dirs='latest:latest/extras' is more flexible,
where '--search-dirs=latest' is the assumed default.

I propose these things, but I don't know enough about the innards of
setup.exe to actually add the functionality. However, do ya'll think
it's a good idea?

=========================================
4) what should be included in the package tarballs (especially
-src.tar.gz) on Cygnus' sourceware ftp site.

I see four options:
---------------
(1) the way I did it at first (sorry, guys...)

o zlib-<pkgversion>-<cygwinver>-src.tar.gz
      === identical to official zlib-<pkgversion>.tar.gz

o zlib-<pkgversion>-<cygwinver>-patch.tar.gz 
      === contains patches necessary for clean building on cygwin to
produce:

o zlib-<pkgversion>-<cygwinver>.tar.gz
      === the binary package

problems: user must download two packages to build; setup.exe unpacks
all the -patch.tar.gz's into the root directory automatically (Oops.)
Setup behavior is easy to fix, but... 

----------------
(2)

o zlib-<pkgversion>-<cygwinver>-src.tar.gz
      contains the same files as the official zlib-<pkgversion>.tar.gz,
but also contains the required patches and stuff for cygwin (user must
apply the patches, then build the package)

o zlib-<pkgverson>-<cygwinver>.tar.gz

headache for maintainers

----------------
(3)

o zlib-<pkgversion>-<cygwinver>-src.tar.gz
      contains the source code for zlib-<pkgversion>, but already
patched up to build cleanly with cygwin. Also contains the necessary
patches, so that the cygwin-specific changes can be reversed by 'patch
-R < cygwin-patch1' etc.

o zlib-<pkgversion>-<cygwinver>.tar.gz

headache for maintainers

-----------------
(4)

o zlib-<pkgversion>-<cygwinver>-src.tar.gz
      contains the source code for zlib-<pkgversion>, but already
patched up to build cleanly with cygwin. NO extra documentation or
patches.

o zlib-<pkgversion>-<cygwinver>.tar.gz

Could get ugly. Keeping the code in an evolving cygwin-apps CVS
repository can help, but this amounts to forking the code. Good? Bad?

------------------
In both (2) and (3), the patchset should probably reside in a uniform
directory in every package, whose name will probably not conflict with
pre-existing directories in any package. 'CYGWIN-PATCHES' ?

I vote for (3). It's clean, it allows our patches to eventually be
merged back into the main codebase if they're clean enough and don't
break other platforms. It's a little bit of a pain for maintainers, but
it's similar to what I've been doing with CygUtils for a long time now.
Finally, personal CVS repositories or sourceware-based CVS can make it
not quite so painful.

===========================================
The End. Whew.

--Chuck


More information about the Cygwin-apps mailing list