Re: Evolution .. Survival of the fittest ...

[Christos Siopis <siopis at umich dot edu>]

[ Apologies for the long posting; I hope it is relevant to this list ]

Manoj,

Thank you for posting this!

I would also like to express my gratitude to the coders of GSL, Python,
Numeric Python and other open source/free numeric software. I have not
written a single line to contribute to these projects, so some might say i
am not entitled to say what i am about to say, but i nevertheless will to
emphasize the user's perspective, and out of interest for the community.

I would like to "complain" about the large degree of "degeneracy" in open
source numeric projects: many projects overlap substantially, not (alas!)  
in terms of code base but in terms of intent. As an example, the Python
interface to GSL comes to mind, which (as a goal) overlaps with the
Numeric Python project. There are several other cases, of course.

The end result is that we have LOTS of programs doing all sorts of things,
but not a SINGLE (or a few) well-done application that can do ALL three
main things the numeric researcher needs today: "array"/vector
calculations, "professional"-grade graphics, and library support for a
large number of standard problems. Commercial programs such as IDL and
Matlab come to mind (which doesn't mean they are perfect or that we cannot
do better). Symbolic support would probably be nice, but i wouldn't touch
this before the other three components are done!

You may think (as Manoj put it) that GSL could function as the basis for
what Numeric Python tries to do. Well, i am not so sure! For instance: how
well could GSL work with Numeric array objects? How many or the array-
oriented operations and ufuncs of Numeric could be implemented in terms of
GSL functions? I suspect that GSL was "made for" serial computations, and
adapting it to the vector environment of Numeric would not be efficient,
except perhaps for some high-level operations.

Does this mean that GSL is redundant? Not at all. It intends to fill in a
free software gap in the niches where NAG, IMSL or the numerical recipes
would otherwise be used. But one can't help wondering if GSL could have
started in a way that other projects, such as Numeric, could benefit from
all the effort that went into it. I believe that one of the nice things
about GSL, compared to using some other free (beer) alternative like
Netlib, is the uniformity in the interface and, perhaps, the expectation
that a similar level of attention went to most functions, and that it is
better maintained than a Netlib program which was "dumped" by its author
some time ago without anyone taking care of it or its bugs any more. In
other words, there is a community behind GSL, which will do a better job
maintaining GSL as long as it does not split its attention among many
GSL-like projects!

Now take a look at Numeric Python: they also want to add much of the
functionality that GSL has or intends to have. They do it in a way which,
in my very humble opinion, is more "anarchic" than GSL, with people
writing their code from scratch or "adapting" code from Netlib and other
projects. There is less consistency in the API of these functions, not to
mention things like error reporting and propagation. Would it not be nice
to be able to use GSL as the foundation for Numeric Python?

Then we have the PyGSL project. Again, i thank the person who started it,
and i do think that there are some cases where it may be useful. But why
not use a full-fledged language like Python to take care of the syntactic
stuff and, instead, re-invent the wheel? Why not put the effort currently 
going into PyGSL to some better (for the numerical community) use?

Of course, there is no way, and probably no sense, in stopping people from
starting their own projects whenever they feel like it. In fact many do it 
not so much for the sake of the project as for the sake of e.g. learning a 
programming language or environment. Fine.

But we should also understand that the "amateur" numerical community's
resources are limited, perhaps more limited than the resources of the
software engineering community that created and maintains GNU/linux. This
may explain why, while that community has managed to create a full-fledged
operating system with all sorts of utilities, the scientific community,
after 7+ years of efforts, still does not have a "standard" platform that
includes production-quality vector computing, graphics and library
support.

I think it is important for our community to decide (somehow) on a minimum
number of "core projects" to work on; e.g., a science library like GSL, a
core plotting/graphics language (possibly based on one of many such
libraries floating around), and a hierarchy of array objects. Once the
basic stuff is there, different communities can then modularly build on
top of it, based on their respective needs (and again, Python is probably
the perfect language for that).

This would not just be a free (as in beer and as in speech) infrastructure
for the numerical scientist to use; it would also offer a much greater
level of transparency (no black boxes!) and consistency: when enough
people use e.g. its FFT transform module, the module will not only
eventually be bug-free, but its overall behaviour would be known and
understood. So whether i link to it through my C code or i use it through
Python's integrated environment, i will expect the same behavior. I will
not wonder, with every new package i use, whether its FFT transforms work
well, because they will all use that same FFT module. The more useful this
environment becomes, the more people will use it, the more people will
contribute to it, and the more useful it will become. This would be a
situation almost competely reverse of the current, fragmented landscape.

My programming skills are not good enough to contribute to the foundations
of this project. In fact, i do not know if the abilities of most numerical
scientists are sufficient. This may be an area where some government
agency should lay the seeds by hiring some professional team to get it
started. But once the overall foundation is there, there are things that i
(and most other numerical scientists) could contribute.

So i would like to ask: is there ANY chance at all that such a project
might one day become reality? Or will we have to forever use the IDL's and
Matlab's of this world?

***************************************************************
/  Christos Siopis              |                             \
/  Postdoctoral Research Fellow |    ''CARPE NOCTUM''         \
/  Department of Astronomy      |_____________________________\
/  University of Michigan       |                             \
/  Ann Arbor, MI 48109-1090     | E-mail : siopis@umich.edu   \
/  U.S.A.             __________|                             \
/                    / http://www.astro.lsa.umich.edu/~siopis \
***************************************************************


On Sat, 19 Oct 2002, Manoj Warrier wrote:

> Hi ppl ...
> 
> First of all a BIG thanks to the GSL team (I use it in my research).
> 
> I used to wonder why there exist so many programs that do allmost
> similar things? For example (octave, Scilab, RLAB, etc .... (MATLAB
> like). Then U have MuPad, Maxima, YACAS, etc doing a fair amount of
> symbolic computation (which I naively refer to as mathematica like).
> A more comprehensible list is available at scilinux.sf.net/mathpack.html
> Possible reasons are:
> 1) They start off covering an area not addressed by the other package,
>    (but then spread to cover the areas covered by other software).
> 2) License dissatisfactions,
> 3) Coding styles,
> 4) Programming satisfaction
> ..... N) etc.
> 
> Initially it seems that it is better if all combine and develop
> a better product together, but on reflection it seems that an evolutionary
> model might be better and we might get a decent software which will
> be better than the best ...
> 
> I also wonder how the GSL interface to PYTHON, and also the numeric
> PYTHON extensions has postured PYTHON as a good alternative to these
> various MATLAB like mathematical packages. Of course it is also a far
> cry from doing symbolic computations like Mathematica or MuPAD does.
> I have been promising myself since a long time to check out Maxima.
> 
> Why is all this relevant to this list? Well, it shows how a nice numerical
> library (GSL) can be the heart of many mathematical packages.
> 
> Manoj Warrier (manoj.warrier@ipp.mpg.de)
> 
> Stellaratortheorie, Max-Planck Institut Fur Plasmaphysik
> TeilInstitut Greifswald Wendelsteinstrasse 1
> D-17491 Greifswald Germany Tel: +49-3834-882434
> 
> --------- History of Computing 10-11-3003 ---------------
> Then there used to be this great user friendly OS which
> overwrote your MBR whenever you installed it.
> ---------------------------------------------------------
> 
> On 18 Oct 2002, James Amundson wrote:
> 
> > On Fri, 2002-10-18 at 12:43, Robert G. Brown wrote:
> > > I for one would love to see a clone of mathematica good enough to
> > > completely replace it, built on top of the GSL.  Good luck.
> >
> > Impolite though it may be to mention another project in response to this
> > query, please let me point out that Maxima, <http://maxima.sf.net>, is a
> > mature GPL'ed mathematics program. We don't utilize GSL yet, but there
> > are plans to do so in the future.
> >
> > --Jim Amundson