classpath+mauve

[Norman Hendrich <hendrich at informatik dot uni-hamburg dot de>]

Hello David and Audrius,

first of all, thanks for your answers to my mauve-setup question(s).
I tried the MakeTestClassList program, which seems to work fine.
However, after spending another few hours trying to get the whole 
testsuite running, my worst fears are confirmed... To summarize:

* writing new testcases is easy

* running individual testcases is easy (from the shell or eclipse)

* running parts of the testsuite via an index file works.

* BUT there is no easy way to run the whole testsuite. A developer
  first has to create a custom list of exclusion testcases, before
  running the remaining fraction of the testsuite... 
 

Despite David's nice program, command line parameters, and exclusion 
testcase lists, I did not manage to run the whole testsuite reliably 
against today's classpath CVS. After passing some tests,
jamvm encounters a test that throws some unexpected exception,
runs into a stack-overflow, or complains about an IllegalStateException
in loadClass(). In all cases, gnu.testlet.runner.Mauve does not run to 
the end, and the user doesn't even get the summary of test results.

This basically leads to the situation that a developer needs to run the
testsuite, try to understand the last error, remove the problematic test 
question from the list of testcases, and re-run. I tried this from scratch
without David's exclusion list, but stopped after the first 20 iterations
of this (with no end in sight). I then proceeded to remove larger chunks
of the tests, and finally got a test report - but this in turn means
that many tests were never run at all.

---

The current all-testcases-at-once model used by gnu.testlet.runner.Mauve
is not robust enough to survive all tests, while the opposite strategy
of running each test in a new VM via (Simple)TestHarness is extremely
wasteful due to the VM initialization overhead.

Perhaps we could switch to a two-level model, where an outer process
watches the progress of the inner runner process, and restarts the 
inner process whenever this crashes or deadlocks?

Also, a test that doesn't compile should be marked as FAIL instead
of stopping the testsuite to run at all...


- Norman

PS: a matrix of today's mauve experiments:

* mauve+jdk1.5.0 with 'make check': doesn't even compile
* mauve+jdk1.4.2 with 'make check': works, but subset of tests only
* mauve+jdk1.4.2 with 'ant': doesn't even compile
* mauve+jdk1.5.0 with 'ant': doesn't even compile
* mauve+classpath+jamvm+jikes:  compiles, but crashes on lots of testcases