Some more Mauve observations

[Mark Wielaard <mark at klomp dot org>]

Hi,

Here are some things that I would like to improve in Mauve.
It is also the beginning of some documentation. Although I do admit that
it looks more like a collection of random thought and observations :)
Let me know what you think about it.

Mauve doesn't have really simple application/system tests only unit
tests. The unit tests are much more accurate but need some extra
framework. I agree with Andrew that real standalone tests are very nice
to have, especially for exchanging tests with people that don't have
Mauve setup. Dalibor is correct that such application tests are much
more fragile then real unit tests but I think it is more important to
have a really easy way to add more tests. Will try to come up with
something that turns a application test into a Mauve unit test without
to much work.

I looked at the unit test framework of Mauve and JUnit (not how to
actually run the test suite, see below) and I don't think one is really
better then the other. For Mauve it works as follows:

When writing a class you implement gnu.testlet.Testlet which defines
just one method void test (TestHarness harness). The TestHarness gives
you the following methods for which comparable ones can also be found in
JUnit:

checkPoint(String) - Gives (base)name for next check/test.

check(boolean) - Test if argument is true.

fail(String) - Give the test a name and fail.
               Basically, checkPoint(name); check(false);

check(X,Y)
check(X,Y,String) for Objects and all primitive types
- checks if the given arguments are the same.
  The optional third argument can be the name of the test.

It might be nice to have something like the JUnit Assert class in Mauve
if people think that makes people that know about JUnit already help
write Mauve tests:
http://junit.sourceforge.net/javadoc/junit/framework/Assert.html
Except for that class I don't think JUnit is that special.
And some of it relies to much on a good working Java environment and
Mauve is often used in a more challenging situation.

The following features are not present in JUnit:

The following TestHarness methods can be used to give more output:

debug(String) and verbose(String)
debug(String, boolean) for (not) adding newlines.
debug(Trowable) and debug(Object[], String)

The TestHarness also implements the gnu.testlet.config
interface that defines methods that give the Strings representing the
SourceDirectory, TempDirectory, PathSeparator and file Separator. These
values are generated by configure.
(The only test that uses this directly is
gnu/testlet/java/io/File/jdk11.java.)

You can also use the following methods from TestHarness:
getResourceReader() and getResourceStream() which give you a handle to a
file (starting from the top of the source tree). The implementation in
SimpleTestHarness uses the configure values found in config.java. These
methods are certainly handy but I hope that they can be done simpler (so
they don't need the configure output) by for example adding the contents
of those resource to a static byte[] in the class itself.
(All of the resources used are less then 500 bytes except for the
Unicode database which is more then 400K.)

Nice about the Mauve SimpleTestHarness is that it interprets the
"xfails" file (in the current directory) when verbose is true. That
makes it easy to find regressions. (JUnit does not seem to have this
because it is based on the XP methodology which has the philosophy that
it is really, really bad to have failing tests and not fixing them
immediately.)

Setting up and running the Mauve framework is not the easiest thing in
the world. The configure and Makefile are to complex because they have
special rules for using gcj. I would like to push the special support
for gcj out of Mauve and in the libgcj tree so we can use more clear and
simple scripts (and hopefully get rid of the whole configure step).

Alex wrote some simple shell scripts for Kissme to easily run Mauve. The
keys file and the tags should make Mauve very flexible but in practice
it is unclear. Needs more documentation. Each file contains a // Tags:
line (and possible a // Uses: line). Which tags imply which other tags
and the mechanism for generating the classlist from a key file is done
in the choose script. I think this mechanism is fine but it is poorly
documented.

What I am personally really missing is a test batch mode. It would be
really nice to have a mechanism for running the compiler and VM for each
test separately. Currently when the compiler cannot compile a test or if
the VM crashes your complete test run stops. libgcj has something that
can do that using dejagnu but I hope it can be done by a simple shell
script.

I hope to implement some of the above suggestions in the next couple of
days.

Cheers,

Mark