This is the mail archive of the mailing list for the Mauve project. See the Mauve home page for more information.

[Date Prev][Date Next][Thread Prev][Thread Next]
[Date Index] [Subject Index] [Author Index] [Thread Index]

Re: What are we testing ?

>>>>> "Artur" == Artur Biesiadowski <> writes:

Artur> We have to decide if mauve is going to be core library test, or
Artur> vm test.  I think that it is designed to be former - I don't
Artur> think that we should use same framework for vm testing.

I thought we had already decided to write a library test suite.  This
decision was made in the initial discussions.

I guess we could change this.  For instance, we could just have two
different test mechanisms in Mauve -- one for VM tests (where each
test case is a separate program), and another for the library tests
(which would preserve the nice properties of the current framework).

I'm probably not going to implement this.

Artur> We have master program, which scan directories for text files with
Artur> tests' main class names. For each test it:
Artur> 1) loads test class (by either Class.forName() or classloader - second
Artur> would allow us to forget about lengthy package names)
Artur> 2) instantiates it through: test = (Test) class.newInstance()
Artur> 3) creates Thread(test)
Artur> 4) sets result harness for test : test.setHarness(harness)
Artur> 5) runs the thread (thus running test in separate thread)
Artur> 6) reads results from harness

I think using threads in the test harness would be a mistake.  As
Anthony points out, not every implementation will have threads (for
instance, our implementation can be configured threadless).

Of course, you're free to make your own subclass of TestHarness that
runs each test in a separate thread, if that is what you want.  I
don't think it gains anything.

Artur> harness.testing("Hashmap.put with null key&value")
Artur> hashmap.put(null,null);

Artur> HashMap should not throw NullPointerException here, but it
Artur> could.  Instead of unknown error, we know exactly at which
Artur> point test failed - it is given in harness variable set by
Artur> testing method.

I just changed SimpleTestHarness to report any uncaught exception in
the test case.  Thanks.

Artur> How exact mauve is supposed to be ? Should it test if given
Artur> class works (more or less) or should it test EVERY aspect of
Artur> visible API ? Said other way, are we designing nice regression
Artur> test, or full blown JCK ?

The tests should be as complete as is reasonable, and should test
compliance to the relevant spec(s).