This is the mail archive of the mailing list for the Mauve project. See the Mauve home page for more information.

[Date Prev][Date Next][Thread Prev][Thread Next]
[Date Index] [Subject Index] [Author Index] [Thread Index]

Re: What are we testing ? (fwd)

Godmar> Sometimes, tests will fail because they don't compile.  Should
Godmar> that be a reason to bail out of the framework?

Anthony> That's a good point.  It definitely shouldn't cause the
Anthony> testsuite run to fail.  If a testcase doesn't compile because
Anthony> the library implementation is wrong, we'll, that should be
Anthony> recorded as a testcase failure, but everything else should
Anthony> continue as usual.  We will need to fix this.

I think this will probably be a pain to implement.  That's because
test case selection, compilation, and execution happen in different
places (`choose', Makefile, and test harness respectively).

Instead I'd rather make it so that uncompilable test cases will be
avoided when appropriate.  For instance, we already do this with
libjava -- if you specify the `libjava' tag in KEYS, then nothing from
java.beans will be built.

Anthony> If a particular build failure is cause problems right now,
Anthony> you can always exclude test cases with a !java.* KEYS value.

I recommend this as a general strategy.  This is what the `mauve-TAG'
files are for (see README).