Re: RFA: Testsuite patches...

[Michael Snyder <msnyder at redhat dot com>]

Donn Terry wrote:
> 
> One of the first rules of testing: don't discard useful tests.
> Thus, I agree that the tests should be added as they are generated.

We do not have to throw away the tests.
Convert them into problem reports.
That is how we traditionally keep track of problems.
We do not traditionally note a problem by adding
a test that will fail.  If we know of a problem, 
there is an established way to keep track of it.

Adding a test that will fail is a very annoying way
to ask someone to fix something, IMHO.


> 
> The problem ISN'T the tests, it's the apparent regression (and the
> work that that causes).  Blaming the tests is like blaming the messenger.
> Having been in the business of doing a new port over a protracted
> period (and of the whole toolchain) it has been a real pain for me
> to deal with this problem, but I would never, ever object to including
> a new (valid, of course) test.
> 
> In the gcc case there has been a database of "who failed what", which gives
> me a clue as to whether it's a new test that most systems don't pass (and
> I sould defer working on it to more immediate (for me) things) or something
> I should fix now.  It's the lack of THAT data that makes including the
> new tests painful.  (The gcc tool is less-than-perfect, but moving
> it to a central server would address the biggest of the problems.)
> 
> XFAIL is a very crude tool: it doesn't really say what "reasonable
> expectation" is for any given platform.  However, it's a help, but
> has often been abused.
> 
> Bright ideas on how to manage the problem of apparent regressions would
> be most useful.  Certainly educating the people who look at the
> regression results that because the set of regressions is growing,
> there will be continuous new failures would be useful.
> 
> (One bright idea: insist that the body of the test describe precisely
> what is being tested and why, the expectation of which systems should
> or should not pass (not in detail, but "RISCs probably will have trouble
> with this because..."), and a date(!!!) will make it much easier to
> make an informed judgement on a new failure.   Another: include date
> of addition in a place where the final report can say something like:
> "Tests younger than 30 days: nnn pass; mmm fail.
>  Tests younger than 90 days..."
> 
> (This makes it easier to show to some manager that it's new tests
> that are a problem.)
> 
> Donn
> 
> > -----Original Message-----
> > From: Daniel Berlin [mailto:dberlin@redhat.com]
> > Sent: Wednesday, August 02, 2000 9:45 AM
> > To: Kevin Buettner
> > Cc: Andrew Cagney; GDB Discussion; GDB Patches Mail List
> > Subject: Re: RFA: Testsuite patches...
> >
> >
> > Kevin Buettner <kevinb@cygnus.com> writes:
> >
> > > On Aug 2,  9:55pm, Andrew Cagney wrote:
> > >
> > > > I think GDB should be accepting tests (provided that they are
> > > > rigiously examined) even when they add failures - just as long as
> > > > the failures examine real bugs.  I think this also better reflects
> > > > what really goes on.
> > >
> > > I agree.
> > >
> > > If we make "no new failures" the criteria for whether a
> > test is added
> > > to the testsuite or not, then it seems to me that we'll end
> > up adding
> > > very few new tests just because it's so difficult for any one person
> > > to test on all affected targets.  (And it really doesn't work to
> > > post a patch and expect everyone affected to try it.)
> > >
> > > It makes sense to me to spread the workload by adding a
> > test and then
> > > expecting the various maintainers to make sure that the test passes
> > > (or gets suitably tweaked) for their targets.
> > >
> > > Kevin
> >
> > This seems like a better idea.
> >
> > In fact, I propose the following, or something like it:
> >
> > We accept all new tests people are willing to contribute, whether GDB
> > passes them or not, on any platform (assuming the test itself is
> > showing a problem with GDB, or something that should eventually work
> > in GDB, like say, virtual function calling).
> >
> > We have a seperate directory in the testsuite for tests that nobody
> > has any idea whether it will pass on all platforms or not, or whether
> > GDB can do that yet or not.
> >
> > That way, even if you XFAIL'd the test (so people didn't bitch about
> > the failures), at least I could look in that test results for
> > that directory when I wanted to know what should be
> > working, but isn't, etc.
> >
> > Or maybe i'm just babbling.
> > --Dan
> >