RFA: Testsuite patches...

Donn Terry donnte@microsoft.com
Wed Aug 2 10:59:00 GMT 2000


One of the first rules of testing: don't discard useful tests.
Thus, I agree that the tests should be added as they are generated.

The problem ISN'T the tests, it's the apparent regression (and the
work that that causes).  Blaming the tests is like blaming the messenger.
Having been in the business of doing a new port over a protracted
period (and of the whole toolchain) it has been a real pain for me
to deal with this problem, but I would never, ever object to including
a new (valid, of course) test.

In the gcc case there has been a database of "who failed what", which gives
me a clue as to whether it's a new test that most systems don't pass (and
I sould defer working on it to more immediate (for me) things) or something
I should fix now.  It's the lack of THAT data that makes including the
new tests painful.  (The gcc tool is less-than-perfect, but moving
it to a central server would address the biggest of the problems.)

XFAIL is a very crude tool: it doesn't really say what "reasonable
expectation" is for any given platform.  However, it's a help, but
has often been abused.

Bright ideas on how to manage the problem of apparent regressions would
be most useful.  Certainly educating the people who look at the
regression results that because the set of regressions is growing,
there will be continuous new failures would be useful.

(One bright idea: insist that the body of the test describe precisely
what is being tested and why, the expectation of which systems should
or should not pass (not in detail, but "RISCs probably will have trouble
with this because..."), and a date(!!!) will make it much easier to
make an informed judgement on a new failure.   Another: include date
of addition in a place where the final report can say something like:
"Tests younger than 30 days: nnn pass; mmm fail.
 Tests younger than 90 days..."

(This makes it easier to show to some manager that it's new tests
that are a problem.)

Donn


> -----Original Message-----
> From: Daniel Berlin [ mailto:dberlin@redhat.com ]
> Sent: Wednesday, August 02, 2000 9:45 AM
> To: Kevin Buettner
> Cc: Andrew Cagney; GDB Discussion; GDB Patches Mail List
> Subject: Re: RFA: Testsuite patches...
> 
> 
> Kevin Buettner <kevinb@cygnus.com> writes:
> 
> > On Aug 2,  9:55pm, Andrew Cagney wrote:
> > 
> > > I think GDB should be accepting tests (provided that they are
> > > rigiously examined) even when they add failures - just as long as
> > > the failures examine real bugs.  I think this also better reflects
> > > what really goes on.
> > 
> > I agree.
> > 
> > If we make "no new failures" the criteria for whether a 
> test is added
> > to the testsuite or not, then it seems to me that we'll end 
> up adding
> > very few new tests just because it's so difficult for any one person
> > to test on all affected targets.  (And it really doesn't work to
> > post a patch and expect everyone affected to try it.)
> > 
> > It makes sense to me to spread the workload by adding a 
> test and then
> > expecting the various maintainers to make sure that the test passes
> > (or gets suitably tweaked) for their targets.
> > 
> > Kevin
> 
> This seems like a better idea.
> 
> In fact, I propose the following, or something like it:
> 
> We accept all new tests people are willing to contribute, whether GDB
> passes them or not, on any platform (assuming the test itself is
> showing a problem with GDB, or something that should eventually work
> in GDB, like say, virtual function calling).
> 
> We have a seperate directory in the testsuite for tests that nobody
> has any idea whether it will pass on all platforms or not, or whether
> GDB can do that yet or not.
> 
> That way, even if you XFAIL'd the test (so people didn't bitch about
> the failures), at least I could look in that test results for 
> that directory when I wanted to know what should be
> working, but isn't, etc.
> 
> Or maybe i'm just babbling.
> --Dan
> 


More information about the Gdb-patches mailing list