Re: [patch/rfc] KFAIL gdb.c++/annota2.exp watch triggered on a.x

[David Carlton <carlton at math dot stanford dot edu>]

On Fri, 3 Jan 2003 16:51:34 -0500, Daniel Jacobowitz <drow@mvista.com> said:

> How do you envision them updating the testsuite?  Certainly not by
> removing the KFAIL's pattern; that defeats the point of having a
> regression test.

That's actually exactly how I expect them to update the testsuite
(though they might want to keep the pattern around in a comment
somewhere, or even leave the pattern intact but replace the kfail by
fail plus a comment).  If a bug is claimed to be fixed but isn't
actually fixed, then that bug isn't a known failure any more, so it
should be FAILed until the PR is reopened.

Consider this scenario: we have a bug, with a test for it; that test
has a PASS pattern (either because people don't like KPASS or because
the bug is intermittent) and a KFAIL pattern.

Then programmer A, a habitual user of platform A, fixes the bug,
closes the PR, happily notices that the test has changed from KFAIL to
PASS, but leaves the KFAIL branch intact.

Except that it turns out that programmer A's fix was more specific to
platform A than he or she realized.  The bug is still present on
platform B; programmer B on platform B didn't notice that the bug was
allegedly fixed (maybe programmer B wasn't even working on GDB at the
time that the bug was fixed), but programmer B does notice that
there's a KFAILed test case corresponding to the bug.  So programmer B
assumes that the bug is, in fact, known, whereas from the point of
view of both programmer A and the GDB bug database, the bug is
supposed to have gone away.

On the other hand, if programmer A had deleted the KFAIL pattern, then
programmer B would see the test start FAILing, would say "hey, the
test suite says that GDB has an unknown bug here", and would then have
a reason to investigate the situation.

Removing the test entirely would the point of having a regression
test.  But removing patterns that handle casses that we don't expect
to occur should make the test more effective rather than less
effective: if we get surprising output from GDB, we want that to be
flagged as prominently as possible.

At least, that's my reasoning.

David Carlton
carlton@math.stanford.edu