This is the mail archive of the
gdb-patches@sources.redhat.com
mailing list for the GDB project.
Re: RFC: KFAIL DejaGnu patch
- From: Michael Elizabeth Chastain <mec at shout dot net>
- To: fnasser at redhat dot com
- Cc: ac131313 at cygnus dot com, drow at mvista dot com, gdb-patches at sources dot redhat dot com, rob at welcomehome dot org
- Date: Sun, 7 Apr 2002 20:40:43 -0500
- Subject: Re: RFC: KFAIL DejaGnu patch
More comments ...
gdb.c++/cplusfuncs.exp has a good place for some KFAIL's. Grep for
hairyfunc5, hairyfunc6, hairyfunc7. They are associated with PR gdb/19.
I added some setup_kfail's to that and it appears to work okay,
except that the bug id is not printed on xpass.
Okay, here's some proofreading comments.
Michael C
=== Distinction between XFAIL and KFAIL
An XFAIL is:
A test is expected to fail in some environment(s) due to some
bug in the environment ...
And a KFAIL is:
A test is known to fail in some environment(s) due to a known bug
in the tool being tested (identified by a bug id string).
I like this distinction. The "K" letter means that it is a known problem
inside the tool, and the "X" letter means that it is an expected problem
outside the tool.
That makes it weird for a KFAIL to turn into an XPASS. I've got test
results here with XPASS for the gcc v2 compilers and KFAIL for the gcc
v3 compilers. I really want KFAIL/KPASS, not KFAIL/XPASS.
A specific note: proc record_test has this code:
XPASS {
set exit_status 1
if { $xfail_prms != 0 } {
set message [concat $message "\t(PRMS $xfail_prms)"]
}
}
In fact that code could use a little refactoring, because
UNRESOLVED/UNSUPPORTD/UNTESTED all have the same code to pick up
kfail_prms and xfail_prms.
So an XPASS that comes from setup_xfail will have a PRMS id,
but an XPASS that comes from setup_kfail will not.
=== KFAIL as a way of hiding problems
Later on in the documentation:
@item KFAIL
...
This exists so that, after a bug is identified and properly registered
in a bug tracking database (Gnats, for instance), the count of failures
can be kept as zero. Having zero has a baseline in all platforms allow
the tool developers to immediately detect regressions caused by changes
(which may affect some platforms and not others).
Conceptually, there is a disconnect here. To me, KFAIL means: "there
is a problem report associated with this test". The problem could be
minor, or it could be a showstopper. But you say that KFAIL means:
"tool developers can ignore this test when they look for regressions".
I think this is fundamentally wrong. The right way to look for
regressions is to compare before-and-after test runs. gdb.sum files
are already pretty good for this; you can just diff them.
I see two practical problems that come out of this wrongness:
(1) If I find a showstopper regression, such as PR gdb/379, and I mark
it with setup_kfail (as I should), then someone who is using the
"# of FAILs" metric is not going to see the regression.
(2) If the test suite has a significant number of setup_kfail (and
it should), then a regression bug may manifest as a transition from
PASS -> KFAIL. I will see that because my regression reports look
at all transitions. But someone looking at "# of FAILs" will not see
this transition, so they won't see the regression.
I believe we already have this problem with XFAIL. People conflate
the idea of "this test fails due to an external problem" with the idea
"this failure is not important enough to care about at this time".
=== ChangeLog entry
DejaGnu 1.4.2 has a ChangeLog, so the patch needs a ChangeLog entry.
=== Tests
testsuite/ needs some tests for the new KFAIL feature.
=== lib/dg.exp
lib/dg.exp has a bunch of XFAIL stuff, so it needs KFAIL stuff.