Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain <mec at shout dot net>]

Hi Rob,

I'm not looking for any changes in 1.4.3.  Think of this more as one
user's feedback.  If you get any good ideas from it, that's fine.

mec> ERRORs and WARNINGs: it would help if all ERRORs and WARNINGs ...
rob> This is probably a good idea.

Now that I know Dejagnu is actively maintained, maybe I'll pick this
one up myself.  I would have to learn more TCL but Dejagnu is not
that large.

mec> Duplicate test names: ...
rob> Um, I don't think it's the responsibility of DejaGnu to work around a
rob> poor programming practice.

Okay, that's fine.  I admit it's a poor practice.  It would have been
nice to move the anti-duplication code into a common place (that isn't
in my scripts, grin).

mec> TIMEOUT: ...
rob> If GDB has crashed, this is supposed to be UNRESOLVED, according to POSIX.
rob> UNRESOLVED is a test case that can't be finished, and needs a human to look
rob> at it.  UNTESTED is when there isn't support in the underlying OS or
rob> hardware to run a test case.

If gdb has crashed, indeed a human has to look at it.  But the tone of
the documentation is that the human is deciding whether the test passed
or not.  Actually a human is needed here to fix the tool, or perhaps fix
the testing environment.  I think TIMEOUT would be more accurate than
using UNRESOLVED for this.

Right now we use FAIL "... (timeout)" for this, so in principle, we're
getting the information we need.  So I can live without this if we really
need it.
  
mec> Split gdb.log file: when I look at gdb.log, I am usually interested in
mec> just one test script.  So I'd like to have a directory of gdb.log files,
mec> one per each test script.  I don't need the giant log.
rob> Hum... I might try that. Mind you, the log files were more for debugging
rob> purposes, than analysing them for test results.

Here is the situation: each week, I run the test suite on a few dozen
configurations.  I have a Perl script that looks for changes in the
results since the last test run.  When I find a negative change, then
I file a bug report.

The problem is what to put in that bug report so that an independent
person can use it effectively.  I'd like to include a copy of the gdb.log
section for that test script, so that people perusing the bug report
can see what's going on without downloading a big tarball from me first.

Another way to look at it: as projects get larger, the roles separate.
I'm planning to publish my scripts so that lots of people can run gdb
tests and mail in useful information.  My end goal is an Internet scale
"gnutest@home" facility, where developers can inject proposed patches and
get some kind of differential analysis over a diverse group of test beds.

rob> Any changes I make have to work generically, since the gdb team is a 
rob> only part of the DejaGnu user community.

Sure, I understand.

Thanks for being here and doing this.

Michael C