This is the mail archive of the mailing list for the systemtap project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Systemtap .sum vs .log

Kris Van Hees wrote:
In followup to our conversation on the conference call...  The summary file
does not capture the ERROR messages from runtest, and those are significant
in the processing of test results.

Whenm e.g. the staprun executable is not setuid root (or equiv privs), some
tests print out an ERROR about this, but the test result is (for some tests)
actually reported as XFAIL.  Now, per the discussion on the call, that should
be interpreted as a PASS.  So...  the summary file ends up listing a test
result that will get reported as a PASS when it most definitely was not a
successful test execution.

That kind of cases can be resolved based on the verbose log.

Another problem (brought to light by a question from David Wilder) is that
the dejagnu summary output seems to be inconsistent with the actual log
messages (both in the summary and verbose logs):

stap Grep ---- ---- PASS (expected passes) 463 462 FAIL (unexpected failures) 19 19 XFAIL (expected failures) 152 152 XPASS (unknown successes) 1 0 KFAIL (known failures 5 5 UNTESTED (untested) 26 26 UNSUPOPRTED (unsupported) 2 2

As you can see, dejagnu is reporting 1 extra PASS, and 1 extra XPASS, yet
a grep on the log cannot find those two entries.  Does anyone have any idea
where they are coming from?

As to the potential for non-public information making its way into the verbose
log...  If and when that occurs, I am certain that it would be treated as a
high priority bug in systemtap or its testsuite, and that it would be resolved
rather quickly.  For example, the samba build (and test) farm has a long
history of doing this kind of stuff (and they have had their share of somewhat
unexpected things happening).  The main thing usually is to ensure that all
testing is done in a sufficiently safe environment, both in terms of not
letting anything leak out that shouldn't, and in terms of ensuring that a
misbehaving test won't cause more damage.


Here is the difference, the systemtap.sum has one more PASS line that is missing from systemtap.log.

# grep ^PASS: systemtap.sum > pass.sum
# grep ^PASS: systemtap.log > pass.log
# diff -Naru pass.log pass.sum
--- pass.log 2007-10-11 14:07:41.000000000 -0700
+++ pass.sum 2007-10-11 14:07:28.000000000 -0700
@@ -346,6 +346,7 @@
PASS: systemtap.printf/string1.stp
PASS: /home/src/stap/src-20071006/testsuite/systemtap.printf/string2.stp passed
PASS: args search for staprun (/usr/local/bin/staprun)
+PASS: args compile
PASS: args search for probe module (args_3085.ko)
PASS: args search for tmpdir (/tmp/stapwfd7KX)
PASS: args search for probe module (/tmp/stapwfd7KX/args_3085.ko)

I don't see any XPASS in ether the .sum or .log. However there is a single KPASS in both logs. KPASS ?? known successes ???, must be a bug.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]