Labrat reports for 2.6.27-RC snapshot tests

[Kris Van Hees]

On Fri, Oct 24, 2008 at 01:34:26PM +0200, Mark Wielaard wrote:
<< snip>>
> There is some noise in the output because the script seems to not
> understand some of the error messages when things fail. e.g.
> 
>    FAIL  -> N/A   test - systemtap.samples/gtod.exp(gtod (0))
>    [...]
>    N/A   -> PASS  test - systemtap.samples/gtod.exp(gtod (300))
> 
> It would be useful to keep either the kernel version or the systemtap
> snapshot version constant between runs. It looks like currently both are
> updated at the same time. Although it isn't completely clear, the
> version is always reported as 0.7.1, it would be good to mention the
> last git commit (as in stap -V).

This is actually not something that can easily be solved without hardcoding
test-specific logic to strip out dynamic content in these messages.  Since
labrat is not a project-specific reporting system, implementing such hardcoded
rules would be less desirable.  Also note that wiring this logic in a way
that is not test specific is somewhat dangerous because you may end stripping
out important information from the messages unless people are extremely
careful in how they write the pass/fail messages.

If there is a very strict convention (one that gets enforced) on how dynamic
data is included in these messages, I can definitely look at implementing the
logic to strip it out based on the established conventions.

	Kris