Re: GDB 7.9.90 available for testing

[Doug Evans <dje at google dot com>]

On Fri, Jul 10, 2015 at 9:33 AM, Pedro Alves <palves@redhat.com> wrote:
> There's no single "base" run, actually.  The baseline is dynamically
> adjusted at each build; it's a moving baseline, and it's per
> test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
> added to the baseline.  That means that if some test is racy, it'll
> sometimes FAIL, and then a few builds later it'll PASS, at which point
> the PASS is recorded in the baseline, and then a few builds again
> later, the test FAIL again, and so the buildbot email report mentions
> the regression against the baseline.  In sum, if a test goes
> FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
> get reports of regressions against the baseline for that racy test.

Time for another plug to change how we manage racy tests?

E.g., if a test fails, run it again a few times.
I can think of various things to do after that.
E.g. if any of the additional runs of the test record a PASS then flag
the test as RACY, and remember this state for the next run, and rerun
the same test multiple times in the next run. If the next time all N
runs pass (or all N runs fail) then switch its state to PASS/FAIL.
That's not perfect, it's hard to be perfect with racy tests. One can
build on that, but there's a pragmatic tradeoff here between being too
complex and not doing anything at all.
I think we should do something. The above keeps the baseline
machine-generated and does minimal work to manage racy tests. A lot of
racy tests get exposed during these additional runs for me because I
don't run them in parallel and thus the system is under less load, and
it's system load that triggers a lot of the racyness.

The ultimate goal is of course to remove racy tests, but first we need
to be more systematic in identifying them, which is one of the goals
of this process.