testsuite and hardcoded timeouts

[Quentin Barnes <qbarnes at urbana dot css dot mot dot com>]

I mentioned this issue as an aside and asked about it a month ago,
but I don't think it got a response at the time.

In porting the Systemtap testsuite to an embedded ARM platform
(~350Mhz CPU with 64MB and running using an NFS root and swap), I
found many of the existing hardcoded timeout parameters are way too
short.  Several of them I had to at least triple if not increase by
a larger factor, sometimes 6x-15x to get them to pass.

How do we want to deal with this portability problem of hardcoded
timeouts?


There are a few ways I can think of to address this:

1) Let the hardcoded numbers stay, but up them large enough to
  handle the slowest platform we might ever run on.  If that's
  still not slow enough someday, up them some more when the time
  comes.

This has the advantage of simplicity, but can greatly slow down
suite runs on faster processors when tests do get stuck.

2) Ban all standalone hardcoded timeouts replacing them with an
  expression involving a multiplier and/or a constant and a
  multiplier.

This is not the cleanest because some tests are slow due to I/O
bandwidth or paging where others are slow due to CPU limitations.
But it does have the advantage that if someone is having timeout
issues, they can up the multiplier value and rerun to see if the
problem goes away without having to edit all sorts of wrapper
scripts and tests.

If we go with a multiplier, the multiplier could be set
automatically by reading the cpuinfo and taking a stab at it based
on the machine's BogoMIPS or MHz.  We'd still need a way to have a user
straightforwardly tweak it beyond that manually.

Unfortunately, I don't understand the Systemtap testsuite framework
yet well enough to make specific suggestions.

Thoughts?

Quentin