testsuite and hardcoded timeouts

[William Cohen]

David Wilder wrote:

> 
> I ran into this issue on s390.   When a time out occurs if the test 
> would simply produce a warning message then restarts the timer, allowing 
> the timeout to be restarted say 4 or 5 times before finally reporting a 
> failure.   Then if something breaks the test will still report a 
> failure.  On slower system the test would still pass.  If a  system/test 
> normally passes with one or two restarts of the timer then something 
> changes and it starts taking 3 or 4 restarts we will know that 
> investigation is needed.
> 


You might luck out with the caching helping the later attempts skip some of the 
phases of the translator and avoid those times on the later runs. However, 
restarting 4 or 5 times is probably not going to help that much if the time 
required to generate the module is way larger than the time out.

The timeout is there to make sure that forward progress is made on the testing. 
We would prefer to have the test fail in a reasonable amount of time than to 
have a test hang for an unreasonable amount of time and not get any results at 
all. The translator internals are pretty much a black box to the testing 
harness, so the timer is used to judge when the the test isn't making forward 
progress. Too bad there couldn't be an equivalent to a watchdog for the testing 
harness, e.g. if the test is making forward progress, leave the test be.

-Will