[PATCH] Run benchmarks for constant time instead of constant iterations

[Siddhesh Poyarekar]

On Wed, Apr 24, 2013 at 04:32:41PM +0200, Ondřej Bílka wrote:
> I work at even better idea, benchmark should run until certain level of
> precision is reached. I decided to have with probability 90% error of 1%.
>
> This depends on preheat cpu patch, collecting data at 800MHz and then
> switching to 2.5GHz will cause running time of minute until slow data
> are ruled out as error.
> 
> Your patch is nice, with it I could make less modifications.
> 
> I want do rougthly following. Perhaps you could merge it to your patch?

That's an interesting idea.  By 'precision' and 'error' I assume you
mean the stability of the measurements and the deviation of the
current measurement from the mean.  Is that a correct interpretation
of what you're saying or do you mean something else?

Howeber, it could be possible that the measurements could stabilize
early on and give us a worse result than it actually is, no?  To avoid
that we may need to set a high enough initial number of iterations,
which again could lead to vastly different runtimes for different
architectures and may even have to be decided and hard-coded on an
individual function basis.  This essentially brings us back to the
problem I'm trying to solve with this, which is to avoid the great
variance in benchmark runtimes across architectures.

Siddhesh