[PATCH] Run benchmarks for constant time instead of constant iterations

[Ondřej Bílka]

On Wed, Apr 24, 2013 at 08:30:27PM +0530, Siddhesh Poyarekar wrote:
> On Wed, Apr 24, 2013 at 04:32:41PM +0200, Ondřej Bílka wrote:
> > I work at even better idea, benchmark should run until certain level of
> > precision is reached. I decided to have with probability 90% error of 1%.
> >
> > This depends on preheat cpu patch, collecting data at 800MHz and then
> > switching to 2.5GHz will cause running time of minute until slow data
> > are ruled out as error.
> > 
> > Your patch is nice, with it I could make less modifications.
> > 
> > I want do rougthly following. Perhaps you could merge it to your patch?
> 
> That's an interesting idea.  By 'precision' and 'error' I assume you
> mean the stability of the measurements and the deviation of the
> current measurement from the mean.  Is that a correct interpretation
> of what you're saying or do you mean something else?
>
Moreless. it means that http://en.wikipedia.org/wiki/Confidence_interval 
is within given fraction of a mean.

> Howeber, it could be possible that the measurements could stabilize
> early on and give us a worse result than it actually is, no?  To avoid
> that we may need to set a high enough initial number of iterations,
> which again could lead to vastly different runtimes for different
> architectures and may even have to be decided and hard-coded on an
> individual function basis.  This essentially brings us back to the
> problem I'm trying to solve with this, which is to avoid the great
> variance in benchmark runtimes across architectures.
> 
This does not happen in practice. Most of time variance is estimated
with sufficient precision within 100 trials. Then estimate to number of
iterations needed is pretty accurate.

> Siddhesh