This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH] Run benchmarks for constant time instead of constant iterations
- From: Siddhesh Poyarekar <siddhesh at redhat dot com>
- To: OndÅej BÃlka <neleai at seznam dot cz>
- Cc: libc-alpha at sourceware dot org
- Date: Wed, 24 Apr 2013 20:30:27 +0530
- Subject: Re: [PATCH] Run benchmarks for constant time instead of constant iterations
- References: <20130424130839 dot GG14810 at spoyarek dot pnq dot redhat dot com> <20130424143241 dot GA5192 at domone dot kolej dot mff dot cuni dot cz>
On Wed, Apr 24, 2013 at 04:32:41PM +0200, OndÅej BÃlka wrote:
> I work at even better idea, benchmark should run until certain level of
> precision is reached. I decided to have with probability 90% error of 1%.
>
> This depends on preheat cpu patch, collecting data at 800MHz and then
> switching to 2.5GHz will cause running time of minute until slow data
> are ruled out as error.
>
> Your patch is nice, with it I could make less modifications.
>
> I want do rougthly following. Perhaps you could merge it to your patch?
That's an interesting idea. By 'precision' and 'error' I assume you
mean the stability of the measurements and the deviation of the
current measurement from the mean. Is that a correct interpretation
of what you're saying or do you mean something else?
Howeber, it could be possible that the measurements could stabilize
early on and give us a worse result than it actually is, no? To avoid
that we may need to set a high enough initial number of iterations,
which again could lead to vastly different runtimes for different
architectures and may even have to be decided and hard-coded on an
individual function basis. This essentially brings us back to the
problem I'm trying to solve with this, which is to avoid the great
variance in benchmark runtimes across architectures.
Siddhesh