This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH] Support separate benchmark outputs
- From: OndÅej BÃlka <neleai at seznam dot cz>
- To: Siddhesh Poyarekar <siddhesh dot poyarekar at gmail dot com>
- Cc: Siddhesh Poyarekar <siddhesh at redhat dot com>, GNU C Library <libc-alpha at sourceware dot org>
- Date: Wed, 17 Apr 2013 08:15:07 +0200
- Subject: Re: [PATCH] Support separate benchmark outputs
- References: <20130416122544 dot GH3063 at spoyarek dot pnq dot redhat dot com> <20130416132838 dot GA29626 at domone dot kolej dot mff dot cuni dot cz> <20130416140355 dot GI3063 at spoyarek dot pnq dot redhat dot com> <20130416155032 dot GA30216 at domone dot kolej dot mff dot cuni dot cz> <CAAHN_R1U_8_OMuzVKJ2q2w8i_XWYrx0mYmvHodBLxmNLiznSkQ at mail dot gmail dot com> <20130416185720 dot GA790 at domone dot kolej dot mff dot cuni dot cz> <CAAHN_R0AVLCRDf=Qp1269U3xxUN9LFh_+32mbjyvOVfqSC-rsw at mail dot gmail dot com>
On Wed, Apr 17, 2013 at 09:18:41AM +0530, Siddhesh Poyarekar wrote:
> On 17 April 2013 00:27, OndÅej BÃlka <neleai@seznam.cz> wrote:
> > How do you want to use data that you get from these benchmarks?
> > Could you say that if I come with new implementation if firefox will run
> > faster or slower?
>
> That's not the point of a microbenchmark. A microbenchmark will give
> you more generic information like performance of the function under
> various defined conditions.
> That way an implementing application can
> see that and decide on the tradeoffs it can make.
And how you describe conditions of your benchmark. Conditions under
which we test should be carefully selected to get as much information as
possible.
> Our goal is to provide acceptable average case performance,
A code that I commented is following:
> + for (i = 0; i < 32; ++i)
> + {
> + HP_TIMING_NOW (start);
> + CALL (impl, dst, src, len);
> + HP_TIMING_NOW (stop);
> + HP_TIMING_BEST (best_time, start, stop);
> + }
It does not provide average case performance. It computes best case performance.
This favours implementation with complex control flow that are in 5% of
cases where conditions are rigth faster but not so in 95% of bulk data.
Please compute average.
> but that definitely does
> not mean that we ignore performance characteristics of our functions
> under specific loads.
>
How will we choose these specific loads?
You must deal with conflicts. How when user submits regression of new
implementation in firefox but all our benchmarks show that new
implementation is faster. Who is true?
Or which happens all the time you have three microbenchmarks.
First say that A is better than B which is better than C.
Second says that B is better than C which is better than A.
Third says that C is better than A which is better than B.
How you choose among A,B,C best implementation?
> > I measure several implementations (glibc, modified, byte, qwords)
> > You measure several implementations (sse3, sse4 , byte, avx)
> >
>
> Difference is that your implementations don't sit together in parallel in glibc.
>
They do nat have to sit together (pedanticaly byte implementation no not
sit in glibc). With bit of hacking I could also test what is in libc.
If somebody wants to add new implementation how would you benchmark
that? Similar question is how would you find regressions?
> Siddhesh
> --
> http://siddhesh.in