Re: [PATCH] benchtests/Makefile: Run the string benchmarks four times by default.

[Will Newton <will dot newton at linaro dot org>]

On 5 September 2013 16:03, OndÅej BÃlka <neleai@seznam.cz> wrote:
> On Thu, Sep 05, 2013 at 08:51:53AM +0100, Will Newton wrote:
>> The intention of my patch - which I may have not made completely clear
>> in the commit message - is to improve test stability. What I mean by
>> this is that with a physically indexed cache the physical pages
>> allocated to the test can have a significant effect on the performance
>> at large (e.g. cache size / ways and above) buffer sizes and this will
>> cause variation when running the same test multiple times. My aim is
>> to average out these differences as it is hard to control for them
>> without understanding the details of the cache subsystem of the system
>> you are running on.
>>
> This can be just explained just by having more data. Simply multiplying
> iteration count by four would then do same job.

No, it wouldn't. That would just mean four times as much data
resulting in a reduced variance but the same systematic error.

>> Your test appears to be addressing concerns of test validity by
>> running a wider range of buffer alignments, which is an important but
>> separate concern IMO.
>>
> No, your patch will pick src pointer at 4 different physical pages (one
> allocated in each run) and calculate average performance.
>
> Mine will pick src pointers in 2000000/4096 = 488 different pages and
> calculate average.

Yes, this would work too. But it has a number of flaws:

1. It does not allow one to analyze the performance of the code across
alignments, everything gets averaged together.
2. It has no mechanism for showing variance, whereas multiple runs of
the same test the variance of the means can at least be seen.
3. It only works for one test (memcpy).

-- 
Will Newton
Toolchain Working Group, Linaro