Re: [RFC] Porting string performance tests into benchtests

[OndÅej BÃlka <neleai at seznam dot cz>]

On Wed, Apr 03, 2013 at 03:41:30PM +0530, Siddhesh Poyarekar wrote:
> Hi,
> 
> I'm looking at porting the string function performance tests
> (string/test-*) into benchtests.  They don't fit into the simple
> scenario of mean execution times - they are characterized by the way
> they scale in relation to other methods.  In memcpy for example, the
> comparison is between a simple byte copy, glibc memcpy and
> __builtin_memcpy.
>
I plan to create string benchmarks, I still do not have it automated, I
to compare it to look at graph. I will post what I have it friday/next week.

Main problem with them that there are various biases that favor one
implementation over other.

One source of bias is branch prediction. Performance characteristic
varies depending on what was called before. Compare following two figures:
http://kam.mff.cuni.cz/~ondra/benchmark_string/core2/strlen_profile/results_seq/result.html
http://kam.mff.cuni.cz/~ondra/benchmark_string/core2/strlen_profile/results_rand/result.html
which is different from what gcc uses.
http://kam.mff.cuni.cz/~ondra/benchmark_string/core2/strlen_profile/results_gcc/result.html

A memcpy etc is even more tricky. It uses jump table. When tested
with same length it looks jump table is free. However this is about 50
cycles from truth, when we also consider cache misses. See following:
http://kam.mff.cuni.cz/~ondra/benchmark_string/core2/memcpy_profile/results_rand/result.html
http://kam.mff.cuni.cz/~ondra/benchmark_string/core2/memcpy_profile/results_gcc/result.html

> I could port these over by simply defining them as 'benchsets', which
> would be independent benchmark runs that execute within the framework
> and generate a separate output (i.e. not within bench.out) with its
> own format.  That way I need to do as less work as possible to port
> the functions over and it also adds a new flexibility to the benchmark
> suite.  Any thoughts on this?