This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: Summary for the glibc benchmark BoF
- From: Jeff Law <law at redhat dot com>
- To: munroesj at linux dot vnet dot ibm dot com, Siddhesh Poyarekar <siddhesh at redhat dot com>
- Cc: libc-alpha at sourceware dot org
- Date: Tue, 18 Aug 2015 13:40:48 -0600
- Subject: Re: Summary for the glibc benchmark BoF
- Authentication-results: sourceware.org; auth=none
- References: <20150818080953 dot GG2415 at spoyarek dot pnq dot redhat dot com> <1439925005 dot 569 dot 13 dot camel at oc7878010663>
On 08/18/2015 01:10 PM, Steven Munroe wrote:
I don't think IBM is saying that the current benchmarks are complete or
completely representative. We are saying that current benchmarks are
what we have, and a assertion that the lack of some hypothetical
"better" benchmark, should not be used as an excuse to block a patch.
That seems reasonable to me.
Building a benchmarking suite is an iterative process -- you start with
what you can and build around it. Over time some tests may be
superceded by more representative tests, but until those more
representative tests exist, you live with what you've got.
I would personally like to see more representative benchmarks based on
actual usage.
I can't think anyone would say that more representative benchmarks would
be a bad idea.
I would also assert that benchmarks should be split into representative
(of normal usage) and extreme (for example testing for quadratic
behavior or only testing for the needle match at the very end of the
haystack) categories. And clearly labeled as such.
Agreed 100%. Creating reasonable buckets for tests is valuable. And
for a performance suite, I can certainly see the value in clearly
marking normal vs extreme/corner case tests.
We've done it in GCC and it's proven helpful. We're testing for
different things, but the fundamental concepts are the same. If test
XYZ fails, the quicker you can parse the impact the better. It's lame
when the developer has to either "just know" or read the source to find
out that the test is some corner case condition.
We have buckets for things like ieee-fp conformance, testing language
limits, extension conformance, quality of debugging output, thread
safety, etc etc.
The one thing we didn't do was go back to the old tests and put them
into the new buckets, though we do try and route new tests into the
right buckets.
Jeff