Re: [PATCH] New numbers in the benchtests.

[Siddhesh Poyarekar <siddhesh at gotplt dot org>]

On Friday 15 December 2017 11:49 PM, Ben Woodard wrote:
> Right now, they demonstrate specific numbers that trigger the
> implementation to go into different modes. With our current
> implementation, this is the fast path, the slow path and the very slow
> path. So these functions are there to show how well the current
> implementation is running when it enters those modes. You can easily
> see if a code change or a compiler change improves or impacts
> performance. This is good for optimizing the implementation for
> performance in the traditional way.
> 
> However, there are also certain inputs to functions which are known to
> be problematic or more difficult to round correctly than others and
> you must compute them out to N-digits to get correct rounding when
> using a particular implementation. The quality of implementation is in
> essence how often we hit those hard cases and how quickly we can deal
> with them. These benchtests that I added are more along those lines.
> For example, they include exp for the last subnormal and the first
> normal floating point number. Together these represent a test case
> which is known by the customer to identify quality of implementation
> issues. They have indicated to us this case comes up frequently enough
> that would like it to be tracked and Carlos suggested adding it to the
> benchtests.

You're right in that specific inputs (or rather results) are more
important than others due to the number of bits required to ensure a
correct rounding regardless of the algorithm used.  However, they assume
importance from a practical standpoint only if they result in vastly
different behaviour in the algorithm.  Slow path vs fast path for
example makes sense if the algorithm actually has a slow and fast path.
In that context, the last subnormal and first normal number are
important, but not more important than other inputs that exercise the
exact same code path.

This is why I suggest merging these inputs into the code path that they
exercise since that sort of partitioning scheme makes more sense for
upstream code than 'redhat-fast-customer-cases' and
'redhat-slow-customer-cases'.  If you want to see those input results
differently then a downstream patch that converts the comment into a tag
"##name: " is a more appropriate solution.

Note that this is different from the concept of workloads that we add to
benchmark inputs where the sequence of calls is also important and hence
they're executed slightly differently.  See for example some CPU2006
inputs that we added to simulate that workload.  If your inputs emulate
a specific and interesting workload then that's interesting as a
separate partition even if it is vendor-specific but it doesn't seem
like that.

Siddhesh