Re: Improving libm-test.inc structure and maintenance

[OndÅej BÃlka <neleai at seznam dot cz>]

On Sun, May 05, 2013 at 02:06:29PM +0000, Joseph S. Myers wrote:
> On Sun, 5 May 2013, Ondrej Bilka wrote:
> 
> > Why do auto-libm-test-out  have to be checked in? If outputs be computed 
> > automaticaly we can save them in test directory. I could write utility
> > that caches mpc results on disk and returns them.
> 
> Generating expected results would dominate the time required for a 
> complete testsuite run if it wasn't checked in; build/test directories are 
> generally expected to be transient.  Also, I don't want to assume the user 
> testing glibc has an MPC version recent enough to contain mpc_log10, but 
> do want to use mpc_log10 when generating the expected results rather than 
> hacking around its absence.  Nor do I want MPFR or MPC bugs to cause 
> different people to get different glibc test results because their 
> expected results were computed differently.
>
Thanks, that was answer I was looking for.

Technical note: could you add test if  auto-libm-test-in changed then 
auto-libm-test-out should change as well.

> > One possibility would be use dryrun format which also could allow
> > automate bug report of inaccurate results. I could combine this with 
> > program that tries random values and looks for inaccuracies. 
> 
> I think testing random values (cf. also Jakub's code to do such testing 
> for fma), probably with appropriate biases to include reasonable 
> proportions of large inputs, small inputs, etc. rather than being uniform 
> across the exponent range, is best as a separate project that could test 
> more libm implementations than just glibc.  See the last point under "libm 
> testsuite" at 
> <http://sourceware.org/glibc/wiki/Development_Todo/Master>.  Humans then 
> need to review results to identify if bugs found are genuinely new before 
> reporting them.
> 
You do not have to review if you do following: 
You have random tester that is periodicaly rebuilds lm and tests 
on results that he knows are inaccurate. Then it tries say 1000000
random inputs.

It reports bug for first wrong result and waits with reporting until 
first wrong result becomes accurate where it can report bug again. 
Repeat. 

This will pair automaticaly generated bugs with manual ones and it is
marking these duplicate has small overhead. 
Well unless there is commit that fixes first input by accident.

I think first part of having system that automates submission and
resolves bug when it is fixed is viable. A simple submission form could
be http://kam.mff.cuni.cz/~ondra/inaccuracy_example

We would need properly configured machines to run tests on them to make
this viable.

You could generate input and output files from database.

> I should perhaps add what I think the goal is for how libm functions 
> behave, and that such random tests should test:
> 
Perhaps you could put it on wiki?

> Each function with a floating-point result (real or complex) should behave 
> as if it computes an infinite-precision result that is within a few ulp 
> (in both real and complex parts, for functions with complex results) of 
> the mathematically correct value (interpreted together with ISO C or POSIX 
> semantics for the function in question) of the function at the exact value 
> passed as the input.  Exceptions should be raised, and errno set, 
> appropriately for this value (provided that there is no requirement to set 
> errno for complex functions) and in accordance with ISO C / POSIX 
> semantics, and it should then be rounded according to the current rounding 
> direction to the result that is returned to the user.
> 
> Functions should behave as if the infinite-precision result computed is 
> zero, infinity or NaN if and only if that is the mathematically correct 
> infinite-precision result.  They should behave as if the 
> infinite-precision result computed always has the same sign as the 
> mathematically correct result.
> 
> If the mathematical result has magnitude well below half the least 
> subnormal magnitude, the returned value should be either zero or the least 
> subnormal (in each case, with the correct sign), according to the current 
> rounding direction and with the underflow exception raised.  If the 
> mathematical result is more than a few ulp above the overflow threshold 
> for the current rounding direction, similarly the value returned should be 
> the appropriate overflow value for the current rounding direction, with 
> the overflow exception raised.
> 
> Where the mathematical result underflows and is not exactly representable 
> as a floating-point value, the underflow exception should be raised (so 
> there may be spurious underflow exceptions in cases where the underflowing 
> result is exact, but not missing underflow exceptions in cases where it is 
> inexact).
> 
> All the above applies to both real and complex parts, for complex 
> functions.
> 
> In addition to the above, for fully-specified functions such as fma, sqrt 
> and rint, the result and corresponding exceptions should be exactly 
> according to the fully-specified semantics of the function; there should 
> be no spurious or missing exceptions, including "inexact" exceptions.  
> This includes detection of tininess for underflow exceptions in the same 
> way as for other operations with binary floating-point results on the 
> architecture in question, in accordance with the requirements of IEEE 
> 754-2008.  (There are a few existing fma tests using 
> UNDERFLOW_EXCEPTION_BEFORE_ROUNDING to verify such architecture-specific 
> requirements for fma.)
> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com