Re: openlibm and libm: mutation testing

[Joseph Myers <joseph at codesourcery dot com>]

I would suggest looking at glibc's libm tests (60 MB of expectations 
generated using MPFR and MPC, plus many tests with manually maintained 
expectations for special cases of particular functions).  In principle it 
should be possible to test other libm implementations using them, although 
certainly at present there are lots of dependencies on glibc-specific 
interfaces and glibc's rules for error handling and accuracy requirements.  
Where the glibc tests do not provide sufficient code coverage, whether for 
glibc's implementations (NB many functions have several different 
implementations used on different architectures, so several architectures 
would need testing to assess coverage well) or for other libms' 
implementations of the same functions, additional test inputs to improve 
coverage would make sense.

I would point out that there are many cases in Sun fdlibm, and thus in 
other implementations derived from it, where there is a lot of 
arbitraryness in the particular value or operation used to e.g. force an 
exception; if the point of doing C * C is to generate "overflow", it's 
expected that you can change the (large) value of C with no change to the 
results of the function; if the point of doing C + x is to generate 
"inexact" (not in glibc's goals for most functions, but various such code 
has yet to be cleaned up), you can change C, or change the operation + to 
-, without any change to the results of the function being expected; if a 
function uses a polynomial approximation, many changes to low-order 
coefficients may result in only small changes to the result of the 
function, within its accuracy goals.  So the fact that mutating the code 
does not affect test results does not necessarily indicate any problem 
with test coverage.

-- 
Joseph S. Myers
joseph@codesourcery.com