incorrectly rounded square root
Paul Zimmermann
Paul.Zimmermann@inria.fr
Thu Jun 3 10:21:46 GMT 2021
Hi Jeff,
I've investigated and found (partially) the cause of the problem. When I
compile my program, gcc uses the system fenv.h, which contains FE_TONEAREST=0,
FE_DOWNWARD=0x400, FE_UPWARD=0x800, and FE_TOWARDZERO=0xc00 in
/usr/include/bits/fenv.h.
However, the values in x86_64/newlib/targ-include/sys/fenv.h are different:
FE_TONEAREST=0, FE_DOWNWARD=1, FE_UPWARD=2, FE_TOWARDZERO=3. Thus the test
at the beginning of fesetround in newlib/libm/machine/x86_64/fenv.c fails
except for round=FE_TONEAREST:
/* Will succeed for any valid value of the input parameter. */
if (round < FE_TONEAREST || round > FE_TOWARDZERO)
return EINVAL;
enter fesetround, round=0 use_sse=1
enter fesetround, round=3072 use_sse=1
enter fesetround, round=2048 use_sse=1
enter fesetround, round=1024 use_sse=1
This explains why the other modes are not set.
A first question is why Newlib and the system libm (here glibc) use different
constants for the rounding modes.
Now if I compile with -I/tmp/x86_64/include (where the Newlib fenv.h is
installed) I get:
enter fesetround, round=0 use_sse=1
enter fesetround, round=3 use_sse=1
enter fesetround, round=2 use_sse=1
enter fesetround, round=1 use_sse=1
but I still do get the same results whatever the rounding mode!
If I look at the cw and mxcsr registers like in Newlib fenv.c, I get with glibc
(after the fesetround call):
fegetround: 0
cw=895 mxcsr=8064
RNDN: 0x1.ff83fp+63
fegetround: 3072
cw=3967 mxcsr=32672
RNDZ: 0x1.ff83eep+63
fegetround: 2048
cw=2943 mxcsr=24480
RNDU: 0x1.ff83fp+63
fegetround: 1024
cw=1919 mxcsr=16288
RNDD: 0x1.ff83eep+63
and with Newlib:
fegetround: 0
cw=895 mxcsr=8064
RNDN: 0x1.ff83fp+63
fegetround: 3
cw=3967 mxcsr=32640
RNDZ: 0x1.ff83fp+63
fegetround: 2
cw=2943 mxcsr=24448
RNDU: 0x1.ff83fp+63
fegetround: 1
cw=1919 mxcsr=16256
RNDD: 0x1.ff83fp+63
The cw values do match but not the mxcsr values.
Best regards,
Paul
> Something weird is going on. I condensed the newlib code into the
> following files and when I compile and run the test,
> it works as expected:
>
> gcc -std=c99 -g3 -O0 test_sqrt.c fesetround.c ef_sqrt.c
> [jjohnstn@localhost shared_x86]$ ./a.out
> RNDN: 0x1.ff83fp+63
> RNDZ: 0x1.ff83eep+63
> RNDU: 0x1.ff83fp+63
> RNDD: 0x1.ff83eep+63
>
> Can you verify that you are calling the fesetround in fenv.c and what
> configuration call did you use for building newlib?
More information about the Newlib
mailing list