Bug 17907

Summary: fma modifies and does not reset SSE rounding mode on 32bit machines
Product: glibc Reporter: Simon Byrne <simonbyrne>
Component: mathAssignee: Not yet assigned to anyone <unassigned>
Status: RESOLVED DUPLICATE    
Severity: normal CC: simonbyrne
Priority: P2 Flags: fweimer: security-
Version: 2.19   
Target Milestone: ---   
Host: Target:
Build: Last reconfirmed:
Attachments: fma rounding mode test function

Description Simon Byrne 2015-01-30 11:59:41 UTC
Created attachment 8091 [details]
fma rounding mode test function

The software fma function modifies the rounding mode for SSE floating point instructions (to round-toward-zero), but does not reset it correctly, on 32bit (686) processors. The x87 rounding mode is unaffected.

Attached is a sample program: compile on a 32bit machine with gcc:

$ gcc fmacheck.c -o fmacheck -lm -msse2 -mfpmath=sse

or clang:

$ clang fmacheck.c -o fmacheck -lm

When run as

$ ./fmacheck 0x1.0000000000001p+0

it should give
0x1.ffffffffffffep-1
if rounding mode is the default (round-to-nearest). If the rounding mode has been incorrectly modified to be round-toward-zero, then you will get
0x1.ffffffffffffdp-1
Comment 1 Joseph Myers 2015-01-30 15:14:18 UTC
This looks like it would have been a consequence of bug 16064, fixed for 2.20.

*** This bug has been marked as a duplicate of bug 16064 ***
Comment 2 jsm-csl@polyomino.org.uk 2015-01-30 15:21:53 UTC
(More specifically: i386 uses sysdeps/ieee754/ldbl-96/s_fma.c, which saves 
state with feholdexcept, sets the rounding mode (twice) with fesetround, 
and then restores state, with exceptions merged, with feupdateenv.  This 
was the sort of pattern that was mishandled before bug 16064 was fixed, 
because fesetround sets the SSE rounding mode but the fenv_t functions 
used not to handle it.)
Comment 3 Simon Byrne 2015-01-30 16:15:06 UTC
Great, thanks for letting me know.