[Bug libc/4943] Inconsistent rounding behaviour for sprintf and IEEE doubles

[paul at inet dot co dot za]

------- Additional Comments From paul at inet dot co dot za  2007-09-22 11:05 -------
(In reply to comment #19)
>   0.125 (or 0.625 or XXXX.625) is another story, because 0.125 indeed is
> an existing value in the IEEE world, but it falls right in between 0.12
> and 0.13. And here, IEEE says that "Nearest even" rounding will round
> half of the time up, half of the time down. For 0.125 it rounds down,
> try for 0.0125 it rounds up. And this is done on purpose to spread the
> errors evenly accross different computations.

This part warrants a separate response.

0.125 is an "existing value" in your definition as it is precisely stored. 
Because of the fact that it is precisely stored, the value does not require
rounding in the IEEE sense at all.  IEEE rounding only comes into effect when
trying to store the result of a calculation, where that result cannot in turn be
stored precisely.  The FPU takes care of all of that, and should not concern you.

So, for example, say you divide the precisely stored value 0.125 by 10.  The FPU
might not be able to store the result 0.0125 precisely, and therefore would FPU
default rounding before storing the result.

This has absolutely nothing to do with conversion to text.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4943

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.