[Bug libc/4943] Inconsistent rounding behaviour for sprintf and IEEE doubles

["paul at inet dot co dot za" <sourceware-bugzilla at sourceware dot org>]

------- Additional Comments From paul at inet dot co dot za  2007-09-22 10:55 -------
(In reply to comment #19)
> Subject: Re:  Inconsistent rounding behaviour for sprintf and IEEE doubles
> 
> On Sat, Sep 22, 2007 at 02:09:47PM +0000, paul at inet dot co dot za wrote:
> > 
> > ------- Additional Comments From paul at inet dot co dot za  2007-09-22
10:09 -------
> > (In reply to comment #17)
> > > The number 2597.525 does not exist.  There are no numbers between
> > > 2597.5249999999996362 and 2597.5250000000000909 in IEEE double.
> > > 
> > 
> > You are wrong.  All numbers between 
> >   2597.5249999999996362 and 
> >   2597.5250000000000909 are 
> >   2597.52500000000 stored to 15 decimals of precision.
> > 
> > You are incorrect to assume that there exists a floating point number such as 
> >   2597.5250000000000909
> > since this is simply
> >   2597.52500000000 stored with an error of 0.0000000000000909
> 
>   No it's not. It's what you _think_ it is. But not what the FPU knows
> it is.
> 
>   Just think about this one second. What if you had stored
> 2597.52500000000009 and not 2597.5250000 ? It would have made no
> difference, because for your FPU, 2597.5250000, 2597.52500000000009 and
> 2597.5250000000000909 have the same representation.
> 
>   So when you ask for the rounded value the libc has to round
> 2597.5250000000000909, not 2597.52500000000009, not 2597.525 but
> 2597.5250000000000909 because it's what is stored in your double value.
> 
>   So it rounds to XXX.53, because default rounding scheme is Nearest
> even.
> 
>   0.125 (or 0.625 or XXXX.625) is another story, because 0.125 indeed is
> an existing value in the IEEE world, but it falls right in between 0.12
> and 0.13. And here, IEEE says that "Nearest even" rounding will round
> half of the time up, half of the time down. For 0.125 it rounds down,
> try for 0.0125 it rounds up. And this is done on purpose to spread the
> errors evenly accross different computations.
> 
>   STOP ARGUING that 2597.5250000000000909 is 2597.52500000000 stored to
> 15 decimals of precision. You're uterly wrong. 2597.5250* does not
> exists in IEEE so once stored, it becomes 2597.5250000000000909. The FPU
> has no way to know you meant 2597.525, it sees 2597.5250000000000909,
> and 2597.5250000000000909 only. Not {2597.5250000000000909 but only the
> 15 first digits are meaningful}.

This is utter rubbish.  Of course the FPU, and you too, have a way of knowing
the precision.  It is stated as per the specification as 15 decimals.  Assuming
anything after the 15th digit for the purposes of conversion to text is
nonsense.  The error portion (as stored after the 15th digit) is useful only
when using the number in further calculations, where the rounding to closest or
rounding to nearest even on boundary applies.

Was all the shouting to try to stop me from pointing out something important
that you folks are clearly missing? No amount of shouting is going to change the
fact that IEEE doubles store at most 15 digits precisely.  How about you pass
this along to someone you know who may be competent in mathematics, who will
verify what I am saying, rather than just sprouting at the gills.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4943

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.