Fw: [PATCH 0/3] libm: Clean up gamma functions

[Keith Packard]

Brian Inglis <Brian.Inglis@SystematicSw.ab.ca> writes:

> Last C17 public FDIS N2176 is available via:
>
> https://web.archive.org/web/20181230041359if_/http://www.open-std.org/jtc1/sc22/wg14/www/abq/c17_updated_proposed_fdis.pdf
>
> see 7.12.8.3 lgamma and 7.12.8.4 tgamma on pp181-182, 7.26 tgmath #5 lgamma,
> tgamma pp272-273, 7.31.1 complex clgamma, ctgamma p332, F10.5.3 lgamma and
> F.10.5.4 tgamma.

Thanks for the link -- the difference for lgamma is that in C99:

        A range error occurs if x is too large. A range error may occur
        if x is a negative integer or zero.

While in C17:

        A range error occurs if positive x is too large. A pole error
        may occur if x is a negative integer or zero.

Frustratingly, tgamma still returns different errors. In C99:

        A domain error or range error may occur if x is a negative
        integer or zero. A range error may occur if the magnitude of x
        is too large or too small.

While in C17:

        A domain error or pole error may occur if x is a negative
        integer or zero. A range error occurs if the magnitude of x is
        too large and may occur if the magnitude of x is too small.

C17 appears to have added this new 'pole' error, which seems to match
what the IEEE 754 calls a 'divideByZero exception'. Let's see if my
understanding of the various errors is correct:

IEEE            C17             POSIX   errno   Exception
divideByZero    Pole            Pole    ERANGE  FE_DIVBYZERO
invalid         Domain          Domain  EDOM    FE_INVALID
overflow        Range           Range   ERANGE  FE_OVERFLOW

POSIX is more specific than C17, but it does appear to have been updated
to follow that spec, including the term 'Pole Error' which wasn't in
C99.

As my tests all refer to the POSIX spec (which defines the
errno/exception values), I think they're correct. Here's the set of
values I'm testing for both tgamma and lgamma (I'm not testing gamma
currently; we need to sort out what it should be doing first):

Value                 tgamma                               lgamma
         POSIX   result   errno  exception      POSIX   result   errno   exception

+0       Pole   +INFINITY ERANGE FE_DIVBYZERO   Pole   +INFINITY ERANGE FE_DIVBYZERO
-0       Pole   -INFINITY ERANGE FE_DIVBYZERO   Pole   +INFINITY ERANGE FE_DIVBYZERO
-1       Domain    NAN    EDOM   FE_INVALID     Pole   +INFINITY ERANGE FE_DIVBYZERO
+3e38f   Range  +INFINITY ERANGE FE_OVERFLOW    Range  +INFINITY ERANGE FE_OVERFLOW
+1.7e308 Range  +INFINITY ERANGE FE_OVERFLOW    Range  +INFINITY ERANGE FE_OVERFLOW
-3e38f   Domain    NAN    EDOM   FE_INVALID     Pole   +INFINITY ERANGE FE_DIVBYZERO
-1.7e308 Domain    NAN    EDOM   FE_INVALID     Pole   +INFINITY ERANGE FE_DIVBYZERO
+inf     None   +INFINITY 0      0              None   +INFINITY 0      0
-inf     Domain    NAN    EDOM   FE_INVALID     None   +INFINITY 0      0

I have rationalized the differences in this way:

        Domain error means there is no defined limit for the function
        approaching the parameter. Pole error means that there *is* a
        defined limit in this case.

        For non-positive inputs, the limit of the value of the tgamma
        function depends on whether that limit is approached from above
        or below.

        For zero, we can separate out 'approach from below' and
        'approach from above' with the sign -- -0 might well be the
        result of an underflow from below, hence returning the value of
        the function as approached from below makes "sense", in this
        case -INFINITY. Similarly, +0 might well be underflow from
        above, so the function can return +INFINITY. In both cases, as
        the limit is well defined, this represents a Pole error rather
        than a Domain error, hence the differences in errno and exceptions.
        
        For negative integers, as the limit depends on whether the value
        is approached from below or above and we have no way of
        distinguishing these two cases from the input, the limit is not
        well defined, so we have a Domain error and return
        NAN/EDOM/FE_INVALID.

        For the lgamma function, the limit for non-positive integers is
        always +INFINITY because it returns ln(|Γ(x)|). This makes it
        well defined and hence a Pole error rather than a Domain error

        Huge negative values (-1e308) are negative integers, and hence
        treated that way, generating a Domain error for tgamma and Range
        error for lgamma

        +INFINITY input and +INFINITY output generate *no* error
        presumably because there's an assumption that the computation
        which resulted in +INFINITY being provided to these functions
        already raised a FE_OVERFLOW exception, and perhaps an ERANGE
        errno.

Please review the table above to see if you agree about what errno and
exception values each of the provided inputs should generate. I'd really
like to know if you can suggest additional test cases to use to validate
the correctness of these functions.

-- 
-keith
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/newlib/attachments/20200827/376450d3/attachment.sig>