[PATCH v3 1/3] math: Make rsqrtf correctly rounded

Adhemerval Zanella Netto adhemerval.zanella@linaro.org
Wed Nov 4 11:57:26 GMT 2026



On 03/11/26 17:53, Wilco Dijkstra wrote:
> Hi Adhemerval,
> 
> This is much faster than the previous version - however it adds a lot of complex
> checks, so the rounding mode variant still beats it...
> 
> +  uint32_t m = ix << 8;
> +
> +  /* The expression (1.0 / x) * sqrt (x) fails to return correct rounding
> +     for x=0x1.7bf15p-128*2^(2k) and x=0x1.ab6f7ap-125*2^(2k) with k>=0, for
> +     rndu (where the returned value is 1 ulp too large).  */
> +  if (__glibc_unlikely (ix == 0 || ix >= EXPONENT_MASK || ix == 0x2f7e2au
> +			|| m == 0xbdf8a800u || m == 0x55b7bd00u))
> +    {
> 
> We can merge one of these checks with the exponent check and use 16 bits for the
> other checks (which is accurate enough). The hot path goes from 23 instructions to 18
> and TP improves by 8%, latency by 0.5%.
> 
>   uint32_t m = (ix >> 8) & 65535;
>   if (__glibc_unlikely ((ix - 0x300000) >= EXPONENT_MASK - 0x300000
>                         || m == 0xbdf8 || m == 0x55b7))
> 
> This gets almost the same performance as my version with the rounding mode check,
> except that denormals are ~7% slower - but denormals are rare.
> This still passes in all rounding modes of course.
I recall I tried something similar, but I double checked this on different CPUs
with gcc 16 and results are mixed:

* latency
- input: [0.5,2)
                 master       patched   improvement
x86_64          28.0730       27.9316         0.51%
x86_64-v2       28.2768       27.9831         1.05%
x86_64-v3       25.6674       25.6647         0.01%
aarch64         10.7039       10.6315         0.68%
powerpc          8.2174        8.0844         1.65%
loongarch        3.2030        3.6530       -12.32%

- input: normal
                 master       patched   improvement
x86_64          27.9410       27.9027         0.14%
x86_64-v2       28.3114       27.9170         1.41%
x86_64-v3       25.6744       25.7589        -0.33%
aarch64         10.7022       10.6341         0.64%
powerpc          8.2174        8.0885         1.59%
loongarch        3.2029        3.6532       -12.33%

- input: subnormal
                 master       patched   improvement
x86_64          27.8963       30.4178        -8.29%
x86_64-v2       28.2242       30.3693        -7.06%
x86_64-v3       25.7171       29.2883       -12.19%
aarch64         10.7048       13.2136       -18.99%
powerpc          8.2179        9.1850       -10.53%
loongarch        3.2526        4.6538       -30.11%

* reciprocal-throughput
- input: [0.5,2)
                 master       patched   improvement
x86_64           9.8835        9.8891        -0.06%
x86_64-v2       10.3604        9.8784         4.88%
x86_64-v3        9.9397       10.4478        -4.86%
aarch64          4.7314        5.3203       -11.07%
powerpc          2.4636        2.3760         3.69%
loongarch        1.8517        2.0390        -9.18%

- input: normal
                 master       patched   improvement
x86_64           9.8684        9.8713        -0.03%
x86_64-v2       10.3600        9.8702         4.96%
x86_64-v3        9.9435       10.4705        -5.03%
aarch64          4.7301        5.3018       -10.78%
powerpc          2.4640        2.3906         3.07%
loongarch        1.8515        2.0391        -9.20%

- input: subnormal
                 master       patched   improvement
x86_64           9.8594       10.6623        -7.53%
x86_64-v2       10.3353       10.4596        -1.19%
x86_64-v3        9.9671       10.6234        -6.18%
aarch64          4.7308        6.9790       -32.21%
powerpc          2.4645        2.6751        -7.87%
loongarch        1.9018        2.1028        -9.56%

* x86_64-v3: Ryzen9 5900X
* aarch64:   Neoverse-N1
* powerpc:   POWER10
* loongarch: Loongson-3C5000L-LL 

It does generate shorted code patch sequences through (instructions):

                              master   patched
      aarch64                     23        18
      x86_64-v3                   18        17
      powerpc64le (POWER8)        26        22
      powerpc64le (POWER10)       23        19
      loongarch64                 22        18
      riscv64 (rv64gc)            33        25
      riscv64 (RVA23)             32        24


More information about the Libc-alpha mailing list