[PATCH v3 1/3] math: Make rsqrtf correctly rounded
Adhemerval Zanella Netto
adhemerval.zanella@linaro.org
Wed Nov 4 11:57:26 GMT 2026
On 03/11/26 17:53, Wilco Dijkstra wrote:
> Hi Adhemerval,
>
> This is much faster than the previous version - however it adds a lot of complex
> checks, so the rounding mode variant still beats it...
>
> + uint32_t m = ix << 8;
> +
> + /* The expression (1.0 / x) * sqrt (x) fails to return correct rounding
> + for x=0x1.7bf15p-128*2^(2k) and x=0x1.ab6f7ap-125*2^(2k) with k>=0, for
> + rndu (where the returned value is 1 ulp too large). */
> + if (__glibc_unlikely (ix == 0 || ix >= EXPONENT_MASK || ix == 0x2f7e2au
> + || m == 0xbdf8a800u || m == 0x55b7bd00u))
> + {
>
> We can merge one of these checks with the exponent check and use 16 bits for the
> other checks (which is accurate enough). The hot path goes from 23 instructions to 18
> and TP improves by 8%, latency by 0.5%.
>
> uint32_t m = (ix >> 8) & 65535;
> if (__glibc_unlikely ((ix - 0x300000) >= EXPONENT_MASK - 0x300000
> || m == 0xbdf8 || m == 0x55b7))
>
> This gets almost the same performance as my version with the rounding mode check,
> except that denormals are ~7% slower - but denormals are rare.
> This still passes in all rounding modes of course.
I recall I tried something similar, but I double checked this on different CPUs
with gcc 16 and results are mixed:
* latency
- input: [0.5,2)
master patched improvement
x86_64 28.0730 27.9316 0.51%
x86_64-v2 28.2768 27.9831 1.05%
x86_64-v3 25.6674 25.6647 0.01%
aarch64 10.7039 10.6315 0.68%
powerpc 8.2174 8.0844 1.65%
loongarch 3.2030 3.6530 -12.32%
- input: normal
master patched improvement
x86_64 27.9410 27.9027 0.14%
x86_64-v2 28.3114 27.9170 1.41%
x86_64-v3 25.6744 25.7589 -0.33%
aarch64 10.7022 10.6341 0.64%
powerpc 8.2174 8.0885 1.59%
loongarch 3.2029 3.6532 -12.33%
- input: subnormal
master patched improvement
x86_64 27.8963 30.4178 -8.29%
x86_64-v2 28.2242 30.3693 -7.06%
x86_64-v3 25.7171 29.2883 -12.19%
aarch64 10.7048 13.2136 -18.99%
powerpc 8.2179 9.1850 -10.53%
loongarch 3.2526 4.6538 -30.11%
* reciprocal-throughput
- input: [0.5,2)
master patched improvement
x86_64 9.8835 9.8891 -0.06%
x86_64-v2 10.3604 9.8784 4.88%
x86_64-v3 9.9397 10.4478 -4.86%
aarch64 4.7314 5.3203 -11.07%
powerpc 2.4636 2.3760 3.69%
loongarch 1.8517 2.0390 -9.18%
- input: normal
master patched improvement
x86_64 9.8684 9.8713 -0.03%
x86_64-v2 10.3600 9.8702 4.96%
x86_64-v3 9.9435 10.4705 -5.03%
aarch64 4.7301 5.3018 -10.78%
powerpc 2.4640 2.3906 3.07%
loongarch 1.8515 2.0391 -9.20%
- input: subnormal
master patched improvement
x86_64 9.8594 10.6623 -7.53%
x86_64-v2 10.3353 10.4596 -1.19%
x86_64-v3 9.9671 10.6234 -6.18%
aarch64 4.7308 6.9790 -32.21%
powerpc 2.4645 2.6751 -7.87%
loongarch 1.9018 2.1028 -9.56%
* x86_64-v3: Ryzen9 5900X
* aarch64: Neoverse-N1
* powerpc: POWER10
* loongarch: Loongson-3C5000L-LL
It does generate shorted code patch sequences through (instructions):
master patched
aarch64 23 18
x86_64-v3 18 17
powerpc64le (POWER8) 26 22
powerpc64le (POWER10) 23 19
loongarch64 22 18
riscv64 (rv64gc) 33 25
riscv64 (RVA23) 32 24
More information about the Libc-alpha
mailing list