[PATCH] aarch64: MTE compatible memrchr
Alex Butler
Alex.Butler@arm.com
Tue Jun 9 16:08:07 GMT 2020
This patch adds an MTE compatible implementation of memrchr.
Please see the benchmark results for the performance uplift.
| length | position | alignment | uplift A72 | uplift A53 | uplift N1 |
|--------+----------+-----------+-------------|-------------|-------------|
| 2048 | 32 | 0 | 0.87x | 0.81x | 0.87x |
| 256 | 64 | 1 | 1.03x | 0.93x | 0.99x |
| 2048 | 32 | 0 | 0.88x | 0.81x | 0.87x |
| 256 | 64 | 1 | 1.06x | 0.93x | 1.01x |
| 256 | 1 | 0 | 0.95x | 0.85x | 0.89x |
| 256 | 1 | 0 | 0.93x | 0.85x | 0.91x |
| 256 | 1 | 1 | 1.01x | 0.92x | 1.03x |
| 256 | 1 | 1 | 1.00x | 0.91x | 1.01x |
| 2048 | 64 | 0 | 0.89x | 0.81x | 0.88x |
| 256 | 64 | 2 | 1.06x | 0.93x | 0.99x |
| 2048 | 64 | 0 | 0.89x | 0.81x | 0.88x |
| 256 | 64 | 2 | 1.07x | 0.93x | 0.99x |
| 256 | 2 | 0 | 0.94x | 0.85x | 0.94x |
| 256 | 2 | 0 | 0.93x | 0.85x | 0.94x |
| 256 | 2 | 2 | 1.01x | 0.91x | 1.02x |
| 256 | 2 | 2 | 1.01x | 0.91x | 1.02x |
| 2048 | 128 | 0 | 0.89x | 0.81x | 0.87x |
| 256 | 64 | 3 | 1.04x | 0.93x | 1.02x |
| 2048 | 128 | 0 | 0.88x | 0.81x | 0.86x |
| 256 | 64 | 3 | 1.04x | 0.93x | 1.00x |
| 256 | 3 | 0 | 0.93x | 0.85x | 0.92x |
| 256 | 3 | 0 | 0.93x | 0.85x | 0.94x |
| 256 | 3 | 3 | 1.01x | 0.91x | 1.00x |
| 256 | 3 | 3 | 1.01x | 0.91x | 0.99x |
| 2048 | 256 | 0 | 0.89x | 0.81x | 0.87x |
| 256 | 64 | 4 | 1.05x | 0.93x | 0.99x |
| 2048 | 256 | 0 | 0.89x | 0.81x | 0.87x |
| 256 | 64 | 4 | 1.03x | 0.93x | 1.01x |
| 256 | 4 | 0 | 0.93x | 0.85x | 0.91x |
| 256 | 4 | 0 | 0.91x | 0.85x | 0.90x |
| 256 | 4 | 4 | 1.00x | 0.91x | 1.02x |
| 256 | 4 | 4 | 1.00x | 0.91x | 1.03x |
| 2048 | 512 | 0 | 0.91x | 0.81x | 0.87x |
| 256 | 64 | 5 | 1.01x | 0.93x | 0.99x |
| 2048 | 512 | 0 | 0.91x | 0.81x | 0.87x |
| 256 | 64 | 5 | 1.03x | 0.93x | 0.99x |
| 256 | 5 | 0 | 0.95x | 0.85x | 0.92x |
| 256 | 5 | 0 | 0.94x | 0.85x | 0.91x |
| 256 | 5 | 5 | 0.99x | 0.92x | 1.01x |
| 256 | 5 | 5 | 0.99x | 0.91x | 1.02x |
| 2048 | 1024 | 0 | 0.92x | 0.82x | 0.86x |
| 256 | 64 | 6 | 1.05x | 0.93x | 1.00x |
| 2048 | 1024 | 0 | 0.92x | 0.82x | 0.86x |
| 256 | 64 | 6 | 1.01x | 0.93x | 1.01x |
| 256 | 6 | 0 | 0.94x | 0.85x | 0.91x |
| 256 | 6 | 0 | 0.92x | 0.85x | 0.91x |
| 256 | 6 | 6 | 1.00x | 0.91x | 0.99x |
| 256 | 6 | 6 | 1.00x | 0.92x | 0.98x |
| 2048 | 2048 | 0 | 0.87x | 0.81x | 0.87x |
| 256 | 64 | 7 | 1.04x | 0.93x | 1.00x |
| 2048 | 2048 | 0 | 0.87x | 0.81x | 0.88x |
| 256 | 64 | 7 | 1.03x | 0.93x | 0.99x |
| 256 | 7 | 0 | 0.93x | 0.85x | 0.92x |
| 256 | 7 | 0 | 0.93x | 0.85x | 0.92x |
| 256 | 7 | 7 | 1.00x | 0.91x | 1.00x |
| 256 | 7 | 7 | 1.00x | 0.91x | 1.02x |
| 2 | 1 | 0 | 1.26x | 1.82x | 1.73x |
| 2 | 1 | 0 | 1.26x | 1.82x | 1.74x |
| 2 | 1 | 1 | 1.26x | 1.78x | 1.73x |
| 2 | 1 | 1 | 1.26x | 1.76x | 1.73x |
| 2 | 1 | 0 | 1.26x | 1.78x | 1.72x |
| 2 | 2 | 0 | 1.36x | 1.50x | 1.72x |
| 3 | 2 | 0 | 1.26x | 1.77x | 1.73x |
| 3 | 2 | 0 | 1.26x | 1.75x | 1.53x |
| 3 | 2 | 2 | 1.26x | 1.78x | 1.52x |
| 3 | 2 | 2 | 1.26x | 1.77x | 1.54x |
| 3 | 1 | 0 | 1.26x | 1.77x | 1.54x |
| 3 | 2 | 0 | 1.26x | 1.82x | 1.54x |
| 4 | 3 | 0 | 1.26x | 1.80x | 1.54x |
| 4 | 3 | 0 | 1.26x | 1.82x | 1.54x |
| 4 | 3 | 3 | 1.26x | 1.76x | 1.55x |
| 4 | 3 | 3 | 1.26x | 1.78x | 1.54x |
| 4 | 1 | 0 | 1.26x | 1.80x | 1.54x |
| 4 | 2 | 0 | 1.26x | 1.78x | 1.55x |
| 5 | 4 | 0 | 1.26x | 1.78x | 1.54x |
| 5 | 4 | 0 | 1.26x | 1.82x | 1.54x |
| 5 | 4 | 4 | 1.26x | 1.77x | 1.53x |
| 5 | 4 | 4 | 1.26x | 1.75x | 1.54x |
| 5 | 1 | 0 | 1.26x | 1.77x | 1.54x |
| 5 | 2 | 0 | 1.26x | 1.81x | 1.54x |
| 6 | 5 | 0 | 1.26x | 1.78x | 1.84x |
| 6 | 5 | 0 | 1.26x | 1.75x | 1.86x |
| 6 | 5 | 5 | 1.26x | 1.78x | 1.73x |
| 6 | 5 | 5 | 1.26x | 1.75x | 1.73x |
| 6 | 1 | 0 | 1.26x | 1.77x | 1.74x |
| 6 | 2 | 0 | 1.26x | 1.77x | 1.73x |
| 7 | 6 | 0 | 1.26x | 1.76x | 1.54x |
| 7 | 6 | 0 | 1.26x | 1.78x | 1.60x |
| 7 | 6 | 6 | 1.26x | 1.76x | 1.60x |
| 7 | 6 | 6 | 1.26x | 1.82x | 1.61x |
| 7 | 1 | 0 | 1.26x | 1.80x | 1.60x |
| 7 | 2 | 0 | 1.26x | 1.82x | 1.60x |
| 8 | 7 | 0 | 1.26x | 1.80x | 1.54x |
| 8 | 7 | 0 | 1.26x | 1.82x | 1.54x |
| 8 | 7 | 7 | 1.26x | 1.76x | 1.53x |
| 8 | 7 | 7 | 1.26x | 1.78x | 1.53x |
| 8 | 1 | 0 | 1.26x | 1.76x | 1.54x |
| 8 | 2 | 0 | 1.26x | 1.82x | 1.54x |
| 9 | 8 | 0 | 1.26x | 1.82x | 1.83x |
| 9 | 8 | 0 | 1.26x | 1.76x | 1.83x |
| 9 | 8 | 0 | 1.26x | 1.82x | 1.84x |
| 9 | 8 | 0 | 1.26x | 1.82x | 1.85x |
| 9 | 1 | 0 | 1.26x | 1.79x | 1.83x |
| 9 | 2 | 0 | 1.26x | 1.82x | 1.84x |
| 10 | 9 | 0 | 1.26x | 1.78x | 1.83x |
| 10 | 9 | 0 | 1.26x | 1.78x | 1.85x |
| 10 | 9 | 1 | 1.26x | 1.82x | 1.84x |
| 10 | 9 | 1 | 1.26x | 1.82x | 1.84x |
| 10 | 1 | 0 | 1.26x | 1.82x | 1.86x |
| 10 | 2 | 0 | 1.26x | 1.82x | 1.83x |
| 11 | 10 | 0 | 1.26x | 1.82x | 1.83x |
| 11 | 10 | 0 | 1.26x | 1.75x | 1.85x |
| 11 | 10 | 2 | 1.26x | 1.77x | 1.85x |
| 11 | 10 | 2 | 1.26x | 1.75x | 1.85x |
| 11 | 1 | 0 | 1.26x | 1.77x | 1.84x |
| 11 | 2 | 0 | 1.26x | 1.81x | 1.92x |
| 12 | 11 | 0 | 1.26x | 1.82x | 1.92x |
| 12 | 11 | 0 | 1.26x | 1.78x | 1.83x |
| 12 | 11 | 3 | 1.26x | 1.76x | 1.85x |
| 12 | 11 | 3 | 1.26x | 1.78x | 1.84x |
| 12 | 1 | 0 | 1.26x | 1.76x | 1.84x |
| 12 | 2 | 0 | 1.26x | 1.82x | 1.84x |
| 13 | 12 | 0 | 1.26x | 1.82x | 1.84x |
| 13 | 12 | 0 | 1.26x | 1.75x | 1.85x |
| 13 | 12 | 4 | 1.26x | 1.78x | 1.92x |
| 13 | 12 | 4 | 1.26x | 1.75x | 1.81x |
| 13 | 1 | 0 | 1.26x | 1.78x | 1.81x |
| 13 | 2 | 0 | 1.26x | 1.82x | 1.73x |
| 14 | 13 | 0 | 1.26x | 1.82x | 1.72x |
| 14 | 13 | 0 | 1.26x | 1.76x | 1.73x |
| 14 | 13 | 5 | 1.26x | 1.78x | 1.85x |
| 14 | 13 | 5 | 1.26x | 1.76x | 1.84x |
| 14 | 1 | 0 | 1.26x | 1.78x | 1.84x |
| 14 | 2 | 0 | 1.26x | 1.82x | 1.84x |
| 15 | 14 | 0 | 1.26x | 1.82x | 1.86x |
| 15 | 14 | 0 | 1.26x | 1.79x | 1.83x |
| 15 | 14 | 6 | 1.26x | 1.77x | 1.84x |
| 15 | 14 | 6 | 1.26x | 1.79x | 1.92x |
| 15 | 1 | 0 | 1.26x | 1.81x | 1.91x |
| 15 | 2 | 0 | 1.26x | 1.82x | 1.84x |
| 16 | 15 | 0 | 1.26x | 1.82x | 1.55x |
| 16 | 15 | 0 | 1.26x | 1.82x | 1.54x |
| 16 | 15 | 7 | 1.26x | 1.78x | 1.54x |
| 16 | 15 | 7 | 1.26x | 1.76x | 1.53x |
| 16 | 1 | 0 | 1.26x | 1.82x | 1.54x |
| 16 | 2 | 0 | 1.26x | 1.82x | 1.52x |
| 17 | 16 | 0 | 1.26x | 1.82x | 1.54x |
| 17 | 16 | 0 | 1.26x | 1.77x | 1.54x |
| 17 | 16 | 0 | 1.26x | 1.82x | 1.54x |
| 17 | 16 | 0 | 1.26x | 1.77x | 1.54x |
| 17 | 1 | 0 | 1.17x | 1.17x | 1.07x |
| 17 | 2 | 0 | 1.17x | 1.19x | 1.06x |
| 18 | 17 | 0 | 1.26x | 1.82x | 1.47x |
| 18 | 17 | 0 | 1.26x | 1.82x | 1.53x |
| 18 | 17 | 1 | 1.26x | 1.78x | 1.53x |
| 18 | 17 | 1 | 1.26x | 1.77x | 1.54x |
| 18 | 1 | 0 | 1.17x | 1.18x | 1.06x |
| 18 | 2 | 0 | 1.17x | 1.19x | 1.06x |
| 19 | 18 | 0 | 1.26x | 1.82x | 1.54x |
| 19 | 18 | 0 | 1.26x | 1.79x | 1.53x |
| 19 | 18 | 2 | 1.26x | 1.77x | 1.54x |
| 19 | 18 | 2 | 1.26x | 1.82x | 1.45x |
| 19 | 1 | 0 | 1.17x | 1.16x | 1.25x |
| 19 | 2 | 0 | 1.17x | 1.18x | 1.26x |
| 20 | 19 | 0 | 1.26x | 1.82x | 1.84x |
| 20 | 19 | 0 | 1.26x | 1.82x | 1.84x |
| 20 | 19 | 3 | 1.26x | 1.78x | 1.82x |
| 20 | 19 | 3 | 1.26x | 1.78x | 1.84x |
| 20 | 1 | 0 | 1.17x | 1.16x | 1.26x |
| 20 | 2 | 0 | 1.17x | 1.19x | 1.25x |
| 21 | 20 | 0 | 1.26x | 1.82x | 1.73x |
| 21 | 20 | 0 | 1.26x | 1.82x | 1.73x |
| 21 | 20 | 4 | 1.26x | 1.78x | 1.74x |
| 21 | 20 | 4 | 1.26x | 1.76x | 1.72x |
| 21 | 1 | 0 | 1.17x | 1.16x | 1.26x |
| 21 | 2 | 0 | 1.17x | 1.19x | 1.26x |
| 22 | 21 | 0 | 1.26x | 1.78x | 1.54x |
| 22 | 21 | 0 | 1.26x | 1.76x | 1.54x |
| 22 | 21 | 5 | 1.26x | 1.78x | 1.53x |
| 22 | 21 | 5 | 1.26x | 1.76x | 1.53x |
| 22 | 1 | 0 | 1.17x | 1.18x | 1.06x |
| 22 | 2 | 0 | 1.17x | 1.19x | 1.25x |
| 23 | 22 | 0 | 1.26x | 1.78x | 1.83x |
| 23 | 22 | 0 | 1.26x | 1.82x | 1.85x |
| 23 | 22 | 6 | 1.26x | 1.78x | 1.85x |
| 23 | 22 | 6 | 1.26x | 1.78x | 1.85x |
| 23 | 1 | 0 | 1.17x | 1.14x | 1.24x |
| 23 | 2 | 0 | 1.17x | 1.19x | 1.22x |
| 24 | 23 | 0 | 1.26x | 1.82x | 1.83x |
| 24 | 23 | 0 | 1.26x | 1.82x | 1.54x |
| 24 | 23 | 7 | 1.26x | 1.77x | 1.55x |
| 24 | 23 | 7 | 1.26x | 1.79x | 1.54x |
| 24 | 1 | 0 | 1.17x | 1.15x | 1.06x |
| 24 | 2 | 0 | 1.17x | 1.19x | 1.06x |
| 25 | 24 | 0 | 1.26x | 1.82x | 1.53x |
| 25 | 24 | 0 | 1.26x | 1.79x | 1.55x |
| 25 | 24 | 0 | 1.26x | 1.76x | 1.54x |
| 25 | 24 | 0 | 1.26x | 1.79x | 1.84x |
| 25 | 1 | 0 | 1.17x | 1.15x | 1.24x |
| 25 | 2 | 0 | 1.17x | 1.19x | 1.26x |
| 26 | 25 | 0 | 1.26x | 1.82x | 1.53x |
| 26 | 25 | 0 | 1.26x | 1.78x | 1.54x |
| 26 | 25 | 1 | 1.26x | 1.76x | 1.55x |
| 26 | 25 | 1 | 1.26x | 1.78x | 1.54x |
| 26 | 1 | 0 | 1.17x | 1.14x | 1.06x |
| 26 | 2 | 0 | 1.17x | 1.19x | 1.07x |
| 27 | 26 | 0 | 1.26x | 1.82x | 1.85x |
| 27 | 26 | 0 | 1.26x | 1.76x | 1.84x |
| 27 | 26 | 2 | 1.26x | 1.78x | 1.84x |
| 27 | 26 | 2 | 1.26x | 1.76x | 1.85x |
| 27 | 1 | 0 | 1.17x | 1.16x | 1.26x |
| 27 | 2 | 0 | 1.17x | 1.19x | 1.18x |
| 28 | 27 | 0 | 1.26x | 1.82x | 1.84x |
| 28 | 27 | 0 | 1.26x | 1.82x | 1.84x |
| 28 | 27 | 3 | 1.26x | 1.78x | 1.79x |
| 28 | 27 | 3 | 1.26x | 1.76x | 1.73x |
| 28 | 1 | 0 | 1.17x | 1.16x | 1.26x |
| 28 | 2 | 0 | 1.17x | 1.19x | 1.27x |
| 29 | 28 | 0 | 1.26x | 1.82x | 1.80x |
| 29 | 28 | 0 | 1.26x | 1.76x | 1.81x |
| 29 | 28 | 4 | 1.16x | 1.54x | 1.58x |
| 29 | 28 | 4 | 1.16x | 1.54x | 1.53x |
| 29 | 1 | 0 | 1.17x | 1.14x | 1.23x |
| 29 | 2 | 0 | 1.17x | 1.19x | 1.26x |
| 30 | 29 | 0 | 1.26x | 1.82x | 1.84x |
| 30 | 29 | 0 | 1.26x | 1.78x | 1.83x |
| 30 | 29 | 5 | 1.15x | 1.53x | 1.57x |
| 30 | 29 | 5 | 1.15x | 1.54x | 1.52x |
| 30 | 1 | 0 | 1.17x | 1.19x | 1.26x |
| 30 | 2 | 0 | 1.17x | 1.19x | 1.26x |
| 31 | 30 | 0 | 1.26x | 1.82x | 1.83x |
| 31 | 30 | 0 | 1.26x | 1.75x | 1.84x |
| 31 | 30 | 6 | 1.16x | 1.53x | 1.58x |
| 31 | 30 | 6 | 1.15x | 1.54x | 1.53x |
| 31 | 1 | 0 | 1.17x | 1.17x | 1.26x |
| 31 | 2 | 0 | 1.17x | 1.19x | 1.26x |
| 32 | 31 | 0 | 1.19x | 1.61x | 1.63x |
| 32 | 31 | 0 | 1.19x | 1.61x | 1.62x |
| 32 | 31 | 7 | 1.15x | 1.54x | 1.58x |
| 32 | 31 | 7 | 1.16x | 1.54x | 1.52x |
| 32 | 1 | 0 | 1.11x | 1.05x | 1.11x |
| 32 | 2 | 0 | 1.11x | 1.05x | 1.12x |
This patch passes the tests with no regressions.
8< --- 8< --- 8<
Add support for MTE to memrchr. Regression tested with xcheck and benchmarked
with glibc's benchtests on the Cortex-A53, Cortex-A72, and Neoverse N1.
The existing implementation assumes that any access to the pages in which the
string resides is safe. This assumption is not true when MTE is enabled. This
patch updates the algorithm to ensure that accesses remain within the bounds
of an MTE tag (16-byte chunks) and improves overall performance.
Co-authored-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-aarch64-add-MTE-compatible-memrchr.patch
Type: text/x-patch
Size: 6772 bytes
Desc: 0001-aarch64-add-MTE-compatible-memrchr.patch
URL: <https://sourceware.org/pipermail/libc-alpha/attachments/20200609/65421141/attachment-0001.bin>
More information about the Libc-alpha
mailing list