[PATCH] riscv: Add memcpy for CPUs with slow misaligned access
Matteo Croce
technoboy85@gmail.com
Fri Nov 13 13:31:05 GMT 2026
On RISC-V CPUs without V whose misaligned accesses are not reported as
fast, memcpy uses the generic C implementation. This copies byte by
byte below 16 bytes and uses _wordcopy_* above.
Add __memcpy_shifted, which only uses naturally aligned loads and
stores. After aligning DEST, it copies 16 words per iteration with
interleaved loads and stores when SRC has the same alignment, then
blocks of 8, 4, 2 and 1 words selected by the bits of the count.
Otherwise it builds each output word from two adjacent aligned source
words with shifts, 8 words per iteration. The same technique was
proposed for the Linux kernel memcpy:
https://lore.kernel.org/linux-riscv/20260311173138.1820-1-teknoraver@meta.com/
Copies below 16 bytes, the bytes that align DEST and the bytes left
after the shifted copy are copied with byte loads and stores in blocks
of 8, 4, 2 and 1 bytes, selected by the bits of the count, instead of
byte loops. On the P550 the cost of a short byte loop jumps by several
cycles at a small iteration count, so that a 2-byte copy with a loop
was slower than a 3-byte copy with these blocks.
Loop heads are 16-byte aligned so that copies do not get slower
depending on where the code is linked.
The memcpy ifunc selects the new variant when misaligned accesses are
not reported as fast (__memcpy_noalignment is still used when they
are).
Benchmarked on a Milk-V Megrez (ESWIN EIC7700X, SiFive P550, no V,
misaligned accesses trapped and emulated), 5 runs pinned to one core,
with the best and worst run of each case dropped and the other 3
averaged. Time saved compared to the generic C version (negative
means slower):
alignment differs alignment matches
size best avg worst best avg worst
1-15 B +35% +23% -5.1% +35% +19% -5.1%
16-511 B +22% +20% +16% +47% +40% +35%
512 B-64 KiB +26% +26% +22% +31% +24% +20%
> 64 KiB +26% +23% +17% +0.5% +0.1% -0.4%
The best and worst columns are the best and worst size in each row,
with sizes grouped per byte below 16 bytes and per power of two above.
The avg column is computed from the geometric mean of the time ratios
over all lengths and alignments in the row.
bench-memcpy-large has no warm-up call, so its first case of each length
(alignment 0/0, DEST below SRC) also times the page faults on the newly
used part of DEST, and only for the first implementation in the list,
which is __memcpy_shifted. That case is left out of the table and of
the bench-memcpy-large mean below (with it: +14.6%, and 25% slower at
16 MiB). In a standalone test with the order swapped, the other
implementation pays them instead.
Overall, from the geometric mean of the time ratios:
bench-memcpy +25.7%
bench-memcpy-large +15.8%
bench-memcpy-random +16.2%
The 5 runs of bench-memcpy-random measured +16.1% to +16.3%. 2-byte
copies are about 5% slower. Copies above 64 KiB with matching
alignment are within 0.5% of the generic version.
Other synthetic benchmarks, run alternately with the generic and the
new version, pinned to one core (time saved, from the mean throughput
of all runs): perf bench mem memcpy with aligned buffers, 5 runs per
size, and stress-ng --memcpy 1 --memcpy-method libc, 3 runs of 30 s:
perf bench mem memcpy, 64 B +38.8%
perf bench mem memcpy, 512 B +31.4%
perf bench mem memcpy, 4 KiB +22.1%
perf bench mem memcpy, 32 KiB +20.3%
perf bench mem memcpy, 1 MiB -0.3%
perf bench mem memcpy, 32 MiB -0.3%
stress-ng --memcpy +9.6%
Real programs, run alternately with the generic and the new version,
pinned to one core, with files in tmpfs, 2 to 5 runs per version.
These are the programs that gained 2% or more in our tests, plus gcc,
where 1.8% still matters because builds take hours (time saved with
the new version, from the mean wall time of all runs):
git blame kernel/fork.c, v6.0..v6.19 +7.6%
zstd -d of a 200 MiB tar of kernel sources +6.5%
sed s/int/long/g on 64 MB of kernel C files +4.5%
gunzip of a 200 MiB tar of kernel sources +3.0%
gawk '{print $1}' on 64 MB of kernel C files +2.6%
ImageMagick -resize 50% of a 40 MP photo +2.5%
mawk '{print $1}' on 64 MB of kernel C files +2.1%
uniq -c on 64 MB of kernel C files +2.0%
gcc (cc1) on 12 preprocessed kernel files +1.8%
The other programs we timed were between -2.0% and +1.9%: gzip, bzip2,
xz and zstd compression, bzip2 and xz decompression, zip, unzip, lame,
ffmpeg, sort, perl, python3, php, sqlite, ImageMagick, rsync, dd, and
git archive, cat-file, diff and log. The slowest were ImageMagick
making a 1920 pixel JPEG thumbnail of the same photo (-2.0%) and sort
(-0.76%). This is not caused by memcpy: the same libc.so built to
select the generic memcpy is just as slow. Differences of about 0.5%
are within the variation between runs.
Checked on riscv64-linux-gnu (string tests).
Signed-off-by: Matteo Croce <technoboy85@gmail.com>
---
sysdeps/riscv/multiarch/memcpy_shifted.S | 396 ++++++++++++++++++
.../unix/sysv/linux/riscv/multiarch/Makefile | 1 +
.../linux/riscv/multiarch/ifunc-impl-list.c | 1 +
.../unix/sysv/linux/riscv/multiarch/memcpy.c | 3 +-
4 files changed, 400 insertions(+), 1 deletion(-)
create mode 100644 sysdeps/riscv/multiarch/memcpy_shifted.S
diff --git a/sysdeps/riscv/multiarch/memcpy_shifted.S b/sysdeps/riscv/multiarch/memcpy_shifted.S
new file mode 100644
index 0000000000..e4e0e51355
--- /dev/null
+++ b/sysdeps/riscv/multiarch/memcpy_shifted.S
@@ -0,0 +1,396 @@
+/* memcpy for RISC-V, using only naturally aligned accesses
+ Copyright (C) 2026 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library. If not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <sysdep.h>
+#include <sys/asm.h>
+
+#if IS_IN(libc)
+/* memcpy optimization for CPUs without V where misaligned accesses are
+ not reported as fast (any RISCV_HWPROBE_MISALIGNED_* value other than
+ FAST).
+
+ Every load and store is naturally aligned. Copies shorter than
+ 2 * SZREG are done with byte loads and stores in blocks of 8, 4, 2
+ and 1 bytes, selected by the bits of LEN, without loops. Longer
+ copies first align DEST the same way, then:
+
+ - if SRC is now aligned too, copy BLOCK_SIZE bytes per iteration with
+ interleaved loads and stores, then blocks of 8, 4, 2 and 1 words
+ selected by the bits of LEN, then a 4/2/1 byte tail;
+
+ - otherwise round SRC down to a word boundary and build each output
+ word from two adjacent aligned source words with shifts
+ ((w0 >> shr) | (w1 << shl)), 8 words per iteration, then at most
+ one block of 4 words, then single words, then copy the remaining
+ bytes in blocks of 4, 2 and 1 bytes.
+
+ Short byte loops are avoided because on the SiFive P550 their cost
+ jumps by several cycles at a small iteration count: copying 2 bytes
+ with a byte loop took longer than 3 bytes with the blocks above.
+
+ The shifted copy reads the aligned word containing the last source
+ byte, so it never reads past the aligned word holding SRC + LEN - 1
+ and cannot cross a page boundary that the copy itself does not.
+
+ Loop heads are aligned to 16 bytes: on the SiFive P550 a 16-byte
+ loop ran about 30% slower when it straddled a 64-byte boundary,
+ which otherwise depends on where the linker places this code. */
+
+# define BLOCK_SIZE (16 * SZREG)
+
+ENTRY (__memcpy_shifted)
+ /* a6 = DEST cursor, a0 is preserved as the return value. */
+ mv a6, a0
+
+ /* If LEN < 2 * SZREG copy bytes. */
+ li a3, 2*SZREG
+ bltu a2, a3, L(byte_copy)
+
+ /* Copy a3 = (-DEST) & (SZREG-1) bytes to align DEST, in blocks of
+ 1, 2 and 4 bytes selected by the bits of a3. */
+ neg a3, a0
+ andi a3, a3, SZREG-1
+ beqz a3, L(dst_aligned)
+ sub a2, a2, a3
+ andi a4, a3, 1
+ beqz a4, 1f
+ lbu a5, 0(a1)
+ sb a5, 0(a6)
+ addi a1, a1, 1
+ addi a6, a6, 1
+1:
+ andi a4, a3, 2
+ beqz a4, 2f
+ lbu a4, 0(a1)
+ lbu a5, 1(a1)
+ sb a4, 0(a6)
+ sb a5, 1(a6)
+ addi a1, a1, 2
+ addi a6, a6, 2
+2:
+# if SZREG == 8
+ andi a4, a3, 4
+ beqz a4, L(dst_aligned)
+ lbu a4, 0(a1)
+ lbu a5, 1(a1)
+ lbu t0, 2(a1)
+ lbu t1, 3(a1)
+ sb a4, 0(a6)
+ sb a5, 1(a6)
+ sb t0, 2(a6)
+ sb t1, 3(a6)
+ addi a1, a1, 4
+ addi a6, a6, 4
+# endif
+
+L(dst_aligned):
+ /* DEST is aligned, LEN > SZREG. */
+ andi a3, a1, SZREG-1
+ bnez a3, L(shifted)
+
+ /* Both aligned. If LEN < BLOCK_SIZE jump to word copy. */
+ li a4, BLOCK_SIZE-1
+ bleu a2, a4, L(word_copy)
+ andi a7, a2, -BLOCK_SIZE
+ add a7, a6, a7
+ .p2align 4
+L(block_loop):
+ REG_L a3, 0(a1)
+ REG_L a4, SZREG(a1)
+ REG_S a3, 0(a6)
+ REG_L a5, (2*SZREG)(a1)
+ REG_S a4, SZREG(a6)
+ REG_L t0, (3*SZREG)(a1)
+ REG_S a5, (2*SZREG)(a6)
+ REG_L a3, (4*SZREG)(a1)
+ REG_S t0, (3*SZREG)(a6)
+ REG_L a4, (5*SZREG)(a1)
+ REG_S a3, (4*SZREG)(a6)
+ REG_L a5, (6*SZREG)(a1)
+ REG_S a4, (5*SZREG)(a6)
+ REG_L t0, (7*SZREG)(a1)
+ REG_S a5, (6*SZREG)(a6)
+ REG_L a3, (8*SZREG)(a1)
+ REG_S t0, (7*SZREG)(a6)
+ REG_L a4, (9*SZREG)(a1)
+ REG_S a3, (8*SZREG)(a6)
+ REG_L a5, (10*SZREG)(a1)
+ REG_S a4, (9*SZREG)(a6)
+ REG_L t0, (11*SZREG)(a1)
+ REG_S a5, (10*SZREG)(a6)
+ REG_L a3, (12*SZREG)(a1)
+ REG_S t0, (11*SZREG)(a6)
+ REG_L a4, (13*SZREG)(a1)
+ REG_S a3, (12*SZREG)(a6)
+ REG_L a5, (14*SZREG)(a1)
+ REG_S a4, (13*SZREG)(a6)
+ REG_L t0, (15*SZREG)(a1)
+ REG_S a5, (14*SZREG)(a6)
+ addi a1, a1, BLOCK_SIZE
+ REG_S t0, (15*SZREG)(a6)
+ addi a6, a6, BLOCK_SIZE
+ bne a6, a7, L(block_loop)
+ andi a2, a2, BLOCK_SIZE-1
+ /* Skip the word blocks if no whole word is left. Coming from the
+ entry LEN >= SZREG + 1 here, so the check is only needed after
+ the block loop. */
+ andi a3, a2, -SZREG
+ beqz a3, L(aligned_tail)
+
+ /* 0 <= LEN < BLOCK_SIZE, SRC and DEST aligned. Copy 8, 4, 2 and 1
+ words selected by the bits of LEN, without loops. */
+L(word_copy):
+ andi a3, a2, 8*SZREG
+ beqz a3, 1f
+ REG_L a3, 0(a1)
+ REG_L a4, SZREG(a1)
+ REG_L a5, (2*SZREG)(a1)
+ REG_L t0, (3*SZREG)(a1)
+ REG_S a3, 0(a6)
+ REG_S a4, SZREG(a6)
+ REG_S a5, (2*SZREG)(a6)
+ REG_S t0, (3*SZREG)(a6)
+ REG_L a3, (4*SZREG)(a1)
+ REG_L a4, (5*SZREG)(a1)
+ REG_L a5, (6*SZREG)(a1)
+ REG_L t0, (7*SZREG)(a1)
+ REG_S a3, (4*SZREG)(a6)
+ REG_S a4, (5*SZREG)(a6)
+ REG_S a5, (6*SZREG)(a6)
+ REG_S t0, (7*SZREG)(a6)
+ addi a1, a1, 8*SZREG
+ addi a6, a6, 8*SZREG
+1:
+ andi a3, a2, 4*SZREG
+ beqz a3, 2f
+ REG_L a3, 0(a1)
+ REG_L a4, SZREG(a1)
+ REG_L a5, (2*SZREG)(a1)
+ REG_L t0, (3*SZREG)(a1)
+ REG_S a3, 0(a6)
+ REG_S a4, SZREG(a6)
+ REG_S a5, (2*SZREG)(a6)
+ REG_S t0, (3*SZREG)(a6)
+ addi a1, a1, 4*SZREG
+ addi a6, a6, 4*SZREG
+2:
+ andi a3, a2, 2*SZREG
+ beqz a3, 3f
+ REG_L a3, 0(a1)
+ REG_L a4, SZREG(a1)
+ REG_S a3, 0(a6)
+ REG_S a4, SZREG(a6)
+ addi a1, a1, 2*SZREG
+ addi a6, a6, 2*SZREG
+3:
+ andi a3, a2, SZREG
+ beqz a3, L(aligned_tail)
+ REG_L a3, 0(a1)
+ REG_S a3, 0(a6)
+ addi a1, a1, SZREG
+ addi a6, a6, SZREG
+
+ /* 0 <= LEN & (SZREG-1) < SZREG, SRC and DEST aligned, so every
+ access below is naturally aligned. */
+L(aligned_tail):
+# if SZREG == 8
+ andi a3, a2, 4
+ beqz a3, 1f
+ lw a4, 0(a1)
+ addi a1, a1, 4
+ sw a4, 0(a6)
+ addi a6, a6, 4
+1:
+# endif
+ andi a3, a2, 2
+ beqz a3, 2f
+ lhu a4, 0(a1)
+ addi a1, a1, 2
+ sh a4, 0(a6)
+ addi a6, a6, 2
+2:
+ andi a3, a2, 1
+ beqz a3, L(ret)
+ lbu a4, 0(a1)
+ sb a4, 0(a6)
+ ret
+
+ /* DEST aligned, SRC not, LEN > SZREG.
+ a3 = SRC & (SZREG-1), 0 < a3 < SZREG. */
+L(shifted):
+ /* t5 = right shift amount, t6 = left shift amount. The shift
+ instructions only use the low log2(XLEN) bits of the amount, so
+ -t5 behaves as XLEN - t5. */
+ slli t5, a3, 3
+ neg t6, t5
+ /* Round SRC down to a word boundary. */
+ sub a1, a1, a3
+ /* a7 = end of the whole words in DEST, t4 = end of the 8 word blocks. */
+ andi a7, a2, -SZREG
+ add a7, a6, a7
+ andi t4, a2, -(8*SZREG)
+ add t4, a6, t4
+ /* a4 = the current aligned source word. */
+ REG_L a4, 0(a1)
+ beq a6, t4, L(shifted_4)
+ .p2align 4
+L(shifted_loop8):
+ REG_L t0, SZREG(a1)
+ REG_L t1, (2*SZREG)(a1)
+ srl a4, a4, t5
+ sll a5, t0, t6
+ or a4, a4, a5
+ REG_L t2, (3*SZREG)(a1)
+ REG_S a4, 0(a6)
+ srl t0, t0, t5
+ sll t3, t1, t6
+ or t0, t0, t3
+ REG_L a4, (4*SZREG)(a1)
+ REG_S t0, SZREG(a6)
+ srl t1, t1, t5
+ sll a5, t2, t6
+ or t1, t1, a5
+ REG_S t1, (2*SZREG)(a6)
+ srl t2, t2, t5
+ sll t3, a4, t6
+ or t2, t2, t3
+ REG_S t2, (3*SZREG)(a6)
+ REG_L t0, (5*SZREG)(a1)
+ REG_L t1, (6*SZREG)(a1)
+ srl a4, a4, t5
+ sll a5, t0, t6
+ or a4, a4, a5
+ REG_L t2, (7*SZREG)(a1)
+ REG_S a4, (4*SZREG)(a6)
+ srl t0, t0, t5
+ sll t3, t1, t6
+ or t0, t0, t3
+ REG_L a4, (8*SZREG)(a1)
+ REG_S t0, (5*SZREG)(a6)
+ srl t1, t1, t5
+ sll a5, t2, t6
+ or t1, t1, a5
+ REG_S t1, (6*SZREG)(a6)
+ srl t2, t2, t5
+ sll t3, a4, t6
+ or t2, t2, t3
+ REG_S t2, (7*SZREG)(a6)
+ addi a1, a1, 8*SZREG
+ addi a6, a6, 8*SZREG
+ bne a6, t4, L(shifted_loop8)
+L(shifted_4):
+ /* At most one 4 word block is left. */
+ andi t4, a2, 4*SZREG
+ beqz t4, L(shifted_word)
+ REG_L t0, SZREG(a1)
+ REG_L t1, (2*SZREG)(a1)
+ srl a4, a4, t5
+ sll a5, t0, t6
+ or a4, a4, a5
+ REG_L t2, (3*SZREG)(a1)
+ REG_S a4, 0(a6)
+ srl t0, t0, t5
+ sll t3, t1, t6
+ or t0, t0, t3
+ REG_L a4, (4*SZREG)(a1)
+ REG_S t0, SZREG(a6)
+ srl t1, t1, t5
+ sll a5, t2, t6
+ or t1, t1, a5
+ REG_S t1, (2*SZREG)(a6)
+ srl t2, t2, t5
+ sll t3, a4, t6
+ or t2, t2, t3
+ REG_S t2, (3*SZREG)(a6)
+ addi a1, a1, 4*SZREG
+ addi a6, a6, 4*SZREG
+L(shifted_word):
+ beq a6, a7, L(shifted_tail)
+ .p2align 4
+L(shifted_loop1):
+ REG_L t0, SZREG(a1)
+ srl a4, a4, t5
+ sll a5, t0, t6
+ or a4, a4, a5
+ REG_S a4, 0(a6)
+ mv a4, t0
+ addi a1, a1, SZREG
+ addi a6, a6, SZREG
+ bne a6, a7, L(shifted_loop1)
+L(shifted_tail):
+ /* Point SRC back at the next unaligned byte. */
+ add a1, a1, a3
+ andi a2, a2, SZREG-1
+
+ /* Copy the remaining 0 <= LEN < 2 * SZREG bytes in blocks of 8, 4,
+ 2 and 1 bytes selected by the bits of LEN. */
+L(byte_copy):
+# if SZREG == 8
+ andi a3, a2, 8
+ beqz a3, 1f
+ lbu a3, 0(a1)
+ lbu a4, 1(a1)
+ lbu a5, 2(a1)
+ lbu a7, 3(a1)
+ sb a3, 0(a6)
+ sb a4, 1(a6)
+ sb a5, 2(a6)
+ sb a7, 3(a6)
+ lbu a3, 4(a1)
+ lbu a4, 5(a1)
+ lbu a5, 6(a1)
+ lbu a7, 7(a1)
+ sb a3, 4(a6)
+ sb a4, 5(a6)
+ sb a5, 6(a6)
+ sb a7, 7(a6)
+ addi a1, a1, 8
+ addi a6, a6, 8
+1:
+# endif
+ andi a3, a2, 4
+ beqz a3, 2f
+ lbu a3, 0(a1)
+ lbu a4, 1(a1)
+ lbu a5, 2(a1)
+ lbu a7, 3(a1)
+ sb a3, 0(a6)
+ sb a4, 1(a6)
+ sb a5, 2(a6)
+ sb a7, 3(a6)
+ addi a1, a1, 4
+ addi a6, a6, 4
+2:
+ andi a3, a2, 2
+ beqz a3, 3f
+ lbu a3, 0(a1)
+ lbu a4, 1(a1)
+ sb a3, 0(a6)
+ sb a4, 1(a6)
+ addi a1, a1, 2
+ addi a6, a6, 2
+3:
+ andi a3, a2, 1
+ beqz a3, L(ret)
+ lbu a3, 0(a1)
+ sb a3, 0(a6)
+L(ret):
+ ret
+END (__memcpy_shifted)
+
+#endif
diff --git a/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile b/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile
index bdbd12a2b6..1bbd691693 100644
--- a/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile
+++ b/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile
@@ -13,6 +13,7 @@ sysdep_routines += \
memcpy-generic \
memcpy-vector \
memcpy_noalignment \
+ memcpy_shifted \
memmove \
memmove-generic \
memmove-vector \
diff --git a/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c b/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c
index ac2b25de74..e97873e261 100644
--- a/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c
+++ b/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c
@@ -48,6 +48,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array,
__memcpy_vector)
IFUNC_IMPL_ADD (array, i, memcpy, fast_unaligned,
__memcpy_noalignment)
+ IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_shifted)
IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_generic))
IFUNC_IMPL (i, name, memset,
diff --git a/sysdeps/unix/sysv/linux/riscv/multiarch/memcpy.c b/sysdeps/unix/sysv/linux/riscv/multiarch/memcpy.c
index 11919d815a..53cdb4ced9 100644
--- a/sysdeps/unix/sysv/linux/riscv/multiarch/memcpy.c
+++ b/sysdeps/unix/sysv/linux/riscv/multiarch/memcpy.c
@@ -32,6 +32,7 @@ extern __typeof (__redirect_memcpy) __libc_memcpy;
extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden;
extern __typeof (__redirect_memcpy) __memcpy_noalignment attribute_hidden;
+extern __typeof (__redirect_memcpy) __memcpy_shifted attribute_hidden;
extern __typeof (__redirect_memcpy) __memcpy_vector attribute_hidden;
static inline __typeof (__redirect_memcpy) *
@@ -46,7 +47,7 @@ select_memcpy_ifunc (uint64_t dl_hwcap, __riscv_hwprobe_t hwprobe_func)
&& (v & RISCV_HWPROBE_MISALIGNED_MASK) == RISCV_HWPROBE_MISALIGNED_FAST)
return __memcpy_noalignment;
- return __memcpy_generic;
+ return __memcpy_shifted;
}
riscv_libc_ifunc (__libc_memcpy, select_memcpy_ifunc);
--
2.56.0
More information about the Libc-alpha
mailing list