[PATCH v3] aarch64: Optimized memcpy and memmove for Kunpeng processor

[Wilco Dijkstra]

Hi Xuelei,

> Copies up to 96 bytes are split into 2 cases: long copies of 96..1024
> align dst address unrolling 64 bytes without prefetching. And large
> copies more than 1024 bytes align dst address unrolling 128 bytes
> with prfm instructions. Gain greater than 20% improvement both in
> aligned cases and misaligned cases through walk-bench.

What about bench-memcpy-random? Particularly the <= 96 cases look
problematic due to having too many branches and complex alignment
code just to align a single load.

> And for memmove, there are two main changes: i) Q register is used
> instead of  X register. ii) dst address is aligned instead of src
> address aligned to improve store operation. Hence, memmove
> implementation also has improvement above 128 bytes, that about 30%
> for 2k to 8M bytes, and about 50% for 32M or more.

Is there a reason to make the memcpy and memmove loops so different?

copy_long copies 64 bytes/iter using integer LDP/STP for 96..1024 bytes
copy_large copies 128 bytes/iter using SIMD LDP/STP for > 1024 bytes
move_long copies 64 bytes/iter using SIMD LDP/STP for > 512 bytes
move_middle copies 64 bytes/iter using SIMD LDR/STR for 96..512 bytes

The difference between move_middle and move_long is a single prefetch...

A few other comments:

+ENTRY_ALIGN (MEMMOVE, 6)
+
+	DELOUSE (0)
+	DELOUSE (1)
+	DELOUSE (2)
+
+	sub	tmp1, dstin, src
+	cmp	count, 512
+	ccmp	tmp1, count, 2, hi
+	b.lo	L(move_long)
+	cmp	count, 96
+	ccmp	tmp1, count, 2, hi
+	b.lo	L(move_middle)

Why not do a single check and do the same as in copy_long? That speeds up all
small moves.

+L(copy_large):
+	ldr	A_q, [src]
+	and	tmp1, dstin, 15
+	sub	src, src, tmp1
+	add	count, count, tmp1
+	add	src, src, 16
+	ldp	B_q, C_q, [src], #32
+	ldp	D_q, E_q, [src], #32

Why all the increments of src?

+	ldp	F_q, G_q, [src], #32
+	stp	B_q, C_q, [dst], #32
+	ldp	H_q, I_q, [src], #32
+	prfm	pldl1strm, [src, MEMCPY_PREFETCH_LDR]
+	ldp	B_q, C_q, [src], #32
+	stp	D_q, E_q, [dst], #32
+	ldp	D_q, E_q, [src], #32
+	stp	F_q, G_q, [dst], #32
+	stp	H_q, I_q, [dst], #32

And here...

+END (MEMCPY)
+	.section	.rodata
+	.p2align	4

What is this for???

Cheers,
Wilco