This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH v3] aarch64: Optimized memcpy and memmove for Kunpeng processor
- From: Wilco Dijkstra <Wilco dot Dijkstra at arm dot com>
- To: Xuelei Zhang <zhangxuelei4 at huawei dot com>, "libc-alpha at sourceware dot org" <libc-alpha at sourceware dot org>, "jiangyikun at huawei dot com" <jiangyikun at huawei dot com>, "yikunkero at gmail dot com" <yikunkero at gmail dot com>
- Date: Wed, 18 Dec 2019 15:49:18 +0000
- Subject: Re: [PATCH v3] aarch64: Optimized memcpy and memmove for Kunpeng processor
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WmNCLb3cAoYUc/tz7d4u63LZQFYbk7Fa+q4QQNkf1H4=; b=DoXmRmyNdA3m83E4qvLzGMAavp+16eDLBax0bBoKDADiB782wQaQoCR2vIiyDB9Xz6Yn7N4X1tmGUhD6vHsqH7VUmgxUlXxdTPK97LWl4c5Av7m34y6X6tJtFQDdppmduUBdNHpFGk1m+V1I3eZn9mjIongx5C0xtUGh10o2Q5ykHGKxUCoECQlPtdP/c30d0OtMLNWa6pT7ZEdjLcjAsjKFoVTPiPNKN/QRZOFREGhA/BdX1haJI8uoWz477rjBMDKSXOMPPUKwH/MFFuRyd3yatQI3MP9XUIBnP1dte4Rc0turvm9AjGQwgyWkQW+bn05W8rfmAKVC+Jq+AdelxQ==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VtHpi18g80JPhodFFdVIw5SmOlFEYFTTby6/IOi3v5BMwdb/lOZmihMRHmXmSlfKSyD1oRwwbvfLCV0hmCb0zvs+SwVNRXJuyAbeUAYROmT30wT35gK4BP0H0N+oQjNDpvKwWZYBwZKX4DciPNkey+I8d9uwVbVOTHBFWRswdPXJOqXMafTQQ/ijgqswjIj8455xTuBa9pK0nl32XsWXCZd7i/8gUnwQpjWRcJ9VJIAOf1LMWLmWWvRKmUpRoAqLADsBezjfRFLDIeXvW9L7D4AvitId2ZYHvFdS8azJaEbu0igZhGQXTGBPcGZ+ppYunw44Ah5Yi4D62Z48Cwy3oQ==
- Original-authentication-results: spf=none (sender IP is ) smtp.mailfrom=Wilco dot Dijkstra at arm dot com;
- References: <20191119133306.20572-1-zhangxuelei4@huawei.com>
Hi Xuelei,
> Copies up to 96 bytes are split into 2 cases: long copies of 96..1024
> align dst address unrolling 64 bytes without prefetching. And large
> copies more than 1024 bytes align dst address unrolling 128 bytes
> with prfm instructions. Gain greater than 20% improvement both in
> aligned cases and misaligned cases through walk-bench.
What about bench-memcpy-random? Particularly the <= 96 cases look
problematic due to having too many branches and complex alignment
code just to align a single load.
> And for memmove, there are two main changes: i) Q register is used
> instead of X register. ii) dst address is aligned instead of src
> address aligned to improve store operation. Hence, memmove
> implementation also has improvement above 128 bytes, that about 30%
> for 2k to 8M bytes, and about 50% for 32M or more.
Is there a reason to make the memcpy and memmove loops so different?
copy_long copies 64 bytes/iter using integer LDP/STP for 96..1024 bytes
copy_large copies 128 bytes/iter using SIMD LDP/STP for > 1024 bytes
move_long copies 64 bytes/iter using SIMD LDP/STP for > 512 bytes
move_middle copies 64 bytes/iter using SIMD LDR/STR for 96..512 bytes
The difference between move_middle and move_long is a single prefetch...
A few other comments:
+ENTRY_ALIGN (MEMMOVE, 6)
+
+ DELOUSE (0)
+ DELOUSE (1)
+ DELOUSE (2)
+
+ sub tmp1, dstin, src
+ cmp count, 512
+ ccmp tmp1, count, 2, hi
+ b.lo L(move_long)
+ cmp count, 96
+ ccmp tmp1, count, 2, hi
+ b.lo L(move_middle)
Why not do a single check and do the same as in copy_long? That speeds up all
small moves.
+L(copy_large):
+ ldr A_q, [src]
+ and tmp1, dstin, 15
+ sub src, src, tmp1
+ add count, count, tmp1
+ add src, src, 16
+ ldp B_q, C_q, [src], #32
+ ldp D_q, E_q, [src], #32
Why all the increments of src?
+ ldp F_q, G_q, [src], #32
+ stp B_q, C_q, [dst], #32
+ ldp H_q, I_q, [src], #32
+ prfm pldl1strm, [src, MEMCPY_PREFETCH_LDR]
+ ldp B_q, C_q, [src], #32
+ stp D_q, E_q, [dst], #32
+ ldp D_q, E_q, [src], #32
+ stp F_q, G_q, [dst], #32
+ stp H_q, I_q, [dst], #32
And here...
+END (MEMCPY)
+ .section .rodata
+ .p2align 4
What is this for???
Cheers,
Wilco