[PATCH] x86: Enable non-temporal memset without ERMS
Noah Goldstein
goldstein.w.n@gmail.com
Tue Jul 9 04:19:18 GMT 2024
On Tue, Jul 9, 2024 at 10:58 AM Feifei Wang <wangfeifei@hygon.cn> wrote:
>
> Thanks for your kindly reply.
>
> > -----邮件原件-----
> > 发件人: Noah Goldstein <goldstein.w.n@gmail.com>
> > 发送时间: 2024年7月8日 13:29
> > 收件人: Feifei Wang <wangfeifei@hygon.cn>
> > 抄送: libc-alpha@sourceware.org; Jing Li <lijing@hygon.cn>
> > 主题: Re: [PATCH] x86: Enable non-temporal memset without ERMS
> >
> > On Fri, Jul 5, 2024 at 3:53 PM Feifei Wang <wangfeifei@hygon.cn> wrote:
> > >
> > > Currently, large memset perf can be improved with non-temporal stores[1].
> > > But it just supports the case in which ERMS feature is enabled.
> > >
> > > For some architectures, ERMS is disabled in cpu feature as default For
> > > example, AMD zen3-milan[2], Or ERMS performance is worse than
> > > vectorized loop, and it is disabled.
> > > For example, HygonGenuine arch.
> > >
> > > Thus, to let these arch achieve perf improvement from non-temporal
> > > memset, and enable non-temporal memset without ERMS, a 'non-temporal
> > > memset branch jump' is added in 'more_2x_vec'case.
> > >
> > > Test Results:
> > > thread: 1
> > > memset store value: 0
> > > function: memset_avx2_unaligned
> > >
> > > hygon1 arch
> > > x86_memset_non_temporal_threshold = 8MB
> > > size new performance / old performance
> > > 128 byte(2x -4x vec case) 1
> > > 256 byte(4x - 8x vec case) 1
> > > 512 byte( > 8x loop case) 1
> > > 1MB 0.994
> > > 4MB 0.996
> > > 8MB 0.670
> > > 16MB 0.343
> > > 32MB 0.355
> > >
> > > hygon2 arch
> > > x86_memset_non_temporal_threshold = 8MB
> > > size new performance / old performance
> > > 128 byte(2x -4x vec case) 1
> > > 256 byte(4x - 8x vec case) 0.653
> > > 512 byte( > 8x loop case) 0.713
> > > 1MB 1
> > > 4MB 0.887
> > > 8MB 1.312
> > > 16MB 0.822
> > > 32MB 0.830
> > >
> > > hygon3 arch
> > > x86_memset_non_temporal_threshold = 8MB
> > > size new performance / old performance
> > > 128 byte(2x -4x vec case) 1
> > > 256 byte(4x - 8x vec case) 1
> > > 512 byte( > 8x loop case) 1
> > > 1MB 1
> > > 4MB 0.990
> > > 8MB 0.737
> > > 16MB 0.390
> > > 32MB 0.401
> > >
> > > For hygon arch with , no performance degradation on '2x - 8x branch
> > > case' when extra branch jump added. And with this patch, non-temporal
> > > stores can improve performance by 20% - 65%.
> > >
> > > amd-zen2-rome arch
> > > x86_memset_non_temporal_threshold = 12MB
> > > size new performance / old performance
> > > 128 byte(2x -4x vec case) 0.986
> > > 256 byte(4x - 8x vec case) 1.057
> > > 512 byte( > 8x loop case) 1.005
> > > 1MB 0.999
> > > 4MB 0.998
> > > 8MB 1.014
> > > 16MB 1.156
> > > 32MB 0.634
> > >
> > > For amd-zen2-rome, no performance degradation on '2x - 8x branch case'
> > > when extra branch jump added. When size >=
> > > x86_memset_non_temporal_threshold, for 'size = 16MB' case, performance
> > > degradation is due to that amd-zen2 non_temporal perf is worse than
> > > temporal[3]. And in 32MB case, it can achieve performance improvement by
> > 37%.
> > >
> > > For intel arch, intel arch ERMS feature is enabled as default, so this
> > > patch has no effect for it.
> > >
> > > Note:
> > > Chengdu Haiguang IC Design Co., Ltd (Hygon) aims at providing high
> > > performance x86 processor for China server market.
> > >
> > > Ref:
> > > [1]
> > > https://sourceware.org/git/?p=glibc.git;a=commit;h=5bf0ab80573d66e4ae5
> > > d94b094659094336da90f [2]
> > > https://lists.gnu.org/archive/html/qemu-devel/2022-01/msg06747.html
> > > [3]
> > >
> > https://docs.google.com/spreadsheets/d/1opzukzvum4n6-RUVHTGddV6RjAEil4
> > > P2uMjjQGLbLcU/edit?gid=120165109#gid=120165109
> > >
> > > Jira: HES-210
> > > Signed-off-by: Feifei Wang <wangfeifei@hygon.cn>
> > > Reviewed-by: Jing Li <lijing@hygon.cn>
> > > Change-Id: I5f9b032c848f4e9d2e54a08138af6b9ae71e1156
> > > ---
> > > sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S | 10 +++++++++-
> > > 1 file changed, 9 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
> > > b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
> > > index 88bf08e..063dd51 100644
> > > --- a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
> > > +++ b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
> > > @@ -229,9 +229,17 @@ L(stosb_more_2x_vec):
> > > cmp __x86_rep_stosb_threshold(%rip), %RDX_LP
> > > ja L(stosb_local)
> > > #endif
> > > - /* Fallthrough goes to L(loop_4x_vec). Tests for memset (2x, 4x]
> > > + /* If rdx is less than __x86_memset_non_temporal_threshold,
> > > + fallthrough goes to L(loop_4x_vec). Tests for memset (2x,
> > > + 4x]
> > > and (4x, 8x] jump to target. */
> > > L(more_2x_vec):
> > > +#if defined USE_MULTIARCH && IS_IN (libc)
> > > + /* Check non-temporal store threshold.
> > > + This is for the hardware which can not support ERMS feature or
> > > + ERMS fearure is disabled. */
> > > + cmp
> > __x86_memset_non_temporal_threshold(%rip), %RDX_LP
> > > + jae L(nt_memset)
> > > +#endif
> > > /* Store next 2x vec regardless. */
> > > VMOVU %VMM(0), (%rdi)
> > > VMOVU %VMM(0), (VEC_SIZE * 1)(%rdi)
> > > --
> > > 2.7.4
> > >
> >
> > I am opposed to this patch getting in as is. I strongly dislike the essentially
> > redundant checks for `__x86_rep_stosb_threshold` and
> > `__x86_memset_non_temporal_threshold` that will occur on most processors.
> > Further, in this case, it messes up the alignment of the loop.
> >
> > What I would like to do instead for processors which prefer non-temporal but
> > not erms, is to have them set `__x86_rep_stosb_threshold ==
> > __x86_memset_non_temporal_threshold` and prefer the `erms` version.
> >
> > Can you implement it like that instead?
> [Feifei] This is also a good choice. But I have a question for this:
> If my understand is right, you means that If __x86_rep_stosb_threshold ==
> __x86_memset_non_temporal_threshold, we can set "Prefer_ERMS" when
> init_cpu_features for servers without ERMS.
>
Yes, if you don't want memcpy to use `erms`, set
`__x86_rep_movsb_stop_threshold == SIZE_MAX` as well.
> However, if like this, I think we ignore the small size case. Though we can use non temporal now,
> But for "size < __x86_rep_stosb_threshold" case, it is forced to use erms whose performance is very bad
> For example, hygon and amd zen2.
Hmm, I'm not sure I understand what you mean.
What I am arguing for is that a processor that wants non-temporal but
doesn't want erms should do the following:
memset:
`__x86_rep_stosb_threshold == __x86_memset_non_temporal_threshold ==
YOUR_PREFERED_NT_THRESHOLD`
This will have the exact same behavior as your patch. Below NT threshold it
will use the normal temporal loop.
Above NT threshold it will use nt-stores. It will never use `rep stosb`.
>
> Thus we add a nt jump for temporal branch, and for this, we can support that :
> 1. small size can use temporal for good performance rather than erms.
> 2.big size can use non temporal to achieve better performance.
>
>
> Best Regards
> Feifei
>
More information about the Libc-alpha
mailing list