[PATCH v2 1/2] x86: Improve large memset perf with non-temporal stores [RHEL-29312]

[Noah Goldstein]

On Fri, Jul 12, 2024 at 4:56 AM DJ Delorie <dj@redhat.com> wrote:
>
>
> Noah Goldstein <goldstein.w.n@gmail.com> writes:
> > Using non-temporal stores can be up to 3x faster on ICX and 2x faster
> > on SKX. Historically, these numbers would not have been so good
> > because of the zero-over-zero writeback optimization that `rep stosb`
> > is able to do. But, the zero-over-zero writeback optimization has been
> > removed as a potential side-channel attack, so there is no longer any
> > good reason to only rely on `rep stosb` for large memsets. On the flip
> > size, non-temporal writes can avoid data in their RFO requests saving
> > memory bandwidth.
>
> I'm actually working on RHEL-29312 that you reference in the subject
> (thanks for the reference, but please don't reference downstream tickets
> in upstream patches).  In that ticket was shown a 50% slowdown in memset
> for the Xeon 4215.  I've reproduced that slowdown locally, but this
> patch doesn't seem to have any affect on it (current git glibc still has
> the slow code).  I can get the faster results with tunables[1].  Will
> there be further tuning of the default thresholds?  Or do we expect
> users to use tunables to get around these regressions, for the specific
> chips that are not optimized?
>
> [1] specifically, libc.cpu.x86_rep_stosb_threshold=0x1000000000 - none
> of the others have any affect.

Okay, I messed up. The SKX machine I used for the SKX benchmarks was
mislabeled, and was in fact an ICX machine.

I have some older SKX/BWD data:
https://docs.google.com/spreadsheets/d/1rcRIWpU8Tcq0FMH9iGPBUViPvioUmHtA9qx3h-UzPa0/edit?usp=sharing
and agree we need to increase the stosb threshold on SKX. I didn't see
the huge (100MB threshold) you saw there, but am re-running benchmarks.

An unfortunate note is BWD seems to prefer ERMS in the same range SKX
doesn't, so we can't just make the decision based on the feature.
On the bright side, it does look like the non-temporal decision (this patch)
still makes sense on SKX.

I am re-running benchmarks now and will see what we can do about raising
the stosb threshold on SKX.

I am deeply sorry for the confusion I have created here.
>