[PATCH 07/20] alpha: deepen memchr prefetch distance for EV7

[Matt Turner]

The alpha memchr already issues a read prefetch ahead of the scan, but only
three cache lines out -- tuned for the 21264 (EV6/67/68) and its external
Bcache.  On the 21364 (EV7), whose on-chip memory controller has a much longer
memory latency, three lines is too close to hide the miss.

Issue the prefetch eight cache lines ahead instead (both the priming prefetches
before the cacheline loop and the one inside it).  This is a read hint, so on
the 21264 a deeper distance is at worst dropped; no IMPLVER dispatch is needed.

Measured on an AlphaServer ES47 (EV7, 1.3GHz), cold cyc/call (full scan):
  size       before    after
  4096         3898     3268  1.19x
  65536       61359    52271  1.17x
  262144     246666   208148  1.19x

The 21264 side of that claim has since been measured rather than assumed: on
an EV68CB the read-prefetch distance curve for a streaming scan is flat from
four cache lines out to eight, so moving from three to eight neither helps nor
hurts there.

An assembly rewrite was prototyped but, with the deeper prefetch, a read-hint
loop measured no faster than this C.  The only further gain came from a
modify-intent (LDS/LDT) prefetch, which is not safe to issue on the read-only
operand of a general-purpose memchr: it requests the line for ownership, which
is wrong for a buffer the caller may share.  It is also not the win it looks
like.  On an EV68CB, modify intent on a pure read stream measures 2x slower
than a plain read prefetch -- for memcmp, 756446 cycles against 368989 on a
cold 256KB compare -- because ReadBlkMod provokes about twice the Mbox replay
traps.  Bcache misses are unchanged, so the cost is not the extra coherence
traffic one might expect; it is the replays.
---
 sysdeps/alpha/memchr.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git ./sysdeps/alpha/memchr.c ./sysdeps/alpha/memchr.c
index d9115923d2..663cd08454 100644
--- ./sysdeps/alpha/memchr.c
+++ ./sysdeps/alpha/memchr.c
@@ -79,10 +79,15 @@ __memchr (const void *s, int xc, size_t n)
   /* If the block is sufficiently large, align to cacheline and prefetch.  */
   if (unlikely (n >= 256))
     {
-      /* Prefetch 3 cache lines beyond the one we're working on.  */
-      prefetch (s_align + 8);
-      prefetch (s_align + 16);
+      /* Prefetch several cache lines beyond the one we're working on.
+	 The 21364 (EV7), with its on-chip memory controller and much longer
+	 memory latency than the 21264's external Bcache, needs the read
+	 prefetch issued considerably further ahead to hide the miss; eight
+	 lines beats the three that suit the 21264, and on the 21264 a deeper
+	 read hint is at worst dropped.  */
       prefetch (s_align + 24);
+      prefetch (s_align + 40);
+      prefetch (s_align + 56);
 
       while ((word)s_align & 63)
 	{
@@ -120,10 +125,10 @@ __memchr (const void *s, int xc, size_t n)
 	} while (0)
 
       /* While there's still lots more data to potentially be read,
-	 continue issuing prefetches for the 4th cacheline out.  */
+	 continue issuing prefetches for the 8th cacheline out.  */
       while (n >= 256)
 	{
-	  prefetch (s_align + 24);
+	  prefetch (s_align + 64);
 	  CACHELINE_LOOP;
 	}
 
-- 
2.54.0