[PATCH 03/20] alpha: use prefetch-with-modify-intent in memcpy on EV7

[Matt Turner]

The unrolled loop in the ev6 memcpy issues wh64 one trip ahead of the
stores to avoid read-for-ownership on whole-line stores into the
destination.  That is tuned for the 21264 (EV6, EV67, EV68) with an
external Bcache.  On the 21364 (EV7), with its on-chip memory controller
and much longer memory latency, one line is far too close: the hint
neither overlaps the fill nor avoids the Mbox replay traps it provokes
in the store stream.  The loop also never prefetches the source, leaning
entirely on software pipelining to hide load latency.

Dispatch on IMPLVER (2 for the 21264 family, 3 for the 21364 family) and
give EV7 its own unrolled loop that software-prefetches both streams six
cache lines ahead of the current block: the source with a plain read
prefetch (LDL to R31) and the destination with a prefetch with modify
intent (LDS to F31, a ReadBlkMod that allocates the line writeable).
Both are reads only, so prefetching past the end of the region is
harmless and no overrun guard is needed.  The distance is not critical:
4 to 12 lines all measure the same on EV7.  The 21264 code path is
unchanged.

Measured on an AlphaServer ES47 (EV7, 1.3GHz), cycles/call:
  size     cold before  cold after      hot before  hot after
  1024            3510        3050 -13%        585        246 -58%
  16384          55299       46369 -16%       8602       4054 -53%
  65536         327197      291205 -11%      61712      27244 -56%
  262144        676953      535414 -21%     248933      98661 -60%
The large hot-cache win comes from removing wh64: it was the source of
hundreds of millions of Mbox replay traps (perf -e r4) in the store
stream, which the read-only prefetch path does not provoke.
---
 sysdeps/alpha/alphaev6/memcpy.S | 61 ++++++++++++++++++++++++++++++++-
 1 file changed, 60 insertions(+), 1 deletion(-)

diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S
index 04bf2422e3..7f5799ecac 100644
--- ./sysdeps/alpha/alphaev6/memcpy.S
+++ ./sysdeps/alpha/alphaev6/memcpy.S
@@ -85,9 +85,29 @@ $single_head_quad:
 	bne	$1, $single_head_quad	# U : still not fully aligned
 
 $do_unroll:
-	addq	$16, 64, $7		# E : Initial (+1 trip) wh64 address
+	/*
+	 * Dispatch on the CPU implementation.  The 21264 (EV6, EV67, EV68)
+	 * issues wh64 one trip ahead of the stores to avoid read-for-ownership
+	 * on whole-line stores into the destination.  That distance is far too
+	 * short for the 21364 (EV7), whose on-chip memory controller has a much
+	 * longer latency: there wh64 neither overlaps the fill nor avoids the
+	 * Mbox replay traps it provokes in the store stream.  EV7 takes its own
+	 * unrolled loop that software-prefetches both streams several lines
+	 * ahead -- the source with a plain read prefetch (LDL to R31) and the
+	 * destination with a prefetch-with-modify-intent (LDS to F31, a
+	 * ReadBlkMod that allocates the line writeable).  Both are reads only,
+	 * so prefetching past the end of the region is harmless and no overrun
+	 * guard is needed.  IMPLVER returns 2 for the 21264 family and 3 for
+	 * the 21364 family.
+	 */
 	cmple	$18, 127, $1		# E : Can we go through the unrolled loop?
 	bne	$1, $tail_quads		# U : Nope
+	implver	$2			# E : 2 = 21264 (EV6/67/68), 3 = 21364
+	subq	$2, 3, $2		# E : EV7 -> 0
+
+	beq	$2, $ev7_unroll		# U : EV7 takes the prefetch-modify path
+	addq	$16, 64, $7		# E : Initial (+1 trip) wh64 address
+	nop				# E :
 	nop				# E :
 
 $unroll_body:
@@ -142,6 +162,45 @@ $unroll_body:
 	stq	$3, -8($16)		# L : bytes 24..31
 	nop				# E :
 	beq	$1, $unroll_body
+	br	$31, $tail_quads	# U : 1..15 trailing quads/bytes
+
+	/*
+	 * EV7 (21364): 64-byte unrolled copy loop that software-prefetches six
+	 * cache lines (384 bytes) ahead of the current block -- the source with
+	 * a read prefetch and the destination with modify intent.  The distance
+	 * is not critical: anything from 4 to 12 lines measures the same on EV7.
+	 * Both src and dest are 0mod8 here (dest is 0mod64), and the prefetches
+	 * only read, so running past the end of the region is harmless.
+	 * $16 - dest, $17 - src, $18 - bytes left, $1..$7,$22 scratch.
+	 */
+	.align 4
+$ev7_unroll:
+	lds	$f31, 6*64($16)		# L : dest prefetch w/ modify intent
+	ldl	$31, 6*64($17)		# L : src prefetch (read)
+	ldq	$1, 0($17)		# L : bytes 0..7
+	ldq	$2, 8($17)		# L : bytes 8..15
+	ldq	$3, 16($17)		# L : bytes 16..23
+	ldq	$4, 24($17)		# L : bytes 24..31
+	ldq	$5, 32($17)		# L : bytes 32..39
+	ldq	$6, 40($17)		# L : bytes 40..47
+	ldq	$7, 48($17)		# L : bytes 48..55
+	ldq	$22, 56($17)		# L : bytes 56..63
+
+	stq	$1, 0($16)		# L :
+	stq	$2, 8($16)		# L :
+	stq	$3, 16($16)		# L :
+	stq	$4, 24($16)		# L :
+	stq	$5, 32($16)		# L :
+	stq	$6, 40($16)		# L :
+	stq	$7, 48($16)		# L :
+	stq	$22, 56($16)		# L :
+
+	subq	$18, 64, $18		# E : count -= 64
+	addq	$17, 64, $17		# E : src += 64
+	addq	$16, 64, $16		# E : dest += 64
+	cmple	$18, 63, $1		# E : less than one more full trip?
+	beq	$1, $ev7_unroll		# U : keep going
+	/* fall through to $tail_quads for the 1..63 byte remainder */
 
 $tail_quads:
 $no_unroll:
-- 
2.54.0