[PATCH 06/20] alpha: finalize the ev6 memcpy wh64 address late to avoid Mbox replays

[Matt Turner]

$7 holds the address the next trip's wh64 issues against, and the cmov that
picks between the two-line hint and the end-of-region fallback ran in the
same group as the loads near the top of the loop.  Letting the hint issue
that early, while the previous trip's stores are still in flight, provokes
Mbox replay traps in the store stream.

Move the cmov down into the group with the loop branch.  The fallback
address has to move from $1 to $23 to get there, because $1 is reused for
the loop condition by the cmple two groups earlier; $23 is otherwise unused
in this file.  Instruction count is unchanged -- the cmov swaps places with
a nop at either end.

Measured on an EV68CB over a cold 256KB copy, this cuts Mbox replay traps
(perf raw event r4 on the 21264) from roughly 1000 per call to roughly 750,
and is worth 1.4-1.8% from 2KB upwards.  Short copies are unaffected at 256
bytes and lose about 1% at 512.

This is worth a comment because the placement looks arbitrary and reads
like something to tidy up.  It is load-bearing: hoisting the cmov back
beside the loads, or reusing $1 so that it has to be hoisted, gives the
gain back.  A variant that instead added a second fallback level reached
the same effect by accident -- it too pushed the last write of $7 into the
final group -- but paid fifteen thousand extra instructions per 256KB copy
and lost 2.3% at 256 bytes for it.

Verified on an EV68CB against the C memcpy over every size class from 0 to
64KB at 72 destination alignments, with 512-byte guard regions either side
checked for stray writes.
---
 sysdeps/alpha/alphaev6/memcpy.S | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S
index 89324be23a..d5b72d4961 100644
--- ./sysdeps/alpha/alphaev6/memcpy.S
+++ ./sysdeps/alpha/alphaev6/memcpy.S
@@ -148,7 +148,7 @@ $unroll_body:
 	nop				# E :
 
 	ldq	$3, 24($17)		# L : bytes 24..31
-	addq	$16, 64, $1		# E : fallback value for wh64
+	addq	$16, 64, $23		# E : fallback wh64 address (see below)
 	nop				# E :
 	nop				# E :
 
@@ -173,8 +173,7 @@ $unroll_body:
 
 	ldq	$6, 0($17)		# L : bytes 0..7
 	ldq	$4, 8($17)		# L : bytes 8..15
-	cmovlt	$2, $1, $7		# E : Latency 2, extra map slot - Use
-					# fallback wh64 address if < 2 more trips
+	nop				# E :
 	nop				# E :
 
 	ldq	$5, 16($17)		# L : bytes 16..23
@@ -187,9 +186,23 @@ $unroll_body:
 	stq	$4, -24($16)		# L : bytes 8..15
 	cmple	$18, 63, $1		# E : At least one more trip?
 
+	/*
+	 * $7 is deliberately finalized late, in the group with the loop branch
+	 * rather than alongside the loads above, and the fallback address lives
+	 * in $23 so that it survives the cmple that reuses $1 for the loop
+	 * condition.  This is not cosmetic: $7 is what the next trip's wh64
+	 * issues against, and letting that hint issue while the previous trip's
+	 * stores are still in flight provokes Mbox replay traps in the store
+	 * stream.  Measured on an EV68CB over a cold 256KB copy, moving the
+	 * cmov down cuts replays (perf r4) from about 1000 per call to about
+	 * 750 and is worth 1.4-1.8% from 2KB upwards, for no extra instructions.
+	 * Moving it back up, or reusing $1 so that it has to move back up,
+	 * gives up that gain.
+	 */
 	stq	$5, -16($16)		# L : bytes 16..23
 	stq	$3, -8($16)		# L : bytes 24..31
-	nop				# E :
+	cmovlt	$2, $23, $7		# E : fallback if < 2 more trips (late; see
+					#     the note above -- do not hoist this)
 	beq	$1, $unroll_body
 	br	$31, $tail_quads	# U : 1..15 trailing quads/bytes
 
-- 
2.54.0