[PATCH 04/20] alpha: issue the ev6 memcpy wh64 two lines ahead

[Matt Turner]

The 21264 memcpy loop issued its wh64 write hint one 64-byte line ahead of
the stores that consume it, which is too close to be much use.  wh64 has to
claim the line before the stores reach it and, unlike a load prefetch,
there is no fill latency for a late hint to hide behind -- a late hint
collides with the store stream and provokes Mbox replay traps instead.  The
same mistake costs memset a factor of three; here it costs less, because
the loads of this loop already put some distance between the hint and the
stores that consume it.

Issue it two lines ahead.  Measured on an EV68CB (DS15, 2MB off-chip
Bcache), cold copies improve by 6-8% from 4KB upwards, and 1KB is
unchanged.

Two lines rather than four: the gain past two lines is under 1%, while the
stretch at the end of the region that has to fall back to hinting the
current block grows with the distance and makes short copies slower.  The
curve at 512 bytes runs 702 cycles at one line, 760 at two, 863 at three
and 870 at four; at 256KB it is 324k, 301k, 300k and 298k.  Two lines is
the only depth that does not lose ground at the short end.  The CWG asks
for at least two blocks of lead per stream, and notes that short trip count
loops may want less than the 8/n blocks its MAF rule suggests, which is
exactly this tradeoff.

Both guards move with the distance, because wh64 makes the entire 64-byte
block UNPREDICTABLE and a hint past the end of the destination would
corrupt memory.  The steady-state hint consumed at trip N+1 covers
[$16+128, $16+192), so that trip needs 192 bytes left and the trip that
computes it needs 256.  The loop is only known to have 128 bytes left on
entry, so the initial hint needs its own check -- it is the one hint that
can overrun where the steady-state one cannot -- and falls back to naming
the first block.

Verified on an EV68CB against the C memcpy over every size class from 0 to
64KB at 72 destination alignments, with 512-byte guard regions either side
checked for stray writes.
---
 sysdeps/alpha/alphaev6/memcpy.S | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S
index 7f5799ecac..cf5c97c1de 100644
--- ./sysdeps/alpha/alphaev6/memcpy.S
+++ ./sysdeps/alpha/alphaev6/memcpy.S
@@ -87,9 +87,16 @@ $single_head_quad:
 $do_unroll:
 	/*
 	 * Dispatch on the CPU implementation.  The 21264 (EV6, EV67, EV68)
-	 * issues wh64 one trip ahead of the stores to avoid read-for-ownership
-	 * on whole-line stores into the destination.  That distance is far too
-	 * short for the 21364 (EV7), whose on-chip memory controller has a much
+	 * issues wh64 two lines ahead of the stores to avoid read-for-ownership
+	 * on whole-line stores into the destination.  wh64 has to claim the line
+	 * before the stores reach it and, unlike a load prefetch, there is no
+	 * fill latency for a late hint to hide behind, so a one-line lead is too
+	 * short to do much good.  Going deeper than two costs more than it
+	 * returns here: the loads of this loop already supply some slack, so the
+	 * gain past two lines is under 1%, while the stretch at the end of the
+	 * region that has to fall back to hinting the current block grows and
+	 * makes short copies slower.  memset, which has no load stream to hide
+	 * behind, needs four.  wh64 is far too
 	 * longer latency: there wh64 neither overlaps the fill nor avoids the
 	 * Mbox replay traps it provokes in the store stream.  EV7 takes its own
 	 * unrolled loop that software-prefetches both streams several lines
@@ -106,9 +113,17 @@ $do_unroll:
 	subq	$2, 3, $2		# E : EV7 -> 0
 
 	beq	$2, $ev7_unroll		# U : EV7 takes the prefetch-modify path
-	addq	$16, 64, $7		# E : Initial (+1 trip) wh64 address
-	nop				# E :
-	nop				# E :
+
+	/*
+	 * Initial wh64 target, two lines (128 bytes) ahead of the first block.
+	 * The loop is only known to have 128 bytes left at this point, so the
+	 * first hint can overrun where the steady-state one cannot; if there is
+	 * not room for it, hint the first block instead.  wh64 makes the whole
+	 * 64-byte block UNPREDICTABLE, so an overrunning hint corrupts memory.
+	 */
+	addq	$16, 128, $7		# E : initial (+2 line) wh64 address
+	subq	$18, 192, $2		# E : room for [$16+128, $16+192)?
+	cmovlt	$2, $16, $7		# E : no -- hint the first block instead
 
 $unroll_body:
 	wh64	($7)			# L1 : memory subsystem hint: 64 bytes at
@@ -134,7 +149,11 @@ $unroll_body:
 
 	stq	$4, 8($16)		# L : bytes 8..15
 	stq	$5, 16($16)		# L : bytes 16..23
-	subq	$18, 192, $2		# E : At least two more trips to go?
+	lda	$2, -256($18)		# E : room for the +2 line hint next trip?
+					#     it covers [$16+128, $16+192) there,
+					#     so that trip needs 192 and this one
+					#     needs 256 (256 exceeds the subq
+					#     literal field, hence lda)
 	nop				# E :
 
 	stq	$3, 24($16)		# L : bytes 24..31
-- 
2.54.0