[PATCH 02/20] alpha: issue the ev6 memset wh64 four lines ahead

[Matt Turner]

The 21264 memset loop issued its wh64 write hint one 64-byte line ahead of
the stores that consume it.  That is far too close, and the cost is not
subtle: on an EV68CB (DS15, 2MB off-chip Bcache) a cold 256KB memset ran
at 453k cycles with the one-line hint against 406k with no hint issued at
all, so the wh64 was not merely ineffective but a net loss.

Issue it four lines (256 bytes) ahead instead.  The same measurement gives
151k cycles, a 3.0x improvement, and the curve is flat from three lines out
to eight, so the depth is not delicately tuned -- it only has to clear the
threshold.  Four lines is at least as fast as one at every size in both the
hot and cold cases, so no size range regresses.  This matches the CWG,
which asks for at least two cache blocks of lead per stream and up to 8/n
blocks for n streams against the 21264's 8-entry MAF; a one-line hint was
below the architectural minimum.

The mechanism is Mbox replay traps, not fill latency.  Counted with perf on
the 21264 (raw event r4), the one-line hint takes 69885 replays per cold
256KB call and the four-line hint takes 11467, an 83% reduction, while
Bcache misses are unchanged at 4162 against 4125.  No extra data is moved;
the loop was simply trapping and replaying its own store stream, at close
to one replay per instruction retired.  wh64 has to claim the line before
the stores reach it, and unlike a load prefetch there is no fill latency
for a late hint to hide behind -- it just collides.

The end guard moves with the distance: the hint consumed at trip N+1 covers
[$5+256, $5+320), so that trip needs 40 quads remaining, and it is computed
at trip N, which therefore needs 48.  When fewer remain the target falls
back to the block the next trip stores, which is always in range.  This
matters because wh64 makes the entire 64-byte block UNPREDICTABLE, so a
hint that runs past the end of the region would corrupt memory.

320 does not fit the 8-bit addq literal field, so the address computation
becomes an lda; both issue on the E box, so the schedule is unchanged.

Verified on an EV68CB against the C memset over every size class from 0 to
64KB at 72 destination alignments, with 512-byte guard regions either side
checked for stray writes.
---
 sysdeps/alpha/alphaev6/memset.S | 31 +++++++++++++++++++++++++------
 1 file changed, 25 insertions(+), 6 deletions(-)

diff --git ./sysdeps/alpha/alphaev6/memset.S ./sysdeps/alpha/alphaev6/memset.S
index 5c1e307cdf..fb01baa795 100644
--- ./sysdeps/alpha/alphaev6/memset.S
+++ ./sysdeps/alpha/alphaev6/memset.S
@@ -158,19 +158,38 @@ $bigalign:
 	 * Scratch registers available: $7, $2, $4, $1
 	 * We know that we'll be taking a minimum of one trip through.
 	 * CWG Section 3.7.6: do not expect a sustained store rate of > 1/cycle
-	 * Assumes the wh64 needs to be for 2 trips through the loop in the future.
-	 * The wh64 is issued on for the starting destination address for trip +2
-	 * through the loop, and if there are less than two trips left, the target
-	 * address will be for the current trip.
+	 *
+	 * The wh64 is issued five trips ahead of the block it is computed on,
+	 * so that when it is consumed at the top of the next trip it names the
+	 * block four lines (256 bytes) beyond the stores of that trip.  A
+	 * shorter lead does not work: wh64 has to claim the line before the
+	 * stores reach it and, unlike a load prefetch, there is no fill latency
+	 * for a late hint to hide behind.  Measured on an EV68CB (DS15, 2MB
+	 * off-chip Bcache), a one-line lead is 453k cycles for a cold 256KB
+	 * memset against 406k for issuing no hint at all -- i.e. it is worse
+	 * than useless -- while four lines is 151k, a 3.0x improvement.  The
+	 * curve is flat from three lines out to eight, so the exact depth in
+	 * that range does not matter; being under it does.
+	 *
+	 * If fewer than four lines remain the hint would run off the end of the
+	 * region, and wh64 makes the whole 64-byte block UNPREDICTABLE, so the
+	 * target falls back to the block the next trip stores.  That is always
+	 * in range and still avoids that block's read-for-ownership.
+	 *
+	 * Guard arithmetic: the hint consumed at trip N+1 covers
+	 * [$5+256, $5+320), so trip N+1 needs 40 quads left; it is computed at
+	 * trip N, which therefore needs 48.
 	 */
 
 $do_wh64:
 	wh64	($4)		# L1 : memory subsystem write hint
-	subq	$3, 24, $2	# E : For determining future wh64 addresses
+	subq	$3, 48, $2	# E : >= 48 quads left? (see guard note above)
 	stq	$17, 0($5)	# L :
 	nop			# E :
 
-	addq	$5, 128, $4	# E : speculative target of next wh64
+	lda	$4, 320($5)	# E : speculative target of next wh64 (4 lines
+				#     ahead of the trip that consumes it; 320
+				#     exceeds the 8-bit addq literal field)
 	stq	$17, 8($5)	# L :
 	stq	$17, 16($5)	# L :
 	addq	$5, 64, $7	# E : Fallback address for wh64 (== next trip addr)
-- 
2.54.0