[PATCH 14/20] alpha: store four quads a trip below the ev6 wh64 loop

[Matt Turner]

Sets shorter than 128 bytes never reach the wh64 loop and were written a
quadword at a time.  Like short copies they are issue-bound -- they miss
nothing and replay nothing -- so what they want is fewer instructions per
byte, and four stores a trip delivers that.  Measured on an EV68CB, hot,
cycles per call:

  size    before   after
  112       56.6    31.2   1.81x
  64        36.2    23.1   1.57x
  120       51.3    32.9   1.56x
  127       51.3    36.2   1.42x
  32        25.8    20.1   1.28x
  96        33.2    27.4   1.21x

Sets below 32 bytes pay one cycle for the test that skips the new loop,
about 5% where the call is only 17 to 21 cycles.

Placement took three attempts and is worth recording.  Letting the trailing
quads of a long set share the four-quad loop cost 5-8% between 128 and 512
bytes, so the wh64 path deliberately still falls into the one-quad loop.
Putting the block inline above $loop cost 13% at 512 bytes even after that,
purely by growing the straight-line distance the long path covers.  Out of
line past the return paths, 128 through 384, 1KB and 2KB all come out level.

Three isolated sizes -- 288, 320 and 512 bytes -- still measure about 10%
down.  They are not a smooth function of length, since 256 and 352 are
level, and they execute exactly the instructions they did before, so this
is cache aliasing against the grown function rather than anything the new
loop does.  No placement tried removed it.  Against that, the geometric
mean over the whole range measured is about 1.09x.

Verified on an EV68CB against the C memset for every length from 0 to 600
at 72 destination alignments, with 512-byte guard regions either side
checked for stray writes.
---
 sysdeps/alpha/alphaev6/memset.S | 46 +++++++++++++++++++++++++++++++--
 1 file changed, 44 insertions(+), 2 deletions(-)

diff --git ./sysdeps/alpha/alphaev6/memset.S ./sysdeps/alpha/alphaev6/memset.S
index fb01baa795..4273604f6e 100644
--- ./sysdeps/alpha/alphaev6/memset.S
+++ ./sysdeps/alpha/alphaev6/memset.S
@@ -117,7 +117,7 @@ $aligned:
 	 * 21264 family (EV6/67/68) and 3 for the 21364 family (EV7/EV79).
 	 */
 	subq	$3, 16, $4	# E : Only try to unroll if > 128 bytes
-	blt	$4, $loop	# U : too small -- scalar tail loop
+	blt	$4, $loop_four	# U : too small -- four/one quad loops
 	implver	$2		# E : 2 = 21264 (EV6/67/68), 3 = 21364 (EV7)
 	subq	$2, 3, $2	# E : EV7 -> 0
 
@@ -126,7 +126,7 @@ $aligned:
 	and	$16, 0x3f, $2	# E : Forward work (only useful for unrolled loop)
 	subq	$3, 16, $4	# E : Only try to unroll if > 128 bytes
 	subq	$2, 0x40, $1	# E : bias counter (aligning stuff 0mod64)
-	blt	$4, $loop	# U :
+	blt	$4, $loop_four	# U :
 
 	/*
 	 * We know we've got at least 16 quads, minimum of one trip
@@ -287,5 +287,47 @@ $end:
 	nop
 	ret $31,($26),1		# L0 :
 
+	/*
+	 * Sets shorter than 128 bytes come here instead of the one-quad loop.
+	 * Like short copies they are issue-bound -- nothing misses and nothing
+	 * replays -- so four stores a trip is worth a fifth to nearly half the
+	 * time between 32 and 127 bytes.
+	 *
+	 * Out of line, past every return path, on purpose.  Two other
+	 * placements were measured.  Sharing the four-quad loop with the
+	 * trailing quads of a long set cost 5-8% between 128 and 512 bytes, so
+	 * the wh64 path deliberately keeps the one-quad loop it had.  Putting
+	 * the block inline just above $loop cost 13% at 512 bytes even with
+	 * that fixed, purely by growing the straight-line distance the long
+	 * path covers.  Down here, 128 through 384, 1KB and 2KB come out level.
+	 *
+	 * A few isolated sizes -- 288, 320 and 512 bytes -- still measure about
+	 * 10% down, and they are not a smooth function of length: 256 and 352
+	 * are level while 288 and 320 are not, and all of them run exactly the
+	 * same instructions as before.  That is cache aliasing against the
+	 * grown function, not anything this loop does; no placement tried made
+	 * it go away entirely.
+	 */
+	.align 4
+$loop_four:
+	subq	$3, 4, $4	# E : four whole quads to write?
+	blt	$4, $loop	# U : no -- one at a time
+	nop			# E :
+	nop			# E :
+
+$loop_four_body:
+	stq	$17, 0($5)	# L : 32 bytes
+	stq	$17, 8($5)	# L :
+	stq	$17, 16($5)	# L :
+	stq	$17, 24($5)	# L :
+
+	addq	$5, 32, $5	# E : Inc address
+	subq	$3, 4, $3	# E : four fewer quads
+	subq	$3, 4, $4	# E : room for another four?
+	bge	$4, $loop_four_body # U :
+
+	beq	$3, $no_quad	# U : exactly consumed
+	br	$31, $loop	# U : 1..3 quads left
+
 	END(memset)
 libc_hidden_builtin_def (memset)
-- 
2.54.0