[PATCH v3] nptl: Optimize trylock for high cache contention workloads (BZ #33704)

[Sunil K Pandey]

Check lock availability before acquisition to reduce cache line
bouncing.  Significantly improves trylock throughput on multi-core
systems under heavy contention.

mutex trylock throughput benchmark runtime

Core  baseline         patch
====  ===============  =============
80    66Min 32Sec      45Sec
24     8Min 4Sec       14Sec
12     5Min 14Sec      34Sec
8      3Min 40Sec      20Sec

Throughput data(Core count: 80)

tcount baseline	pthread_mutex_trylock  Improvement
====== ======== =====================  ===========
80     232112   3119301                13.44x
64     247737   937424                 3.78x
32     441441   888738                 2.01x
16     863201   1529987                1.77x
8      1587950  2775892                1.75x
4      3208012  6487683                2.02x
2      7729917  11811184               1.53x

Tested on x86_64.

Fixes BZ #33704.

Add recursive.
---
 nptl/pthread_mutex_trylock.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/nptl/pthread_mutex_trylock.c b/nptl/pthread_mutex_trylock.c
index 94621cc254..fe9ba8da67 100644
--- a/nptl/pthread_mutex_trylock.c
+++ b/nptl/pthread_mutex_trylock.c
@@ -47,7 +47,8 @@ ___pthread_mutex_trylock (pthread_mutex_t *mutex)
 	  return 0;
 	}
 
-      if (lll_trylock (mutex->__data.__lock) == 0)
+      if (atomic_load_relaxed (&(mutex->__data.__lock)) == 0
+	  && lll_trylock (mutex->__data.__lock) == 0)
 	{
 	  /* Record the ownership.  */
 	  mutex->__data.__owner = id;
@@ -60,7 +61,10 @@ ___pthread_mutex_trylock (pthread_mutex_t *mutex)
     case PTHREAD_MUTEX_TIMED_NP:
     case PTHREAD_MUTEX_ADAPTIVE_NP:
     case PTHREAD_MUTEX_ERRORCHECK_NP:
-      if (lll_trylock (mutex->__data.__lock) != 0)
+      /* Mutex type is already loaded, lock check overhead should
+         be minimal.  */
+      if (atomic_load_relaxed (&(mutex->__data.__lock)) != 0
+	  || lll_trylock (mutex->__data.__lock) != 0)
 	break;
 
       /* Record the ownership.  */
-- 
2.51.0