Race in pthread_mutex_lock while promoting to PTHREAD_MUTEX_ELISION_NP.

[Stefan Liebler]

Hi,

while looking for a possible reason for "pthread_mutex_lock.c:79: 
__pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.", 
I've found a race condition if pthread_mutex_lock() is called for the 
"first time" by multiple threads.
Note: Lock elision has to be supported and enabled (export 
GLIBC_TUNABLES=glibc.elision.enable=1).

The attached test program creates threads which runs multiple iterations 
of the following sequence:
-pthread_mutex_init(): called only by one thread. Initializing with 
default mutex attributes.
-pthread_mutex_lock()
-pthread_mutex_unlock()
-pthread_mutex_destroy(): called only by one thread.

Here are some code snippets in order to follow the explanation below:
-nptl/pthread_mutex_lock.c:
  62int
  63__pthread_mutex_lock (pthread_mutex_t *mutex)
  64{
  65  unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex);
  ...
  73  if (__glibc_likely (type == PTHREAD_MUTEX_TIMED_NP))
  74    {
  75      FORCE_ELISION (mutex, goto elision);
  76    simple:
  77      /* Normal mutex.  */
  78      LLL_MUTEX_LOCK (mutex);
  79      assert (mutex->__data.__owner == 0);
  80    }
  81#ifdef HAVE_ELISION
  82  else if (__glibc_likely (type == PTHREAD_MUTEX_TIMED_ELISION_NP))
  83    {
  84  elision: __attribute__((unused))
  85      /* This case can never happen on a system without elision, 

  86         as the mutex type initialization functions will not 

  87         allow to set the elision flags.  */
  88      /* Don't record owner or users for elision case.  This is a 

  89         tail call.  */
  90      return LLL_MUTEX_LOCK_ELISION (mutex);
  91    }
  92#endif
...
156  /* Record the ownership.  */
157  mutex->__data.__owner = id;
158#ifndef NO_INCR
159  ++mutex->__data.__nusers;
160#endif


-sysdeps/unix/sysv/linux/s390/force-elision.h:
Note: This file is identical to the equivalent files for x86 and powerpc!
19/* Automatically enable elision for existing user lock kinds.  */
20#define FORCE_ELISION(m, s) \
21  if (__pthread_force_elision \
22      && (m->__data.__kind & PTHREAD_MUTEX_ELISION_FLAGS_NP) == 0) \
23    { \
24      mutex->__data.__kind |= PTHREAD_MUTEX_ELISION_NP; \
25      s; \
26    }

-nptl/pthread_mutex_unlock.c:
  36__pthread_mutex_unlock_usercnt (pthread_mutex_t *mutex, int decr)
  37{
  38  int type = PTHREAD_MUTEX_TYPE_ELISION (mutex);
...
  43  if (__builtin_expect (type, PTHREAD_MUTEX_TIMED_NP)
  44      == PTHREAD_MUTEX_TIMED_NP)
  45    {
  46      /* Always reset the owner field.  */
  47    normal:
  48      mutex->__data.__owner = 0;
  49      if (decr)
  50        /* One less user.  */
  51        --mutex->__data.__nusers;
...
  60  else if (__glibc_likely (type == PTHREAD_MUTEX_TIMED_ELISION_NP))
  61    {
  62      /* Don't reset the owner/users fields for elision.  */
  63      return lll_unlock_elision (mutex->__data.__lock, 
mutex->__data.__elision,
  64                                      PTHREAD_MUTEX_PSHARED (mutex));

-nptl/pthread_mutex_destroy.c:
26__pthread_mutex_destroy (pthread_mutex_t *mutex)
27{
28  LIBC_PROBE (mutex_destroy, 1, mutex);
29
30  if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0
31      && mutex->__data.__nusers != 0)
32    return EBUSY;



Assumption: Two threads are calling pthread_mutex_lock at the same time 
and have already loaded the mutex-type in line 65 with the type 
PTHREAD_MUTEX_TIMED_NP.

Thread 1 promotes the mutex to PTHREAD_MUTEX_ELISION_NP (see 
FORCE_ELISION in line 24) and jumps to LLL_MUTEX_LOCK_ELISION (see line 90).

Thread 2 is checking the mutex type in FORCE_ELISION (line 22).
As the mutex-type is already set to PTHREAD_MUTEX_ELISION_NP, the 
condition is false and the "Normal mutex" (line 77) is processed.
Note: The "normal mutex" records the ownership (see line 156).

Both threads are calling pthread_mutex_unlock and are loading mutex-type 
in line 38 with the type PTHREAD_MUTEX_TIMED_ELISION_NP.
Thus the ownership is not resetted (see line 62)!

The call to pthread_mutex_destroy is returning with EBUSY
as __nusers == 1 (see line 31).
Note: Although thread 2 has successfully called pthread_mutex_unlock, it 
is marked as the current owner of the mutex.

If a third thread has also loaded the mutex-type PTHREAD_MUTEX_TIMED_NP 
in pthread_mutex_lock, then the "assert (mutex->__data.__owner == 0)" in 
line 79 is triggered.



When I am running the attached test program on s390x, I can trigger the 
case "pthread_mutex_destroy()==EBUSY" within some thousand iterations.
And I could trigger the assertion while single-stepping in GDB.

Can somebody test and confirm this on x86, power, s390?

How can we solve this issue?

Bye
Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tst_pthread_lock_assertion.c
Type: text/x-csrc
Size: 3961 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20180524/5de33eaf/attachment.bin>