[PATCH 2/2] New pthread rwlock that is more scalable.

[Xin Zhao]

Dear Torvald,

On Wed, 27 Jul 2016 23:47:00 -0000 Jiri Slaby <jirislaby@kernel.org> wrote:

> +   This leaves small windows where blocking using futexes is not possible
> +   although a primary writer exists, but in turn decreases complexity of the
> +   writer--writer synchronization and does not affect correctness.


In our system, we encountered a situation where a writer repeatedly executes the
atomic_load_relaxed operation in the code snippet below, causing the CPU to be
fully utilized. Since our system has high real-time requirements, there are
certain real-time threads. If a real-time thread A acquires the
PTHREAD_RWLOCK_WRLOCKED flag but has not yet executed and is preempted by another
higher-priority real-time thread B, it can lead to the lower-priority real-time
thread being unable to get scheduled, causing the CPU to be fully utilized and
unable to recover, which can result in RCU stalls.

	    {
	      /* If we cannot block on __writers_futex because there is no
		 primary writer, or we cannot set PTHREAD_RWLOCK_FUTEX_USED,
		 we retry.  We must reload __readers here in case we cannot
		 block on __writers_futex so that we can become the primary
		 writer and are not stuck in a loop that just continuously
		 fails to block on __writers_futex.  */
---->     r = atomic_load_relaxed (&rwlock->__data.__readers);
	      continue;
	    }

In the situation described above, if thread A is a regular CFS thread, this situation
might be relatively better since CFS will always have a chance to get scheduled. However,
during the period when CFS is preempted and unable to get scheduled, higher-priority
tasks may repeatedly retry to see when the bit 0 of wf becomes 1. This polling wait can
cause CPU spikes, leading to stability issues in the system.