NPTL mutex and the scheduling priority

[Atsushi Nemoto <anemo at mba dot ocn dot ne dot jp>]

Hi.  I found that it seems NPTL's mutex does not follow the scheduling
parameter.  If some threads were blocked by getting a single
mutex_lock, I expect that a thread with highest priority got the lock
first, but current NPTL's behaviour is different.

Here is a sample program.  This creates four FIFO-class thread with
different priorities and these threads try to get a same mutex.

--- foo.c ---
#include <stdio.h>
#include <pthread.h>
#include <time.h>

static pthread_mutex_t mutex;
static volatile int val;

static void *thread_func(void *arg)
{
	int v;
	pthread_mutex_lock(&mutex);
	v = val++;
	pthread_mutex_unlock(&mutex);
	printf("thread-%ld got %d\n", (long)arg, v);
	return NULL;
}

int main(int argc, char **argv)
{
	struct sched_param param;
	struct timespec ts;
	pthread_t tid[4];
	pthread_attr_t attr;
	int i;

#if 0
	int policy;
	pthread_getschedparam(pthread_self(), &policy, &param);
	policy = SCHED_FIFO;
	param.sched_priority = 99;
	pthread_setschedparam(pthread_self(), policy, &param);
#endif

	pthread_mutex_init(&mutex, NULL);
	pthread_mutex_lock(&mutex);
	pthread_attr_init(&attr);
	pthread_attr_setschedpolicy(&attr, SCHED_FIFO);
	pthread_attr_getschedparam(&attr, &param);
	pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);
	for (i = 0; i < sizeof(tid) / sizeof(tid[0]); i++) {
		param.sched_priority = 50 + i * 10;
		pthread_attr_setschedparam(&attr, &param);
		pthread_create(&tid[i], &attr, thread_func, (void *)i);
		printf("thread-%d pri %d\n", i, param.sched_priority);
	}
	
	ts.tv_sec = 3;
	ts.tv_nsec = 0;
	nanosleep(&ts, NULL);

	val++;
	pthread_mutex_unlock(&mutex);

	for (i = 0; i < sizeof(tid) / sizeof(tid[0]); i++)
		pthread_join(tid[i], NULL);
	return 0;
}
--- foo.c ---

I thought a thread with highest priority (thread-3) will get the
mutex first, so I expected:

thread-0 pri 50
thread-1 pri 60
thread-2 pri 70
thread-3 pri 80
thread-3 got 1
thread-2 got 2
thread-1 got 3
thread-0 got 4

but with NPTL (glibc 2.4, kernel 2.6.16, mips/i386) I got:

thread-0 pri 50
thread-1 pri 60
thread-2 pri 70
thread-3 pri 80
thread-3 got 4
thread-2 got 3
thread-1 got 2
thread-0 got 1

I can get the expected result with linuxthreads (glibc 2.3.6).

I also found that I can get expected result with NPTL if I enabled the
"#if 0" block in the sample program.

Is this glibc/NPTL issue, or kernel/futex issue?  (or my expectation
is wrong?)

---
Atsushi Nemoto