nptl: futex_lock_pi deadlock detection provides valuable information but it is turned into a rather cryptic assertion failure

[Moritz KLAMMLER (FERCHAU)]

Dear glibc maintainers,

in the event that two or more threads are about to mutually deadlock
each other via a cyclic lock chain made up of two or more
priority-inheriting futexes, the Linux kernel can, and will (by default,
free of any extra charge), detect and reliably report this situation.
It will do so by returning -EDEADLK from the FUTEX_LOCK_PI syscall.

The documentation on the respective man page might be somewhat
misleading in that regard (it only mentions deadlock via the same thread
attempting to lock a futex which it already owns, which is already
checked for beforehand by NPTL and not what's happening here).
But the kernel code is quite clear about its intent
(c.f. __rt_mutex_start_proxy_lock, currently defined in rtmutex_api.c
but moved around not too long ago):

/* We enforce deadlock detection for futexes */
ret = task_blocks_on_rt_mutex(lock, waiter, task, NULL, RT_MUTEX_FULL_CHAINWALK, wake_q);

This feedback could be immensely useful.  Both, for potentially
resolving the situation in the application at runtime, if it was written
to handle it, or at least as a hint to the developer debugging the
issue.

Alas, the usefulness of this information is currently reduced by the
NPTL layer, which doesn't return it to the application but runs into the
following assertion (in __pthread_mutex_lock_full).

assert(e != EDEADLK || (kind != PTHREAD_MUTEX_ERRORCHECK_NP && kind != PTHREAD_MUTEX_RECURSIVE_NP));

Not only does this deprive the application of any chance to handle the
problem, the generated error message might also be rather cryptic to
decipher for the average programmer.

I think the "problem" here is that glibc seems to assume that after
checking that the current thread doesn't already own this particular
(recursive or error-checked) mutex, the FUTEX_LOCK_PI syscall can't
return -EDEADLK anymore.  But it still might, if the (non-trivial)
deadlock would occur via two or more separate mutexes.

I'm insufficiently educated to judge whether POSIX has any requirements
regarding what should happen in this particular situation.  But it seems
to me that running into an assertion could only be considered compliant
if this situation constitutes undefined behavior in the first place.  In
which case any other course of action might be just as valid and the
matter could be considered a QoI topic for glibc's pthread
implementation.

The most desirable behavior, in my personal opinion, would be for
pthread_mutex_lock to return EDADLK to the application.  Which then
might either try to resolve the situation or decide to terminate itself
after all.  Of course, once glibc implements this, programmers will
start depending on it, so that decision might be difficult if not
impossible to revise again in the future and therefore probably
shouldn't be taken lightly.  I also don't know how committed the Linux
developers are to guarantee that full deadlock detection on PI futexes
can be depended on in the future.

The second best choice, in my opinion, would be to keep this as a fatal
error inside glibc for now, but improve the error message.  For example:

> Circular lock dependency chain (deadlock) detected by thread $ID
> attempting to acquire mutex $ADDR.

At least, this would tell the programmer unfamiliar with glibc internals
where to start looking for the problem (i.e. in their own code).

I'm not advocating for this third option, but it would of course also be
a possibility to actually deadlock the application (as would happen for
non-PI mutexes).

If there is interest in either of these changes, I'd be happy to supply
a concrete patch for further discussion.

For the time being, please consider the below example program to
reproduce and observe the current behavior.

Best Regards,
Moritz
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: main.c
URL: <https://sourceware.org/pipermail/libc-alpha/attachments/20251216/9ef73930/attachment-0001.c>