nptl: futex_lock_pi deadlock detection provides valuable information but it is turned into a rather cryptic assertion failure

[Adhemerval Zanella Netto]

On 16/12/25 07:15, Moritz KLAMMLER (FERCHAU) wrote:
> Dear glibc maintainers,
> 
> in the event that two or more threads are about to mutually deadlock
> each other via a cyclic lock chain made up of two or more
> priority-inheriting futexes, the Linux kernel can, and will (by default,
> free of any extra charge), detect and reliably report this situation.
> It will do so by returning -EDEADLK from the FUTEX_LOCK_PI syscall.
> 
> The documentation on the respective man page might be somewhat
> misleading in that regard (it only mentions deadlock via the same thread
> attempting to lock a futex which it already owns, which is already
> checked for beforehand by NPTL and not what's happening here).
> But the kernel code is quite clear about its intent
> (c.f. __rt_mutex_start_proxy_lock, currently defined in rtmutex_api.c
> but moved around not too long ago):
> 
> /* We enforce deadlock detection for futexes */
> ret = task_blocks_on_rt_mutex(lock, waiter, task, NULL, RT_MUTEX_FULL_CHAINWALK, wake_q);
> 
> This feedback could be immensely useful.  Both, for potentially
> resolving the situation in the application at runtime, if it was written
> to handle it, or at least as a hint to the developer debugging the
> issue.
> 
> Alas, the usefulness of this information is currently reduced by the
> NPTL layer, which doesn't return it to the application but runs into the
> following assertion (in __pthread_mutex_lock_full).
> 
> assert(e != EDEADLK || (kind != PTHREAD_MUTEX_ERRORCHECK_NP && kind != PTHREAD_MUTEX_RECURSIVE_NP));
> 
> Not only does this deprive the application of any chance to handle the
> problem, the generated error message might also be rather cryptic to
> decipher for the average programmer.
> 
> I think the "problem" here is that glibc seems to assume that after
> checking that the current thread doesn't already own this particular
> (recursive or error-checked) mutex, the FUTEX_LOCK_PI syscall can't
> return -EDEADLK anymore.  But it still might, if the (non-trivial)
> deadlock would occur via two or more separate mutexes.
> 
> I'm insufficiently educated to judge whether POSIX has any requirements
> regarding what should happen in this particular situation.  But it seems
> to me that running into an assertion could only be considered compliant
> if this situation constitutes undefined behavior in the first place.  In
> which case any other course of action might be just as valid and the
> matter could be considered a QoI topic for glibc's pthread
> implementation.
> 
> The most desirable behavior, in my personal opinion, would be for
> pthread_mutex_lock to return EDADLK to the application.  Which then
> might either try to resolve the situation or decide to terminate itself
> after all.  Of course, once glibc implements this, programmers will
> start depending on it, so that decision might be difficult if not
> impossible to revise again in the future and therefore probably
> shouldn't be taken lightly.  I also don't know how committed the Linux
> developers are to guarantee that full deadlock detection on PI futexes
> can be depended on in the future.
> 
> The second best choice, in my opinion, would be to keep this as a fatal
> error inside glibc for now, but improve the error message.  For example:
> 
>> Circular lock dependency chain (deadlock) detected by thread $ID
>> attempting to acquire mutex $ADDR.
> 
> At least, this would tell the programmer unfamiliar with glibc internals
> where to start looking for the problem (i.e. in their own code).
> 
> I'm not advocating for this third option, but it would of course also be
> a possibility to actually deadlock the application (as would happen for
> non-PI mutexes).
> 
> If there is interest in either of these changes, I'd be happy to supply
> a concrete patch for further discussion.
> 
> For the time being, please consider the below example program to
> reproduce and observe the current behavior.
> 
> Best Regards,
> Moritz

My understanding is per POSIX EDEADLK *shall* be returned only for ERRORCHECK
mutexes [1], and this is what glibc current does. In your example, if you
change PTHREAD_MUTEX_RECURSIVE to PTHREAD_MUTEX_ERRORCHECK the testcase will
get the alarm clock instead. One caveat is we don't have a recursive/error-check
mutex type, so relock is always reported as an error.

However POSIX also states that EDEADLK is also a *may* failure so I think we
can also return instead of assert here.  If kernel does give us enough information
I agree that it should a QoI to not require a error-check mutex to have the
proper error return. 

[1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/pthread_mutex_lock.html