[PATCH] malloc: Fix ABBA deadlock in fork handlers

[H.J. Lu]

On Tue, May 12, 2026 at 2:58 PM Cuishuangjin <YS.cuishuangjin@h3c.com> wrote:
>
> Hi Florian,
>
> Thank you for your feedback. I apologize for the earlier garbled patch summary — let me clarify the actual issue and how it differs from bug 34021.
>
> Summary of the Issue
> This is not the same as bug 34021. While both involve allocator locks during fork(), the lock inversion occurs in a different code path and involves different locks.
>
> Lock Order Difference
> ScenarioLocks InvolvedOrder in Thread AOrder in Thread B
> Bug 34021allocator_lock ↔ _IO_proc_file_chain_lockallocator_lock → _IO_proc_file_chain_lock_IO_proc_file_chain_lock → allocator_lock
> This Issueatfork_lock ↔ malloc_lockatfork_lock → malloc_lockmalloc_lock → atfork_lock
> Code Path Analysis
> Bug 34021 occurs when:
>
> Thread A (fork()): __run_prefork_handlers() → allocator_lock → _IO_proc_file_chain_lock
> Thread B (popen()): _IO_proc_file_chain_lock → malloc() → allocator_lock
> This issue occurs in the atfork handler registration/execution path:
>
> Thread A (__run_prefork_handlers):
>
> Holds atfork_lock
> Releases atfork_lock to execute prepare handlers
> Calls ptmalloc_lock_all() → acquires malloc_lock
> Attempts to re-acquire atfork_lock (blocked)
> Thread B (__register_atfork):
>
> Acquires atfork_lock
> Calls fork_handler_list_emplace()
> Calls malloc() → attempts to acquire malloc_lock (blocked)
> Visual Timeline
> Timeline │ Thread A (__run_prefork_handlers)      │ Thread B (__register_atfork)
> ─────────┼────────────────────────────────────────┼─────────────────────────────
>  T1      │ Holds atfork_lock                      │
>  T2      │   lll_unlock(atfork_lock) ───────────► │
>  T3      │ Executes prepare_handler()             │   lll_lock(atfork_lock)
>  T4      │ └─► ptmalloc_lock_all()                │ Holds atfork_lock
>  T5      │ Holds malloc_lock                      │   fork_handler_list_emplace()
>  T6      │   lll_lock(atfork_lock) ◄─ Blocked ─── │ └─► malloc() ◄─ Blocked ─┐
> 复制
> Timeline │ Thread A (__run_prefork_handlers)      │ Thread B (__register_atfork)
> ─────────┼────────────────────────────────────────┼─────────────────────────────
>  T1      │ Holds atfork_lock                      │
>  T2      │   lll_unlock(atfork_lock) ───────────► │
>  T3      │ Executes prepare_handler()             │   lll_lock(atfork_lock)
>  T4      │ └─► ptmalloc_lock_all()                │ Holds atfork_lock
>  T5      │ Holds malloc_lock                      │   fork_handler_list_emplace()
>  T6      │   lll_lock(atfork_lock) ◄─ Blocked ─── │ └─► malloc() ◄─ Blocked ─┐
>
> Why This Matters
> This deadlock affects any custom malloc implementation that:
>
> Uses pthread_atfork for fork-safety
> Registers handlers dynamically during runtime
> Has prepare handlers that acquire malloc locks
> The fix requires ensuring consistent lock ordering between atfork_lock and allocator locks in both the execution path (__run_prefork_handlers) and registration path (__register_atfork).
>
> Proposed Approach
> Weaken the strong definitions of __malloc_fork_lock_parent and related symbols in glibc
> Strengthen malloc_lock management in __register_atfork prepare handlers
> Document the requirements for custom malloc implementations
> As you noted, we'll need to:
>
> Clearly mention custom malloc implications in commit messages
> Update the "Replacing malloc" documentation
> Add appropriate symbol versioning (GLIBC_2.44) if new interfaces are introduced
> Please let me know if this clarifies the distinction from bug 34021.
>
> Best regards,
> ShuangJin Cui
>

Please open a glibc bug report with a testcase.

-- 
H.J.