[PATCH] malloc: Fix ABBA deadlock in fork handlers
H.J. Lu
hjl.tools@gmail.com
Thu Jun 18 04:26:21 GMT 2026
On Thu, Jun 18, 2026 at 11:04 AM Cuishuangjin <YS.cuishuangjin@h3c.com> wrote:
>
> Dear H.J. and glibc maintainers,
>
> Thank you for reviewing our findings on the ABBA deadlock issue. Based on your suggestion, I have opened a glibc Bugzilla report with all relevant details:
>
> Summary of Provided Materials
> The report includes:
>
> Complete Test Case - Demonstrates the deadlock under two different allocator environments
> Reproduction Steps - Clear instructions to reproduce the issue
> GDB Backtraces - Showing the deadlock call stacks
> Lock Ordering Analysis - Detailed analysis of the lock inversion
> Two Independent Reproductions:
> Using ptmalloc standalone allocator
> Using a minimal allocator interposer (independent of ptmalloc internals)
> Key Technical Insights
> The analysis confirms that the issue is not implementation-specific but rather a design-level lock ordering problem affecting any malloc implementation that:
>
> Uses pthread_atfork() for fork-safety
> Dynamically registers handlers at runtime
> Has prepare handlers that acquire malloc locks
> Request for Guidance
> Could you please help review whether:
>
> The analysis is accurate - Does our understanding of the lock inversion match glibc's internal design?
> The fix direction is appropriate - Is the proposed approach of ensuring consistent lock ordering between atfork_lock and allocator locks suitable for glibc?
> There are additional considerations - Are there other aspects we should address in the patch?
> Next Steps Suggestion
> If the analysis is deemed correct, we would appreciate guidance on:
Do you have a glibc bug?
> The preferred patch submission process
> Any specific coding standards or guidelines to follow
> Whether this should target a particular glibc branch
> We are ready to prepare a formal patch once we receive confirmation that the approach is acceptable.
>
> Thank you for your time and expertise.
>
> Best regards,
> ShuangJin Cui
> H3C Technologies
>
>
> -----邮件原件-----
> 发件人: cuishuangjin ys43405(CW, RD)
> 发送时间: 2026年5月26日 14:26
> 收件人: 'H.J. Lu' <hjl.tools@gmail.com>
> 抄送: Florian Weimer <fweimer@redhat.com>; libc-alpha@sourceware.org; zhangchun (操作系统开发部/内核开发部, RD) <zhang.chunA@h3c.com>; lumingxiang (RD) <lu.mingxiang@h3c.com>
> 主题: 答复: [PATCH] malloc: Fix ABBA deadlock in fork handlers
>
> Hi H.J.,
>
> Thank you for the suggestion.
>
> I reproduced the deadlock using the same testcase under two different allocator environments.
>
> The testcase continuously:
> - registers `pthread_atfork()` handlers
> - performs concurrent `fork()`
> - creates heavy allocator contention
>
> Testcase:
>
> #define _GNU_SOURCE
>
> #include <pthread.h>
> #include <stdatomic.h>
> #include <sched.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <sys/wait.h>
> #include <unistd.h>
>
> static atomic_int running = 1;
>
> static void prepare(void)
> {
> /*
> * Intentionally enlarge the prefork handler lock window
> */
> usleep(1000000);
> }
>
> static void parent(void)
> {
> }
>
> static void child(void)
> {
> }
>
> /*
> * Register atfork handlers at high frequency
> *
> * Target:
> * __register_atfork
> * -> lll_lock(atfork_lock)
> * -> realloc(...)
> * -> ptmalloc arena lock
> */
> static void *register_thread(void *arg)
> {
> while (atomic_load(&running)) {
>
> pthread_atfork(
> prepare,
> parent,
> child);
>
> /*
> * Force allocator activity
> */
> void *p = malloc(128 * 1024);
>
> memset(p, 0, 128 * 1024);
>
> free(p);
> }
>
> return NULL;
> }
>
> /*
> * High-frequency fork
> *
> * Target:
> * fork
> * -> __run_prefork_handlers
> * -> __malloc_fork_lock_parent
> * -> arena lock
> */
> static void *fork_thread(void *arg)
> {
> while (atomic_load(&running)) {
>
> pid_t pid = fork();
>
> if (pid == 0) {
> _exit(0);
> }
>
> if (pid > 0) {
> waitpid(pid, NULL, 0);
> }
> }
>
> return NULL;
> }
>
> /*
> * Aggressively create arena contention
> */
> static void *malloc_thread(void *arg)
> {
> while (atomic_load(&running)) {
>
> void *p[256];
>
> for (int i = 0; i < 256; i++) {
> p[i] = malloc(4096);
>
> if (p[i])
> memset(p[i], 0xaa, 4096);
> }
>
> for (int i = 0; i < 256; i++) {
> free(p[i]);
> }
> }
>
> return NULL;
> }
>
> int main(void)
> {
> /*
> * Reduce the number of arenas
> * to increase ptmalloc lock contention probability
> */
> setenv("MALLOC_ARENA_MAX", "1", 1);
>
> pthread_t reg_threads[4];
> pthread_t malloc_threads[8];
> pthread_t fork_tid;
>
> for (int i = 0; i < 4; i++) {
> pthread_create(
> ®_threads[i],
> NULL,
> register_thread,
> NULL);
> }
>
> for (int i = 0; i < 8; i++) {
> pthread_create(
> &malloc_threads[i],
> NULL,
> malloc_thread,
> NULL);
> }
>
> pthread_create(
> &fork_tid,
> NULL,
> fork_thread,
> NULL);
>
> pthread_join(fork_tid, NULL);
>
> return 0;
> }
>
>
> Build:
> gcc -O0 -g -pthread testcase.c -o testcase
>
>
> Environment 1: ptmalloc standalone allocator =============================================
>
> Run:
> LD_PRELOAD=/usr/lib64/libptmalloc_standalone.so \
> MALLOC_ARENA_MAX=1 \
> ./testcase
>
> This reliably reproduces the deadlock.
>
> Observed gdb backtrace:
>
>
> Thread 14 (Thread 0x7fdefb56e6c0 (LWP 108102) "new_test11"):
> #0 0x00007fdf015fed9b in __lll_lock_wait_private () from /usr/lib64/libc.so.6
> #1 0x00007fdf01668ea8 in __run_prefork_handlers () from /usr/lib64/libc.so.6
> #2 0x00007fdf016500ae in fork () from /usr/lib64/libc.so.6
> #3 0x0000000000401273 in fork_thread ()
> #4 0x00007fdf01601fbe in start_thread () from /usr/lib64/libc.so.6
> #5 0x00007fdf01680fb4 in clone () from /usr/lib64/libc.so.6
>
> Thread 13 (Thread 0x7fdefbd6f6c0 (LWP 108101) "new_test11"):
> #0 0x00007fdf015fee00 in __lll_lock_wait () from /usr/lib64/libc.so.6
> #1 0x00007fdf01604ec2 in pthread_mutex_lock@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
> #2 0x00007fdf0178403f in internal_malloc () from /usr/lib64/libptmalloc_standalone.so
> #3 0x00007fdf0178450b in malloc () from /usr/lib64/libptmalloc_standalone.so
> #4 0x00000000004012fa in malloc_thread ()
> #5 0x00007fdf01601fbe in start_thread () from /usr/lib64/libc.so.6
> #6 0x00007fdf01680fb4 in clone () from /usr/lib64/libc.so.6
>
>
> Additional threads are blocked inside:
> internal_malloc
> arena_get2
> pthread_mutex_lock
>
> and registration threads are blocked in:
> __register_atfork
>
> This demonstrates the ABBA deadlock between:
> atfork_lock -> malloc_lock
> malloc_lock -> atfork_lock
>
> Environment 2: abstract minimal allocator =========================================
>
> I also reproduced the same issue using a simplified allocator interposer that only models:
> - allocator arena locking
> - fork prepare locking
> - malloc serialization
>
> The allocator is implemented via `LD_PRELOAD` and does not depend on ptmalloc internals.
>
> Simplified allocator:
>
> #define _GNU_SOURCE
>
> #include <dlfcn.h>
> #include <pthread.h>
> #include <stddef.h>
> #include <unistd.h>
>
> static pthread_mutex_t arena_lock =
> PTHREAD_MUTEX_INITIALIZER;
>
> static void *(*real_malloc)(size_t);
> static void (*real_free)(void *);
>
> static void prepare(void)
> {
> /*
> * fork:
> * atfork_lock -> arena_lock
> */
> pthread_mutex_lock(&arena_lock);
>
> /*
> * Enlarge the deadlock window
> */
> usleep(1000);
> }
>
> static void parent(void)
> {
> pthread_mutex_unlock(&arena_lock);
> }
>
> static void child(void)
> {
> pthread_mutex_unlock(&arena_lock);
> }
>
> __attribute__((constructor))
> static void init(void)
> {
> real_malloc = dlsym(RTLD_NEXT, "malloc");
> real_free = dlsym(RTLD_NEXT, "free");
>
> pthread_atfork(
> prepare,
> parent,
> child);
> }
>
> void *malloc(size_t sz)
> {
> pthread_mutex_lock(&arena_lock);
>
> /*
> * Simulate arena lock hold time
> */
> usleep(100);
>
> void *p = real_malloc(sz);
>
> pthread_mutex_unlock(&arena_lock);
>
> return p;
> }
>
> void free(void *p)
> {
> real_free(p);
> }
>
>
> Build:
> gcc -shared -fPIC -ldl -pthread alloc.so.c -o alloc.so
>
> Run:
> LD_PRELOAD=./alloc.so ./testcase
>
> This reproduces the same lock inversion independently of ptmalloc internals.
>
> The issue therefore appears to be in the lock ordering between:
> - `__run_prefork_handlers`
> - `__register_atfork`
> - allocator fork locking
>
> rather than being specific to a particular malloc implementation.
>
> I will open a glibc Bugzilla report with:
> - the testcase
> - reproduction steps
> - gdb backtraces
> - lock ordering analysis
>
> Best regards,
> ShuangJin Cui
>
> -----邮件原件-----
> 发件人: H.J. Lu [mailto:hjl.tools@gmail.com]
> 发送时间: 2026年5月12日 15:00
> 收件人: cuishuangjin ys43405(CW, RD) <YS.cuishuangjin@h3c.com>
> 抄送: Florian Weimer <fweimer@redhat.com>; libc-alpha@sourceware.org; zhangchun (操作系统开发部/内核开发部, RD) <zhang.chunA@h3c.com>; lumingxiang (RD) <lu.mingxiang@h3c.com>
> 主题: Re: [PATCH] malloc: Fix ABBA deadlock in fork handlers
>
>
> 温馨提示： 此邮件来自公司外部，请核实发件人信息，慎点链接与附件。This is an external email. Please verify the sender's information and proceed with caution when clicking links or downloading attachments.
>
>
>
> On Tue, May 12, 2026 at 2:58 PM Cuishuangjin <YS.cuishuangjin@h3c.com> wrote:
> >
> > Hi Florian,
> >
> > Thank you for your feedback. I apologize for the earlier garbled patch summary — let me clarify the actual issue and how it differs from bug 34021.
> >
> > Summary of the Issue
> > This is not the same as bug 34021. While both involve allocator locks during fork(), the lock inversion occurs in a different code path and involves different locks.
> >
> > Lock Order Difference
> > ScenarioLocks InvolvedOrder in Thread AOrder in Thread B Bug
> > 34021allocator_lock ↔ _IO_proc_file_chain_lockallocator_lock →
> > _IO_proc_file_chain_lock_IO_proc_file_chain_lock → allocator_lock This
> > Issueatfork_lock ↔ malloc_lockatfork_lock → malloc_lockmalloc_lock →
> > atfork_lock Code Path Analysis Bug 34021 occurs when:
> >
> > Thread A (fork()): __run_prefork_handlers() → allocator_lock →
> > _IO_proc_file_chain_lock Thread B (popen()): _IO_proc_file_chain_lock
> > → malloc() → allocator_lock This issue occurs in the atfork handler registration/execution path:
> >
> > Thread A (__run_prefork_handlers):
> >
> > Holds atfork_lock
> > Releases atfork_lock to execute prepare handlers Calls
> > ptmalloc_lock_all() → acquires malloc_lock Attempts to re-acquire
> > atfork_lock (blocked) Thread B (__register_atfork):
> >
> > Acquires atfork_lock
> > Calls fork_handler_list_emplace()
> > Calls malloc() → attempts to acquire malloc_lock (blocked) Visual
> > Timeline
> > Timeline │ Thread A (__run_prefork_handlers) │ Thread B (__register_atfork)
> > ─────────┼────────────────────────────────────────┼─────────────────────────────
> > T1 │ Holds atfork_lock │
> > T2 │ lll_unlock(atfork_lock) ───────────► │
> > T3 │ Executes prepare_handler() │ lll_lock(atfork_lock)
> > T4 │ └─► ptmalloc_lock_all() │ Holds atfork_lock
> > T5 │ Holds malloc_lock │ fork_handler_list_emplace()
> > T6 │ lll_lock(atfork_lock) ◄─ Blocked ─── │ └─► malloc() ◄─ Blocked ─┐
> > 复制
> > Timeline │ Thread A (__run_prefork_handlers) │ Thread B (__register_atfork)
> > ─────────┼────────────────────────────────────────┼─────────────────────────────
> > T1 │ Holds atfork_lock │
> > T2 │ lll_unlock(atfork_lock) ───────────► │
> > T3 │ Executes prepare_handler() │ lll_lock(atfork_lock)
> > T4 │ └─► ptmalloc_lock_all() │ Holds atfork_lock
> > T5 │ Holds malloc_lock │ fork_handler_list_emplace()
> > T6 │ lll_lock(atfork_lock) ◄─ Blocked ─── │ └─► malloc() ◄─ Blocked ─┐
> >
> > Why This Matters
> > This deadlock affects any custom malloc implementation that:
> >
> > Uses pthread_atfork for fork-safety
> > Registers handlers dynamically during runtime Has prepare handlers
> > that acquire malloc locks The fix requires ensuring consistent lock
> > ordering between atfork_lock and allocator locks in both the execution path (__run_prefork_handlers) and registration path (__register_atfork).
> >
> > Proposed Approach
> > Weaken the strong definitions of __malloc_fork_lock_parent and related
> > symbols in glibc Strengthen malloc_lock management in
> > __register_atfork prepare handlers Document the requirements for
> > custom malloc implementations As you noted, we'll need to:
> >
> > Clearly mention custom malloc implications in commit messages Update
> > the "Replacing malloc" documentation Add appropriate symbol versioning
> > (GLIBC_2.44) if new interfaces are introduced Please let me know if
> > this clarifies the distinction from bug 34021.
> >
> > Best regards,
> > ShuangJin Cui
> >
>
> Please open a glibc bug report with a testcase.
>
> --
> H.J.
> -------------------------------------------------------------------------------------------------------------------------------------
> 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出的个人或群组。
> 禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。
> 如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
> This e-mail and its attachments contain confidential information from New H3C, which is intended only for the person or entity whose address is listed above.
> Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited.
> If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
--
H.J.
More information about the Libc-alpha
mailing list