Replacing malloc in large Java application leads to glibc assert attached_threads > 0

[Steven Schlansker]

Hi libc-help, we are experiencing process crashes due to an internal
assertion in glibc.

We run a large Java 26 application on Rocky Linux 10 (thousands of
threads, 16GB heap) including Apache Kafka, which links with rocksdb
(C++ library).
The Kafka documentation suggests that the built-in glibc (2.39)
allocator is appropriate for general desktop use,
but for more demanding use, you might achieve better peak performance
and less rss used by replacing it with e.g. jemalloc or tcmalloc [1].

In order to enjoy this benefit, we run our process with e.g.
LD_PRELOAD=/usr/lib/libjemalloc.so.2
Unfortunately, when we do this, we experience a crash in our burn-in
environment:

Fatal glibc error: arena.c:907 (__malloc_arena_thread_freeres):
assertion failed: a->attached_threads > 0

I managed to get a coredump with a backtrace:

#0  __pthread_kill_implementation (threadid=<optimized out>,
signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007b0baa299ff3 in __pthread_kill_internal (threadid=<optimized
out>, signo=6) at pthread_kill.c:78
#2  0x00007b0baa243f56 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#3  0x00007b0baa22b8fa in __GI_abort () at abort.c:79
#4  0x00007b0baa22cb69 in __libc_message_impl
(fmt=fmt@entry=0x7b0baa398fe0 "Fatal glibc error: %s:%s (%s):
assertion failed: %s\n") at ../sysdeps/posix/libc_fatal.c:134
#5  0x00007b0baa23c2e5 in __libc_assert_fail
(assertion=assertion@entry=0x7b0baa395b93 "a->attached_threads > 0",
file=file@entry=0x7b0baa3959be "arena.c", line=line@entry=907,
    function=function@entry=0x7b0baa39dcf0 <__PRETTY_FUNCTION__.0>
"__malloc_arena_thread_freeres") at __libc_assert_fail.c:31
#6  0x00007b0baa2a8cc6 in __malloc_arena_thread_freeres () at
/usr/src/debug/glibc-2.39-58.el10_1.7.x86_64/malloc/arena.c:907
#7  0x00007b0baa2ab015 in __libc_thread_freeres () at thread-freeres.c:41
#8  0x00007b0baa298000 in start_thread (arg=<optimized out>) at
pthread_create.c:459
#9  0x00007b0baa308afc in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

The assert seems to possibly be triggered by process exit - the
burn-in test completes successfully, but then the test runner fails
due to the child process crashing.
I was not able to determine what type of thread this is that crashed -
whether Java thread, rocksdb native background thread, jemalloc
internal.
The Java instrumentation does not track this thread so I suspect
native thread, but that could be because it is in the process of being
torn down, and already exited "Java-land".

I read through the "Replacing malloc" section of the glibc manual [2],
and found this concerning note:

> Care must be taken not to use functionality from the GNU C Library that uses malloc internally.
> For example, the fopen, opendir, dlopen, and pthread_setspecific functions currently use the malloc subsystem internally

For an application as large as the JVM with native libraries linked
in, none of which I authored, it seems impossible that I could promise
this.

This documentation page makes it seem impossible to me to safely
replace malloc in a program without keeping extremely careful control
over your dependencies,
since most authors involved in open source projects are not
necessarily avoiding these forbidden functions, and the documentation
explicitly leaves room for them to change over time...

I read through the rocksdb source code, and indeed it seems to
implement a `util/thread_local.cc` type using pthread_setspecific.

Does this mean that the Kafka project guidance to use jemalloc via
LD_PRELOAD is fundamentally unsafe, and should not be recommended?
I see that rocksdb uses pthread_setspecific, and I have no way to
guarantee that the JVM does not use e.g. opendir.

Or maybe there's a bug to fix around this assertion that could improve
the situation? It feels like it almost works - but this crash is quite
concerning.

Thank you for any guidance you might provide!
Steven

1: https://kafka.apache.org/41/streams/developer-guide/memory-mgmt/#rocksdb
2: https://sourceware.org/glibc/manual/2.43/html_node/Replacing-malloc.html