Replacing malloc in large Java application leads to glibc assert attached_threads > 0

[Florian Weimer]

* Steven Schlansker via Libc-help:

> I managed to get a coredump with a backtrace:
>
> #0  __pthread_kill_implementation (threadid=<optimized out>,
> signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> #1  0x00007b0baa299ff3 in __pthread_kill_internal (threadid=<optimized
> out>, signo=6) at pthread_kill.c:78
> #2  0x00007b0baa243f56 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/posix/raise.c:26
> #3  0x00007b0baa22b8fa in __GI_abort () at abort.c:79
> #4  0x00007b0baa22cb69 in __libc_message_impl
> (fmt=fmt@entry=0x7b0baa398fe0 "Fatal glibc error: %s:%s (%s):
> assertion failed: %s\n") at ../sysdeps/posix/libc_fatal.c:134
> #5  0x00007b0baa23c2e5 in __libc_assert_fail
> (assertion=assertion@entry=0x7b0baa395b93 "a->attached_threads > 0",
> file=file@entry=0x7b0baa3959be "arena.c", line=line@entry=907,
>     function=function@entry=0x7b0baa39dcf0 <__PRETTY_FUNCTION__.0>
> "__malloc_arena_thread_freeres") at __libc_assert_fail.c:31
> #6  0x00007b0baa2a8cc6 in __malloc_arena_thread_freeres () at
> /usr/src/debug/glibc-2.39-58.el10_1.7.x86_64/malloc/arena.c:907
> #7  0x00007b0baa2ab015 in __libc_thread_freeres () at thread-freeres.c:41
> #8  0x00007b0baa298000 in start_thread (arg=<optimized out>) at
> pthread_create.c:459
> #9  0x00007b0baa308afc in clone3 () at
> ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
>
> The assert seems to possibly be triggered by process exit - the
> burn-in test completes successfully, but then the test runner fails
> due to the child process crashing.

If there isn't a (not yet reported) glibc bug in this area, it's either
imperfect replacement of glibc's malloc (the replacement malloc does not
cover all malloc routines the application uses, or the way dlopen/dlsym
are used exposes glibc malloc routines instead of their replacements),
or memory corruption: something might have overwritten the thread_arena
variable.  It may make sense to dump *thread_arena using GDB, to check
if the values look reasonable.

> I was not able to determine what type of thread this is that crashed -
> whether Java thread, rocksdb native background thread, jemalloc
> internal.

You should be able to view the pthread_create callback arguments using:

  print ((struct pthread *) $fs_base)->start_routine

from GDB.  This may provide a hint regarding the thread type.

> The Java instrumentation does not track this thread so I suspect
> native thread, but that could be because it is in the process of being
> torn down, and already exited "Java-land".
>
> I read through the "Replacing malloc" section of the glibc manual [2],
> and found this concerning note:
>
>> Care must be taken not to use functionality from the GNU C Library
>> that uses malloc internally.  For example, the fopen, opendir,
>> dlopen, and pthread_setspecific functions currently use the malloc
>> subsystem internally
>
> For an application as large as the JVM with native libraries linked
> in, none of which I authored, it seems impossible that I could promise
> this.

This only applies to the replacement malloc subsystem, not the
application outside of malloc.  I thought the rest of the paragraph
makes this pretty clear, but perhaps we need to reword this paragraph.

Thanks,
Florian