glibc 2.3 fork() on i386-gnu crashes if malloc is overridden

[Marcus Brinkmann <Marcus dot Brinkmann at ruhr-uni-bochum dot de>]

Hi,

bash uses its own malloc implementation.  Because of that, ptmalloc_init()
is never called in bash, and the malloc internal variables like main_arena
stay uninitialized.  In particular, main_arena.next is a null pointer.

Now, bash calls fork(), which runs the fork prepare handlers in the Hurd,
among them ptmalloc_lock_all.  The implementation of that crashes if
main_arena.next is a null pointer, because the for loop doesn't terminate.
>From malloc/arena.c:

static void
ptmalloc_lock_all __MALLOC_P((void))
{
  mstate ar_ptr;

  (void)mutex_lock(&list_lock);
  for(ar_ptr = &main_arena;;) {
    (void)mutex_lock(&ar_ptr->mutex);
    ar_ptr = ar_ptr->next;
    if(ar_ptr == &main_arena) break;
  }
  save_malloc_hook = __malloc_hook;
  save_free_hook = __free_hook;
  __malloc_hook = malloc_atfork;
  __free_hook = free_atfork;
  /* Only the current thread may perform malloc/free calls now. */
  tsd_getspecific(arena_key, save_arena);
  tsd_setspecific(arena_key, ATFORK_ARENA_PTR);
}

ar_ptr is NULL the second time through, causing a segmentation fault.

So, either ptmalloc_init must always be called for us, or the fork hook
code needs to be made robust not to fail if malloc wasn't initialized, or
the hook functions shouldn't be registered if ptmalloc_init wasn't called,
or another solution I can't think of.

Thanks,
Marcus


-- 
`Rhubarb is no Egyptian god.' GNU      http://www.gnu.org    marcus@gnu.org
Marcus Brinkmann              The Hurd http://www.gnu.org/software/hurd/
Marcus.Brinkmann@ruhr-uni-bochum.de
http://www.marcus-brinkmann.de/