when malloc runs out...

[DJ Delorie <dj at redhat dot com>]

I've recently been working on some problem cases wrt when malloc runs
out of resources, and it's caused me to investigate all the failure
modes therein.

It seems to me that we have very few cases where a thread will switch
arenas once it's assigned one.

Specifically, a thread changes arenas in only a few cases:

* It has no arena, so chooses one

* An arena is corrupted (and malloc_printerr() is called) and we
  switch to a "free" arena.

* mmap() fails while extending a non-main arena, and the thread
  switches to the main heap.

* sbrk() fails, and we switch to a non-main arena.

The problem is, the third option is only temporary - the tests in
arena_get_retry() are not symmetrical!  The switch from sbrk to mmap
heaps goes through arena_get2(), which sets thread_arena to the new
choice, but the switch from mmap to sbrk heaps does *not* set
thread_arena.

This means, if a thread's arena's mmap() fails, the thread will try
the main heap, but EVERY time that thread tries to malloc(), it will
repeat this map-fail-sbrk sequence, which kills performance due to the
excess system calls.

The reason for this could be intentional - one supposes that the
system will eventually be able to call mmap() again, or it may be
unintentional - a side effect of adding arena support to dlmalloc a
long time ago.

I'm considering what the "right" thing to do in this situation is.  I
think we should at least remember the decision to use the main arena.
This means that all threads may eventually end up there, until sbrk()
fails and they start moving back to other arenas.  Eventually we'll
encounter two failures in a row and malloc() will return NULL, even if
other arenas might have free chunks in them.

Long term, we might need some smarter way to scan through all the
arenas, forcing a coalescing (i.e. purge the unsorted and fast bins)
and checking to see if there's available chunks, when we know that the
system is low on memory or mmaps.  Or some way to periodically or
deterministically re-assigning threads to arenas to balance them.

Thoughts?  Ideas?