malloc: Improve documentation of malloc tunables

[Wilco Dijkstra]

Hi Dimitri,

> It would be very interesting if glibc malloc allocator could somehow
> sense if it is in a container or not (even if by checking the
> container environment variable) and for example choosing to lower
> MALLOC_ARENA_MAX from the 8 times count of observable cpus to a fixed
> lower value.
> I wish containers would universally hint the weight and total amount
> of cpu shares dynamically adjusted for the number of deployed
> containers - but I don't believe this is available today.

So is the issue you are seeing due to a bad default of MALLOC_ARENA_MAX?
Lots of arenas implies malloc isn't trying to use arenas efficiently. I think that
can occur if you have extreme numbers of threads - the current malloc design
tries to give each thread its own arena (which seems like a bad idea).

Setting it to 2 might well be better as a default. However it's not obvious why
we ever need more than say 64 arenas even on systems with lots of cores.

> I think the desire to change the default mostly stems from lack of
> good levers for applying the default tuning; or lack of automatic
> dynamic  tuning based on a given environment.

The goal is to provide good results out of the box. I don't believe in forcing
people to fine tune every feature in their system in order to get it to work
properly.

Cheers,
Wilco