malloc: Improve documentation of malloc tunables

[Wilco Dijkstra]

Hi Dimitri,

>> So is the issue you are seeing due to a bad default of MALLOC_ARENA_MAX?
>
> I am relaying different sets of feedback, from very different
> deployments with very different setups / workload / hardware / etc;
> from different customers / end-users.
>
> The MALLOC_ARENA_MAX=2 helped migrating of glibc 2.35 to 2.43 on x86_64

Interesting since x86_64 does not use THP by default.

>> Lots of arenas implies malloc isn't trying to use arenas efficiently. I think that
>> can occur if you have extreme numbers of threads - the current malloc design
>> tries to give each thread its own arena (which seems like a bad idea).
>
> Yes, in this deployment there is JVM with many threads; inside
> containers; and without MALLOC_ARENA_MAX=2 the performance degraded by
> 30% (total throughput, but also number of total and concurrent
> requests being able to service). And the characteristics were similar
> - very dense container deployment, low memory to cpu ratio, and yes
> each container has a single process with many threads.

There was a talk at the Cauldron about JVM/MySQL malloc issues:
https://conf.gnu-tools-cauldron.org/opo25/talk/LXPUYR/

Since GLIBC 2.43 you can increase the maximum size of allocations handled
by tcache via glibc.malloc.tcache_max=N. This allows one to run with only 1
arena without a performance hit (and no fragmentation after days of use).

> I offered to set MALLOC_ARENA_MAX=2 by default universally for that
> deployment and it had mixed perception - as if it might impact a
> single large container allocated to run by-itself on a node;
> effectively privileged / bare-metal container full speed ahead
> workload.

Yes that would be a bit low...

> It does feel like uncapped linear 8 times CPU cores is excessive after
> a high enough number of cores. Was 8 times CPU cores been set back in
> the day before 192 CPU cores public cloud instances generally
> available; which with hyperthreading may appear as 384 cores; and
> times by 8 would be 3072 arenas => which feels like very excessive
> amount; especially when in practice this host actually runs 500 pods.

The bad commit is from 2009!

 https://sourceware.org/git/?p=glibc.git;a=commit;h=425ce2edb9d

Obviously zero explanation or benchmarks as to why this value was chosen...
So it looks incorrect from the first commit. I think 2 (or even 1) is a better
value, especially since we removed the slow consolidation pass and added
tcache for small blocks. It's just a matter of running some large applications
to confirm there are no major slowdowns.

> I wonder if we can keep 8 times CPU count; but cap it at 32 arenas
> maybe? Unfortunately I don't have benchmark data that I could easily
> trigger to simulate and measure the impact of changing this.

We can lower the 8 times (for laptops) as well as the maximum (for big servers).

Cheers,
Wilco