malloc: Improve documentation of malloc tunables

[Adhemerval Zanella Netto]

On 10/03/26 14:03, Wilco Dijkstra wrote:
> Hi Adhemerval,
> 
> On 10/03/26 12:56, Wilco Dijkstra wrote:
>>
>> Update default for tcache_count tunable.  Remove existing documentation and
>> mention removal of fastbins in mxfast tunable.  Improve wording of hugetlb
>> tunable, including default for AArch64.
>>
>> OK for commit?
> 
>> It does not address the value returned by --list-tunables, which can be
>> misleading; nor the issues about memory consumption raised by Dimitri.
> 
> I'm testing a patch that shows it in --list-tunables, I'll post it when I get some
> results back.
> 
>> I really think we should be more conservative and *not* make this default
>> and only enable through the tunable.  With system-wide tunables we will
>> have an option to enable this as side-wide if user do want it.
> 
> We have been super conservative for way too long already. It's around 15 years
> since it was enabled by default for distros and 4 years since the hugetlb tunable
> was added to GLIBC. How many applications actually set glibc.malloc.hugetlb?
> 
> https://codesearch.debian.net/search?q=glibc.malloc.hugetlb&literal=1&perpkg=1
> 
> So the question is, how many more years should we wait before developers
> figure out how to use the tunables?
> 
> Or we can use the distro settings and use THP when enabled. People can
> disable it if it doesn't work well for them.

I think the main point of contention here is that glibc is the default system-wide
allocator, which users expect to work well across different workloads, and another
expectation is that the same allocator prioritizes performance for specific workloads.

I am focusing on the former because it is not clear that THP usage is beneficial
across all possible workloads and environments (especially given some glibc
ptmalloc shortcomings, such as fragmentation handling in some scenarios), or that
changing the defaults will always benefit users.

>From Dimitri's report, I don't think it is an uncommon high-CPU container cluster
with a low memory-to-CPU ratio. This is exactly what cloud providers are targeting
for current Aarch64 deployments with current chips.

And it worries me that, in such scenarios, sysadmins will need to be aware of
additional configuration to avoid potential pitfalls on aarch64, especially when
comparing with other ABIs that also support THP but do not enable it by default.

Checking on different memory allocators, it really depends on which kind of
workload it targets:

* TCMalloc [1] has baked in THP support and not switch to disable its usage. It
  seems widely used by Google and MongoDB [2].

* jemalloc [3] only enables it as an opt-in feature (with the MALLOC_CONF="thp..."
  option).

* mimalloc [4] also enabled THP usage and set it by default (MIMALLOC_ALLOW_THP).

The TCMalloc case is interesting because Google uses its system-wide profile
(GWP) to drive allocator development to avoid bias from microbenchmarks or specific
benchmarks. It also takes into consideration, from the start, data structures and
heuristics to avoid fragmentation across different workloads. The mimalloc in similar
wrt to design (that takes in consideration THP).

And I do not think we have that amount of research to certify that glibc THP usage
yields the same performance gain across the myriad of workloads it is used for. We
already have a fair number of corner issues [6][7][8][9][10] that might be exacerbated
by using THP by default.

Another worry is that this kind of change might create even more attrition and
misconceptions, leading people to believe that a malloc replacement would be
better than glibc’s.

So I think it would be a good way to first advertise THP support and how to properly
use the tunable, and I would say it is a failure on our part that this isn't more
widespread. This kind of information is really very project-specific, and I don’t
have a good answer on how to make it more widespread.

[1] https://github.com/google/tcmalloc
[2] https://www.mongodb.com/docs/manual/administration/tcmalloc-performance/
[3] https://jemalloc.net/
[4] https://github.com/microsoft/mimalloc
[5] https://paulcavallaro.com/blog/tcmalloc-temeraire-hugepage-aware-allocator/https://paulcavallaro.com/blog/tcmalloc-temeraire-hugepage-aware-allocator/
[6] https://sourceware.org/bugzilla/show_bug.cgi?id=31556
[7] https://sourceware.org/bugzilla/show_bug.cgi?id=30769
[8] https://sourceware.org/bugzilla/show_bug.cgi?id=15321
[9] https://sourceware.org/bugzilla/show_bug.cgi?id=26969
[10] https://sourceware.org/bugzilla/show_bug.cgi?id=21731