malloc: Improve documentation of malloc tunables

[Dev Jain]

On 11/03/26 9:21 pm, Dimitri John Ledkov wrote:
> On Wed, 11 Mar 2026 at 14:07, Adhemerval Zanella Netto
> <adhemerval.zanella@linaro.org> wrote:
>>
>>
>>
>> On 10/03/26 14:03, Wilco Dijkstra wrote:
>>> Hi Adhemerval,
>>>
>>> On 10/03/26 12:56, Wilco Dijkstra wrote:
>>>>
>>>> Update default for tcache_count tunable.  Remove existing documentation and
>>>> mention removal of fastbins in mxfast tunable.  Improve wording of hugetlb
>>>> tunable, including default for AArch64.
>>>>
>>>> OK for commit?
>>>
>>>> It does not address the value returned by --list-tunables, which can be
>>>> misleading; nor the issues about memory consumption raised by Dimitri.
>>>
>>> I'm testing a patch that shows it in --list-tunables, I'll post it when I get some
>>> results back.
>>>
>>>> I really think we should be more conservative and *not* make this default
>>>> and only enable through the tunable.  With system-wide tunables we will
>>>> have an option to enable this as side-wide if user do want it.
>>>
>>> We have been super conservative for way too long already. It's around 15 years
>>> since it was enabled by default for distros and 4 years since the hugetlb tunable
>>> was added to GLIBC. How many applications actually set glibc.malloc.hugetlb?
>>>
>>> https://codesearch.debian.net/search?q=glibc.malloc.hugetlb&literal=1&perpkg=1
>>>
>>> So the question is, how many more years should we wait before developers
>>> figure out how to use the tunables?
>>>
>>> Or we can use the distro settings and use THP when enabled. People can
>>> disable it if it doesn't work well for them.
>>
>> I think the main point of contention here is that glibc is the default system-wide
>> allocator, which users expect to work well across different workloads, and another
>> expectation is that the same allocator prioritizes performance for specific workloads.
>>
>> I am focusing on the former because it is not clear that THP usage is beneficial
>> across all possible workloads and environments (especially given some glibc
>> ptmalloc shortcomings, such as fragmentation handling in some scenarios), or that
>> changing the defaults will always benefit users.
>>
>> From Dimitri's report, I don't think it is an uncommon high-CPU container cluster
>> with a low memory-to-CPU ratio. This is exactly what cloud providers are targeting
>> for current Aarch64 deployments with current chips.
> 
> 
> Most container deployments in such hardware / workload density
> configurations do not currently use cpuset cgroups. Meaning all
> containers typically see the host cpu count, rather than a cpuset
> subset. The consequence is that one has to typically manually tweak
> MALLOC_ARENA_MAX, most famously this heroku guidance
> https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior I
> don't know how container/cgroups aware glibc is.
> 
> It would be very interesting if glibc malloc allocator could somehow
> sense if it is in a container or not (even if by checking the
> container environment variable) and for example choosing to lower
> MALLOC_ARENA_MAX from the 8 times count of observable cpus to a fixed
> lower value.
> I wish containers would universally hint the weight and total amount
> of cpu shares dynamically adjusted for the number of deployed
> containers - but I don't believe this is available today.
> 
> Also, no config choice can be universal. The high-performance
> deployments typically already tune these things. But it seems like the
> number of workloads and glibc loads is no longer bare-metal heavy; but
> is VM heavy and container heavy. (a given deployment is most often
> likely to load glibc in a constrained container).
> 
> One size does not fit all; but more dynamic defaults would be nice.
> 
> I saw patches to add support for setting tunables via a config file.
> It would be nice to ship multiple example config files for bare-metal,
> vm, container => even if they just set malloc_arena_max 2. Then
> distributions will get a hint to package these, and provide them as
> config files with like update-alternatives. And possibly ship one
> edition by default in "container" builds, another in "vm" builds, and
> so on.
> 
> This approach worked really well for tuning ext4 with the e2fsprogs
> default config file sensitive to the total size of filesystem w.r.t.
> reserve space / inode reservation / etc. Such that creating ext4 on a
> small sd-card results in a different performance profile, compared to
> a multi-terabyte filesystem.
> 
> I think the desire to change the default mostly stems from lack of
> good levers for applying the default tuning; or lack of automatic
> dynamic  tuning based on a given environment.

FYI you may take a look at [1], at the heading "process THP controls". In
short, you can make your container workload override the sysctl and set THP
= never for it. Not sure if this suits your case, or how widely this is
used. But yeah we have to improve the malloc THP support in any case.

[1]
https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/mm/transhuge.rst

> 
>>
>> And it worries me that, in such scenarios, sysadmins will need to be aware of
>> additional configuration to avoid potential pitfalls on aarch64, especially when
>> comparing with other ABIs that also support THP but do not enable it by default.
>>
>> Checking on different memory allocators, it really depends on which kind of
>> workload it targets:
>>
>> * TCMalloc [1] has baked in THP support and not switch to disable its usage. It
>>   seems widely used by Google and MongoDB [2].
>>
>> * jemalloc [3] only enables it as an opt-in feature (with the MALLOC_CONF="thp..."
>>   option).
>>
>> * mimalloc [4] also enabled THP usage and set it by default (MIMALLOC_ALLOW_THP).
>>
>> The TCMalloc case is interesting because Google uses its system-wide profile
>> (GWP) to drive allocator development to avoid bias from microbenchmarks or specific
>> benchmarks. It also takes into consideration, from the start, data structures and
>> heuristics to avoid fragmentation across different workloads. The mimalloc in similar
>> wrt to design (that takes in consideration THP).
>>
>> And I do not think we have that amount of research to certify that glibc THP usage
>> yields the same performance gain across the myriad of workloads it is used for. We
>> already have a fair number of corner issues [6][7][8][9][10] that might be exacerbated
>> by using THP by default.
>>
>> Another worry is that this kind of change might create even more attrition and
>> misconceptions, leading people to believe that a malloc replacement would be
>> better than glibc’s.
>>
>> So I think it would be a good way to first advertise THP support and how to properly
>> use the tunable, and I would say it is a failure on our part that this isn't more
>> widespread. This kind of information is really very project-specific, and I don’t
>> have a good answer on how to make it more widespread.
>>
>> [1] https://github.com/google/tcmalloc
>> [2] https://www.mongodb.com/docs/manual/administration/tcmalloc-performance/
>> [3] https://jemalloc.net/
>> [4] https://github.com/microsoft/mimalloc
>> [5] https://paulcavallaro.com/blog/tcmalloc-temeraire-hugepage-aware-allocator/https://paulcavallaro.com/blog/tcmalloc-temeraire-hugepage-aware-allocator/
>> [6] https://sourceware.org/bugzilla/show_bug.cgi?id=31556
>> [7] https://sourceware.org/bugzilla/show_bug.cgi?id=30769
>> [8] https://sourceware.org/bugzilla/show_bug.cgi?id=15321
>> [9] https://sourceware.org/bugzilla/show_bug.cgi?id=26969
>> [10] https://sourceware.org/bugzilla/show_bug.cgi?id=21731
>>
>>
> 
>