[PATCH v2] malloc: Enable 2MB THP by default on Aarch64

[Adhemerval Zanella Netto]

On 17/12/25 01:15, Dev Jain wrote:
> 
> On 16/12/25 8:01 pm, Adhemerval Zanella Netto wrote:
>>
>> On 11/12/25 20:28, Wilco Dijkstra wrote:
>>> Hi Adhemerval,
>>>
>>>> The only thing bothering me is now there is no option users can *disable*
>>>> it for whatever reason. We should at least update the manual/tunables.texi
>>>> documentation that glibc.malloc.hugetlb=1 is the default for AArch64, and
>>>> that it uses a different Huge Page than the one defined by the kernel
>>>>  (/sys/kernel/mm/transparent_hugepage/hpage_pmd_size).
>>>>
>>>> Maybe it would be better to work toward adding an aarch64-specific tunable
>>>> knob to set glibc.malloc.hugetlb=1 as the default. At least it aligns better
>>>> with other architectures and gives aarch64 an option to disable this if
>>>> users want the previous semantics.
>>> The idea was to avoid using the glibc.malloc.hugetlb=1 code path by default since
>>> that does several expensive system calls at startup even if you never use malloc.
>>> We could easily use glibc.malloc.hugetlb=0 as the disable option (eg. set thp_mode
>>> to never in that case and check that in thp_init() instead).
>> I don't think making the 2MB as the only option for AArch64 a problem, although
>> afaik it does only work as expected on Linux 6.9+ with Contiguous PTE enabled.
> 
> For 4K pages, we will benefit from 2MB even without contiguous pte. A 2M hugepage
> will be mapped as a block mapping in the second last level of the pagetable. The
> contigous pte feature opens up the possibility of mapping 64K granularity on 4K
> basepage, and 2M granularity on 64K basepage, with a single TLB entry.

I meant that this change only is really beneficial on 64k page size systems
with Linux 6.9+. But from previous discussion it is also debatable if
such system would benefit from glibc.malloc.hugetlb=1 anyway.

> 
>>
>> It also has the side effect of disabling  glibc.malloc.hugetlb=1 support on
>> kernel older than 6.9 with 64kb page size.  I am not sure if using 512M is
>> really 'unusable' and 'impractical' as Dev has stated in the patch; specially
>> because the tunable is optional. But also I am not sure how wildly this configuration
>> is used, nor the performance implications. Do we have SPECcpu2017 numbers of
>> the performance difference between using 2MB and 512MB mTHP?
> 
> I am of the opinion that if an app wants 512M mappings, there are two much better things
> to do:
> 
> 1) Don't use malloc. I mean otherwise, malloc is handling small chunks in the order of bytes
> and kilobytes, and also handling chunks in order of gigabytes. Also, an sbrk(512M) may fail.
> Surely the app would be better off using mmap() directly.

I think the whole idea of a malloc tunable is exactly to not require code
changes.

> 
> 2) Use static hugepages (hugetlbfs). One can reserve hugetlbpages at boot time. There is no
> such method for 512M THP. It is difficult to get 512M at fault time, because the memory
> is now fragmented. Also such an allocation worsens fragmentation. hugetlbpages get allocated
> through the CMA allocator in the kernel, which is a dedicated area from which to pick
> large blocks - they won't fragment rest of the physical memory.

I don't think there is a way to allocate 512MB on such systems.  On Linux
6.18 on N1 I see:

$ ls -d1 /sys/kernel/mm/hugepages/hugepages-*
/sys/kernel/mm/hugepages/hugepages-1048576kB
/sys/kernel/mm/hugepages/hugepages-2048kB
/sys/kernel/mm/hugepages/hugepages-32768kB
/sys/kernel/mm/hugepages/hugepages-64kB

The hugetlbfs is way more restrict of what kind of pages you are allowed
to reserve. 

In any case, I don't think is an issue. 

> 
>>
>>> In terms of huge page sizes, we have to use reasonable limits rather than just using
>>> the system or user sizes. The current malloc has no checks whatsoever, so it would
>>> fail quickly due to wasting most memory or trying to create arenas that are too large.
>>> Does it sound reasonable to round up a 1.0001GB allocation to 2GB just to use 1GB
>>> huge pages?
>> I though initially to add a way to specify the expected huge page size to use
>> along with the tunable, something like glibc.malloc.hugetlb=1:2097152.  If
>> the value is set it would bypass the hpage_pmd_size parsing.  I am not sure
>> how useful it would be; although it would give users a way to check different
>> page size with Contiguous PTE.
> 
> This sounds reasonable to me on the surface. The edge case I can think of
> is using 16K or 64K basepages, on a non-contiguous pte kernel. We will allocate
> 2M blocks without the TLB compression benefit. I think Wilco's patch disabling
> THP-default should be enough for this.
> 
> One of my own motivations of doing this patch was to provide *less* tunables. We have shot our foot in the
> kernel before when we provide a lot of configurability and then the user does not
> know the right config, or the other extreme case is that the user figures out
> some magic configuration which best suits their workloads, and then removing/massaging
> that config later becomes difficult because such user will complain that a
> particular config works best for them.

I completely agree in this regard.  The original motivation of my hugetlb support
on glibc is to give a way users relying on libhugetlbfs to continue backing malloc 
calls with HUGETLB mmap calls (instead or rolling out its own malloc or relying
on a different malloc implementation). 

I implemented the transparent part a bonus, but I haven't dig into much details
 on how to improve its usage (thanks for continuing the work btw!).

> 
>>
>> Another change that I am not fully confortable is that aarch64 now behaves slight 
>> different than other ABIs and that this is a very server oriented tunable. It is
>> not clear when users starts to use glibc on aarm64 desktop environments if this
>> will incurs in higher memory usage compared to other ABIs.