[PATCH v2] malloc: Enable 2MB THP by default on Aarch64

[Dev Jain]

On 16/12/25 8:01 pm, Adhemerval Zanella Netto wrote:
>
> On 11/12/25 20:28, Wilco Dijkstra wrote:
>> Hi Adhemerval,
>>
>>> The only thing bothering me is now there is no option users can *disable*
>>> it for whatever reason. We should at least update the manual/tunables.texi
>>> documentation that glibc.malloc.hugetlb=1 is the default for AArch64, and
>>> that it uses a different Huge Page than the one defined by the kernel
>>>  (/sys/kernel/mm/transparent_hugepage/hpage_pmd_size).
>>>
>>> Maybe it would be better to work toward adding an aarch64-specific tunable
>>> knob to set glibc.malloc.hugetlb=1 as the default. At least it aligns better
>>> with other architectures and gives aarch64 an option to disable this if
>>> users want the previous semantics.
>> The idea was to avoid using the glibc.malloc.hugetlb=1 code path by default since
>> that does several expensive system calls at startup even if you never use malloc.
>> We could easily use glibc.malloc.hugetlb=0 as the disable option (eg. set thp_mode
>> to never in that case and check that in thp_init() instead).
> I don't think making the 2MB as the only option for AArch64 a problem, although
> afaik it does only work as expected on Linux 6.9+ with Contiguous PTE enabled.

For 4K pages, we will benefit from 2MB even without contiguous pte. A 2M hugepage
will be mapped as a block mapping in the second last level of the pagetable. The
contigous pte feature opens up the possibility of mapping 64K granularity on 4K
basepage, and 2M granularity on 64K basepage, with a single TLB entry.

>
> It also has the side effect of disabling  glibc.malloc.hugetlb=1 support on
> kernel older than 6.9 with 64kb page size.  I am not sure if using 512M is
> really 'unusable' and 'impractical' as Dev has stated in the patch; specially
> because the tunable is optional. But also I am not sure how wildly this configuration
> is used, nor the performance implications. Do we have SPECcpu2017 numbers of
> the performance difference between using 2MB and 512MB mTHP?

I am of the opinion that if an app wants 512M mappings, there are two much better things
to do:

1) Don't use malloc. I mean otherwise, malloc is handling small chunks in the order of bytes
and kilobytes, and also handling chunks in order of gigabytes. Also, an sbrk(512M) may fail.
Surely the app would be better off using mmap() directly.

2) Use static hugepages (hugetlbfs). One can reserve hugetlbpages at boot time. There is no
such method for 512M THP. It is difficult to get 512M at fault time, because the memory
is now fragmented. Also such an allocation worsens fragmentation. hugetlbpages get allocated
through the CMA allocator in the kernel, which is a dedicated area from which to pick
large blocks - they won't fragment rest of the physical memory.

>
>> In terms of huge page sizes, we have to use reasonable limits rather than just using
>> the system or user sizes. The current malloc has no checks whatsoever, so it would
>> fail quickly due to wasting most memory or trying to create arenas that are too large.
>> Does it sound reasonable to round up a 1.0001GB allocation to 2GB just to use 1GB
>> huge pages?
> I though initially to add a way to specify the expected huge page size to use
> along with the tunable, something like glibc.malloc.hugetlb=1:2097152.  If
> the value is set it would bypass the hpage_pmd_size parsing.  I am not sure
> how useful it would be; although it would give users a way to check different
> page size with Contiguous PTE.

This sounds reasonable to me on the surface. The edge case I can think of
is using 16K or 64K basepages, on a non-contiguous pte kernel. We will allocate
2M blocks without the TLB compression benefit. I think Wilco's patch disabling
THP-default should be enough for this.

One of my own motivations of doing this patch was to provide *less* tunables. We have shot our foot in the
kernel before when we provide a lot of configurability and then the user does not
know the right config, or the other extreme case is that the user figures out
some magic configuration which best suits their workloads, and then removing/massaging
that config later becomes difficult because such user will complain that a
particular config works best for them.

>
> Another change that I am not fully confortable is that aarch64 now behaves slight 
> different than other ABIs and that this is a very server oriented tunable. It is
> not clear when users starts to use glibc on aarm64 desktop environments if this
> will incurs in higher memory usage compared to other ABIs.