[PATCH v2] malloc: Enable 2MB THP by default on Aarch64

[Wilco Dijkstra]

Hi Adhemerval,

>> So without making the huge page support significantly smarter (and correct),
>> I don't believe such huge page sizes work at all for GLIBC.
>
> I agree that this 512mb is impractical for most if not all workloads. I was just
> curious if we have tried to actually use it to see if there is any conformance
> issue or performance differences with spec.

I know a colleague tried SPEC with 512MB huge pages, and he was complaining
that malloc would use up huge pages for alignment padding or internal heap
and fail without errors to use huge pages in the cases he wanted it. You need
to supply far more huge pages than you think...

> I was thinking is use the reported values of /sys/kernel/mm/hugepages/ instead
> of /sys/kernel/mm/transparent_hugepage/ because it was how libhugetlbfs used to
> work while reserving huge pages.  The mTHP does allow a lot of more page size
> and I agree that this unlikely to be useful for most cases.
>
> A possible /sys/kernel/mm/hugepages/ side selection does make sense for for
> glibc.malloc.hugetlb=2 though.

I think you just want the smallest huge page rather than going with something
really huge.

Like with caches, you get almost all of the gains from the first few doublings - 
nobody uses a 1MB L1 cache!

> And it seems that there are some users trying hugetlb=2, specially for the case
> where the workload used to rely on libhugetlbfs (and they are not supported anymore
> since we remove the malloc hooks).  I did fixed some issues [1], after a coworker
> reported some issues with an specific workload.

Yes that particular issue affects THP as well. There are some interesting alignment
issues around the very first sbrk() call, where we won't get an aligned huge page until
we allocate more. I'm wondering whether we could use mmap instead (placed near
the sbrk region) or just sbrk() till we are aligned and never use the initial unaligned
part (not sure it is legal to unmap it - Dev?).

> What I am not sure if we will start to get reports that this change increase resource
> usage and will need to reevaluate the default strategy.  I noted that you sent a
> change to override DEFAULT_THP_PAGESIZE with glibc.malloc.hugetlb=0 [2], which is on
> my backlog and I would like to have it for 2.43.

It can increase memory usage of course, but that can be improved by moving towards
fewer and larger heaps. Linux does not have to use huge pages if it is low on memory.
But having an override like you suggested seemed like a good idea.

> I still think we will need to evaluate it to make it the default for other
> ISA that also support hugepages. But I think we can start with aarch64 and
> see how it goes on next release.

Agreed. It's also the only way to find out if there are other issues with GLIBCs THP
implementation - using it in 1 or 2 applications is not the same as it being the
default for all.

Cheers,
Wilco