Documentation updates required for 2MB THP by default on Aarch64

[Dimitri John Ledkov]

Hi,

The feature to enable glibc.malloc.hugetlb by default on arm64 with a
2MB THP by default on Aarch64 is currently under documented. And I
fear it is very difficult to locate and discover, especially when it
takes out all staging environments with OOM across the board, all
clusters, all container hosts.

The release notes suggest that it is on by default and uses 2MB of
THP. However the documentation for memory allocation tunables says
that this feature is off. and ld-linux-aarch64.so.1 --list-tunables
also says that it is off. When chasing unexplained OOM caused by THP
usage, it is hard to discover this.

Can the 2.43+ documentation at
https://sourceware.org/glibc/manual/2.43/html_node/Memory-Allocation-Tunables.html#index-glibc_002emalloc_002ehugetlb
please be updated that the value is 1 on Aarch64 from glibc 2.43?

Can the output of --list-tunables on Aarch64 please be updated to say
that glibc.malloc.hugetlb is 1? Possibly dynamic, cause it should say
0 if hugepages are not available on the given kernel.

And also do this for any new architectures that also see patches to
enable THP by default on the mailing list.

As far as I understand in glibc 2.43 on aarch64 2M THP is enabled by
default and the behaviour is the same if glibc.malloc.hugetlb tunable
is not set; and when it is set to 1; and one can opt-out by explicitly
setting it to zero. As if the true default is not zero, but 'auto',
yet appears as zero in list-tunables. And it is confusing that it is
undocumented behaviour that setting tunable to zero that is already
reported as zero changes behaviour. Please correct me, if this is
wrong understanding of the behaviour on 2.43.

This is important information for people operating high-cpu container
clusters (low ratio of memory to cpu count). Because this feature
requires operators of such clusters to adjust the relevant memory and
hugetlb controllers for this new behaviour, which previously was not
needed.

Specifically, there are two options to control THP usage allocation
with cgroups, which most container deployments today do not perform on
Aarch64. By default most distributions do not set
memory_hugetlb_accounting mount option on cgroups mount. This mount
option is helpful to charge THP usage towards a given container memory
quota. (see https://docs.kernel.org/admin-guide/cgroup-v2.html)
Without this option containers can potentially now use more memory
than they were allowed to. With a large number of containers, busting
memory quotas quickly leads to OOM kills. Secondly one can configure
hugetlb controller - but it is not clear how to control quotas or
limits on per-container.

I welcome the improved performance this feature brings in 2.43.
However troubleshooting this OOM was not fun for everyone involved.

The update to documentation and to the output of list-tunables would
be greatly appreciated.

If you agree, I can work on the patch to do this; unless people more
familiar with glibc tunables and docs can do this quicker than me.

-- 
Regards,

Dimitri.