RFC: malloc automatic THP opt-in breaks madvise contract for container hosts

[H.J. Lu]

On Fri, Apr 17, 2026 at 12:33 PM Sun, Yutong <yutongsu@amazon.com> wrote:
>
> Hi,
>
>
>
> I'd like to raise a concern about the automatic Transparent Huge Pages
>
> (THP) opt-in behavior introduced in malloc, specifically the commits:
>
>
>
>   321e1fc7 - malloc: Enable 2MB THP by default on Aarch64
>
>   1c588a21 - malloc: Improve thp_init
>
>   bd569425 - malloc: Fix clang build after 1c588a2187
>
>
>
> We maintain Bottlerocket, a Linux distribution purpose-built for running
>
> containers. We've had to carry a revert patch for these changes [1] and
>
> would like to discuss a proper upstream solution.
>
>
>
> ## The Problem
>
>
>
> When the kernel is configured with THP in `madvise` mode, glibc now
>
> automatically calls madvise(MADV_HUGEPAGE) on heap allocations for every
>
> glibc-linked process. This changes the semantics of `madvise` mode: the
>
> whole point of `madvise` (vs `always`) is that applications explicitly
>
> opt in. glibc opting in on behalf of every process defeats this.
>
>
>
> On a container host, the impact is particularly counterproductive. These
>
> are measured values from a Bottlerocket node running Kubernetes pods on
>
> aarch64:
>
>
>
>   - kubelet:              ~24 MB AnonHugePages
>
>   - containerd:           ~22 MB AnonHugePages
>
>   - containerd-shim (per pod): ~22-28 MB AnonHugePages
>
>
>
> The container shims — small management processes with minimal heap
>
> working sets — receive huge page allocations, while the actual container
>
> workloads that might benefit from THP do not. On a node running, say,
>
> 200 pods, shim processes alone would account for 4-5 GB of huge page
>
> allocations. After applying our revert patch, all of these drop to 0 kB.
>
>
>
> ## Alternatives Considered
>
>
>
>   - GLIBC_TUNABLES (glibc.malloc.hugetlb=0): Tunables are explicitly
>
>     not part of the stable ABI [2] and the mechanism has had security
>
>     vulnerabilities (CVE-2023-4911 [3]). As a distribution, we cannot
>
>     depend on unstable interfaces for correctness.
>
>
>
>   - Disabling THP system-wide: Too broad — penalizes workloads that
>
>     genuinely benefit from THP.
>
>
>
>   - Carrying a revert patch: This is what we're doing today [1]. It
>
>     works but is not sustainable long-term.
>
>
>
> ## What We'd Like to Discuss
>
>
>
> We recognize the THP optimization has value for workloads with large
>
> heaps. The concern is that the current default removes choice from
>
> distributions and operators who configure `madvise` mode precisely to
>
> retain that control.
>
>
>
> We're open to whatever approach the community prefers and happy to
>
> contribute. We expect other distributions will encounter this as
>
> glibc 2.43 rolls out more broadly.
>
>
>
> [1] https://github.com/bottlerocket-os/bottlerocket-core-kit/pull/905
>
> [2] https://sourceware.org/glibc/manual/2.43/html_mono/libc.html#Tunables
>
> [3] https://lwn.net/Articles/947736/
>
>
>
> Thanks,
>
> Yutong

Will my GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE proposal:

https://inbox.sourceware.org/gnu-gabi/CAMe9rOqRQLJ_03gfy1PvJ2-fPuMMGXBNCx2_paC+ZM_n_-5jsQ@mail.gmail.com/T/#mf03cb1820f19c60967b3ffac7f90c0677c915def

help you?  My proposal is to opt-in THP for read-only segments.   We can
add another bit to opt-in THP for other memories, like read-write segments
and heap.


--
H.J.