RFC: malloc automatic THP opt-in breaks madvise contract for container hosts

[H.J. Lu]

On Fri, Apr 17, 2026 at 12:48 PM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> On Fri, Apr 17, 2026 at 12:33 PM Sun, Yutong <yutongsu@amazon.com> wrote:
> >
> > Hi,
> >
> >
> >
> > I'd like to raise a concern about the automatic Transparent Huge Pages
> >
> > (THP) opt-in behavior introduced in malloc, specifically the commits:
> >
> >
> >
> >   321e1fc7 - malloc: Enable 2MB THP by default on Aarch64
> >
> >   1c588a21 - malloc: Improve thp_init
> >
> >   bd569425 - malloc: Fix clang build after 1c588a2187
> >
> >
> >
> > We maintain Bottlerocket, a Linux distribution purpose-built for running
> >
> > containers. We've had to carry a revert patch for these changes [1] and
> >
> > would like to discuss a proper upstream solution.
> >
> >
> >
> > ## The Problem
> >
> >
> >
> > When the kernel is configured with THP in `madvise` mode, glibc now
> >
> > automatically calls madvise(MADV_HUGEPAGE) on heap allocations for every
> >
> > glibc-linked process. This changes the semantics of `madvise` mode: the
> >
> > whole point of `madvise` (vs `always`) is that applications explicitly
> >
> > opt in. glibc opting in on behalf of every process defeats this.
> >
> >
> >
> > On a container host, the impact is particularly counterproductive. These
> >
> > are measured values from a Bottlerocket node running Kubernetes pods on
> >
> > aarch64:
> >
> >
> >
> >   - kubelet:              ~24 MB AnonHugePages
> >
> >   - containerd:           ~22 MB AnonHugePages
> >
> >   - containerd-shim (per pod): ~22-28 MB AnonHugePages
> >
> >
> >
> > The container shims — small management processes with minimal heap
> >
> > working sets — receive huge page allocations, while the actual container
> >
> > workloads that might benefit from THP do not. On a node running, say,
> >
> > 200 pods, shim processes alone would account for 4-5 GB of huge page
> >
> > allocations. After applying our revert patch, all of these drop to 0 kB.
> >
> >
> >
> > ## Alternatives Considered
> >
> >
> >
> >   - GLIBC_TUNABLES (glibc.malloc.hugetlb=0): Tunables are explicitly
> >
> >     not part of the stable ABI [2] and the mechanism has had security
> >
> >     vulnerabilities (CVE-2023-4911 [3]). As a distribution, we cannot
> >
> >     depend on unstable interfaces for correctness.
> >
> >
> >
> >   - Disabling THP system-wide: Too broad — penalizes workloads that
> >
> >     genuinely benefit from THP.
> >
> >
> >
> >   - Carrying a revert patch: This is what we're doing today [1]. It
> >
> >     works but is not sustainable long-term.
> >
> >
> >
> > ## What We'd Like to Discuss
> >
> >
> >
> > We recognize the THP optimization has value for workloads with large
> >
> > heaps. The concern is that the current default removes choice from
> >
> > distributions and operators who configure `madvise` mode precisely to
> >
> > retain that control.
> >
> >
> >
> > We're open to whatever approach the community prefers and happy to
> >
> > contribute. We expect other distributions will encounter this as
> >
> > glibc 2.43 rolls out more broadly.
> >
> >
> >
> > [1] https://github.com/bottlerocket-os/bottlerocket-core-kit/pull/905
> >
> > [2] https://sourceware.org/glibc/manual/2.43/html_mono/libc.html#Tunables
> >
> > [3] https://lwn.net/Articles/947736/
> >
> >
> >
> > Thanks,
> >
> > Yutong
>
> Will my GNU_PROPERTY_1_NEEDED_TRANSPARENT_HUGEPAGE proposal:
>
> https://inbox.sourceware.org/gnu-gabi/CAMe9rOqRQLJ_03gfy1PvJ2-fPuMMGXBNCx2_paC+ZM_n_-5jsQ@mail.gmail.com/T/#mf03cb1820f19c60967b3ffac7f90c0677c915def
>
> help you?  My proposal is to opt-in THP for read-only segments.   We can
> add another bit to opt-in THP for other memories, like read-write segments
> and heap.
>
>

I opened:

https://sourceware.org/bugzilla/show_bug.cgi?id=34081

-- 
H.J.