[PATCH v6 5/6] elf: Align large load segments to PMD huge page size for THP

[WANG Rui]

Hi Adhemerval,

On Tue, Mar 10, 2026 at 11:06 PM Adhemerval Zanella Netto
<adhemerval.zanella@linaro.org> wrote:
>
> On 09/03/26 22:01, WANG Rui wrote:
> > Hi,
> >
> > On Tue, Mar 10, 2026 at 2:25 AM Adhemerval Zanella Netto
> > <adhemerval.zanella@linaro.org> wrote:
> >>
> >> On 09/03/26 14:36, Wilco Dijkstra wrote:
> >>> Hi Adhemerval,
> >>>
> >>>> There is also the potential memory usage increase of using this tunable as
> >>>> default, as we saw a report for the malloc one on AArch64.  I think it would
> >>>> less than the malloc, but on multi-layer environment with a lot of process
> >>>> this could compound to a significant value.
> >>>>
> >>>> I think we should *not* make this the default for now.
> >>>
> >>> Would it round up a 33MB load segment to 64MB? If so, that could increase
> >>> memory usage if you had lots of processes of that size. However the OS could
> >>> trivially drop any readonly hugepage at the first hint of memory pressure and
> >>> revert to base pages. So the fallback mechanism should be pretty robust.
> >>
> >> Even if _dl_map_segment_align only considers !PROT_WRITE and aligned to the
> >> thp_size, _dl_map_object_from_fd will align all PT_LOAD segments:
> >>
> >> 1228     /* Align all PT_LOAD segments to the maximum p_align.  */
> >> 1229     for (size_t i = 0; i < nloadcmds; i++)
> >> 1230       loadcmds[i].mapalign = p_align_max;
> >>
> >> We can refactor it to just align some specific segments.
> >>
> >> But even with change, I would not do a system-wide change and rely on kernel
> >> OOM mechanism to proper handle memory pressure. We already had some complains
> >> for the AArch64 malloc one (which I think we should revert) so I think we
> >> should be conservative here and make this a opt-in feature.
> >
> > Regarding virtual address space: increasing the alignment can indeed
> > introduce larger gaps between shared library mappings, which wastes
> > some virtual address space. On 64-bit this is basically a non-issue.
> > On 32-bit we avoid extreme cases by capping it with MAX_THP_PAGESIZE.
> >
> > As for physical memory pressure, if a mapping happens to satisfy the
> > THP size and alignment requirements, the kernel may still collapse it
> > into a huge page even when only a few normal pages are actually
> > populated. In that case more physical memory will be allocated.
> >
> > From my understanding, the kernel THP collapse decision should
> > ultimately be responsible for this, and the control we have here is
> > actually quite weak.
>
> Besides higher memory usage and possible increase of number of memory
> mappings (due gapping filling in _dl_map_segments), it also has the side
> effect of silent lowering ASLR bits.  We had some issue in the past
> (check here [1] for some history), so I think we should avoid doing
> changing the mapping alignment without users explicitly asking for this.

I agree with this. It should lower the ASLR bits, but the maximum 32M
address alignment shouldn't cause ASLR to fail, and in 64-bit systems,
the disturbance might be pretty small.

> >
> >> In fact, I think the best option for this THP enablement would to work
> >> towards adding a ELF marking as hint to enable it.  It can also help on
> >> the static and binary/ld.so case, since it is up to the kernel to the
> >> alignment.
> >
> > I would suggest that we avoid requiring changes to ELF binaries for
> > this. Otherwise a large number of existing binaries would simply miss
> > this opt.
>
> Other system uses similar strategies to accomplish it, AIX with
> XCOFF Auxiliary Header and Solaris with MPSS and PF_SUNW_LARGEPAGE.
>
> Having a proper ELF marking will close the missing kernel gap for
> static and the loader, where it requires using extra tools like BOLT
> hugify tool to accomplish.
>
> Also, having the tunable it would be always possible to enable it
> and with a system-wide tunable enforce it on the system.

Regarding modifying the ELF, one possible option would be to adjust
p_align directly in the ELF. However, either recompiling or manually
editing the linked ELF would be a fairly high barrier, and in practice
it would be hard for end users to actually make use of it.

Overall, we're on the same page here. Opting in through
glibc.elf.hugetlb on most arches seems like the way to go.

Thanks,
Rui