[PATCH v6 5/6] elf: Align large load segments to PMD huge page size for THP

[Wilco Dijkstra]

Hi Rui,

> Mapping segments that are at least the size of a PMD huge page to
> huge-page-aligned addresses helps make them eligible for Transparent
> Huge Pages (THP).
>
> This patch introduces a Linux-specific helper, `_dl_map_segment_align`,
> to determine an appropriate maximum alignment for ELF load segments based
> on the system THP policy. The optimization is enabled only when the glibc
> tunable `glibc.elf.hugetlb=1` is set and THP is configured to be used
> unconditionally.
>
> The optimization depends on Linux kernel support for file-backed THP,
> specifically:
>
> * `CONFIG_READ_ONLY_THP_FOR_FS` (available since Linux kernel 5.4), and

Do you know the status of this in the kernel? It used to be experimental, and
I think it is still off by default in most distros.

> * `CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS`.
>
> When enabled, the helper queries the default THP page size and uses it
> to align sufficiently large load segments that are already properly
> aligned in both virtual address and file offset (e.g., zero).
>
> For eligible segments, the alignment is bumped to the THP page size,
> which improves THP eligibility, reduces TLB pressure, and improves
> performance for large objects. To avoid excessive address space padding
> on systems with very large THP sizes, the alignment is capped at 32MB.
> The optimization is applied only to non-writable segments, matching
> typical THP usage.

This is all reasonable, but note there will be an issue with enabling this by
default (see below).

+#include <dl-map-segment-align.h>
+#include <dl-tunables.h>
+#include <hugepages.h>
+
+#define MAX_THP_PAGESIZE (32 * 1024 * 1024)

Should this be in generic hugepages.h? In malloc there is currently no check
that rejects really huge pages, so something like this is useful. Having it in the
generic header also allows targets to override it if necessary.

+ElfW (Addr)
+_dl_map_segment_align (const struct loadcmd *c, ElfW (Addr) p_align_max)
+{
+  static enum thp_mode_t thp_mode = thp_mode_not_supported;
+  static unsigned long int thp_pagesize;
+
+  if (TUNABLE_GET (glibc, elf, hugetlb, int32_t, NULL) == 0)
+    return p_align_max;
+
+  if (__glibc_unlikely (thp_mode == thp_mode_not_supported
+                       || thp_pagesize == 0))
+    {
+      thp_mode = __get_thp_mode ();
+      thp_pagesize = DL_MAP_DEFAULT_THP_PAGESIZE ? : __get_thp_size ();
+    }

When the tunable is enabled by default, this will always call __get_thp_mode()
which may fail (or even abort) on some containers. I think we could add a
similar workaround for this: if DL_MAP_DEFAULT_THP_PAGESIZE is set, use that
and force thp_mode_always without calling __get_thp_mode(). The alignment is
essentially for free, it's just wasting virtual space.

+  /* Aligning load segments that are large enough to the PMD size helps
+     improve THP eligibility and reduces TLB pressure.
+     We cap the huge page size at MAX_THP_PAGESIZE to avoid over-aligning
+     on systems with very large normal pages (like 64K pages with 512M
+     huge pages). */
+  if (thp_mode == thp_mode_always && thp_pagesize <= MAX_THP_PAGESIZE
+      && ((c->mapstart | c->mapoff) & (thp_pagesize - 1)) == 0
+      && (c->mapend - c->mapstart) >= thp_pagesize
+      && p_align_max < thp_pagesize && (c->prot & PROT_WRITE) == 0)
+    return thp_pagesize;
+
+  return p_align_max;
+}

Cheers,
Wilco