[PATCH] elf: Align large load segments to PMD huge page size for THP

[Adhemerval Zanella Netto]

On 24/02/26 15:29, Wilco Dijkstra wrote:
> Hi Wang,
> 
> +#ifdef __LP64__
> +	  /* Dynamically calculate the PMD huge page size based on the normal
> +	     page size (e.g., 2M for 4K pages, 32M for 16K pages). Aligning
> +	     load segments that are large enough to the PMD size helps improve
> +	     THP eligibility and reduces TLB pressure.
> +	     We cap the huge page size at 32M to avoid over-aligning on systems
> +	     with very large normal pages (like 64K pages with 512M huge pages). */
> +	  size_t page_size = GLRO(dl_pagesize);
> +	  size_t huge_page_size = page_size * (page_size / sizeof (void *));
> +	  size_t huge_page_mask = huge_page_size - 1;
> +	  if (huge_page_size <= (32 * 1024 * 1024)
> +	      && ((c->mapstart | c->mapoff) & huge_page_mask) == 0
> +	      && (c->mapend - c->mapstart) >= huge_page_size
> +	      && p_align_max < huge_page_size)
> +	    p_align_max = huge_page_size;
> +#endif
> 
> So I think the idea of this is great, but like with malloc, it seems best to make
> this opt-in via a header (maybe similar mechanism as malloc-hugepages.h)
> and allow different sizes rather than calculate a single huge page size.
> For example on AArch64 the best huge page size is 2MB irrespectively of the
> base pagesize. On other targets, this calculation might not be valid or there
> are no huge pages.
> 

I would also prefer to enable this for glibc.malloc.hugetlb=1, instead of making
this the default.  And I agree that we need to either add a arch-specific
size, or query this from the kernel instead of assuming ad-hoc value.

> Also I don't see any madvise call in the elf/ directory. Is that needed as well
> or is that automatic based on the alignment?
Using MADV_HUGEPAGE unconditionally for text segments has some issues:

  1. It is currently gated through CONFIG_READ_ONLY_THP_FOR_FS (Linux 5.4),
     so it might be wasteful for some configurations.

  2. The collapsing of memory segment to a huge page is controlled by
     max_ptes_none [1].  The default argument for most distros are high
     enough to work most usercases, but lower values might add extra issues
     because large parts of a text segment might be initialization code or
     rarely executed where khugepaged will continuously scan the memory
     and see the regions 'empty'.

  3. There are additional latency when khugepaged promotes the region to
     huge pages.

  4. Under heavy memory pressure they might additional pressure page cache
     thrashing because the kernel cannot easily evict just 4KB of a 2MB 
     huge page.

So I think it should opt-in tuning. 

[1] /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none