[RFC] Add GNU_PROPERTY_1_NEEDED_THP_ON_[READ_ONLY_SEGMENTS|MALLOC]

[H.J. Lu]

Transparent Huge Pages (THP) has been enabled in glibc to use THP on
THP eligible read-only segments and malloc to improve application
performance.  But there are some common issues & trade-offs of THP:

1. Latency Spikes: The background process (khugepaged) that consolidates
pages can cause temporary system freezes or high CPU usage.
2. Memory Waste: Using a 2MB page for only 4KB of data can lead to
internal fragmentation and higher memory consumption.
3. Database Incompatibility: Many databases (e.g., Oracle, MongoDB,
SAP ASE) recommend disabling THP because their non-contiguous access
patterns often trigger performance degradations.

There is a glibc bug:

https://sourceware.org/bugzilla/show_bug.cgi?id=34081

THP shouldn't be always enabled on THP eligible read-only segments nor
in malloc blindly without the knowledge whether THP usage will improve
or hurt performance for the application.  To give application developers
per-application control over THP usage under madvise THP mode, add
the following bits to GNU_PROPERTY_1_NEEDED which indicates the properties
needed by object file for correctness or performance:

1. GNU_PROPERTY_1_NEEDED_THP_ON_READ_ONLY_SEGMENTS

 #define GNU_PROPERTY_1_NEEDED_THP_ON_READ_ONLY_SEGMENTS (1U << 1)

to opt-in THP usage on read-only segments in an application.  When this
bit is set on executable, run-time loader should call madvise with
MADV_HUGEPAGE on all Transparent Huge Pages (THP) eligible read-only
segments in executable and its loaded shared libraries after they are
mapped into memory under madvise THP mode.

2. GNU_PROPERTY_1_NEEDED_THP_ON_MALLOC

 #define GNU_PROPERTY_1_NEEDED_THP_ON_MALLOC (1U << 2)

to opt-in THP usage in malloc in an application.  When this bit is set on
executable, malloc should call madvise with MADV_HUGEPAGE if the
allocated size is greater than or equal to THP page size after calling
mmap to allocate memory under madvise THP mode.

Update GNU_PROPERTY_1_NEEDED description to clarify that it is for
correctness or performance.

Linker will set these bits in application if these bits are set in any
relocatable inputs.

Run-time and tool integration:

1. Add linker options to enable and disable THP:

-z thp=rosegment:
  Turn on the GNU_PROPERTY_1_NEEDED_THP_ON_READ_ONLY_SEGMENTS bit

-z thp=malloc:
  Turn on the GNU_PROPERTY_1_NEEDED_THP_ON_MALLOC bit

-z thp=all
  Turn on both bits

-z nothp=rosegment
  Turn off the GNU_PROPERTY_1_NEEDED_THP_ON_READ_ONLY_SEGMENTS bit

-z nothp=malloc
  Turn off the GNU_PROPERTY_1_NEEDED_THP_ON_MALLOC bit

-z nothp=all
  Turn off both bits

2. Add compiler options to generate these bits in the object output.

3. Add elfedit options to disable and enable these bits.

4. Update the C run-time to check these bits in the executable to enable
THP at load-time.

5. Add GNU_PROPERTY_1_NEEDED to crt1.o files with bits set if THP should
be enabled in the application by default.  THP can be disabled by elfedit
later if needed.

-- 
H.J.