[RFC PATCH 0/1] malloc: madvise interior free chunks above a threshold

[Xavier Roche]

Since glibc 2.26 introduced tcache, free() no longer returns
physical pages to the kernel for chunks that sit in the interior
of an arena heap.  Periodic malloc_trim(0) recovers the memory,
but free() itself cannot.  A bisect across 2.23-2.27 with the
reproducer (attached as malloc/tst-madvise-threshold.c) shows
a 4x RSS jump between 2.25 and 2.26.

BZ #33886 is the latest thread on this.  BZ #15321, #18910,
#27976 describe older aspects of the same behavior.  Several
projects ship workarounds: systemd, OpenJDK, Python, and our
own production search engine, which overrides free() and
triggers malloc_trim(0) on a background thread.

Mechanism
---------
_int_free_maybe_trim already runs for consolidated chunks
>= 64 KB and calls systrim / heap_trim on the top chunk.  This
patch extends it to madvise the page-aligned interior of the
consolidated chunk (same page-alignment logic as mtrim()).

An early version triggered madvise on every qualifying free,
which Wilco Dijkstra pointed out could produce ~1M syscalls
when a million small frees merge into one chunk (BZ #33886
comment 10).  To address that, the patch uses the caller's
original (pre-merge) chunk size as the gate:

  - orig_size >= one page + chunk header: madvise immediately.
  - orig_size smaller: accumulate in a per-arena counter, and
    fire madvise only when the accumulator crosses 256 KB.

In the worst case (one million 100-byte frees merging into a
100 MB chunk), this produces ~400 madvise calls instead of ~1M.
Normal workloads pay one accumulator increment per small free
and one madvise per 256 KB of pressure.

The per-arena counter is accessed only under av->mutex
(verified at both call sites: _int_free_merge_chunk and
_int_memalign).

For advice type: MADV_FREE for moderate chunks (lazy release,
no re-fault on quick reuse); MADV_DONTNEED for chunks >= 128 KB
(immediate RSS reduction matches what operators expect from
mtrim).  This split is an open design point; simplifying to
MADV_DONTNEED unconditionally is an option if preferred.

Impact
------
Reproducer: 16 threads, 256 MB live working set, 10 GB of
short-lived allocations below mmap threshold.

                          baseline    patched
  RSS after free           1247 MB    296 MB
  malloc_trim recovery      962 MB     14 MB   (free did the work)
  Runtime                  0.34 s     0.50 s   (tight malloc loop)

Runtime overhead is only visible in synthetic malloc-heavy
benchmarks.  Real applications spend a fraction of a percent
of their time in free(); the amortised cost is negligible.
Algolia production (the motivating workload) recovers 100+ GB
of previously-wasted RSS with equivalent semantics.

Background: https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-precious-memory

Testing
-------
- malloc test suite: all PASS, including the new
  tst-madvise-threshold that validates the RSS drop.
- Full make check on x86_64-linux-gnu: no new regressions
  vs baseline.
- Cross-built via build-many-glibcs.py for
  aarch64-linux-gnu, arm-linux-gnueabihf,
  powerpc64le-linux-gnu, s390x-linux-gnu: check-compilers,
  build and compile-only check passed on all four with no
  new warnings on malloc.c.  (i686 is covered by patchwork.)

Open questions
--------------
1. MADV_FREE vs MADV_DONTNEED split at 128 KB: keep the two-tier
   design, or always use MADV_DONTNEED (matches mtrim())?
2. Default-on vs tunable: the current patch is default-on on the
   argument that this is a regression fix since 2.26.  A tunable
   kill-switch (glibc.malloc.madvise_interior, default on) could
   be added if preferred.
3. Threshold values (64 KB entry gate, 256 KB accumulator, 128 KB
   MADV_DONTNEED boundary) are picked as multiples of
   ATTEMPT_TRIMMING_THRESHOLD; open to tuning.

Acknowledgments
---------------
Thanks to Wilco Dijkstra for the earlier review on BZ #33886
that surfaced the million-madvise-calls concern and redirected
the design toward the accumulator approach.  The bisect and
tcache-as-root-cause framing also came out of that discussion.

Related prior art: Mel Gorman's 2015 RFC
(https://sourceware.org/legacy-ml/libc-alpha/2015-02/msg00193.html)
explored a related madvise refault-avoidance question; this
patch is narrower in scope.

AI disclosure
-------------
AI assistance (Claude) was used during analysis and drafting:
reading the ptmalloc2 source, reviewing the regression bisect,
exploring design alternatives, and iterating on the patch text.
The design decisions, the benchmark runs, and the final code
were reviewed and validated by the author, who takes
responsibility for correctness.

Xavier Roche (1):
  malloc: madvise interior free chunks above a threshold

 malloc/Makefile                |   1 +
 malloc/malloc.c                |  61 +++++++++++++---
 malloc/tst-madvise-threshold.c | 128 +++++++++++++++++++++++++++++++++
 3 files changed, 182 insertions(+), 8 deletions(-)
 create mode 100644 malloc/tst-madvise-threshold.c


base-commit: e3e8f814e53d32da99db03ac38ed3af86651b233
-- 
2.43.0