[PATCH v5] malloc: Mark pages with MADV_DONTNEED to shrink and grow rather than mremap()

[Florian Weimer]

* William Hunt:

> When reallocating mmap()ed chunks, use madvise() if shrinking to mark
> unused pages as MADV_DONTNEED, only making a call to mremap() on
> failure. Allow growing within MADV_DONTNEED pages for later calls to
> realloc under the original size. This improves the efficiency of
> shrinking large mmap()ed chunks, as madvise() is significantly faster
> than mremap(). It also provides robustness, as if mremap() fails when
> shrinking, another madvise() without the threshold check is attempted,
> and if this fails the pointer is returned as to avoid a potential
> malloc+memcpy+free failure. Since mremap() fragments the VAS by
> usually shrinking in-place, using madvise() will keep the VAS intact
> while freeing the physical frames backing the unused pages, such that
> they will be zero-filled before the first access. To avoid the
> process' VAS from being exhausted, create a threshold for the maximum
> relative size of an mmap()ed chunk that can be marked MADV_DONTNEED.

I think fragmentation stays largely the same with either approach, only
the type changes.

I'm not too familiar with the Linux MM code.  How does MADV_DONTNEED
affect the commit charge?

We have a similar task during heap management (inside an arena), and
there we use check_may_shrink_heap to determine whether to use
MADV_DONTNEED, or mmap with MAP_FIXED and PROT_NONE to get rid of the
memory.  I'm a bit confused, though.  The check seems to in the wrong
direction: force the use of mmap with PROT_NONE if *not* using
vm.overcommit_memory=2.

Besides the commit charge matter, does MADV_DONTNEED split the VMA?
That could be problematic.  If it just rewrites the PTEs, it should be
okay.  The current mremap approach is also likely split VMAs, but the
scenarios are different.  Adding more splits would be a risky change.

> Create a bench-realloc-shrink.c benchtest to show a 210% increase in
> reallocs/sec when shrinking up until an arbitrary limit for the process, 
> verifying that realloc does handle shrinking large mmap()ed chunks more 
> efficiently when using madvise() rather than mremap().

Is this with or without MTE active?

Thanks,
Florian