[PATCH v5] malloc: Mark pages with MADV_DONTNEED to shrink and grow rather than mremap()

[Wilco Dijkstra]

Hi Florian,

(adding Dev Jain who has experience with Linux kernel page management code)

> I'm not too familiar with the Linux MM code.  How does MADV_DONTNEED
> affect the commit charge?

It doesn't affect the VAS (and thus commit charge) - it just reduces RSS by
releasing the physical pages.

> We have a similar task during heap management (inside an arena), and
> there we use check_may_shrink_heap to determine whether to use
> MADV_DONTNEED, or mmap with MAP_FIXED and PROT_NONE to get rid of the
> memory.  I'm a bit confused, though.  The check seems to in the wrong
> direction: force the use of mmap with PROT_NONE if *not* using
> vm.overcommit_memory=2.

check_may_shrink_heap() returns true if you're either in secure mode or if
vm.overcommit_memory=2, so that looks right to me. 

Interestingly the code that tries to grow a heap looks broken since it tries to
use mprotect to grow an unmapped region which should fail and would result
in new heaps being created (thus increasing VAS usage...).

> Besides the commit charge matter, does MADV_DONTNEED split the VMA?
> That could be problematic.  If it just rewrites the PTEs, it should be
> okay.  The current mremap approach is also likely split VMAs, but the
> scenarios are different.  Adding more splits would be a risky change.

No it doesn't, and that is one of the advantages. It also means you can later on
grow back to previous size if needed without having to use any system calls.

We could add a check for check_may_shrink_heap() - however there is a general
issue with granularity as discussed before. There is no point in using munmap()
for 1 or 2 pages (risking extra fragmentation and exceeding max mmap limit),
so all this needs to become smarter and do it on blocks of pages.

>> Create a bench-realloc-shrink.c benchtest to show a 210% increase in
>> reallocs/sec when shrinking up until an arbitrary limit for the process, 
>> verifying that realloc does handle shrinking large mmap()ed chunks more 
>> efficiently when using madvise() rather than mremap().
>
> Is this with or without MTE active?

No MTE.

Cheers,
Wilco