[PATCH v7] Mark pages with MADV_DONTNEED to shrink and grow rather than mremap()
William Hunt
williamhuntdev@gmail.com
Thu Sep 11 15:25:05 GMT 2025
When reallocating mmap()ed chunks, use madvise() if shrinking to mark unused
pages as MADV_DONTNEED, only making a call to mremap() on failure. Allow
growing within MADV_DONTNEED pages for later calls to realloc under the
original size. This improves the efficiency of shrinking large mmap()ed
chunks, as madvise() is significantly faster than mremap(). It also provides
robustness, as if mremap() fails when shrinking, another madvise() without the
threshold check is attempted, and if this fails the pointer is returned as to
avoid a potential malloc+memcpy+free failure. Since mremap() fragments the VAS
by usually shrinking in-place, using madvise() will keep the VAS intact while
freeing the physical frames backing the unused pages, such that they will be
zero-filled before the first access. To avoid the process' VAS from being
exhausted, create a threshold for the maximum relative size of an mmap()ed
chunk that can be marked MADV_DONTNEED. Additionally, prevent an madvise()
call if not in secure mode and overcommit memory is not being used.
Place the logic for reallocating mmap()ed chunks into an _int_realloc_mmapped
function to increase modularity.
Create a tst-realloc-madvise.c test to verify that the relative threshold
works as intended. Check that the RSS is the same when reallocating above the
relative threshold, and the RSS changes if realloc is expected to call mremap.
Create a bench-realloc-shrink.c benchtest to show a 210% increase in
reallocs/sec when shrinking up until an arbitrary limit for the process,
verifying that realloc does handle shrinking large mmap()ed chunks more
efficiently when using madvise() rather than mremap().
Update malloc-check.c to call _int_realloc_mmapped directly when testing
realloc if the chunk is mmapped.
Changes from v1:
- Used the correct tst-realloc-madvise.c, v1 had an older incorrect version.
Changes from v2:
- Add the prev_size before calculating the max madvise size, v2 failed on i386.
Changes from v3:
- Remove unused oldmem param in _int_realloc_mmapped.
- Use mmap_base and mmap_size macros for better readability.
- Check in tst-realloc-madvise for RSS change after memset and madvise calls.
- Check in tst-realloc-madvise if THPs back up to the nearest HugeTLB page.
Changes from v4:
- Fixed bug in tst-realloc-madvise where memory past the allocation was set.
Changes from v5:
- Don't MADV_DONTNEED if using secure mode or not using overcommit memory.
- Add check_can_keep_vma() in Linux's malloc sysdeps before calling madvise().
- Refactored check_may_shrink_heap() to use checks from check_can_keep_vma().
Changes from v6:
- Fixed issue in malloc-sysdep.h with static variable not being assigned to.
- Used atomics to read/write to static variable to fix threading bug.
- Removed unnecessary check for __libc_enable_secure.
- Refactored into check_may_overcommit() for both arenas and mmap'd chunks.
- Called check_may_overcommit() before first madvise() option only.
Passed regress, OK for commit?
Signed-off-by: William Hunt <williamhuntdev@gmail.com>
---
benchtests/Makefile | 3 +
benchtests/bench-realloc-shrink.c | 180 ++++++++++++++++
malloc/Makefile | 6 +-
malloc/arena.c | 2 +-
malloc/malloc-check.c | 24 +--
malloc/malloc.c | 113 +++++++---
malloc/tst-realloc-madvise.c | 276 ++++++++++++++++++++++++
sysdeps/unix/sysv/linux/malloc-sysdep.h | 37 ++--
8 files changed, 568 insertions(+), 73 deletions(-)
create mode 100644 benchtests/bench-realloc-shrink.c
create mode 100644 malloc/tst-realloc-madvise.c
diff --git a/benchtests/Makefile b/benchtests/Makefile
index a52f10ed50..6aa1287573 100644
--- a/benchtests/Makefile
+++ b/benchtests/Makefile
@@ -356,10 +356,12 @@ bench-malloc := \
malloc-simple \
malloc-tcache \
malloc-thread \
+ realloc-shrink \
# bench-malloc
else
bench-malloc := $(filter malloc-%,${BENCHSET})
bench-malloc += $(filter calloc-%,${BENCHSET})
+bench-malloc += $(filter realloc-%,${BENCHSET})
endif
ifeq (${STATIC-BENCHTESTS},yes)
@@ -486,6 +488,7 @@ VALIDBENCHSETNAMES := \
malloc-tcache \
malloc-thread \
math-benchset \
+ realloc-shrink \
stdio-benchset \
stdio-common-benchset \
stdlib-benchset \
diff --git a/benchtests/bench-realloc-shrink.c b/benchtests/bench-realloc-shrink.c
new file mode 100644
index 0000000000..8dedc28afd
--- /dev/null
+++ b/benchtests/bench-realloc-shrink.c
@@ -0,0 +1,180 @@
+/* Measure shrinking mmap()'ed chunks with madvise.
+ Copyright (C) 2025 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <assert.h>
+#include <malloc.h>
+#include <stdbit.h>
+#include <stddef.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <sys/resource.h>
+#include <unistd.h>
+#include "bench-timing.h"
+#include "json-lib.h"
+
+#define PAGESIZE getpagesize()
+#define MMAP_RELATIVE_DONTNEED (1.0 / 4.0)
+#define MAX_MMAP_DONTNEED_MEM (size_t) 4194304
+#define MAX_SYSTEM_DONTNEED_MEM (size_t) 134217728
+#define START_SIZE (size_t) (MAX_MMAP_DONTNEED_MEM + PAGESIZE)
+#define NUM_ALLOCS (size_t) (MAX_SYSTEM_DONTNEED_MEM / MAX_MMAP_DONTNEED_MEM)
+#define NUM_ITERS 10
+#define MAX_PAGES_DIFF (size_t) (((START_SIZE - PAGESIZE) * (1 - MMAP_RELATIVE_DONTNEED)) / PAGESIZE)
+#define TEST_NAME "realloc-shrink"
+
+static size_t num_pages_diff = 1;
+#define get_num_shrinks() (size_t) (MAX_PAGES_DIFF / num_pages_diff)
+static size_t num_shrinks = 1;
+static size_t num_realloc_iters = 0;
+static void *ps[NUM_ALLOCS];
+
+typedef struct
+{
+ size_t num_iters;
+ double mmap_relative_dontneed;
+ size_t max_mmap_dontneed_mem;
+ timing_t elapsed;
+} realloc_bench_args;
+
+static realloc_bench_args args;
+
+void
+alloc_pointers (void)
+{
+ for (size_t i = 0; i < NUM_ALLOCS; ++i)
+ ps[i] = malloc (START_SIZE - PAGESIZE / 2);
+}
+
+void
+free_pointers (void)
+{
+ for (size_t i = 0; i < NUM_ALLOCS; ++i)
+ {
+ free (ps[i]);
+ ps[i] = NULL;
+ }
+}
+
+void
+shrink_max_times (void)
+{
+ for (size_t i = 0; i < NUM_ALLOCS; ++i)
+ {
+ size_t shrink_size = PAGESIZE * num_pages_diff;
+ for (size_t j = 0; j < num_shrinks; ++j)
+ {
+ ++num_realloc_iters;
+ ps[i] = realloc (ps[i], (size_t) (START_SIZE - (PAGESIZE / 2) - shrink_size));
+ shrink_size += (PAGESIZE * num_pages_diff);
+ assert (ps[i] != NULL);
+ }
+ }
+}
+
+void
+grow_max_times (void)
+{
+ for (size_t i = 0; i < NUM_ALLOCS; ++i)
+ {
+ size_t shrink_size = PAGESIZE * num_pages_diff * (num_shrinks - 1);
+ for (size_t j = 0; j < num_shrinks; ++j)
+ {
+ ++num_realloc_iters;
+ ps[i] = realloc (ps[i], (size_t) (START_SIZE - (PAGESIZE / 2) - shrink_size));
+ shrink_size -= (PAGESIZE * num_pages_diff);
+ assert (ps[i] != NULL);
+ }
+ }
+}
+
+static void
+do_benchmark (realloc_bench_args *args)
+{
+ timing_t start, stop;
+ size_t num_iters = args->num_iters;
+ double mmap_relative_dontneed = args->mmap_relative_dontneed;
+ size_t max_mmap_dontneed_mem = args->max_mmap_dontneed_mem;
+
+ const size_t max_diff = (size_t) ((max_mmap_dontneed_mem * mmap_relative_dontneed) / PAGESIZE);
+ assert ((max_diff & (max_diff - 1)) == 0);
+ alloc_pointers ();
+
+ TIMING_NOW (start);
+
+ for (size_t i = 0; i < num_iters; ++i)
+ {
+ for (size_t i = 0; i <= __builtin_ctzll (MAX_PAGES_DIFF); ++i)
+ {
+ num_pages_diff = (2 << i) / 2;
+ num_shrinks = get_num_shrinks ();
+ shrink_max_times ();
+ grow_max_times ();
+ }
+ }
+
+ TIMING_NOW (stop);
+ TIMING_DIFF (args->elapsed, start, stop);
+}
+
+void
+bench (void)
+{
+ args.num_iters = NUM_ITERS;
+ args.mmap_relative_dontneed = MMAP_RELATIVE_DONTNEED;
+ args.max_mmap_dontneed_mem = MAX_MMAP_DONTNEED_MEM;
+
+ do_benchmark (&args);
+
+ free_pointers ();
+
+ json_ctx_t json_ctx;
+ json_init (&json_ctx, 0, stdout);
+ json_document_begin (&json_ctx);
+ json_attr_string (&json_ctx, "timing_type", TIMING_TYPE);
+ json_attr_object_begin (&json_ctx, "functions");
+ json_attr_object_begin (&json_ctx, TEST_NAME);
+
+ json_attr_uint (&json_ctx, "start_size", START_SIZE);
+
+ struct rusage usage;
+ getrusage (RUSAGE_SELF, &usage);
+ json_attr_uint (&json_ctx, "max_rss", usage.ru_maxrss);
+ json_attr_double (&json_ctx, "reallocs/sec", num_realloc_iters / (args.elapsed / 1e9f));
+
+ json_attr_object_end (&json_ctx);
+ json_attr_object_end (&json_ctx);
+
+ json_document_end (&json_ctx);
+}
+
+static void usage (const char *name)
+{
+ fprintf (stderr, "%s\n", name);
+ exit (1);
+}
+
+int
+main (int argc, char **argv)
+{
+ if (argc != 1)
+ usage (argv[0]);
+
+ bench ();
+
+ return 0;
+}
\ No newline at end of file
diff --git a/malloc/Makefile b/malloc/Makefile
index cc012e2921..fcc61df17e 100644
--- a/malloc/Makefile
+++ b/malloc/Makefile
@@ -61,6 +61,7 @@ tests := \
tst-pvalloc \
tst-pvalloc-fortify \
tst-realloc \
+ tst-realloc-madvise \
tst-reallocarray \
tst-safe-linking \
tst-tcfree1 tst-tcfree2 tst-tcfree3 tst-tcfree4 \
@@ -113,6 +114,7 @@ tests-exclude-malloc-check = \
tst-memalign-2 \
tst-memalign-3 \
tst-mxfast \
+ tst-realloc-madvise \
tst-safe-linking \
tst-tcfree4 \
# tests-exclude-malloc-check
@@ -143,7 +145,8 @@ tests-exclude-hugetlb1 = \
# overlapping region.
tests-exclude-hugetlb2 = \
$(tests-exclude-hugetlb1) \
- tst-free-errno
+ tst-free-errno \
+ tst-realloc-madvise
tests-malloc-hugetlb1 = \
$(filter-out $(tests-exclude-hugetlb1), $(tests))
tests-malloc-hugetlb2 = \
@@ -189,6 +192,7 @@ tests-exclude-mcheck = \
tst-memalign-2 \
tst-memalign-3 \
tst-mxfast \
+ tst-realloc-madvise \
tst-safe-linking \
# tests-exclude-mcheck
diff --git a/malloc/arena.c b/malloc/arena.c
index 4cc9881d82..c4a954497c 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -502,7 +502,7 @@ shrink_heap (heap_info *h, long diff)
/* Try to re-map the extra heap space freshly to save memory, and make it
inaccessible. See malloc-sysdep.h to know when this is true. */
- if (__glibc_unlikely (check_may_shrink_heap ()))
+ if (__glibc_unlikely (!check_may_overcommit ()))
{
if ((char *) MMAP ((char *) h + new_size, diff, PROT_NONE,
MAP_FIXED) == (char *) MAP_FAILED)
diff --git a/malloc/malloc-check.c b/malloc/malloc-check.c
index 40b215ea83..850b5b5d9d 100644
--- a/malloc/malloc-check.c
+++ b/malloc/malloc-check.c
@@ -285,28 +285,8 @@ realloc_check (void *oldmem, size_t bytes)
if (chunk_is_mmapped (oldp))
{
-#if HAVE_MREMAP
- mchunkptr newp = mremap_chunk (oldp, chnb);
- if (newp)
- newmem = chunk2mem_tag (newp);
- else
-#endif
- {
- size_t oldsize = memsize (oldp);
- if (oldsize >= rb)
- newmem = oldmem; /* do nothing */
- else
- {
- /* Must alloc, copy, free. */
- top_check ();
- newmem = _int_malloc (&main_arena, rb);
- if (newmem)
- {
- memcpy (newmem, oldmem, oldsize);
- munmap_chunk (oldp);
- }
- }
- }
+ top_check ();
+ newmem = _int_realloc_mmapped (oldmem, bytes);
}
else
{
diff --git a/malloc/malloc.c b/malloc/malloc.c
index 802318d143..c8c0c378e6 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -1102,6 +1102,7 @@ static INTERNAL_SIZE_T _int_free_create_chunk (mstate,
static void _int_free_maybe_consolidate (mstate, INTERNAL_SIZE_T);
static void* _int_realloc(mstate, mchunkptr, INTERNAL_SIZE_T,
INTERNAL_SIZE_T);
+static void* _int_realloc_mmapped (void *oldmem, size_t bytes);
static void* _int_memalign(mstate, size_t, size_t);
#if IS_IN (libc)
static void* _mid_memalign(size_t, size_t);
@@ -3588,7 +3589,7 @@ __libc_realloc (void *oldmem, size_t bytes)
{
size_t difference = usable - bytes;
if ((unsigned long) difference < 2 * sizeof (INTERNAL_SIZE_T))
- return oldmem;
+ return oldmem;
}
/* its size */
@@ -3610,35 +3611,7 @@ __libc_realloc (void *oldmem, size_t bytes)
nb = checked_request2size (bytes);
if (chunk_is_mmapped (oldp))
- {
- void *newmem;
-
-#if HAVE_MREMAP
- newp = mremap_chunk (oldp, nb);
- if (newp)
- {
- void *newmem = chunk2mem_tag (newp);
- /* Give the new block a different tag. This helps to ensure
- that stale handles to the previous mapping are not
- reused. There's a performance hit for both us and the
- caller for doing this, so we might want to
- reconsider. */
- return tag_new_usable (newmem);
- }
-#endif
- /* Return if shrinking and mremap was unsuccessful. */
- if (bytes <= usable)
- return oldmem;
-
- /* Must alloc, copy, free. */
- newmem = __libc_malloc (bytes);
- if (newmem == NULL)
- return NULL; /* propagate failure */
-
- memcpy (newmem, oldmem, oldsize - CHUNK_HDR_SZ);
- munmap_chunk (oldp);
- return newmem;
- }
+ return _int_realloc_mmapped (oldmem, bytes);
ar_ptr = arena_for_chunk (oldp);
@@ -5114,6 +5087,86 @@ _int_realloc (mstate av, mchunkptr oldp, INTERNAL_SIZE_T oldsize,
return tag_new_usable (chunk2mem (newp));
}
+/* Equivalent to a relative threshold of 1/4 for mmap()'ed chunks. */
+static __always_inline size_t
+max_madvise (size_t madvise_sz)
+{
+ return madvise_sz - (madvise_sz >> 2);
+}
+
+/* Shrink the region by marking pages as MADV_DONTNEED.
+ The physical frames are released, but madvise() preserves the VAS.
+ This prevents fragmenting the address space like mremap() would do. */
+static __always_inline bool
+_int_realloc_madvise (mchunkptr oldp, size_t difference)
+{
+ char *madv_start = (char *) mmap_base (oldp) + mmap_size (oldp) - difference;
+ return __madvise (madv_start, difference, MADV_DONTNEED) != -1;
+}
+
+static void *
+_int_realloc_mmapped (void *oldmem, size_t bytes)
+{
+ const mchunkptr oldp = mem2chunk (oldmem);
+ const INTERNAL_SIZE_T oldsize = chunksize (oldp);
+ size_t nb = checked_request2size (bytes);
+ size_t usable = musable (oldmem);
+ size_t difference = usable - bytes;
+
+ /* For mmap()'ed chunks where the size is within the region's VAS. */
+ if (bytes <= usable)
+ {
+ if (difference < GLRO (dl_pagesize))
+ return oldmem;
+
+ /* Only MADV_DONTNEED if allowing overcommit memory, otherwise there
+ is a risk of exhausting the process' pre-determined VMA fraction. */
+ if (__glibc_likely (check_may_overcommit ()))
+ {
+ difference = ALIGN_DOWN (difference, GLRO (dl_pagesize));
+ /* Don't shrink more than the relative threshold for the chunk. */
+ if (difference <= max_madvise (mmap_size (oldp)) &&
+ _int_realloc_madvise (oldp, difference))
+ return oldmem;
+ }
+ }
+
+ void *newmem;
+
+#if HAVE_MREMAP
+ void *newp = mremap_chunk (oldp, nb);
+ if (newp)
+ {
+ newmem = chunk2mem_tag (newp);
+ /* Give the new block a different tag. This helps to ensure
+ that stale handles to the previous mapping are not
+ reused. There's a performance hit for both us and the
+ caller for doing this, so we might want to
+ reconsider. */
+ return tag_new_usable (newmem);
+ }
+#endif
+
+ /* Last attempt to prevent malloc+memcpy+free when shrinking the region. */
+ if (bytes <= usable)
+ {
+ /* Attempt another madvise() call without the previous threshold check.
+ Even if the call to madvise() failed, since we are shrinking it is
+ safer to just return, rather than risk a malloc+memcpy+free error. */
+ _int_realloc_madvise (oldp, difference);
+ return oldmem;
+ }
+
+ /* Must alloc, copy, free if the region grows as a last resort. */
+ newmem = __libc_malloc (bytes);
+ if (newmem == NULL)
+ return NULL; /* propagate failure */
+
+ memcpy (newmem, oldmem, oldsize - CHUNK_HDR_SZ);
+ munmap_chunk (oldp);
+ return newmem;
+}
+
/*
------------------------------ memalign ------------------------------
*/
diff --git a/malloc/tst-realloc-madvise.c b/malloc/tst-realloc-madvise.c
new file mode 100644
index 0000000000..724419c8ac
--- /dev/null
+++ b/malloc/tst-realloc-madvise.c
@@ -0,0 +1,276 @@
+/* Test for realloc madvise use in shrinking and growing mmap()'ed chunks.
+ Copyright (C) 2025 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <errno.h>
+#include <libc-pointer-arith.h>
+#include <stdbool.h>
+#include <stdlib.h>
+#include <string.h>
+#include <support/check.h>
+#include <unistd.h>
+
+#include "malloc-size.h"
+#include "tst-malloc-aux.h"
+
+static int pagesize;
+static size_t system_dontneed_mem = 0;
+static struct mallinfo2 original_mi;
+static size_t original_size;
+/* If testing with THPs, the kernel may promote to a THP which will back the
+ region with physical frames and max out the RSS, but only once. */
+bool using_thp = false;
+
+static __always_inline size_t
+unusable (void *p)
+{
+ return (uintptr_t) p - ALIGN_DOWN ((uintptr_t) p, pagesize);
+}
+
+static __always_inline void *
+checked_malloc (void)
+{
+ /* Take off half a page to safely handle architectural alignment.
+ The region will be rounded up to the nearest page size anyways. */
+ void *p = malloc (original_size - pagesize / 2);
+ if (p == NULL && errno == ENOMEM)
+ FAIL_UNSUPPORTED ("Not enough memory for the minimum mmap threshold");
+ TEST_VERIFY (p != NULL);
+ memset (p, 42, original_size - pagesize / 2);
+ return p;
+}
+
+static size_t
+smaps_data (void *mmap_base, size_t mmap_size, bool expect_same)
+{
+ FILE *f = fopen ("/proc/self/smaps", "r");
+ if (!f) return -1;
+ char line[512]; bool in = false;
+ long long kb = -1;
+
+ while (fgets (line, sizeof line, f))
+ {
+ if (in)
+ {
+ if (sscanf (line, "Rss: %lld kB", &kb) == 1)
+ {
+ fclose(f);
+ return (size_t) kb * 1024;
+ }
+ }
+
+ unsigned long long start, end;
+ if (sscanf (line, "%llx-%llx", &start, &end) != 2) continue;
+ if ((uintptr_t) mmap_base != (uintptr_t) start) continue;
+ if (expect_same)
+ TEST_VERIFY ((uintptr_t) mmap_base + mmap_size == (uintptr_t) end);
+ else
+ TEST_VERIFY ((uintptr_t) mmap_base + mmap_size != (uintptr_t) end);
+ in = true;
+ }
+ fclose (f);
+ return (size_t) -1;
+}
+
+static void *
+shrink_page (void *p, size_t old_size, bool expect_same)
+{
+ TEST_VERIFY (p != NULL);
+ const size_t new_size = old_size - pagesize - pagesize / 2;
+
+ void *oldp = p;
+ p = realloc (p, new_size);
+ /* If only marking pages as MADV_DONTNEED, the new pointer will not change.
+ But mremap may not shrink in-place due to alignment or fragmentation. */
+ size_t num_frames = smaps_data (p - unusable (p),
+ original_size, expect_same);
+
+ /* The kernel may promote to THP and physically back the VAS arbitrarily. */
+ TEST_VERIFY (num_frames == new_size + pagesize / 2 ||
+ (using_thp && num_frames > new_size + pagesize / 2));
+ if (expect_same)
+ TEST_VERIFY (p == oldp);
+
+ size_t mmap_dontneed_mem;
+ size_t chunksize = malloc_usable_size (p) + unusable (p) + SIZE_SZ;
+
+ if (original_size == chunksize)
+ {
+ TEST_VERIFY (expect_same);
+ mmap_dontneed_mem = ALIGN_DOWN (original_size - new_size - unusable (p),
+ pagesize);
+ system_dontneed_mem += pagesize;
+ TEST_VERIFY (system_dontneed_mem == mmap_dontneed_mem);
+ }
+ else
+ {
+ TEST_VERIFY (!expect_same);
+ mmap_dontneed_mem = 0;
+ }
+
+ struct mallinfo2 new_mi = mallinfo2 ();
+ size_t min_used_size = original_size >> 2;
+ /* If there are no more MADV_DONTNEED pages, ignore the threshold. */
+ if (mmap_dontneed_mem && mmap_dontneed_mem <= chunksize - min_used_size)
+ {
+ TEST_VERIFY (expect_same);
+ TEST_VERIFY (new_mi.hblkhd - original_mi.hblkhd == original_size);
+ return p;
+ }
+ TEST_VERIFY (!expect_same);
+
+ TEST_VERIFY (ALIGN_DOWN (new_mi.hblkhd, pagesize) - original_mi.hblkhd ==
+ ALIGN_DOWN (min_used_size - 1, pagesize));
+
+ return p;
+}
+
+static void *
+grow_page (void *p, size_t old_size, size_t old_dontneed, bool expect_same)
+{
+ TEST_VERIFY (p != NULL);
+ const size_t usable = original_size - unusable (p);
+ size_t new_size = old_size + pagesize / 2;
+
+ void *oldp = p;
+ p = realloc (p, new_size);
+ memset(p - unusable (p) + old_size, 42, pagesize);
+ size_t num_frames = smaps_data (p - unusable (p),
+ original_size, expect_same);
+
+ TEST_VERIFY (num_frames == new_size + pagesize / 2 ||
+ (using_thp && num_frames > new_size + pagesize / 2));
+
+ /* If the region was not extended the returned pointer will not change.
+ * If it did extend it may have done so in-place, this cannot be tested. */
+ if (expect_same)
+ TEST_VERIFY (p == oldp);
+
+ new_size = ALIGN_UP (new_size, pagesize);
+
+ struct mallinfo2 new_mi = mallinfo2 ();
+ /* If growing within the MADV_DONTNEED pages. */
+ if (new_size - unusable (p) <= usable && new_size > usable - old_dontneed)
+ {
+ TEST_VERIFY (expect_same);
+ TEST_VERIFY (ALIGN_DOWN (new_mi.hblkhd, pagesize) - original_mi.hblkhd ==
+ original_size);
+ }
+ /* Otherwise the allocation should have grown by a page. */
+ else
+ {
+ TEST_VERIFY (!expect_same);
+ TEST_VERIFY (ALIGN_DOWN (new_mi.hblkhd, pagesize) - original_mi.hblkhd ==
+ original_size + pagesize);
+ }
+
+ return p;
+}
+
+static void
+test_allocation_size_threshold (void)
+{
+ void *p = checked_malloc ();
+
+ struct mallinfo2 new_mi = mallinfo2 ();
+ /* All previous allocations should be done within the main arena. */
+ TEST_VERIFY (new_mi.hblkhd == original_size);
+
+ /* Shrink down to the relative size threshold. */
+ size_t size = original_size;
+ const size_t min_used_size = original_size >> 2;
+ while (size > min_used_size)
+ {
+ p = shrink_page (p, size, true);
+ size -= pagesize;
+ }
+ /* This should exceed the relative size threshold. */
+ p = shrink_page (p, size, false);
+ /* Get a fresh region for testing growing within a region. */
+ free (p);
+ p = checked_malloc ();
+ system_dontneed_mem = 0;
+
+ /* Reallocate down to the relative threshold. */
+ size = min_used_size;
+ p = realloc(p, size - unusable (p));
+ new_mi = mallinfo2 ();
+ TEST_VERIFY (ALIGN_DOWN (new_mi.hblkhd, pagesize) - original_mi.hblkhd ==
+ original_size);
+
+ /* Grow back up to the original size. */
+ while (size < original_size)
+ {
+ p = grow_page (p, size, original_size - size,
+ true);
+ size += pagesize;
+ }
+ /* This should exceed the size of the allocation. */
+ p = grow_page (p, size, original_size - size,
+ false);
+
+ /* New pointer to test shrinking immediately past the relative threshold. */
+ free (p);
+ p = checked_malloc ();
+ system_dontneed_mem = 0;
+
+ /* Shrink a page past the size threshold, this should call mremap(). */
+ /* Must take into account the usable size of the original allocation. */
+ p = realloc (p, min_used_size - pagesize - pagesize / 2);
+ new_mi = mallinfo2 ();
+ TEST_VERIFY (ALIGN_DOWN (new_mi.hblkhd, pagesize) - original_mi.hblkhd ==
+ min_used_size - pagesize);
+
+ /* Grow a page to the threshold, this should call mremap(). */
+ p = realloc (p, min_used_size - pagesize / 2);
+ new_mi = mallinfo2 ();
+ TEST_VERIFY (ALIGN_DOWN (new_mi.hblkhd, pagesize) - original_mi.hblkhd ==
+ min_used_size);
+
+ /* Free so the pointer doesn't effect mallinfo for later tests. */
+ free (p);
+ system_dontneed_mem = 0;
+}
+
+static int
+do_test (void)
+{
+ pagesize = getpagesize ();
+ char *tunables = getenv ("GLIBC_TUNABLES");
+ if (tunables != NULL)
+ {
+ if (strstr (tunables, "hugetlb=1"))
+ using_thp = true;
+ else if (strstr (tunables, "hugetlb=2"))
+ FAIL_UNSUPPORTED ("Huge pages madvise unsupported in glibc\n");
+ }
+
+ /* Prevent MORECORE from being used for large regions. */
+ mallopt (M_MMAP_THRESHOLD, 131072);
+ /* The mmap minimum threshold has been set to 128K via tunables to prevent
+ unexpected calls, so there are no data between comparisons using mmap. */
+ original_mi = mallinfo2();
+
+ /* To ensure that the mmap()'ed region's relative size threshold is tested,
+ make the original allocation size far below max_mmap_dontneed_mem. */
+ original_size = 16 * 1024 * 1024;
+ test_allocation_size_threshold ();
+
+ return 0;
+}
+
+#include <support/test-driver.c>
\ No newline at end of file
diff --git a/sysdeps/unix/sysv/linux/malloc-sysdep.h b/sysdeps/unix/sysv/linux/malloc-sysdep.h
index 778d8971d5..1a0ee742a1 100644
--- a/sysdeps/unix/sysv/linux/malloc-sysdep.h
+++ b/sysdeps/unix/sysv/linux/malloc-sysdep.h
@@ -18,6 +18,7 @@
#include <fcntl.h>
#include <not-cancel.h>
+#include <stdatomic.h>
/* The Linux kernel overcommits address space by default and if there is not
enough memory available, it uses various parameters to decide the process to
@@ -26,34 +27,32 @@
that, a process is only allowed to use the maximum of a pre-determined
fraction of the total address space. In such a case, we want to make sure
that we are judicious with our heap usage as well, and explicitly give away
- the freed top of the heap to reduce our commit charge. See the proc(5) man
- page to know more about overcommit behavior.
+ the freed top of the heap to reduce our commit charge. We also want to unmap
+ unused sections of mmap'd chunks to prevent the VMA from being exhausted.
+ See the proc(5) man page to know more about overcommit behavior.
Other than that, we also force an unmap in a secure exec. */
static inline bool
-check_may_shrink_heap (void)
+check_may_overcommit (void)
{
- static int may_shrink_heap = -1;
+ static int may_overcommit = -1;
- if (__builtin_expect (may_shrink_heap >= 0, 1))
- return may_shrink_heap;
+ if (__glibc_likely (atomic_load_relaxed (&may_overcommit) >= 0))
+ return may_overcommit;
- may_shrink_heap = __libc_enable_secure;
-
- if (__builtin_expect (may_shrink_heap == 0, 1))
+ int fd = __open_nocancel ("/proc/sys/vm/overcommit_memory",
+ O_RDONLY | O_CLOEXEC);
+ if (fd >= 0)
{
- int fd = __open_nocancel ("/proc/sys/vm/overcommit_memory",
- O_RDONLY | O_CLOEXEC);
- if (fd >= 0)
- {
- char val;
- ssize_t n = __read_nocancel (fd, &val, 1);
- may_shrink_heap = n > 0 && val == '2';
- __close_nocancel_nostatus (fd);
- }
+ char val;
+ ssize_t n = __read_nocancel (fd, &val, 1);
+ int expected = -1;
+ atomic_compare_exchange_weak (&may_overcommit, &expected,
+ !(n > 0 && val == '2'));
+ __close_nocancel_nostatus (fd);
}
- return may_shrink_heap;
+ return may_overcommit;
}
#define HAVE_MREMAP 1
--
2.43.0
More information about the Libc-alpha
mailing list