Bug 13276 - assertion failure in realloc when running out of virtual mappings
Summary: assertion failure in realloc when running out of virtual mappings
Alias: None
Product: glibc
Classification: Unclassified
Component: libc (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Ulrich Drepper
Depends on:
Reported: 2011-10-08 16:24 UTC by Andi Kleen
Modified: 2014-06-27 11:53 UTC (History)
2 users (show)

See Also:
Last reconfirmed:
fweimer: security-

test case (348 bytes, text/plain)
2011-10-08 20:01 UTC, Rich Felker

Note You need to log in before you can comment on or make changes to this bug.
Description Andi Kleen 2011-10-08 16:24:05 UTC
When a process runs out of virtual mappings on Linux
(more mmaps than vm.max_map_count) then munmap can fail because it may 
need to split a mapping.

In this case when there is a realloc() it will get an assertation 
failure because it doesn't expect munmap to fail.

Seen with gcc with a specific input file that fragments memory badly:

lto1: malloc.c:3551: munmap_chunk: Assertion `ret == 0' failed.

realloc should return NULL in this case, not assert. free should cleanly 
Comment 1 Rich Felker 2011-10-08 18:40:07 UTC
Making realloc fail in this case is not entirely a solution. The new memory has already been allocated at this point, and the exact same error (inability to munmap) could happen when trying to free this new allocation. (Maybe this isn't possible, however, if the new allocation is either always on the heap or performed atomically via mremap, i.e. if realloc never uses mmap+munmap.)

The most robust solution would be to ensure that each allocation is always its own vma by putting guard pages between them, i.e. mmap(size+1page) with PROT_NONE to begin with, then mmap size over top of that with MAP_FIXED. Unfortunately this increases the kernelspace memory usage quite a bit (more vmas) and adds an extra syscall to every allocation (probably an unacceptable performance cost).

An alternative solution might be merge the region that munmap fails to free into the free lists managed for non-mmap-serviced allocations.
Comment 2 Andi Kleen 2011-10-08 19:04:31 UTC
I guess it would be reasonable to just leave the mapping around in this case
Normally the program will error out anyways when this happens, otherwise
it should happen rarely.

Just it shouldn't assertation failure.
Comment 3 Rich Felker 2011-10-08 20:01:02 UTC
Created attachment 5967 [details]
test case
Comment 4 Rich Felker 2011-10-08 20:01:30 UTC
I can't reproduce the failure for realloc (calling realloc to resize a mmap-sized block down to size 1 results in mremap to PAGE_SIZE rather than servicing it from the heap), but I was able to reproduce it with free instead of realloc. Test case is attached.
Comment 5 Ulrich Drepper 2011-10-29 20:40:10 UTC
I checked in a patch.