When a process runs out of virtual mappings on Linux (more mmaps than vm.max_map_count) then munmap can fail because it may need to split a mapping. In this case when there is a realloc() it will get an assertation failure because it doesn't expect munmap to fail. Seen with gcc with a specific input file that fragments memory badly: lto1: malloc.c:3551: munmap_chunk: Assertion `ret == 0' failed. realloc should return NULL in this case, not assert. free should cleanly return.
Making realloc fail in this case is not entirely a solution. The new memory has already been allocated at this point, and the exact same error (inability to munmap) could happen when trying to free this new allocation. (Maybe this isn't possible, however, if the new allocation is either always on the heap or performed atomically via mremap, i.e. if realloc never uses mmap+munmap.) The most robust solution would be to ensure that each allocation is always its own vma by putting guard pages between them, i.e. mmap(size+1page) with PROT_NONE to begin with, then mmap size over top of that with MAP_FIXED. Unfortunately this increases the kernelspace memory usage quite a bit (more vmas) and adds an extra syscall to every allocation (probably an unacceptable performance cost). An alternative solution might be merge the region that munmap fails to free into the free lists managed for non-mmap-serviced allocations.
I guess it would be reasonable to just leave the mapping around in this case Normally the program will error out anyways when this happens, otherwise it should happen rarely. Just it shouldn't assertation failure.
Created attachment 5967 [details] test case
I can't reproduce the failure for realloc (calling realloc to resize a mmap-sized block down to size 1 results in mremap to PAGE_SIZE rather than servicing it from the heap), but I was able to reproduce it with free instead of realloc. Test case is attached.
I checked in a patch.