Possible malloc/mmap bug?

[Matthew Chapman]

It should be possible to increase the number of mappings allowed per
process via /proc/sys/vm/max_map_count.  This would probably solve the
immediate problem, although I'd be somewhat concerned about the impact
on the performance of page faults, subsequent mmaps, etc. - remember
these mappings are stored in a red-black tree in the kernel.

Matt


On Sat, Nov 19, 2011 at 8:03 AM, Carlos O'Donell
<carlos@systemhalted.org> wrote:
> On Fri, Nov 18, 2011 at 3:33 PM, Charles Hines <hines@cert.org> wrote:
>>> Your application is a pathological case of VM fragmentation.
>> But in a 64 bit operating system w/ the amount of virtual address space in the Exabyte range, and the fact that while we are processing large amounts of data (pre-reduction) and us preparing batches to feed to the program that won't exceed (pre-reduction) the system RAM (48GB), it's not an issue here.
>
> It's the same problem, but just takes longer to reach the limit of the
> kernel VM fragmentation.
>
> ~~~
> ...
> -----
>>>>>>
>>>>>> ERROR: malloc returned NULL at call number 65569 <<<<<
>>>>>>
> -----
> total allocated: 32.27 GB, total allocated after reallocs: 0.25 GB
> [elapsed time: 32 sec]
> malloc_stats() data:
> Arena 0:
> system bytes     =     663552
> in use bytes     =     135696
> Total (incl. mmap):
> system bytes     =  549068800
> in use bytes     =  548540944
> max mmap regions =      65536
> max mmap bytes   =  548405248
> -----
> ~~~
> 1321649768.153047 mmap(NULL, 532480, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f113ffe3000
> 1321649768.153132 mremap(0x7f113ffe3000, 532480, 8192, MREMAP_MAYMOVE)
> = -1 ENOMEM (Cannot allocate memory)
> ...
> 1321649768.153170 brk(0)                = 0x19f0000
> 1321649768.153197 brk(0x1a92000)        = 0x1a92000
> 1321649768.153415 brk(0x1a12000)        = 0x1a92000
> ...
> 1321649768.155301 mmap(NULL, 1048576, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> 1321649768.155323 mmap(NULL, 134217728, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot
> allocate memory)
> 1321649768.155343 mmap(NULL, 67108864, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot
> allocate memory)
> 1321649768.155362 mmap(NULL, 134217728, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot
> allocate memory)
> 1321649768.155382 mmap(NULL, 67108864, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot
> allocate memory)
> 1321649768.155410 write(1, ">>>>>\n", 6) = 6
> 1321649768.155439 write(1, ">>>>> ERROR: malloc returned NUL"..., 61) = 61
> 1321649768.155461 write(1, ">>>>>\n", 6) = 6
> 1321649768.155480 write(1, "-----\n", 6) = 6
> ...
> ~~~
> * mremap fails to find a new mapping for the pages.
> * glibc starts trying to use kernel brk call to get new memory but
> that fails too.
> * final set of mmap allocations fails and glibc gives up.
>
> Fix your application or fix the kernel.
>
> Cheers,
> Carlos.
>