Re: Amount of virtual Memory larger than heap peak

[Johannes Singler <singler at ira dot uka dot de>]

Hi!

>> My program allocates and deallocates large chunks (Megabytes) of memory
>> many times (using new[], but this should be mapped to malloc).  For
>> example, using memusage.sh, I find out that the heap peak is about
>> 100MB.  However, the virtual memory size of my program has grown to
>> about twice this size, as shown by top.  The resident set size stays at
>> about 100MB, as expected.
> 
> You'll have to see what morecore does when freeing memory but I've
> been told that when malloc wants to create a new malloc arena that it
> moves sbrk.  Free can't move sbrk back down because another malloc
> call may be using the arena.

The chunks we are allocating are 1,5GB in size, so they are
allocated via mmap anyways.  I have double-checked that using strace.

>> I don't use mmap, but read and write large amounts of data from disk
>> using read(int fd, void *buf, size_t count) and the corresponding write.
> 
> You should probably use mmap to allocate and use mallopt to force mmap
> usage.  Therefore you can reclaim memory when you're done with it.

In the end, the problem was that some library set TOP_PAD to 128K.  So
the large chunk + 128K was allocated, and some minor allocations
prevented the block from being freed.  This lead to the memory
fragmentation.  The problem is solved by setting TOP_PAD to 0.

Thanks for your answer!

Johannes