[Bug malloc/21731] Bad malloc heuristic result in 30-40x memory consumption

["adhemerval.zanella at linaro dot org" <sourceware-bugzilla at sourceware dot org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=21731

--- Comment #12 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
(In reply to Yichao Yu from comment #10)
> As I said in the very first post, the program does have a small "leak".
> However, I think it is reasonable to expect that if the program allocate
> <100kB per loop, the peak memory consumption (measured by overall system
> memory usage and also the maxrss of the process) should not grow 10x or 100x
> faster than this "leak". It is perfectly valid in a program to accumulate
> small allocations overtime until they are free'd later. Being able to return
> to the same memory consumption (measured in whatever way) is not the only
> measure of a bug-free memory manager, the ratio between the peak memory
> usage and the memory actually needed is too.

Again your example is a bad one to try show any point. As from my comment #6
RSS at end of execution is roughly 61MB and valgrind is showing a memory leak
of 54MB. This is hardly '10x' increase and it shows more an issue of the
workload/example than on GLIBC.

I think you might not be clear here and using 'memory consumption' as virtual
memory increase. If you check VMSize from /proc/<pid>/status it does increase
over time:

VmSize:    81920 kB
VmSize:   107704 kB
[...]
VmSize:  1164740 kB
VmSize:  1190520 kB

But this does not actually count as actual memory consumption (RSS) and this is
indeed an issue for your program (since it does seems to generate a lot of
memory fragmentation due the leaks). One option could use the MALLOC_TOP_PAD_
(environment variable) / M_TOP_PAD (mallopt).  This will avoid the arena
fragmentation by forcing mmap allocations over sbrk:

# this is greping VmSize instead
$ LD_PRELOAD=./libintercept.so MALLOC_TOP_PAD_=0 ./malloc
VmSize:    81376 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
VmSize:   107160 kB
[...]
VmSize:   107160 kB

GLIBC memory allocation will try to coalesce memory on newer allocations, but
fragmentation might impede and thus why the increase of virtual memory
allocation (some information on the algorithm used can be found at section
'Malloc Algorithm' [1]).

For the workload in question, are these 'leak' intentional? If it the case it
could be a better strategy to create a different allocation strategy, for
instance a external memory poll.

[1] https://sourceware.org/glibc/wiki/MallocInternals

-- 
You are receiving this mail because:
You are on the CC list for the bug.