[Bug malloc/21731] Bad malloc heuristic result in 30-40x memory consumption

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=21731

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos at redhat dot com

--- Comment #11 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Adhemerval Zanella from comment #8)
> The main problem is the example you provided is *leaking* memory and using
> this an a driver to state that glibc is resulting in increase memory
> consumption does not make sense. Either you need to provide an example that
> does not leak memory and thus shows that this is indeed an internal glibc
> issue or fix it in your program.

You can have an application that allocates a cache and does not free the cache.
So from the glibc allocator perspective it is not leaking memory, it's just not
being returned because it could be in use. So asking for a perfect program that
doesn't leak just results in a slightly altered test case that has roughly the
same semantics anyway.

The question that remains is:

Where is the memory going?

* external/internal malloc fragmentation?
* program requirements growing with time?
* an application leak?

You have to answer one of these three questions.

For example see:
https://sourceware.org/glibc/wiki/389-ds-malloc

It is not a trivial analysis.

A leak of ~1.6k/loop shows up when you graph the results as a slop in your RSS
consumption. You need to be able to analyze that graph and take into account
the various changes in your application usage.

DJ Delorie and I have done several of these analyses for larger applications
and we have used the malloc simulator and tooling to produce graphs to help
users understand the application memory usage.

I have my own tooling here:
https://pagure.io/glibc-malloc-trace-utils

Then on dj's dj/malloc branch, we have the trace and simulator code (in malloc/
directory) that lets you capture a malloc workload, and then re-run it outside
of your application environment over and over again to look at the behaviour
against various tunables.

I would start by graphing the exact memory usage your application requests for
each API call and see what that looks like e.g. sum after every call.

Then graph the program growth in VmRSS and VmSize and see how the two lines
relate. The VmRSS growth should be linearly dependent upon the requests, and if
it's not then you have to be able to answer the question about about where the
memory is going.

-- 
You are receiving this mail because:
You are on the CC list for the bug.