Excessive memory consumption when using malloc()

[Konstantin Kharlamov]

On Thu, 2021-11-25 at 20:46 +0300, Konstantin Kharlamov via Libc-help wrote:
> On Thu, 2021-11-25 at 18:20 +0100, Christian Hoff via Libc-help wrote:
> > Hello all,
> > 
> > we are facing the a problem with the memory allocator in glibc 2.17 on
> > RHEL 7.9. Or application allocates about 10 GB of memory (split into
> > chunks that are each around 512 KB large). This memory is used for some
> > computations and released afterwards. After a while, the application is
> > running the same computations again, but this time in different threads.
> > The first issue we are seeing is that - after the computations are done
> > - the 10 GB of memory is not released back to the operating system. Only
> > after calling malloc_trim() manually with GDB, the size of the process
> > shrinks dramatically from ~10GB to 400 MB. So, at this point, the unused
> > memory from the computations is finally returned to the OS.
> > 
> > Our wish would be that the memory is returned to the OS without us
> > having to call malloc_trim(). And I understand that glibc also trims the
> > heap when there is sufficient free space in top of it (the
> > M_TRIM_THRESHOLD in mallopt() controls when this should happen). What
> > could be the reason why this is not working in our case? Could it be
> > related to heap fragmentation? But assuming that is the reason, why is
> > malloc_trim() nevertheless able to free this memory?
> > 
> 
> I assume the bug you stumbled upon is this one
> https://sourceware.org/bugzilla/show_bug.cgi?id=27103
> 
> I'm not sure there is anything more to say except that it is a long-standing
> glibc bug (it seems to have been known long before I reported it in 2020), and
> malloc_trim is the official workaround to it.
> 
> For you purposes though you could perhaps try other malloc implementations such
> as jemalloc. Try and see if it fixes these problems. Please report back if you
> try it, I am curious if that can be used as another workaround.

Hah, FYI, the bug was just closed by Glibc dev with the following, I'm quoting:

> the allocator is under no requirement to
free back the 50MiB to the OS if it believes that the memory will be used again
and that performance is improved by caching it in userspace

So there you go, you 10G of unreleased memory is a Glibc feature, no complaints
;-P