Bug 27103

Summary: glibc fails to free memory
Product: glibc Reporter: Hi-Angel <hi-angel>
Component: mallocAssignee: Not yet assigned to anyone <unassigned>
Status: RESOLVED NOTABUG    
Severity: normal CC: carlos
Priority: P2    
Version: 2.32   
Target Milestone: ---   
See Also: https://sourceware.org/bugzilla/show_bug.cgi?id=14581
Host: Target:
Build: Last reconfirmed:

Description Hi-Angel 2020-12-21 22:58:35 UTC
This is suspected to be the reason for this bugreport¹ and was discussed in glibc ML here².

# Steps to reproduce (in terms of terminal commands)

     λ cat test.cpp
    // credits to https://stackoverflow.com/a/48652734/2388257#glibc-application-holding-onto-unused-memory-until-just-before-exit
    #include <cstdlib>
    #include <cstdio>
    #include <unistd.h>

    int main() {
        static const int N = 50000;
        static void *arr[N];

        for (int i = 0; i < N; i++)
            arr[i] = std::malloc(1024);

        // reverse to simplify allocators job
        for (int i = N - 1; i >= 0; i--)
            std::free(arr[i]);
        puts("Measure");
        sleep(1e5);
    }
     λ g++ test.cpp -o a
     λ ./a &
    [1] 647847
    Measure
     λ smem -kc "name pid pss" | grep -P "\ba\b"
    a                        647847    50.4M

## Expected

Memory doesn't take up 50M because these 50M were freed in the point of measure

## Actual

Memory takes up 50M despite that memory have been freed.

# Links

1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45200
2: https://sourceware.org/pipermail/libc-help/2020-December/thread.html#005592
Comment 1 Hi-Angel 2021-01-24 14:35:35 UTC
To add to the list of affected apps: Ruby¹, glib².

And yeah, calling `malloc_trim(0);` next to each `free()` resolves the problem, but it is merely a workaround. Because it is glibc-specific function and one can't reasonably expect every developer of each existing app to be able to write code for each existing libc-implementation. Glibc `free()` should call `malloc_trime()` themselves whenever they deem appropriate.

1: https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#investigating-fragmentation-at-the-memory-allocator-level
2: https://gitlab.gnome.org/GNOME/glib/-/issues/2057
Comment 2 Carlos O'Donell 2021-11-25 18:05:39 UTC
(In reply to Hi-Angel from comment #1)
> To add to the list of affected apps: Ruby¹, glib².
> 
> And yeah, calling `malloc_trim(0);` next to each `free()` resolves the
> problem, but it is merely a workaround. Because it is glibc-specific
> function and one can't reasonably expect every developer of each existing
> app to be able to write code for each existing libc-implementation. Glibc
> `free()` should call `malloc_trime()` themselves whenever they deem
> appropriate.
> 
> 1:
> https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-
> bloat.html#investigating-fragmentation-at-the-memory-allocator-level
> 2: https://gitlab.gnome.org/GNOME/glib/-/issues/2057

The ruby case is a known issue with aligned allocations.
https://sourceware.org/bugzilla/show_bug.cgi?id=14581

We have a patch to improve the ruby case significantly, but it needs more work.

The gnome case is unrelated and is just about freeing back unused pages under memory pressure which is exactly what malloc_trim() will do when called.

This particular issue is not a bug, the allocator is under no requirement to free back the 50MiB to the OS if it believes that the memory will be used again and that performance is improved by caching it in userspace.