Excessive memory consumption when using malloc()

[Patrick McGehearty]

I wonder if a useful heuristic would be for under some
circumstances for the malloc library to call malloc_trim()
every so often, based on passage of time in addition to
the current calls to malloc_trim() based on user calls to free().

If free memory is cached but unused for a 'substantial'
time while the program is doing 'other' things, it gets
freed for other use by the OS. Setting the default to
a time is large enough to avoid excess syscall overhead
while being below typical human perception would be right
for most contexts.

This option could be under tunable control (at least
until we understand better heuristics) with two
variables:
1) A variable that says yes/no we will call malloc_trim()
based on time. (Initial default: NO)
2) A variable that specifies the time frequency.
(Initial default: 1/10 second or reasonable approximation)

This approach avoids the need to have malloc_trim()
or calls to other malloc internals appear in user
applications.

I have not made any investigation of implementation
details. If anyone wants to take this idea and run
with it, they are welcome to do so.

- Patrick McGehearty


On 11/26/2021 12:10 PM, Christian Hoff via Libc-help wrote:
> Hello Adhemerval,
>
> On 11/25/21 9:56 PM, Adhemerval Zanella wrote:
>> What I think we might improve is to maybe add an heuristic to call
>> malloc_trim once a certain level of fragmentation in the main_arena 
>> is found.
>> The question is which metric and threshold to use.  The trimming does 
>> have
>> a cost, however I think it worth to decrease fragmentation and memory 
>> utilization.
>
> Yes, I think this is a very good idea. It is difficult for us to tell
> our customers that our application consumes so much memory even while it
> is not running any computations. Of course, we can call malloc_trim() as
> a workaround, but even this is difficult as this is a Java application,
> so calling system functions is not so straightforward. Any glibc
> improvements in this area would be highly welcome.
>
>
> Best regards,
>
>    Christian
>