[Bug libc/167] New: malloc() eats excess ram

["bluefoxicy at linux dot net" <sourceware-bugzilla at sources dot redhat dot com>]

This isn't really a 'bug', just a restriction due to the nature of the heap. 
The issue is that when using malloc(), memory fragmentation holds the heap in an
overly large state in many cases, which results in greater ram/swap usage and
excess swapping.  This can be fixed on Linux by using mmap() with private,
anonymous, read/write memory rather than brk().

The basic function of malloc() with brk(), to the best of my understanding, is
to grow and shrink the heap and manage the partitioning of the memory within. 
When there is not a large enough chunk of contiguous, free memory on the heap to
satisfy a malloc(), a brk() is called to get enough for the request to be satisfied:

* -- Used ram
- -- unused, but allocated ram
n -- Newly allocated ram in this example
N -- Used, newly allocated ram in this example

[*-**--*] malloc(3) ->
[*-**--*NNN]

This raises one problem that can be manifested in two ways.  The less extreme
problem is seen above:  There is unused, but allocated ram.  The more extreme
problem is best illustrated with a new example.

Let's say that your task is a file manager.  It indexes and displays many files,
sometimes thousands at once.  Let's say that in one particular directory, you
bring up 10,000 files, the rendering of thumbnail previews for which eat a grand
total of 500M of ram.

[*--*-] malloc() ->
[*****NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
 NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
 NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
 NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN]

(not to scale)

Now let's say that after this, you allocate one more piece of ram.  Let's say
that you're using Konqueror or Nautilus, so this could be for a web browser
activity or a new icon display on the desktop, or just one permenant allocation.


[**********************************************************
 **********************************************************
 **********************************************************
 **********************************************************] malloc() ->
[**********************************************************
 **********************************************************
 **********************************************************
 **********************************************************
 N]

Now, let's say that the directory is left.  All that ram is freed.  What's the
result?

[**********************************************************
 **********************************************************
 **********************************************************
 **********************************************************
 *] free() ->


[*--*-*----------------------------------------------------
 ----------------------------------------------------------
 ----------------------------------------------------------
 ----------------------------------------------------------
 *]

In the case of things like Nautilus, you could keep this large chunk of unused
memory allocated for days (nautilus runs as the desktop too), and still only
ever use a fraction of it again.  Even if it gets swapped out, it still uses
space in swap that could be used for other things; becomes a target for the OOM
killer; and causes an unnecessary disk write on swap out, and an unnecessary
disk read on swap in, which could be used for a freeing and a reallocation (many
times faster than a disk seek, unless they trigger a swapping operation, which
is <100%).

This may explain why many applications, especially X applications, eat more ram
than they're supposed to, and always seem to have memory leaks.

I propose that the Linux implimentation of malloc() use instead of brk() calls
to mmap() for anonymous, read-write pages.  This would be functionally the same
as malloc(), except it would not use the heap and thus would not have the above
issues.

A small bit of code I made a while back just to play with mmap() is attatched. 
With a little modification, you can add it to your ld preload list and drop it
in as a replacement for malloc().  It will need to be changed to have the proper
function names (instead of mmalloc() mcalloc() etc) and to set errno on errors.
 I haven't gone over the code in a long time, so it may be a bit wonky.  :)

-- 
           Summary: malloc() eats excess ram
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: gotom at debian dot or dot jp
        ReportedBy: bluefoxicy at linux dot net
                CC: glibc-bugs at sources dot redhat dot com


http://sources.redhat.com/bugzilla/show_bug.cgi?id=167

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.