Re: [patch] Make the mmap/brk threshold in malloc dynamic to improve performance

[Valerie Henson <val_henson at linux dot intel dot com>]

On Thu, Mar 02, 2006 at 03:03:31PM -0600, Steve Munroe wrote:
> 
> Also you may want to look the existing bugzilla 
> http://sources.redhat.com/bugzilla/show_bug.cgi?id=1541 and see if the 
> problem you found was similar and if the test case attached to the this 
> bugzilla shows similar improvement.

Hey Steve,

This patch was inspired by an application we worked on together, if
you recall that particular customer engagement.  Since we can't say
who it is, or describe their application, I wrote a benchmark that
replicates it that I am currently trying to open source.  Here's a
description of it I sent to l-k:

* Allocate memory
* Write a pattern to it
* Spawn sufficient threads to keep your cpus busy

Each thread does:

* Allocate a little more memory and copy part of memory to it
* Search for a key within its copy
* Free the memory
* Repeat

You can select either all mmap or no mmap behavior, similar to your
test program.

It took me a long time to figure out where the performance hit with
all mmap was mainly coming from - remember we thought it was mmap_sem
contention?  While that has an effect, it is second-order.  The big
win is:

* Not clearing all memory every time we free and reallocate it.

Mmaping to allocate memory is a loss if we are going to free it and
reuse it, because we have to do this clearing unnecessarily.

The real question Arjan and I have is what are the conditions under
which allocating using mmap is a win?  Here's what I can think of:

* When we're running out of brk() space - when does this happen?
* When we don't ever touch most of the allocated memory
* When it will reduce fragmentation in the brk()'d memory

It seems like having larger address spaces is obviating most reasons
for using mmap.

-VAL