Re: [PATCH][RFC] Allow explicit shrinking of arena heaps using anenvironment variable

[Siddhesh Poyarekar <siddhesh at redhat dot com>]

On Sun, 5 Aug 2012 06:54:56 -0400, KOSAKI wrote:
> I don't have any knowledge of hurd. then I'd like to explain Linux
> behavior. On Linux, both MADV_DONTNEED and mprotect don't decommit
> virtual memory charge. Only munmap (included implicit munmap by
> MAP_FIXED) makes decommit.

Sorry I took so long to reply. I did a few tests today based around a
little test program[1], which I started out writing to check the times
taken by each of the options regardless of the implications of commit
charge:

1) madvise
2) mmap (MAP_FIXED)
3) madvise + mprotect

The program performs 1024 iterations of mapping 1GB, writing into it
and then 'dropping' it using one of the above methods, thus ending up
with a mapping of 1TB. To compile:

For mmap (MAP_FIXED):
gcc -o bench -D_USE_MAP_FIXED bench.c

For madvise:
gcc -o bench -D_USE_MADVISE bench.c

For mprotect:
gcc -o bench -D_USE_MPROTECT bench.c

For madvise+mprotect:
gcc -o bench -D_USE_MADVISE -D_USE_MPROTECT bench.c

At /proc/sys/vm/overcommit_memory set to 0 the commit charge does not
matter directly, so I went ahead and did the performance tests at this
level. The average and total time taken for all the 'drops' are below.
To conclude, I don't see much in the way of performance gain to glean
out of doing an madvise as against MAP_FIXED, let alone
madvise+mprotect.

The other factor in favour of MAP_FIXED is what KOSAKI mentioned, and I
can now back up experimentally by running the program with
overcommit_memory set to 2 and overcommit_ratio set to 50, where commit
charges make a direct difference. Here, only memory that is either
originally mmapped as PROT_NONE or is modified to PROT_NONE by using
mmap (MAP_FIXED) has any impact on the commit charge - it is not
directly proportional though, i.e. all of it does not get subtracted
from the commit charge from what I can tell. This is evident from the
fact that the above program runs for up to 70+ iterations with
MAP_FIXED and not all the way as I would have expected theoretically.
Other methods fare worse, only allowing 3 iterations.

In retrospect, I can relate this to several reports of OOM kills due to
high address space usage that I had come across on RHEL-6 (which has
arenas-per-thread, as opposed to RHEL-5, which doesn't). We would 'fix'
those problems by using MALLOC_ARENA_MAX to lower the number of arenas,
but I think using MAP_FIXED unconditionally should be a better idea.

Given this, I propose dropping the usage of madvise within the
shrink_heap altogether so that unused parts of the arenas do not add up
to the commit charge of a program. There isn't much in the way of
performance that we're gaining here (25 usec avg.) and it is useless
when overcommit is turned off or controlled.

I have not attached a patch for this yet, but I will if the argument
above makes sense for all -- it is quite a straightforward change as
far as the code is concerned and I've got the testsuite running
already to ensure that it doesn't break any of the current test cases.

Regards,
Siddhesh

Environment:

* Intel Core i5 (2 hyperthreaded cores == 4 visible)
* 4GB RAM and 5GB swap
* Using Fedora 16 x86_64: kernel-3.4.6-1.fc16.x86_64
* Load: Three browser windows (firefox), xfce desktop and terminal
  windows

Average times:

Using madvise only:
================================================================================
Total time elapsed: 6098515.000000 usec
Average time per iteration: 5955.581055 usec

Using madvise and mprotect:
================================================================================
Total time elapsed: 6103235.000000 usec
Average time per iteration: 5960.190430 usec

Using MAP_FIXED:
================================================================================
Total time elapsed: 6122586.000000 usec
Average time per iteration: 5979.087891 usec

[1] The program:

#include <sys/mman.h>
#include <assert.h>
#include <stdio.h>
#include <string.h>
#include <time.h>

#define SIZE 1024 * 1024 * 1024UL
#define ITERS 1024

int
main()
{
  int i = ITERS;
  double total = 0.0;
  while (i--)
    {
      struct timespec before, after;
      double elapsed;
      void *m = mmap (NULL, SIZE, PROT_READ|PROT_WRITE,
		      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

      if (m == MAP_FAILED)
	{
	  perror ("mmap failed");
	  return 1;
	}

      memset (m, 42, SIZE);

#ifdef _USE_MAP_FIXED
      clock_gettime (CLOCK_MONOTONIC_RAW, &before);
      m = mmap (m, SIZE, PROT_NONE,
		MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE,
		-1, 0);
      clock_gettime (CLOCK_MONOTONIC_RAW, &after);

      if (m == MAP_FAILED)
	{
	  perror ("mmap failed");
	  return 1;
	}
#else
      clock_gettime (CLOCK_MONOTONIC_RAW, &before);
# if defined(_USE_MADVISE) || defined(_USE_MPROTECT)
#  ifdef _USE_MADVISE
      madvise (m, SIZE, MADV_DONTNEED);
#  endif
#  ifdef _USE_MPROTECT
      mprotect (m, SIZE, PROT_NONE);
#  endif
# else
#  error "Define either _USE_MAP_FIXED or _USE_MPROTECT or _USE_MADVISE"
# endif
      clock_gettime (CLOCK_MONOTONIC_RAW, &after);
#endif
      elapsed = (after.tv_sec - before.tv_sec) * 1000000;
      elapsed += (after.tv_nsec - before.tv_nsec) / 1000;
      total += elapsed;
      printf ("%d:\t%lf\n", ITERS - i, elapsed);
      fflush (stdout);
    }

  for (i = 0; i< 80; i++)
    putchar ('=');
  putchar ('\n');
  printf("Total time elapsed: %lf usec\n", total);
  printf("Average time per iteration: %lf usec\n", total / ITERS);
  fflush (stdout);
}