Question about madvise(DONTNEED) in glibc malloc

[KOSAKI Motohiro <kosaki dot motohiro at gmail dot com>]

Hi all,

Now, we linux MM folks discuss are discussing about new memory discarding feature.
(https://lkml.org/lkml/2013/3/12/105). The motivation is similar wtih MADV_FREE, 
but more efficient. (http://lwn.net/Articles/230799)

And I played ebizzy benchmark a bit because jemalloc claims jemalloc is faster than glibc
by using it (http://people.freebsd.org/~kris/scaling/ebizzy.html) and the patch auther
claimed vrange patch improves that. And I've found current glibc's MADV_DONTNEED usage
is crazy wrong.

Please look at following result. MADV_DONTNEED makes 5 milion minor page fault and it
decrease transaction performance (record/s) from 73259 to 168333. My machine is typical 
laptop. core i7 4cpu (8 threads) w/ 2G ram. When using larger machine, MADV_DONTNEED decrease
a performance more.


% perf stat ./ebizzy -S 3
16833 records/s
real  3.00 s
user  6.83 s
sys  17.09 s

 Performance counter stats for './ebizzy -S 3':

      23914.067812 task-clock                #    7.941 CPUs utilized          
             2,609 context-switches          #    0.109 K/sec                  
               137 CPU-migrations            #    0.006 K/sec                  
         4,803,074 page-faults               #    0.201 M/sec                  

%   MALLOC_DISCARD_HEAP=0  perf stat ./ebizzy -S 3
73259 records/s
real  3.00 s
user 23.84 s
sys   0.05 s


 Performance counter stats for './ebizzy -S 3':

      23919.162533 task-clock                #    7.945 CPUs utilized          
             2,533 context-switches          #    0.106 K/sec                  
                77 CPU-migrations            #    0.003 K/sec                  
             4,256 page-faults               #    0.178 K/sec                  


More unfortunately, current MADV_DONTNEED usage was introduced following
commit and it doesn't have ChangeLog. So, I couldn't find the discussion
at that time.


commit 0ecb606cb6cf65de1d9fc8a919bceb4be476c602
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Jul 12 18:26:36 2007 +0000

    2.5-18.1


So, a lot of questions was raised in my brain.

- MADV_DONTNEED assume discarded memory is 99.999% reused. but current glibc's assumption is
  clealy opposite. glibc assume it is very light weight when glibc prediction is not correct.
  I have no idea where this mismatch come from.

- HPC folks want an allocator never return memory to OS. They are one of MALLOC_TRIM_THRESHOLD 
  main user. however current MADV_DONTNEED usage don't have disabling knob. I couldn't found
  any reasonable reason.

- Which benchmark were used in glibc malloc developers?


Please tell me historical reason, a pointer of past discussion, the design of current code, 
etc, etc if anyone know.
I hope not to improve only jemalloc, but also glibc malloc. Then, any feedback and information
is welcome.
In this week, we kernel MM folks plan to discuss on Linux MM summit (http://events.linuxfoundation.org/events/lsfmm-summit).
then, quick response is much appreciate even if not accuate.

Thank you.

Attachment: 0001-add-MALLOC_DISCARD_HEAP.patch
Description: Text document