Performance issues with glibc stdio and multithreading on SMP machines

[Eric Dumazet <dada1 at cosmosbay dot com>]

glibc stdio default buffer allocation uses ALLOC_BUF() macro defined in 
libio/libioP.h, with a size of 4096 bytes (on linux at least).

This macro uses a mmap() call :  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)

At fclose() time, the allocated buffer is then freed with munmap() call.

typicall strace call for { fopen(); fread(...);fclose(..);} is

 open("/path_to_file", O_RDONLY) = 3
 fstat64(3, {st_mode=S_IFREG|0644, st_size=70001, ...}) = 0
 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7d46000
 read(3, 0xb7d46000, 4096) = 4096
...
 close(3)                          = 0
 munmap(0xb7d46000, 4096)          = 0


It is a *big* performance problem for multithreaded apps because 
mmap()/munmap() calls have scalability issues.
These calls must lock the list of vmas of the process, and invalidate 
TLB (using expensive inter processor messages in SMP machines). Also, 
the first access to the newly mmaped() buffer done at read() time 
triggers a page fault, that might take a spinlock or rwlock in kernel to 
resolve the fault.

User apps can solve this by using setvbuf() call after fopen(), giving a 
buffer that is allocated by a normal malloc() call. But the buffer must 
be freed at fclose() time too.

Could glibc use :

- ordinary malloced buffers
- or a (small) cache of buffers (to avoid munmap() and then a further 
mmap())

Thank you
Eric Dumazet