This is the mail archive of the mailing list for the glibc project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Performance issues with glibc stdio and multithreading on SMP machines

glibc stdio default buffer allocation uses ALLOC_BUF() macro defined in libio/libioP.h, with a size of 4096 bytes (on linux at least).

This macro uses a mmap() call : mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)

At fclose() time, the allocated buffer is then freed with munmap() call.

typicall strace call for { fopen(); fread(...);fclose(..);} is

open("/path_to_file", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=70001, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d46000
read(3, 0xb7d46000, 4096) = 4096
close(3) = 0
munmap(0xb7d46000, 4096) = 0

It is a *big* performance problem for multithreaded apps because mmap()/munmap() calls have scalability issues.
These calls must lock the list of vmas of the process, and invalidate TLB (using expensive inter processor messages in SMP machines). Also, the first access to the newly mmaped() buffer done at read() time triggers a page fault, that might take a spinlock or rwlock in kernel to resolve the fault.

User apps can solve this by using setvbuf() call after fopen(), giving a buffer that is allocated by a normal malloc() call. But the buffer must be freed at fclose() time too.

Could glibc use :

- ordinary malloced buffers
- or a (small) cache of buffers (to avoid munmap() and then a further mmap())

Thank you
Eric Dumazet

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]