Piecemeal library loading causes slow startup of big apps

[Lorenzo Colitti]

Steve Munroe wrote:
> It looks like the kernel is already doing some (8-32) pages of read-ahead 
> for you. And this is not enough? how big is this library (in megabytes?)

No, it's is not enough: if I cat the library to /dev/null before loading 
it with the dynamic linker, I gain about 0.5-1s. The library is 9MB (as 
reported by ls)

> LD_BIND_NOW will initialize the PLT sooner but will not touch any 
> additional text pages (which is what I think you want).

Yes, that's right. Ideally, in this particular case it would be better 
if all the pages in the library were loaded together at the beginning.

> Most of the time we want to defer loading pages until we know we need 
> them. So replacing mmap with read is not a general solution.

I wonder if this is really true in real-world cases? Of course with very 
general libraries like libc it doesn't make sense to load the whole 
library at startup time, but has anyone measured this? It might be one 
of those counter-intuitive things. :-)

> It seems like you need something more like madvise(POSIX_MADV_SEQUENTIAL) 
> or POSIX_MADV_WILLNEED to get the effect you want. This will probably 
> require some loader (ld.so) changes and some way to indicate this desire 
> in the ELF (special ELF note to ask the loader to call madvise(
> POSIX_MADV_WILLNEED) on the text segment, or an new PT_LOAD flag?)

Do you think it might be worthwile to include such an optimization in 
ld.so? It seems like a good idea to me, but I don't see the big picture...


Cheers,
Lorenzo