Re: Speeding up the dynamic linker with 100s of DSOs?

[Andrew Chatham <chatham at google dot com>]

On 1/6/06, Roland McGrath <roland@redhat.com> wrote:
> > I'm going to try to fix this for us somehow and release a patch,
> > whether it goes into the official glibc or not. Perhaps my efforts
> > would be better directed towards speeding up prelink. I haven't
> > actually looked at the implementation yet.
>
> What we'll encourage is whatever efforts help to optimize cases that affect
> a lot of users.  Linking something and then running it only once, is common
> for you and me, but not common among dynamic linker invocations overall.
> If you find improvements to the dynamic linker that improve your case
> without hurting others, nor (further) overly complicating the code nor
> other downsides, we will of course be receptive.  There are certainly other
> cases of programs that load way too many DSOs, and some of those are with
> dlopen and thus much harder to help with prelink.  But algorithm
> improvements or tunings need to be beneficial across the board, or at least
> beneficial to worthwhile cases while not harming other common cases.

Fair enough, though it seems similar LD_BIND_NOW, perhaps a bit more
complicated. My current patch is a 50 lines touching the dynamic
linker and 150 to implement the bloom filters. I'll just continue on
my own and put the patch on code.google.com when I can.

A broader takeaway is that the code for searching the ELF hash tables
is pretty slow, if just putting a bloom filter in front of it sped
things up so much. This affects everyone using DSOs, though perhaps
not enough to notice. Not sure there's much room to fix it, though,
since the hash table is part of the ELF spec and looks pretty
reasonable. As I said before, Michael had good luck just making the
hash tables bigger.

Andrew