This is the mail archive of the
mailing list for the glibc project.
Re: Speeding up the dynamic linker with 100s of DSOs?
- From: Andrew Chatham <chatham at google dot com>
- To: michael dot meeks at novell dot com
- Cc: libc-alpha at sources dot redhat dot com
- Date: Thu, 26 Jan 2006 14:43:53 -0800
- Subject: Re: Speeding up the dynamic linker with 100s of DSOs?
- References: <firstname.lastname@example.org>
What I've ended up doing is this:
After the call to _dl_map_object_deps(), when all of the required
objects have been loaded, I build up a table. For every symbol, I
record the earliest position in the global scope where that symbol
occurs. I just store the minimum in table[hash(symbol_name)] rather
than having a real hash table with chaining.
Then when a symbol is being looked up in the global scope, I look it
up in my hash table and skip to that object. So it's O(#symbols)
instead of O(#symbols * #DSOs), speeding up a few of my horrible test
cases by 95%. Even at 5% of the previous time, they're still slow. So
I may try your patches at some point, though I'm having enough trouble
getting this through at the moment.
I just left dlopened objects alone, since we don't use dlopen much. So
their symbols get handled the slow way, but it wouldn't be too hard to
handle them in the same way.
Still waiting on the copyright assignment, so the patch still has to
wait. Also, it looks terrible :)
On 1/25/06, michael meeks <email@example.com> wrote:
> Hi Andrew,
> So - we have the same problem in OO.o - but we have 'only' 150DSOs
> loaded instead of 300 almost all dlopened.
> I have several patches addressing this posted to the binutils list -
> particularly implementing -Bdirect linking (which stores the
> compile-link-time information as to which library a symbol is defined
> in) so that we can look there 1st at run-time. That of course, removes
> 75% or so of the O(num-libraries) lookups from the link.
> Of course - a facile implementation of -Bdirect would have serious
> problems with C++ - because of the vague linkage / ODR issues - so, the
> patch detects which symbols are vague during library compile/link, then
> propagates that information to other libraries, so life is good.
> In addition; by sorting relocations & dynsym & dynstr entries one can
> get another ~25% win, and by storing & comparing pre-computed hash
> values one can get a ~50% win - so, all in all if we could get this
> stuff into general glibcs ;-), performance of large, modularized C++
> apps would get a lot better. [ there are 2 final, more maginal tweaks
> queued pending this getting rationalized too :-].
> -Bdirect: http://sourceware.org/ml/binutils/2005-11/msg00380.html
> sorting: http://sourceware.org/ml/binutils/2006-01/msg00024.html
> hashvals http://sourceware.org/ml/binutils/2006-01/msg00171.html
> The latest patches .suse.ized are at:
> As one blogs about the work, I've had the Troll-tech guys & a random
> ISV play with various features - suffering the same problems. I'd love
> to know what your experience of testing these is - that is if you are ok
> with re-building your glibc :-)
>  - the other 25% of relocations in my case being vague type info.
>  - SL 10.1 should ship with some of this, though it's not there yet.
> firstname.lastname@example.org <><, Pseudo Engineer, itinerant idiot