hwcaps subdirectory selection in the dynamic loader

Florian Weimer fweimer@redhat.com
Thu May 28 10:54:25 GMT 2020


* Adhemerval Zanella via Libc-alpha:

>> The implementation largely happens via elf/dl-hwcaps.c, dl-procinfo.h,
>> elf/dl-load.c, and elf/ldconfig.c, elf/dl-cache.c for ld.so.cache.  On
>> typical targets, the kernel provides hardware capability bits via
>> AT_HWCAP auxiliary vector entry, and a platform string AT_PLATFORM.
>
> By typical do you mean all but x86 that uses a different mechanism
> (cpuid through sysdeps/x86/cpu-features.c)?

Yes, indeed.

> So it seems that currently we have x86, i686, powerpc32, powerpc64,
> powerpc64le, sparc32, sparc64, s390, s390x, and aarch64 that use the
> hwcap subdirectories, right?

I think 32-bit Arm uses it too:

sysdeps/unix/sysv/linux/arm/dl-procinfo.h:#define HWCAP_IMPORTANT		(HWCAP_ARM_VFP | HWCAP_ARM_NEON)

>> This is no sysdeps override for this search path construction.  An
>> architecture can only affect how the hwcap bits are computed, to which
>> strings individal bits correspond, and what the platform subdirectory is
>> called.  The fake two bits (TLS and platform) and the power-set
>> construction always apply.
>
> First question is whether still does make sense to provide the fake TLS bit
> to add "tls" in patch construction.

I don't think it does.  We really should deprecate it.

>> # Discussion
>> 
>> I think there a couple of problems with this approach.  One subtle
>> problem involves the AT_PLATFORM encoding in the cache file (bug 25938).
>> But I think there are other issues.
>> 
>> The LD_LIBRARY_PATH/non-cache case is rather wasteful in terms of system
>> calls, even with the blacklisting in place.
>
> Indeed, one option might be to use a different scheme than nesting string
> capabilities by appending them.  So instead of:
>
>    zEC12/dfp/eimm/ldisp/zarch
>    zEC12/dfp/eimm/ldisp
>    zEC12/dfp/eimm/zarch
>    zEC12/dfp/eimm
>    zEC12/dfp/ldisp/zarch
>
> we might have
>
>    zEC12-dfp-eimm-ldisp-zarch
>    zEC12-dfp-eimm-ldisp
>    zEC12-dfp-eimm-zarch
>    zEC12-dfp-eimm
>    zEC12-dfp-ldisp-zarch
>
> And a thus a openat plus a getdents might be less costly than the multiple
> opens loaders issues today. It should scale better also with new inclusions
> or permutations.

It's hard to tell what's better for the file system.  The intermediate
subdirectory could swing things in favor of getdents64.  But listing
everything that is on LD_LIBRARY_PATH strikes me as problematic because
we don't know how large those directories are.  Performance
characteristics will change again if the kernel ever implements negative
dentries.

However, I do not think these bitmask-based hwcaps are actually useful
to programmers.  It's simply too much choice.

>> The heuristics for choosing the implementation is not very obvious.  Of
>> course, with bitmasks of opaque CPU features, there is no generic
>> winner.  For example, on s390x z13, a library in a subdirectory
>> ldisp/zarch would be preferred over one in vx because the former has
>> more matching hwcap bits and comes earlier in the ld.so.cache sort order
>> (but not the LD_LIBRARY_PATH order).  This is counter-intuitive because
>> vx (the z13 vector capability) should imply the other capabilities—the
>> library was just placed into the wrong directory.
>
> It was not clear no me why exactly the heuristics used in cached lookups
> differs from non-cached way.

It's readdir vs path probing.  There's considerable more flexibility
with readdir in terms of directory layout.

>> The complexity of the trade-offs here suggests to me that we (the GNU
>> toolchain as a whole) should try to pre-define names for collections of
>> hwcap flags, so that we can get a monotonic progression of features
>> under a clearly defined name.  This will allow programmers to optimize
>> for subsequent microarchitecture revisions.  So instead of "x86_64" we
>> would have pseudo-capabilities like "x86-200", "x86-201", "x86-202" and
>> so on, more or less mirroring the "zEC12", "z13" &c platform directories
>> on s390x, even though the kernel does not provide such platform names on
>> x86-64.  Even on platforms that provide an AT_PLATFORM name, in most
>> cases, it would make sense to use *earlier* platform names as a fallback
>> (so that z15 system would also use z14- and z13-optimized libraries if
>> available).  This would mean that the dynamic loader would need to know
>> more about these relationships.
>
> I agree with you, my understanding is the current scheme try to free the
> platform maintainer to pre-define such list (where itself has its own
> drawbacks as the AMD/Intel selection one is showing us).

Yes, and the selection approach we have on x86 uses hwcap in this way
not because hwcap is a good fit to the architecture requirements, but
because it's the only mechanism we've got.

I suspect s390x is similar in that regard.

>> The way we store hwcap bits in ld.so.cache is also not ideal.  It would
>> be nice if ldconfig could be hwcap-agnostic, not having to care at all
>> of the correspondence between subdirectory name and hwcap bit (or
>> AT_PLATFORM pseudo-hwcap bit).  I think I have a way to encode that
>> while still maintaining ld.so.cache backwards compatibility (basically,
>> set the currently unused bit 62 on those new hwcap entries, so that
>> older loaders ignore them because of a missed hwcap requirement).
>
> From previous discussion I take the current searched glibc defined
> path are not part of the ABI so we are free to tune in future
> releases.

I'm not so sure.  We should probably make changes incrementally.  Add a
new mechanism, preferred over the existing one, wait a bit, and then
remove the old mechanism.

>> If we put new hwcap subdirectories under a *single* subdirectory (say
>> "glibc-hwcaps"), then we could prune paths more aggressively, and use
>> the new scheme in parallel to the old without much impact on performance
>> until these subdirectories are actually used.  ldconfig could also treat
>> the presence of a glibc-hwcaps subdirectory has an instruction to
>> descend into each subdirectory of the glibc-hwcaps directory, but not
>> further, and store the names of those subdirectories in ld.so.cache, so
>> that the loader can match them at run time.
>> 
>> In any case, I do not see a way to make good progress on bug 23249 (the
>> "haswell" platform subdirectory issue on various x86-64 variants)
>> without tackling some of these isssues.
>> 
>> Thoughts?
>
> Agreed, what I think what we need to do is move the logic of providing
> the paths from generic to arch-specific bits.  One thing that was clear
> is not every hwcap bits represents a meaningful ABI variant where developers
> will actively provided optimized builds.

Right, that's currently expressed as HWCAP_IMPORTANT.

> So one option I see is to create a architecture hook similar to 
> _dl_hwcap_string where loader provides the AT_PLATFORM, the hwcaps,
> and any other meaningful information and architecture specific code
> creates the desirable search list (which might either use a pre-defined
> database or compute based on hwcap bits).

Yes, and ldconfig puts the strings into the cache (not bits or indices).

> Final question is whether will use still continue to use current scheme
> in parallel or if the idea it to eventually phase this out.

I think we'll have both in parallel for a while.  We can use an unused
hwcap bit (like bit 62) in the cache file entries to make old glibc
ignore the new entries, and add some secondary data later in the file
which is ignored by current ld.so.

Thanks,
Florian



More information about the Libc-alpha mailing list