[RFC PATCH] elf: Add opt-in per-object resolution notes
Domen Kožar
domen@dev.si
Thu Nov 5 13:09:42 GMT 2026
Hi Adhemerval,
Thanks for putting this together. Your results look promising, and
avoiding a persistent format and writer synchronization makes this a
compelling direction to investigate.
One integration concern: skipping an absent name before open_verify
also skips la_objsearch. An audit module can redirect a candidate
whose original path does not exist. The l_auditing check excludes
auditor namespaces, but ordinary audited applications would still use
the filter. Should this fast path be disabled while auditing is
active?
It would also help to define visibility of files created during a
single load operation: discarding listings between operations limits
staleness, but a listing can still hide a file added before that
operation finishes.
Comparing this on actual Nix workloads, including small applications
where directory scanning could add overhead, seems the most useful
next step.
Regards,
Domen
On Thu, 05 Nov 2026 09:34:15 -0300, Adhemerval Zanella Netto
<adhemerval.zanella@linaro.org> wrote:
> On 30/10/26 12:44, Domen Kožar wrote:
> > Hello,
> >
> > Programs with long DT_RUNPATH lists can spend many failed file lookups
> > resolving each library at startup. This patch lets glibc use
> > per-object ELF notes containing precomputed library paths to avoid
> > repeated directory searches.
> >
> > Each note records ordered exact-file candidates and directories that
> > still need to be searched at runtime. The loader consults it after
> > LD_LIBRARY_PATH and before the normal DT_RUNPATH search. The feature
> > defaults off and is enabled with glibc.rtld.resolution_cache=1.
>
> Indeed this is know problem on some specific areas. I recall reading about
> the stat storm in HPC area where workloads are usually deployed through
> some network filesystem and during some workload startup thousands of
> dynamic linked binaries would trigger a *lot* of network traffic.
>
> They solved this by using a caching mechanism based on audit modules [1].
> Guix, in the other hand, patched glibc to provide a per-application loader
> cached (ld.so.cache) [2]. I think the Guix issue is similar to the Nix-like
> layout.
>
> There is also Shrinkwrap, which rewrites every transitive DT_NEEDED
> dependency as an absolute path, which has the same effect as caching
> the lookups and avoids the failed attempts on each RUNPATH entry [3].
>
> This approach has some concerns, where a per-object ELF note is a new
> on-disk ABI that glibc would have to support indefinitely, and every
> record in it depends on filesystem state that the loader cannot check:
>
> 1.1 It is a permanent ABI.
>
> The new note adds an ABI constraint that add extra maintenance: binaries
> carrying the note exist for good, and every later glibc has to keep
> parsing them, including any variant that appears before the format
> settles. And it seems that we already have two: this RFC uses owner
> "glibc" with type 1 and rejects the nixpkgs/patchelf predecessor,
> whose identifiers differ.
>
> 1.2 It records negative facts that the loader cannot verify.
>
> The ld.so can check none of the list deployers must guarantee to make the
> note notes consistent:
>
> - skipped files and directories never change, including those where
> no library was found when the note was written;
> - mutable directories stay in the note as runtime searches, in order;
> - the writer preserves hardware-capability selection;
> - access rights and filesystem namespaces match the writer's view;
> - any change to DT_RUNPATH, DT_NEEDED or a recorded path regenerates
> or removes the note.
>
> The prelink support had similar constraints, with cached resolution
> results, per-library timestamps and checksums in .gnu.liblist,
> plus DT_GNU_PRELINKED and DT_CHECKSUM. It added a lot of extra code,
> and it was bitrotten with ABI extensions.
>
> 1.3 It must stay in sync with data that other tools rewrite.
>
> I would require to add patchelf and any other tool that manipulates
> DT_RUNPATH and DT_NEEDED support to read/parse/update this note.
>
> 1.4 It does not cover everything and add extra costs
>
> This RFC states it only for direct startup dependencies in the
> initial namespace, missing dlopen and its dependencies, dlmopen, preload
> and audit modules, LD_LIBRARY_PATH and DT_RPATH. This add inconsistency
> for the dynamic loading semantic.
>
>
> The same problem can be fixed inside the loader at runtime, with no
> format, no writer tool and nothing that can go stale, and that recovers
> nearly all of the gain.
>
> The loader already remembers missing directories (status[] in r_search_path_elem).
> What remains is one failed openat per pair of existing directory and library.
> I have added a very WIP patch that implements it (caveat: I haven't tested
> this extensively and there is no tests):
>
> - It lists each absolute search directory with getdents64 the first
> time a load operation uses it. A load operation is process
> startup or one dlopen call, and its listings are dropped when it
> ends.
>
> - It keeps 32-bit hashes of the entry names (the GNU hash, computed
> once per search) and skips the open when the name is not in the
> set. A hash hit still goes through open, so a collision costs one
> syscall and never changes the result.
>
> - A directory that is missing, or has no glibc-hwcaps entry, settles
> all of its glibc-hwcaps subdirectories without probing them. As
> with today's probes, each directory status is decided once per
> process.
>
> - The listings and a 16 KiB scratch buffer share one 256 KiB
> anonymous mapping per operation, released when the operation ends.
> Only the pages that are touched use memory.
>
> - With the tunable off, nothing is mapped and only one static pointer
> is added. No malloc is used, since interposed malloc may call
> dlopen.
>
> The results seems promising, I have crated a benchmarks trying to mimic
> how Guix/Niox folder setup and evaluate binaries with 10/50/100 runpaths.
>
> The first value is your proposal:
>
> $ ./repro.py /home/azanella/Projects/glibc/build/x86_64-linux-gnu-study
> build: /home/azanella/Projects/glibc/build/x86_64-linux-gnu-study
> glibc.rtld.dircache: no, glibc.rtld.resolution_cache: yes
> N=10
> variant median us min us failed openat
> baseline 440.2 426.4 77
> note 367.0 364.6 2
> absolute 364.6 352.9 2
> N=50
> variant median us min us failed openat
> baseline 2206.1 2191.1 1377
> note 1082.4 1054.4 2
> absolute 1029.8 1021.7 2
> N=100
> variant median us min us failed openat
> baseline 6341.6 6317.0 5252
> note 2100.5 2093.3 2
> absolute 2009.0 2002.7 2
>
> And the following is the proposed approach:
>
> $ ./repro.py /home/azanella/Projects/glibc/build/x86_64-linux-gnu-study-alternate
> build: /home/azanella/Projects/glibc/build/x86_64-linux-gnu-study-alternate
> glibc.rtld.dircache: yes, glibc.rtld.resolution_cache: no
> N=10
> variant median us min us failed openat
> baseline 425.3 420.8 77
> dircache 378.8 370.1 0
> absolute 360.3 347.4 2
> N=50
> variant median us min us failed openat
> baseline 2185.2 2175.2 1377
> dircache 1102.6 1096.3 0
> absolute 1023.3 1022.1 2
> N=100
> variant median us min us failed openat
> baseline 6330.6 6285.4 5252
> dircache 2165.1 2157.8 0
> absolute 2011.2 2001.9 2
>
>
> There still some open issues, where it does not handle case-insentive; and
> the directories that do not fit the scratch buffer are ignored.
>
> Also, I am still checking on the GUIX since it seem the fastest strategy because
> it basically simulates what Shrinkwrap does through the per-application cache.
> But it adds the per-application cache, which I am not fully sure about security
> application and it adds the requirement of the extra deployment step.
> It also changes the documented search order (cache before DT_RUNPATH).
>
> >
> > The cache is safe only while it selects the same libraries as a normal
> > search. Files and directories whose searches are skipped must remain
> > unchanged, including paths where no library was found when the note
> > was written. Mutable directories must remain in the note as runtime
> > searches, in their original order. The writer must also preserve
> > hardware-capability selection and account for differences in access
> > rights and filesystem namespaces. Changing DT_RUNPATH, DT_NEEDED, or
> > recorded paths requires regenerating or removing the note; the loader
> > cannot detect stale records.
> >
> > The implementation covers direct startup dependencies with DT_RUNPATH
> > in the initial namespace. It ignores notes during secure execution,
> > auditing, dlopen dependency searches, preload/audit module searches,
> > and legacy RPATH searches. Missing or rejected notes, or exhausted
> > candidates, fall back to ordinary library search.
> >
> > The RFC proposes a glibc-owned note format, with no
> > distribution-specific paths or policy. Its note identifiers are
> > provisional. Existing patchelf --build-resolution-cache output uses
> > different identifiers and would need a matching writer update.
> >
> > Questions for review:
> > 1. Are these restrictions sufficient for an opt-in cache, or should
> > the loader detect stale records before using them?
>
> IMHO no. For upstream, is not enough (1.2). And I think we can the
> runpath/rpath caching without the need of the extra note with the
> extra cost of some syscalls.
>
> > 2. Should the note bind its records to the object's DT_RUNPATH and
> > DT_NEEDED values from the outset?
> > 3. Would extending ld.so.cache be preferable, and how should it
> > preserve each requesting object's RUNPATH search order?
>
> The ld.so.cache is a global soname -> path map and it has to preserve
> each requesting object's RUNPATH order. It becomes the per-object note
> stored *out of line* (with an ldconfig regeneration lifecycle on top).
>
> I don't think either Guix or this proposal is preferable to fixing the
> search itself.
>
> >
> > This is adapted from Nixpkgs PR #535735 and pennae's earlier per-DSO
> > work; the references below include the existing patchelf writer.
> >
> > Provenance: Codex (GPT-6) generated substantial parts of the
> > adaptation and tests. The patch retains attribution and Assisted-by
> > trailers; no DCO signoff is asserted.
> >
> > References:
> > https://github.com/NixOS/nixpkgs/pull/535735
> > https://github.com/NixOS/nixpkgs/pull/207893
> > https://github.com/NixOS/patchelf/pull/647
> >
> > Regards,
> > Domen
>
> [1] https://github.com/llnl/Spindle
> [2] https://guix.gnu.org/sr/blog/2021/taming-the-stat-storm-with-a-loader-cache/
> [3] https://github.com/fzakaria/shrinkwrap
More information about the Libc-alpha
mailing list