[PATCH v3 1/1] manual: document audit interface

[Adhemerval Zanella Netto]

On 13/10/26 13:35, DJ Delorie wrote:
> 
> This is my attempt at initial documentation for the la_* audit
> interface, based on reading the code...
> ---
> Changes in v3:
>   - expanded intro to loading
>   - clarify that la_activity happens before la_objopen
>   - clarify need for _GNU_SOURCE wrt link.h
>   - clarify that DT_AUDIT and DT_DEPAUDIT are for the main object
>     only.
>   - Fix audit source order.
>   - clarify that secure auditors aren't loaded via ld.so.cache
>   - clarify when la_version() causes an error message.
>   - typos in LA_ACT_*
>   - clarify how la_objsearch searches can be cancelled.
>   - clarify la_objopen is called for main/ld.so/vdso objects.
>   - expand la_symbind prototypes
>   - clarify how LA_FLG_BIND* works
>   - expand pltenter/pltexit info
> 
> Changes in v2:
>   - fixed typos.
>   - elaborated on what a cookie is
>   - elaborated on what "isolated" means for auditors
>   - replaced "cookie" with "private data" and rewrote the intro for it
>   - various clarifications based on review feedback.
>   - mention that auditors are not "I can do anything" hooks
>   - Explain DT_DEPAUDIT better
>   - explain how la_version checks work
>   - mention performance issues with PLT auditing
>   - mention non-mentioning of decoding arguments

Hi DJ,

This patch looks good in general, it addresses most of my v2 comments, and the manual
now builds (info and html; check-deftype.sh and check-safety.sh pass). There are a 
few new issues and some open points from v2 below.

You may also want to check some recent fixed I sent for the case of multiple auditor
(and you are right that this case is buggy) [1].

[1] https://patchwork.sourceware.org/project/glibc/list/?series=67128

> 
>  manual/dynlink.texi | 381 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 381 insertions(+)
> 
> diff --git a/manual/dynlink.texi b/manual/dynlink.texi
> index ad4da753a5..e7e5d6c966 100644
> --- a/manual/dynlink.texi
> +++ b/manual/dynlink.texi
> @@ -17,6 +17,7 @@ Dynamic linkers are sometimes called @dfn{dynamic loaders}.
>  * Dynamic Linker Environment Variables:: Environment variables that control the
>                                            dynamic linker.
>  * Dynamic Linker Introspection::    Interfaces for querying mapping information.
> +* Dynamic Linker Auditing::	Auditing interface.
>  * Indirect Functions::          Selecting function implementations at load time.
>  * Dynamic Linker Hardening::    Avoiding unexpected issues with dynamic linking.
>  @end menu
> @@ -736,6 +737,386 @@ A bit mask used to signal that the object contains SFrame data.  See
>  
>  @end table
>  
> +@node Dynamic Linker Auditing
> +@section Dynamic Linker Auditing
> +@cindex Auditing
> +@cindex LD_AUDIT
> +
> +The dynamic linker's auditing interface allows external code to
> +observe and affect the loading of dynamic objects.  As this allows the
> +external code to modify the operation of the program, great care
> +should be taken when creating and deploying audit modules (auditors).
> +
> +Audit modules should include @code{<link.h>} to import constants and
> +prototypes used by the audit API.  Note that the audit interface is an
> +extention to @theglibc{} so requires _GNU_SOURCE to be defined before
> +including @code{<link.h>}.

s/extention/extension/, and @code{_GNU_SOURCE}.  The manual uses
@file{link.h} for headers (this also applies to @samp{<link.h>} in
la_version).

> +
> +Note that while auditors can do pretty much anything to the audited
> +program, anything other than the monitoring documented here is not
> +supported.

The section also documents redirection (the la_objsearch, la_symbind
and la_pltenter return values), so "anything other than the uses
documented here" would be more accurate.

> +
> +@menu
> +* Audit Module Loading::  How modules are selected and loaded.
> +* Audit API Callbacks::	  Audit functions called by the dynamic linker.
> +@end menu
> +
> +@node Audit Module Loading
> +@subsection Audit Module Loading
> +
> +An audit module (auditor) is a shared object (plus its DT_NEEDED
> +dependencies) that is loaded very early in the dynamic loading process
> +in its own link namespace. An auditor can load a set of support
> +libraries into its link namespace and the symbols that these libraries
> +define will be invisible to the main executable and its
> +dependencies. For example, each auditor may load its own libstdc++,
> +which may be incompatible with those used by the main executable
> +and/or other auditors, and will have its own copy of glibc itself
> +(separate malloc, heap, stdio, etc). Despite the fact that an auditor

s/etc)/etc.)/.  It may also be worth saying that the dynamic linker
itself is not duplicated: there is a single ld.so for all namespaces.

> +in its own link namespace is somewhat isolated from the main
> +executable and other auditors, auditors and the main executable share
> +process state, including Linux namespaces (distinct from link
> +namespaces), process id, and file descriptors.
> +
> +Once an auditor is loaded, the dynamic loader will call functions in
> +the auditor to report key actions in the load process — associated

The manual uses "---" rather than a literal em dash.

> +with itself, other auditors, the executable, and any shared objects.
> +Example actions include opening a shared object, binding a symbol, or
> +completion of changes to the link map. While an auditor receives
> +notifications of all dynamic linker actions on behalf of the main
> +executable and its shared objects, as explained later, an auditor
> +only receives limited notifications about its own actions or those of
> +other auditors.
> +
> +Auditors are specified by four methods, each of which specifies a
> +colon-separated list of auditors:
> +
> +@table @code
> +
> +@item LD_AUDIT
> +The @code{LD_AUDIT} environment variable.
> +
> +@item --audit
> +If the dynamic linker is invoked manually, it allows an @code{--audit}
> +command line parameter followed by a list of auditors.
> +
> +@item DT_AUDIT
> +If the main shared object has a DT_AUDIT dynamic tag, that tag
> +specifies a list of auditors.
> +
> +@item DT_DEPAUDIT
> +If the main shared object has a DT_DEPAUDIT dynamic tag, that tag is a

s/main shared object/main program/ in both, since it is the executable.

> +separate tag for auditors that are requested not by the main object,
> +but by one of the shared objects it was linked against.
> +
> +@end table
> +
> +The above methods are listed in the order indicated above; auditors
> +loaded by @code{LD_AUDIT} will be loaded first, etc.  For each

This reads a bit circular.  Maybe: "Auditors are loaded in the order
of the table above: first those from @code{LD_AUDIT}, then those from
@code{--audit}, @code{DT_AUDIT}, and @code{DT_DEPAUDIT}."

> +callback the dynamic loader calls, the modules will be called in the
> +order in which they were loaded.
> +
> +Note that --audit may occur multiple times.  The limit on the number
> +of colon-separated lists is currently 16.  The limit on the number of
> +auditors current defaults to 12.  @Theglibc{} will refuse to load if
> +too many auditors are requested.

s/current defaults/currently defaults/.  The 12 comes from the
glibc.rtld.nns tunable (the limit is 16 minus its value), so I would
say that and add @pxref{Dynamic Linking Tunables}.  Exceeding either
limit is a fatal error at startup, so "the program fails to start"
is more precise than "will refuse to load".

It would also help to say what happens when an auditor cannot be
loaded (not found, no la_version, and so on), where the dynamic linker
prints an error and the program runs without it.  It still counts
toward the limit, because the names are counted before loading.

> +
> +As each auditor is loaded, its @code{la_version} callback is called
> +to ensure the module supports the API version the dynamic loader is
> +using.  Auditors which do not are unloaded.  Once an auditor is
> +loaded, it may audit the loading of further auditors@footnote{The
> +current implementation sends events for the searching and loading, but
> +nothing significant after that.  This may be intentional or it may be
> +a bug, and auditors should assume auditing auditors is unsupported but
> +plan for such events.}.  Each auditor is loaded into its own namespace,
> +that is, any symbols exported by the audit will not be visible to the
> +main program or other auditors.

s/the audit/the auditor/.

> +
> +If the program running is AT_SECURE (for example, set-user-id
> +programs), auditors are limited to secure auditors - auditor names
> +must not include @samp{/} characters and will only be searched for in
> +standard directories (ignoring @file{ld.so.cache}).  Additionally,
> +such auditors must themselves be set-user-id, and have names less than
> +255 characters long.
> +
> +@node Audit API Callbacks
> +@subsection Audit API Callbacks
> +
> +Each auditor provides one or more callbacks in the form of global
> +functions that the dynamic linker may call.  @code{la_version} is
> +required of all auditors, but every other callback is optional.  The
> +auditor defines the callback if it wants those events.
> +
> +Each auditor may attach private data to each shared object in the
> +system via a word stored the shared object (as an intptr_t, so type
> +casting is always needed when storing pointers).  There is one word of
> +private data per shared object, per auditor.

Still from v2: the type is uintptr_t, and the word is kept by the dynamic
linker, not stored in the object ("stored the" is also missing a word).
link.h and rtld-audit(7) call it the "cookie", and your la_pltexit example
uses refcook/defcook, so I would mention that name once.

> +
> +This word may be an index into an array, a bitmask, or a pointer, and
> +the address of it is passed to most callbacks.  This word is
> +initialized just before calling @code{la_objopen} (or, for the first
> +object in a namespace, @code{la_activity}), to be a pointer to the
> +link map for the shared object.  The auditor may choose to initialize

The la_activity case holds for dlmopen namespaces but not for the
base namespace: at startup the order is la_objopen (main), la_objopen
(ld.so), then la_activity (LA_ACT_ADD).  The word is set when the
link map is created, so maybe:

  This word is set to the address of the object's link map when the
  dynamic linker creates the link map, before any callback receives
  it.  @code{la_objopen} is the first callback to see it with the 
  exception is the first object of a namespace created by
  @code{dlmopen} (which @code{la_activity} sees first).

> +it to something else (typically in @code{la_objopen()}, as this is
> +called right after initializing it) in order to store more or
> +different data per object.  There is no guaranteed way to free or
> +clean up allocated data used for this purpose, but a reasonable option
> +is to clean it up in @code{la_objclose()} @emph{and} set the word to
> +zero at that time.  With that, all callbacks would need to check for a
> +zero word and assume the related object has been or is being closed.
> +
> +@deftypefun {unsigned int} la_version (unsigned int version)
> +This mandatory callback is called when the auditor is loaded.  It is
> +passed the API version the dynamic linker is using.  It should return
> +the API version the module is using, or zero to silently unload the
> +module.  If the module and the dynamic linker are using incompatible
> +API versions, the auditor is unloaded and an error is printed.  The
> +expectation is that the module will return the @code{LAV_CURRENT}
> +value from @samp{<link.h>}.
> +
> +Note that @code{la_version()} may return values other than LAV_CURRENT
> +if it has knowledge of what the various versions provide and can adapt
> +to the @var{version} given, but doing this is outside the scope of
> +this document.  This may be complicated by LAV_CURRENT being different
> +for different architectures.
> +
> +@end deftypefun
> +
> +@deftypefun void la_activity (uintptr_t *privdata, unsigned int flag)
> +This callback is used to signal activity in a namespace.
> +@var{privdata} points to the private data for a shared object in that
> +namespace.  The @var{flag} is one of the following:

It is always the first object of the namespace (the main program for
LM_ID_BASE).  la_activity is also never called for the auditors' own
namespaces.

> +
> +@table @code
> +
> +@item LA_ACT_ADD
> +A shared object has been added to the namespace.  Further additions
> +are not signalled until the object is marked consistent again.

For dlopen and dlmopen, LA_ACT_ADD is sent before the la_objopen calls
for the new objects, so "New objects are being added to the
namespace".  Also s/the object is marked/the namespace is marked/.

> +
> +@item LA_ACT_DELETE
> +One or more shared objects will be removed from the namespace.
> +Further deletions will not be signalled until the link map is marked
> +consistent again.

s/link map/namespace/

> +
> +@item LA_ACT_CONSISTENT
> +All adds or deletes are complete for this namespace.  After this
> +callback, the namespace is considered consistent, and further
> +additions or deletions will cause another activity callback.  Note
> +that this callback will not happen if the namespace is empty.
> +
> +@end table
> +
> +@end deftypefun
> +
> +@deftypefun {char *} la_objsearch (const char *name, uintptr_t *privdata, unsigned int flag)
> +Called before the dynamic linker tries to locate a shared object to
> +load it.  @var{name} is the name of the object to be loaded.  The
> +callback must return @var{name} if it does not want to change the
> +loader's behavior.  Otherwise, it may return a different string to
> +change the object to be searched for or loaded, or NULL to prevent the
> +object from being searched for at all (for @code{LA_SER_ORIG}) or in
> +that path (for others).
> +
> +If the auditor returns an alternate string for this callback, the
> +string must remain valid for the duration of the application.  There
> +is no mechanism for freeing any allocated memory.  The alternate
> +string will be passed to the remaining auditors in the audit chain,
> +but if an auditor returns a NULL name, further auditors will not be
> +called for that path.
> +
> +The @var{privdata} points to the private data for the object doing the
> +search.  The @var{flag} is one of the following:
> +
> +@table @code
> +
> +@item LA_SER_ORIG
> +The dynamic linker is starting a search for @var{name}, usually via a
> +call to @code{dlopen()} or a @code{DT_NEEDED} entry.  A callback with
> +this flag will happen before a callback with any of the other flags,
> +which indicate progress through the search.
> +
> +@item LA_SER_LIBPATH
> +The dynamic loader is searching for @var{name} in one of the paths
> +specified by @code{LD_LIBRARY_PATH}.  If @var{name} is changed, it
> +must be to an existing file that can be opened without searching for
> +it.

The second sentence applies to every flag except LA_SER_ORIG (the
returned name is opened directly), so I would move it to the general
description.  Also, names that contain a slash only get the
LA_SER_ORIG call.

> +
> +@item LA_SER_RUNPATH
> +The dynamic loader is searching for @var{name} in a path specified
> +by @code{DT_RPATH} or @code{DT_RUNPATH}.
> +
> +@item LA_SER_CONFIG
> +The dynamic loader found a candidate for @var{name} in the paths
> +stored in @file{ld.so.cache}.
> +
> +@item LA_SER_DEFAULT
> +The dynamic loader found a candidate for @var{name} in the default
> +paths hardcoded in the dynamic linker itself.

Like LIBPATH and RUNPATH, this is called before each open attempt,
not after a candidate is found.  A dlopen of a library that does not
exist gets one LA_SER_DEFAULT call for each default directory and
each of their glibc-hwcaps subdirectories.  So "The dynamic loader is
searching for @var{name} in the default paths ...".

> +
> +@item LA_SER_SECURE
> +Unused.
> +
> +@end table
> +
> +@end deftypefun
> +
> +@deftypefun {unsigned int} la_objopen (struct link_map *map, Lmid_t lmid, uintptr_t *privdata)
> +Called when a shared object is loaded, including the main, ld.so, and
> +vDSO objects.  Unlike @code{la_activity} this callback is called once
> +for each object.  @var{map} is the link map for the object.  The
> +@var{privdata} points to the private data for the object being loaded,
> +is initialized to @var{map} before this callback@footnote{As if this
> +callback did @code{*privdata = (uintptr_t) map}}, and may be
> +initialized to point to auditor-specific data by this callback.
> +@var{lmid} is the namespace the object was loaded into, either
> +@code{LM_ID_BASE} for the base namespace, or some other value for
> +namespaces created with @code{dlmopen()}.
> +
> +The value returned by this callback is a bitmask of zero or more of
> +these flags:
> +
> +@table @code
> +
> +@item LA_FLG_BINDTO
> +Audit symbols bound to this object.
> +
> +@item LA_FLG_BINDFROM
> +Audit symbols bound from this object.
> +
> +@end table
> +
> +@end deftypefun
> +
> +@deftypefun {unsigned int} la_objclose (uintptr_t *privdata)
> +Called when a shared object (including the main, ld.so, and vDSO
> +objects) is unloaded.  For the main objects, this is just before the
> +program exits.  This is called after @code{la_activity} signals its
> +deletion.  The return value is ignored.

The vDSO never gets la_objclose: at exit, _dl_fini only calls it for
objects that went through initialization, and the vDSO does not.
So maybe:

  Called when a shared object is unloaded by @code{dlclose}, and at
  process exit, after its destructors have run, for every object
  that is still loaded (except the vDSO).

Another option would to issue la_objclose for vDSO as well.

> +
> +@end deftypefun
> +
> +@deftypefun void la_preinit (uintptr_t *privdata)
> +This callback is called just before the dynamic linker passes control
> +to the application (after ELF constructors, before @code{main}).  The

The part in parentheses is right, but the caller is not the dynamic
linker: __libc_start_main calls la_preinit after all constructors,
including the main program's, have run.  Maybe "This callback is
called after all ELF constructors have run, just before @code{main}
is called."

> +@var{privdata} always points to the private data for the main
> +application.
> +
> +@end deftypefun
> +
> +@deftypefun uintptr_t la_symbind32 (Elf_Sym32 *sym, unsigned int ndx, uintptr_t *refdata, uintptr_t *defdata, unsigned int *flags, const char *symname)
> +@deftypefunx uintptr_t la_symbind64 (Elf_Sym64 *sym, unsigned int ndx, uintptr_t *refdata, uintptr_t *defdata, unsigned int *flags, const char *symname)

The types are Elf32_Sym and Elf64_Sym, as in link.h.  The same fix is
needed in "Elf_Sym32 or Elf_Sym64 parameters" below.

> +Called when a binding occurs between the use of a symbol @var{sym}
> +with name @var{symname} and its definition.  @var{refdata} points to
> +the private data for the object referencing the symbol, and
> +@var{defdata} points to the private data for the object providing
> +the definition of the symbol.  @var{ndx} is the index of the symbol in
> +the symbol table of the shared object in which it is defined.
> +
> +This callback will be called only for symbols in objects where
> +@code{la_objopen} returned a nonzero value.  It will also only be
> +called for symbols that represent a function call, and use the PLT for
> +that call, and for queries from @code{dlsym}/@code{dlsym}.  For

s/dlsym\/dlsym/dlsym\/dlvsym/ (twice).


Two v2 points are still missing here:

  - @var{sym} is a copy of the symbol table entry whose st_value is
    the symbol's run-time address.  That is why returning
    sym->st_value leaves the binding unchanged.

  - *flags is not always zero on input.  LA_SYMB_DLSYM is set for
    dlsym.  When the PLT entry is bound at load time (LD_BIND_NOW=1,
    or -z now when no auditor defines la_pltenter or la_pltexit),
    la_symbind is still called, with LA_SYMB_NOPLTENTER |
    LA_SYMB_NOPLTEXIT already set.  I see flags=0x3 in both cases.

Current master also passes one auditor's flags on to the next one
(BZ 34687).  That is a bug that is being fixed separately, so the
documentation should not describe it.

> +@code{dlsym}/@code{dlsym}, either LA_FLG_BINDFROM must have been
> +returned for the referencing object, or LA_FLG_BINDTO for the defining
> +object.  For all other cases, both flags must be set.
> +
> +This callback is actually one of two callbacks, either
> +@code{la_symbind32} or @code{la_symbind64}, with corresponding
> +@code{Elf_Sym32} or @code{Elf_Sym64} parameters.  Both callbacks
> +function the same other than the symbol size.  The auditor may define
> +either or both, but if both, only one is ever called - the one that
> +corresponds to the application's architecture.
> +
> +@var{flags} points to a bitmask which may include any of the following:
> +
> +@table @code
> +@item LA_SYMB_NOPLTENTER
> +@itemx LA_SYMB_NOPLTEXIT
> +The auditor may add these flags to @var{*flags} to request skipping

*@var{flags}

> +callbacks for the architecture-specific la_pltenter and/or la_pltexit
> +events.  Note that to avoid the performance overhead associated with
> +the PLT callback hooks, all auditors must have neither an la_pltenter
> +nor an la_pltexit callback.
> +
> +@item LA_SYMB_STRUCTCALL
> +Unused.
> +
> +@item LA_SYMB_DLSYM
> +The symbol is being bound due to a @code{dlsym()} call.
> +
> +@item LA_SYMB_ALTVALUE
> +The symbol's value has been altered by a previous @var{la_symbind}
> +callback.

@code instead of @var here, and maybe "by an earlier auditor's
la_symbind callback".

> +
> +@end table
> +
> +The value returned by @code{la_symbind()} is used as the address of
> +the symbol.  The auditor should return @code{sym->st_value} if it does
> +not wish to modify the application's behavior.
> +
> +@end deftypefun
> +
> +@deftypefun  uintptr_t la_pltenter (@dots{})
> +This ``callback'' represents one of many architecture-defined
> +callbacks listed in @code{<link.h>}, refer to that for specifics.  The
> +parameters are similar to @code{la_symbind}, with the addition of one
> +or more parameters for passing register contents.  For example, the
> +x86-64 prototype is:
> +
> +@example
> +Elf64_Addr la_x86_64_gnu_pltenter (Elf64_Sym *sym,
> +    unsigned int ndx, uintptr_t *refdata, uintptr_t *defdata,
> +    La_x86_64_regs *regs, unsigned int *flags,
> +    const char *symname, long int *framesizep);
> +@end example
> +
> +Note that decoding the arguments is very dependent on the platform and
> +compiler flags, and is outside the scope of this document.
> +
> +@c FIXME add architecture-specific prototpes and documentation

s/prototpes/prototypes/

> +
> +The @code{la_pltenter} callback is called when a caller is about to
> +call a function through a PLT entry which is lazily bound, if the
> +caller and callee objects have been marked for auditing by
> +@code{la_objopen()}.  The callback returns the address to actually
> +call, or @code{sym->st_value} if it does not wish to modify the call.
> +Note that the PLT remains lazily bound for the life of the program if
> +this callback is defined.
> +
> +Note that hardening an executable by disabling lazy binding
> +(i.e. using @code{-fno-plt} when building it or setting
> +@code{LD_BIND_NOW=1}) may block the ability to trace calls via the
> +PLT.
> +

The usual hardening option, -z now, does not block tracing since the
dynamic linker ignores DT_BIND_NOW when an auditor defines
la_pltenter or la_pltexit.  Readers will think of -z now first, so I
would say this explicitly:

  Calls are not traced if the PLT is bound at load time with
  @code{LD_BIND_NOW=1}, or if the code does not use the PLT at all
  (for example, when built with @code{-fno-plt}).  Linking with
  @code{-z now} does not prevent tracing, because the dynamic linker
  keeps lazy binding when an auditor defines @code{la_pltenter} or
  @code{la_pltexit}.

> +The callback may modify the frame size used for @code{la_pltexit} via
> +the @var{framesizep} pointer.  The call will use the largest requested
> +frame size.  If no auditors modify this, @code{la_pltexit} will not be
> +called.  This frame size is the amount of the caller's stack frame
> +which is copied to the callee's stack frame, as the presence of the
> +callback forces another call frame between the two (so the callee
> +returns to the callback, not to the callee).

The last "callee" should be "caller". The text should say how to choose
the value, since a wrong one silently breaks the program, I would suggest:

  *@var{framesizep} is initially -1, which means @code{la_pltexit} is
  not wanted.  To receive @code{la_pltexit}, set it to the number of
  bytes of the caller's stack (the arguments passed on the stack)
  that must be copied for the call.  The dynamic linker calls the
  function from its own frame, so that the function returns to the
  dynamic linker, and copies that many bytes into that frame.  If
  the value is too small, the function reads garbage for its stack
  arguments; 0 is fine for functions with no stack arguments.  If
  several auditors set it, the largest value is used.

(This is tricky API and maybe we should rethink this eventually).

> +
> +@end deftypefun
> +
> +@deftypefun {unsigned int} la_pltexit (@dots{})
> +Architecture-specific function similar to @code{la_pltenter}.  This
> +callback is called when a function, called through the PLT, returns to
> +the caller.  It will only be called if @code{la_pltenter} specified a
> +frame size for it.  The parameters are architecture-dependent, but all
> +include an in-registers parameter, which contains the original
> +registers at the time of @code{la_pltenter}, and an out-registers
> +parameter, which contains the return value of the function, which the
> +auditor may modify.  For example, the x86_64 function is:
> +
> +@example
> +unsigned int la_x86_64_gnu_pltexit (Elf64_Sym *sym,
> +    unsigned int ndx, uintptr_t *refcook, uintptr_t *defcook,
> +    const La_x86_64_regs *inregs, La_x86_64_retval *outregs,
> +    const char *symname);
> +@end example
> +
> +@end deftypefun
> +
>  @node Indirect Functions
>  @section Indirect Functions
>  @cindex indirect function