[PATCH v1 1/1] manual: document audit interface

[Ben Woodard]

> This is my attempt at initial documentation for the la_* audit
> interface, based on reading the code...
> ---
>
> Note: I was asked to add this to the manual, but given there was no
> documentation to begin with, I had to guess on some of the things.  If
> you disagree with what should be supported or not, or documented or
> not, please discuss ;-)
>
>   manual/dynlink.texi | 315 ++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 315 insertions(+)
>
> diff --git a/manual/dynlink.texi b/manual/dynlink.texi
> index ad4da753a5..0d10401f49 100644
> --- a/manual/dynlink.texi
> +++ b/manual/dynlink.texi
> @@ -17,6 +17,7 @@ Dynamic linkers are sometimes called @dfn{dynamic loaders}.
>   * Dynamic Linker Environment Variables:: Environment variables that control the
>                                             dynamic linker.
>   * Dynamic Linker Introspection::    Interfaces for querying mapping information.
> +* Dynamic Linker Auditing::	Auditing interface.
>   * Indirect Functions::          Selecting function implementations at load time.
>   * Dynamic Linker Hardening::    Avoiding unexpected issues with dynamic linking.
>   @end menu
> @@ -736,6 +737,320 @@ A bit mask used to signal that the object contains SFrame data.  See
>   
>   @end table
>   
> +@node Dynamic Linker Auditing
> +@section Dynamic Linker Auditing
> +@cindex Auditing
> +@cindex LD_AUDIT
> +
> +The dynamic linker's auditing interface allows external code to
> +observe and affect the loading of dynamic objects.  As this allows the
> +external code to modify the operation of the program, great care
> +should be taken when creating and deploying audit modules (auditors).
> +
> +Audit modules should include @code{<link.h>} to import constants and
> +prototypes used by the audit API.
> +
> +@menu
> +* Audit Module Loading::  How modules are selected and loaded.
> +* Audit API Callbacks::	  Audit functions called by the dynamic linker.
> +@end menu
> +
> +@node Audit Module Loading
> +@subsection Audit Module Loading
> +
> +An audit module (auditor) is a shared object (plus its DT_NEEDED
> +dependencies) that is loaded very early in the dynamic loading
> +process, and isolated from the main application and other auditors.
I think that the word "isolated" here needs more explanation here. You 
and I know what "isolated" means in this context but I think most 
readers will not. I think that you need to say that its linkage 
namespace is isolated from other auditors. Then explain that this allows 
auditors to have their own libc and libstdc++.

Also isolation does not mean one auditor can't affect another auditor. 
While you don't get symbind events for other auditors, you do at least 
get objopen for the other auditors. I can't remember off the top of my 
head but I believe that one auditor can substitute another library for 
other auditors. That isn't exactly isolation.

The point is "isolated" in this context really refers to their link maps.

I could see people mis-understanding "isolated" to mean something more 
like the isolation between processes.

Now you don't want to mention this because it is a bug that needs to be 
fixed but that isolation creates problems when dealing with TLS. Because 
each glibc that uses TLS will try to initialize its own TLS and that 
causes it to reinitialize TLS and breaks everything.
> +Once loaded, the dynamic loader will call functions in the auditor at
> +key points in the load process.  Auditors are specified by four
> +methods, each of which specifies a colon-separated list of auditors:
> +
> +@table @code
> +
> +@item --audit
> +If the dynamic linker is invoked manually, it allows an @code{--audit}
> +command line parameter followed by a list of auditors.
> +
> +@item LD_AUDIT
> +The @code{LD_AUDIT} environment variable
> +
> +@item DT_AUDIT
> +If a shared object has a DT_AUDIT dynamic tag, that tag specifies a
> +list of auditors.
> +
> +@item DT_DEPAUDIT
> +Similarly for the DT_DEPAUDIT dynamic tag.
I think that you need to explain this better. It took me a while to 
understand this and I'm still not sure that I'm correct. DT_DEPAUDIT is 
an advisory flag. It basically says "I do not have any DT_AUDIT 
auditor's specified but one of the dependencies does." It took me ages 
to figure that out. The Sun docs aren't exactly clear on that point.
> +
> +@end table
> +
> +The above methods are listed in order of precedence; auditors loaded
> +by @code{--audit} will be loaded first, etc.  For each callback the
> +dynamic loader calls, the modules will be called in the order in which
> +they were loaded.
> +
> +Note that, other than the environment variable, these may occur
> +multiple times.  The limit on the number of occurances (not the number
> +of audits) is currently 16.
Should that be "not the number of auditors"
> +
> +As each auditor is loaded, it's @code{la_version} callback is called
> +to ensure the module supports the API version the dynamic loader is
> +using.  Auditors which do not are unloaded.  Once an auditor is
> +loaded, it may audit the loading of further auditors@footnote{The
> +current implementation sends events for the searching and loading, but
> +nothing significant after that.  This may be intentional or it may be
> +a bug,
I have been arguing and will continue to argue that this is a bug. I 
personally know someone who was in the room and was a participant when 
the LD_AUDIT interface was designed. He's like 90 now and long since 
retired. This was not the intent. The initial implementation in Solaris 
had bugs in it. Then Sun collapsed and work pretty much stopped. It was 
this buggy behavior that was copied by glibc.

This creates problems for tools that now currently exist. In particular, 
we have one performance tool which does PC sampling. To be able to 
properly attribute each sample that it gets, it needs to know about the 
complete process address space. This includes other auditors because a 
PC sample can legitimately occur in another auditor. While former 
auditors are notified about the initial loading of subsequent auditors 
and are also notified about changes made to the application's namespace, 
they are not notified about objects that auditors add to their own link 
map. i.e. if an auditor dlopen's or dlmopen's an object. This leaves the 
PC sampling auditor's notion of the complete process address space 
incomplete. We believe that this is obviously a bug. We would like to 
see la_activity, la_objsearch, la_objopen/la_objclose events for other 
auditors.

So far, we do not know of a compelling justification for supporting 
symbind or pltenter/pltexit of other auditors. The one exception is we 
do need to symbind the library constructor and library destructor for 
certain libraries. Currently, all known cases of this in our tools 
happen during auditor load time when we do get la_objopen events and so 
we do not have a strong case for needing symbind being supported by 
other auditors.

> and auditors should assume auditing auditors is unsupported but
> +plan for such events.}.  Each auditor is loaded into its own namespace,
> +that is, any symbols exported by the audit will not be visible to the
> +main program or other auditors.
However auditors can add things to other auditor's and to the 
application's namespace. They just have to have the cookie. This allows 
you to effectively LD_PRELOAD libraries as needed. This is an important 
behavior and needs to be documented.I just had to explain how to do this 
to one of our tool developers earlier this morning.
> +
> +If the program running is AT_SECURE (for example, set-user-id
> +programs), auditors are limited to secure auditors - auditor names
> +must not include @samp{/} characters and will only be searched for in
> +secure directories.  Additionally, such auditors must themselves
> +be set-user-id, and have names less than 255 characters long.
> +
> +@node Audit API Callbacks
> +@subsection Audit API Callbacks
> +
> +Each auditor provides one or more callbacks in the form of global
> +functions that the dynamic linker may call.  @code{la_version} is
> +required of all auditors, but every other callback is optional.  The
> +auditor defines the callback if it wants those events.

> +
> +Some callbacks use a @emph{cookie} to store data specific to each
> +shared object.  This cookie is initialized by @code{la_objopen} to a
> +unique but arbitrary value, but as a pointer to it is passed to the
> +callbacks, the auditor may choose to replace the cookie with something
> +of its own design (for example, a pointer to the link map for the
> +object, a structure which includes such a pointer, or an index into an
> +array of data).
One design problem with this is the first time you see some cookies is 
in the la_activity whenever a new namespace is being initialized. As far 
as I know you can't change the cookie at that time. That cookie is then 
used in la_objsearch and once again, I'm not sure if you can change it 
then. Finally, it is used in la_objopen where you can finally change it.

If you can change the cookie at either of those two callbacks, then it 
should be documented.

I have not tried this but one of the things that remain unclear to me is 
if I change the cookie in la_objopen for the first object in a new 
namespace does it retroactively change the value of the cookie sent in 
la_activity and la_objsearch? In other words if I change the cookie when 
doing la_objopen for the first object in a namespace, will I get 
la_objsearch events for objects subsequently loaded in that namespace? 
Also when I get a la_activity will it be the cookie that I changed in 
la_objopen or the original one when I got the first la_activity 
introducing me to the new namespace.

To me it seems like there are really two kinds of cookies here. There 
are namespace cookies used in la_activity and la_objsearch and there are 
object cookies used in la_objopen and la_objclose. Currently, the 
namespace cookie and the first object in that namespace's cookie are the 
same but the fact that you see two callbacks la_activity and 
la_objsearch before you get to change that cookie in the la_objopen 
makes them feel distinct.

I think that several things need to be documented:

1) are namespace cookies actually distinct from object cookies. i.e. if 
I change an object's cookie in la_objopen does it change the namespace's 
cookie for la_activity and la_objsearch?

2) can you change the namespace cookie in la_activity or la_objsearch?

3) what kind of cookie does la_preinit get? Is it a namespace cookie or 
an object cookie? I think it is a namespace cookie. If it is, then why 
doesn't la_preinit get called when the initial loading of each namespace 
is complete. It really doesn't make sense to me if it is object cookie.

It is confusion about these issues is why I tell tool developers that I 
work with not to change cookies just use them as a handle.

What I would like is if there were distinct (proper math term would be 
disjoint) namespace cookies used for la_activity la_objsearch, and 
la_preinit and distinct object cookies used for la_objsearch and 
la_objclose. If you want, you can add the capability to change a 
namespace cookie in la_activity like you can in la_objopen.

> Each auditor has its own set of private cookies,
> +there is one per auditor per shared object.
This is useful to know. I did not realize this. But are you talking 
about object cookies or namespace cookies?
> There is no guaranteed
> +way to free or clean up allocated data used as a cookie, but a
> +reasonable option is to clean up the cookie in @code{la_objclose()}
> +@emph{and} set the cookie to zero at that time.  With that, all
> +callbacks would need to check for a zero cookie and assume the related
> +object has been or is being closed.
> +
> +@deftypefun {unsigned int} la_version (unsigned int version)
> +This mandatory callback is called when the auditor is loaded.  It is
> +passed the API version the dynamic linker is using.  It should return
> +the API version the module is using, or zero to unload the module.  If
> +the module and the dynamic linker are using incompatible API versions,
> +the auditor is unloaded.  The expectation is that the module will
> +return the @code{LAV_CURRENT} value from @samp{<link.h>}.
> +@end deftypefun
> +

There are a bunch of things that you can't do in la_version(), for 
example you cannot dlopen or dlmopen other libraries from there. If you 
do, then you crash because you are reentering the loader. I think that 
these need to be documented. I figured some of them out while writing 
mux_audit https://github.com/woodard/mux-audit and audit_hook 
https://github.com/woodard/audit_hook I'll have to go back and review 
some of my notes and experiments to figure out what all the limitations 
that I discovered were. The point is, you can't really treat la_version 
as a init function where you setup everything you want to setup. You are 
not really operating as an auditor at that point. It is not until you 
return from la_version and then get a subsequent callback that you can 
start setting up your environment as an auditor.

> +@deftypefun void la_activity (uintptr_t *cookie, unsigned int flag)
> +This callback is used to signal activity in a namespace.  @var{cookie}
> +is the cookie for a shared object in that namespace.  The @var{flag}
> +is one of the following:
> +
> +@table @code
> +
> +@item LC_ACT_ADD
> +A shared object has been added to the namespace.  Further additions
> +are not signalled until the object is marked consistent again.
> +
> +@item LC_ACT_DELETE
> +One or more shared object will be removed from the namespace.  Further
> +deletions will not be signalled until the link map is marked
> +consistent again.
> +
> +@item LC_ACT_CONSISTENT
> +All adds or deletes are complete for this namespace.  After this
> +callback, the namespace is considered consistent, and further
> +additions or deletions will cause another activity callback.  Note
> +that this callback will not happen if the namespace is empty.
> +
> +@end table
> +
> +@end deftypefun
> +
> +@deftypefun {char *} la_objsearch (const char *name, uintptr_t *cookie, unsigned int flag)
> +Called before the dynamic linker tries to locate a shared object to
> +load it.  @var{name} is the name of the object to be loaded.  The
> +callback must return @var{name} if it does not want to change the
> +loader's behavior.  Otherwise, it may return a different string to
> +change the object to be searched for or loaded, or NULL to prevent the
> +object from being searched for (LA_SER_ORIG) or loaded (other flags).
> +
> +If the auditor returns an alternate string for this callback, the
> +string must remain valid for the duration of the application.  There
> +is no mechanism for freeing any allocated memory.
> +
> +The @var{cookie} is the cookie for the object doing the search.  The
> +@var{flag} is one of the following:
> +
> +@table @code
> +
> +@item LA_SER_ORIG
> +The dynamic linker is starting a search for @var{name}, usually via a
> +call to @code{dlopen()} or a @code{DT_NEEDED} entry.  A callback with
> +this flag will happen before a callback with any of the other flags,
> +which indicate progress through the search.
> +
> +@item LA_SER_LIBPATH
> +The dynanmic loader is searching for @var{name} in the paths specified
> +by @code{LD_LIBRARY_PATH}, and a candidate object has been found.  If
> +@var{name} is changed, it must be to an existing file that can be
> +opened without searching for it.
> +
> +@item LA_SER_RUNPATH
> +The dynanmic loader found a candidate for @var{name} in the paths
> +specified by @code{DT_RPATH}.
> +
> +@item LA_SER_CONFIG
> +The dynanmic loader found a candidate for @var{name} in the paths
> +stored in @file{ld.so.cache}.
> +
> +@item LA_SER_DEFAULT
> +The dynanmic loader found a candidate for @var{name} in the default
> +paths hardcoded in the dynamic linker itself.
> +
> +@item LA_SER_SECURE
> +Unused in Linux.
For multi-auditor situations, there also kind of needs to be something 
like LA_SER_AUDIT or something like that for a value which was replaced 
by the previous auditor.
> +
> +@end table
> +
> +@end deftypefun
> +
> +@deftypefun {unsigned int} la_objopen (struct link_map *map, Lmid_t lmid, uintptr_t *cookie)
> +Called when a shared object is loaded.  Unlike @code{la_activity} this
> +callback is called once for each object.  @var{map} is the link map
> +for the object.  The @var{cookie} is the cookie for the object being
> +added, is initialized to @var{map} before this callback@footnote{As if
> +this callback did @code{*cookie = (uintptr_t) map}}, and may be
> +modified by this callback.  @var{lmid} is one of the following:
> +
> +@table @code
> +
> +@item LM_ID_BASE
> +The object is in the base namespace.
> +
> +@item LM_ID_NEWLM
> +A new namespace has been requested via @code{dlmopen()}.
I think it is wrong. You do not really see LM_ID_NEWLM you just see 
la_activity with a value you haven't seen before. It is also wrong in 
the rtld-audit man page.

Yes you pass in LM_ID_NEWLM when you call dlmopen and you want it to 
start a new namespace but auditors never see LM_ID_NEWLM.

> +
> +@end table
> +
> +The value returned by this callback is a bitmask of zero or more of
> +these flags:
> +
> +@table @code
> +
> +@item LA_FLG_BINDTO
> +Audit symbols bound to this object.
> +
> +@item LA_FLG_BINDFROM
> +Audit symbols bound from this object.
> +
> +@end table
> +
> +@end deftypefun
> +
> +@deftypefun {unsigned int} la_objclose (uintptr_t *cookie)
> +Called when a shared object is unloaded.  This is called after
> +@code{la_activity} signals its deletion.  The return value is ignored.
> +
> +@end deftypefun
> +
> +@deftypefun void la_preinit (uintptr_t *cookie)
> +This callback is called just before the dynamic linker passes control
> +to the application.  The @var{cookie} is always the cookie for the
> +main application.
> +
> +@end deftypefun
> +
> +@deftypefun uintptr_t la_symbind (Elf_Sym *sym, unsigned int ndx, uintptr_t *refcookie, uintptr_t *defcookie, unsigned int *flags, const char *symname)
> +@findex la_symbind32
> +@findex la_symbind64
> +Called when a binding occurs between the use of a symbol @var{sym}
> +with name @var{symname} in the object referenced by @var{refcookie}
> +and the definition of the symbol in the object referenced by
> +@var{defcookie}.  @var{ndx} is the index of the symbol in the symbol
> +table of the shared object in which it is defined.
> +
> +This callback will be called only for symbols in objects where
> +@code{la_objopen} returned a nonzero value.  It will also only be
> +called for symbols that represent a function call, and that use the
> +PLT for that call.
> +
> +This callback is actually one of two callbacks, either
> +@code{la_symbind32} or @code{la_symbind64}, with corresponding
> +@code{Elf32_Sym} or @code{Elf_Sym64} parameters.  Both callbacks
> +function the same other than the symbol size.  The auditor may define
> +either or both, but if both, only one is ever called - the one that
> +corresponds to the application's architecture.
> +
> +@var{flags} points to a bitmask which may include any of the following:
> +
> +@table @code
> +@item LA_SYMB_NOPLTENTER
> +@item LA_SYMB_NOPLTEXIT
> +The auditor may add these flags to @var{*flags} to request skipping
> +callbacks for the architecture-specific la_pltenter and/or la_pltexit
> +events.
> +
I think that right here you may want to discuss the performance 
implications of not finalizing the symbol in the GOT.
> +@item LA_SYMB_STRUCTCALL
> +Unused in Linux.
> +
> +@item LA_SYMB_DLSYM
> +The symbol is being bound due to a @code{dlsym()} call.
> +
> +@item LA_SYMB_ALTVALUE
> +The symbol's value has been altered by a previous @var{la_symbind()}
> +callback
> +
> +@end table
> +
> +The value returned by @code{la_symbind()} is used as the address of
> +the symbol.  The auditor should return @code{sym->st_value} if it does
> +not wish to modify the application's behavior.
> +
> +@end deftypefun
> +
> +@deftypefun  uintptr_t la_pltenter (@dots{})
> +This callback is one of many architecture-defined callbacks listed in
> +@code{<link.h>}, refer to that for specifics.  The parameters are
> +similar to @code{la_symbind}, with the addition of one or more
> +parameters for passing register contents.  For example, the x86-64
> +prototype is:
> +
> +@example
> +Elf64_Addr la_x86_64_gnu_pltenter (Elf64_Sym *sym,
> +    unsigned int ndx, uintptr_t *refcook, uintptr_t *defcook,
> +    La_x86_64_regs *regs, unsigned int *flags,
> +    const char *symname, long int *framesizep);
> +@end example
> +
> +@c FIXME add architecture-specific prototpes and documentation
> +
> +The @code{la_pltenter} callback is called when a caller is about to
> +call a function through a PLT entry which is lazily bound, if the
> +caller and/or callee objects have been marked for auditing by
> +@code{la_objopen()}.  The callback returns the address to actually
> +call, or @code{sym->st_value} if it does not wish to modify the call.
> +
> +Note that hardening an executable by disabling lazy binding
> +(i.e. using @code{--enable-bind-now} or @code{-z now} when building
> +it) blocks the ability to trace calls via the PLT.
Are you sure about that. I'm pretty sure you still get the symbind you 
just get it earlier and then if you mark the function for auditing, I'm 
pretty sure you sill get the pltenter and exit callbacks.
> +
> +The callback may modify the frame size used for @code{la_pltexit} via
> +the @var{framesizep} pointer.  The call will use the largest requested
> +frame size.  If no auditors modify this, @code{la_pltexit()} will not
> +be called.
Some more explanation and maybe an example of how you can use framesizep 
to change the size of the frame used by the function if the changes you 
are making to the function's parameters change the size of the stack 
being used by the function. I've never really understood how this could 
be made to actually work. Luckily, It hasn't come up with any of the 
tool developers that I work with but my expectation is if anyone 
actually tried to do that, we would immediately discover bugs.

-ben

> +
> +@end deftypefun
> +
> +@deftypefun  void la_pltexit (@dots{})
> +Architecture-specific function similar to @code{la_pltenter}.  This
> +callback is called when a function, called through the PLT, returns to
> +the caller.  It will only be called if @code{la_pltenter} specified a
> +frame size for it.  @var{inregs} contains the original registers at
> +the time of @code{la_pltenter}, and @var{outregs} contains the return
> +value of the function, which the auditor may modify.
> +
> +@end deftypefun
> +
>   @node Indirect Functions
>   @section Indirect Functions
>   @cindex indirect function