A collection of LD_AUDIT bugs that are important for tools (with better formatting for this list)

[John Mellor-Crummey]

I was encouraged to notify this list about several LD_AUDIT bugs that
have significant impact on performance tools that we are developing
for Linux in general and US Department of Energy (DOE) parallel
supercomputers in particular.

My team develops the HPCToolkit performance tools
(https://hpctoolkit.org, https://github.com/HPCToolkit/hpctoolkit)
under funding from the DOE. We have been modifying our measurement
subsystem to interpose itself between an application and the OS using
glibc’s LD_AUDIT capability.

For such tools to succeed, we need many of LD_AUDIT’s features to
work. Over the last eight months, we identified six bugs on systems we
use. We believe that four are still problems in upstream glibc and two
may be fixed (as noted).

We would like confirmed fixes for the following bugs in upstream
glibc:

----------------------------------------------------------
Priority   | Issue 
—————————————————————————————
VERY       | When using an auditor, there is an unacceptable
HIGH       | performance degradation of over 10x for PLT
           | calls to small procedures even when neither
           | la_pltenter or la_pltexit is present.
----------------------------------------------------------
HIGH       | When auditing, a dlmopen of a shared library
           | causes a SEGV.
----------------------------------------------------------
HIGH       | la_symbind isn't always called when appropriate.
           | We observed that glibc 2.26 calls la_symbind
           | when appropriate; glibc 2.28 does not.
----------------------------------------------------------
HIGH       | glibc does not save all necessary registers
           | (e.g. X8 - the indirect result register, truncated
           | SIMD registers) when auditing on aarch64 since
           | the beginning of time.
----------------------------------------------------------
LOW        | When auditing, a dlopen of a shared library
           | that uses R_X86_64_TLSDESC causes a SEGV. This
           | is reportedly fixed in glibc 2.34.
----------------------------------------------------------
LOW   | An auditor added to an executable at link time
           | with --audit=auditor.so and noted in the DT_AUDIT
           | entry of the dynamic section is not called at
           | runtime. This is reportedly fixed in glibc 2.32.
----------------------------------------------------------



A repository of reproducers for these bugs can be found here:
https://github.com/hpctoolkit/auditor-tests.

A detailed writeup of everything known about each of these bugs,
including links to Red Hat and Sourceware Bugzilla entries, if any are
known to exist, can be found here:
https://docs.google.com/document/d/1dVaDBdzySecxQqD6hLLzDrEF18M1UtjDna9gL5BWWI0/edit?usp=sharing

Technical stakeholders for platforms that are HPCToolkit’s principal
targets under DOE funding, especially the exascale computing program:

----------------------------------------------------------
Stakeholder | Why
----------------------------------------------------------
Intel       | Prime contractor on Aurora exascale system at
            | Argonne National Laboratory
----------------------------------------------------------
IBM         | Prime contractor and processor vendor for Summit
            | and Sierra supercomputers at Oak Ridge National
            | Laboratory and Lawrence Livermore  National
            | Laboratory.  
----------------------------------------------------------
ARM         | Stakeholder who wants all ARM Linux platforms
            | to succeed, including Sandia National Laboratory's
            | Astra supercomputer and SUNY Stony Brook's
            | A64FX-based Ookami.
----------------------------------------------------------
AMD         | Processor vendor for Frontier and El Capitan
            | exascale supercomputers at Oak Ridge and Lawrence
            | Livermore National Laboratories.  
----------------------------------------------------------
SuSE        | Linux distribution provider for Cray systems to be
            | delivered to Oak Ridge and Lawrence Livermore
            | National Laboratories and the A64FX-based system
            | installed at SUNY Stony Brook.  
----------------------------------------------------------
Red Hat     | Linux distribution provider for Oak Ridge
            | National Laboratory s Summit, Lawrence Livermore
            | National Laboratory s Sierra.
----------------------------------------------------------
Cray        | Prime contractor and system vendor for Oak Ridge
            | and Lawrence Livermore National Laboratories,
            | and SUNY Stony Brook; system vendor for Argonne
            | National Laboratory.
----------------------------------------------------------


For reference, here is a pointer to the portion of our tool that uses
the LD_AUDIT interface:
https://github.com/HPCToolkit/hpctoolkit/blob/master/src/tool/hpcrun/audit/auditor.c

Here are some of the capabilities of LD_AUDIT that we need to work and why:

-  We use LD_AUDIT’s la_objopen and la_objclose to track what objects are
   in an application’s address space so that our measurement subsystem
   can unwind the call stack when a profiling signal is
   received. Tracking libraries by wrapping dlopen is problematic for
   several reasons. For instance, a wrapper would need to implement RPATH
   and RUNPATH semantics because glibc does not provide an alternate
   dlopen interface (like _dlsym) so that a wrapper can provide the
   return address in the requesting library as an argument which glibc
   needs to determine the R_PATH and RUNPATH to use when trying to find
   the library and its dependencies.

-  We want to use LD_AUDIT’s la_symbind32 and la_symbind64 to interpose
   wrappers around key functions, e.g. pthread_create. This enables a
   tool to intercept functions invoked through pointers obtained with
   dlsym, which preloaded wrappers can’t do. (Note: We don’t use
   la_symbind for interposition yet, but we plan to when it works
   everywhere.)

-  We need auditing to work when an application or a tool library (e.g.,
   Intel’s GT-Pin) opens a shared library with dlmopen.

-  We need auditing to work when opening a dynamic library with TLS
   dialect gnu2 relocations on x86_64 (R_X86_64_TLSDESC). We don’t have
   any special interest in such relocations; at present, they cause a
   SEGV when auditing and that must be avoided.

-  We want to add an auditor to an application at link time, noted in the
   DT_AUDIT entry of the dynamic section. Loading the DT_AUDIT entry as a
   program is launched enables our profiler to be injected into an
   application’s address space without a wrapper script that sets
   LD_AUDIT and LD_PRELOAD.

-  LD_AUDIT needs to work on aarch64, which is an important target for
   our tools. The fact that _dl_runtime_profile does not save register x8
   (the indirect result register) is often fatal for applications, which
   makes LD_AUDIT unusable for any purpose on aarch64.

-  LD_AUDIT needs to support auditing of inter-object calls on aarch64
   when SVE registers are in use.

As a final thing to consider: we understand that there is a tension
between security and auditability. We are concerned that changes being
considered for security may compromise observability for tools. For
tools, we would need a way to authorize full observability even in the
cases when that may theoretically reduce security. Perhaps setting
DT_AUDIT could be considered as authorizing full observability.

--
John Mellor-Crummey         Professor
Dept of Computer Science    Rice University
email: johnmc@rice.edu      phone: 713-348-5179