Loading glibc in a new namespace fails vtable check

[Moshe Rubin]

Hi,

**Executive Summary**

For technical reasons, a project of mine attempts to load a second copy of
glibc to a new namespace using the dlmopen(LM_ID_NEWLM, <libc-path>,
RTLD_GLOBAL | RTLD_LAZY) command.  The function fails internally with the
message "Fatal error: glibc detected an invalid stdio handle, Aborted (core
dumped)".

Analysis shows that loading a second copy of libc fails the internal glibc
function _IO_vtable_check() in libio/vtables.c.  This does not happen in
all cases:

- When I run a "proof-of-concept" C++ app that calls dlmopen("glibc.so")
under target OSs (both x86 and MIPS), the program executes flawlessly.
- When I run a production application that calls dlmopen("glibc.so") under
target OSs (both x86 and MIPS) I receive a fatal error.

**Question**

Can someone please explain why dlmopen() et al works flawlessly in one case
but fails in the other?  If the same shared library object failed in both
cases, I would understand, but why are there different results?

**Proof-of-concept (POC) application**

To understand what I'm talking about, here is my POC C++ code:

<code>
// To build: g++ main.cpp -o test_dlmopen -g -ldl
// To run: ./ test_dlmopen

#include <string>
#include <vector>
#include <fcntl.h> // Include the <fcntl.h> header file
#include <unistd.h>
#include <regex>
#include <dlfcn.h> // Include the <dlfcn.h> header file

bool getGlibcPath(std::string& path)
{
  const char* proc_maps_path = "/proc/self/maps";
  bool retval = false;

  int fd = open(proc_maps_path, O_RDONLY);
  if (fd == -1) {
    printf("open(%s) failed\n", proc_maps_path);
    return retval;
  }

  std::string buffer(4096, 0);
  ssize_t bytesRead = read(fd, (void*) buffer.data(), buffer.size() - 1);

  if (bytesRead == -1) {
    printf("read() failed\n");
    return retval;
  }

  close(fd);

  std::smatch match;
  std::regex rgx("(/.*/?libc[.-].*)\n");

  if (std::regex_search(buffer, match, rgx)) {
      path = match[1].str();
      retval = true;
  }

  return retval;
}

int main(int argc, char*argv[])
{
  std::string glibc_path;
  if (!getGlibcPath(glibc_path)) {
    return -1;
  }

  printf("glibc path: %s\n", glibc_path.c_str());

  // Load a second copy of glibc to a new namespace
  void* handle = dlmopen(LM_ID_NEWLM, glibc_path.c_str(), RTLD_LAZY);
  if (!handle) {
    printf("Failed to load glibc using dlmopen\n");
    return -1;
  }

  // Get the address of the several functions from the new glibc
  void* (*mallocPtr)(size_t) = (void* (*)(size_t))dlsym(handle, "malloc");
  void* (*freePtr)(void*) = (void* (*)(void*))dlsym(handle, "free");
  int (*openPtr)(const char*, int) = (int (*)(const char*,
int))dlsym(handle, "open");
  void (*closePtr)(int) = (void (*)(int))dlsym(handle, "close");

  // Call malloc in the new namespace
  void* ptr = mallocPtr(100);

  if (ptr) {
    printf("Allocated memory using dlmopen!\n");
    freePtr(ptr);
  } else {
    printf("Failed to allocate memory using dlmopen\n");
    return -1;
  }

  // Call open in the new namespace
  int fd = openPtr("/proc/self/maps", O_RDONLY);
  if (fd != -1) {
    printf("Opened file using dlmopen!\n");
    closePtr(fd);
  } else {
    printf("Failed to open file using dlmopen\n");
    return -1;
  }

  return 0;
}
</code>

When compiled, the correct typical output is:

<output>
# ./test_dlmopen
glibc path: /lib/libc-2.28.so
Allocated memory using dlmopen!
Opened file using dlmopen!
#
</output>

**The production application**

The production application is written in C++ and naturally has a much more
complicated code base than the POC.  They both, however, perform the same
operation of calling dlmopen() to load a second copy of glibc.so.  The
production app functioned correctly until I added code that calls
dlmopen().  Now it throws a fatal error in the early shared library loading
stage.  Here is the gdb backtrace of the failure:

<backtrace>
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x000000fff401d1e4 in __GI_abort () at abort.c:79
#2  0x000000fff405c51c in __libc_message (action=action@entry=(do_abort |
do_backtrace), fmt=fmt@entry=0xfff414b630 "%s") at
../sysdeps/posix/libc_fatal.c:181
#3  0x000000fff405c568 in __GI___libc_fatal (message=0xfff414d6e0 "Fatal
error: glibc detected an invalid stdio handle\n") at
../sysdeps/posix/libc_fatal.c:191
#4  0x000000fff405cfc8 in _IO_vtable_check () at vtables.c:72
#5  0x000000fff4061740 in IO_validate_vtable (vtable=0xfff3b646b0) at
libioP.h:838
#6  __GI___uflow (fp=0xfff38c8270) at genops.c:323
#7  0x000000fff4051380 in __GI__IO_getline_info (fp=fp@entry=0xfff38c8270,
buf=buf@entry=0xfffba99348 "\001", n=511, delim=<optimized out>,
extract_delim=<optimized out>, eof=eof@entry=0x0) at iogetline.c:60
#8  0x000000fff40514f8 in __GI__IO_getline (fp=fp@entry=0xfff38c8270,
buf=buf@entry=0xfffba99348 "\001", n=<optimized out>, delim=delim@entry=10,
extract_delim=extract_delim@entry=1) at iogetline.c:34
#9  0x000000fff404fbbc in _IO_fgets (buf=0xfffba99348 "\001", n=<optimized
out>, fp=0xfff38c8270) at iofgets.c:53
#10 0x000000fff45789cc in fgets (__stream=<optimized out>, __n=<optimized
out>, __s=<optimized out>) at
/production/ci-sw/ci/promotions/sysroot/ssw_stable_cc523583/eyeq5-4_19/sysroot/usr/include/bits/stdio2.h:265
#11 is_process_loading_address_valid () at src/contig_mm_os_device.c:47
#12 0x000000fff4569878 in on_library_load () at src/contig_mm_api.c:98
#13 0x000000fff47ae560 in call_init (l=<optimized out>, argc=argc@entry=72,
argv=argv@entry=0xfffba99698, env=env@entry=0xfffba998e0) at dl-init.c:72
#14 0x000000fff47ae6a4 in call_init (env=0xfffba998e0, argv=0xfffba99698,
argc=72, l=<optimized out>) at dl-init.c:30
#15 _dl_init (main_map=0xfff47d28c0, argc=<optimized out>,
argv=0xfffba99698, env=0xfffba998e0) at dl-init.c:119
#16 0x000000fff479e76c in _dl_start_user () from
/production/shared/compilers/gcc-7.4.0-mips-img-linux-gnu-2019.02-05/sysroot/mipsel-r6-hard/lib64/
ld-2.28.so
Backtrace stopped: frame did not save the PC
(gdb)
</backtrace>

Question: Can I assume this trace shows the second glibc copy being loaded
into the new namespace?

**The _IO_vtable_check() function in libio/vtables.c**

Here is the source code of the _IO_vtable_check() function (taken from
https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=libio/vtables.c;hb=e64a1e81aadf6c401174ac9471ced0f0125c2912)
which triggers the fatal error message:

<code>
503 void attribute_hidden
504 _IO_vtable_check (void)
505 {
506 #ifdef SHARED
507   /* Honor the compatibility flag.  */
508   void (*flag) (void) = atomic_load_relaxed
(&IO_accept_foreign_vtables);
509   PTR_DEMANGLE (flag);
510   if (flag == &_IO_vtable_check)
511     return;
512
513   /* In case this libc copy is in a non-default namespace, we always
514      need to accept foreign vtables because there is always a
515      possibility that FILE * objects are passed across the linking
516      boundary.  */
517   {
518     Dl_info di;
519     struct link_map *l;
520     if (!rtld_active ()
521         || (_dl_addr (_IO_vtable_check, &di, &l, NULL) != 0
522             && l->l_ns != LM_ID_BASE))
523       return;
524   }
525
526 #else /* !SHARED */
527   /* We cannot perform vtable validation in the static dlopen case
528      because FILE * handles might be passed back and forth across the
529      boundary.  Therefore, we disable checking in this case.  */
530   if (__dlopen != NULL)
531     return;
532 #endif
533
534   __libc_fatal ("Fatal error: glibc detected an invalid stdio
handle\n");
535 }
</code>

We're only interested in the "#ifdef SHARED" code block.  It is obvious
that the "if" expression on lines 520-522 is false, falling into the
__libc_fatal() call at the end.  Can anyone explain why this test might
succeed in the POC but fail in production?  They're both using the
identical glibc.so.

Based on the code's comment, lines 508-524 check whether the library should
accept "foreign" vtables (vtables from outside the library). If the library
is in a non-default namespace, it always accepts foreign vtables because
`FILE *` objects might be passed across the linking boundary.  The second
copy is indeed being loaded into a non-standard namespace, why is there a
problem?

*The original commit comment for vtables.c (23 June 2016)*

Here is the detailed commit comment when vtables.c was originally committed:

<quote>
commit db3476aff19b75c4fdefbe65fcd5f0a90588ba51
Author: Florian Weimer <fweimer@redhat.com>
Date:   Thu Jun 23 20:01:40 2016 +0200
    libio: Implement vtable verification [BZ #20191]
    This commit puts all libio vtables in a dedicated, read-only ELF
    section, so that they are consecutive in memory.  Before any indirect
    jump, the vtable pointer is checked against the section boundaries,
    and the process is terminated if the vtable pointer does not fall into
    the special ELF section.

    To enable backwards compatibility, a special flag variable
    (_IO_accept_foreign_vtables), protected by the pointer guard, avoids
    process termination if libio stream object constructor functions have
    been called earlier.  Such constructor functions are called by the GCC
    2.95 libstdc++ library, and this mechanism ensures compatibility with
    old binaries.  Existing callers inside glibc of these functions are
    adjusted to call the original functions, not the wrappers which enable
    vtable compatiblity.

    The compatibility mechanism is used to enable passing FILE * objects
    across a static dlopen boundary, too.
</quote>

**The GNU C Library Security Policy**

The GNU C Library Security Policy (
https://github.com/bminor/glibc/security#post-exploitation-countermeasures)
alludes to vtable validation for stdio stream handles:

<quote>
Post-exploitation countermeasures

Certain features have been added to the library only to make exploitation
of security bugs (mainly for code execution) more difficult. Examples
includes the stack smashing protector, function pointer obfuscation, *vtable
validation for stdio stream handles*, and various heap consistency checks.
Failure of such countermeasures to stop exploitation of a different
vulnerability is not a security vulnerability in itself. By their nature,
these countermeasures are based on heuristics and will never offer complete
protection, so the original vulnerability needs to be fixed anyway."
</quote>

**Select portions of running readelf on the MIPS glibc.so**

<quote>
 ELF Header:
 Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
 Class:                             ELF64
 Data:                              2's complement, little endian
 Version:                           1 (current)
 OS/ABI:                            UNIX - System V
 ABI Version:                       0
 Type:                              DYN (Shared object file)
 Machine:                           MIPS R3000
 Version:                           0x1
 Entry point address:               0x3df18
 Start of program headers:          64 (bytes into file)
 Start of section headers:          1823552 (bytes into file)
 Flags:                             0xa0000407, noreorder, pic, cpic,
nan2008, mips64r6
 Size of this header:               64 (bytes)
 Size of program headers:           56 (bytes)
 Number of program headers:         11
 Size of section headers:           64 (bytes)
 Number of section headers:         66
 Section header string table index: 65
</quote>

Here's a segment related to "vtable":

<quote>
Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES
Flg Lk Inf Al
  . . .
  [25] __libc_IO_vtables PROGBITS        00000000001abb88 19bb88 000dc8 00
 WA  0   0  8
</quote>

*cat /proc/self/maps before calling dlmopen()*

<quote>
 # cat /proc/self/maps
aaac4dc000-aaac5e0000 r-xp 00000000 00:02 9951
/bin/busybox
aaac5ec000-aaac5f0000 r--p 00100000 00:02 9951
/bin/busybox
aaac5f0000-aaac5f4000 rw-p 00104000 00:02 9951
/bin/busybox
fff4c98000-fff4cb8000 r-xp 00000000 00:02 9874
/lib/libpthread-2.28.so
fff4cb8000-fff4cc4000 ---p 00020000 00:02 9874
/lib/libpthread-2.28.so
fff4cc4000-fff4cc8000 rw-p 0001c000 00:02 9874
/lib/libpthread-2.28.so
fff4cc8000-fff4ccc000 rw-p 00000000 00:00 0
fff4ccc000-fff4e68000 r-xp 00000000 00:02 9873
/lib/libc-2.28.so
fff4e68000-fff4e74000 ---p 0019c000 00:02 9873
/lib/libc-2.28.so
fff4e74000-fff4e7c000 r--p 00198000 00:02 9873
/lib/libc-2.28.so
fff4e7c000-fff4e84000 rw-p 001a0000 00:02 9873
/lib/libc-2.28.so
fff4e84000-fff4e88000 rw-p 00000000 00:00 0
fff4e88000-fff4e8c000 r-xp 00000000 00:02 9867
/lib/libssp.so.0.0.0
fff4e8c000-fff4e98000 ---p 00004000 00:02 9867
/lib/libssp.so.0.0.0
fff4e98000-fff4e9c000 rw-p 00000000 00:02 9867
/lib/libssp.so.0.0.0
fff4e9c000-fff4eb0000 r-xp 00000000 00:02 9847
/lib/libresolv-2.28.so
fff4eb0000-fff4ec0000 ---p 00014000 00:02 9847
/lib/libresolv-2.28.so
fff4ec0000-fff4ec4000 rw-p 00014000 00:02 9847
/lib/libresolv-2.28.so
fff4ec4000-fff4ee8000 r-xp 00000000 00:02 8833
/usr/lib/libtirpc.so.3.0.0
fff4ee8000-fff4ef4000 ---p 00024000 00:02 8833
/usr/lib/libtirpc.so.3.0.0
fff4ef4000-fff4ef8000 rw-p 00020000 00:02 8833
/usr/lib/libtirpc.so.3.0.0
fff4ef8000-fff4f20000 r-xp 00000000 00:02 9883
/lib/ld-2.28.so
fff4f24000-fff4f2c000 rw-p 00000000 00:00 0
fff4f2c000-fff4f30000 rw-p 00024000 00:02 9883
/lib/ld-2.28.so
fffbb00000-fffbb24000 rw-p 00000000 00:00 0
 [stack]
fffbff0000-fffbff4000 r-xp 00000000 00:00 0
ffff3d8000-ffff3e0000 r--p 00000000 00:00 0
 [vvar]
ffff3e0000-ffff3e4000 r-xp 00000000 00:00 0
 [vdso]
</quote>