Contents

  1. Testing a glibc build
    1. Building glibc without installing
    2. Building glibc with intent to install
    3. Compile normally, run under new glibc
    4. Compile against glibc build tree
    5. Compile against glibc in an installed location
    6. Required gdb setup
    7. Building with completely new files

Testing a glibc build

So you want to test out a glibc build, and you don't want to install it over your existing system glibc. Not installing glibc over your existing system glibc is the normal process. Installing a new glibc over your system glibc may break your system, do not do this unless you really know what you're doing.

You need to do two things:

First we go through the two ways to build glibc:

Building glibc without installing

To build glibc without installing you can do the standard configure and make e.g. 

$ mkdir $HOME/src
$ cd $HOME/src
$ git clone git://sourceware.org/git/glibc.git
$ mkdir -p $HOME/build/glibc
$ cd $HOME/build/glibc
$ $HOME/src/glibc/configure --prefix=/usr
$ make
$ make check

Do not run make install.

By building with the prefix /usr you have created a glibc that will load and use all configuration files from the standard locations. The prefix /usr is considered the correct prefix for a system glibc. More advanced users will note that the prefix is actually part of the glibc ABI.

Lastly if you want to build with an alternate set of Linux kernel headers you will need to use --with-headers= to point to a unified installation of Linux headers and other headers required by glibc during the build which may or may not include SELinux headers (--with-selinux) or NSS headres (--enable-nss-crypt).

Building glibc with intent to install

Do configure with a standard --prefix=/usr directory, then make and make install with the DESTDIR GNU standard variable set to some temporary installation directory, e.g. 

$ DESTDIR=<path to the GLIBC install directory>
$ mkdir $HOME/src
$ cd $HOME/src
$ git clone git://sourceware.org/git/glibc.git
$ mkdir -p $HOME/build/glibc
$ cd $HOME/build/glibc
$ $HOME/src/glibc/configure --prefix=/usr
$ make
$ make check
$ make install DESTDIR=${DESTDIR}
$ make localedata/install-locales DESTDIR=${DESTDIR}
$ make localedata/install-locale-files DESTDIR=${DESTDIR}

You now have the newly built glibc installed in subdirectories of $DESTDIR and you can build applications that run against it. Keep in mind that the build will reference configuration files relative to the '--prefix configured path of /usr.

Then you will need a set of linux kernel headers: 

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux
$ make headers_install INSTALL_HDR_PATH="${DESTDIR}/usr"

Lastly you will need gcc's helper library for cancellation: 

$ cp /lib64/libgcc* "${DESTDIR}/lib64/"

This leaves you with a ready to use system root in ${DESTDIR}.

You will need to use lib on 32-bit systems or 64-bit systems that use lib like Ubuntu.

Compile normally, run under new glibc

Use this if you want to be able to run your application easily with the newly built glibc and then again with the system glibc for comparison.

Compile your test program, and invoke it with the new loader you built. This is also how the make check runs tests. Note: if the executable under test resides in the current directory, don't forget to use ./ before the name.

Use either testrun.sh in the build directory e.g. 

$ cd $HOME/build/glibc
$ ./testrun.sh /path/to/test/application

Or do this manually like this: 

GLIBC=<path to the GLIBC build directory>

GCONV_PATH=${GLIBC}/iconvdata LC_ALL=C     \
${GLIBC}/elf/ld.so.1 --library-path \
${GLIBC}:\
${GLIBC}/math:\
${GLIBC}/elf:\
${GLIBC}/dlfcn:\
${GLIBC}/nss:\
${GLIBC}/nis:\
${GLIBC}/rt:\
${GLIBC}/resolv:\
${GLIBC}/crypt:\
${GLIBC}/nptl:\
${GLIBC}/dfp \
<executable to test> <arguments>

Please note that such a compilation does not make use of new C runtime objects i.e. crt1.o, crti.o, and crtn.o provided by glibc. Changes made to these objects require a more complex compilation, see later instructions for the details. Compiling with -Wl,-Map,linker.map will show you exactly which objects were used in the final link.

Compile against glibc build tree

Use this method if you want to easily debug your application but haven't installed glibc. 

GLIBC=<path to the GLIBC build directory>

gcc \
  -Wl,-rpath=${GLIBC}:\
${GLIBC}/math:\
${GLIBC}/elf:\
${GLIBC}/dlfcn:\
${GLIBC}/nss:\
${GLIBC}/nis:\
${GLIBC}/rt:\
${GLIBC}/resolv:\
${GLIBC}/crypt:\
${GLIBC}/nptl:\
${GLIBC}/dfp \
  -Wl,--dynamic-linker=${GLIBC}/elf/ld.so
  <other compiler flags> -o <application> <application>.c

Please note that such a compilation does not make use of new headers or C runtime objects i.e. crt1.o, crti.o, and crtn.o provided by glibc. Changes made to headers or objects require a more complex compilation, see later instructions for the details. Compiling with -Wl,-Map,linker.map will show you exactly which objects were used in the final link.

Compile against glibc in an installed location

Please note that the installed glibc is an incomplete C runtime. In order to complete the C runtime you may need to copy in additional headers that match the compiler you are using since the use of --sysroot will restrict their lookup to the sysroot.

Use this method if you want to easily debug your application.

Compile your test program and give some extra options to gcc to use the install directory, and some options to the linker to set the shared library search path and dynamic linker. It is a good idea to verify the location of the dynamic linker using "readelf" and the library search paths with "ldd". (See loader tips and tricks) 

SYSROOT=<path to the GLIBC install directory>
gcc \
  -L${SYSROOT}/usr/lib64 \
  -I${SYSROOT}/include \
  --sysroot=${SYSROOT} \
  -Wl,-rpath=${SYSROOT}/lib64 \
  -Wl,--dynamic-linker=${SYSROOT}/lib64/ld-2.18.90.so\
  <other compiler flags> -o <application> <application>.c

The built application will now always use the dynamic loader and libraries from the paths you compiled it with.

If your static linker lacks sysroot support you can try this instead: 

SYSROOT=<path to the GLIBC install directory>
gcc \
  -L${SYSROOT}/usr/lib64 \
  -I${SYSROOT}/include \
  -Wl,-rpath=${SYSROOT}/lib64 \
  -Wl,--dynamic-linker=${SYSROOT}/lib64/ld-2.18.90.so \
  <other compiler flags> -o <application> <application>.c

You will need to use lib on 32-bit systems or 64-bit systems that use lib like Ubuntu.

You will need to adjust ld-2.18.90.so for the version of glibc.

Required gdb setup

Debugging programs using a new glibc build requires a couple of additional steps if the program is multi-threaded or if environment variables are to be passed to the program. The glibc build system provides a helper script for such purposes, so, after running make and make check:

Use either debugglibc.sh in the build directory e.g. 

$ cd $HOME/build/glibc
$ ./debugglibc.sh /path/to/test/application

Or perform these steps manually using the instructions in the following subsections:

Thread Setup

One special step needs to be taken to debug a threaded application using a new glibc build with gdb. The thread db library must be explicitly set. The thread db library allows the debugger to inspect various internal library states (including threading) and is needed for thread debugging. The debugger can't easily guess the correct version and location of the thread db library. If you installed glibc the debugger will often find libthread_db.so without any problems, but if you didn't install glibc then you will certainly have problems trying to debug threaded code using the system library. Therefore it's always safest to specify exactly where to search for the thread db library.

Execute this in gdb before debugging: 

set auto-load safe-path <path to libthread_db.so.1 e.g. /build/glibc/nptl_db or /install/lib64>:$debugdir:$datadir/auto-load
set libthread-db-search-path <path to libthread_db.so.1 e.g. /build/glibc/nptl_db or /install/lib64/>

Environment Setup

Passing environment variables to the application to debug is not always as straight forward as you might expect. Some environment variables may negatively impact the shell that is used by gdb to start the application. In these cases you should use exec-wrapper to ensure that only the debugged application has the intended environment variables set.

Execute this in gdb before debugging: 

set exec-wrapper env 'LD_PRELOAD=libmalloc-extras.so'

Debugging a Test Case

The test cases that are run in glibc (i.e. via make check) typically need to be run in a glibc-specific way, and some need to run inside a test container. To debug these, glibc provides a special makefile target: 

WAIT_FOR_DEBUGGER=1 make test t=nss/tst-nss-test2

When the test is run this way, it will pause just before running the body of the test, and print instructions for attaching to it via gdb.

Note that omitting the WAIT_FOR_DEBUGGER=1 part will simply re-run the single specified test.

Building with completely new files

As mentioned earlier it is very complicated to build an application using the newly built crt files. Likewise static builds that use only completely new files is also hard. In general the process taken involves building the application using the system compiler with -v, reviewing the static linker map file from -Map, and then tweaking the link line to use all the new objects you have just built.

The following script gives a template that might help those users that need to test changes to the crt files, or that want to build statically using all the newly built pieces available from the runtime.

These instructions assume you have built but not installed glibc. It is easier to do this if you have a fully built system root because you would just use --sysroot (you must do this if you want to new the C library headers), but assume you don't have that and these instructions will help you assemble the application. We provide an example with threads because that's most common, you can add in any other glibc library right after the thread libraries in the final link line.

Tested on Fedora 23. 

# glibc pieces:
BUILD=/home/carlos/build/glibc
DYNLINKER=--dynamic-linker="${BUILD}"/elf/ld.so
LINKPATH=-rpath="${BUILD}":"${BUILD}"/math:"${BUILD}"/elf:"${BUILD}"/dlfcn:"${BUILD}"/nss:"${BUILD}"/nis:"${BUILD}"/rt:"${BUILD}"/resolv:"${BUILD}"/crypt:"${BUILD}"/mathvec:"${BUILD}"/nptl
CRT1="${BUILD}"/csu/crt1.o
CRT1_PIE="${BUILD}"/csu/Scrt1.o
CRTI="${BUILD}"/csu/crti.o
CRTN="${BUILD}"/csu/crtn.o

# gcc pieces:
CRTBEGIN_STATIC=$(gcc -print-file-name="crtbeginT.o")
CRTBEGIN_PIE=$(gcc -print-file-name="crtbeginS.o")
CRTBEGIN_NORMAL=$(gcc -print-file-name="crtbegin.o")
CRTEND=$(gcc -print-file-name="crtend.o")
CRTEND_PIE=$(gcc -print-file-name="crtendS.o")
GCCINSTALL=$(gcc -print-search-dirs | grep 'install:' | sed -e 's,^install: ,,g')
LD=$(gcc -print-prog-name="collect2")

# Application pieces:
PROG_NAME_STATIC=threads-static
PROG_NAME_NORMAL=threads-normal
PROG_NAME_PIE=threads-pie
PROG_SOURCE=threads.c
PROG_OBJ=threads.o
PROG_OBJ_PIE=threads.os
MAP_STATIC=mapfile-static.txt
MAP_NORMAL=mapfile-normal.txt
MAP_PIE=mapfile-pie.txt
CFLAGS="-g3 -O0"

# Compile the application.
rm -f $PROG_NAME_STATIC
rm -f $PROG_NAME_NORMAL
rm -f $PROG_NAME_PIE
rm -f $PROG_OBJ
rm -f $PROG_OBJ_PIE
rm -f $MAP_STATIC
rm -f $MAP_NORMAL
rm -f $MAP_PIE

# Once for static and normal builds and once for shared (PIE).
# These compilations still use the old C library headers.
gcc $CFLAGS -c $PROG_SOURCE -o $PROG_OBJ
gcc -pie -fPIC $CFLAGS -c $PROG_SOURCE -o $PROG_OBJ_PIE

# Link it against a hybrid combination of:
# - Newly build glibc.
# - Split out libpthread because the .so is a linker script.
# - C development environment present on the system.
# Notes:
# - LTO is not supported.
# - Profiling is not supported (-pg).
# - Only works for gcc.
# - Only works for x86_64.
# - Assumes we are using only the first and default multlib.

# Static build:
$LD --build-id --no-add-needed --hash-style=gnu -m elf_x86_64 -static -o \
$PROG_NAME_STATIC $CRT1 $CRTI $CRTBEGIN_STATIC \
-L$GCCINSTALL \
-L$GCCINSTALL/../../../../lib64 \
-L/lib/../lib64 \
-L/usr/lib/../lib64 \
-L$GCCINSTALL/../../.. \
-Map $MAP_STATIC \
$PROG_OBJ \
$BUILD/nptl/libpthread.a \
--start-group -lgcc -lgcc_eh $BUILD/libc.a --end-group \
$CRTEND $CRTN

# Normal build:
$LD --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 \
$DYNLINKER $LINKPATH  -o \
$PROG_NAME_NORMAL $CRT1 $CRTI $CRTBEGIN_NORMAL \
-L$GCCINSTALL \
-L$GCCINSTALL/../../../../lib64 \
-L/lib/../lib64 \
-L/usr/lib/../lib64 \
-L$GCCINSTALL/../../.. \
-Map $MAP_NORMAL \
$PROG_OBJ \
${BUILD}/nptl/libpthread.so.0 ${BUILD}/nptl/libpthread_nonshared.a \
-lgcc --as-needed -lgcc_s --no-as-needed \
${BUILD}/libc.so.6 ${BUILD}/libc_nonshared.a --as-needed ${BUILD}/elf/ld.so --no-as-needed \
-lgcc --as-needed -lgcc_s --no-as-needed \
$CRTEND $CRTN

# PIE build:
$LD --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 \
$DYNLINKER $LINKPATH -pie -o \
$PROG_NAME_PIE $CRT1_PIE $CRTI $CRTBEGIN_PIE \
-L$GCCINSTALL \
-L$GCCINSTALL/../../../../lib64 \
-L/lib/../lib64 \
-L/usr/lib/../lib64 \
-L$GCCINSTALL/../../.. \
-Map $MAP_PIE \
$PROG_OBJ_PIE \
${BUILD}/nptl/libpthread.so.0 ${BUILD}/nptl/libpthread_nonshared.a \
-lgcc --as-needed -lgcc_s --no-as-needed \
${BUILD}/libc.so.6 ${BUILD}/libc_nonshared.a --as-needed ${BUILD}/elf/ld.so --no-as-needed \
-lgcc --as-needed -lgcc_s --no-as-needed \
$CRTEND_PIE $CRTN

None: Testing/Builds (last edited 2023-07-07 15:38:34 by CarlosODonell)