The glibc testsuite
Contents
Testsuite targets
Developers should use "make check" to test the built glibc. When running Testsuite in a cross-compiled setup, make check is necessary as it builds the testroot container [1].
A typical test case writes out a file with a .out extension in the build directory, that contains the output of the test. This file may be inspected in case of a test case failure to determine what the problem is. If you report a test case failure in bugzilla, be sure to include the contents of the relevant .out file as well.
There is no need to use "-k" when running "make check"; a failing test will not stop the testing. If make does not finish by printing "Summary of test results" then there is a problem in the testing process somewhere.
[1] https://sourceware.org/pipermail/libc-alpha/2020-March/111815.html
Architectures, specific tests, containerization
For requirements, failures, and background of specific tests, please see a separate page with notes.
Containerization can have a strong impact on the test results; this is also discussed on a separate page.
Test results are tracked on the pages of the specific glibc releases. In addition, there is a set of pages where results can be summarized by architecture.
Testing just one test
Run make help to see if you have support for single test testing. If you see test as a valid option then you can re-run a test with just e.g. make test t=wcsmbs/test-wcsnlen to run the test-wcsnlen test from the wcsmbs subdirectory. Otherwise follow the instructions to run the check target in just one directory (runs all tests in that directory). Due to limits in the test framework the top-level tests in the top-level Makefile cannot be run with make test t= and instead can only be run with with make check.
To test just one test again start by removing the singular *.out file for the test of interest and rerun the testing. The infrastructure will only re-run the tests with the missing output file.
To accelerate this process and limit it to a single directory worth of tests you have to run with -C <path to glibc source>/<top-level directory tests to run e.g. make -r PARALLELMFLAGS="-j4" -C /home/carlos/src/glibc/inet objdir=`pwd` check runs the standard tests for just the inet top-level directory.
Alternatively, you may do make subdirs=login -j8 check to run all the tests in the login subdirectory in parallel. Note that -jN is automatically ignored in the nptl directory since several of those tests use multi-threading.
ABI check
The glibc testsuite contains a number of tests to check that the ABI of glibc does not change. It compares symbol names and versions and static variable sizes in generated binaries with those maintained in *.abilist in the source. The test runs as part of 'make check'. You can also run these separately via "make check-abi".
In order to increase the quality of ABI analysis one can run the following additional tests:
Writing a test case
Use the test driver (in support/test-driver.c) to launch your test case. To do this, you would typically write your test case as:
/* This is your test case 'main' method. */
static int
do_test (void)
{
/* Test goes here. */
}
#include <support/test-driver.c>A minimal complete example can be found in the file support/README-testing.c.
- Diagnostic or informative messages in the test case should be printed to standard output. This includes messages that flag a test case failure.
- If you want to print a warning in the test case that is not a failure, then print it also to standard output. This could be used for cases like skipping a test if a certain feature is unavailable. We don't want to print to stderr because it looses the ordering between stdout and stderr, is lost in the output of the testsuite run, and isn't a useful distinction. That is to say that stdout and stderr distinction is less useful than the distinction between PASS and FAIL.
It is recommended to use info:, warning:, and error: or FAIL prefixes in diagnostic output, to clearly indicate which output reflects an error condition.
You can include <support/test-driver.h> and use the test_verbose variable to check if the test program was run with the --verbose argument, to avoid printing to many diagnostic messages by default.
If you print an error message, you need to make sure that you indicate failure. You can do this by calling exit (1) (which may not work after a fork). If you want to continue execution, call support_record_failure () from <support/check.h> (which works after a fork because it uses a shared memory segment, but it will not work after an execve or from a dynamic shared object). Do not use abort or assert.
EXIT_UNSUPPORTED from <support/test-driver.h> is a magic exit status (77) which indicates that the test is not supported on this particular system (perhaps due to lack of hardware or kernel support detected at run time).
The test driver in <support/test-driver.c> offers some hooks for customization, such as custom argument list parsing. For details, see the documentation in the source file.
Testing with a cross-compiler
While building and testing glibc on a system with native tools is the simplest way to do glibc testing, it is possible to build a cross-compilation toolchain that includes glibc and run the glibc testsuite using that toolchain.
To do cross-compilation testing you need to use the scripts/cross-test-ssh.sh script in the glibc sources and do the build on the host machine in a directory that is also visible, with the same path, on the target machine. It is also necessary to be able to use ssh to access the target machine from the host machine. There are details on this in the cross-test-ssh.sh script. This script can be used to run "make check", "make xcheck", or "make bench".
Because both the host and target machines are touching files during the make, changes on one machine need to be immediately visible on the other machine. This is not the default behaviour with NFS due to caching. You may need to use the 'noac' option on NFS in order to get this to work. This problem can cause messages similar to the following when running make:
touch: cannot touch `/home/sellcey/gcc/gcc_cross_testing/obj-mips-mti-linux-gnu/glibc/obj_default/localedata/de_DE.ISO-8859-1/LC_CTYPE': No such file or directory
If you build a cross-toolchain by building binutils, an initial GCC, glibc, and then rebuild GCC (a common build sequence), then running the glibc testsuite may result in glibc getting rebuilt before any tests are run. This is because the glibc tests have make dependencies on glibc and glibc has make dependencies on GCC (or at least some GCC headers). If your second GCC build overwrites the first one then make will see this change and rebuild many parts (if not all) of glibc before doing any testing. If you know you are going to run the glibc testsuite you may want to rebuild glibc after rebuilding GCC and before running "make check". You cannot skip the second GCC build because the initial GCC (if configured using --without-headers) will not have all the headers and libraries needed to build the glibc test programs.