Differences between revisions 17 and 18
Revision 17 as of 2016-02-29 18:48:14
Size: 7484
Editor: SteveEllcey
Comment:
Revision 18 as of 2016-05-30 20:35:31
Size: 7925
Comment: Update to mention how to run tests in one directory.
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
== Testsuite targets ==
Line 6: Line 8:

To repeat a test that failed, simply remove the .out file associated with that test, since the presence of the file in the build directory (with a more recent timestamp) implies that the test has already been run.
Line 16: Line 16:

== Testing just one test ==

To test just one test you have to have already run the entire testsuite (creates output files for all the tests). Next you remove the singular `*.out` file for the test of interest and rerun the testing. The infrastructure will only re-run the tests with the missing output file.

To accelerate this process and limit it to a single directory worth of tests you have to run with `-C <path to glibc source>/<top-level directory tests to run` e.g. {{{make -r PARALLELMFLAGS="-j4" -C /home/carlos/src/glibc/inet objdir=`pwd` check}}} runs the standard tests for just the `inet` top-level directory.

The glibc testsuite

Testsuite targets

A typical test case writes out a file with a .out extension in the build directory, that contains the output of the test. This file may be inspected in case of a test case failure to determine what the problem. If you report a test case failure in bugzilla, be sure to include the contents of the relevant .out file as well.

There's an additional test target: make xcheck. Running make xcheck includes all the tests of make check but adds some additional tests. These additional tests have requirements on the execution environment that are available for normal execution, e.g. network connectivity, but not necessarily for builds as part of build systems.

Developers should use "make xcheck" instead of "make check".

There is no need to use "-k" when running "make check" or "make xcheck", a failing test will not stop the testing. If the make does not finish by printing "Summary of test results" then there is a problem in the testing process somewhere.

Testing just one test

To test just one test you have to have already run the entire testsuite (creates output files for all the tests). Next you remove the singular *.out file for the test of interest and rerun the testing. The infrastructure will only re-run the tests with the missing output file.

To accelerate this process and limit it to a single directory worth of tests you have to run with -C <path to glibc source>/<top-level directory tests to run e.g. make -r PARALLELMFLAGS="-j4" -C /home/carlos/src/glibc/inet objdir=`pwd` check runs the standard tests for just the inet top-level directory.

ABI check

The glibc testsuite contains a number of tests to check that the ABI of glibc does not change. It compares symbol names and versions and static variable sizes in generated binaries with those maintained in *.abilist in the source. The test runs as part of 'make check'. You can also run these separately via "make check-abi".

In order to increase the quality of ABI analysis one can run the following additional tests:

Details about make xcheck specific tests

The list below describes the requirements on the execution environment of the tests that are part of only make xcheck:

  1. resolv/tst-leaks2 nss/bug-erange.c posix/bug-ga2.c
    Requires enough network connectivity to do a DNS lookup on www.gnu.org

  2. nptl/tst-setuid1.c and nptl/tst-setuid1-static.c
    Requires user "nobody" in /etc/passwd or similar, and the ability as the current user to setuid to it.

  3. nptl/tst-mutexpp1.c nptl/tst-mutexpp6.c nptl/tst-mutexpp10.c
    Requires enough privileges to set mutex priority ceiling via pthread_mutexattr_setprioceiling

  4. sunrpc/tst-getmyaddr.c
    Requires at least one non-loopback IP address configured on the system.

  5. sunrpc/thrsvc.c
    Requires loopback to be up and operational with IP address 127.0.0.1

  6. iconv/test-iconvconfig
    Requires an existing gconv-modules.cache to be installed under $(inst_gconvdir)

Writing a test case

  • Use the test skeleton ($(topsrcdir)/test-skeleton.c) to launch your test case. To do this, you would typically write your test case as:

  /* This is your test case 'main' method.  */
  static int
  do_test (void)
  {
    /* Test goes here.  */
  }
  
  #define TEST_FUNCTION do_test ()
  #include "../test-skeleton.c"

If you don't define TEST_FUNCTION, then your do_test is considered to have the following signature:

int do_test (int argc, char *argv[])
  • Diagnostic or informative messages in the test case should be printed to standard output. This includes messages that flag a test case failure.
  • If you want to print a warning in the test case that is not a failure, then print it also to standard output. This could be used for cases like skipping a test if a certain feature is unavailable. We don't want to print to stderr because it looses the ordering between stdout and stderr, is lost in the output of the testsuite run, and isn't a useful distinction. That is to say that stdout and stderr distinction is less useful than the distinction between PASS and FAIL.

Testing with a cross-compiler

While building and testing glibc on a system with native tools is the simplest way to do glibc testing, it is possible to build a cross-compilation toolchain that includes glibc and run the glibc testsuite using that toolchain.

To do cross-compilation testing you need to use the scripts/cross-test-ssh.sh script in the glibc sources and do the build on the host machine in a directory that is also visible, with the same path, on the target machine. It is also necessary to be able to use ssh to access the target machine from the host machine. There are details on this in the cross-test-ssh.sh script. This script can be used to run "make check", "make xcheck", or "make bench".

Because both the host and target machines are touching files during the make, changes on one machine need to be immediately visible on the other machine. This is not the default behaviour with NFS due to caching. You may need to use the 'noac' option on NFS in order to get this to work. This problem can cause messages similar to the following when running make:

touch: cannot touch `/home/sellcey/gcc/gcc_cross_testing/obj-mips-mti-linux-gnu/glibc/obj_default/localedata/de_DE.ISO-8859-1/LC_CTYPE': No such file or directory

If you build a cross-toolchain by building binutils, an initial GCC, glibc, and then rebuild GCC (a common build sequence), then running the glibc testsuite may result in glibc getting rebuilt before any tests are run. This is because the glibc tests have make dependencies on glibc and glibc has make dependencies on GCC (or at least some GCC headers). If your second GCC build overwrites the first one then make will see this change and rebuild many parts (if not all) of glibc before doing any testing. If you know you are going to run the glibc testsuite you may want to rebuild glibc after rebuilding GCC and before running "make check". You cannot skip the second GCC build because the initial GCC (if configured using --without-headers) will not have all the headers and libraries needed to build the glibc test programs.

Known testsuite failures

abi-check

You might see a check failure due to a different size for _nl_default_dirname if you build for a different prefix using the --prefix configure option. The size of _nl_default_dirname depends on the prefix and /usr/share/locale is considered the default and hence the value 0x12. If you see such a difference, you should check that the size corresponds to your prefix, i.e. (length of prefix path + 1) to ensure that you haven't really broken abi with your change.

bug-atexit3 and nptl tests

These failures are again seen when a build is configured for a non-standard prefix (i.e. not /usr). This occurs because the built dynamic linker is unable to find libstdc++ and libgcc_s.so, since its default search path is the build directory with a fallback to the ld.so.cache in the prefix directory i.e. $prefix/etc/ld.so.cache. One could work around these failures in one of the following ways:

  • Preload libstdc++ and libgcc_s using the LD_PRELOAD environment variable

  • Create a symlink to libstdc++ and libgcc_s in the build directory

  • Generate an ld.so.cache that caches the location of libstdc++ and libgcc_s and place it in $prefix/etc/

tst-eintr1

One may see this failure sporadically. This happens because the kernel fails to reap exiting threads fast enough, eventually resulting an EAGAIN when pthread_create is called within the test. This is currently seen as a kernel limitation.

None: Testing/Testsuite (last edited 2016-05-30 20:35:31 by CarlosODonell)