• Testing
• LSB_Core

LSB Core Tests

This article describes how to run the official LSB Core test suite (OLVER) against the local development version of glibc. The test suite covers 1193 glibc functions (see LSB 12.3. "Interfaces for libc") and checks them for conformance with LSB, SUSv2, SUSv3, SUSv4, LFS and SVID.4 standards. The test suite is a part of the LSB certification tests for Linux distributions.

1. Installation

The test suite requires LSB SDK, LSB support package (e.g. redhat-lsb in Fedora, lsb in Debian, etc.), GCC, Perl 5, JDK 6, zlib, ncurses, Linux-PAM (development files), GNU Automake and Libtool to be installed on your host.

By default the test suite checks glibc installation from /usr system directory. But it's not convenient for testers because installing latest glibc to /usr is very difficult and dangerous operation for the system. Unfortunately, the test suite is not configurable enough yet to easily switch the path of tested glibc installation. This section describes a workaround, that allows you to run the test suite against the local glibc install tree.

Install glibc to a local directory (/home/user/glibc-2.17 for example)

$ wget http://ftp.gnu.org/gnu/glibc/glibc-2.17.tar.gz
$ tar -xzf glibc-2.17.tar.gz
$ cd glibc-2.17
$ mkdir build
$ cd build
$ ../configure --prefix=/usr
$ make
$ make install install_root=/home/user/glibc-2.17

Install LSB SDK

$ wget http://ftp.linuxfoundation.org/pub/lsb/bundles/released-4.1.0/sdk/lsb-sdk-4.1.5-1.ia32.tar.gz
$ tar -xzf lsb-sdk-4.1.5-1.ia32.tar.gz
$ cd lsb-sdk
$ sudo ./install.sh

Install lsbcc compiler to a local directory (/home/user/local-sdk for example)

$ bzr branch http://bzr.linuxfoundation.org/lsb/devel/build_env
$ cd build_env/lsbdev-cc
$ LSBCC_LSBVERSION=4.1 INSTALL_ROOT=/home/user/local-sdk  make
$ LSBCC_LSBVERSION=4.1 INSTALL_ROOT=/home/user/local-sdk  make install

Configure LSB SDK

$ sudo mv /opt/lsb/bin/lsbcc /opt/lsb/bin/lsbcc-orig

$ sudo vim /opt/lsb/bin/lsbcc
    
    #!/bin/bash
    /home/user/local-sdk/lsbcc -Wl,-rpath=/home/user/glibc-2.17/lib $@

$ sudo chmod +x /opt/lsb/bin/lsbcc
$ sudo ln -sf /home/user/glibc-2.17/lib/ld-linux.so.2 /lib/ld-lsb.so.3

Checkout the test suite

$ bzr branch http://bzr.linux-foundation.org/lsb/devel/olver-core-tests

Build the test suite

$ cd olver-core-tests
$ sudo ./configure.sh
$ ./build_conf_tests.sh
$ sudo ./install.sh

2. Usage

2.1. Run tests

The list of scenarios to run is bin/testplan. Some non-glibc scenarios can be removed from the testplan: util_pam_scenario (libpam), util_compress_scenario (libz), ncurses_* (libncurses). Some scenarios are disabled by default and you can enable them back by removing leading '#'.

To execute tests run (may take several hours)

$ ./bin/olver_run_tests

2.2. Reports

Summary report - shows number of executed scenarios, tested functions, failed requirements and divides found problems into groups of known problems and new ones

/var/opt/lsb/test/olver-core/DATE/summary.htm

Execution report - shows information on executed scenarios

/var/opt/lsb/test/olver-core/DATE/report/index.html

Requirements coverage - shows covered requirements of standards

/var/opt/lsb/test/olver-core/DATE/result.htm

TET report - machine readable report in the TET format

/var/opt/lsb/test/olver-core/DATE/tet/nice_tet.log

See detailed tutorial on OLVER reports in "OLVER Reports: Reference Guide".

2.3. Debugging

The test suite has a model-based architecture. Each target function is called by the agent program (./bin/agent). The model program (./bin/olver) tells the agent program which arguments should be passed to the function and receives its return value(s) back. Then the model checks constraints on the return value, parameters and the state of function.

Click on the failure to see description of failed requirement (Requirement failed: {localtime.01} The localtime() function shall convert the time in seconds into a local time for example), name of the scenario (time_conversion_scenario for example) and name of the specification function (localtime_spec() for example). You can find the implementation of the specification function under the ./src/model directory (*_model.sec file) and the code of the requirement by its id.

To debug the agent program you should run the model first in one terminal

$ ./bin/olver -c ./etc/olver.conf SCENARIO

The SCENARIO is one of the scenarios from the testplan (e.g. math_integer_scenario).

Then run the agent program under the debugger in the second terminal

$ gdb ./bin/agent
$ b FUNCTION
$ run
$ ...

The FUNCTION is one of the tested functions in the target SCENARIO (e.g. abs, div, etc.).

3. Current results

This section describes current test results for various glibc versions on Fedora host.

The test suite already has its own database of known problems, so it automatically divides found problems into a group of known problems and a group of new problems in the summary report.

3.1. glibc 2.19

Fixed 5 problems: bug516_2(cexp), bug841(cproj), bug504(sched_setscheduler), bug808(strptime), bug810(strptime). See this page for more info on these bugs.

Fixed 16 requirements: math.cexp.cexp{,f,l}.{01, 02}, math.complex.cproj{,f,l}.01, util.crypt.crypt.09, util.format.time.strptime.{06.01, 06.23, 06.25, 06.29, 12.01, 17.02}.

3.2. glibc 2.18

Fixed problems bug561(wcsftime), bug543(hypotl), bug512(tgamma), bug511_1(initstate) and bug803(getdate). Fixed requirements: math.exp.hypotl.08, math.gamma.tgamma.11.01, math.rand.initstate.11 and util.format.time.getdate.{01, 03, 04, 05, 07, 13, 18.01}.

New problem (!): failed requirement pthread_mutexattr_gettype.01.01

4. Known problems

TODO: new problems should be analyzed by maintainers and the result should be recorded here

util_dl_scenario [dlsym.03] Requirement failed: {dlsym.03} correct symbol not found

#include <dlfcn.h>

int main()
{
    void* h = dlopen(NULL, RTLD_NOW | RTLD_GLOBAL);
    void* r = dlsym(h, "authnone_create");
    
    return 0;
}

The issue is that dlsym cannot find a symbol authnone_create in the libc. There is only authnone_create\@GLIBC_2.0 symbol. The problem is not reproducible on glibc versions older than 2.14.

util_crypt_scenario [crypt.09] Requirement failed: Unexpected error code returned: [EINVAL] code 0x16

#include <unistd.h>
#include <errno.h>

int main()
{
    int e1 = 0;
    int e2 = 0;
    int r = 0;
    
    e1 = errno;
    r = crypt("Key", "a");
    e2 = errno;
    
    return 0;
}

The issue is that the new implementation of the crypt function in glibc 2.17 requires the salt parameter to be longer than two characters and returned an error code EINVAL (22). This error code is not described in the standard for this function.

5. Samples

1. Test report for glibc 2.17 in Fedora 18 on x86 (Linux 3.6.10-4.fc18.i686.PAE)

2. Test report for glibc 2.17 in ROSA 2012 on x86 (Linux 3.0.28-nrj-desktop-2rosa.lts)