Re: Smoke test for glibc

[Rich Felker <dalias at aerifal dot cx>]

On Mon, Oct 22, 2012 at 09:23:13AM -0400, Carlos O'Donell wrote:
> On Mon, Oct 22, 2012 at 9:00 AM, Andrey Ponomarenko
> <aponomarenko@rosalab.ru> wrote:
> > I've recently written an article [1] about smoke testing of Glibc API using
> > a modern approach. This kind of test is needed to verify that most of the
> > library functions pass a simple test in the particular environment, i.e.
> > don't crash, don't emit any signal, don't hang and don't exit with non-zero
> > error code. The test generator takes only a path to the installed glibc tree
> > in order to compose input arguments for all functions (reasonable in most,
> > but unfortunately not all, cases) and then create, build and run the test
> > cases.
> >
> > I've added a link to this article on the Release Process page (section 5.6.
> > Testing), so that glibc maintainers can run this additional test when
> > preparing the release and look at the behaviour of all functions under the
> > smoke test. The sample test report for 2.16 looks like this one [2].
> 
> Andrey,
> 
> Thanks! This looks very interesting.
> 
> Will you continue to work on getting these results clean for glibc?
> 
> Going through 154 failed testcases from the smoke test is not something
> that I want a release manager to be doing.

I did a quick look through them, and a lot come from the test
generator not knowing how to satisfy the interface contracts. For
instance, it's passing {0,1,2,3,4,5} to longjmp and then getting
surprised when that causes a crash.

Further, it's passing uninitialized memory to at least some functions,
which can't be meaningful unless they only use the pointer for output.
For example, the hang in pthread_spin_lock is simply due to passing a
pointer to an indeterminate value.

In order to be meaningful, this sort of testing can only be applied to
functions for which the entire input domain is valid (i.e. mainly
mathematical functions; certainly nothing that takes a pointer), or it
must be enhanced to understand the interface contracts and satisfy
them.

Rich