[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

12.5 Adding a Test Suite

Automake has very flexible support for automated test-suites within a project distribution, which are discussed more fully in the Automake manual. I have added a simple shell script based testing facility to Sic using this support – this kind of testing mechanism is perfectly adequate for command line projects. The tests themselves simply feed prescribed input to the uninstalled sic interpreter and compare the actual output with what is expected.

Here is one of the test scripts:

 

The tricky part of this script is the first part which discovers the location of (and loads) ‘$srcdir/defs’. It is a little convoluted because it needs to work if the user has compiled the project in a separate build tree – in which case the ‘defs’ file is in a separate source tree and not in the actual directory in which the test is executed.

The ‘defs’ file allows me to factor out the common definitions from each of the test files so that it can be maintained once in a single file that is read by all of the tests:

 

Having written a few more test scripts, and made sure that they are working by running them from the command line, all that remains is to write a suitable ‘Makefile.am’ so that automake can run the test suite automatically.

 

I have used the ‘testsubdir’ macro to run the tests in their own subdirectory so that the directory containing the actual test scripts is not polluted with lots of fallout files generated by running the tests. For completeness I have used a hook target(26) to remove this subdirectory when the user types:

 
$ make distclean
...
rm -rf testSubDir
...

Adding more tests is accomplished by creating a new test script and adding it to the list in noinst_SCRIPTS. Remembering to add the new ‘tests’ subdirectory to ‘configure.in’ and the top-level ‘Makefile.am’, and reconfiguring the project to propagate the changes into the various generated files, I can run the whole test suite from the top directory with:

 
$ make check

It is often useful run tests in isolation, either when developing new tests, or to examine more closely why a test has failed unexpectedly. Having set this test suite up as I did, individual tests can be executed with:

 
$ VERBOSE=1 make check TESTS=incomplete.test
make  check-TESTS
make[1]: Entering directory
/tmp/sic/tests
=== Running test incomplete.test
1
2
3
PASS: incomplete.test
==================
All 1 tests passed
==================
make[1]: Leaving directory /tmp/sic/tests
$ ls testSubDir/
err   errok   in.sic   ok   out

The ‘testSubDir’ subdirectory now contains the expected and actual output from that particular test for both ‘stdout’ and ‘stderr’, and the input file which generated the actual output. Had the test failed, I would be able to look at these files to decide whether there is a bug in the program or simply a bug in the test script. Being able to examine individual tests like this is invaluable, especially when the test suite becomes very large – because you will, naturally, add tests every time you add features to a project or find and fix a bug.

Another alternative to the pure shell based test mechanism I have presented here is the Autotest facility by François Pinard, as used in Autoconf after release 2.13.

Later in A Complex GNU Autotools Project, the Sic project will be revisited to take advantage of some of the more advanced features of GNU Autotools. But first these advanced features will be discussed in the next several chapters – starting, in the next chapter, with a discussion of how GNU Autotools can help you to make a tarred distribution of your own projects.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]

This document was generated by Ben Elliston on May 30, 2015 using texi2html 1.82.