Back: Including Texinfo Documentation
Forward: Rolling Distribution Tarballs
FastBack: A Large GNU Autotools Project
Up: A Large GNU Autotools Project
FastForward: Rolling Distribution Tarballs
Top: Autoconf, Automake, and Libtool
Contents: Table of Contents
Index: Index
About: About this document

12.5 Adding a Test Suite

Automake has very flexible support for automated test-suites within a project distribution, which are discussed more fully in the Automake manual. I have added a simple shell script based testing facility to Sic using this support -- this kind of testing mechanism is perfectly adequate for command line projects. The tests themselves simply feed prescribed input to the uninstalled sic interpreter and compare the actual output with what is expected.

Here is one of the test scripts:

## -*- sh -*-
## incomplete.test -- Test incomplete command handling

# Common definitions
if test -z "$srcdir"; then
    srcdir=echo "$0" | sed s,[^/]*$,,'
    test "$srcdir" = "$0" && srcdir=.
    test -z "$srcdir" && srcdir=.
    test "${VERBOSE+set}" != set && VERBOSE=1
. $srcdir/defs

# this is the test script
cat <<\EOF > in.sic
echo "1

# this is the output we should expect to see
cat <<\EOF >ok

cat <<\EOF >errok

# Run the test saving stderr to a file, and showing stdout
# if VERBOSE == 1
$RUNSIC in.sic  2> err | tee -i out >&2

# Test against expected output
if ${CMP} -s out ok; then
    echo "ok:" >&2
    cat ok >&2
    exit 1

# Munge error output to remove leading directories, `lt-' or
# trailing `.exe'
sed -e "s,^[^:]*[lt-]*sic[.ex]*:,sic:," err >sederr && mv sederr err

# Show stderr if doesnt match expected output if VERBOSE == 1
if "$CMP" -s err errok; then
    echo "err:" >&2
    cat err >&2
    echo "errok:" >&2
    cat errok >&2
    exit 1

The tricky part of this script is the first part which discovers the location of (and loads) `$srcdir/defs'. It is a little convoluted because it needs to work if the user has compiled the project in a separate build tree -- in which case the `defs' file is in a separate source tree and not in the actual directory in which the test is executed.

The `defs' file allows me to factor out the common definitions from each of the test files so that it can be maintained once in a single file that is read by all of the tests:

#! /bin/sh

# Make sure srcdir is an absolute path.  Supply the variable
# if it does not exist.  We want to be able to run the tests
# stand-alone!!
if test ! -d $srcdir ; then
    echo "defs: installation error" 1>&2
    exit 1

#  IF the source directory is a Unix or a DOS root directory, ...
case "$srcdir" in
    /* | [A-Za-z]:\\*) ;;
    *) srcdir=`\cd $srcdir && pwd` ;;

case "$top_builddir" in
    /* | [A-Za-z]:\\*) ;;
    *) top_builddir=`\cd ${top_builddir-..} && pwd` ;;

progname=`echo "$0" | sed 's,^.*/,,'`
testname=`echo "$progname" | sed 's,-.*$,,'`


# User can set VERBOSE to prevent output redirection
case x$VERBOSE in
    xNO | xno | x0 | x)
        exec > /dev/null 2>&1

rm -rf $testsubdir > /dev/null 2>&1
mkdir $testsubdir
cd $testsubdir \
   || { echo "Cannot make or change into $testsubdir"; exit 1; }

echo "=== Running test $progname"


Having written a few more test scripts, and made sure that they are working by running them from the command line, all that remains is to write a suitable `' so that automake can run the test suite automatically.

## -- Process this file with automake to produce

EXTRA_DIST              = defs $(TESTS)

testsubdir              = testSubDir

TESTS_ENVIRONMENT       = top_builddir=$(top_builddir)

TESTS                   =                       \
                        empty-eval.test         \
                        empty-eval-2.test       \
                        empty-eval-3.test       \
                        incomplete.test         \

        -rm -rf $(testsubdir)

I have used the `testsubdir' macro to run the tests in their own subdirectory so that the directory containing the actual test scripts is not polluted with lots of fallout files generated by running the tests. For completeness I have used a hook target(26) to remove this subdirectory when the user types:

$ make distclean
rm -rf testSubDir

Adding more tests is accomplished by creating a new test script and adding it to the list in noinst_SCRIPTS. Remembering to add the new `tests' subdirectory to `' and the top-level `', and reconfiguring the project to propagate the changes into the various generated files, I can run the whole test suite from the top directory with:

$ make check

It is often useful run tests in isolation, either when developing new tests, or to examine more closely why a test has failed unexpectedly. Having set this test suite up as I did, individual tests can be executed with:

$ VERBOSE=1 make check TESTS=incomplete.test
make  check-TESTS
make[1]: Entering directory
=== Running test incomplete.test
PASS: incomplete.test
All 1 tests passed
make[1]: Leaving directory /tmp/sic/tests
$ ls testSubDir/
err   errok   in.sic   ok   out

The `testSubDir' subdirectory now contains the expected and actual output from that particular test for both `stdout' and `stderr', and the input file which generated the actual output. Had the test failed, I would be able to look at these files to decide whether there is a bug in the program or simply a bug in the test script. Being able to examine individual tests like this is invaluable, especially when the test suite becomes very large -- because you will, naturally, add tests every time you add features to a project or find and fix a bug.

Another alternative to the pure shell based test mechanism I have presented here is the Autotest facility by Fran@,cois Pinard, as used in Autoconf after release 2.13.

Later in 20. A Complex GNU Autotools Project, the Sic project will be revisited to take advantage of some of the more advanced features of GNU Autotools. But first these advanced features will be discussed in the next several chapters -- starting, in the next chapter, with a discussion of how GNU Autotools can help you to make a tarred distribution of your own projects.

This document was generated by Gary V. Vaughan on February, 8 2006 using texi2html