Re: Some more Mauve observations (test batch mode)

[Mark Wielaard <mark at klomp dot org>]

Hi,

On Fri, 2002-07-19 at 01:36, Mark Wielaard wrote:
> What I am personally really missing is a test batch mode. It would be
> really nice to have a mechanism for running the compiler and VM for each
> test separately. Currently when the compiler cannot compile a test or if
> the VM crashes your complete test run stops. libgcj has something that
> can do that using dejagnu but I hope it can be done by a simple shell
> script.

I made something by adapting/splitting up the 'choose' script and I am
really happy with how it works. I have been testing it with jikes and
gcj -C as compilers and gij or kissme as runtime VMs. It also works for
compiling to native code (completely from source, not yet
source->class->native but that is not so interesting for me at the
moment). If someone has a working kjc and kaffe installation please try
it out. To use it just set a variable in the 'runner' an 'batch_run'
script and run it with ./batch_run > results_file. (Mauve should have
been correctly configured though.)

Attached you find the following files:

choose-classes:
- generates a 'test_classes' file given some keys/tags.

uses-list:
- generates a 'uses_files' file containing all
  the java source files used by a given test.

runner:
- Runs a given test class and generates a FAIL output
  when the test crashes the VM of times out.

batch_run:
- Uses choose-classes to generate a list of all the
  tests in mauve. Tries to compile all the needed files
  (using the uses-list script) and generates a FAIL
  when compilation fails and finally runs the test using
  the runner script.

There is also a small patch for SimpleTestHarness to add a -resultsonly
flag to make it only output the PASS or FAIL results which is used by
the runner script.

These scripts should be usable with the current Mauve installation.
The really tricky part is integrating them with the configure/build
environment. I have not yet really thought about it since the Makefile
looks like black magic to me. Hopefully someone is interested in helping
me integrate them...

Cheers,

Mark
#! /bin/sh

# Copyright (c) 1998, 1999 Cygnus Solutions
# Written by Tom Tromey <tromey@cygnus.com>

# Copyright (c) 2002 Free Software Foundation, Inc.
# Adapted by Mark Wielaard <mark@klomp.org>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.  */

# Choose the tests we want to run. Output generated in file 'test_classes'.
# Usage: choose-classes [-verbose] output-directory [tag] ...
# Run in the source directory.
# See README for information on tags.

verbose=
spectags=
tags=
file_specs=

if test "$1" = "-verbose"; then
   verbose=yes
   shift
fi

outdir="$1"
shift
test -z "$outdir" \
    && echo "Usage: choose-classes [-verbose] output-directory [tag] ..." \
    && exit -1

# Some tags imply other tags.  For instance, JDK1.2 implies JDK1.1 and
# JDK1.0.  This makes it easier to keep the tags up to date in the
# source -- we only have to keep track of differences (as opposed to
# editing every source file when a new tag is added).  If we see an
# otherwise-unrecognized tag, and the file `mauve-TAG' exists, then we
# treat the contents of that file as a list of implied tags.
taglist=${1+"$@"}
seen_dashdash=
while test -n "$taglist"; do
   set $taglist
   i="$1"
   shift
   taglist=${1+"$@"}

   case "$i" in
    X--)
       seen_dashdash=yes
       ;;
    !!java.*)
       echo "Invalid tag: $i" 1>&2
       exit 1
       ;;
    !java.* | java.* | !javax.* | javax.*)
       file="`echo $i | tr . /`"
       file_specs="$file_specs $file"
       ;;
    JAVA2)
       tags="$tags JDK1.0 JDK1.1 JDK1.2 $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JDK1.2)
       tags="$tags JDK1.0 JDK1.1 JAVA2 $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JDK1.1)
       tags="$tags JDK1.0 JLS1.1 $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JDK1.0)
       tags="$tags JLS1.0"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JLS1.1)
       tags="$tags JDK1.0 JDK1.1"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JLS1.0)
       tags="$tags JDK1.0"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    *)
       file=
       # Always prefer a tag file in the build directory.
       if test -f "$outdir/mauve-$i"; then
	  file="$outdir/mauve-$i"
       else
	  if test -f "mauve-$i"; then
	     file="mauve-$i"
	  fi
       fi
       if test -n "$file"; then
	  # We don't want tags from a file to be treated as specified
	  # on the command line.  So we add a `X--' before the
	  # expansion.  We don't use just `--' as this would confuse
	  # `set'.
	  taglist="$taglist X-- `sed -e 's/#.*$//g' $file`"
       fi
       tags="$tags $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
   esac
done

# By default, use JDK1.1.
test -z "$tags" && tags="JDK1.0 JDK1.1"

test -n "$verbose" && echo "tags = $tags"
test -n "$verbose" && echo "spectags = $spectags"
test -n "$verbose" && echo "file_specs = $file_specs"

classes=/tmp/classes-$$
: > $classes

(cd gnu/testlet; find java javax -name '*.java' -print) |
while read file; do
   exact_match=no
   test -n "$verbose" && echo "Examining $file"
   if test -n "$file_specs"; then
      ok=yes
      for pat in $file_specs; do
	 # PAT can be like `java.lang' or `!java.lang'.
	 # If `!FILE' matches `PAT', then we know the pattern
	 # starts with `!' and is an exclusion pattern.
	 # Otherwise, if `!FILE' matches `!PAT', then we know that
	 # the file should be included.
	 case "!$file" in
	  ${pat}*)
	     ok=no
	     test -n "$verbose" && echo "  ... excluded by $pat"
	     ;;
	  !${pat}.java)
	     # If the tag list includes an exact match, then we
	     # unconditionally accept it.
	     ok=yes
	     exact_match=yes
	     test -n "$verbose" && echo "  ... included by $pat (exact match)"
	     ;;
	  !${pat}*)
	     ok=yes
	     test -n "$verbose" && echo "  ... included by $pat"
	     ;;
	 esac
      done
   else
      ok=yes
   fi

   if test "$ok" = no; then
      continue
   fi

   # Surround value with spaces so that case statements will work
   # correctly.
   taglist=" `grep '^// Tags:' gnu/testlet/$file | sed -e 's,^// Tags:,,'` "
   istest=yes
   case "$taglist" in
    *\ not-a-test\ *)
       istest=no
       ;;
   esac

   # For an exact match, we don't look at the tags.
   if test "$exact_match" = no; then
      ok=no
      # If any tag on the tag list matches, then we are ok.
      for tag in $tags; do
	 case "$taglist" in
	    *" ${tag} "*)
	    ok=yes
	    break
	    ;;
	 esac
      done

      # If any specified tag hits a `not' tag, then we are not ok.
      if test "$ok" = yes; then
	 for tag in $spectags; do
	    case "$taglist" in
	       *" !${tag} "*)
	         ok=no
		 test -n "$verbose" && echo "File $file has tag !$tag"
		 break
		 ;;
	    esac
	 done
      else
	 test -n "$verbose" && echo "  ... excluded because no tag matched"
      fi
   fi

   if test "$ok" = yes; then
      class="gnu.testlet.`echo $file | sed -e 's/\.java$//' | tr / .`"
      if test "$istest" = yes; then
	 echo $class >> $classes
	 test -n "$verbose" && echo "Chose $class"
      else
      	 test -n "$verbose" && echo " ... excluded because not-a-test"
      fi
   fi
done

if test -f $outdir/test_classes \
   && cmp $classes $outdir/test_classes > /dev/null 2>&1; then
   # Files are the same.
   rm $classes
else
   mv $classes $outdir/test_classes
fi

exit 0
#!/bin/sh

# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>
# Based on the choose script written by Tom Tromey <tromey@cygnus.com>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.

# Given a (list of) test classes or java files generates a list of all java
# files used for that test in the file 'uses_files'.

usage="uses-list [-verbose] output-directory classes_or_files"

if test "$1" = "-verbose"; then
    verbose=yes
    shift
else
    verbose=
fi 

outdir="$1"
shift
test -z "$outdir" && echo "$usage" && exit -1

usesfile=/tmp/uses-file-$$
uses=/tmp/uses-$$

class="$1"
shift
test -z "$class" && echo "$usage" && exit -1

while test ! -z "$class"; do
    test -n "$verbose" && echo "Processing $class"

    # Strip trailing .java or .class if given and replace . with /
    file="`echo $class | sed -e 's/\.java$//' | sed -e 's/\.class$//' | tr . /`"
    file="$file.java"
    dir="`dirname $file`"
    test ! -f "$file" && echo "file $file does not exist" && exit -1

    useslist=" `grep '^// Uses:' $file | sed -e 's,^// Uses:,,'` "

    filelist=$file
    echo "$file" >> "$usesfile"
    for used in $useslist; do
	usedfile=$dir/$used.java
	if test ! -f "$usedfile"; then
	    echo "file $used file used by $file does not exist"
	    exit -1
	fi
	echo "$usedfile" >> "$usesfile"
	filelist="$filelist $usedfile"
    done
    test -n "$verbose" && echo " ... uses: $filelist"

    class="$1"
    shift
done

# Now uniquify the list of auxiliary files we used.
# I think neither `uniq' nor `sort -u' is portable, so we do it by hand.
previous=
sort "$usesfile" |
while read next; do
    if test "$previous" != "$next"; then
	echo "$next" >> $uses
	previous=$next
    fi
done

if test -f $outdir/uses_files \
    && cmp $uses $outdir/uses_files > /dev/null 2>&1; then
    # Files are the same.
    rm $uses
else
    mv $uses $outdir/uses_files
fi
rm $usesfile
#!/bin/bash

# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.

# Tries to run one test case. Produces a FAIL output if the test crashes
# or doesn't finish after some timeout occurs.
# RUNTIME command can be set in the script to select the VM.
# For testing (gcj) native code set RUNTIME to 'native' it will then try to
# execute the program SimpleTestHarness in the current directory.
# Called from batch_run script.
#
# Usage: runner test_class_name
# Run in the source directory.

#RUNTIME="gij"
RUNTIME="Kissme"
#RUNTIME="native"

# Number of seconds to wait for the test to finish
WAIT=60

# Called when timeout occurs
timeout()
{
    kill -9 $PID 2>&1 > /dev/null
}

# Install timeout
trap timeout SIGALRM
sleep $WAIT && kill -s SIGALRM $$ 2>&1 > /dev/null &
TIMEOUT=$!

testclass=$1
# Run process
if test "$RUNTIME" = "native"; then
    echo $testclass | ./SimpleTestHarness -resultsonly &
else
    echo $testclass | $RUNTIME gnu.testlet.SimpleTestHarness -resultsonly &
fi
PID=$!
wait $PID
result=$?

# Terminated unexpectedly?
if test $result != 0 -a $result != 1; then
    echo "FAIL: $testclass abnormal termination $result CRASH or TIMEOUT"
fi

# cleanup timeout
trap '' SIGALRM
kill $TIMEOUT 2>&1 > /dev/null
#!/bin/bash

# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.

# Tries to compile and run all possible tests.
# Produces a FAIL output if compilation fails.
# COMPILER command can be set in the script
# For testing (gcj) native code compilation set NATIVE to 'true', otherwise
# don't set it.
# Uses runner script to run each individual test.
#
# Usage: run_batch > result_output
# Run in the source directory.

# Include every key/tag we can think of but always exclude the unicode test.
KEYS="JDK1.0 JDK1.1 JDK1.2 JDK1.3 JDK1.4 JLS1.0 JLS1.2 JDBC1.0 JDBC2.0 JAVA2"
KEYS="$KEYS !java.lang.Character.unicode"

#COMPILER="gcj -C"
#COMPILER="gcj"
COMPILER="jikes"

COMPILER_FLAGS="-g"

NATIVE=
#NATIVE="true"

# Cleanup
find gnu/testlet -name "*.class" | xargs rm -f
if test ! -z "$NATIVE"; then
    rm -f SimpleTestHarness
fi

# Create Mauve Framwork
framework_sources="gnu/testlet/SimpleTestHarness.java gnu/testlet/TestHarness.java gnu/testlet/Testlet.java gnu/testlet/ResourceNotFoundException.java gnu/testlet/config.java"

if test -z "$NATIVE"; then
    $COMPILER $COMPILER_FLAGS $framework_sources
fi

# Generate test classes list
./choose-classes . "$KEYS"

# Compile and run all the tests
for testclass in `cat test_classes`; do
    # Get all needed java sources
    ./uses-list . $testclass

    # Try to compile sources
    if test -z "$NATIVE"; then
        $COMPILER $COMPILER_FLAGS @uses_files
    else
        $COMPILER -o SimpleTestHarness --main=gnu.testlet.SimpleTestHarness \
	          $framework_sources `cat uses_files`
    fi
    if test $?  == 0; then
        ./runner $testclass
    else
        echo "FAIL: $testclass COMPILE FAILED"
    fi
done
Index: gnu/testlet/SimpleTestHarness.java
===================================================================
RCS file: /cvs/mauve/mauve/gnu/testlet/SimpleTestHarness.java,v
retrieving revision 1.29
diff -u -r1.29 SimpleTestHarness.java
--- gnu/testlet/SimpleTestHarness.java	18 Feb 2002 01:43:07 -0000	1.29
+++ gnu/testlet/SimpleTestHarness.java	19 Jul 2002 22:57:12 -0000
@@ -39,6 +39,7 @@
   private int total = 0;
   private boolean verbose = false;
   private boolean debug = false;
+  private boolean results_only=false;
   private String description;
   private String last_check;
 
@@ -59,13 +60,13 @@
 	      System.out.println (desc);
 	      ++failures;
 	    }
-	  else if (verbose)
+	  else if (verbose || results_only)
 	    {
 	      System.out.println ("X" + desc);
 	      ++xfailures;
 	    }
 	}
-      else if (verbose)
+      else if (verbose || results_only)
 	{
 	  if (expected_xfails.contains (getDescription ("FAIL")))
 	    {
@@ -235,18 +236,26 @@
 
   protected int done ()
     {
-      System.out.println(failures + " of " + total + " tests failed");
-      if (xpasses > 0)
-        System.out.println(xpasses + " of " + total + " tests unexpectedly passed");
-      if (xfailures > 0)
-        System.out.println(xfailures + " of " + total + " tests expectedly failed");
+      if (!results_only)
+	{
+          System.out.println(failures + " of " + total + " tests failed");
+          if (xpasses > 0)
+            System.out.println(xpasses + " of " + total
+			    + " tests unexpectedly passed");
+          if (xfailures > 0)
+            System.out.println(xfailures + " of " + total
+			    + " tests expectedly failed");
+	}
       return failures > 0 ? 1 : 0;
+
     }
 
-  protected SimpleTestHarness (boolean verbose, boolean debug)
+  protected SimpleTestHarness (boolean verbose, boolean debug,
+		               boolean results_only)
     {
       this.verbose = verbose;
       this.debug = debug;
+      this.results_only= results_only;
 
       try
         {
@@ -269,6 +278,7 @@
     {
       boolean verbose = false;
       boolean debug = false;
+      boolean results_only = false;
       int i;
 
       for (i = 0; i < args.length; i++) 
@@ -277,12 +287,18 @@
 	    verbose = true;
 	  else if (args[i].equals("-debug")) 
 	    debug = true;
+	  else if (args[i].equals("-resultsonly")) 
+	    {
+	      results_only = true;
+	      verbose = false;
+	      debug = false;
+	    }
 	  else
 	    break;
         }
 
       SimpleTestHarness harness
-	= new SimpleTestHarness (verbose, debug);
+	= new SimpleTestHarness (verbose, debug, results_only);
 
       BufferedReader r
 	= new BufferedReader (new InputStreamReader (System.in));