testsuite/run_dump_test improvement

[Nick Clifton]

Hi Fangrui,

> 2) Can we write comments in *.d?

Yes.  There are plenty of example of this.  Just prefix them with a # character.
For an example see gas/testsuite/gas/elf/bignums.d


> 3) Can we allow free-form shell commands instead of specialized directives?

No - the *.d files are not shell scripts.  They are Tcl command files,

> Then, why can't we have multiple dump programs?

We can - we just need to write a proc to support such behaviour.  Or extend
an existing proc.

> Frequently we need to check
> multiple properties of an output and a single dump program just does not
> work well. 

But you can have multiple *.d files that reference the same source file
and which use different dump tools to check different properties of the
output.  Maybe not as elegant as having all the checks in one file, but
certainly do-able with the current infrastructure.


> 4) It may be nice for the pattern matching tool to recognize a substring by default.
> Let {{^}}, {{$}}, or some other metacharacters to denote that "the full line should be matched"

I am not clear as to what you mean here.  If you need an exact match of
a line of output then you just include those characters as a line in the
.d file.  eg:

  must match this line exactly

If you want to insist that a line is present but you do not care about
the characters on the line then you can just you a glob-regexp, eg:

  .*

If you want to match multiple lines up until a specific line then you
can use #...  For example:

  must match this line
  #...
  any number of lines were matched until this line, but this line must be present

Plus if you no longer care about the rest of the output and you just want
the test to pass you can use:

  #pass


> 5) It may be nice to use more descriptive filenames than *-1.d, *-2.d, or prXXXXX.d
>   prXXXXX can be mentioned as comments if we allow 2)
>   As it stands, many tests require guesswork before you open them in an editor.

True - but this is down to the testcase writers.  Or a maintainer to 
come along and rename them....


> 6) Parallelism
> 
>   runtest --tool ld --srcdir ../../ld/testsuite/ took more than 2 hours
>   to run on my machine. I'd expect a few seconds for a more pleasant experience.

I think that the gcc testsuite has the concept of expensive tests which can
be disabled via an environment option.  Maybe we need something similar for
the binutils testsuites.


> 7) Run an individual test, rather than runtest ...
> 
> I figured out that I can run partial tests with:
> 
>   # my build directory is $srcdir/Debug
>   cd Debug/ld
>   runtest --tool ld --srcdir ../../ld/testsuite export-dynamic-symbol.exp

Which can also be done as:

    make check RUNTESTFLAGS="<dir>/<driver>"
eg:
    make check-ld RUNTESTFLAGS="ld-elf/elf.exp"

> but it would be convenient to be able to run just one test.

When I need to do this I just create a dummy test driver file
that only runs the test I want.  eg:

  % cat ld/testsuite/ld-elf/dummy.exp
  run_dump_test seg

  % make check-ld RUNTESTFLAGS="ld-elf/dummy.exp"
  [...]
  Running /work/sources/binutils/current/ld/testsuite/ld-elf/dummy.exp ...

		=== ld Summary ===

  # of expected passes		1

It is ungainly I agree, but it does work.

Cheers
  Nick