Re: [RFC] README for the testsuite

[Pedro Alves <palves at redhat dot com>]

On 09/06/2013 01:11 AM, Stan Shebs wrote:
> As Pedro pointed out, the internals manual has some prescriptive
> material on testsuite parameters and variables that ought to be under
> version control in the sources somewhere.  So I constructed a README
> for the testsuite directory, generally writing new descriptions adapted
> for the readme context.

Thanks!

> 
> It's almost all ready to go, but the description of INTERNAL_GDBFLAGS
> got kind of muddled, and stopped making sense partway through.  So I
> could use a little help on clarifying what it is supposed to do exactly.
> 
> (There's another bit of info from the internals manual that should be in
> the sources, which is our official GDB coding standard.  It's partly
> in gdb/CONTRIBUTE, but not complete.)
> 
> Stan
> stan@codesourcery.com
> 
> 
> README
> 
> 
> This is a collection of tests for GDB.
> 
> The file gdb/README contains basic instructions on how to run the
> testsuite, while this file documents additional options and controls
> that are available.  The GDB wiki may also have some pages with ideas
> and suggestions.
> 
> 
> Running the Testsuite
> *********************
> 
> There are two ways to run the testsuite and pass additional parameters
> to DejaGnu.  The first is with `make check' in the main build
> directory and specifying the makefile variable `RUNTESTFLAGS':
> 
> 	 make check RUNTESTFLAGS='TRANSCRIPT=y gdb.base/a2-run.exp'
> 
> The second is to cd to the testsuite directory and invoke the DejaGnu
> `runtest' command directly.
> 
> 	cd testsuite
> 	make site.exp
> 	runtest TRANSCRIPT=y
> 
> (The `site.exp' file contains a handful of useful variables like host
> and target triplets, and pathnames.)
> 
> Testsuite Parameters
> ********************
> 
> The following parameters are DejaGNU variables that you can set to
> affect the testsuite run globally.
> 
> TRANSCRIPT
> 
> You may find it useful to have a transcript of the commands that the
> testsuite sends to GDB, for instance if GDB crashes during the run,
> and you want to reconstruct the sequence of commands.
> 
> If the DejaGNU variable TRANSCRIPT is set (to any value), each
> invocation of GDB during the test run will get a transcript file
> written into the DejaGNU output directory.  The file will have the
> name transcript.<n>, where <n> is an integer.  The first line of the
> file shows the invocation command with all the options passed to it,
> while subsequent lines are the GDB commands.
> 
>       make check RUNTESTFLAGS=TRANSCRIPT=y

Hmm, without the @smallexample markup, it read like this line appears
out of nowhere.  Sounds like we need to explicitly write "Example:"
or some such.

> 
> The transcript may not be complete, as for instance tests of command
> completion may show only partial command lines.
> 
> GDB
> 
> By default, the testsuite exercise the GDB in the build directory, but

exercises

> you can set GDB to be a pathname to a different version.  For
> instance,
> 
>     make check RUNTESTFLAGS=GDB=/usr/bin/gdb
> 
> runs the testsuite on the GDB in /usr/bin.
> 
> GDBSERVER
> 
> You can set GDBSERVER to be a particular GDBserver of interest.
> 
>     make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver"

Likewise.

> 
> checks installed GDB and GDBserver.
> 
> [
> INTERNAL_GDBFLAGS
> 
> When running the testsuite normally, you usually don't want whatever
> is in ~/.gdbinit to interfere with the tests, therefore the test
> harness passes `-nx' to GDB by default.  Likewise, you usually don't
> want any of the windowed interfaces, which you suppress with `-nw'.
> This is achieved via INTERNAL_GDBFLAGS.
> 
> set INTERNAL_GDBFLAGS "-nw -nx"
> 
> This is all well and good, except when testing an installed GDB
> that has been configured with `--with-system-gdbinit'.  Here you
> do not want ~/.gdbinit loaded but may want the system
> .gdbinit file loaded.  This can be achieved by pointing $HOME
> at a directory without a .gdbinit and by overriding
> INTERNAL_GDBFLAGS and removing -nx.
> 
> 	cd testsuite
> 	HOME=`pwd` runtest \
> 	  GDB=/usr/bin/gdb \
> 	  GDBSERVER=/usr/bin/gdbserver \
> 	  INTERNAL_GDBFLAGS=-nw
> ]

Well, my initial reaction was, "this is an internal flag,
WTH are we documenting it here?"  But I see that this
text is almost as is from gdbint.texinfo...

Here's my suggestion:

INTERNAL_GDBFLAGS

Command line options passed to all GDB invocations.

The default is "-nw -nx".

`-nw' disables any of the windowed interfaces.
`-nx' disables ~/.gdbinit, so that it doesn't interfere with
the tests.

This is actually considered an internal variable, and you
won't normally want to change it.  However, in some situations,
this may be tweaked as last resort if the testsuite doesn't
have direct support for the specifics of your environment.
The testsuite does not override a value provided by the user.

As an example, when testing an installed GDB that has been
configured with `--with-system-gdbinit', like by default,
you do not want ~/.gdbinit to interfere with tests, but, you
may want the system .gdbinit file loaded.  As there's no way to
ask the testsuite, or GDB, to load the system gdbinit but
not ~/.gdbinit, a workaround is then to remove `-nx' from
INTERNAL_GDBFLAGS, and point $HOME at a directory without
a .gdbinit.  For example:

	cd testsuite
	HOME=`pwd` runtest \
	  GDB=/usr/bin/gdb \
	  GDBSERVER=/usr/bin/gdbserver \
	  INTERNAL_GDBFLAGS=-nw

> Board Settings
> **************
> 
> DejaGNU includes the concept of a "board file", which specifies
> testing details for a particular target (which are often bare circuit
> boards, thus the name).
> 
> In the GDB testsuite specifically, the board file may include a
> number of "board settings" that test cases may check before deciding
> whether to exercise a particular feature.  For instance, a board
> lacking any I/O devices, or perhaps simply having its I/O devices
> not wired up, should set `noinferiorio'.
> 
> Here are the supported board settings:
> 
> gdb,cannot_call_functions
> 
> The board does not support inferior call, that is, invoking inferior
> functions in GDB.
> 
> gdb,can_reverse
> 
> The board supports reverse execution.
> 
> gdb,no_hardware_watchpoints
> 
> The board does not support hardware watchpoints.

I suggest indenting the descriptions, like in a man
page, to have better guidance on what is what.  E.g.:

gdb,cannot_call_functions

  The board does not support inferior call, that is, invoking inferior
  functions in GDB.

gdb,can_reverse

  The board supports reverse execution.

gdb,no_hardware_watchpoints

  The board does not support hardware watchpoints.

etc.

The "Testsuite Parameters" section likewise, perhaps.

-- 
Pedro Alves