The use of the name tmpdir in the tests

[Ruud van der Pas]

Hi Nick,

Thanks very much for the response!

> Wouldn't it be better for gprofng to store the data in a directory
> named after the experiment ?  That way if the user runs multiple,
> different experiments it would still be possible to determine which
> directory contains the relevant data.

Thanks for the suggestion!

We have tried different approaches and given the kind of things
we would like to test (e.g. comparison of experiments), we came
up with a structure where each code/test has its own main directory
in tmpdir. It is named after the test (e.g. "mxv").

Within this directory there will be one or more subdirectories
with a name that is set in the .exp file for that test.

For example, in my current set up, this is what you see in tmpdir:

$ ls tmpdir
mxv/
$ ls tmpdir/mxv
no-opt/  opt/
$

This means there are two different main tests for the "mxv" code
and as suggested by the name, one is for an executable compiled
without optimizations enabled and the other one uses an executable
compiled with certain optimization flags.

The compiler flags are set in the same .exp file by the way.

This is an example line from the table:

{ "mxv"  "opt"  "-g -O"  "-p on"  "-m 4000 -n 2000" "2" "4"}

The last two numbers are the initial and end value for the
thread count used. This example will run the experiments
for 2, 3, and 4 threads. One thing I'm considering is to also
include an increment to reduce on the number of cases.

Within each subdirectory, there are one or more log and result files,
plus the corresponding experiment directories. It also contains the
generated gprofng script file that has been used to generate the results.

The log files are named after the functionality (e.g. collect-app.log).
The result files include the name of the gprofng tool and command
used, plus the number of threads (e.g. display-text.functions.2.thr.txt).

The name of the experiment directory uses some of the settings
in the table that defines the experiment. In particular, the
name of the test and the number of threads used. For example
mxv.2.thr.er.

Since we are in a subdirectory we already know more, like "no-opt",
and by keeping the name short we avoid cluttering the directory
with (too) long names.

We typically want to extract more than one result from an experiment
directory, because we'd like to test various commands (e.g. -header,
-threads, -functions, etc.) and eventually also different tools.

Essentially, these subdirectories contain everything needed to 
manually reproduce the results (the generated script file handles
everything) and/or conduct some different "what if" experiments.

In this way, we hope to make it easier to quickly find the test(s)
that failed (e.g. mxv/no-opt) and ease the troubleshooting.

At this point, I have this framework working, but have not yet
gotten to work on the verification part. Which is of course the
crucial next step.

> Probably.  There are lots of places in the binutils sources where
> the "tmpdir" directory name is explicitly referenced, so you would
> need a large patch to change them all.  Plus I would not be at all
> surprised to find that there are users of the binutils who expect
> to find certain things in the tmpdir directories.  So renaming
> would definitely cause them problems.

Thanks. It is clear then :-) We should not change that name.

> How about "tmpdir-<experiment_name>" or "resultdir-<experiment_name>" ?

Thanks for the suggestions!

With the above structure, I hope we make it easy for people to do the
troubleshooting, but as always, suggestions are very welcome!

Kind regards, Ruud