Improving the situation around check, xcheck and developer testing.

[Sam James]

Carlos O'Donell <carlos@redhat.com> writes:

> The xcheck target is unique to glibc, and has the following deficiencies:
>
>  * Not documented in the GNU Coding Standards or Maintainers Guide.
>
>  * Not documented anywhere the glibc manual.
>
>  * Individual tests have varying critical requirements that are not documented
>    except here:
>    https://sourceware.org/glibc/wiki/Testing/Tests
>
>  * Developers don't run it.
>
>  * Distributions don't run it.

I had no idea it existed until today. I'd seen the xtest patch Florian
posted and wanted to look into what on earth it was..

>
> All of these deficiencies stack up to produce a kind of poorly maintained
> target that only senior developers really know about, and that doesn't get
> fixed because it doesn't impact day-to-day developer workflows. Since the
> target isn't a "normal" target it never gets executed by downstream
> distributions.
>
> That we know there are cleanups that need to be made is exactly the
> reason I'd like to see 'make xcheck' merged into 'make check', or at the 
> very least new tests added to *both* targets.

If at all possible, we should have only one target. Note that packaging
often runs 'make check' by default (for us, we do something like:
run check target if it exists; if not, run test target if it exists,
unless overridden).

Florian's patch for the xcheck bits introduces run-built-tests=no [0]. These
variables generally make me uncomfortable. The reason is that they
*look* like internal variables and it's never clear to me if we should
be using them or not.

For example: objdir, subdirs (both mentioned on the wiki [1]) are often
used by recursive make setups and shouldn't be overridden by the user.

[0] https://inbox.sourceware.org/libc-alpha/87a5g2gnej.fsf@oldenburg.str.redhat.com/
[1] https://sourceware.org/glibc/wiki/Testing/Testsuite#Testing_just_one_test

We should have a single list of variables we promise are part of the
test interface.

>
> Let me sketch a suggestion for us to discuss:
>
>  * Add a quickcheck target for developers, put it in the manual in install.texi
>
>    - The quickcheck target runs ALL the tests but sets SCALE_CHECK=1 env var,
>      valid values are 1-100.
>
>  * Default quickcheck and check to be the same.
>
>    - By default today quickcheck and check would be the same but we could
>      change that.
>

I think this sounds good. Perhaps should print a line when using 'make
check' linking to a wiki page describing the differences.

"If you hit unexpected failures as a distributor, please see ...".

>  * Default xcheck to run with SCALE_CHECK=100 env var.
>
>  * Add your xcheck test to check *and* xcheck.
>
>    - Tune the test to run as short as possible with SCALE_CHECK=1, and as long
>      as required with SCALE_CHECK=100.
>
> Outcomes:
>
>  * There are no tests we don't run.
>
>    - Provides compile-time and run-time coverage for all tests.
>

This is important. We've had tests bitrot before.

>  * Design some tests to have coverage as a sliding scale.
>
>    - Coverage can be a function of distribution choices.
>

This would match what other projects let us do sometimes.

>    - The math library testing could be scale sensitive too.
>

Yeah, this is often a problem.

>  * Developers wanting quicker local testing use 'make test t=test' (already present)
>    or 'make quickcheck' for local iterative testing.
>
> Future work:
>
>  * Migrate each xcheck to check and add a sliding scale with SCALE_CHECK.
>
>    - Tests that need permissions go into containers.
>
>    - Tests that are destructive go into containers (postclean).
>
>    - Tests that need specific configurations go into containers.
>
>    - Tests that can't be scaled up or down become UNSUPPORTED.

I think the proposal sounds good.