Re: RFC: make fastcheck

[Carlos O'Donell <carlos at redhat dot com>]

On 7/18/19 4:13 AM, Florian Weimer wrote:
> * Carlos O'Donell:
>
> > Florian,
> >
> > What do you think of making a 'fastcheck' that can run all tests which have
> > timeouts less than or equal to the default timeout?
> >
> > My idea here is as follows:
> >
> > - 'make fastcheck' is for developer smoke testing during edit/build/test cycle.
> > - 'make check' if for broader developer testing, and will, if required, update
> >    the chroot for container testing (would have fixed Rafal's observed breakage).
> > - 'make xcheck' runs everything including the expensive tests and tests requiring
> >    explicit permissions.
> >
> > Then you could switch your workflow to using 'fastcheck'?
> >
> > This is one part of fixing the workflow.
> >
> > The other part is actually fixing 'make' to go faster, but DJ and I talked about
> > this recently and DJ has some ideas.
>
> Build time on my test machine:
>
> real    1m40.633s
> user    8m19.845s
> sys     2m6.309s
>
> The test time is:
>
> real    14m5.543s
> user    29m29.308s
> sys     7m27.912s
>
>
> Now apply this patch:
>
> diff --git a/support/support_test_main.c b/support/support_test_main.c
> index 7e7b9edbb0..f196990a3f 100644
> --- a/support/support_test_main.c
> +++ b/support/support_test_main.c
> @@ -209,6 +209,9 @@ adjust_exit_status (int status)
>   int
>   support_test_main (int argc, char **argv, const struct test_config *config)
>   {
> +  if (config->timeout > DEFAULT_TIMEOUT)
> +    return 0;
> +
>     if (test_main_called)
>       {
>         printf ("error: test_main called for a second time\n");
>
> The new testing time is:
>
> real    10m35.001s
> user    25m55.020s
> sys     5m21.004s
>
> So there is a definite improvement.  But I don't think it doesn't solve
> the core problem because it's still too long.

Sorry, I failed to list a critical part of my own reasoning.

I intended to write that the test-container tests would *not* run in
make fastcheck.

How much does that improve the timing by?

> It also doesn't help with build-many-glibcs.py because that doesn't run
> any tests which could time out.

That's the second part of this problem. I agree it's a problem.

However, testing, even if we fix the build infrastructure, will always
grow in duration, and we need a way to handle this with a "profile" which
allows developers to smoke test their build.

--
Cheers,
Carlos.