Improving the situation around check, xcheck and developer testing.

[Sam James]

Florian Weimer <fweimer@redhat.com> writes:

> * Carlos O'Donell:
>
>> On 9/20/24 12:34 PM, Florian Weimer wrote:
>>> * Carlos O'Donell:
>>> 
>>>>  * Add a quickcheck target for developers, put it in the manual in install.texi
>>>>
>>>>    - The quickcheck target runs ALL the tests but sets SCALE_CHECK=1 env var,
>>>>      valid values are 1-100.
>>>>
>>>>  * Default quickcheck and check to be the same.
>>>>
>>>>    - By default today quickcheck and check would be the same but we could
>>>>      change that.
>>> 
>>> I don't see why this would be an improvement over xcheck?
>>
>> The first and foremost improvement I would like to see it that all tests are always compiled.
>>
>> This ensures tests don't fail to compile.
>>
>> Do you agree that always compiling all tests would be an improvement?
>
> I think it makes sense.  We don't seem to have any xtests that trigger
> pathological GCC behavior during compilation, I think, so it should be
> okay.
>
>> Do you agree that at least running the test and having it check
>> preconditions is an improvement?
>
> Again, some tests don't have preconditions.  If they check preconditions
> and can produce UNSUPPORTED, we should consider turning them into
> regular tests.
>
> We should not do that for potentially destructive tests that require
> root privileges, they should remain xtests.
>
>>> This assumes that tests can be scaled like this.  That's not always
>>> true.  For example, the mkstemp tests perform at least 238,328 file
>>> system operations before they can report a meaningful test result.
>>
>> Then we build the test, the pre-conditions are not met for
>> SCALE_CHECK=1, and we return UNSUPPORTED. But the test is built (check
>> for building tests), and the test is run (check that infrastructure
>> around the test works), and at runtime we determine the preconditions
>> are not met and return UNSUPPORTED.
>
> Seems quite a bit of work for very little improvement, to be honest.  I
> think we are better off if we upstream our upstream-testsuite wrapper to
> Fedora, along with the planned xcheck enhancement.  Other distributions
> could use that as a template.

This would be fine for something like contrib/ but I don't feel it
solves the problem here. Not least because it's RPM-specific ;)

Could you clarify what you'd like to see here? The wrapper + which
changes to xcheck specifically?

thanks,
sam