Improving the situation around check, xcheck and developer testing.

[Carlos O'Donell]

On 9/20/24 3:05 PM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> On 9/20/24 12:34 PM, Florian Weimer wrote:
>>> * Carlos O'Donell:
>>>
>>>>  * Add a quickcheck target for developers, put it in the manual in install.texi
>>>>
>>>>    - The quickcheck target runs ALL the tests but sets SCALE_CHECK=1 env var,
>>>>      valid values are 1-100.
>>>>
>>>>  * Default quickcheck and check to be the same.
>>>>
>>>>    - By default today quickcheck and check would be the same but we could
>>>>      change that.
>>>
>>> I don't see why this would be an improvement over xcheck?
>>
>> The first and foremost improvement I would like to see it that all tests are always compiled.
>>
>> This ensures tests don't fail to compile.
>>
>> Do you agree that always compiling all tests would be an improvement?
> 
> I think it makes sense.  We don't seem to have any xtests that trigger
> pathological GCC behavior during compilation, I think, so it should be
> okay.

Oh my. Yes, that's something I hadn't considered.

Thanks for adjusting your other patch in this direction (now reviewed).


>> Do you agree that at least running the test and having it check
>> preconditions is an improvement?
> 
> Again, some tests don't have preconditions.  If they check preconditions
> and can produce UNSUPPORTED, we should consider turning them into
> regular tests.

Agreed.

> We should not do that for potentially destructive tests that require
> root privileges, they should remain xtests.

I would like to review such tests, because I think only test-container is a good
fit for that.

Have you come across a destructive test?

>>> This assumes that tests can be scaled like this.  That's not always
>>> true.  For example, the mkstemp tests perform at least 238,328 file
>>> system operations before they can report a meaningful test result.
>>
>> Then we build the test, the pre-conditions are not met for
>> SCALE_CHECK=1, and we return UNSUPPORTED. But the test is built (check
>> for building tests), and the test is run (check that infrastructure
>> around the test works), and at runtime we determine the preconditions
>> are not met and return UNSUPPORTED.
> 
> Seems quite a bit of work for very little improvement, to be honest.  I
> think we are better off if we upstream our upstream-testsuite wrapper to
> Fedora, along with the planned xcheck enhancement.  Other distributions
> could use that as a template.

The biggest improvement for me would be to make sure xcheck tests are built and that
all the infrastructure around them is runnable, and I agree that anything beyond that
is diminishing returns. Cleaning them up would be a welcome next step. If Fedora starts
running xcheck in rawhide CI that would be great.

-- 
Cheers,
Carlos.