Re: How to handle long running tests in nptl.

[Carlos O'Donell <carlos at redhat dot com>]

On 12/13/19 6:15 AM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> On 9/12/19 9:21 AM, Stefan Liebler wrote:
>>> Hi,
>>>
>>> the nptl tests are currently running in sequence. Some of them are running very long:
>>> -tst-cond16: 20s
>>> -tst-cond17: 20s
>>> -tst-cond18: 20s
>>> -tst-rwlock-tryrdlock-stall: 20s
>>> -tst-mutex10: 16s
>>> -tst-rwlock20: 10s
>>> -tst-rwlock-trywrlock-stall: 10s
>>> -tst-rwlock-pwn: 10s
>>>
>>> The listed tests are responsible for over two minutes of runtime of "make check". They all are running a test in a loop for a large amount of iterations or seconds in order to trigger e.g. a race condition.
>>>
>>> How to handle those long running tests with respect of "make check" runtime?
>>> - reduce runtime by reducing number of iterations or seconds
>>> - move those tests to "make xcheck"
>>> - reduce runtime while running "make check" and rerun with unchanged runtime in "make xcheck"
>>> - change nothing
>>> - other ideas?
>>
>> The use of 'make xcheck' is for tests that need specific persmissions
>> to run, like root.
> 
> It's also used for tests that require many minutes to run or require
> special firewall settings, e.g., resolv/tst-resolv-qtypes.
> 
> Unfortunately, we do not capture data of legitimate test failures, so
> it's hard to tell how valuable such tests are.
> 
> Many of the timeout-heavy nptl tests do not actually need a quiet
> system, though, so they could be moved to a separate subdirectory that
> contains only such tests.  Or we could add some markup to the tests and
> add a more intelligent test scheduler.

Let me set what I see is in or out of scope. Feel free to argue either way.

The quality of tests is out of scope for this discussion. Consider that
the project grows the number of regression tests into infinity, and consider
what a process would look like for handling this.

Recording the PASS/FAIL metrics is out of scope for this discussion.
I consider it a distinct discussion to make a database and start recording
the results of our test runs (which can grow out of CI infrastructure).

In scope is the tests themselves and how we group them (profiles) and what
tests are run by default and when.

We need to define what is a reasonable amount of time for a developer to
wait for make check to complete.

How long is this? What does it cover?

We need to define a broader testing target that includes longer running tests
that do not require root or special privileges. Some of these tests can also
be moved into test-container tests. DJ and I were talking about trying to
simulate setuid root in the container for DT_RUNPATH/DT_RPATH DST testing
(I have *hundreds* of permutation tests to verify correct operation of the
ELF gABI DST handling rewrite).

I would like to see make xcheck deprecated, and all tests moved, rewritten
to test-container, or removed. We should be able to do everything in a container.

I think make check could take an argument.

make check (Run default test profile "p=dev")
make xcheck (Deprecated, force users to consciously pick a profile)
make p=dev check (Run developer test profile, run iteratively by developer)
make p=rel check (Run release test profile, run minimally at release boundaries or by distro builds)
make p=torture check (Run torture test profile, run minimally at release boundaries)
make p=ci check (Run CI test profile adequate for CI testing, run per commit, limit to control CI costs/time)

Where 'p' is the profile of the test you want and the profile can be assembled
based on the requirements at hand for the person running the test.

Next steps:
- Define a developer test profile.
- Have 'make check' default to the developer test profile.
- Add the capability for other profiles to exist.
- Deprecate xcheck.
- Migrate xheck tests to profiles and in the process either rewrite or remove them.
- Continue active work to reshuffle the profiles as the project evolves.

Thoughts?

-- 
Cheers,
Carlos.