Re: RFC: Test hook for nss_files testing

["Carlos O'Donell" <carlos at redhat dot com>]

On 12/11/2015 07:15 AM, Florian Weimer wrote:
> On 12/11/2015 07:51 AM, Carlos O'Donell wrote:
> 
>>> I'm not so sure.  Setting up the chroot in a realistic way is quite
>>> difficult.  You need to populat /dev and /dev/pts in some way, and
>>> arrange for the necessary GCC  and platform bits being present, whatever
>>> they are. âmake installâ may not actually produce a tree layout that is
>>> used by any downstream distribution.  And so on.  The list of issues is
>>> quite long.
>>
>> In which regard do you think this is not ideal?
> 
> It's quite a bit of work and very tightly integrated with individual
> distributions.  Test hooks and in-tree testing directly benefit everyone
> who looks at test results.

You raise a good point here and that's perhaps the strongest argument
I've heard in favour of white-box testing for this particular interface.

Test hooks do result in immediate benefits, and I'm not against them for
testing intermediate state that can't otherwise be accessed via whole
system tests.

We need to move developers beyond the immediate `make check` and get
them doing broader testing upstream and earlier to accomodate the changes
they are working on.

>> Should we rely on distro images?
> 
> It's not just that, you'd also need to build glibc according to the
> distribution's needs, regarding file system layout and so on.  So it's
> really difficult to see how this can be part of the upstream testing.

Agreed. However, I don't see how that means it can't be part of the upstream
testing. There is no line in the sand that says upstream stops here and the
distribution starts here.

>> Lastly, we may have to rely on some basic building blocks for doing this kind
>> of testing. Without those building blocks we mark the test UNSUPPORTED.
>> It is entirely plausible that at the start the test only works on x86_64,
>> requires docker, and runs slim Fedora container for testing.
> 
> But that still looks like it needs network access, which builders
> generally lack (and rightly so).

You always need network access to clone the repository. Therefore this
would be an extension of the checkout process that would require you to
run enough initialization to be able to use the results to do the testing.
Beyond the image download there won't be much else required.

Cheers,
Carlos.