Re: Chroot testsuite

[Ryan Arnold <ryan dot arnold at linaro dot org>]

On Thu, Dec 19, 2013 at 10:28 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 12/19/2013 10:04 PM, Ryan Arnold wrote:
>>> I agree.
>>>
>>> We need some kind of VM-based or chroot-based testing.
>>>
>>> Cheers,
>>> Carlos.
>>
>> I'll actually be working on a chroot based testsuite for glibc for my
>> work at Linaro starting in January.  I should have a preliminary
>> design I can put forward in a few weeks for comment.
>>
>> The first pass might not be as full featured as some people might want.
>
> Please do not let that stop you. Full featured is not a required
> bullet point on any design I've been thinking about. Simply
> adding chroot based testing would allow us to begin adding
> tests for ldconfig, networking, the resolver etc. etc.
>
> Cheers,
> Carlos.
>

I want to lay out the scope of initial objectives for adding what we
might call a glibc host testsuite (to be run in a chroot).

Scope: Expand the existing glibc test framework to include tests that
will be executed on the 'host' once glibc has been installed into a
chroot.

Whether the 'host' chroot is on a remote (potentially cross) machine
or local build machine depends on the compatibility of 'host' code to
run on the 'build' machine.  That determination is outside the scope
of this proposal.  I assume the user will tell configure where host
testing is to take place.

Creating the chroot is out-of-scope for glibc since the methodology
for doing so will vary based on operating system distribution (for
example, Ubuntu might use schroot and debootstrap while for Fedora
might use systemd-nspawn and yum --installroot).  A wiki how-to would
be helpful in any case.  I imagine an OS distribution could also
provide a package which does this for their specific src package
manager.

Glibc will need to know how to install itself (and it's host tests)
remotely for remote hosts.  Local host chroot installs are already
covered by install_root.  I've thought through a few scenarios, such
as a script in /scripts which copies files from the install_root over
to the host chroot, as well as overriding the existing make install
methodology.  Any ideas are appreciated.

Glibc will need to know how to startup local and remote chroots and
how to run tests in them.  Systemd-nspawn and schroot are both session
capable.  Running programs in the chroot of a remote host isn't
terribly complicated but will vary by chroot methodology, and glibc
will need to know how to do this.  Perhaps we can do something similar
to what was done with the test-wrapper variables added for cross
testing and provide various scripts in the scripts/ directory that
know how to access different types of containers on the remote or
local machines.

For instance, with a remote schroot session called glibc_test already
started we might have glibc dispatch a remote host test via an ssh
tunnel in the following way:

ssh -p 9999 ryanarn@localhost "schroot -r -c glibc_test â
/home/ryanarn/glibc_test/foo"

(Tunneling isn't required, but it'll greatly improve performance since
ssh handshaking isn't performed repeatedly).

Collecting the output of the tests and collating the results is done
via stdout, and sterr.

>From the earlier discussion it's clear that we want other project
testsuites (like ruby) to be able to dynamically link against this
newly installed libc and run in the chroot.  While direct support for
running these tests suites from within the glibc testsuite is
out-of-scope, if we focus on what is in-scope, it should make this
relatively easy.

A secondary objective (for me) is to be able to repurpose as many
existing builddir tests into host/install tests as possible.  This
removes the burden of having to copy the source and builddirs to a
remote machine which might have space storage space challenges
(especially on embedded machines).

Is there any coverage that's lost by not performing builddir tests
remotely if they're all repurposed as install/host tests?

Having a properly configured test-farm and some scaffolding around
glibc, one should also be able to do full breadth ifunc coverage
testing as well (by installing and running the same libraries and
tests onto a variety of remote hosts).  I'll be working on automating
such a system in Linaro for platforms we're interested in since we
already have some of the scaffolding in place.

I suppose we could also use the same framework for installing and
running benchmarks on hosts for ifunc performance testing coverage.

Thoughts?

Ryan