Basic requirements for supporting OS and machine ports.

[Joseph Myers]

On Thu, 20 Apr 2017, Tulio Magno Quites Machado Filho wrote:

> > Well, we need a glibc-specific script to collect whatever information we 
> > think is relevant from the results of a test run for sending to 
> > libc-testresults (for use for both manual and buildbot builds).
> 
> Usually BuildBot would notify a build failed and send a link to the place
> where you could get more information, including log files.  So it wouldn't
> be necessary to send all of that data to the mailing list.

I think buildbot logs have disappeared at least once when the bot moved 
from one system to another.  That's a clear reason to send the key 
information to the mailing list, so it stays available.

> But it seems you want a much broader solution.  Could you elaborate what you'd
> like to see in those messages?

Key information about the compile-time configuration of glibc, such as 
configure options, glibc host, GCC and binutils versions, glibc commit 
being tested.

Key information about the test configuration, such as the run-built-tests 
value and the kernel version on the host used for testing if 
run-build-tests is yes.

A list of the tests with unexpected results (anything that's not PASS or 
XFAIL) and the numbers of each result, as at the end of the output of 
"make check".

I think that's the basic minimum.  It would be expected that when doing 
pre-release testing for an architecture you send the results to 
libc-testresults and include a link on the per-release wiki page.  Bots, 
whether or not linked to the buildbot master, would also send results to 
libc-testresults.

It should be possible for people to grep a libc-testresults mailbox to see 
"is this test failing for other people, and if so, in what 
configurations?".  (Which is a use I make of gcc-testresults from time to 
time.)

-- 
Joseph S. Myers
joseph@codesourcery.com