Re: [BuildBot] News and announcements

[Sergio Durigan Junior <sergiodj at sergiodj dot net>]

On Wednesday, June 17 2015, Doug Evans wrote:

> I have successfully run the gdb testsuite on the aix machine on the gcc farm,
> so it is doable. I can (try to) help (if I can find the time).

Thanks, Doug.  I will let you and Joel know when I have time to tackle
this again.

> Let me take the opportunity repeat some of my feature requests. :-)
>
> Instead of running a build per commit, just pick up the current tree
> on each iteration of the builder (and early exit if nothing has changed).

I don't know if I understood the request well, but there is no such
thing as "iteration of the builder".  The way BuildBot works is: if
there is a new commit in the repository, it triggers builds for every
builder (a.k.a. "build configurations").  This makes each builder to
update its own copy of the upstream repository, and build the specified
commit (usually HEAD, but it's possible to force them to build other
commits as well).

> And if there's a problem, bisect.

With the way BuildBot works (described above), there is no need to
bisect something: if a commit introduces a (valid) failure on the
testsuite, we will know right away (i.e., when the builders build it).
Also, and as I said in my first message, if the commit breaks GDB's
compilation, we will also know right away, and the commit author will be
notified directly.

Hm, but now I think I understood what you proposed above.  You are
proposing to make something like nightly builders, is that right?  I.e.,
instead of building every commit in the repository, just build git HEAD
at the same time every day.  That can be done with BuildBot (although I
don't know how hard it would be to implement a "bisect").

> This should scale better when there are more frequent commits.

I agree, although I like the idea of testing every commit as it hits the
tree.

> Commits usually don't break the build so picking up the tree
> where it is should be just fine (has been for me with my
> nightly build of trunk for years).
>
> If a test fails, run it several times.
> This should help catch hard fails vs flaky fails.

Yeah, this last part would be neat (although I am also not sure how hard
it would be to implement this on BuildBot).

> IWBN if one build handled multiple test configs.

This was something I could have thought about when I was designing how
our BuildBot was going to work.  Currently, we have one builder for
every build & test configuration.  I agree it should be possible to have
e.g. one builder to test multiple test configs...  This will require a
major redesign of how our BuildBot works, so I don't know when I'll have
the time to tackle this.

> As for new test configs I'd like to see:
> gcc vs llvm
> gdb-generated index vs gold-generated index vs no index
> dwarf2 vs dwarf4 (with type units)
> fission vs fission-with-dwp vs no-fission
> stabs
> gdbserver vs no gdbserver
> Each of these would use the same gdb, it's just
> the test run that's different.

Noted.  I don't know if it will be possible to easily implement the "vs"
configs; what can be done is running "gcc" on a builder and then "llvm"
on another...

>> Problematic testcases
>> =====================
>>
>> Efforts on reducing the number of problematic testcases are being done,
>> mostly by Pedro.  Thanks, Pedro!  Hopefully, sometime in the future we
>> will reach a state when all the notifications of regressions will
>> actually be trustworthy.
>
> It's difficult to prevent flaky tests from being added in the future,
> which is why I'd like to see the build-bot help find them.

BuildBot is already "finding" them: you can compare the test results (on
gdb-testers) and see that there are lots of racy testcases being
reported for various builders.  As I may have mentioned before, there is
a way to make our BuildBot ignore tests that are racy, but this is just
a workaround, of course.

>>   - Update the existing x86_64 machine to Fedora 22
>>
>>   - Get the new x86_64 machines and set them up for our BuildBot
>>
>>   - Get the AIX buildslave up and running
>>
>>   - Continue trying to convince others to donate buildslaves to us :-)
>
> Working on it ...

Thanks.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/