Re: buildbot for glibc

[Sergio Durigan Junior <sergiodj at redhat dot com>]

On Wednesday, February 11 2015, Roland McGrath wrote:

> Google is now hosting a new buildbot setup for glibc as a service to the
> community.  Things are extremely primitive at the moment, but we hope to
> improve and expand the setup in the future (with your help!).

Heya :-).

I am hosting a BuildBot for GDB as well:

  <http://gdb-build.sergiodj.net/waterfall>

>From your message, you probably didn't know :-).

> The hosts are virtual machines provided by Google Compute Engine.  The
> machine resources and setup help are being donated by Google's Dynamic
> Tools group (maintainers of AddressSanitizer et al).  Special thanks to
> Sergey Matveev for doing the initial installation and for ongoing help with
> the setup and administration.
>
> You can see the build status at:
> 	http://glibc-build.hack.frob.com/waterfall
> That name (on my personal server) is a simple redirect to the actual page,
> which does not yet have a pretty URL.  We'll need some help figuring out
> exactly how best to set things up to yield a more pleasant browsing
> experience.

What I did was setting up a reverse proxy in my Apache server.
Everything just worked.  I can provide this configuration if needed.

> You can currently get the buildbot configuration and build scripts with:
> 	svn co https://address-sanitizer.googlecode.com/svn/trunk/glibc_buildbot
> We want eventually to find a more appropriate place for that stuff to live,
> where the glibc community can collaborate directly on maintaining the setup.

We are still tweaking hard our configuration, which is temporarily
living in my personal git repo
(<http://git.sergiodj.net/?p=gdb-buildbot.git;a=summary>), but when it
is in a good state I will submit it upstream and host it on GDB's
repository, probably under contrib/ or so.

> For now, there are two "bots", both actually running on the same one
> virtual machine (called a "slave" in Buildbot lingo).  One does the basic
> x86_64-linux-gnu build, and one the basic i686-linux-gnu build.  

One thing I am trying very hard to maintain is the configuration
compatibility between buildslaves.  For example, if 2 slaves are
building the same builder, then they both should have the same distro
version, with the same packages (i.e., same versions) installed.  This
helps to narrow the search for causes of regressions when something
happens.

> Fresh builds are kicked off every time there is a new commit to master.
> It polls every few minutes to notice.

I had some trouble polling from sourceware every 3 minutes (my default
setup).  This eventually led to the implementation of a gitorious mirror
of GDB (<https://gitorious.org/gdb>).  Just FYI, in case you have
misterious "Connection reset" errors.

> The slave OS installation is based on Debian wheezy, which has only GCC
> 4.7.  The i686 build always fails in 'make check' because I haven't figured
> out what packages to install so that 'g++ -m32' will find -lstdc++.
> (Actually, I just installed another one and it might work next cycle.)

I maintain a wiki page:

  <https://sourceware.org/gdb/wiki/BuildBot>

That tries to explain how you should set up your buildslave.  Of course,
the page still needs improvement, but it's a good start for people who
want to donate slaves.

> The x86_64 build has a few test failures (but all the tests do build),
> which I have not looked into yet.  The build logs do not get much
> information about how they fail, just the summary emitted at the end.  I
> can log into the slave and poke around, but I don't think I'll be able to
> offer that access to people outside Google.  Perhaps someone should whip
> up a script that examines tests.sum and prints out the .test-result and
> .out files for each FAIL--though that is by no means necessarily enough
> information to understand the failure mode.

This is one of the hardest parts, indeed.  I tried many configurations,
but what I am using is some old code created by Tromey (but heavily
modified by me):

  <http://git.sergiodj.net/?p=gdb-buildbot.git;a=blob;f=lib/sumfiles.py>
  <http://git.sergiodj.net/?p=gdb-buildbot.git;a=blob;f=lib/gdbcommand.py>

I can explain what it does if needed.  It even has a XFAIL mechanism so
that you can specify XFAIL tests per builder, per branch.

I am also storing test results using git:

  <http://gdb-build.sergiodj.net/cgit>

Because I don't have too much disk space to spare.  I tried to explain
how it works when I sent the announcement message to gdb@:

  <https://sourceware.org/ml/gdb/2015-01/msg00026.html>

> As with all infrastructure for the project, there is no big supply of paid
> hacker time to work on this.  It's intended to grow to be of great use to
> us developers and maintainers, but the only way that will happen is if more
> folks in the community pitch in to improve and maintain it.

So...  Would Google be willing to host our buildbot as well?  We would
obviously need access to the machine in order to perform tweaks, but
since you're being so kind with glibc, I figured it wouldn't hurt to ask
on behalf of GDB :-).

Also, we are always in need of buildslaves, too!  We already have:

- Fedora x86_64
- Debian i686/x86_64
- PPC64BE
- PPC64LE

Of course, the more, the better :-).  Maybe we can trade some slaves
here and there...?  :-)

Cheers,

-- 
Sergio
GPG key ID: 0x65FC5E36
Please send encrypted e-mail if possible
http://sergiodj.net/