[GDB BuildBot] New "Try Server"

[Sergio Durigan Junior <sergiodj at redhat dot com>]

Hi there,

0. Introduction
===============

I am happy to announce that I have finally finished configuring our
BuildBot instance to accept "try jobs".  What this means is that a
developer will be able to upload a patch to our BuildBot, and this patch
will then be tested against a set of builders we have there.  In the
paragraphs below I will explain how this feature works, who can use it
(and how), and try to answer a few questions that I think you will have.

1. Concepts
===========

As you know, our BuildBot master runs on my personal server, and we have
several "slaves" running on different locations.  The master doesn't do
any builds itself, but rather controls all the details about the builds
that need to be performed.  The slaves do not take any decision by
themselves; they just do what the master tells them to do.

On top of the slaves, there is an abstraction called "builder".  The
builder is basically a recipe of how to do a build.  This includes the
compilation flags used, environment variables, and testsuite options.  A
slave can have multiple builders; a builder can also be assigned to
multiple slaves (for example, when we have more than one slave with the
same hardware configuration, as is the case of the Fedora x86_64
slaves).

Then, we have the schedulers.  They are responsible for monitoring
events and triggering builds.  Our main scheduler monitors our
repository, and triggers a full build on all builders whenever a new
commit has been pushed.  We also have "cron job" schedulers, used to
trigger racy testcase detection builds every week at a certain time.
And now, we have the "try job" scheduler, responsible for triggering a
build on a set of builders when a developer has submitted a patch for
testing.

2. Build priority
=================

Before the try scheduler, builds were basically treated in a "first
come, first served" basis.  No special priority was assigned to a build.

Now, I decided to try a new algorithm.  Considering that a developer
usually want to test patches when they are almost ready for submission,
it would be really annoying if she had to wait in line until her build
is processed; it is not rare to see some builders having 10 or more
builds in the queue.  So, my decision was to "move" the try builds to
the top of the queue.  This will obviously delay the testing of the
official repository, but I am hoping that the impact will not be very
big.  If it is, then this decision will have to be revisited.

3. Security
===========

I am currently talking to the buildslave owners off-list in order to
make sure that we are on the same page here, but there are a few
considerations to be made about the security of a try job.  The obvious
one is that a developer may upload a malicious patch to be "tested", and
there is no easy way for BuildBot to detect this.  Since the BuildBot
master does not build anything, it would be the slaves who would suffer
From this kind of attack.

But there is another thing to consider.  In order to be able to upload a
patch to be tested, the developer must have some kind of access to the
BuildBot master.  Yes, my personal server.  No, I am not happy about
this.  I will write more on the authentication mechanism below.

So, after giving a long thought about this problem, I decided that I
will only provide accounts to:

  - Global maintainers.

  - Someone I know personally, or that I feel confident enough that is a
    trustworthy person.  (yeah, this is very subjective, but it is my
    machine :-/)

If you do not fit into one of these categories, you can ask someone from
one of the above categories to vouch for you.  This person will need to
contact me.

4. Authentication
=================

The BuildBot try command can use SSH for authentication, so that is what
I have chosen.  Basically, you will need to send me your *public* SSH
key; no passwords are needed/accepted.  You will be using a common user
to log in, and will not receive a PTY (i.e., will not have shell
access).  If you ever used gitolite, then you know the drill.  If not,
it's pretty simple.

I will provide a configuration file below.

5. Installing buildbot
======================

In order to use the new feature, you will have to install the "buildbot"
package.  You don't need the "slave" package!  On Debian, an "apt-get
install buildbot" should do the trick.  On Fedora, "dnf install
buildbot".  However, Fedora enables the BuildBot systemd service after
the installation; I strongly recommend you disable that, unless you
intend to run a BuildBot master on your machine.

6. The "buildbot try" command
=============================

After installing the package, the only command you need to know is
"buildbot try".  It will be responsible for generating a patch out of
the git tree you're in (assuming you don't provide a patch explicitly),
connecting to the server and submitting the patch to be tested.

I strongly recommend that you read the command's documentation at:

  <http://docs.buildbot.net/current/manual/cmdline.html#try>

However, the usage is pretty basic.  There are two modes: you can either
be in a local branch and ask "buildbot try" to automatically figure out
the patch to be tested, or you can provide the patch file explicitly.

In the first mode, "buildbot try" will generate a diff of your local
branch against your "master" branch, and will instruct the BuildBot
master to test the patch *using the master's branch HEAD revision*.
This means that if you want to generate a patch to be tested against the
latest commit of the official repository, you have to do a "git pull" on
your master branch first (and rebase your local branch, of course).

The second mode accepts a patch file via the "--diff" parameter.
Originally, "buildbot try" will not specify a revision against which the
patch needs to be tested; however, I have modified that behaviour on the
server side, and the provided patch will always be tested against the
latest commit of the official repository.

7. Configuration file for "buildbot try"
========================================

Here is the configuration file that you need to use for the "buildbot
try" command to work.  This file needs to be named
<$HOME/.buildbot/options>.

  $> cat $HOME/.buildbot/options
  # The method of connection to the BuildBot master.
  try_connect = 'ssh'

  # The host of the BuildBot master.
  try_host = 'gdb-build.sergiodj.net'

  # Our username.  We will always use this username!
  try_username = 'gdb-buildbot'

  # The jobdir on BuildBot master (i.e., the directory where "buildbot
  # try" will put the request files.
  try_jobdir = 'gdb/try_ssh_jobdir'

  # The VCS we are using for the project.
  try_vc = 'git'

  # The branch against which we will try our patches.  Currently, we
  # only support 'master'.
  try_branch = 'master'

  # Who am I?
  # NOTE: YOU NEED TO USE THE FORMAT "NAME <EMAIL>" IN THIS FIELD!
  try_who = 'John Smith <john@example.com>'

  # The builders I want to test my patch in.
  # The list of available builder can be found in:
  #  <http://sourceware.org/gdb/wiki/BuildBot#Available_Builders>
  try_builders = [ "Fedora-x86_64-m64",
                   "Fedora-x86_64-m32",
                   "Fedora-x86_64-native-gdbserver-m64",
                   "Fedora-x86_64-native-gdbserver-m32",
                   "Fedora-x86_64-native-extended-gdbserver-m64",
                   "Fedora-x86_64-native-extended-gdbserver-m32",
                   "Fedora-x86_64-cc-with-index",
                   "Fedora-i686" ]
  $>

It is good to mention that most of these options can be overriden by
command line options, so you can (for example) choose to test your patch
against only a subset of the available try_builders.

8. Notifications
================

The "buildbot try" command offers a way to wait for the builds to
complete on the command line, but this doesn't work when you're using
SSH.  However, you will receive e-mails from our BuildBot notifying you
about the results of all of your try builds.  The e-mails will be very
similar to those sent to the gdb-testers mailing list.

9. Build specifics
==================

As mentioned above, the try builds will almost always be performed
against the latest commit on the official repository.  They will also
run using the very same flags that are being used for each builder
(unfortunately, there is no way to change these flags on try builds).
And the results will be compared against the last results that our
BuildBot has on file.

For example, if our BuildBot has just tested the commit 0abcd on a
certain builder, and then you request it to test your patch on the same
builder, BuildBot will run the full build of your patch, generate the
testsuite logs, and compare them against the testsuite logs generated by
testing 0abcd.  The known XFAIL files for that specific builder will
also be taken into account, i.e., they will be discarded when comparing
the testsuite logs.

Last, but not least, your try build will generate its own testsuite
logs, which will be recorded in that builder's git repository, available
at:

  <http://gdb-build.sergiodj.net/cgit>

10. More info
=============

I guess this e-mail already covers quite a lot, but I also intend to
populate our wiki with more info.  The page will be:

  <https://sourceware.org/gdb/wiki/BuildBot#Try_Jobs>

And, of course, I am open to suggestions for improvements (can't promise
I'll be able to dedicate much time these days, though).


----

I guess that is it.  Please let me know if you have any questions, and I
will be happy to (try to) answer!

Cheers,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Attachment: signature.asc
Description: PGP signature