GDB BuildBot

This wiki page describes how the GDB BuildBot works. Read it if you are interested in adding a new buildslave to the system, if you want to know who is responsible for each existing buildslave, or if you are just curious about how the BuildBot is configured.

1. DISCLAIMER

/!\ PLEASE READ CAREFULLY

It has become clear that it is important to define the roles of each individual/community when it comes to our BuildBot. Please read the excerpt below, taken from Joel Brobecker's e-mail message at https://sourceware.org/ml/gdb-patches/2017-12/msg00417.html:

The GDB project being a volunteer-based community-driven project,
a port, and the associated resources, can only survive if there is
a group of people with time and motivation to match both the port
and the available resources. When resources are put in place for
those platforms, and those resources fail to be useful, the people
to turn to for answers are those who pushed for them to be put
in place. Asking the community in general to be responsible does
not seem like an attainable goal, nor is it attractive to me.

That is not to say that the community is entirely care free either.
I think we see nearly every day people cooperating on subjects
that do not directly matter to them, and as far as I am concerned
going well beyond the call of duty. But there has to be a reasonable
limit to what we ask of people to help for other platforms.
Contributing hardware is not a fire-and-forget thing. For the issues
with the buildbots, there has to be someone who has a stake in
those platforms, and willing to drive the effort in getting those
back in shape. These are the people we should turn to, not the
person who provides the service for those who want to. And if there
are none and the resource becomes a nuisance rather than a help,
then the logical answer is to remove it.

If anything, what this discussion leads me to, is that the lessons
learned here is that we should be clearer about an identified group
who would be responsible for what before a new buildbot is being put
in place, and also about what level of "service" can be expected
from the people maintaining the infrastructure, vs what concerns
the platforms the buildbots build and test.

In other words:

2. TL;DR

The web interface for the GDB BuildBot is https://gdb-build.sergiodj.net/. Specifically, you may be interested in going directly to the Waterfall page, which displays in a nice manner the builds.

The URL for connecting your buildslave to the GDB BuildBot is gdb-build.sergiodj.net:16123. You must contact us first in order to obtain permission to do that (and a password); send a message to <gdb@sourceware.org> and put <sergiodj@redhat.com> in the Cc.

The interface for viewing the results of the builds is https://gdb-build.sergiodj.net/results .

Regression reports are sent to the gdb-testers mailing list.

3. New terms

In this page, we will use a few specific terms to describe things, so it is good to understand what they mean.

4. How to view previous build logs

All the logs for all builds in the last 6 months are stored at https://gdb-build.sergiodj.net/results/. As you can see when you enter the page, the logs are organized by builder, then by the first two letters of the commit hash, and then by the commit hash itself. So, for example, if you wanted to see the build logs generated by the Fedora-x864_64-m64 builder for a fictitious commit 0a1b2c3d4e, you would have to go to https://gdb-build.sergiodj.net/results/Fedora-x864_64-m64/0a/0a1b2c3d4e.

The logs are compressed using xz in order to save space, so you have to uncompress before viewing them. A direct way to look at the gdb.log of the commit mentioned above, for example, is:

wget https://gdb-build.sergiodj.net/results/Fedora-x864_64-m64/0a/0a1b2c3d4e/gdb.log.xz -O - | xzcat | pager

We also store logs for try builds (see below for more information on try builds). To access them, go to the try directory inside the builder directory. For example, https://gdb-build.sergiodj.net/results/Fedora-x864_64-m64/try/0a/0a1b2c3d4e.

5. How to add your buildslave

If you are interested in adding a new buildslave (and possibly a new builder) to the GDB BuildBot, thank you! There are some steps you need to perform in order to make sure everything is OK.

5.1. Buildslave specs

Your machine needs to:

It is also recommended that your machine is constantly updated (e.g., on Fedora you might consider running a cronjob to dnf update every day), because GDB's test results are impacted by updates on GCC, glibc, binutils, etc.

5.2. Buildslave owner responsibilities

You will be the responsible for the buildslave, which means that we may contact you if there is anything you should do (i.e., update the buildbot-slave package, restart your buildslave, etc.). Be sure to fill the contact fields on your buildslave configuration file (see below).

5.3. Contact us first

Please, contact us in order to discuss further details, and to let us know about you. The best way to contact the GDB community is to write to <gdb@sourceware.org>. Please, make sure you also include Sergio Durigan Junior in the Cc (e-mail: <sergiodj@redhat.com>), because he is the current maintainer of the GDB BuildBot.

5.4. Adding your buildslave to the configuration

You will need to patch the following files and add your buildslave configuration there:

On config.json, you will have to add your buildslave information on the slaves section. You should provide:

This is an example of how a new entry should look like:

"slaves" : [ { "name" : "fedora-x86-64-1", "arch" : "x86_64", "jobs" : "4" },
             ...
             { "name" : "debian-x86-64-1", "arch" : "x86_64", "jobs" : "2" }
           ],

Then, you will have to edit the builders section. In this section we describe what kind of build we are interested in, and what are the buildslaves that will perform it. If you are adding a buildslave with a new architecture, then you will most probably want to create a new builder for it. However, you can also be interested in adding more buildslaves to an existing builder. In both cases, you will have to edit the following properties:

"builders" : [ { "name" : "Fedora-x86_64-m64", "type" : "Plain_c64t64",
                 "builddir" : "fedora-x86-64",
                 "slavenames" : [ "fedora-x86-64-1" ] },
               ...
               { "name" : "Debian-x86_64-m64", "type" : "Plain_c64t64",
                 "builddir" : "debian-x86-64",
                 "slavenames" : [ "debian-x86-64-1" ] },

At last, if you created a new builder, you will also want to include its name in the builderNames property of the schedulers section.

After you have done this, please send a patch to the <gdb-patches@sourceware.org> mailing list. You will be provided a password used by the buildslave to connect to the buildmaster.

5.5. Buildslave configuration

The first thing you will need to do is to install the buildbot-slave, version 0.8.14, package on your system. To do that, run (as root):

pip install buildbot-slave==0.8.14

/!\ You may need to install the Python development files in order to build buildbot-buildslave.

If this doesn't work, you will have to find the specific version on BuildBot's website. If you are running Windows, take a look at BuildBot's instructions.

5.5.1. Fedora-specific instructions

You will need to install the dependencies to compile and test GDB. For Fedora systems:

(set -ex;dnf install fedpkg rpm-build;fedpkg clone -a gdb;cd gdb;fedpkg sources;rpmbuild --define "_specdir $PWD" --define "_sourcedir $PWD" --define "_srcrpmdir $PWD" -bs --with testsuite *.spec;dnf builddep gdb-*.src.rpm)

/!\ Please make sure to re-run this command whenever you upgrade your Fedora! New dependencies can be added to GDB, and it is important to keep up-to-date with them.

Do not forget to disable the abrt-ccpp service on Fedora. It will make ABRT be invoked every time a coredump is generate, which consumes time and is totally unnecessary. To disable it, do (as root):

$ systemctl stop abrt-ccpp.service
$ systemctl disable abrt-ccpp.service

If you're running a version of Fedora that has abrtd, you can also disable it:

$ systemctl stop abrtd.service
$ systemctl disable abrtd.service

Also make sure that core dumps can be generated in the current directory, so that core dump tests are not skipped. E.g., on Fedora:

$ echo "kernel.core_pattern=core" > /etc/sysctl.d/50-coredump.conf
$ sysctl --system

If you're having permission problems or any other strange problem, try disabling SELinux:

$ setenforce 0

If it works, you will have to permanently disable it by editing the file /etc/selinux/config and setting SELINUX=disabled.

5.5.2. Debian-specific instructions

For Debian-based distributions, you can do:

$ apt-get build-dep gdb
$ apt-get install texinfo flex bison
$ apt-get install emacs

/!\ Please make sure to re-run this command whenever you upgrade your Debian! New dependencies can be added to GDB, and it is important to keep up-to-date with them.

5.5.3. General instructions

/!\ Because of the make TAGS step, you will also have to install emacs.

/!\ If your machine will not test 32-bit packages, you do not need to install the 32-bit (i.e., .i686) versions of the packages above.

/!\ It has been reported that the tests won't run properly if the SHELL variable is not set to a valid shell (e.g. /sbin/nologin instead of /bin/bash). Make sure that the user running the buildslave has a valid shell or override the SHELL environment variable somehow.

It is also recommended that you create a separate user to run the buildslave. This user should not need/have root nor SSH access; it only needs to be able to build and test GDB.

For example, let's assume this new user is called buildbot. Log into its account in the system, and create a new buildslave. To do that, you are going to need three things:

Then, you can issue the command:

$ buildslave create-slave DIR URL:PORT NAME PASSWORD

Where DIR is the directory that will be used by the buildslave to store local files (it can be the same as NAME), URL:PORT is the address of the buildmaster, and NAME is the name of the buildslave. The PASSWORD needs to be obtained from the BuildBot admin.

5.5.4. Adding information about yourself

You must add your contact information on your buildslave. To do that, edit the file named DIR/info/admin (assuming that DIR is the name of your buildslave, as per the example above) and add your name and e-mail address there:

$ cat DIR/info/admin
My Name <my@email.com>

5.5.5. Adding information about the machine

You should also edit the file DIR/info/host and include information about the machine that is running the buildslave. This is very useful when a failure happens and someone is trying to reproduce it, because knowing the versions of the Linux kernel, g++ and other tools can help narrow down the problem. In order to do that, it is recommended that you put at least the following information in your DIR/info/host file:

<Distro name and release>
<Linux kernel information (from uname)>

<Version of g++>
<Version of binutils>

To see an example of how this information is displayed, see the "Slave information" section here: https://gdb-build.sergiodj.net/buildslaves/fedora-x86-64-1

A good idea is to create a script that updates this information every time the system reboots, because that is usually a sign that some packages have been upgraded. In other words, it's your responsibility to keep this information up-to-date on your buildslave :-).

5.5.6. Starting the buildslave

After that, you can start the buildslave:

$ buildslave start NAME

Make sure it is started OK. It is also recommended that you create some special cronjob to restart this on every reboot. An example would be:

$ crontab -l
# restart buildslaves on every reboot

@reboot buildslave restart NAME

6. Try Jobs

The Try scheduler is now configured on our BuildBot. What this means is that a developer will be able to upload a patch to our BuildBot, and this patch will then be tested against a set of builders we have there.

6.1. Concepts

As you know, our BuildBot master runs on my (Sergio) personal server, and we have several "slaves" running on different locations. The master doesn't do any builds itself, but rather controls all the details about the builds that need to be performed. The slaves do not take any decision by themselves; they just do what the master tells them to do.

On top of the slaves, there is an abstraction called "builder". The builder is basically a recipe of how to do a build. This includes the compilation flags used, environment variables, and testsuite options. A slave can have multiple builders; a builder can also be assigned to multiple slaves (for example, when we have more than one slave with the same hardware configuration, as is the case of the Fedora x86_64 slaves).

Then, we have the schedulers. They are responsible for monitoring events and triggering builds. Our main scheduler monitors our repository, and triggers a full build on all builders whenever a new commit has been pushed. We also have "cron job" schedulers, used to trigger racy testcase detection builds every week at a certain time. And now, we have the "try job" scheduler, responsible for triggering a build on a set of builders when a developer has submitted a patch for testing.

6.2. Build priority

Before the try scheduler, builds were basically treated in a "first come, first served" basis. No special priority was assigned to a build.

Now, I decided to try a new algorithm. Considering that a developer usually want to test patches when they are almost ready for submission, it would be really annoying if she had to wait in line until her build is processed; it is not rare to see some builders having 10 or more builds in the queue. So, my decision was to "move" the try builds to the top of the queue. This will obviously delay the testing of the official repository, but I am hoping that the impact will not be very big. If it is, then this decision will have to be revisited.

6.3. Security

There are a few considerations to be made about the security of a try job. The obvious one is that a developer may upload a malicious patch to be "tested", and there is no easy way for BuildBot to detect this. Since the BuildBot master does not build anything, it would be the slaves who would suffer from this kind of attack.

But there is another thing to consider. In order to be able to upload a patch to be tested, the developer must have some kind of access to the BuildBot master. Yes, my personal server. No, I am not happy about this. I will write more on the authentication mechanism below.

So, after giving a long thought about this problem, I decided that I will only provide accounts to people who already have commit access to the GDB repository.

If you want access, you need to contact me and provide a proof that you have commit access (a commit made by you should be enough).

6.4. Authentication

The BuildBot try command can use SSH for authentication, so that is what I have chosen. Basically, you will need to send me your public SSH key; no passwords are needed/accepted. You will be using a common user to log in, and will not receive a PTY (i.e., will not have shell access). If you ever used gitolite, then you know the drill. If not, it's pretty simple.

I will provide a configuration file below.

6.5. Installing buildbot

In order to use the new feature, you will have to install the "buildbot" package. You don't need the "slave" package! On Debian, an "apt-get install buildbot" should do the trick. On Fedora, "dnf install buildbot". However, Fedora enables the BuildBot systemd service after the installation; I strongly recommend you disable that, unless you intend to run a BuildBot master on your machine.

6.6. The "buildbot try" command

After installing the package, the only command you need to know is "buildbot try". It will be responsible for generating a patch out of the git tree you're in (assuming you don't provide a patch explicitly), connecting to the server and submitting the patch to be tested.

I strongly recommend that you read the command's documentation at:

http://docs.buildbot.net/current/manual/cmdline.html#try

However, the usage is pretty basic. There are two modes: you can either be in a local branch and ask "buildbot try" to automatically figure out the patch to be tested, or you can provide the patch file explicitly.

In the first mode, "buildbot try" will generate a diff of your local branch against your "master" branch, and will instruct the BuildBot master to test the patch *using the master's branch HEAD revision*. This means that if you want to generate a patch to be tested against the latest commit of the official repository, you have to do a "git pull" on your master branch first (and rebase your local branch, of course).

The second mode accepts a patch file via the "--diff" parameter. Originally, "buildbot try" will not specify a revision against which the patch needs to be tested; however, I have modified that behaviour on the server side, and the provided patch will always be tested against the latest commit of the official repository.

6.7. Configuration file for "buildbot try"

Here is the configuration file that you need to use for the "buildbot try" command to work. This file needs to be named $HOME/.buildbot/options.

$> cat $HOME/.buildbot/options
# The method of connection to the BuildBot master.
try_connect = 'ssh'

# The host of the BuildBot master.
try_host = 'gdb-build.sergiodj.net'

# Our username.  We will always use this username!
try_username = 'gdb-buildbot'

# The jobdir on BuildBot master (i.e., the directory where "buildbot
# try" will put the request files.
try_jobdir = 'gdb/try_ssh_jobdir'

# The VCS we are using for the project.
try_vc = 'git'

# The branch against which we will try our patches.  Currently, we
# only support 'master'.
try_branch = 'master'

# Who am I?
# NOTE: YOU NEED TO USE THE FORMAT "NAME <EMAIL>" IN THIS FIELD!
try_who = 'John Smith <john@example.com>'

# The builders I want to test my patch in.
# The list of available builder can be found in:
#  <http://sourceware.org/gdb/wiki/BuildBot#Available_Builders>
try_builders = [ "Fedora-x86_64-m64",
                 "Fedora-x86_64-m32",
                 "Fedora-x86_64-native-gdbserver-m64",
                 "Fedora-x86_64-native-gdbserver-m32",
                 "Fedora-x86_64-native-extended-gdbserver-m64",
                 "Fedora-x86_64-native-extended-gdbserver-m32",
                 "Fedora-x86_64-cc-with-index",
                 "Fedora-i686",
                 "Fedora-s390x-m64",
                 "Debian-s390x-m64",
                 "Debian-s390x-native-gdbserver-m64",
                 "Debian-s390x-native-extended-gdbserver-m64",
                 "AIX-POWER8-plain",
                 "RHEL-s390x-m64",
                 "Ubuntu-AArch64-m64",
                 "Ubuntu-AArch64-native-gdbserver-m64" ]
$>

It is good to mention that most of these options can be overriden by command line options, so you can (for example) choose to test your patch against only a subset of the available try_builders.

6.8. Notifications

The "buildbot try" command offers a way to wait for the builds to complete on the command line, but this doesn't work when you're using SSH. However, you will receive e-mails from our BuildBot notifying you about the results of all of your try builds. The e-mails will be very similar to those sent to the gdb-testers mailing list.

6.9. Build specifics

As mentioned above, the try builds will almost always be performed against the latest commit on the official repository. They will also run using the very same flags that are being used for each builder (unfortunately, there is no way to change these flags on try builds). And the results will be compared against the last results that our BuildBot has on file.

For example, if our BuildBot has just tested the commit 0abcd on a certain builder, and then you request it to test your patch on the same builder, BuildBot will run the full build of your patch, generate the testsuite logs, and compare them against the testsuite logs generated by testing 0abcd. The known XFAIL files for that specific builder will also be taken into account, i.e., they will be discarded when comparing the testsuite logs.

Last, but not least, your try build will generate its own testsuite logs, which will be stored inside the builder's directory, at:

https://gdb-build.sergiodj.net/results

6.10. Available Builders

The list of available builders for a try build can be found at:

https://git.sergiodj.net/gdb-buildbot.git/tree/lib/config.json

Search for the entry named GDB_Try_Jobdir, and look at the builderNames list. That is the most updated list of try builders.

None: BuildBot (last edited 2018-01-18 01:38:48 by SergioDuriganJr)

All content (C) 2008 Free Software Foundation. For terms of use, redistribution, and modification, please see the WikiLicense page.