GDB BuildBot

This wiki page describes how the GDB BuildBot works. Read it if you are interested in adding a new buildslave to the system, if you want to know who is responsible for each existing buildslave, or if you are just curious about how the BuildBot is configured.

1. TL;DR

The web interface for the GDB BuildBot is http://gdb-build.sergiodj.net/. Specifically, you may be interested in going directly to the Waterfall page, which displays in a nice manner the builds.

The URL for connecting your buildslave to the GDB BuildBot is gdb-build.sergiodj.net:16123. You must contact us first in order to obtain permission to do that (and a password); send a message to <gdb@sourceware.org> and put <sergiodj@redhat.com> in the Cc.

The interface for viewing the results of the builds is http://gdb-build.sergiodj.net/cgit .

Regression reports are sent to the gdb-testers mailing list.

2. New terms

In this page, we will use a few specific terms to describe things, so it is good to understand what they mean.

3. How to use the cgit interface to view the logs

The interface for looking at the log files generated by the testsuite (i.e., gdb.sum and gdb.log files) is http://gdb-build.sergiodj.net/cgit. When you enter this page, you will many git repositories whose names are exactly the same names of the existing builders. Inside each repository, you will notice that there is only the master branch, and lots of tags. Each tag is named using the following convention:

TIMESTAMP-COMMIT

Where TIMESTAMP is composed by YEARMONTHDAY-HOURMINUTESECOND, and COMMIT is the SHA1 hash representing the GDB git repository's commit that has been tested. Therefore, if you pushed a commit to the GDB repository (say, 0abcd), and you want to take a look at the log files generated by testing your patch against the builder Fedora-x86_64-native-gdbserver-m64, you just have to go the repository named Fedora-x86_64-native-gdbserver-m64/.git and search (there is a search box in the top-right corner) for 0abcd.

4. How to add your buildslave

If you are interested in adding a new buildslave (and possibly a new builder) to the GDB BuildBot, thank you! There are some steps you need to perform in order to make sure everything is OK.

4.1. Buildslave specs

Your machine needs to:

It is also recommended that your machine is constantly updated (e.g., on Fedora you might consider running a cronjob to yum update every day), because GDB's test results are impacted by updates on GCC, glibc, binutils, etc.

4.2. Buildslave owner responsibilities

You will be the responsible for the buildslave, which means that we may contact you if there is anything you should do (i.e., update the buildbot-slave package, restart your buildslave, etc.). Be sure to fill the contact fields on your buildslave configuration file (see below).

4.3. Contact us first

Please, contact us in order to discuss further details, and to let us know about you. The best way to contact the GDB community is to write to <gdb@sourceware.org>. Please, make sure you also include Sergio Durigan Junior in the Cc (e-mail: <sergiodj@redhat.com>), because he is the current maintainer of the GDB BuildBot.

4.4. Adding your buildslave to the configuration

You will need to patch the following files and add your buildslave configuration there:

On config.json, you will have to add your buildslave information on the slaves section. You should provide:

This is an example of how a new entry should look like:

"slaves" : [ { "name" : "fedora-x86-64-1", "arch" : "x86_64", "jobs" : "4" },
             ...
             { "name" : "debian-x86-64-1", "arch" : "x86_64", "jobs" : "2" }
           ],

Then, you will have to edit the builders section. In this section we describe what kind of build we are interested in, and what are the buildslaves that will perform it. If you are adding a buildslave with a new architecture, then you will most probably want to create a new builder for it. However, you can also be interested in adding more buildslaves to an existing builder. In both cases, you will have to edit the following properties:

"builders" : [ { "name" : "Fedora-x86_64-m64", "type" : "Plain_c64t64",
                 "builddir" : "fedora-x86-64",
                 "slavenames" : [ "fedora-x86-64-1" ] },
               ...
               { "name" : "Debian-x86_64-m64", "type" : "Plain_c64t64",
                 "builddir" : "debian-x86-64",
                 "slavenames" : [ "debian-x86-64-1" ] },

At last, if you created a new builder, you will also want to include its name in the builderNames property of the schedulers section.

After you have done this, please send a patch to the <gdb-patches@sourceware.org> mailing list. You will be provided a password used by the buildslave to connect to the buildmaster.

4.5. Buildslave configuration

The first thing you will need to do is to install the buildbot-slave, version 0.8.12, package on your system. To do that, run (as root):

pip install buildbot-slave==0.8.12

If this doesn't work, you will have to find the specific version on BuildBot's website. If you are running Windows, take a look at BuildBot's instructions.

4.5.1. Fedora-specific instructions

You will need to install the dependencies to compile and test GDB. For Fedora systems:

(set -ex;fedpkg clone -a gdb;cd gdb;fedpkg sources;rpmbuild --define "_specdir $PWD" --define "_sourcedir $PWD" --define "_srcrpmdir $PWD" -bs --with testsuite *.spec;yum-builddep gdb-*.src.rpm)

Do not forget to disable the abrt-ccpp service on Fedora. It will make ABRT be invoked every time a coredump is generate, which consumes time and is totally unnecessary. To disable it, do (as root):

$ systemctl stop abrt-ccpp.service
$ systemctl disable abrt-ccpp.service

Also make sure that core dumps can be generated in the current directory, so that core dump tests are not skipped. E.g., on Fedora:

echo "kernel.core_pattern=core" > /etc/sysctl.d/50-coredump.conf

4.5.2. Debian-specific instructions

For Debian-based distributions, you can do:

$ apt-get build-dep gdb
$ apt-get install texinfo flex bison
$ apt-get install emacs

4.5.3. General instructions

/!\ Because of the make TAGS step, you will also have to install emacs.

/!\ If your machine will not test 32-bit packages, you do not need to install the 32-bit (i.e., .i686) versions of the packages above.

/!\ It has been reported that the tests won't run properly if the SHELL variable is not set to a valid shell (e.g. /sbin/nologin instead of /bin/bash). Make sure that the user running the buildslave has a valid shell or override the SHELL environment variable somehow.

It is also recommended that you create a separate user to run the buildslave. This user should not need/have root nor SSH access; it only needs to be able to build and test GDB.

For example, let's assume this new user is called buildbot. Log into its account in the system, and create a new buildslave. To do that, you are going to need three things:

Then, you can issue the command:

$ buildslave create-slave DIR URL:PORT NAME PASSWORD

Where DIR is the directory that will be used by the buildslave to store local files (it can be the same as NAME), URL:PORT is the address of the buildmaster, and NAME is the name of the buildslave. The PASSWORD needs to be obtained from the BuildBot admin.

4.5.4. Adding information about yourself and the machine

You must add your contact information on your buildslave. To do that, edit the file named DIR/info/admin} (assuming that {{{DIR is the name of your buildslave, as per the example above) and add your name and e-mail address there:

$ cat DIR/info/admin
My Name <my@email.com>

You should also edit the file DIR/info/host and include a one-line information about the machine. For example, if it is running a Fedora GNU/Linux 23 on x86_64, you would put:

Fedora 23 x86_64

This info will show up in your buildslave's page, and it's important to keep it up-to-date whenever you upgrade or change your machine.

4.5.5. Starting the buildslave

After that, you can start the buildslave:

$ buildslave start NAME

Make sure it is started OK. It is also recommended that you create some special cronjob to restart this on every reboot. An example would be:

$ crontab -l
# restart buildslaves on every reboot

@reboot buildslave restart NAME

5. Try Jobs

The Try scheduler is now configured on our BuildBot. What this means is that a developer will be able to upload a patch to our BuildBot, and this patch will then be tested against a set of builders we have there.

5.1. Concepts

As you know, our BuildBot master runs on my (Sergio) personal server, and we have several "slaves" running on different locations. The master doesn't do any builds itself, but rather controls all the details about the builds that need to be performed. The slaves do not take any decision by themselves; they just do what the master tells them to do.

On top of the slaves, there is an abstraction called "builder". The builder is basically a recipe of how to do a build. This includes the compilation flags used, environment variables, and testsuite options. A slave can have multiple builders; a builder can also be assigned to multiple slaves (for example, when we have more than one slave with the same hardware configuration, as is the case of the Fedora x86_64 slaves).

Then, we have the schedulers. They are responsible for monitoring events and triggering builds. Our main scheduler monitors our repository, and triggers a full build on all builders whenever a new commit has been pushed. We also have "cron job" schedulers, used to trigger racy testcase detection builds every week at a certain time. And now, we have the "try job" scheduler, responsible for triggering a build on a set of builders when a developer has submitted a patch for testing.

5.2. Build priority

Before the try scheduler, builds were basically treated in a "first come, first served" basis. No special priority was assigned to a build.

Now, I decided to try a new algorithm. Considering that a developer usually want to test patches when they are almost ready for submission, it would be really annoying if she had to wait in line until her build is processed; it is not rare to see some builders having 10 or more builds in the queue. So, my decision was to "move" the try builds to the top of the queue. This will obviously delay the testing of the official repository, but I am hoping that the impact will not be very big. If it is, then this decision will have to be revisited.

5.3. Security

There are a few considerations to be made about the security of a try job. The obvious one is that a developer may upload a malicious patch to be "tested", and there is no easy way for BuildBot to detect this. Since the BuildBot master does not build anything, it would be the slaves who would suffer from this kind of attack.

But there is another thing to consider. In order to be able to upload a patch to be tested, the developer must have some kind of access to the BuildBot master. Yes, my personal server. No, I am not happy about this. I will write more on the authentication mechanism below.

So, after giving a long thought about this problem, I decided that I will only provide accounts to people who already have commit access to the GDB repository.

If you want access, you need to contact me and provide a proof that you have commit access (a commit made by you should be enough).

5.4. Authentication

The BuildBot try command can use SSH for authentication, so that is what I have chosen. Basically, you will need to send me your public SSH key; no passwords are needed/accepted. You will be using a common user to log in, and will not receive a PTY (i.e., will not have shell access). If you ever used gitolite, then you know the drill. If not, it's pretty simple.

I will provide a configuration file below.

5.5. Installing buildbot

In order to use the new feature, you will have to install the "buildbot" package. You don't need the "slave" package! On Debian, an "apt-get install buildbot" should do the trick. On Fedora, "dnf install buildbot". However, Fedora enables the BuildBot systemd service after the installation; I strongly recommend you disable that, unless you intend to run a BuildBot master on your machine.

5.6. The "buildbot try" command

After installing the package, the only command you need to know is "buildbot try". It will be responsible for generating a patch out of the git tree you're in (assuming you don't provide a patch explicitly), connecting to the server and submitting the patch to be tested.

I strongly recommend that you read the command's documentation at:

http://docs.buildbot.net/current/manual/cmdline.html#try

However, the usage is pretty basic. There are two modes: you can either be in a local branch and ask "buildbot try" to automatically figure out the patch to be tested, or you can provide the patch file explicitly.

In the first mode, "buildbot try" will generate a diff of your local branch against your "master" branch, and will instruct the BuildBot master to test the patch *using the master's branch HEAD revision*. This means that if you want to generate a patch to be tested against the latest commit of the official repository, you have to do a "git pull" on your master branch first (and rebase your local branch, of course).

The second mode accepts a patch file via the "--diff" parameter. Originally, "buildbot try" will not specify a revision against which the patch needs to be tested; however, I have modified that behaviour on the server side, and the provided patch will always be tested against the latest commit of the official repository.

5.7. Configuration file for "buildbot try"

Here is the configuration file that you need to use for the "buildbot try" command to work. This file needs to be named $HOME/.buildbot/options.

$> cat $HOME/.buildbot/options
# The method of connection to the BuildBot master.
try_connect = 'ssh'

# The host of the BuildBot master.
try_host = 'gdb-build.sergiodj.net'

# Our username.  We will always use this username!
try_username = 'gdb-buildbot'

# The jobdir on BuildBot master (i.e., the directory where "buildbot
# try" will put the request files.
try_jobdir = 'gdb/try_ssh_jobdir'

# The VCS we are using for the project.
try_vc = 'git'

# The branch against which we will try our patches.  Currently, we
# only support 'master'.
try_branch = 'master'

# Who am I?
# NOTE: YOU NEED TO USE THE FORMAT "NAME <EMAIL>" IN THIS FIELD!
try_who = 'John Smith <john@example.com>'

# The builders I want to test my patch in.
# The list of available builder can be found in:
#  <http://sourceware.org/gdb/wiki/BuildBot#Available_Builders>
try_builders = [ "Fedora-x86_64-m64",
                 "Fedora-x86_64-m32",
                 "Fedora-x86_64-native-gdbserver-m64",
                 "Fedora-x86_64-native-gdbserver-m32",
                 "Fedora-x86_64-native-extended-gdbserver-m64",
                 "Fedora-x86_64-native-extended-gdbserver-m32",
                 "Fedora-x86_64-cc-with-index",
                 "Fedora-i686",
                 "Fedora-s390x-m64",
                 "Debian-s390x-m64",
                 "Debian-s390x-native-gdbserver-m64",
                 "Debian-s390x-native-extended-gdbserver-m64",
                 "Fedora-ppc64be-m64",
                 "Fedora-ppc64le-m64",
                 "AIX-POWER8-plain",
                 "RHEL-s390x-m64" ]
$>

It is good to mention that most of these options can be overriden by command line options, so you can (for example) choose to test your patch against only a subset of the available try_builders.

5.8. Notifications

The "buildbot try" command offers a way to wait for the builds to complete on the command line, but this doesn't work when you're using SSH. However, you will receive e-mails from our BuildBot notifying you about the results of all of your try builds. The e-mails will be very similar to those sent to the gdb-testers mailing list.

5.9. Build specifics

As mentioned above, the try builds will almost always be performed against the latest commit on the official repository. They will also run using the very same flags that are being used for each builder (unfortunately, there is no way to change these flags on try builds). And the results will be compared against the last results that our BuildBot has on file.

For example, if our BuildBot has just tested the commit 0abcd on a certain builder, and then you request it to test your patch on the same builder, BuildBot will run the full build of your patch, generate the testsuite logs, and compare them against the testsuite logs generated by testing 0abcd. The known XFAIL files for that specific builder will also be taken into account, i.e., they will be discarded when comparing the testsuite logs.

Last, but not least, your try build will generate its own testsuite logs, which will be recorded in that builder's git repository, available at:

http://gdb-build.sergiodj.net/cgit

5.10. Available Builders

Here you can see the available builders to try your patches on.

try_builders = [ "Fedora-x86_64-m64",
                 "Fedora-x86_64-m32",
                 "Fedora-x86_64-native-gdbserver-m64",
                 "Fedora-x86_64-native-gdbserver-m32",
                 "Fedora-x86_64-native-extended-gdbserver-m64",
                 "Fedora-x86_64-native-extended-gdbserver-m32",
                 "Fedora-x86_64-cc-with-index",
                 "Fedora-i686"
                 "Fedora-i686",
                 "Fedora-s390x-m64",
                 "Debian-s390x-m64",
                 "Debian-s390x-native-gdbserver-m64",
                 "Debian-s390x-native-extended-gdbserver-m64",
                 "Fedora-ppc64be-m64",
                 "Fedora-ppc64le-m64",
                 "AIX-POWER8-plain",
                 "RHEL-s390x-m64" ]

None: BuildBot (last edited 2016-09-27 20:42:08 by SergioDuriganJr)

All content (C) 2008 Free Software Foundation. For terms of use, redistribution, and modification, please see the WikiLicense page.