BuildBot
Goal
Test GDB for each new repository commit and report any regressions for it. Test also regressions for user-provided patches.
Regressions get detected by running the thorough GDB testsuite and (a bit fuzzy) comparison of its results against the previous build results. (optional, can be done by the GDB team)
Roadmap
- Make builds more reliable e.g add local git mirror to avoid git timeouts
- Move all native builds to cross builds.
- Support releases as well as trunk
- Support testing various configs within one "build", e.g., with/without .gdb_index, with/without dwarf4 type units, with gcc or clang, etc., etc., etc.
- Add new operating systems like windows, freebsd, openbsd
- Add new architectures like mips, powerpc, ia64
- Run tests for available targets and store results in a database
- Write a script which detects regressions
- Add a ARI step
- Add performance testsuite
Opens
- How build fails and regression should be reported e.g. via irc, email to the offender or gdb-testers
- How to trigger build/test runs prior to commits to the CVS
- How to deal with reporting false regression positives due to fuzzy testsuite results
- E.g., rerun all failing tests multiple times, and then report as "consistently failing" or "flaky" (or whatever)
- Performance testsuite
- Results are reported differently, both within one build and over time
- Results are better with otherwise unused machines
Feature list
Different configs (even within one "build") have different expected fails. It is often easier to maintain expected fails in separate files associated with the config, and use that to assist in report generation.
Compatibility is required with GDB repository in both CVS and GIT.
Support various provided (as rpms, distros or upstream repository branches/tags) GCC versions for building GDB and its testing workload.
Support multiple testing hosts with various architectures. Beaker test system integration must not be required for x86* but it should be supported to be able to cover also other arches.
Support running tests inside various kinds of virtualized hosts. Either mock (=chroot) or the full KVM virtual host. (It may be just provided by Beaker, interfaced probably by libvirt.)
Support running multi-host testsuite using gdbserver (remote GDB backend).
Properly report kernel crashes of hosts under the test. Associate the kernel crash reports with the GDB workload under test.
(optional) Integration with Fedora/RHEL mock (OS distribution provisioning) and the rpm packaging to test also new Fedora/RHEL builds with it.
(optional) Besides GDB to regression test also binutils.
Supporting other features of existing buildbot solutions.
Analyze existing buildbot solutions, we could just deploy one of them:
Mozilla http://buildbot.net ,
GCC testing http://gcc.gnu.org/testing
- a bot in use for GDB in Red Hat by Jan Kratochvil:
Hudson: http://hudson-ci.org/docs/HudsonIntro.pdf last page has some summary of such other existing tools
- others