Moving buildbot master to sourceware

[Mark Wielaard]

Hi Carlos,

On Mon, Apr 04, 2022 at 06:15:01PM -0400, Carlos O'Donell via Overseers wrote:
> > Maintenance of the master is fairly minimal and it doesn't require
> > that many resources. The current state database, git poller and build
> > logs total less than 3.5G for hundreds to thousands of builds for all
> > of the above projects.
> > 
> > The bike shed items for installing this on sourceware are:
> > 
> > - Whether to install buildbot from pip or epel. My preference is
> >   throug pip under a user account since that is how I run it
> >   locally. But using the epel package is fine too.
> > - Running it under sourceware.org/buildbot or adding a vhost
> >   builder.sourceware.org. My preference is for a vhost.
> > - A git repo for the master.cfg file builder.git (*)
> >   Plus a git trigger to reload to config by the buildbot master.
> > - A user, group and homedir to store the state and password/key files
> >   /sourceware/home/builder/
> > - The initial members of the builder group which can update the git
> >   repo and key files. The people on CC?
> > - A mailinglist to coordinate builder@sourceware.org or we could
> >   simply use the overseers mailinglist.
> 
> Can we deploy this as a buildbot master container with some customization?
> 
> This would make the service easier to keep going during sourcware updates because
> dependencies would be decoupled.
> 
> It would allow developers to test pre-production changes locally by using the same
> container.

The most natural "container" for a pure python app that we'll be
maintaining through pip is a python virtualenv. I have scripts to
setup and update the virtualenv so it is easy to replicate and play
with independently. You can wrap those in a whole system image of
course. I'll make sure to document that process.

> This is how upstream patchwork changes are being developed, which DJ and I are
> looking at because we will need to update patchwork to v3.1 upstream when it's
> released with DJ's new changes.

Is there documentation on how sourceware patchwork is currently
deployed? Are you actually running a patches upstream version? I am
happy to help upgrading.

I haven't had to make any changes to upstream/packaged/pip buildbot
because all configuration is done through a python file (so if you do
really have to monkey patch the upstream version you can add those to
the config file).

> My plan was to work with downstream distributions, ISVs, and IHVs who are interested
> in a particular project to donate workers. In order to do that though we have to make
> the act of installing and setting up a worker very low friction and very safe.
> 
> The workers should arguably have a net positive impact for the downstream, or ISV, or
> IHV, so they need a way to measure that they prevented a patch that breaks their
> configuration of interest. So I also need to make sure they can gather metrics for
> reporting on the resources they are spending e.g. how soon after going red on their
> build was it corrected?
> 
> For example, could Red Hat or IBM donate workers?
> 
> Can we grow best practice CI adoption in our mature projects by making it easy?

All good ideas and, if you have the matching project developers, then
adding more workers is in general a good thing.

There are a few lessons about making CI most effective:

- Start small with a minimal environment and a minimal subset of tests
  that just have to work.
- Always keep the builder green.
- If it is a flaky test leave it out.
- If there is a test that is failing consistently, but nobody fixes
  it, file a bug and take it out.
- Only expand if you have developers that care to keep that environment green.
- Only add workers if you have devxelopers on the project that can take
  care of the new build environments created for it.
- If you have a worker that is unique in some way (say for a
  non-common architecture) arrange for project maintainers to get
  access to it outside the buildbot so they can actually debug
  problems.

For tracking actual success/failures/added/removed tests we could look
at feeding the test logs from buildbot to bunsen.

Cheers,

Mark