CI/CD for glibc

This page documents the design of the continuous integration and continuous development framework that will be used by glibc.

Scope

The following items are in scope:

The following items are out of scope:

Moving away from email-based workflow.
The actions taken by the CI/CD system (we care only that you can run something, not what it is).
Existing glibc build system.
Modifying patchwork.
Events which are not associated with a patch (or series) and/or its lifecycle.

Framework requirements are as follows:

Integrates with email-based workflow.
Do not modify patchwork downstream. Make changes upstream.
Provides automatic feedback to submitted patches.
Provides a quick review quickly e.g. patch applies to current tree, patch passes linter, patch passes format for bug designation.
Provides automatic feedback about build results on key architectures.
Provide a method for authenticated reviewers to trigger try bots on restricted resources.

Project - this is a combination of a patchwork project, a mailing list, and a git repo.
Curator - this is the part that takes in all the events happening in patchwork, mail, and git, and "curates" the event stream, to produce a smaller set of events for runners to trigger off.
Runner - this is one of many servers which watch for curated events, and trigger one or more Trybots when events they care about happen.
Trybot - this is a module which knows how to perform some service and report status back
Event - this is some action, change, or request, associated with a patch, including all the related meta data.

What does an example review session look like for a reviewer:

Patch author sends email to libc-alpha with patch formatted following patch contribution guidelines.
Patchwork v2 detects the patch and generates a new patch object for the patch.
CI/CD Framework detects patchwork v2 patch creation event (REST API).
Framework authenticates user.
Framework decides to generate trybot request based on authorization for user and policy.
Framework exposes curated event list for Runners to review.
Runners review curated event list and decide which local trybot to run based on event type.
Runner may have additional policy and ignore the request.
Runner passes event to local trybot.
Trybot carries out action e.g. applies patch, does a built, etc.
Trybot interfaces directly to patchwork and mailing list as needed to send results.
- Trybot updates patchwork check status for SUCCESS, WARNING or FAIL.
- Trybot sends email to the mailing list with status.
- Trybot must not be able to modify patch status, external project maintainer scripts can be used to mark patches as rejected if they fail basic trusted linters etc.

Components:

CI/CD Framework that talks to Patchwork aka Curator (Python scripts)
Fully documented REST API for the CI/CD Framework
Sample CI/CD Runner (Python + Shell)
Sample Buildbot master configuration
Sample Trybots for building glibc and binutils
- Includes examples for updating Patchwork check statuses and sending results to mailing list.
Ansible playbook to setup Runner, buildbot master, and trybots in containers on a single VM.
- Documentation on how to configure the samples to interface with the CI/CD Framework.

With...

buildbot - buildbot's master could be one of the TryBots, perhaps wrapped in a thin shim
patchwork bots - if the bot changes the patchwork state in a way that generates an event, we'll work off that event.

2021-02-12:

We have had frequent developer requests for retrying patches in the review queue against current main branch.

CICDmainflow.svg is generated from CICDmainflow.gv using dot