Sourceware GNU Toolchain Infrastructure (*) and beyond

Presenter Notes

The title is a bit of a lie, let's try again.

HTML created by python landslide. Press 'h' for help. For more background info see presentor notes. Press 'p'.

New services for sourceware projects

sourceware.org is one of the original free software project hosting services. It hosts dozens of projects, including some of the GNU toolchain components.

It is community operated, Red Hat subsidized infrastructure.

Presenter Notes

This is about infrastructure that the GNU toolchain projects use on Sourceware, together with many other projects. And it isn't directly about the Infrastructure itself, but about how you want to use it and maybe about which policies projects want to adopt that the infrastructure makes possible (setting project policies isn't the job of sourceware overseers, but we love to help making sure the infrastructure is ready).

Goal and vision

Email is awesome, but how can we combine our discussion based patch reviews with patch tracking and test automation?

Provide zero maintenance infrastructure for tracking and automation of patches, testing and analyzing testresults.

And how do we keep improving and innovating as a community run free software infrastructure project?

Sourceware was started in 1998, what about the next 24 years?

Presenter Notes

At Red Hat we are trying to push CI testing upstream. We have to push fixes upstream if we find them in products. So try to test and find regressions earlier. Younger generation has trouble contributing to projects that primarily use email. For elfutils we got some new contributors who struggle with providing patches. gccrs frontend. project on github, but wants to integrate with gcc. There are two main things that make them productive, automated testing and easy patch tracking. Finally while setting up the new services this year we started discussing a good structure for making sure Sourceware keeps around for another 24 years. Best to have this discussion while there are no known problems or issues.

Discussions

Discuss how to use the infrastructure, builder, try/ci/full buildbots, bunsen testsuite analyzis, patchwork patch tracking, CICD trybot/buildbot integration, handling patches/email with public-inbox and the sourcehut mirror and other automation to make you more productive and happy to contribute.

Discuss the future of the Sourceware infrastructure as a Software Freedom Conservancy member project (having a fiscal sponsor).

Presenter Notes

So we believe we are providing the infrastructure services for projects to automate email based patch reviews with patch tracking, but it is up to the projects to see how much they use the services and whether they want to set policies on how to use them.

After discussion with the communty the Software Freedom Conservancy voted to accepted us as member project so we should soon have a fiscal sponsor.

builder.sourceware.org

An installation of Buildbot (python), along with a local community, mailinglist, git repo, container files, self-configuring.

Includes various compute resources:

  • virtual machines, bare metal, containers
  • distros (fedora, debian, centos, ubuntu, opensuse)
  • architectures (i386, x86_64, ppc64le, s390x, ppc64, armhf, arm64)
  • Thanks: Brno University, Marist University, Thomas Fitzsimmons, Mark Wielaard, Frank Eigler, IBM, The Works on Arm initiative and OSUOSL.
  • More please!

Presenter Notes

Show https://builder.sourceware.org/

Three kinds of builders

  • Try builders - "pre-commit" - reports to author only. (binutils, elfutils, gdb and libabigail)
  • CI builders - "quick regression tests", reports to author and list on regression. (bzip2, dwz, debugedit, libabigail, elfutils, glibc, gcc, gccrs, valgrind, gdb, binutils)
  • Full builders - "all (slow) tests, not all commits, not all arches" (gcc x86_64/arm64/armhf, glibc x86_64/arm64, gdb-binutils x86_64, gccrust-bootstrap x86_64/arm64)

All results, reported or not, go into bunsen. 10.000+ builds a month.

Presenter Notes

Try builders normally run all tests as the CI builders. Full builders are only needed for those projects that have heavy/slow tests.

Try Example

  • git checkout -b frob
  • hack, hack, hack... OK, looks good to submit
  • git commit -a -m "Awesome hack"
  • git push origin frob:users/mark/try-frob
  • ... wait for the emails to come in or watch buildbot logs ...
  • Send in patches and mention what the try bot reported

After your patches have been accepted you can delete the branch again: - git push origin :users/mark/try-frob

Currently enabled for binutils, elfutils, gdb and libabigail.

Presenter Notes

My favorite kind of builder!

bunsen testrun storage & analysis

An installation of bunsen (git://sourceware.org/git/bunsen.git) (python), under rapid development. irc: #bunsen on irc.libera.chat.

Stores all test-related logs + metadata from each testrun into one commit of a ordinary dedicated git repo (git://sourceware.org/git/bunsendb.git).

Small python analysis scripts parse new log files from git into an sqlite database. Analysis passes build on each other. Testruns are clustered by metadata similarities.

Small command line reporting tools browse/search git + sqlite data, show diffs/regressions. More coming.

Includes a simple web frontend (https://builder.sourceware.org/testruns/).

Easy to run your own! https://sourceware.org/git/?p=bunsen.git;a=blob;f=README

Presenter Notes

Sourceware testrun logs includes numerous & huge dejagnu, automake, autoconf log files, and more. Bunsen stores them in git.

Git compresses amazingly: 1TB of raw text from all 45.000 buildbot runs fits into 10GB. Use your knowledge of ordinary git to replicate, subset, garbage-collect, authenticate.

Let's flip through some screenshots of the web interface.

bunsen web: list testruns

Presenter Notes

Each testrun uses its git commit hash as identifier.

bunsen web: testrun metadata

Presenter Notes

Here we see the metadata for this particular testrun.

From the "filelist" and other tabs, one can explore what the system has found out about this testrun: all the files, all the test cases.

The table's right columns let one navigate to related testruns that share given metadata values, or else precede or follow it. Let's compare this testrun to one that came from the previous elfutils source version: "source.gitdescribe", "prev", "delta".

bunsen web: compare two testruns

Presenter Notes

Here we see the differences in test results between the two builds. Nothing too worrysome. Notice the autoconf values are also differenced. We can click on each log file to instantly see the exact line context of each test case.

The regression mode limits differences to only pass->fail type transitions, omitting skipped results and other such less serious results.

bunsen web: see details of one test case

Presenter Notes

Here we see the specific logfile region that accounts for the skipped run-large-elf-file.sh.trs case.

There is much more to see, clicking around. Please excuse the plain looks, it's old school HTML with simple self-contained server tech.

patchwork.sourceware.org

An installation of patchwork (python/django).

git pw is awesome

patchwork plus CI/CD - Let's use those buildbot workers too

Presenter Notes

I use it as a personal TODO for elfutils, not really a team effort. gdb used it but abandoned glibc does seem to use it as a group with weekly patch reviews some gcc hackers use it? Why does patchwork work for some, but not others?

inbox.sourceware.org

An installation of public-inbox (perl).

A better mail archive and so much more....

Allows people to "subscribe" to the list through atom, nntp, imap

Easy way to have mailinglist mirrors, including local (git like) mirror

Try out piem (public-inbox emacs mode) or b4 tools.

Presenter Notes

All based on Message-IDs.

Make email lists like git.

Not a push, but a pull resource.

b4 example

.git/config

[b4]
    midmask = https://inbox.sourceware.org/dwz/%s
    linkmask = https://inbox.sourceware.org.org/dwz/%s

 $ b4 am a2bf576d-3598-385d-2139-cae0d4f11074@suse.cz
 Looking up https://inbox.sourceware.org/dwz/a2bf576d-3598-385d-2139-cae0d4f11074%40suse.cz
 Grabbing thread from inbox.sourceware.org/dwz/a2bf576d-3598-385d-2139-cae0d4f11074%40suse.cz/t.mbox.gz
 Analyzing 1 messages in the thread
 Checking attestation on all messages, may take a moment...
 ---
   ✓ [PATCH] Use grep -E instead of egrep.
   ---
   ✓ Signed: DKIM/suse.cz
 ---
Total patches: 1
---
 Link: https://inbox.sourceware.org.org/dwz/a2bf576d-3598-385d-2139-cae0d4f11074@suse.cz
 Base: applies clean to current tree
       git am ./20220907_mliska_use_grep_e_instead_of_egrep.mbx

Presenter Notes

b4 example (cont)

$ b4 ty --auto
Auto-thankanating commits in master
Found 2 of your commits since 1.week
Calculating patch hashes, may take a moment...
  Located: [PATCH] Use grep -E instead of egrep.
  Located: [PATCH] Fix executable stack warning from linker
---
Generating 2 thank-you letters
  Writing: ./mliska_suse_cz_patch_use_grep_e_instead_of_egrep_.thanks
  Writing: ./mliska_suse_cz_patch_fix_executable_stack_warning_from_linker.thanks
---
You can now run:
  git send-email ./*.thanks

Presenter Notes

Patch attestation

Either:

  • Awesome way to "close" the secure software supply chain
  • Security "theater" that will exclude people and reduce code reviews to "has the submitter jumped through code signing hoops"

Presenter Notes

Example showed DKIM verification, but this can also be done for gpg signed email messages or special patchatt "signed" patch emails.

Experiment: sourcehut

https://sr.ht/~sourceware/

A more webby git workflow alternative

git send-email without the email

Presenter Notes

Not many people seem to use it, it might not be known well enough. Was tried to submit gccrs patches, but it wasn't a success?

If it doesn't work out we can still keep the git mirrors.

Sourceware as Conservancy member

First off: things are fine and stable.

The next 24 years of Sourceware?

Why? In case of future financial or organizational needs.

Reached out to Software Freedom Conservancy a few months ago. Wrote up an application last month, reached out to "all" sourceware users. SFC offered project membership to Sourceware last week.

What is a fiscal sponsor?

Does anything change? Hardly.

Independent from any guest projects!

In particular the GNU toolchain projects have the FSF as fiscal sponsor, nothing changes about that.

Continue to have public discussions on overseers mailing list and public video chats with Conservancy.

Presenter Notes

There currently isn't really a need, so now is a good time to set something up. This is just about sourceware providing free software project hosting for the next few decades. Projects still decide how or if they want to use the services. We don't want to decide any policy for any guest project. Talked to FSF to make sure this isn't in conflict with the FSF providing fiscal sponsorship to the GNU Toolchain projects or a way to undermine GNU projects leadership. We believe this is a non-event. But we want to be as transparent as possible. If this isn't a non-event we are doing it wrong. We want to know if any of this is controversial. All discussions are open and public on the overseers mailinglist. Video chats with Conservancy have been open to all on BBB.