Meeting Minutes - Office Hours for CTI - 2026-08-28

[Mark Wielaard]

Hi Zack,

On Sun, Sep 06, 2026 at 10:54:22AM -0400, Zack Weinberg wrote:
> On Fri, Sep 4, 2026, at 12:40 PM, Siddhesh Poyarekar wrote:
> > FWIW, you say that those issues could be fixed in situ, but they
> > were summarily dismissed when we raised them in 2022.  Changes have
> > happened for the better over the years since, but they have been
> > limited because there's no funding and people are working in their
> > spare time to support this.
> 
> So it was my impression that the "CTI" initiative did in fact secure
> funding, that there were now people paid to be overseers, and that
> there was now organizational willingness to move forward with
> incremental service isolation.
> 
> Mark, is this accurate?  Could we maybe get a report on what has been
> done for service isolation already, and what more could still be done
> without hardware upgrades?  And, while I'm asking, what the current
> state of funding is for sourceware, and to what extent it is still
> being maintained on a volunteer basis?  As a reminder, for the reasons
> given up top, I'm not looking at anything that has only been linked
> here.  If you have documents that already address these questions,
> please post their full text to the mailing list.

I can only talk about what Sourceware has been doing technically and
as organization. We already have done the hardware upgrades, which was
the funding goal for last year. We now have various physical and
virtual machines in two datacenters supported by two different
organizations.
https://sourceware.org/sourceware-wiki/servers-and-services-2026/
(Text version below [1])

Maintenance is split between staff at RH OSPO (taking care of the
hardware and connectivity in RDU3), OSUOSL (same and some virtual
machines in SDC), SFC (taking care of all administrativia, contracts,
funding matters, etc) and "volunteers" (but mostly as project
maintainers/developers are volunteers, for some part of their job is
helping out with some services).

Funding comes from individuals, corporations (both in-kind and
financial) and grant makers. We just hit our goal to be able to hire
some contractor(s) to help with migrating the remaining services into
their own isolated VMs. A strech goal is to hire a part-time Junior
System Administrator. https://sourceware.org/financials.html
(Text version below [2])

Note those are goals for 2026/2027, current numbers were posted to
this list last month:
https://sourceware.org/pipermail/libc-alpha/2026-August/180073.html

I am not sure who summarily dismissed which issues in 2022, but I
think that must have been some misunderstanding. We have worked hard
since 2022 to turn Sourceware into a reliable organization and
technical infrastructure.

Cheers,

Mark

[1] Sourceware Servers and Services 2026

End of 2025, start of 2026, Sourceware servers were moved into new
datacenters from Red Hat (RDU3) and OSUOSL (SDC). All servers are
VM-first now and there are no public services running on bare metal
anymore. We also take advantage of the RH OSPO cloud (RDU3) and OSUOSL
x86_64 and arm64 openstack setups (SDC) for extra compute, redundancy
and backups.

Bare-metal servers in RH RDU3 (running vms)

    server1, 112 cores, 1.5Ti mem, 14T raid6 disk
    server2, 64 cores, 500Gi mem, 4.4T raid6 disk
    server3, 64 cores, 188Gi mem, 4.4T raid6 disk

These servers aren't directly accessible, but do provide 8 public ips
throughout various vms (vm01 to vm08) that are publicly accessible.

The vms in use are:

    core services (dns, httpd, ssh, git) on vm01 (56 vcores, 500G mem, 4TB disk)
    forge (forgejo) on vm02 (16 vcores, 48G mem, 200G disk)
    forge-stage (forgejo) on vm08 (2 vcores, 4G mem, 40G disk)
    debuginfod (elfutils debuginfod) on vm07 (2 vcores, 8G mem, 50GB disk).

Future vms:

    inbox (public-inbox, 6 vcores, 16GB mem, 160GB disk)
    patchwork (patchwork, 4 vcores, 12GB mem, 30GB disk)
    buildbot (buildbot, 4 vcores, 32GB mem, 48GB disk)
    bunsen (bunsen, 8 vcores, 12GB mem, 500GB disk).

With additional vms possible for cygwin, gitweb/cgit, bugzilla,
dwarfstd and valgrind.

RH OSPO cloud server

    rh-ospo-sourceware01, 3 vcores, 6Gi mem, 8G+60G vdisks (old forge, now backup)

Bare-metal server in OSUOSL SDC (running vms)

    sourceware-builder3, 112 cores, 754Gi mem, 465G raid1 disk, 2*932G nvme

This has been partitioned into 2 buildbot workers for
https://builder.sourceware.org and 2 forgejo runners for
https://forge.sourceware.org

    sw3bb1: 40 vcpus, 256GB mem, 500GB disk (Fedora Core OS, containers)
    sw3bb2: 16 vcpus, 96GB mem, 420GB disk (Fedora Core OS, containers)
    sw3runner1: 40 vcpus, 256GB mem, 500GB disk (Debian, Forgejo Runner)
    sw3runner2: 16 vcpus, 96GB mem, 420GB disk (Debian, Forgejo Runner)

OSL cloud servers

    snapshots, 4 vcores, 8Gi mem, 620G vdisk (x86_64, alma9)
    osuosl-arm64, 8 vcores, 16Gi mem, 160G vdisk (arm64, fedora)
    osuosl-arm64-2, 16 vcores, 32Gi mem, 320G vdisk (arm64, debian)

Other servers and services

The following servers and services have been donated by companies,
organizations or individuals but are not administrated directly by
Sourceware overseers.

    Linaro CI https://ci.linaro.org does CI builds against patchwork and forge merge requests (Linaro, Maxim Kuvyrkov, https://gitlab.com/LinaroLtd/tcwg/ci/)
    redhat-pt-bot does try builds against patchwork (Red Hat, DJ Delorie, https://gitlab.com/djdelorie/glibc-cicd)
    fedora-ppc64le buildbot worker (Brno University, Dan Horák)
    fedora-s390x buildbot worker (Marist University, Dan Horák)
    debian-ppc64[be] buildbot worker (Thomas Fitzsimmons)
    debian-i386 and debian-armhf buildbot workers (Mark Wielaard)
    fedrawhide-x86_64 buildbot worker (Frank Eigler)
    ibm_power9 and ibm_power10 buildbot workers (IBM, Carl Love)
    starfive-[1-4] buildbot workers, VisionFive-2 risc-v boards (donated by Starfive, run by Mark Wielaard)
    Milk-V Pioneer Box donated by RISC-V International and SOPHGO (decommissioned)
    stap-*-x86_64 buildbot workers (Red Hat, Serhei Makarov)

[2] Sourceware financials 2026/2027

We ended the previous year with the successful completion of our
infrastructure hardware refresh cycle. Doubling of individual
donations and increased (corporate) "in-kind" support for hosting and
networking support.

For the 2026/2027 fiscal year, we would like to shift our budget focus
to spend less on "iron" and more on "services and people". We would
like to finally provide financial support/compensation to our partner
OSUOSL. Hire paid staff/consultants to help with services upgrades and
VM migrations. And look into funding upstream Forgejo improvements.

Our budget and funding targets three "layers" of sponsors. Individual
sponsors, Corporate sponsors and Grant Makers.

Individual Sponsors: "Empowering the Maintainers"
Independence and a worry-free developer experience

Individual developers and contributors who rely on Sourceware for
their daily work. Sourceware is the worry-free home for the tools you
build and use every day. Our focus is on making your workflow faster,
more secure and something you don't have to think about. It is just
there. Thanks to your help we moved to a modern, VM-first
infrastructure and are experimenting with a dedicated Forge to
streamline collaboration. Your small, monthly contributions are the
lifeblood of our independence, ensuring that the core infrastructure
remains community-run and developer-focused.

    2026/2027 target: $4,000-$6,000

        $500 Domain fees, banking, and essential operational overhead
        $1,000 Contribution toward OSUOSL colocation and bandwidth
        $3,000 Forgejo improvements
        $1,500 Emergency hardware replacement fund

Corporate Sponsors: "Securing the Global Supply Chain"
Infrastructure security and accelerating engineering velocity

Sourceware hosts the projects at the very heart of the Free Software
supply chain. We are seeking corporate partners to help us
professionalize our security posture. By sponsoring Sourceware, you
aren’t just donating; you are insuring the integrity of the tools your
business is built upon. Our focus is on modernizing, isolating, and
automating our services to stay ahead of the evolving cybersecurity
landscape. And eliminating administrative tasks so engineers can focus
on their projects.

    2026/2027 target: $4,000-$12,000

        4x $2,500-$4,000 Consultants to upgrade, VM isolation and
        automation for bugzilla, public-inbox, patchwork and buildbot.
        In kind donations of builder machines and server hosting

Grant Makers: "Sustainable Infrastructure for the Next 25 Years"
Sustainability, Governance, and Global Impact

Sourceware provides the foundational infrastructure that has anchored
the core Free Software toolchain and developer tools for over 25
years. We are seeking institutional partners to help us preserve this
deep institutional knowledge and safeguard the digital commons for the
next quarter century. As a member project of the Software Freedom
Conservancy, we pair our historical expertise with professional asset
stewardship and independent community governance. Through our
collaboration with corporate partners and the Open Source Lab (OSUOSL)
we provide Free Infrastructure for Free Software projects. Grant
funding will empower us to scale this critical cooperation, formalize
our standard operating procedures, and transfer decades of technical
expertise to the next generation of admins and developers.

    2026/207 target $20,000-$40,000

        To hire a part-time Junior System Administrator (potentially
        in partnership with OSUOSL). This role will document Standard
        Operating Procedures (SOPs), handle daily administrative
        tasks, and ensure the long-term sustainability of the
        infrastructure.