Sourceware infrastructure updates for Q2 2025

[Mark Wielaard]

Hi Carlos,

Thanks for your interest in the Sourceware infrastructure
improvements.

On Wed, Jul 30, 2025 at 09:35:47AM -0400, Carlos O'Donell wrote:
> On 7/29/25 8:35 PM, Mark Wielaard wrote:
> >= Sourceware servers on the move
> >
> >   All our servers will be moving later this year because both our
> >   hardware services partners will move datacenters. The Sourceware PLC
> >   decided to take advantage of this move by adding more/bigger
> >   machines.
> >
> >   - Red Hat Community Cage server move
> >
> >   This https://www.osci.io/tenants/ impacts server2 (main server),
> >   server3 (backup server) and forge.sourceware.org. We will add a new
> >   bigger server which has 3x memory [24x64GB], 10x storage [6x3.84TB],
> >   2x cpu ish [2x28 cores] compared to the current servers. The new
> >   data center also has a a faster/bigger network pipe.
> >
> >   The new server1 was made possible thanks to the FUTO grant,
> >   individual Sourceware donations and Red Hat OSPO CommInfra & IT
> >   teams. It has already been installed in the new RDU3 data
> >   center. But doesn't have network yet (will be added in two weeks).
>
> I really appreciate that Sourceware is attempting to address the SSDLC
> issues that we've been talking about since 2022.

Yes, this is the implementation of the security vision we published
back then https://sourceware.org/sourceware-security-vision.html

> However, now I have *further* concerns that Sourceware has 3 servers
> that need ongoing financial support, but I haven't seen a budget that
> outlines:
>
>  * All the costs involved in operations, including estimates for
>    the OSCI provided services in the event we need to find replacements.
>
>  * Future costs involved with the replacement of 3, instead of 2, servers.
>
> Does Sourceware have funding for these costs from sponsors?
>
> Does the Sourceare PLC have ongoing yearly sponsors?

Yes, see also our yearly reports, which include some financial data as
well. It was important to the PLC that we would be able to replace any
new hardware from the ongoing donations in ~3 years and still have
enough money left in our hardware replacement fund to replace any one
server in case it has to be replaced immediately.

We have taken guidance from the SFC, which has helped dozens of
projects ramp up from zero revenue to having sustainable revenue to
support their volunteer operations. We're right on track; ramping up
funds without clear planning and demonstrated need often backfires.

We also have multiple infrastructure partners so we can fallback to
another in case one of them might stop providing some or all of their
services. To make sure servers can easily be moved to another
infrastructure partner we also have agreements in place about the
hardware being owned by us.

> The purpose of CTI is to resolve this by finding larger corporate
> sponsors to support the costlier core infrastructure with recurring
> membership from LF members. All of which is a mechanism that existing
> corporate members understand and know how to fund.

That would be really nice. See https://sourceware.org/donate.html or
if the CTI, LF or OpenSSF want to know how to best sponsor Sourceware
plans, donate hardware or services, email sponsor@sourceware.org.  For
larger and/or more ongoing sponsorships, we can set up a conversation
between your leads and Karen at SFC (as Sourceware is an SFC member
project).

Additionally, last year, thanks to Zoe (FSF's Executive Director), the
PLC had some nice conversation with the OpenSSF and they seemed
willing to fund some of the plans directly
https://sourceware.org/sourceware-security-vision.html#plans
But some of the OpenSSF staff left and they changed general managers
twice since then. We have reengaged contact and they are still
interested in helping us improve the security infrastructure, but they
don't seem to have the funds to sponsor any FOSS infrastructure right
now.

> >   We like to have the new server1 setup and in production before the
> >   move of the other two servers so there is a minimum of downtime. We
> >   discussed a plan to do this and how we can use this for moving some
> >   services in their own isolated VMs, and which resources need to be
> >   untangled for that at the last Open Office hour.
> >
> >   https://sourceware.org/sourceware-wiki/Migration2025/
>
> Have you discussed this plan with the GNU Toolchain projects?

Yes, this particular plan comes from the discussion during the last
few Open Office meetings where we explicitly asked for feedback:
https://inbox.sourceware.org/20250615235100.GA14424@gnu.wildebeest.org
https://inbox.sourceware.org/20250709223052.GG13630@gnu.wildebeest.org
The PLC has several members who are maintainers and/or stewards of
various projects. And we are making sure to announce each step in our
monthly Open Office call for participation, these Quarterly reports
and our year reports.

Note that we are moving deliberately slowly making sure people have
enough time to give feedback. Also there is no urgency, the Red Hat
OSPO CommInfra & IT teams know we want to get this right and will take
a couple of months to first setup the new server1 before doing the
actual move.

> Can we review the plan against glibc's SSDLC plan?

Please feel free. You probably also want to review the Sourceware
Cyber Security FAQ and Recommendations for Sourceware hosted projects:
https://sourceware.org/cyber-security-faq.html

> It is not sufficient for us to just say what we're doing, we need to
> engage with the project leadership and ensure they understand the
> impact of the change and the choices available (if any).

Hope this has helped. We don't want to force change on any hosted
project, but we do want to be pro-active guiding the hosted projects
onto a more powerful and modern infrastructure.

> >   - OSUOSL datacenter move
> >
> >   This https://osuosl.org/communities/ impacts sourceware-builder1,
> >   sourceware-builder2, arm64-1, arm64-2 and the snapshots server. The
> >   OSL might be able to upgrade the first two CI x86_64 builders which
> >   would be great since we expect the experimental forge to also want
> >   to add CI for merge requests. But they would like us to cover some
> >   of the co-location hosting costs if possible.
>
> It makes complete sense that OSUOSL would start asking for funding.
>
> What are those costs and how large are they?
>
> How do you plan to fund those costs?

We can easily pay the requested costs for a couple of years from our
current funds available. But we first want to discuss our budget with
the SFC to make sure that doesn't mean we run out of funds and
recurring donations that we have reserved to pay for other stuff. We
plan to do a full budgeting process with SFC in the next 3-6 months.
SFC has 20 years experience budgeting for volunteer-oriented FOSS
projects.

Note that this really isn't a demand of the OSUOSL. They don't want it
to become a burden for us. It really is a larger community issue for
making the OSUOSL sustainable:
https://osuosl.org/blog/osl-future-update/
We support their efforts, but we also don't rely solely on their
infrastructure.

Again we encouraged the OpenSSF to sponsor OSUOSL, but they said they
don't have the funds at the moment to help out.

Cheers,

Mark