Sourceware infrastructure updates for Q2 2025

[Carlos O'Donell]

On 7/30/25 5:43 PM, Mark Wielaard wrote:
> Hi Carlos,
> 
> Thanks for your interest in the Sourceware infrastructure
> improvements.
> 
> On Wed, Jul 30, 2025 at 09:35:47AM -0400, Carlos O'Donell wrote:
>> On 7/29/25 8:35 PM, Mark Wielaard wrote:
>>> = Sourceware servers on the move
>>>
>>>    All our servers will be moving later this year because both our
>>>    hardware services partners will move datacenters. The Sourceware PLC
>>>    decided to take advantage of this move by adding more/bigger
>>>    machines.
>>>
>>>    - Red Hat Community Cage server move
>>>
>>>    This https://www.osci.io/tenants/ impacts server2 (main server),
>>>    server3 (backup server) and forge.sourceware.org. We will add a new
>>>    bigger server which has 3x memory [24x64GB], 10x storage [6x3.84TB],
>>>    2x cpu ish [2x28 cores] compared to the current servers. The new
>>>    data center also has a a faster/bigger network pipe.
>>>
>>>    The new server1 was made possible thanks to the FUTO grant,
>>>    individual Sourceware donations and Red Hat OSPO CommInfra & IT
>>>    teams. It has already been installed in the new RDU3 data
>>>    center. But doesn't have network yet (will be added in two weeks).
>>
>> I really appreciate that Sourceware is attempting to address the SSDLC
>> issues that we've been talking about since 2022.
> 
> Yes, this is the implementation of the security vision we published
> back then https://sourceware.org/sourceware-security-vision.html
> 
>> However, now I have *further* concerns that Sourceware has 3 servers
>> that need ongoing financial support, but I haven't seen a budget that
>> outlines:
>>
>>   * All the costs involved in operations, including estimates for
>>     the OSCI provided services in the event we need to find replacements.
>>
>>   * Future costs involved with the replacement of 3, instead of 2, servers.
>>
>> Does Sourceware have funding for these costs from sponsors?
>>
>> Does the Sourceare PLC have ongoing yearly sponsors?
> 
> Yes, see also our yearly reports, which include some financial data as
> well. It was important to the PLC that we would be able to replace any
> new hardware from the ongoing donations in ~3 years and still have
> enough money left in our hardware replacement fund to replace any one
> server in case it has to be replaced immediately.
> 
> We have taken guidance from the SFC, which has helped dozens of
> projects ramp up from zero revenue to having sustainable revenue to
> support their volunteer operations. We're right on track; ramping up
> funds without clear planning and demonstrated need often backfires.
> 
> We also have multiple infrastructure partners so we can fallback to
> another in case one of them might stop providing some or all of their
> services. To make sure servers can easily be moved to another
> infrastructure partner we also have agreements in place about the
> hardware being owned by us.

Just for clarity Sourceware PLC has ~$10K in the Sourceware PLC accounts.

And some of this was spent on the new server?

In ~3 years you would have another ~$10K added (based on estimates of
$250/month in donations) for a total of ~$20K?

While this is a good amount of money, it isn't a budget.

For example, I would expct:

  * CAPEX - What needs replacing in 3 years?
  * OPEX  - What is the estimated cost of operations in a data center?

May the Sourceware PLC please put together a budget that includes forcasts?

>> The purpose of CTI is to resolve this by finding larger corporate
>> sponsors to support the costlier core infrastructure with recurring
>> membership from LF members. All of which is a mechanism that existing
>> corporate members understand and know how to fund.
> 
> That would be really nice. See https://sourceware.org/donate.html or
> if the CTI, LF or OpenSSF want to know how to best sponsor Sourceware
> plans, donate hardware or services, email sponsor@sourceware.org.  For
> larger and/or more ongoing sponsorships, we can set up a conversation
> between your leads and Karen at SFC (as Sourceware is an SFC member
> project).

Please note that CTI's goal is to pay for the infrastructure services
directly, not to give an arm's length donation to a charity.

My experience is that donations which don't have a direct link to a
particular use e.g. paying for hardware, paying for IT staff, are much
harder to acquire from sponsors.

CTI's goals are not to provide another mechanism for charitable donations.
  
> Additionally, last year, thanks to Zoe (FSF's Executive Director), the
> PLC had some nice conversation with the OpenSSF and they seemed
> willing to fund some of the plans directly
> https://sourceware.org/sourceware-security-vision.html#plans
> But some of the OpenSSF staff left and they changed general managers
> twice since then. We have reengaged contact and they are still
> interested in helping us improve the security infrastructure, but they
> don't seem to have the funds to sponsor any FOSS infrastructure right
> now.

It may be the case that they don't have the funds to make a "charitable
donation" because such a donation is an arm's length giving of funds for
use in whatever ways the charity deems meets the mission.

CTI is still funded, and in the future the CTI TAC would continue to work
with the same sponsors to fund directed hardware projects, but not charitable
donations (much harder to acquire).

>>>    We like to have the new server1 setup and in production before the
>>>    move of the other two servers so there is a minimum of downtime. We
>>>    discussed a plan to do this and how we can use this for moving some
>>>    services in their own isolated VMs, and which resources need to be
>>>    untangled for that at the last Open Office hour.
>>>
>>>    https://sourceware.org/sourceware-wiki/Migration2025/
>>
>> Have you discussed this plan with the GNU Toolchain projects?
> 
> Yes, this particular plan comes from the discussion during the last
> few Open Office meetings where we explicitly asked for feedback:
> https://inbox.sourceware.org/20250615235100.GA14424@gnu.wildebeest.org
> https://inbox.sourceware.org/20250709223052.GG13630@gnu.wildebeest.org
> The PLC has several members who are maintainers and/or stewards of
> various projects. And we are making sure to announce each step in our
> monthly Open Office call for participation, these Quarterly reports
> and our year reports.

Please note that sending emails out is not the same as engagement.

Please also note that the Sourceware PLC does not have GNU Project
maintainer representation from any of the GNU Toolchain projects.
While it has some developers from the projects, that isn't the same
as having direct engagement with those responsible for the projects.

I encourage you to discuss these issues directly with more than just
mail outs that might be missed by busy maintainers.
  
> Note that we are moving deliberately slowly making sure people have
> enough time to give feedback. Also there is no urgency, the Red Hat
> OSPO CommInfra & IT teams know we want to get this right and will take
> a couple of months to first setup the new server1 before doing the
> actual move.

Thanks.

>> Can we review the plan against glibc's SSDLC plan?
> 
> Please feel free. You probably also want to review the Sourceware
> Cyber Security FAQ and Recommendations for Sourceware hosted projects:
> https://sourceware.org/cyber-security-faq.html
> 
>> It is not sufficient for us to just say what we're doing, we need to
>> engage with the project leadership and ensure they understand the
>> impact of the change and the choices available (if any).
> 
> Hope this has helped. We don't want to force change on any hosted
> project, but we do want to be pro-active guiding the hosted projects
> onto a more powerful and modern infrastructure.
> 
>>>    - OSUOSL datacenter move
>>>
>>>    This https://osuosl.org/communities/ impacts sourceware-builder1,
>>>    sourceware-builder2, arm64-1, arm64-2 and the snapshots server. The
>>>    OSL might be able to upgrade the first two CI x86_64 builders which
>>>    would be great since we expect the experimental forge to also want
>>>    to add CI for merge requests. But they would like us to cover some
>>>    of the co-location hosting costs if possible.
>>
>> It makes complete sense that OSUOSL would start asking for funding.
>>
>> What are those costs and how large are they?
>>
>> How do you plan to fund those costs?
> 
> We can easily pay the requested costs for a couple of years from our
> current funds available. But we first want to discuss our budget with
> the SFC to make sure that doesn't mean we run out of funds and
> recurring donations that we have reserved to pay for other stuff. We
> plan to do a full budgeting process with SFC in the next 3-6 months.
> SFC has 20 years experience budgeting for volunteer-oriented FOSS
> projects.

Looking forward to seeing the budget.

> Note that this really isn't a demand of the OSUOSL. They don't want it
> to become a burden for us. It really is a larger community issue for
> making the OSUOSL sustainable:
> https://osuosl.org/blog/osl-future-update/
> We support their efforts, but we also don't rely solely on their
> infrastructure.
> 
> Again we encouraged the OpenSSF to sponsor OSUOSL, but they said they
> don't have the funds at the moment to help out.

That falls under the same problem I've discussed earlier, you are not
asking for help, you're asking for a no strings attached donation that
will be used to meet the charitable mission of the project.

The OpenSSF and other organizations prefer to have a more direct approach
to solving the specific community security needs.

-- 
Cheers,
Carlos.