Meeting Minutes - Office Hours for CTI - 2026-08-28

[Siddhesh Poyarekar]

On 2026-09-06 10:54, Zack Weinberg wrote:
> This is a matter of sociology, of small-group interpersonal dynamics,
> of perceived legitimacy and validity in decision making.  Reason is
> often a poor guide to these matters.
> 
> I'm not doing this to be difficult, nor because I am implacably
> imposed to your proposed move.  I'm insisting that every last word of
> the debate be posted to the libc-alpha mailing list, nowhere else,
> because *nothing else will work.*  In order to convince all
> stakeholders that this is not a stitch-up of some kind; in order to
> convince the opponents of the move that their objections have
> genuinely been heard and responded to; in order to build real
> consensus of the entire community; a debate conducted exclusively
> on libc-alpha is *what you have to do*.
> 
> It's not enough for a subgroup to reach consensus in a synchronous
> meeting, because some stakeholders cannot attend synchronous meetings.
> libc-alpha is the only venue that we know every stakeholder can use.

Yes, and the final consensus discussion did happen on the list, i.e. the 
thread in Jan/Feb.  We keep multiple avenues open because not everyone 
is comfortable expressing non-technical opinions on a mailing list, 
which was a learning for me too thanks to the CTI discussion over the years.

> It's not enough for people to declare their support on libc-alpha,
> because their arms could be being twisted in private.  The *debate*
> has to happen on libc-alpha.
> 
> It's not enough to post links, because documents behind links can be
> edited at any time.  All the text of everyone's arguments needs to be
> posted to libc-alpha so that there is a permanent record stored in an
> archive that everyone has confidence in.
> 
> I spoke up *specifically* because you responded to Mark Wielaard
> saying that he was insisting on "rehashing" points in order to "keep
> dragging this on".  He did that *because* you haven't conducted the
> entire debate on libc-alpha.  If you want this argument actually ever
> to end, the *only way* to accomplish that is to start over and conduct
> the entire debate on libc-alpha.  Nothing else will work.
> 
> ---
> 
> Moving away from the meta-discussion...
> 
>> I've mentioned in those threads the main motives I support for the
>> move: 1) a lack of scalable funding to the levels that will actually
>> support growing resource requirements of the GNU toolchain projects
>> 2) lack of a real growth plan for the sourceware overseers volunteer
>> group.
> 
> As a very occasional contributor, it *seems to me* that the glibc
> project in particular does not *need* either of these things.  It
> seems to be ticking along just fine with the resources currently
> available to it.  However, I could easily not be aware of problems
> behind the scenes.  In order to understand why you believe additional
> funding and a growth plan are needed, I would like to see time series
> of disk usage, CPU load, network bandwidth, and human sysadmin time
> consumed by sourceware.org, both in aggregate and on a per-project and
> per-service basis, for at least the past ten years.  If graphs are
> not already available, raw numbers are fine.
> 
> If it is actually one of the bigger projects hosted on sourceware
> that's outgrowing it, then why is *that* project not taking on the
> risk and the switching costs?

All of the toolchain projects are hosted on *one* set of servers (a load 
balancing pair and a backup IIRC, I don't remember how the 3 machines 
are set up at the moment) and we're basically switching the smallest 
project because that's the lowest risk from a GNU toolchain front. 
glibc has historically also been the most nimble project in terms of 
embracing large changes.

>> Finally, the proposal at the moment is limited to moving critical
>> infrastructure (git and email) to LF IT, not replace sourceware.
>> Since some of the overseers decided to continue supporting
>> sourceware (and we came to terms with it) we realized that it's
>> better to have the use of both infrastructures.
> 
> If anything is to be moved at all, it seems to me that the most
> critical infrastructure (git and email, as you say) should move
> *last*.  Wouldn't it be easier for everyone involved, and also
> substantially *safer* -- much more practical to revert or abandon if
> something goes wrong -- if we started by having LF provide
> *non-essential* services?  A build farm, perhaps, or a read-only
> mirror of the website and/or the git repos, or an experimental
> alternative patch tracker.  The possibility of moving over more
> important services could then be revisited, in a few years, after
> everyone had had a chance to build confidence in LF hosting both on a
> technical and an organizational level.

We don't need LF IT level funding for mirrors.  It is the primary 
infrastructure and admin team that hasn't been scaling, not the 
mirroring.  In fact, mirrors that we've tried to maintain routinely go 
out of sync (the gitlab gnu toolchain mirrors for example) because the 
primary servers have some challenge or another due to AI scrapers.

>> I don't think any commercial organization is immune from the LLM
>> garbage.  I "trust" the Linux Foundation as much as I trust Red Hat
>> (my employer, although that hat is strictly off when I communicate
>> upstream) or even SFC or FSF, which is limited.  Over the years I've
>> realized that every commercial (including non-profit) organization
>> is more than willing to bend and manipulate public emotions and
>> sometimes even facts to their benefit.
> 
> That's fair, but, again, we've been doing fine on sourceware for
> decades, and, unlike LF, nobody has accused them of being a vehicle
> for a corporate takeover of the project.  It seems to me that this
> is a "better the devil we know" kind of situation.

No we've not been doing fine, we've been adjusting for many years now 
and the LF IT transition is the opportunity to get better resources that 
we have decided to take advantage of.

>> https://sourceware.org/pipermail/libc-alpha/2026-January/174602.html
>> https://sourceware.org/pipermail/libc-alpha/2026-February/174850.html
>> https://sourceware.org/pipermail/libc-alpha/2026-February/174852.html
> 
> The first of these is just a list of named supporters of the move;
> I explained above why that's not a useful response.

It's not a convincing explanation just because it speaks to your 
argument.  Like I said, most people feel deeply comfortable getting 
involved in discussions that are not patch reviews.  If you sample 
non-technical discussion threads on libc-alpha for example, you'll see a 
very small, specific set of people who tend to engage.  It's not because 
they don't have opinions, it's because they don't want to get tied up in 
it.  Please accept that most regular glibc contributors are actually OK 
with this move, as you've seen in the responses to the Jan/Feb threads 
and meeting notes from the CTI office hours.

> The second and third are rather more substantial, and I do regret
> having missed them in the archives.  Let me respond here (quotes
> shuffled around for ease of response)
> 
>> I expressed the concern in one of the Cauldrons that there were
>> barely a handful of us doing this work in our spare time and that
>> doesn't seem scalable, nor the fact that we're not really full time
>> system administrators (I guess I'm not a "real programmer").  This
>> was apparently a known concern for a while, especially because the
>> infrastructure story was struggling on multiple fronts
> ...
>> Our users are working around our deficiencies; wolfi and others use
>> mirrors for their downstream distros because sourceware cannot handle
>> that load and in some cases, has outright banned their IPs.
> 
> I don't believe in scaling for scaling's sake.  It's far too easy to
> build something over-complicated and over-expensive in the name of
> scaling, that actually winds up being less efficent and less resilient
> to actual conditions.
> 
> I would also be inclined to say that if particular downstream users of
> our projects find themselves repeatedly pulling the same large files
> off of sourceware, then they *should* be hosting their own caches of
> those files.  Perhaps that's dreadfully old-fashioned of me, but it
> puts the expenses on the people creating the costs, so it's the
> economically correct solution.

Many of those organizations are willing to fund this infrastructure and 
dedicated adminstration services through the Linux Foundation, which is 
what we're using with this move.

> Again, I'd like to see time series of actual resource consumption by
> sourceware.org, broken down by project, over a substantial period of
> time - ten years or more.
> 
>> the number of active volunteer overseers were dwindling (2 then and
>> still the same now) and the sourceware hardware that hosts core
>> services for the GNU toolchain was (and still is) funded solely by
>> Red Hat, hosted in a Red Hat datacentre.  It was 2 machines then (3
>> now) including backups, running *all* of the core infrastructure.
> 
> I do agree that it would be better to have organizationally redundant
> hosting, and professional sysadmin support, but I think these things
> could be more easily achieved with incremental steps from where we are
> now, than with a "big bang" move.  I'll address people power below;
> regarding organizationally redundant hosting, again, why are we
> beginning with a *move* of the two most critical services -- the most
> disruptive and risky step we could take, and one that *doesn't* create
> redundancy -- and not something safer, such as read-only mirrors?

Because this is not about setting up mirroring, it's about moving and 
scaling primary infrastructure.

>> Having a registered vendor account with large companies is one thing
>> and getting those large companies to actually pay up for a cause is
>> quite another.  I've seen Carlos and David run from pillar to post
>> all these years, trying to drum up sponsors for everything from the
>> Cauldron to the infrastructure, it's hard
> 
> This actually makes me *less* inclined to support the move!  Funding
> that can only be had by agreeing to a big list of conditions is
> funding that is liable to grow even more conditions in the future,
> or dry up altogether.  It is, in my opinion, probably *not* in the
> project's long-term interest to accept such funding, *even if*
> it's the only funding available at present, and even if that causes
> major short-term problems.

The funding was not conditional, we requested this.  We want the full 
time IT support in addition to scalable infrastructure.

>> Everything ran on those machines, alongside each other.
>> Compromising one service would mean pretty much everything would be
>> compromised.  I remember Carlos talking about service isolation (and
>> how we can't really do it on sourceware because we don't have the
>> hardware capacity to do that) as far back as our first conversation
>> in 2016 when I complained about patchwork maintenance.
> 
> Sandboxing is a desirable thing, but it's been my experience that the
> combination of good old Unix user- and group-based access controls,
> with systemd's various service sandboxing controls, is more than
> sufficient to achieve the same level of *practical* isolation
> guarantees that you can get with anything more heavyweight
> (e.g. bolt-on mandatory access control, docker-type containers,
> virtual machines).  It's also much, much easier to deploy on an
> existing server, and much easier to fix when something goes wrong.

I disagree, we need physical separation where we can, not just to reduce 
performance cross-talk but also to isolate the infrastructure.

>> FWIW, you say that those issues could be fixed in situ, but they
>> were summarily dismissed when we raised them in 2022.  Changes have
>> happened for the better over the years since, but they have been
>> limited because there's no funding and people are working in their
>> spare time to support this.
> 
> So it was my impression that the "CTI" initiative did in fact secure
> funding, that there were now people paid to be overseers, and that
> there was now organizational willingness to move forward with
> incremental service isolation.

The CTI funding has not been secured for sourceware, it is for git and 
email transition and administrator support.  I am not aware of any paid 
full time overseers.

Sid