Meeting Minutes - Office Hours for CTI - 2026-08-28

[Zack Weinberg]

On Fri, Sep 4, 2026, at 12:40 PM, Siddhesh Poyarekar wrote:
> On 2026-09-04 12:04, Zack Weinberg wrote:
>> On Fri, Sep 4, 2026, at 11:27 AM, Siddhesh Poyarekar wrote:
>> In <https://sourceware.org/pipermail/libc-alpha/2026-January/174676.html>
>> you said "we did the groundwork" and posted a link to
>> <https://cti.coretoolchain.dev/projects/enum.html>.  *Only* text
>> posted to this mailing list counts, so I did not review the linked
>> document.  (To be clear, I'm being utterly inflexible about
>> this because the mailing list archive is an *archive*.  We can count
>> on what's in the mailing list archive staying the same for decades
>> into the future.  No such guarantee exists for documents posted on
>> external sites.)
>
> I don't think that's a reasonable position to argue from.

This is a matter of sociology, of small-group interpersonal dynamics,
of perceived legitimacy and validity in decision making.  Reason is
often a poor guide to these matters.

I'm not doing this to be difficult, nor because I am implacably
imposed to your proposed move.  I'm insisting that every last word of
the debate be posted to the libc-alpha mailing list, nowhere else,
because *nothing else will work.*  In order to convince all
stakeholders that this is not a stitch-up of some kind; in order to
convince the opponents of the move that their objections have
genuinely been heard and responded to; in order to build real
consensus of the entire community; a debate conducted exclusively
on libc-alpha is *what you have to do*.

It's not enough for a subgroup to reach consensus in a synchronous
meeting, because some stakeholders cannot attend synchronous meetings.
libc-alpha is the only venue that we know every stakeholder can use.

It's not enough for people to declare their support on libc-alpha,
because their arms could be being twisted in private.  The *debate*
has to happen on libc-alpha.

It's not enough to post links, because documents behind links can be
edited at any time.  All the text of everyone's arguments needs to be
posted to libc-alpha so that there is a permanent record stored in an
archive that everyone has confidence in.

I spoke up *specifically* because you responded to Mark Wielaard
saying that he was insisting on "rehashing" points in order to "keep
dragging this on".  He did that *because* you haven't conducted the
entire debate on libc-alpha.  If you want this argument actually ever
to end, the *only way* to accomplish that is to start over and conduct
the entire debate on libc-alpha.  Nothing else will work.

---

Moving away from the meta-discussion...

> I've mentioned in those threads the main motives I support for the
> move: 1) a lack of scalable funding to the levels that will actually
> support growing resource requirements of the GNU toolchain projects
> 2) lack of a real growth plan for the sourceware overseers volunteer
> group.

As a very occasional contributor, it *seems to me* that the glibc
project in particular does not *need* either of these things.  It
seems to be ticking along just fine with the resources currently
available to it.  However, I could easily not be aware of problems
behind the scenes.  In order to understand why you believe additional
funding and a growth plan are needed, I would like to see time series
of disk usage, CPU load, network bandwidth, and human sysadmin time
consumed by sourceware.org, both in aggregate and on a per-project and
per-service basis, for at least the past ten years.  If graphs are
not already available, raw numbers are fine.

If it is actually one of the bigger projects hosted on sourceware
that's outgrowing it, then why is *that* project not taking on the
risk and the switching costs?

> Finally, the proposal at the moment is limited to moving critical
> infrastructure (git and email) to LF IT, not replace sourceware.
> Since some of the overseers decided to continue supporting
> sourceware (and we came to terms with it) we realized that it's
> better to have the use of both infrastructures.

If anything is to be moved at all, it seems to me that the most
critical infrastructure (git and email, as you say) should move
*last*.  Wouldn't it be easier for everyone involved, and also
substantially *safer* -- much more practical to revert or abandon if
something goes wrong -- if we started by having LF provide
*non-essential* services?  A build farm, perhaps, or a read-only
mirror of the website and/or the git repos, or an experimental
alternative patch tracker.  The possibility of moving over more
important services could then be revisited, in a few years, after
everyone had had a chance to build confidence in LF hosting both on a
technical and an organizational level.

> I don't think any commercial organization is immune from the LLM
> garbage.  I "trust" the Linux Foundation as much as I trust Red Hat
> (my employer, although that hat is strictly off when I communicate
> upstream) or even SFC or FSF, which is limited.  Over the years I've
> realized that every commercial (including non-profit) organization
> is more than willing to bend and manipulate public emotions and
> sometimes even facts to their benefit.

That's fair, but, again, we've been doing fine on sourceware for
decades, and, unlike LF, nobody has accused them of being a vehicle
for a corporate takeover of the project.  It seems to me that this
is a "better the devil we know" kind of situation.

> https://sourceware.org/pipermail/libc-alpha/2026-January/174602.html
> https://sourceware.org/pipermail/libc-alpha/2026-February/174850.html
> https://sourceware.org/pipermail/libc-alpha/2026-February/174852.html

The first of these is just a list of named supporters of the move;
I explained above why that's not a useful response.

The second and third are rather more substantial, and I do regret
having missed them in the archives.  Let me respond here (quotes
shuffled around for ease of response)

> I expressed the concern in one of the Cauldrons that there were
> barely a handful of us doing this work in our spare time and that
> doesn't seem scalable, nor the fact that we're not really full time
> system administrators (I guess I'm not a "real programmer").  This
> was apparently a known concern for a while, especially because the
> infrastructure story was struggling on multiple fronts
...
> Our users are working around our deficiencies; wolfi and others use
> mirrors for their downstream distros because sourceware cannot handle
> that load and in some cases, has outright banned their IPs.

I don't believe in scaling for scaling's sake.  It's far too easy to
build something over-complicated and over-expensive in the name of
scaling, that actually winds up being less efficent and less resilient
to actual conditions.

I would also be inclined to say that if particular downstream users of
our projects find themselves repeatedly pulling the same large files
off of sourceware, then they *should* be hosting their own caches of
those files.  Perhaps that's dreadfully old-fashioned of me, but it
puts the expenses on the people creating the costs, so it's the
economically correct solution.

Again, I'd like to see time series of actual resource consumption by
sourceware.org, broken down by project, over a substantial period of
time - ten years or more.

> the number of active volunteer overseers were dwindling (2 then and
> still the same now) and the sourceware hardware that hosts core
> services for the GNU toolchain was (and still is) funded solely by
> Red Hat, hosted in a Red Hat datacentre.  It was 2 machines then (3
> now) including backups, running *all* of the core infrastructure.

I do agree that it would be better to have organizationally redundant
hosting, and professional sysadmin support, but I think these things
could be more easily achieved with incremental steps from where we are
now, than with a "big bang" move.  I'll address people power below;
regarding organizationally redundant hosting, again, why are we
beginning with a *move* of the two most critical services -- the most
disruptive and risky step we could take, and one that *doesn't* create
redundancy -- and not something safer, such as read-only mirrors?

> Having a registered vendor account with large companies is one thing
> and getting those large companies to actually pay up for a cause is
> quite another.  I've seen Carlos and David run from pillar to post
> all these years, trying to drum up sponsors for everything from the
> Cauldron to the infrastructure, it's hard

This actually makes me *less* inclined to support the move!  Funding
that can only be had by agreeing to a big list of conditions is
funding that is liable to grow even more conditions in the future,
or dry up altogether.  It is, in my opinion, probably *not* in the
project's long-term interest to accept such funding, *even if*
it's the only funding available at present, and even if that causes
major short-term problems.

> Everything ran on those machines, alongside each other.
> Compromising one service would mean pretty much everything would be
> compromised.  I remember Carlos talking about service isolation (and
> how we can't really do it on sourceware because we don't have the
> hardware capacity to do that) as far back as our first conversation
> in 2016 when I complained about patchwork maintenance.

Sandboxing is a desirable thing, but it's been my experience that the
combination of good old Unix user- and group-based access controls,
with systemd's various service sandboxing controls, is more than
sufficient to achieve the same level of *practical* isolation
guarantees that you can get with anything more heavyweight
(e.g. bolt-on mandatory access control, docker-type containers,
virtual machines).  It's also much, much easier to deploy on an
existing server, and much easier to fix when something goes wrong.

> FWIW, you say that those issues could be fixed in situ, but they
> were summarily dismissed when we raised them in 2022.  Changes have
> happened for the better over the years since, but they have been
> limited because there's no funding and people are working in their
> spare time to support this.

So it was my impression that the "CTI" initiative did in fact secure
funding, that there were now people paid to be overseers, and that
there was now organizational willingness to move forward with
incremental service isolation.

Mark, is this accurate?  Could we maybe get a report on what has been
done for service isolation already, and what more could still be done
without hardware upgrades?  And, while I'm asking, what the current
state of funding is for sourceware, and to what extent it is still
being maintained on a volunteer basis?  As a reminder, for the reasons
given up top, I'm not looking at anything that has only been linked
here.  If you have documents that already address these questions,
please post their full text to the mailing list.

zw