Meeting Minutes - Office Hours for CTI - 2026-08-28

[Zack Weinberg]

On Tue, Sep 8, 2026, at 6:48 PM, Siddhesh Poyarekar wrote:
> On 2026-09-06 10:54, Zack Weinberg wrote:
>> I'm not doing this to be difficult, nor because I am implacably
>> imposed to your proposed move.  I'm insisting that every last word of
>> the debate be posted to the libc-alpha mailing list, nowhere else,
>> because *nothing else will work.*  In order to convince all
>> stakeholders that this is not a stitch-up of some kind; in order to
>> convince the opponents of the move that their objections have
>> genuinely been heard and responded to; in order to build real
>> consensus of the entire community; a debate conducted exclusively on
>> libc-alpha is *what you have to do*.
>>
>> It's not enough for a subgroup to reach consensus in a synchronous
>> meeting, because some stakeholders cannot attend synchronous
>> meetings. libc-alpha is the only venue that we know every stakeholder
>> can use.
>
> Yes, and the final consensus discussion did happen on the list, i.e.
> the thread in Jan/Feb.  We keep multiple avenues open because not
> everyone is comfortable expressing non-technical opinions on a mailing
> list, which was a learning for me too thanks to the CTI discussion
> over the years.

There *is no final consensus*, and there cannot be, not until and unless
the *entire planning process and debate, from the beginning* is re-done
on the mailing list. The Jan/Feb thread was not enough, because it
started from a plan already mostly hashed out, and which therefore could
be understood as a stitch-up.

I am quite aware that some people are uncomfortable speaking up on
nontechnical topics on what is nominally a technical mailing list;
however, that has to be balanced against the fact that other people
cannot participate in synchronous meetings, and, perhaps more
importantly right now, the fact that a technical mailing list is the
only venue for project-wide decision making that *already had consensus
support for use as that venue.*

People who are uncomfortable speaking up directly can bring their
concerns to people who they trust to relay those concerns to the mailing
list; anonymized or pseudonymized reporting would be fine, as long as
someone is in a position to attest that no sock puppetry is going on.
That's the farthest compromise position I can accept under present
circumstances.

It might make sense, down the road, *after* the hosting argument is
finally resolved, to take a hard look at our decision-making procedures
and think about how they could be improved, particularly with respect
to encouraging more people to express their opinions; but changing our
decision-making procedures *in the middle of a controversial decision
process* is completely unacceptable.  That would look like the process
was being manipulated to force a particular outcome, whether or not it
actually was.

So, again, I say:
>> If you want this argument actually ever to end, the *only way* to
>> accomplish that is to start over and conduct the entire debate on libc-
>> alpha.  Nothing else will work.

...
> All of the toolchain projects are hosted on *one* set of servers (a
> load balancing pair and a backup IIRC, I don't remember how the 3
> machines are set up at the moment)

And that might be more than enough!  If you have evidence that it's not,
please present it.  I asked Mark for the same thing just now, but if you
have access to those time series of load metrics that I mentioned on
Tuesday, that would really help.

> We don't need LF IT level funding for mirrors.  It is the primary
> infrastructure and admin team that hasn't been scaling, not the
> mirroring.

Again, I need to see concrete, quantitative evidence here.  If the
problem is really as bad as you say, the numbers to prove it should not
be hard to come by.

Please understand that until I see data to warrant it, I will be
feeling no sense of urgency whatsoever.  Right now, I have no concrete
reason to think that there's any actual technical problem with
remaining on the existing server hardware and network infrastructure
for the foreseeable future.

> In fact, mirrors that we've tried to maintain routinely go out of sync
> (the gitlab gnu toolchain mirrors for example) because the primary
> servers have some challenge or another due to AI scrapers.

Another really helpful piece of evidence that you (or anyone!) could
provide here is specifics of what has already been done to block AI
scrapers and to what extent it helped.  I know Anubis has been tried and
hasn't been much help.  Has iocaine (with aggressive firewalling) been
tried?  Has anything else been tried?

> Please accept that most regular glibc contributors are actually OK
> with this move, as you've seen in the responses to the Jan/Feb threads
> and meeting notes from the CTI office hours.

No.  Not good enough, on two fronts: "most regular glibc contributors"
is not the full set of people who need to agree that the move is a good
idea, and, as stated above, the *process failure* can, at this point,
only be remedied by restarting the entire planning process from the
beginning.

Please accept that *there is no consensus*, and that consensus *cannot
be achieved* starting from where we are.  The entire thing has to be
redone from scratch or we'll never reach a decision that all
stakeholders are actually okay with.

>> I would also be inclined to say that if particular downstream users
>> of our projects find themselves repeatedly pulling the same large
>> files off of sourceware, then they *should* be hosting their own
>> caches of those files.  Perhaps that's dreadfully old-fashioned of
>> me, but it puts the expenses on the people creating the costs, so
>> it's the economically correct solution.
>
> Many of those organizations are willing to fund this infrastructure
> and dedicated adminstration services through the Linux Foundation,
> which is what we're using with this move.

That's more complicated and indirect than having those organizations run
their own local caches.  Is there a concrete reason why local caches
don't work for them, or for us?

>> again, why are we beginning with a *move* of the two most critical
>> services -- the most disruptive and risky step we could take, and one
>> that *doesn't* create redundancy -- and not something safer, such as
>> read-only mirrors?
>
> Because this is not about setting up mirroring, it's about moving and
> scaling primary infrastructure.

Well, again, I want to see actual data backing up the claim that the
primary infrastructure needs to be moved, but *even if* that really is
the most urgent need, just the risk of something going
catastrophically wrong during the move and producing an extended
outage and/or need to restore from backups is enough, in my book, to
make "move the critical services first" a bad idea.  Debug the
*procedure for moving things over* on the non-critical services first:
that's the only sensible way to do it.

Moving, or, better, replicating non-critical services first would also
be better from a consensus-building perspective.  You have confidence in
LF, but several other people don't; *why not* give them a chance to see
for themselves that their worries about LF are unfounded?

>>> Having a registered vendor account with large companies is one thing
>>> and getting those large companies to actually pay up for a cause is
>>> quite another.  I've seen Carlos and David run from pillar to post
>>> all these years, trying to drum up sponsors for everything from the
>>> Cauldron to the infrastructure, it's hard
>>
>> This actually makes me *less* inclined to support the move!  Funding
>> that can only be had by agreeing to a big list of conditions is
>> funding that is liable to grow even more conditions in the future, or
>> dry up altogether.  It is, in my opinion, probably *not* in the
>> project's long-term interest to accept such funding, *even if* it's
>> the only funding available at present, and even if that causes major
>> short-term problems.
>
> The funding was not conditional, we requested this.  We want the full
> time IT support in addition to scalable infrastructure.

You misunderstand me.  "I've seen Carlos and David run from pillar to
post all these years, trying to drum up sponsors for everything from the
Cauldron to the infrastructure, it's hard" implies that the only way you
*did* manage, in the end, to get funding for this move was to accept
strings -- particularly, reading between the lines of what else has been
said, that the funding is only available if funneled through LF and
ultimately used to pay for LF-based services.  And that's a yellow flag.
That suggests that there might be more strings in the future.  Hypothetically,
we might find ourself in the same position as the Python Software
Foundation was last year, forced to choose between accepting conditions
incompatible with the project's mission, and losing a big chunk of funding.
Only worse, since in this scenario, "losing a big chunk of funding" might
equate to "losing the use of the primary git server".

>> it's been my experience that the
>> combination of good old Unix user- and group-based access controls,
>> with systemd's various service sandboxing controls, is more than
>> sufficient to achieve the same level of *practical* isolation
>> guarantees that you can get with anything more heavyweight (e.g. bolt-
>> on mandatory access control, docker-type containers, virtual
>> machines).  It's also much, much easier to deploy on an existing
>> server, and much easier to fix when something goes wrong.
>
> I disagree, we need physical separation where we can, not just to
> reduce performance cross-talk but also to isolate the infrastructure.

Isolate the infrastructure against what threat?  We're not at any
realistic risk of one of the other projects on sourceware.org trying
to sabotage us.

>> So it was my impression that the "CTI" initiative did in fact secure
>> funding, that there were now people paid to be overseers, and that
>> there was now organizational willingness to move forward with
>> incremental service isolation.
>
> The CTI funding has not been secured for sourceware, it is for git and
> email transition and administrator support.  I am not aware of any
> paid full time overseers.

My apologies, I misremembered "CTI" as the work Mark and company have
been doing to secure sponsors for sourceware, not your initiative.

zw