Meeting Minutes - Office Hours for CTI - 2026-08-28

[Siddhesh Poyarekar]

On 2026-09-10 17:30, Zack Weinberg wrote:
> On Tue, Sep 8, 2026, at 6:48 PM, Siddhesh Poyarekar wrote:
>> On 2026-09-06 10:54, Zack Weinberg wrote:
>>> I'm not doing this to be difficult, nor because I am implacably
>>> imposed to your proposed move.  I'm insisting that every last word of
>>> the debate be posted to the libc-alpha mailing list, nowhere else,
>>> because *nothing else will work.*  In order to convince all
>>> stakeholders that this is not a stitch-up of some kind; in order to
>>> convince the opponents of the move that their objections have
>>> genuinely been heard and responded to; in order to build real
>>> consensus of the entire community; a debate conducted exclusively on
>>> libc-alpha is *what you have to do*.
>>>
>>> It's not enough for a subgroup to reach consensus in a synchronous
>>> meeting, because some stakeholders cannot attend synchronous
>>> meetings. libc-alpha is the only venue that we know every stakeholder
>>> can use.
>>
>> Yes, and the final consensus discussion did happen on the list, i.e.
>> the thread in Jan/Feb.  We keep multiple avenues open because not
>> everyone is comfortable expressing non-technical opinions on a mailing
>> list, which was a learning for me too thanks to the CTI discussion
>> over the years.
> 
> There *is no final consensus*, and there cannot be, not until and unless
> the *entire planning process and debate, from the beginning* is re-done
> on the mailing list. The Jan/Feb thread was not enough, because it
> started from a plan already mostly hashed out, and which therefore could
> be understood as a stitch-up.

I disagree, the entire planning process, even if conversations did not 
necessarily start on libc-alpha, is on record here over multiple years, 
with the final revival in Jan/Feb and has extensive documentation across 
libc-alpha and the cti mailing list archives, not to mention the various 
wikis you've been pointed at that you're refusing to look at.  It's 
unreasonable to ask that the whole thing start over when we're already 
far enough along.

> I am quite aware that some people are uncomfortable speaking up on
> nontechnical topics on what is nominally a technical mailing list;
> however, that has to be balanced against the fact that other people
> cannot participate in synchronous meetings, and, perhaps more
> importantly right now, the fact that a technical mailing list is the
> only venue for project-wide decision making that *already had consensus
> support for use as that venue.*
> 
> People who are uncomfortable speaking up directly can bring their
> concerns to people who they trust to relay those concerns to the mailing
> list; anonymized or pseudonymized reporting would be fine, as long as
> someone is in a position to attest that no sock puppetry is going on.
> That's the farthest compromise position I can accept under present
> circumstances.

That doesn't make sense; you're willing to accept potentially strawman 
arguments but not statements of support from actual people because it 
satisfies the image of having had a debate?

> It might make sense, down the road, *after* the hosting argument is
> finally resolved, to take a hard look at our decision-making procedures
> and think about how they could be improved, particularly with respect
> to encouraging more people to express their opinions; but changing our
> decision-making procedures *in the middle of a controversial decision
> process* is completely unacceptable.  That would look like the process
> was being manipulated to force a particular outcome, whether or not it
> actually was.
> 
> So, again, I say:
>>> If you want this argument actually ever to end, the *only way* to
>>> accomplish that is to start over and conduct the entire debate on libc-
>>> alpha.  Nothing else will work.
> 
> ...
>> All of the toolchain projects are hosted on *one* set of servers (a
>> load balancing pair and a backup IIRC, I don't remember how the 3
>> machines are set up at the moment)
> 
> And that might be more than enough!  If you have evidence that it's not,
> please present it.  I asked Mark for the same thing just now, but if you
> have access to those time series of load metrics that I mentioned on
> Tuesday, that would really help.
> 
>> We don't need LF IT level funding for mirrors.  It is the primary
>> infrastructure and admin team that hasn't been scaling, not the
>> mirroring.
> 
> Again, I need to see concrete, quantitative evidence here.  If the
> problem is really as bad as you say, the numbers to prove it should not
> be hard to come by.
> 
> Please understand that until I see data to warrant it, I will be
> feeling no sense of urgency whatsoever.  Right now, I have no concrete
> reason to think that there's any actual technical problem with
> remaining on the existing server hardware and network infrastructure
> for the foreseeable future.

While I personally don't have quantitative data, I have several 
anecdotal experiences over the last decade+ where cross service noise 
led to performance issues, e.g. bugzilla slowing down git clones.

>> In fact, mirrors that we've tried to maintain routinely go out of sync
>> (the gitlab gnu toolchain mirrors for example) because the primary
>> servers have some challenge or another due to AI scrapers.
> 
> Another really helpful piece of evidence that you (or anyone!) could
> provide here is specifics of what has already been done to block AI
> scrapers and to what extent it helped.  I know Anubis has been tried and
> hasn't been much help.  Has iocaine (with aggressive firewalling) been
> tried?  Has anything else been tried?

One thing that was tried recently was to ban IP blocks, which went awry 
because it ended up locking legimitate users out, across services. 
Eventually Mark worked with Fedora admins to figure out a way to do this 
more reasonably so that we don't get locked out.

>> Please accept that most regular glibc contributors are actually OK
>> with this move, as you've seen in the responses to the Jan/Feb threads
>> and meeting notes from the CTI office hours.
> 
> No.  Not good enough, on two fronts: "most regular glibc contributors"
> is not the full set of people who need to agree that the move is a good
> idea, and, as stated above, the *process failure* can, at this point,
> only be remedied by restarting the entire planning process from the
> beginning.
> 
> Please accept that *there is no consensus*, and that consensus *cannot
> be achieved* starting from where we are.  The entire thing has to be
> redone from scratch or we'll never reach a decision that all
> stakeholders are actually okay with.

I apologize, I've been saying consensus where I've meant a significant 
majority with only a couple of sustained objections.  I don't think 
consensus can be achieved in this case because the sustained objections 
are based on conditions that cannot be met, e.g. by your insistence of 
restarting the conversation to attempt to achieve consensus or 
objections over SaaSS, antagonistic feelings for the LF or the 
perception that there's no need to scale infrastructure or have 
professional, full time infrastructure administration.

>> Many of those organizations are willing to fund this infrastructure
>> and dedicated adminstration services through the Linux Foundation,
>> which is what we're using with this move.
> 
> That's more complicated and indirect than having those organizations run
> their own local caches.  Is there a concrete reason why local caches
> don't work for them, or for us?

I don't understand the argument you're making here, why shouldn't they 
simply fund upstream infrastructure when they have an easy opportunity 
to do so instead of standing up their own mirrors?

>>> again, why are we beginning with a *move* of the two most critical
>>> services -- the most disruptive and risky step we could take, and one
>>> that *doesn't* create redundancy -- and not something safer, such as
>>> read-only mirrors?
>>
>> Because this is not about setting up mirroring, it's about moving and
>> scaling primary infrastructure.
> 
> Well, again, I want to see actual data backing up the claim that the
> primary infrastructure needs to be moved, but *even if* that really is
> the most urgent need, just the risk of something going
> catastrophically wrong during the move and producing an extended
> outage and/or need to restore from backups is enough, in my book, to
> make "move the critical services first" a bad idea.  Debug the
> *procedure for moving things over* on the non-critical services first:
> that's the only sensible way to do it.
> 
> Moving, or, better, replicating non-critical services first would also
> be better from a consensus-building perspective.  You have confidence in
> LF, but several other people don't; *why not* give them a chance to see
> for themselves that their worries about LF are unfounded?

Because there's no real value in moving non-critical services over to LF.

>> The funding was not conditional, we requested this.  We want the full
>> time IT support in addition to scalable infrastructure.
> 
> You misunderstand me.  "I've seen Carlos and David run from pillar to
> post all these years, trying to drum up sponsors for everything from the
> Cauldron to the infrastructure, it's hard" implies that the only way you
> *did* manage, in the end, to get funding for this move was to accept
> strings -- particularly, reading between the lines of what else has been
> said, that the funding is only available if funneled through LF and
> ultimately used to pay for LF-based services.  And that's a yellow flag.

Most companies find it easier to support upstream projects through 
consortiums like LF or Linaro because it's streamlined for them.  For 
example if ChipMakerCo is alread a LF member, it's easier for them to 
direct a portion of their membership funding (or kick up their funding 
by, say, 5%) to the CTI for a setup they know well (professional IT 
support) than to make a fresh case for a new vendor to fund volunteer 
sysadmins; the hardware bit is less important in this context.  Add to 
that the fact that we're looking for full time administration; I think 
it's better to pay for an established professional, full time IT team 
that also manages the Linux kernel and is prominently involved in the 
FOSS ecosystem, than to try and build one out ourselves.

> That suggests that there might be more strings in the future.  Hypothetically,
> we might find ourself in the same position as the Python Software
> Foundation was last year, forced to choose between accepting conditions
> incompatible with the project's mission, and losing a big chunk of funding.
> Only worse, since in this scenario, "losing a big chunk of funding" might
> equate to "losing the use of the primary git server".

This is precisely why we're focused on making our setup fork-liftable.

Sid