Meeting Minutes - Office Hours for CTI - 2026-08-28

[Mark Wielaard]

Hi Zack,

On Thu, 2026-09-10 at 18:47 -0400, Zack Weinberg wrote:
> On Thu, Sep 10, 2026, at 6:02 PM, Siddhesh Poyarekar wrote:
> > Does blocking https for git clones count as a critical service failure? 
> > This had happened some months ago in an attempt (IIRC) to counter AI 
> > scraper bots.  The change was not reverted for a while, arguing that 
> > git:// was good enough but eventually it was, I don't remember the final 
> > justification for it all.
> 
> Oh, for sure.  Loss of anonymous https-based access to the main git server
> means anyone with no SSH account, stuck behind a restrictive outbound firewall,
> can't clone or pull, so that's going to be a big deal for a lot of people.
> And the logic of the change doesn't hold up; I've heard any number of
> reports that the bots don't even *try* to clone, they just beat on gitweb.

As Frank already pointed out, that is sadly not true. While most of the
pain of the AI scraperbots comes from the botnet being plain dumb with
hundreds of hits a second on bugzilla, cgit, gitweb, moinmoin, etc
through https. We are sadly also seeing git clone attacks through the
git protocol over https. And even on other protocols/services, like
rsync or the imap interface we run on inbox.sourceware.org for people
to read the mailinglists directly in their mua.

> So now we're actually getting somewhere maybe.  This is a concrete example
> of a critical service failure on sourceware that would have been prevented
> by more disciplined change management procedures.
> [...]
> Mark: Same questions for the overseers; additionally, if you know
> anything about this specific incident, can you explain what happened
> from your perspective?

I am not sure which incident we are specifically talking about. I am
aware of 3 in the last year, only one of which might be related to
"change management". All our changes of course go through git and can
easily be reverted if they turn out to be bad. Which is exactly what
happened with the incident of a nginx config change clashing with a
httpd security update (I think it was resolved in about an hour after
it happened). It won't happen again since now we know how/where to put
the config where it sticks.
https://inbox.sourceware.org/gcc/20260908222609.GB23490@gnu.wildebeest.org

The other two incidents were git clones over https were temporarily
were simple operational issues when the AI scraperbot attacks became so
bad that we needed to add more mitigations. One happened soon after
some of our servers moved to a new datacenter. It looked like the AI
scraperbits smelled there was more bandwidth to be used and increased
their hit frequency a hundred fold both on gitweb/cgit and doing https
git clones. There was some scramble to deal with that. Especially
because this was during the end of year break (December 29). But
service was restored within 24 hours and a full fix was available Jan
4th.
https://inbox.sourceware.org/overseers/20251229200052.GC6902@redhat.com/


A similar issue, occurred last month. Which was when we put the nginx
limiter in front of the httpd fleet. Again it took a bit to get the
config completely right (one issue was that we had configured gitweb
and git clone on the same pool because both started with
https://sourceware.org/git meaning that an attack on gitweb also
limited git clones over https). This did indeed take a couple of days
to figure out. We now also have pools for ips that should never be
limited that you can get into by loggin into bugzilla for example, are
automatically created for CI buildbots, or can be added manually.
https://inbox.sourceware.org/overseers/0475bd28feb9f78dcea1c13d6315aac79966ca53.camel@klomp.org

Cheers,

Mark