scraperbot protection - Patchwork and Bunsen behind Anubis

[Guinevere Larsen]

On 4/21/25 12:59 PM, Mark Wielaard wrote:
> Hi hackers,
>
> TLDR; When using https://patchwork.sourceware.org or Bunsen
> https://builder.sourceware.org/testruns/ you might now have to enable
> javascript. This should not impact any scripts, just browsers (or bots
> pretending to be browsers). If it does cause trouble, please let us
> know. If this works out we might also "protect" bugzilla, gitweb,
> cgit, and the wikis this way.
>
> We don't like to hav to do this, but as some of you might have noticed
> Sourceware has been fighting the new AI scraperbots since start of the
> year. We are not alone in this.
>
> https://lwn.net/Articles/1008897/
> https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
>
> We have tried to isolate services more and block various ip-blocks
> that were abusing the servers. But that has helped only so much.
> Unfortunately the scraper bots are using lots of ip addresses
> (probably by installing "free" VPN services that use normal user
> connections as exit point) and pretending to be common
> browsers/agents.  We seem to have to make access to some services
> depend on solving a javascript challenge.

Jan Wildeboer, on the fediverse, has a pretty interesting lead on how AI 
scrapers might be doing this: 
https://social.wildeboer.net/@jwildeboer/114360486804175788 (this is the 
last post in the thread because it was hard to actually follow the 
thread given the number of replies, please go all the way up and read 
all 8 posts).

Essentially, there's a library developer that pays developers to just 
"include this library and a few more lines in your TOS". This library 
then allows the app to sell the end-user's bandwidth to clients of the 
library developer, allowing them to make requests. This is how big 
companies are managing to have so many IP addresses, so many of those 
being residential IP addresses, and it also means that by blocking those 
IP addresses we will be - necessarily - blocking real user traffic to 
our platforms.

I'm happy to see that the sourceware is moving to a more comprehensive 
solution, and if this is successful, I'd suggest that we also try to do 
that to the forgejo instance, and remove the IPs blocked because of this 
scraping.

>
> So we have installed Anubis https://anubis.techaro.lol/ in front of
> patchwork and bunsen. This means that if you are using a browser that
> identifies as Mozilla or Opera in their User-Agent you will get a
> brief page showing the happy anime girl that requires javascript to
> solve a challenge and get a cookie to get through. Scripts and search
> engines should get through without. Also removing Mozilla and/or Opera
> from your User-Agent will get you through without javascript.
>
> We want to thanks Xe Iaso who has helped us set this up and worked
> with use over the Easter weekend solving some of our problems/typos.
> Please check out if you want to be one of their patrons as thank you.
> https://xeiaso.net/notes/2025/anubis-works/
> https://xeiaso.net/patrons/
>
> Cheers,
>
> Mark
>

-- 
Cheers,
Guinevere Larsen
She/Her/Hers