Using 444

[Jeffrey Walton]

On Sat, Sep 27, 2025 at 2:28 PM Paul <paul at stormy.ca> wrote:
>
> [...]
> Maxim, many thanks.  Currently battling a DDoS including out of control
> "AI". Front end nginx/1.18.0 (Ubuntu) easily handles volume (CPU usage
> rarely above 1%) but proxied apache2 often runs up to 98% across 12
> cores (complex cgi needs 20-40 ms per response.)
>
> I'm attempting to mitigate.  Your advice appreciated. I've "snipped"
> below for readability:

My apologies if this wanders too off-topic.

A lot of folks are having trouble due to AI Agents scraping their
sites for training data.  It hit the folks at GNU particularly hard.
If AI is so smart, then why does it not clone a project instead of
scraping source code presented as web pages???

You might consider putting a box on the front-end to handle the abuse
from AI agents.  Anibus, go-away and several others are popular.
go-away provides a list of similar projects at
<https://git.gammaspectra.live/git/go-away#other-similar-projects>.
In fact, go-away names Nginx's ngx_http_js_challenge_module as a
mitigation for the problem.

Jeff