Nerd-talk warning...
I'd been using Apache's mod_evasive module to try and trap the bots, but that didn't distinguish between a page load and an image load within a page. People reading a screenshot thread or doing a file search with lots of thumbnails to display would hit the limits in place to catch the bots largely because of the image loading.
As of about 24 hours ago, I've moved to a different tool called mod_qos (quality of service), which is able to focus only on base page reuests, and can exclude images, stylesheets, scripts, etc. from that equation. The result seems to be working as I'm not hearing of any new reports since that was turned on and evasive was turned off.
What all that means in simple terms....
I've changed the screening method a bit to help balance out the needs for regular users and throw some harder limits at the bad actors.
- Global Cap: the server is currently throttled to 200 simultaneous open user connections. Visitor 201 may have to wait until one of the existing threads is done.
- Per-IP Limits: To keep single users (or an aggressive bot / scraper) from eating up all the available bandwidth, each IP address will be capped to 15 simultaneous connections -- each page load uses about 4-8 connections over a second or three until a page is fully loaded. If you do manage to blow thru that cap... you might wind up in server jail for up to 60 seconds.
- Page Limits: Every IP will be limited to around 10 page requests per minute. If you blow thru that cap... you might wind up in server jail for up to 60 seconds.
For 99% of you just browsing or downloading files, you’ll probably never notice these guardrails.
For the automated scrapers and bad-actors.... they're going to get slowed down, and hopefully give up because us hitting the brakes every 60 seconds is going to tie up -their- resources.
For the really bad actors... those firing off random page addresses hoping to find a hole... I did enable a separate module called Fail2Ban, which focuses on various failed transaction attempts. As of a few minutes ago, that had banned a dozen or so IP's for a much longer timeframe... 24 hours.
If you do find that you ran into the 24 hour ban... please send an email to ipban@trainsim.com with your IP address and I'll make sure it's unblocked.
Hopefully this puts up enough of a defense that we don't see a recurrence anytime soon.
Aloha, Jeff Link
Yeah, Fail2Ban can kick some butt, even if it is a bit of a cruder instrument. But this is war, and thus the enemy better expect to be clobbered. What I often see these days are attempts from IPV4 A.B.C.0/24 at random but continuous. I concur with the trouble and effort, but it seems to work!
Steve