Bots again..

[Dave]

It seems that recently there has been a resurgence of Facebook bots, as well as others, on the sites. I thought I'd ask Google AI how to solve it and what it came up with was the code below.

Before I added the code there were 263 bots looking at my site, now I know that's not many compared to this site, at the point of writing this post there were 3546 visitors, but what they were doing is causing extra load on the server and giving some members and I '504 bad gateway' errors.

After adding the code to the .htaccess file the numbers gradually dropped significantly.

There was another option for the code which was use the mod_rewrite option but if the host does not have that enabled then the code will not work but using the 'If' option will.

Code Select Expand

<If "%{HTTP_USER_AGENT} =~ /(FacebookBot|meta-externalagent|Meta-ExternalFetcher|facebookexternalhit|Amazonbot|Bingbot|AdIdxBot|BingPreview|MicrosoftPreview|ClaudeBot|GPTBot|ChatGPT-User)/i">
    Require all denied
</If>
I have posted this on SMFH also

[Chen Zhen]

Were the bad gateway errors happening on DJ or Quiz?

[Dave]

They were happening on Quiz. ShawnB replied to the post I created in SMFH and suggested that I disable hostname look ups in Admin and that I remove the 'facebookexternalhit' and 'Bing' which I have.

Using the code I posted has reduced the bots tremendously though, maybe you should try it here, as I write this you have 2700+ bots and looking at the statistics 'Most Online Ever: 86,212 (30 July 2026, 11:05:19)' so you probably could do with something to cut them down

[Chen Zhen]

I have applied some mods/tactics to cut down on all these bots and spammers and I will try what you posted or perhaps something similar.

[Dave]

This is the code I'm currently using in my .htaccess, note that all other code relating to remove etc has been deleted from the file.

I added the Alibaba and Wowrack myself, not sure if they will work or not.

Code Select Expand

<If "%{HTTP_USER_AGENT} =~ /(FacebookBot|meta-externalagent|Meta-External|Amazonbot|AdIdxBot|Alibaba|Wowrack|BingPreview|MicrosoftPreview|ClaudeBot|GPTBot|ChatGPT-User)/i">
    Require all denied
</If>

[Chen Zhen]

Here is some updated code which is recommended for Apache 2.4+
It requires the setenvif module (common environment variable module preloaded with Apache2.4).

Code Select Expand

<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent (FacebookBot|meta-externalagent|Meta-External|Amazonbot|AdIdxBot|Alibaba|Wowrack|BingPreview|MicrosoftPreview|ClaudeBot|GPTBot|ChatGPT-User) bad_ua
    <RequireAll>
        Require all granted
        Require not env bad_ua
    </RequireAll>
</IfModule>


[Skhilled]

I might try it myself...after this very long cPanel update finishes. LOL

EDIT:  I've had a big issue with it updating as expected. But... Now it is updating as expected, although it's slow as expected because of the large update...

[Dave]

Here is some updated code which is recommended for Apache 2.4+
It requires the setenvif module (common environment variable module preloaded with Apache2.4).

Code Select Expand

<IfModule mod_setenvif.c>
    SetEnvIfNoCase User-Agent (FacebookBot|meta-externalagent|Meta-External|Amazonbot|AdIdxBot|Alibaba|Wowrack|BingPreview|MicrosoftPreview|ClaudeBot|GPTBot|ChatGPT-User) bad_ua
    <RequireAll>
        Require all granted
        Require not env bad_ua
    </RequireAll>
</IfModule>


Thanks Chen, unfortunately using that on Quiz breaks it with 'Internal Server Error'. It's seems to be OK on DJ my guess is it's something to do with the hosts config on Quiz.

I have removed it from DJ

I see this morning you have more than 4500 guests

[Chen Zhen]

Are you using Apache 2.4 on Quiz?

[Dave]

PHPinfo tells me it's Apache 2.0

I cannot find anything related to it the the hosts panel, it's their own

They don't make it easy to know if it's 2.2 or 2.4 but I did find reference to 2.4 in documentation.

They don't make it easy to ask the question either

[Chen Zhen]

There must be something in the file or my code that is somehow causing it.
I have tried with quotes instead of brackets where both seem to work on this webserver.

I had the same error when attempting to add more bots & it's likely something in the code that it didn't like.
Also I have to remember to always test it on my localhost first so I don't make my site go down temporarily.   

I have a huge list now for multiple user-agents/bots and IP's.
Almost all the remaining ones seem to be in my Arcade.
 

[Chen Zhen]

I've been reading up on this & plan to make further changes. Some bots that you've suggested for blocking are not necessarily bad bots & may be controlled by robots.txt, llms.txt & llms-full.txt.

Known bad bots should be blocked of course but a current updated list is needed.

[Dave]

Have a look here https://www.smfhelper.com/index.php?topic=1765.msg13785#new and then also look at Shawn's links he's added in his reply to me, it might help your search for info

[Dave]

I've now spoken to the customer services on the phone and they confirm it's Apache 2.4. But as you know it also depends on what modules they have enabled. You will know a lot more than me but look at this page, it might help

https://httpd.apache.org/docs/trunk/new_features_2_4.html