Bots again..

[Dave]

It seems that recently there has been a resurgence of Facebook bots, as well as others, on the sites. I thought I'd ask Google AI how to solve it and what it came up with was the code below.

Before I added the code there were 263 bots looking at my site, now I know that's not many compared to this site, at the point of writing this post there were 3546 visitors, but what they were doing is causing extra load on the server and giving some members and I '504 bad gateway' errors.

After adding the code to the .htaccess file the numbers gradually dropped significantly.

There was another option for the code which was use the mod_rewrite option but if the host does not have that enabled then the code will not work but using the 'If' option will.

Code Select Expand

<If "%{HTTP_USER_AGENT} =~ /(FacebookBot|meta-externalagent|Meta-ExternalFetcher|facebookexternalhit|Amazonbot|Bingbot|AdIdxBot|BingPreview|MicrosoftPreview|ClaudeBot|GPTBot|ChatGPT-User)/i">
    Require all denied
</If>
I have posted this on SMFH also

[Chen Zhen]

Were the bad gateway errors happening on DJ or Quiz?

[Dave]

They were happening on Quiz. ShawnB replied to the post I created in SMFH and suggested that I disable hostname look ups in Admin and that I remove the 'facebookexternalhit' and 'Bing' which I have.

Using the code I posted has reduced the bots tremendously though, maybe you should try it here, as I write this you have 2700+ bots and looking at the statistics 'Most Online Ever: 86,212 (30 July 2026, 11:05:19)' so you probably could do with something to cut them down

[Chen Zhen]

I have applied some mods/tactics to cut down on all these bots and spammers and I will try what you posted or perhaps something similar.

[Dave]

This is the code I'm currently using in my .htaccess, note that all other code relating to remove etc has been deleted from the file.

I added the Alibaba and Wowrack myself, not sure if they will work or not.

Code Select Expand

<If "%{HTTP_USER_AGENT} =~ /(FacebookBot|meta-externalagent|Meta-External|Amazonbot|AdIdxBot|Alibaba|Wowrack|BingPreview|MicrosoftPreview|ClaudeBot|GPTBot|ChatGPT-User)/i">
    Require all denied
</If>

[Chen Zhen]

Here is some updated code which is recommended for Apache 2.4+
It requires the setenvif module (common environment variable module preloaded with Apache2.4).

Code Select Expand

<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent (FacebookBot|meta-externalagent|Meta-External|Amazonbot|AdIdxBot|Alibaba|Wowrack|BingPreview|MicrosoftPreview|ClaudeBot|GPTBot|ChatGPT-User) bad_ua
    <RequireAll>
        Require all granted
        Require not env bad_ua
    </RequireAll>
</IfModule>


[Skhilled]

I might try it myself...after this very long cPanel update finishes. LOL

EDIT:  I've had a big issue with it updating as expected. But... Now it is updating as expected, although it's slow as expected because of the large update...