robots.txt does not block bots, it is just a suggestion. There are no legal teeth behind robots.txt and AI training bots can and do just ignore it. There are a myriad of methods to actually block bots, here is just one of them (and it will block Google's crawler as their http library is out of date). I use the ban variable to put them in a different access log and dump their IP into NFTables ipsets. This is mostly for small sites that need not be crawled or archives that get abused by poorly written or rude bots.
# nginx rule to block some crawlers/bots, permit HTTP/2.0 and HTTP/3.0, block all other protocols.
if ($server_protocol = HTTP/2.0) { set $ok 1; }
if ($http3 != "") { set $ok 1; }
if ($ok != 1) { set $ban 1; return 444 ''; }
robots.txt does not block bots, it is just a suggestion. There are no legal teeth behind robots.txt and AI training bots can and do just ignore it. There are a myriad of methods to actually block bots, here is just one of them (and it will block Google's crawler as their http library is out of date). I use the ban variable to put them in a different access log and dump their IP into NFTables ipsets. This is mostly for small sites that need not be crawled or archives that get abused by poorly written or rude bots.