from all aspects of AI, i find this the one that is easiest to ignore. AI might poison your reading, don't let it poison your writing. Why not continue doing your thing? This feeling of estrangement is not new in a world where every usefulness has to express itself as exchange value. Just have a service provider handle the additional load and don't stop being a thinking, writing, picture taking and social human being.
I wrote an algorithm to instantly solve Boggle, but it’s still a delightful game to play without technology. I don’t think that AI ruins the satisfaction of doing things yourself.
That doesn't ruin a good game. It might spoil a plot-based game. It doesn't ruin a game where you creatively interact with mechanisms and rules, because all a cheat does is push some of those mechanisms and rules aside, so that you skip interacting with the substantial part of the game.
Writing English in the AI era is like going for an hour-long walk in the park, when previously it took you 15 minutes via a crowded subway, where you were stabbed in the neck by a raging lunatic while a nearby passenger blasted Coldplay at 100 dB.
It’s like walking in the park when everyone else is zipping past you on scooters, and there’s nobody going your speed for you to talk with or even smile at.
I think "previously" meant "previously, using AI," not "previously to AI." The comment is confusing because "the AI era" seems like the antecedent of "previously" but it isn't.
Not necessarily. Having an “open web” doesn’t require you to be a network peer. You have to pay someone to bring the bandwidth to you. A CDN just moves that connection point a little. It actually allows you the freedom to run whatever server you want with much lower hardware and bandwidth requirements — even if 100% of your traffic were true humans.
That said, a CDN is only good for standard web traffic, not some custom game/chat/video service you might invent, but then again crawlers aren’t hitting those.
Oh? People keep parroting that but the small detail missing is that there needs to be enough volume of correct information to counter your adversarial disinfo. Otherwise the LLM is one agentic call away from stating it as fact.
Honestly, doing the same, I would love to know what the true count is.
I quickly looked over my access logs just now (eyeballs and grep, not goaccess) and see a couple of patterns...
One, there are indeed lots of bots - over a thousand hits from Claude. Some from search, some from users.
There are some bots I've never even heard of (mj12bot?).
There are some things which look like legitimate traffic coming from a common household Web browser, looking at blog posts and such, until suddenly they make a 404 request for a nonexistent API endpoint and go away.
And there are _so many_ requests for `.env` files, `swagger.json`, `backup.sql`, `phpinfo.php`, WordPress (always!)... Sorry folks, it's nginx in a container, serving a bunch of static content.
Unless you run full behaviour analysis, it's going to be hard to identify all bots from a normal access log. If you want more data, you can add JA4 hash to your logs that will surface huge amounts of things pretending to be the latest chrome.
At scale it’s vastly more resource intensive than scripts, so it does stop the very cheapest layer of spammers, but not anyone in any way resourced or determined.
I do realise that scrapers are also using residential IP ranges via SDK pitched to app developers that add backdoors to utilise the connection for "other things"
I share the same sentiment. It breaks my heart because I don't want the internet to go this way, but I fear there's no cure for this.
The only thing I am holding on to (like someone else said) is that they might include my writings in the training data, and bias towards my opinions ever so slightly as a result. :')
> moved to iocaine which is slightly different in what it does, and that was fine until so many scrapers hit my site that it saturated my upload with generated content feeding the them
You're feeding bots poison without rate limiting them all to some small fraction of your pipe?
iocaine has shipped firewall support to temporarily drop traffic from IPs after you've fed them an appropriate amount poison. More people should enable it.
You should not traffic shape the egress lest you run out of workers handling legitimate traffic. You should also make sure connections are appropriately terminated on your side for the same reason.
Don't give up. Blocking, redirecting and tarpitting bots can be turned into a fun game or puzzle of sorts. The bots evolve and so the game must evolve. I slow the residential bots down just a little and the obvious bots to somewhere between 32 and 96 bytes per second. I thought most would abort but many do not. They hang around for the slow dribble.
Here's just one example. It's kindof rude, especially for legit people behind corporate firewalls but such people should not be on my site anyway. Adjust as needed. It's just a silly example.
if ($server_protocol = HTTP/2.0) { set $limit_rate 384k; set $ok 1; }
if ($http3 != "") { set $ok 1; } # no rate limit for quic
if ($ok != 1) { set $limit_rate 32; limit_rate_after 128; set $trace "${trace}not-h2h3 "; add_header Content-Type text/html;return 200 '<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>body{width:35em;margin:0 auto;font-family:Tahoma,Verdana,Arial,sans-serif;}</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p></body></html>'; }
People not behind fascist firewalls and not on proxies will be able to use QUIC (via alt-svc header and dns) for full speed. Most real VPN's should be able to use QUIC.
This is just one silly example. One can use nginx geo in combination with FireHOL and other spammer IP repositories (consolidating with iprange) to also slow or tarpit or give dummy HTML and keep them on your edge servers and away from the origin or app servers or whatever.
That's a good one. Also in general, return 200 with invalid content to them or 404. Don't explicitly block with 403 or they will try different networks/clients/approaches to avoid that.
or they will try different networks/clients/approaches
Absolutely. Most of the bots are simple but a few of them were created and actively maintained by clever people that are here on HN. I like to think it's a playful arms race. They are the red team and I am the wrinkled blue team.
I'm glad this is fun for people, but I think I would rather play Monopoly with money under free parking instead.
I have noticed among my tech friends that one spectrum you can put us on is "sysadmin vs. not" - some people love tinkering with home labs, setting up systems, being able to reboot things from their watches, etc. And some people, like me, would strongly prefer not to be on-call for their hobby.
Hell, my personal server has been down for at least a week now because I started upgrading Ubuntu (that was very, very far behind), and one upgrade caused it to lose network connectivity, and I just haven't bothered to even start troubleshooting. If I had to guess, I'll probably be arsed to get it up sometime in December, though 2027 isn't out of the question.
Agreed it's not for everyone. As a side note, reboot your Ubuntu node and capture
uname -a;dmesg -T; lspci -vv; lsmod; ip link; systemctl; nmcli;
to a thumb drive and paste it into an LLM if you are not feeling like debugging such things. I used to call that lazy but I was corrected and now call it efficient.
serve mostly static pages cached by a CDN, and cache the odd dynamic ones you don't expect to be hit very often yourself at the server side, and you'll be fine
Picking a site in Cloudflare Bandwith Alliance, putting Cloudflare in front, and running your backend server on IPv6 only drops bandwidth charges to zero which tends to make bots ignorable.
I appreciate the decision to not run proof of work challenges. They punish all customers , and they especially punish outliers with excessive challenges : long challenge factors costing minutes, challenges that are presented many times a day, or ones where the challenge interferes with multi-screen oauth steps.
I haven’t seen an installation where the operator is tracking customer impact. They check the logs and say “see , we’ve screened 80% of the AI bots”. Meanwhile , among the 80% of bot traffic is a few percent of customers who are aggressively being punished without the operator being aware.
> I looked at Anubis and decided I didn't want the delay loading for regular viewers
There really isn’t that much of a delay, and once a visitor is confirmed, the rest of the site can still be zippy AF.
The downside with Anubis is that a properly-secured web browser will not be able to run its anti-AI code, and will fail. Not “you are a bot” fail, but “cannot run our checks” fail.
This is at a much more stringent level than CloudFlare’s maliciously invasive and user-hostile “protections”, so even fewer site visitors will be impacted, but can still be annoying to a privacy-oriented user.
I've thought of this and realized it's probably better for AI to be trained on my blog. I've very good opinions (obviously), and by letting AI bots scrape my writing I'm slightly biasing their training towards agreeing with me.
I'm struggling with the bots at repology.org too, but it instead makes me want do it even more.
For the fun of it, I'm developing my own anti-bot solution which acts as a proxy between nginx and the application, and is able to modify and override application responses (I anubis works the same way). What I've tried so far:
- Introducing a tarpit - infinite amount of cheap dynamically generated interlinked garbage pages, a hidden nofollow+noindex link to which is added to legitimate pages.
- Ability to introduce artificial response delays to tarpit or other pages.
- Ban IPs accessing a tarpit (and some other broken path patterns which browsers will never visit).
- Set a simple boolean cookie to know whether session has already visited the site.
- Set another boolean cookie for clients accessing a tarpit (in case bots learn to store cookies or reuse sessions; not tried to use it).
- For heaviest application endpoints (which need protection the most), require either "already visited" cookie, or show simple captcha.
- Also show captcha to banned IPs.
Each of these seems to help to some extent, and there's a lot of room for both experiments and for the case where bots adapt to some of these mechanisms.
Findings:
- Tarpit seems to divert a lot of traffic from legitimate pages.
- However, bots understood how well it performs and increased request rate from 40 to 160 rps.
- Introducing a delay helped with that. 9 sec delay has brought the rate down to 40 rps immediately. Longer delay makes bots timeout and drop the connection - not useful in my case as I want to feed more garbage links to them.
- It suggests the important idea of that the botnet has some centralized rate control mechanism which adapts to site performance, so you just need to make the site look slow and underperforming to bots.
- Most basic cookie based logic works great (for now). No need for cryptography or complex captchas, just set already_visited=1, or passed_captcha=1, where "captcha" is just an automated js which sets a cookie and reloads the page.
- IP bans work too, but simple `HashMap<IpAddr>` does not scale. Will try to come with more clever solution someday, probably involving a tree of IP address space regions, or a registry of nets from WHOIS, gathers statistics on per-network basis and acts based on that. For instance, nets with no legitimate requests over large period of time may be banned on firewall to reduce server load. Other nets with high rate of bot hits may be considered untrusted and always require captcha.
- Botnets ignore a `rel="nofollow noindex"`, so you can probably safely use these to guide the botnet to tarpit, while legitimate crawlers should not be affected.
Bot-resistant application design wisdom:
- Obviously, make it fast. You can no longer afford the application to perform slow (over 100ms) SQL queries. That's ok for humans, but bots will crush you. You can no longer afford slow backend language. I've rewritten repology to rust a couple of years ago, which resulted in around 10-100x performance and memory optimization. Had I not done it, the site would already be down by now.
- Don't ignore dynamic solutions. I myself am an oldschool who prefers to generate HTML on backend and avoid javascript. Guess it's time to reconsider. In my case, doing pagination in js would a) hide most pages from the botnet b) avoid generating infinite amount of distinct pagination pages, while these pages being heaviest on the site.
- Nginx before the application currently consumes more CPU than the application and anti-bot solution combined, so I'm considering retiring it.
Questions to community:
- I'm not satisfied with tarpit pages with just links to other tarpit pages. Any ideas of feeding the botnet with actual content to poison the dataset? I can come up with fake API reference pages with nonexistent methods.
The part about saturating a home upload link is what makes this different from ordinary bot noise. On a rented box a crawler wave is an annoyance; on a home connection it takes the whole household offline.
One middle ground between "home server takes everything" and "put a large CDN in front": a small rented VPS as a caching reverse proxy, with the home machine reachable only through a tunnel from it. I run a small site and a demo on a few-euro VPS, and at that price the bandwidth is no longer mine to worry about. The crawlers hit the cache, rate limiting happens there, and the home line only sees cache misses. It is still paying someone, but it is a commodity box you can move in an afternoon, with no dependency on one provider's edge.
Blocking by ASN instead of hand-picked /16s might also save you some evenings, since the cloud ranges keep changing.
Does the photography site need to be public at all for what you want from it, or would a password shared at events get you most of the way there without going fully to print?
thinking running a website never got you 99.9% of requests from bot is just delusional.
anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.
I got to watch traffic shift from cloud IPs to residential proxies in real time after I blocked the cloud IPs. It’s endless whack-a-mole.
What really pisses me off was Meta’s bots ignoring robots.txt when I tried to prevent high levels of traffic hitting meaningless URL variants on a Mediawiki site. Everybody else stopped hitting those URLs but not Meta which was flooding me with so much traffic it was impacting site performance. There source IPs and user agents are now banned.
Look at the US midterm results and how many elected candidates had AI regulation and data center moratoriums as part of their platform. The incoming Texas governor is saying she will halt data center approvals until new legislation is passed.
I find the defining of things that have happened more than once to be "cycles" in order to somehow rationalize closing off speculation about the future to be simple-minded.
Pretending that everything that has happened will always happen is just brain atrophy, which is why it's mistaken for maturity.
from all aspects of AI, i find this the one that is easiest to ignore. AI might poison your reading, don't let it poison your writing. Why not continue doing your thing? This feeling of estrangement is not new in a world where every usefulness has to express itself as exchange value. Just have a service provider handle the additional load and don't stop being a thinking, writing, picture taking and social human being.
> Why not continue doing your thing?
Writing software or english in the AI era is the same feeling as playing a video game without cheat codes you already beat using cheat codes.
The game is now ruined permanently.
I wrote an algorithm to instantly solve Boggle, but it’s still a delightful game to play without technology. I don’t think that AI ruins the satisfaction of doing things yourself.
That doesn't ruin a good game. It might spoil a plot-based game. It doesn't ruin a game where you creatively interact with mechanisms and rules, because all a cheat does is push some of those mechanisms and rules aside, so that you skip interacting with the substantial part of the game.
Writing English in the AI era is like going for an hour-long walk in the park, when previously it took you 15 minutes via a crowded subway, where you were stabbed in the neck by a raging lunatic while a nearby passenger blasted Coldplay at 100 dB.
It's more like replacing the park with plastic immitation trees.
No the park hasn’t changed.
It’s like walking in the park when everyone else is zipping past you on scooters, and there’s nobody going your speed for you to talk with or even smile at.
I think "previously" meant "previously, using AI," not "previously to AI." The comment is confusing because "the AI era" seems like the antecedent of "previously" but it isn't.
Maybe for you!
> Just have a service provider handle the additional load
That implies having a "service provider" in the first place, which is itself a bastardization of the open web.
Not necessarily. Having an “open web” doesn’t require you to be a network peer. You have to pay someone to bring the bandwidth to you. A CDN just moves that connection point a little. It actually allows you the freedom to run whatever server you want with much lower hardware and bandwidth requirements — even if 100% of your traffic were true humans.
That said, a CDN is only good for standard web traffic, not some custom game/chat/video service you might invent, but then again crawlers aren’t hitting those.
It seems to me like he’s going to keep doing his thing. Just not on the open web.
He doesn’t stop being all that. What he does is stop spending money so someone else can take his work and use it for its own commercial purposes.
Or poison your own writing for AI.
This is the future -- adversarial prose and adversarial source code. It is already working as seen with LLMs citing disinfo directly from Reddit.
No, it really isn't. If you think a 0.1% of writing being "poisoned" is going to trick AI, you're wildly underestimating AI's capabilities.
https://lyra.horse/blog/2026/11/disinformation-machine/
https://infosec.exchange/@rebane2001/117601277347377053
Research showed AI training could be poisoned by as few as five documents in a training set of millions.
Oh? People keep parroting that but the small detail missing is that there needs to be enough volume of correct information to counter your adversarial disinfo. Otherwise the LLM is one agentic call away from stating it as fact.
I also run a small side project site, and I get hundreds of thousands of bots visiting every week..
Honestly, doing the same, I would love to know what the true count is.
I quickly looked over my access logs just now (eyeballs and grep, not goaccess) and see a couple of patterns...
One, there are indeed lots of bots - over a thousand hits from Claude. Some from search, some from users.
There are some bots I've never even heard of (mj12bot?).
There are some things which look like legitimate traffic coming from a common household Web browser, looking at blog posts and such, until suddenly they make a 404 request for a nonexistent API endpoint and go away.
And there are _so many_ requests for `.env` files, `swagger.json`, `backup.sql`, `phpinfo.php`, WordPress (always!)... Sorry folks, it's nginx in a container, serving a bunch of static content.
Unless you run full behaviour analysis, it's going to be hard to identify all bots from a normal access log. If you want more data, you can add JA4 hash to your logs that will surface huge amounts of things pretending to be the latest chrome.
How long before bots just use the chrome crypto API to make connections?
At scale it’s vastly more resource intensive than scripts, so it does stop the very cheapest layer of spammers, but not anyone in any way resourced or determined.
They already can. But they largely don't. We still get lots of really dumb requests that don't do basic hiding.
> There are some bots I've never even heard of (mj12bot?).
That's for a search engine that has been around before the AIpocalypse.
I like that term but I also like “AI winter” like we used to talk about a nuclear winter.
Because the world isn’t so much violently destroyed as it is… quietly cold, barren and lifeless.
What are they doing?
I do realise that scrapers are also using residential IP ranges via SDK pitched to app developers that add backdoors to utilise the connection for "other things"
Can anyone corroborate this?
The residential proxy SDKs for Smart TVs have been discussed at length at least. [1]
[1]: https://news.ycombinator.com/item?id=48635954
Thanks for the link. That’s atrocious
https://nickk.info/2023/11/16/api-limits.html
It works too well.
I share the same sentiment. It breaks my heart because I don't want the internet to go this way, but I fear there's no cure for this.
The only thing I am holding on to (like someone else said) is that they might include my writings in the training data, and bias towards my opinions ever so slightly as a result. :')
> moved to iocaine which is slightly different in what it does, and that was fine until so many scrapers hit my site that it saturated my upload with generated content feeding the them
You're feeding bots poison without rate limiting them all to some small fraction of your pipe?
iocaine has shipped firewall support to temporarily drop traffic from IPs after you've fed them an appropriate amount poison. More people should enable it.
You should not traffic shape the egress lest you run out of workers handling legitimate traffic. You should also make sure connections are appropriately terminated on your side for the same reason.
Don't give up. Blocking, redirecting and tarpitting bots can be turned into a fun game or puzzle of sorts. The bots evolve and so the game must evolve. I slow the residential bots down just a little and the obvious bots to somewhere between 32 and 96 bytes per second. I thought most would abort but many do not. They hang around for the slow dribble.
Here's just one example. It's kindof rude, especially for legit people behind corporate firewalls but such people should not be on my site anyway. Adjust as needed. It's just a silly example.
People not behind fascist firewalls and not on proxies will be able to use QUIC (via alt-svc header and dns) for full speed. Most real VPN's should be able to use QUIC.This is just one silly example. One can use nginx geo in combination with FireHOL and other spammer IP repositories (consolidating with iprange) to also slow or tarpit or give dummy HTML and keep them on your edge servers and away from the origin or app servers or whatever.
That's a good one. Also in general, return 200 with invalid content to them or 404. Don't explicitly block with 403 or they will try different networks/clients/approaches to avoid that.
or they will try different networks/clients/approaches
Absolutely. Most of the bots are simple but a few of them were created and actively maintained by clever people that are here on HN. I like to think it's a playful arms race. They are the red team and I am the wrinkled blue team.
I'm glad this is fun for people, but I think I would rather play Monopoly with money under free parking instead.
I have noticed among my tech friends that one spectrum you can put us on is "sysadmin vs. not" - some people love tinkering with home labs, setting up systems, being able to reboot things from their watches, etc. And some people, like me, would strongly prefer not to be on-call for their hobby.
Hell, my personal server has been down for at least a week now because I started upgrading Ubuntu (that was very, very far behind), and one upgrade caused it to lose network connectivity, and I just haven't bothered to even start troubleshooting. If I had to guess, I'll probably be arsed to get it up sometime in December, though 2027 isn't out of the question.
Agreed it's not for everyone. As a side note, reboot your Ubuntu node and capture
serve mostly static pages cached by a CDN, and cache the odd dynamic ones you don't expect to be hit very often yourself at the server side, and you'll be fine
Picking a site in Cloudflare Bandwith Alliance, putting Cloudflare in front, and running your backend server on IPv6 only drops bandwidth charges to zero which tends to make bots ignorable.
I feel the same way, I've landed on astro and vanilla css as a safe haven for now. No more frameworks, no more bloat. Maybe htmx for some stuff.
I'm not quite sure what the author wants to tell me.
I think it might be lacking context being posted here and emotionally resonates more with the actual intended viewership?
I appreciate the decision to not run proof of work challenges. They punish all customers , and they especially punish outliers with excessive challenges : long challenge factors costing minutes, challenges that are presented many times a day, or ones where the challenge interferes with multi-screen oauth steps.
I haven’t seen an installation where the operator is tracking customer impact. They check the logs and say “see , we’ve screened 80% of the AI bots”. Meanwhile , among the 80% of bot traffic is a few percent of customers who are aggressively being punished without the operator being aware.
> I looked at Anubis and decided I didn't want the delay loading for regular viewers
There really isn’t that much of a delay, and once a visitor is confirmed, the rest of the site can still be zippy AF.
The downside with Anubis is that a properly-secured web browser will not be able to run its anti-AI code, and will fail. Not “you are a bot” fail, but “cannot run our checks” fail.
This is at a much more stringent level than CloudFlare’s maliciously invasive and user-hostile “protections”, so even fewer site visitors will be impacted, but can still be annoying to a privacy-oriented user.
"Running web sites makes me not want to" is one of the greatest, perhaps THE greatest, tech headline of the year.
Gosh, that's a supremely weird font!
The font is bearable but the neon flow effect behind it in a dark background makes it unbearable to read.
Reading web sites in this font makes me not want to.
I've thought of this and realized it's probably better for AI to be trained on my blog. I've very good opinions (obviously), and by letting AI bots scrape my writing I'm slightly biasing their training towards agreeing with me.
Nice, working for the LLM corps for free! I see no issues with that whatsoever.
I'm struggling with the bots at repology.org too, but it instead makes me want do it even more.
For the fun of it, I'm developing my own anti-bot solution which acts as a proxy between nginx and the application, and is able to modify and override application responses (I anubis works the same way). What I've tried so far:
Each of these seems to help to some extent, and there's a lot of room for both experiments and for the case where bots adapt to some of these mechanisms.Findings:
Bot-resistant application design wisdom: Questions to community:Maybe I'm just old but the font on this site is unreadable for me. But reader mode saves me. At least the clankers can read it...
The e’s in particular are driving me crazy. I found the content interesting but I closed it because of the font.
The part about saturating a home upload link is what makes this different from ordinary bot noise. On a rented box a crawler wave is an annoyance; on a home connection it takes the whole household offline.
One middle ground between "home server takes everything" and "put a large CDN in front": a small rented VPS as a caching reverse proxy, with the home machine reachable only through a tunnel from it. I run a small site and a demo on a few-euro VPS, and at that price the bandwidth is no longer mine to worry about. The crawlers hit the cache, rate limiting happens there, and the home line only sees cache misses. It is still paying someone, but it is a commodity box you can move in an afternoon, with no dependency on one provider's edge.
Blocking by ASN instead of hand-picked /16s might also save you some evenings, since the cloud ranges keep changing.
Does the photography site need to be public at all for what you want from it, or would a password shared at events get you most of the way there without going fully to print?
[flagged]
[dead]
thinking running a website never got you 99.9% of requests from bot is just delusional.
anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.
I got to watch traffic shift from cloud IPs to residential proxies in real time after I blocked the cloud IPs. It’s endless whack-a-mole.
What really pisses me off was Meta’s bots ignoring robots.txt when I tried to prevent high levels of traffic hitting meaningless URL variants on a Mediawiki site. Everybody else stopped hitting those URLs but not Meta which was flooding me with so much traffic it was impacting site performance. There source IPs and user agents are now banned.
Do not explicitly block that traffic. Return 200 with fake content or 404 and they won't try to workaround it.
[flagged]
Some of us deployed websites before bots. Heck once upon a time you had to ask nicely for them to even come visit.
[flagged]
I didn't find this comment particularly insightful, novel, or interesting.
Look at the US midterm results and how many elected candidates had AI regulation and data center moratoriums as part of their platform. The incoming Texas governor is saying she will halt data center approvals until new legislation is passed.
I think you'll find that paid astroturf and ideolog touts collectively make way more noise than their detractors.
I assure you, the corporate tech and AI simps are incredibly vocal here as well. And have you seen LinkedIn lately?
I find the defining of things that have happened more than once to be "cycles" in order to somehow rationalize closing off speculation about the future to be simple-minded.
Pretending that everything that has happened will always happen is just brain atrophy, which is why it's mistaken for maturity.
"The DDoS will continue until morale improves."