> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
That's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.
Hah. I have a homelab with a couple of sites, including a personal Forgejo installation.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.
I have a medium-sized Discord server of web sysadmin people (mostly wiki operators) that came together after I wrote a similar blog post [1] about how the LLM scrapers/resproxies are making it suck to run wikis. Not sure if there's other private communities out there, but feel free to email me if you want to join
Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage.
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
That's missing the title and URL, as I suspected it would, though the submitter UID is available.
To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.
> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware.
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.
And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.
The article was called "The one we're commenting on"
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.
Does that sound like your typical self-hosted blog?
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
ive seen unscrupulous founders fake traction during diligence, which is my day job
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
It's not a sign about that project in any way. I had never heard of it and have no opinion about it one way or the other.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue.
The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.
For what it's worth I'm working on hashx support. It's just going to take a bit to ship while I do browser testing with broken browser configs.
Interesting. How do they tell the difference between legitimate and forged ip owner records?
At least anubis works for me. (I run umatrix)
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
> Unfortunately whatever HN is using routinely blocks my login with "Sorry."
I believe that's the HN application itself, not a WAF in front of it.
HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented.
Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
Or you can go full Reddit and just block anything that seems even remotely suspicious.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
Even worse, not blocked, shadowbanned.
> Your screen brightness too low? Believe or not, blocked.
... What?!
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
> a badly configured scraping setup?
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
I kind of wish the recent Google monopoly court ruling had forced Google to open up their index to anyone, not just Perplexity/other big players.
That's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.
Maybe someone is paid per scrape, without reduction in payment for duplicates.
Maybe every web query for Linux commands in $LARGE_COUNTRY checks all the Linux websites again.
Maybe they are aggressively scanning for updates on the page
Hah. I have a homelab with a couple of sites, including a personal Forgejo installation.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.
Thanks for making common crawl as good as it is. It’s a really important part of making the Internet better
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.
Feels like it would be a good time for freenet and the like to catch on.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
I have essentially this at work, but the scrapers ignore it. (Or at least many, many scrapers ignore it.)
that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
Yeah. Though I guess the point I thought of was like a deals site. That would have infinite pages and content
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
From the article:
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
[0]: https://github.com/jow-/nlbwmon
[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...
[2]: https://docs.mossdef.org/adblock-fast/
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.
If you scrape 6TB from across the web or grab it from one place, it's still 6TB
One article mentioned in the OP was discussed here:
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?
It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.
As someone dealing with similar on a large site, I'd love to see a private community to discuss some of these issues.
I have a medium-sized Discord server of web sysadmin people (mostly wiki operators) that came together after I wrote a similar blog post [1] about how the LLM scrapers/resproxies are making it suck to run wikis. Not sure if there's other private communities out there, but feel free to email me if you want to join
[1] - https://weirdgloop.org/blog/clankers
Ive been seeing a 'sorry' message occasionally when accessing older pages.
Is that a side effect of whatever you are doing?
It was, but there were too many legit users getting affected, so we turned most of that off a few days ago. Are you still seeing it?
Not in last day no.
Ok that's good - if you get it again can you let us know at hn@ycombinator.com? We definitely don't want to exclude legit users.
Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage.
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
[1] - https://news.ycombinator.com/item?id=48416693
And the backup is field-tested to fall over within a few hours of the primary ;-)
<https://news.ycombinator.com/item?id=32048148>
<https://news.ycombinator.com/item?id=32031243>
(AFAIK that specific failure mode has in fact been addressed.)
For one datapoint ...
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
<https://news.ycombinator.com/item?id=36078578>
<https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
HN is exported to firebase, which you can hit directly, for that sort of purpose
https://github.com/HackerNews/API
I know that.
I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.
API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.
That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
I feel less bad hammering firebase in a "while True:" loop vs hitting HN's servers.
12,000 times less bad? <https://news.ycombinator.com/item?id=48868910>
Looking at the API ...
... it's starting to make sense, but ...
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
<https://news.ycombinator.com/front?day=2008-11-05>
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
"A Year on Hacker News" <https://news.ycombinator.com/item?id=31295219>
"Top Hacker News commenters of 2021" <https://news.ycombinator.com/item?id=29778994>
"What Happened This Year on Hacker News (2021)" <https://news.ycombinator.com/item?id=29769470>
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
OK, looking at a recent dead atomic128 comment:
So userID is visible.And from a current dead submission in the New queue:
That's missing the title and URL, as I suspected it would, though the submitter UID is available.To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...
I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
As someone who operates a wiki, this does not solve the problem.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.
At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.
> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
I think they also have to operate in countries that don't mind shady things like this.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
TIL:
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
WTF. That's just botnets.
Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware.
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.
by consent they mean a dialog/EULA with careful wording was display and the user clicked ok.
And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
Thank god for residential proxies.
Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet
By providing a way for corporate AI scrapers to operate with impunity and force the last few independently-run websites to move to the cloud?
No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.
The article was called "The one we're commenting on"
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.
Does that sound like your typical self-hosted blog?
i know a few very large startups that used it to fake their way into an exit
unethical yes but really raises the question as to what we see is real or not
Money is real. DAU that don't pay subscriptions, or don't lead to paid conversions on hosted ads, are worthless.
"Raises the question of what we see is real"
No they really don't, dishonest founders do that.
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
ive seen unscrupulous founders fake traction during diligence, which is my day job
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.
Google itself is a huge database.Who makes these rules depends on who's leading the market.
https://archive.fo/PAcF5
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Has no one noticed their miniflux instance failing to fetch feeds because of this?
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
Is reasonable! Many others are not
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
If it weren't a real problem, these types of articles and services wouldn't exist.
Why the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?
"He who has the gold makes the rules" is older than the pyramids.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
Common Crawl's dataset was downloaded in full 100 times in 2025.
We agree that it would be great if it was even more widely used.
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
Common Crawl's archive has metadata that says when each record (html file) was crawled.
But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.
> widespread scraping of web sites in search of training data for large language models and related projects
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
https://news.ycombinator.com/newsguidelines.html
Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.
Seriously, dang?
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.
I made it all the way back to https://news.ycombinator.com/posts?id=atomic128&next=4628060... (6 months ago) before seeing posts about anything else, only to find that there was a different agenda before that. Not cool.
Edit: and before all that, there was this: https://news.ycombinator.com/posts?id=atomic128&next=4164795.... This is obviously not using HN as intended.
Exceptions?
<https://news.ycombinator.com/item?id=47116093> (LLM but not PF).
<https://news.ycombinator.com/item?id=47095664> (a16h)
<https://news.ycombinator.com/item?id=46695693> (vuln exploits) 2026-1-20
<https://news.ycombinator.com/item?id=46280602> (???, but not PF)
<https://news.ycombinator.com/item?id=46195234> (Monero / Dark Web) 2025-12-8
<https://news.ycombinator.com/item?id=45894305> and <https://news.ycombinator.com/item?id=45826273> (Tor hidden service) Nov 2025
That's from the past 20 comments.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
> Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I took a look at the most recent comments from both accounts and they don't look similar to me in this respect.
I think there are two questions here though:
1. Was the violation egregious?
2. Did it deserve an immediate ban, or did they deserve a warning etc.?
Seems to me the answer to (1) is yes, but the answer to (2) I'm less sure about.
[delayed]
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
I'm pretty confident it's not that.
HN's prime directive is "anything that gratifies one's intellectual curiosity": <https://news.ycombinator.com/newsguidelines.html> and many, many, many dang comments.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Occasional alts are explicitly permitted, though not to engage in abuse (e.g., mutual admiration societies, sock-puppetry). See: <https://news.ycombinator.com/item?id=9963551> <https://news.ycombinator.com/item?id=9823379> (both against sock puppetry) and <https://news.ycombinator.com/item?id=9122086> and <https://news.ycombinator.com/item?id=7504621> (on where throwaways are/aren't permitted).
Where HN does favour persistent accounts the stated claim is to foster community, rather than for nefarious tracking purposes: <https://news.ycombinator.com/item?id=18082346> and <https://news.ycombinator.com/newsguidelines.html>. From that last:
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
Just curious dang, did you warn them before banning?
Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.
No, but if they have a change of heart and genuinely want to use HN as intended, they're welcome to let us know. I've added that in an edit now.
This is a strong positive sign that poison fountain works.
I wasn't aware of this project. Thanks for the heads up.
It's not a sign about that project in any way. I had never heard of it and have no opinion about it one way or the other.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
Yeah. A fun thing to do is to try and actually read common crawl!
Really makes you think, what we're feeding them...
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
Again, why do we allow China on the Internet?
Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.
>types of operator running residential-proxy networks to attack web sites.
This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product.
>paid for hijacking their users' network connections
Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors.
>Recently, LWN was subjected what was, by far, the heaviest scraper attack yet.
LWN is a static site. To me it seems more expensive to use Anubis than just serve the actual page.
>will now check for NetNut-infected apps
Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).