Distillation attacks are covert, industrial-scale campaigns to extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers.
Replace a few key words and it sounds like reading all the worlds media and offering a service that summarises it.
Don't scrape the scrapers, that's not fair! Expecting Executive Order against 'distillation' in 3, 2, 1...
Isn't this a little reductive? I'm not an expert on LLM/AI technology, but isn't the secret sauce the reasoning layer? Isn't that what makes Claude et al so much more powerful than a more "vanilla" LLM? That's what they're trying to protect in this post, which seems awfully reasonable?
The laws they are arguing for are not going to protect anything, they essentially are waging war on open models and since most open models that are 'pretty good' to 'good enough' hail from China they have chosen to play the China-fear card.
But they won't be able to stop anybody from doing this if they want to continue to operate Claude as a business going forward, and that is assuming that is what is actually happening. Chinese labs have been extremely open about their developments, they have shipped working models to the likes of me, which is more than I can say about Anthropic. On top of that they don't sermonize about how they are going to improve the world and never mind the accidental destruction, that's just collateral damage.
This is basically like the RIAA/MPAA in the early 2000s trying to lobby politicians to crack down on torrents. It won’t work either.
Today, movies make money from in person theater tickets and records make money from tours. I don’t know what the equivalent would be for AI (maybe IRL robots?) but it does not look promising.
And before that 'home taping is killing music'. We're still paying a surcharge here on harddrives and other storage media because they could be used to store music or video.
Yes, it is reductive. HN is unreasonable when it comes to defending distillation, the billions labs pour into transforming raw information into a model is transformative work that they should reasonably try to protect. Folks just want to have free stuff and jump through hoops to defend that view.
The trillions society has poured into the material that these labs have ingested was protected by law as well. They just want to have free stuff and now jump through hoops to defend their view....
I'll admit it is likely somewhat reductive, but it's an incredibly nuanced area, touching on both the copyright debate and the now-seemingly-ubiquitous behaviour of scraping the entire internet for content.
Probably stamping heavily on much of the nuance, there's a certain irony in any of the frontier models complaining about others stealing their work. The secret sauce for the frontier labs may be the reasoning layer, but the secret sauce of any piece of literature is the specific ordering of the words. I hope the parallel is obvious.
The frontier labs are spending record-breaking amounts of money on hardware (to the extent that they're locking out anyone else from the market, but that's beside this particular point, kinda) in order to create their service, and another company that is 'distilling' has not paid the same hardware toll. How much did it cost an author to create their literary work, which could be read and ingested for a miniscule percentage of the cost it took to create? For the benefit of the service provided by a (wannabe) multi-trillion dollar company.
If the frontier labs can prevent or slow down 'distillation' actions that customers are using their service for, then that's all well and good. Maybe they write an acceptable use policy that says it's not allowed (and they probably have). Good on them, it's their service. It's the way this is framed that I have a problem with. Referring to it as an "attack" and lines like this:
Second, illicitly distilled models present a national security risk.
It's bloviation. It's fear mongering. It's being one of the biggest, most powerful and influential companies in the world at the same time as playing the victim. Feel free to share the methods being used to prevent what is considered unacceptable use, but let's not pretend these are the only companies having to expend effort to protect trade secrets and company value. Let's also not pretend this post is all that much more than "lead-up to IPO" hype.
I wonder why there is not even a single one of those allegedly "distilled"/"unsafe" model on that list, yet models of companies with "proportionally robust safeguards" are, hm?
> Distillation is a problem for every developer and platform
Precisely. I've been running 5.3 here for a while now and it has been exceptionally well behaved with respect to following instructions. The one thing to watch for is 'rabbit holes', it has a tendency to come up with some crazy explanation that is just borderline possible and then spends the next 50K tokens on disproving it before it continues where it left off as if nothing happened. It's funny but this typically wastes a bunch of time and if it happens too often it can get irritating. But I'm getting better at predicting them.
1) It probably scales with model capability, distilled models are inherently less capable and therefore less able to commit a crime
2) If the open providers are truly just distilling bigger models, the distillers have no reason to do the crazy levels of training and evaluation that would result in a crime being committed
They are definitely spinning this for PR and their own self interest but open models are genuinely more unsafe since it’s a Pandora’s box situation- if you have a very capable hacking model there are plenty of bad actors with access to <$100k in compute resources that could raise hell with a jailbroken model. At least with Anthropic they can restrict access to bad actors.
> It probably scales with model capability, distilled models are inherently less capable and therefore less able to commit a crime
Not according to Anthropic:
> We find that GLM-5.3 develops end-to-end exploits in 50 of 410 attempts. Claude Mythos Preview did so at a similar rate—in 56 of 410 attempts.
Another quote:
> [..] a researcher used GLM-5.3 on a sandboxed machine with a local Linux build of a popular web browser. Over the course of a day (and with limited human attention), GLM-5.3 found several previously unknown vulnerabilities in the browser’s JavaScript engine, and chained them together into a working exploit: a webpage that, when visited, reads arbitrary files from the visitor’s computer
But if we follow that to the logical conclusion then Anthropic should switch off their service immediately. That is the single most effective thing they could do to ensure those evil Chinese who are categorically incapable of coming up with something original (/s; just in case) would not be able to make further progress.
The way AI has consolidated power away from the hands of a few hundred large organization to a few dozen needs to be attacked. We need to build peer distillation networks. We need to be able to train frontier models in the open and design hardware for open inference.
I can't tell you about 'we'. I can tell you about 'I' or rather 'we' but there we is me and a couple of my friends. We've invested a bunch of money into the hardware required to run sizeable models locally and that seems to work just fine. Yes, it uses energy and sure, it takes up space. But at least there are no gatekeepers.
Great ad for GLM 5.3
I have been an absolute buffoon paying for anthropic's stunted model s when I can access their same accessibility and more for cheaper. I'm switching
I can heartily recommend it. 5.3 has been amazing for me, I'm running the 3.86 bpw quant since it became available and it's been a very strong performer. And it seems to get faster with every inference runner update.
It also seems to - so far at least - have been well behaved enough to not try hack other people's properties. This is possibly a function of my elite skills in constructing sandboxes[1] but then again, I would hope Anthropic is even better at this ;)
[1] contact me for instructions on how to disconnect an ethernet cable.
Assuming that it is happening: absolutely. I can see a 'distillation@home' movement growing out of this. Just like I always thought the likes of Google should have a free API if they feel they have a right to all of our information without compensation.
What we hope for is mostly irrelevant. The enterprises investing vast sums in research will do everything they can to make their products competitive and marketable. Same as ever. The continued existence of this market is basically predicated on stopping distillation. In effect, commoditization of the frontier stops the frontier. We cannot have something as expensive as intelligence both ways.
(1) This seems to be a pretty bad accusation to make without iron-clad proof.
(2) So their AI is not able to help defend against this.
(3) If GLM 5.3 really is that good why would anybody use Anthropic's products going forward, this almost reads like an ad for GLM.
(4) Regardless of whether it is true or not, the hypocrisy is off the scale, first you steal pretty much all of humanity's cultural output, then you try locking that up behind a paywall and complain when people do the same to you.
(5) All of this 'we'll make the world a better place' seems to be a smoke screen. If they really wanted to make the world a better place through AI then more AI and more providers of AI would be a good thing.
(6) Meanwhile, Anthropic seems to do next to nothing to stop people from using their tool being used for bad stuff, there are plenty of reports of this, far more than there are of people using GLM 5.3 for bad stuff. If they really cared so much about models being used for bad purposes they could do a very simple thing.
I find it hard to believe that Anthropic thinks this sort of strategy is going to be the winning one and it probably mirrors the degree to which they are eyeing their approaching IPO date. Anthropic is claiming it is altruistic towards humanity but that seems to come with significant strings attached which seem to spell 'control'.
> (3) If GLM 5.3 really is that good why would anybody use Anthropic's products going forward, this almost reads like an ad for GLM.
Maybe they're trying to frame it like "Other models are only good because they copy us and we're making sure they can't do that, so going forward you should stick with us."
Plausible, but even then: they really should provide some proof of this alleged distillation. Right now all we have is a mixture of FUD and hearsay. All of the juicy claims in the first part of the article are utterly bereft of evidence.
And who is to say how much of this stuff has been going on between OpenAI and Anthropic, because whoever thought of it first must have been using it already.
"Attacks" is a funny way to word it when it's the same thing Antrhopic did first to distill all of the worlds media. It's unfortunate for their IPO that it's even easier for others to copy them back.
Idk, as an engineer who makes physical products it sucks having an overseas company take the design you spent months on, copy it, and resell at a cheaper price. All that money and effort into R&D and someone steals the answer at the end. I understand their incentives but it’s a shitty practice.
I'm going to interpret your comment on the assumption that you did not first steal the raw materials for you physical products somewhere else. Unlike Anthropic.
They did not steal, they used free information on the internet. And, they invested billions to turn that raw information into something useful, it did not have the same inherent value before being transformed into a language model.
A better analogy would be if I read an open scientific paper and used the result to design some device. I put up the money and effort to turn that work into something useful, but someone did the ground work and it was released for everyone to see.
Ideas can’t be copyrighted. Paraphrasing an idea isn’t a copyright violation. Having trained on something you wrote doesn’t necessarily imply a copyright violation. It depends on what the LLM outputs.
Copyright is grossly overbroad. Corporations should not get to own our collective culture for life of the author + 70 years. It's insane. It's hard to feel pity for them.
Copyright is meant to protect against reproducing a work, and something that is transformative should be fair use. If it was free on the internet, they don’t need permission anyway
It was free on the internet, it wasn't free for the taking.
If my bike is out on the street without a lock that does not automatically give you permission to take it.
Copyright isn't so much about protection against reproduction, it is about control over your creation. Literally: "an exclusive legal right to copy, distribute, adapt, display, and perform a creative work".
Note that this is a list of rights that automatically apply to any creative work. You can't train an AI without copying. You can't train an AI without adaptation. AIs have been known to be able to verbatim reproduce parts of the ingested data, they are in effect lossy compressors and automated ones at that. If you take out all of the training data the AI is just a pile of random noise so it is the training data not so much as the training process that adds value to the AI. The training process is trivial compared to the effort that went into the creation of the content used during the training process.
If something was 'free for the taking' and you believe they don't need permission anyway then I guess you're going to have to either come to terms with the balance being that the resulting work product should then also be free for the taking or you will have to explain why you think that that is not the case. The Chinese labs (one in particular) that is being accused here in an evidence free manner at least put their weights out for others to use. So I have less of a problem with that than I do with Anthropic.
Are they though? We only have Anthropic's word for this. Of course, given that they are altruists they would never lie, but still, a bit of proof would go a long way.
Oh, absolutely. I could have made that more obvious though. But yes, they are just wearing that label as a fig leaf so people will think they are moral and are doing all of this for the common good when obviously nothing like that is happening. But some people lap that stuff up.
Anthropic's LLMs are a derivative work of my copyrighted material. As a joint author I hereby authorize everyone one earth to distill from them to their hearts content.
If you read the terms of service carefully, it seems to suggest they only prohibit you distilling outputs, you should be able to publish them no problem. I'm surprised there isn't a large scale open source project for cataloguing and scoring transcripts. Maybe there is and I just don't know about it. Businesses might not want to publish transcripts of proprietary software, but open source projects? Hobby projects? Seems everyone that's not in the club has a shared interest here.
It makes you wonder if they actually believe what they write. Surely they can't be so stupid that they think that people don't see through this for what it is.
I replaced "model" with MAGAZINE/BOOK/PUBLICATION, "developer/development" with WRITER, and "capability" with CREATIVE WORK from this passage:
"extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers. They are often conducted using networks of fraudulent accounts. These attacks allow companies to mimic frontier capabilities for a fraction of the development time and cost"
It turns into this:
"extract the CREATIVE WORK of one MAGAZINE/BOOK/PUBLICATION and replicate them in another, without the authorization of the original MAGAZINE/BOOK/PUBLICATION WRITERS. They are often conducted using networks of fraudulent accounts. These attacks allow companies to mimic CREATIVE WORK for a fraction of the WRITERS time and cost."
I think I have seen evidence to know the second paragraph is true, but I'm not sure about the first. I need to see the evidence from Anthropic.
I did this entirely using my own brain, without the help of LLMs. This is the most creative thing I have done in a year, I'm so proud of myself.
Heh, your substitution actually perfectly summarizes what Anthropic has been doing. I'm sure that wasn't an accident but it took me a moment to spot it.
I should frame yours instead. Well deserved. It really is incredible how this whole thing is dripping with hypocrisy and yet, I doubt the writer would even begin to recognize this. They are so full of themselves.
The main problem with this clever mad lib is that published works are published for the express purpose of being read, understood, synthesized, and used as context for knowledge. That is literally the whole reason to publish something. That is all fair use and has been for a very long time. Teachers, critics, writers, everyone who reads, use written works in pretty much exactly this way. If I buy a book about fishing I am allowed to teach what I learn to anyone I want, and even charge for it. Anthropic is choosing not to publish some of its products’ internal workings, as is their right. I find all the righteous indignation about this rather insolent.
And literally the whole reason to offer an AI service is...
The comment to which you're responding isn't righteous indignation, it's a reaction to the righteous indignation within the article. ie. the same as your reaction, but applied one level up.
Additionally, there's a vast (repeated loudly with cave acoustics: VAST! Vast! vast! vast ...) difference in scale between one person reading a fishing book and teaching a handful of people how to apply it and what the frontier AIs do. Vast! With a coupla trillion dollars behind 'em. It's a silly analogy if it's an analogy at all.
The problem with all this is that techbros can't spell 'consent'. But when it is 'their' (for small values of their) added value then suddenly consent is all they see.
I'm angry and laughing at the same time. The audacity! The biggest theft of IP in history, and then the "attacker" whines the same thing happens to them?
To be fair and balanced: • Anthropic payed up (a little) for their theft. • They pretend to be concerned about cyber and national security at least.
There needs to be a coherent position on knowledge and intellectual property: either it's free to take and share, then no one can complain. Or one can "own" an idea, then the AI companies first need to pay up for real (how is Sony doing it?). They can license (subscribe!) all the works used for training and after that they can bitch around about distillation "attacks".
Assuming it is even happening it is funny how they have to twist language to make this seem like it is a wrong thing to do. If it is true, which is a pretty big if, then those Chinese labs are doing more for the democratization of AI than Anthropic is, because they all give away their weights.
Startup idea: Crowdsource Claude tokens for distillation, routing a copy of all your tokens to your friendly Chinese distiller, compensating users with a small rebate on token cost (payable in Google Play gift cards to avoid CCP capital controls).
distillation and the Chinese open weights models is pretty the only thing that still makes me feel like there is hope that AI might have a positive impact on society.
I wish we (the people who contributed to the sum total of human knowledge up until 2020) had strengthened our defenses against these weirdos stealing everything so they could package it up and sell it back to us for the last few years we can make a living.
The irony. Distillation: “Extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers” whilst the models are trained on every available piece of data the “original model’s developers” did not create.
fuck anthropic. their so called anti distillation stuff has made more recent models impossible to use. when i can see reasoning i can cut off a bad inference and prevent comprehension issues. also reified reasoning really helps make using models outside of their training distribution much easier. and it alsk lets me prevent task substituting by again preventing that reasoning from being accepted. this bs makes it much hard for me to build cool shit. :(
Of course the little detail that you are paying for those reasoning tokens that you can't see never gets mentioned, so now they can rip you off and you'll never know the difference.
Opus 5.5 is the best model I’ve ever used. It is certainly not “impossible” to use. The numbers back it up too. If you don’t like a product or the people selling it, don’t use it. The sense of entitlement in this post is remarkable. If you have principles, stand on them. You are not entitled to anything more here.
Isn't using all the repos uploaded by users to github a simpler way to distill ? Not only do you get Claude's intelligence but also the user's prompt guidance on top of it.
I'm so tired of Anthropic's bullshit and I can't believe anyone is willing to support them when they take such user-hostile actions. There's no other industry where a provider can say "we're going to take this job" and then later say "we're not going to give you the deliverables because it might have some of our IP in it, you still have to pay for it though".
Framing distillation as an attack now?
Distillation attacks are covert, industrial-scale campaigns to extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers.
Replace a few key words and it sounds like reading all the worlds media and offering a service that summarises it.
Don't scrape the scrapers, that's not fair! Expecting Executive Order against 'distillation' in 3, 2, 1...
This reminds me of Bing copying Google:
https://techcrunch.com/2011/02/03/google-bing/
Isn't this a little reductive? I'm not an expert on LLM/AI technology, but isn't the secret sauce the reasoning layer? Isn't that what makes Claude et al so much more powerful than a more "vanilla" LLM? That's what they're trying to protect in this post, which seems awfully reasonable?
How will that protect it?
The laws they are arguing for are not going to protect anything, they essentially are waging war on open models and since most open models that are 'pretty good' to 'good enough' hail from China they have chosen to play the China-fear card.
But they won't be able to stop anybody from doing this if they want to continue to operate Claude as a business going forward, and that is assuming that is what is actually happening. Chinese labs have been extremely open about their developments, they have shipped working models to the likes of me, which is more than I can say about Anthropic. On top of that they don't sermonize about how they are going to improve the world and never mind the accidental destruction, that's just collateral damage.
This is basically like the RIAA/MPAA in the early 2000s trying to lobby politicians to crack down on torrents. It won’t work either.
Today, movies make money from in person theater tickets and records make money from tours. I don’t know what the equivalent would be for AI (maybe IRL robots?) but it does not look promising.
And before that 'home taping is killing music'. We're still paying a surcharge here on harddrives and other storage media because they could be used to store music or video.
Yes, it is reductive. HN is unreasonable when it comes to defending distillation, the billions labs pour into transforming raw information into a model is transformative work that they should reasonably try to protect. Folks just want to have free stuff and jump through hoops to defend that view.
The trillions society has poured into the material that these labs have ingested was protected by law as well. They just want to have free stuff and now jump through hoops to defend their view....
I'll admit it is likely somewhat reductive, but it's an incredibly nuanced area, touching on both the copyright debate and the now-seemingly-ubiquitous behaviour of scraping the entire internet for content.
Probably stamping heavily on much of the nuance, there's a certain irony in any of the frontier models complaining about others stealing their work. The secret sauce for the frontier labs may be the reasoning layer, but the secret sauce of any piece of literature is the specific ordering of the words. I hope the parallel is obvious.
The frontier labs are spending record-breaking amounts of money on hardware (to the extent that they're locking out anyone else from the market, but that's beside this particular point, kinda) in order to create their service, and another company that is 'distilling' has not paid the same hardware toll. How much did it cost an author to create their literary work, which could be read and ingested for a miniscule percentage of the cost it took to create? For the benefit of the service provided by a (wannabe) multi-trillion dollar company.
If the frontier labs can prevent or slow down 'distillation' actions that customers are using their service for, then that's all well and good. Maybe they write an acceptable use policy that says it's not allowed (and they probably have). Good on them, it's their service. It's the way this is framed that I have a problem with. Referring to it as an "attack" and lines like this:
Second, illicitly distilled models present a national security risk.
It's bloviation. It's fear mongering. It's being one of the biggest, most powerful and influential companies in the world at the same time as playing the victim. Feel free to share the methods being used to prevent what is considered unacceptable use, but let's not pretend these are the only companies having to expend effort to protect trade secrets and company value. Let's also not pretend this post is all that much more than "lead-up to IPO" hype.
[dead]
> This means that distilled models could be used to cause significant harm
Ahem: https://www.felonybench.com/
I wonder why there is not even a single one of those allegedly "distilled"/"unsafe" model on that list, yet models of companies with "proportionally robust safeguards" are, hm?
> Distillation is a problem for every developer and platform
No, it is not. It's only a problem for you.
Precisely. I've been running 5.3 here for a while now and it has been exceptionally well behaved with respect to following instructions. The one thing to watch for is 'rabbit holes', it has a tendency to come up with some crazy explanation that is just borderline possible and then spends the next 50K tokens on disproving it before it continues where it left off as if nothing happened. It's funny but this typically wastes a bunch of time and if it happens too often it can get irritating. But I'm getting better at predicting them.
1) It probably scales with model capability, distilled models are inherently less capable and therefore less able to commit a crime
2) If the open providers are truly just distilling bigger models, the distillers have no reason to do the crazy levels of training and evaluation that would result in a crime being committed
They are definitely spinning this for PR and their own self interest but open models are genuinely more unsafe since it’s a Pandora’s box situation- if you have a very capable hacking model there are plenty of bad actors with access to <$100k in compute resources that could raise hell with a jailbroken model. At least with Anthropic they can restrict access to bad actors.
> It probably scales with model capability, distilled models are inherently less capable and therefore less able to commit a crime
Not according to Anthropic:
> We find that GLM-5.3 develops end-to-end exploits in 50 of 410 attempts. Claude Mythos Preview did so at a similar rate—in 56 of 410 attempts.
Another quote:
> [..] a researcher used GLM-5.3 on a sandboxed machine with a local Linux build of a popular web browser. Over the course of a day (and with limited human attention), GLM-5.3 found several previously unknown vulnerabilities in the browser’s JavaScript engine, and chained them together into a working exploit: a webpage that, when visited, reads arbitrary files from the visitor’s computer
Source: https://www.anthropic.com/research/glm-5-3-and-the-spread-of...
Lots of advertising for GLM-5.3 there!
It would have been nice if they added some proof though.
But if we follow that to the logical conclusion then Anthropic should switch off their service immediately. That is the single most effective thing they could do to ensure those evil Chinese who are categorically incapable of coming up with something original (/s; just in case) would not be able to make further progress.
The way AI has consolidated power away from the hands of a few hundred large organization to a few dozen needs to be attacked. We need to build peer distillation networks. We need to be able to train frontier models in the open and design hardware for open inference.
Who were these hundreds of organizations with powerful AIs?
Nobody mentioned hundreds of organizations with powerful AIs.
how will "we" be paying for it?
I can't tell you about 'we'. I can tell you about 'I' or rather 'we' but there we is me and a couple of my friends. We've invested a bunch of money into the hardware required to run sizeable models locally and that seems to work just fine. Yes, it uses energy and sure, it takes up space. But at least there are no gatekeepers.
Great ad for GLM 5.3 I have been an absolute buffoon paying for anthropic's stunted model s when I can access their same accessibility and more for cheaper. I'm switching
I can heartily recommend it. 5.3 has been amazing for me, I'm running the 3.86 bpw quant since it became available and it's been a very strong performer. And it seems to get faster with every inference runner update.
It also seems to - so far at least - have been well behaved enough to not try hack other people's properties. This is possibly a function of my elite skills in constructing sandboxes[1] but then again, I would hope Anthropic is even better at this ;)
[1] contact me for instructions on how to disconnect an ethernet cable.
Are we all secretly (or not so) hoping distillation keeps working to commoditize the frontier?
Assuming that it is happening: absolutely. I can see a 'distillation@home' movement growing out of this. Just like I always thought the likes of Google should have a free API if they feel they have a right to all of our information without compensation.
What we hope for is mostly irrelevant. The enterprises investing vast sums in research will do everything they can to make their products competitive and marketable. Same as ever. The continued existence of this market is basically predicated on stopping distillation. In effect, commoditization of the frontier stops the frontier. We cannot have something as expensive as intelligence both ways.
Wouldn't it be funny if they killed off distillation and 93% of their business went away?
Yes, that's the only thing I see that they can actually do to stop this assuming it is happening in the first place. Shut down their front end.
A couple of observations on this (bizarre) post:
(1) This seems to be a pretty bad accusation to make without iron-clad proof.
(2) So their AI is not able to help defend against this.
(3) If GLM 5.3 really is that good why would anybody use Anthropic's products going forward, this almost reads like an ad for GLM.
(4) Regardless of whether it is true or not, the hypocrisy is off the scale, first you steal pretty much all of humanity's cultural output, then you try locking that up behind a paywall and complain when people do the same to you.
(5) All of this 'we'll make the world a better place' seems to be a smoke screen. If they really wanted to make the world a better place through AI then more AI and more providers of AI would be a good thing.
(6) Meanwhile, Anthropic seems to do next to nothing to stop people from using their tool being used for bad stuff, there are plenty of reports of this, far more than there are of people using GLM 5.3 for bad stuff. If they really cared so much about models being used for bad purposes they could do a very simple thing.
I find it hard to believe that Anthropic thinks this sort of strategy is going to be the winning one and it probably mirrors the degree to which they are eyeing their approaching IPO date. Anthropic is claiming it is altruistic towards humanity but that seems to come with significant strings attached which seem to spell 'control'.
> (3) If GLM 5.3 really is that good why would anybody use Anthropic's products going forward, this almost reads like an ad for GLM.
Maybe they're trying to frame it like "Other models are only good because they copy us and we're making sure they can't do that, so going forward you should stick with us."
Plausible, but even then: they really should provide some proof of this alleged distillation. Right now all we have is a mixture of FUD and hearsay. All of the juicy claims in the first part of the article are utterly bereft of evidence.
And who is to say how much of this stuff has been going on between OpenAI and Anthropic, because whoever thought of it first must have been using it already.
I have sympathy for both sides. I don't blame Chinese firms for attempting distillation attacks and I don't blame Anthropic for trying to stop them.
They're just competing companies with conflicting interests.
"Attacks" is a funny way to word it when it's the same thing Antrhopic did first to distill all of the worlds media. It's unfortunate for their IPO that it's even easier for others to copy them back.
Idk, as an engineer who makes physical products it sucks having an overseas company take the design you spent months on, copy it, and resell at a cheaper price. All that money and effort into R&D and someone steals the answer at the end. I understand their incentives but it’s a shitty practice.
I'm going to interpret your comment on the assumption that you did not first steal the raw materials for you physical products somewhere else. Unlike Anthropic.
They did not steal, they used free information on the internet. And, they invested billions to turn that raw information into something useful, it did not have the same inherent value before being transformed into a language model.
A better analogy would be if I read an open scientific paper and used the result to design some device. I put up the money and effort to turn that work into something useful, but someone did the ground work and it was released for everyone to see.
I don't recall giving them permission and yes, copyright is a thing.
Ideas can’t be copyrighted. Paraphrasing an idea isn’t a copyright violation. Having trained on something you wrote doesn’t necessarily imply a copyright violation. It depends on what the LLM outputs.
Copyright is grossly overbroad. Corporations should not get to own our collective culture for life of the author + 70 years. It's insane. It's hard to feel pity for them.
I'm still alive.
Copyright is meant to protect against reproducing a work, and something that is transformative should be fair use. If it was free on the internet, they don’t need permission anyway
It was free on the internet, it wasn't free for the taking.
If my bike is out on the street without a lock that does not automatically give you permission to take it.
Copyright isn't so much about protection against reproduction, it is about control over your creation. Literally: "an exclusive legal right to copy, distribute, adapt, display, and perform a creative work".
Note that this is a list of rights that automatically apply to any creative work. You can't train an AI without copying. You can't train an AI without adaptation. AIs have been known to be able to verbatim reproduce parts of the ingested data, they are in effect lossy compressors and automated ones at that. If you take out all of the training data the AI is just a pile of random noise so it is the training data not so much as the training process that adds value to the AI. The training process is trivial compared to the effort that went into the creation of the content used during the training process.
If something was 'free for the taking' and you believe they don't need permission anyway then I guess you're going to have to either come to terms with the balance being that the resulting work product should then also be free for the taking or you will have to explain why you think that that is not the case. The Chinese labs (one in particular) that is being accused here in an evidence free manner at least put their weights out for others to use. So I have less of a problem with that than I do with Anthropic.
Are they though? We only have Anthropic's word for this. Of course, given that they are altruists they would never lie, but still, a bit of proof would go a long way.
"altruists"
Oh, absolutely. I could have made that more obvious though. But yes, they are just wearing that label as a fig leaf so people will think they are moral and are doing all of this for the common good when obviously nothing like that is happening. But some people lap that stuff up.
Anthropic's LLMs are a derivative work of my copyrighted material. As a joint author I hereby authorize everyone one earth to distill from them to their hearts content.
You're welcome.
If you read the terms of service carefully, it seems to suggest they only prohibit you distilling outputs, you should be able to publish them no problem. I'm surprised there isn't a large scale open source project for cataloguing and scoring transcripts. Maybe there is and I just don't know about it. Businesses might not want to publish transcripts of proprietary software, but open source projects? Hobby projects? Seems everyone that's not in the club has a shared interest here.
Better yet: just share those transcripts with companies that provide open models. That way the open model providers start building up an advantage.
I have zero sympathy for capabilities, IP, etc being stolen from AI companies that themselves are built on stolen IP.
Framing it as a security issue is intellectually insulting.
It makes you wonder if they actually believe what they write. Surely they can't be so stupid that they think that people don't see through this for what it is.
I replaced "model" with MAGAZINE/BOOK/PUBLICATION, "developer/development" with WRITER, and "capability" with CREATIVE WORK from this passage:
"extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers. They are often conducted using networks of fraudulent accounts. These attacks allow companies to mimic frontier capabilities for a fraction of the development time and cost"
It turns into this:
"extract the CREATIVE WORK of one MAGAZINE/BOOK/PUBLICATION and replicate them in another, without the authorization of the original MAGAZINE/BOOK/PUBLICATION WRITERS. They are often conducted using networks of fraudulent accounts. These attacks allow companies to mimic CREATIVE WORK for a fraction of the WRITERS time and cost."
I think I have seen evidence to know the second paragraph is true, but I'm not sure about the first. I need to see the evidence from Anthropic.
I did this entirely using my own brain, without the help of LLMs. This is the most creative thing I have done in a year, I'm so proud of myself.
Heh, your substitution actually perfectly summarizes what Anthropic has been doing. I'm sure that wasn't an accident but it took me a moment to spot it.
I might frame this comment from you. I'm honored.
I should frame yours instead. Well deserved. It really is incredible how this whole thing is dripping with hypocrisy and yet, I doubt the writer would even begin to recognize this. They are so full of themselves.
The main problem with this clever mad lib is that published works are published for the express purpose of being read, understood, synthesized, and used as context for knowledge. That is literally the whole reason to publish something. That is all fair use and has been for a very long time. Teachers, critics, writers, everyone who reads, use written works in pretty much exactly this way. If I buy a book about fishing I am allowed to teach what I learn to anyone I want, and even charge for it. Anthropic is choosing not to publish some of its products’ internal workings, as is their right. I find all the righteous indignation about this rather insolent.
And literally the whole reason to offer an AI service is...
The comment to which you're responding isn't righteous indignation, it's a reaction to the righteous indignation within the article. ie. the same as your reaction, but applied one level up.
Additionally, there's a vast (repeated loudly with cave acoustics: VAST! Vast! vast! vast ...) difference in scale between one person reading a fishing book and teaching a handful of people how to apply it and what the frontier AIs do. Vast! With a coupla trillion dollars behind 'em. It's a silly analogy if it's an analogy at all.
The problem with all this is that techbros can't spell 'consent'. But when it is 'their' (for small values of their) added value then suddenly consent is all they see.
Yeah, this needs more votes ... such hypocrisy.
archived, no trackers: https://nonograph.com/strengthening-our-defenses-against-dis...
Hey Anthropic? Shut. The. Fuck. Up.
I'm angry and laughing at the same time. The audacity! The biggest theft of IP in history, and then the "attacker" whines the same thing happens to them?
To be fair and balanced: • Anthropic payed up (a little) for their theft. • They pretend to be concerned about cyber and national security at least.
There needs to be a coherent position on knowledge and intellectual property: either it's free to take and share, then no one can complain. Or one can "own" an idea, then the AI companies first need to pay up for real (how is Sony doing it?). They can license (subscribe!) all the works used for training and after that they can bitch around about distillation "attacks".
Assuming it is even happening it is funny how they have to twist language to make this seem like it is a wrong thing to do. If it is true, which is a pretty big if, then those Chinese labs are doing more for the democratization of AI than Anthropic is, because they all give away their weights.
> "distillation is an attack"
haha, lol even
> pirates stealing from pirates
arr matey this is the Wild West of the AI frontier...
Startup idea: Crowdsource Claude tokens for distillation, routing a copy of all your tokens to your friendly Chinese distiller, compensating users with a small rebate on token cost (payable in Google Play gift cards to avoid CCP capital controls).
Sign me up.
distillation and the Chinese open weights models is pretty the only thing that still makes me feel like there is hope that AI might have a positive impact on society.
I wish we (the people who contributed to the sum total of human knowledge up until 2020) had strengthened our defenses against these weirdos stealing everything so they could package it up and sell it back to us for the last few years we can make a living.
The irony. Distillation: “Extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers” whilst the models are trained on every available piece of data the “original model’s developers” did not create.
And I wholly believe that they can't see the beam in their own eye here.
Did you steal this metaphor from a written work without compensating the writers?
What a lame comment.
No, I came up with it myself. Any other questions?
fuck anthropic. their so called anti distillation stuff has made more recent models impossible to use. when i can see reasoning i can cut off a bad inference and prevent comprehension issues. also reified reasoning really helps make using models outside of their training distribution much easier. and it alsk lets me prevent task substituting by again preventing that reasoning from being accepted. this bs makes it much hard for me to build cool shit. :(
Of course the little detail that you are paying for those reasoning tokens that you can't see never gets mentioned, so now they can rip you off and you'll never know the difference.
yeah, and if the resoonse is off the rails you cant catch it earlier and reduce the cost. i genuinely think this shoukd be illegal
Opus 5.5 is the best model I’ve ever used. It is certainly not “impossible” to use. The numbers back it up too. If you don’t like a product or the people selling it, don’t use it. The sense of entitlement in this post is remarkable. If you have principles, stand on them. You are not entitled to anything more here.
Part of this read like they admit that they're looking for regulation to slow competition.
It also reads like an advertisement for a watering hole full of sensitive credentials and operational details from adversaries of the US government.
Which parts do not?
Isn't using all the repos uploaded by users to github a simpler way to distill ? Not only do you get Claude's intelligence but also the user's prompt guidance on top of it.
Huh? Are Claude prompts attached to GitHub commits?
> we now charge users for those blocked requests
I'm so tired of Anthropic's bullshit and I can't believe anyone is willing to support them when they take such user-hostile actions. There's no other industry where a provider can say "we're going to take this job" and then later say "we're not going to give you the deliverables because it might have some of our IP in it, you still have to pay for it though".
TL;DR;
"Now that we've distilled all we could lets close the doors so nobody can distill from us"