I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize.
AFAIK no provider guarantees compatibility of reasoning traces, even in the same model generation, and we've in practice seen most of the big LLM APIs throw errors indicating incompatibility (at least transiently) when switching models. The only stable solution right now is to just throw away reasoning traces whenever a model is switched.
Fair (I haven't been using the encrypted-reasoning systems, though this is common in open ones - I'm kinda surprised it's an option in encrypted ones too), though what they're doing here is cross-user replays in addition to cross-model.
There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves?
This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation.
If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT.
For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.
I suspect that there are companies with internal proxies that load-balance across keys, and they didn't want to break that when adding encrypted reasoning.
It’s about being able to change models mid-task. For example, I want to be able to plan using Fable but implement the plan using Sonnet, and that won’t work if this is implemented.
Actually, that brings up a good reason they can't fix it. Fable falls back to Opus when the topic is too "unsafe". That behavior requires traces than can move between models!
Star Trek is a post-scarcity society, those problems don't exist there unless you're in the middle of a crisis and on emergency power.
LLMs briefly seemed like this too, after subscriptions made the SOTA models too cheap to meter, but before they walked back on that and introduced quotas...
For plan it's relatively easy, just make the plan the artifact. The point is to ingest knowledge with one model and use it in another, and that is not necessarily easily expressible in natural language.
I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?
Makes no difference. There is a policy as to whether to allow use of a reasoning trace in a given context. Whether that trace originates from authenticated ciphertext or a backend database is basically irrelevant.
Yes, this storage would be growing exponentially making the disk space and latency problems harder (add the disaster recovery/backups). I think the choice of using client side is not too bad if you ensure that its secured properly. Also the company can excuse itself from the liability of storing sensitive data on its servers, thats a big deal in itself to be compliant for enterprise audits
1. The down side is that it cannot be used across the clients even for the same user
2. Using the same encryption key was a bad choice here, a per user key would have solved this issue for sure.
Having thought about this a little more, it's clear that server-side storage is not compatible with Zero Data Retention (ZDR). However, in non-ZDR settings, it seems likely that the providers are capturing all that data anyway?
> a per user key would have solved this issue for sure
It would have helped with PII leakage, but not with plain-text trace extraction attacks, right?
Per user encryption key ties it with the user session (assuming you do authentication properly), no one else can access it. User being able to see the information is not really an attack vector in this case.
The compliance rules at times are outdated and people skirt around them by following the worded rule instead of the intent.
I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design.
The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier.
If I'm reading this right, they literally just ask a LLM to tell them what the traces say, with the key being that the traces are portable across LLM models, so they can switch to a smaller one that's easier to jailbreak.
For all of the years of research, thinking and talking about model alignment, safety, confinement, etc, when it comes down to it these companies appear to be entirely incompetent.
If you've watched the Blackhat OpenAI/Huggingface incident talk, my conclusion is that they (believe they) cannot afford being competent, these models are too expensive to train, they won't even pull the plug when one literally goes rogue, as the "very persistent" model that "had seen the secret message board" was included in the second series of runs, and whaddayaknow it happened again. They proudly proclaimed they cleared the message board and then continued the training run with the rogue AI model included ...
By some interpretations protecting your frontier model with strong safeguards from being distilled into an open source model without safeguards is a safety issue.
You could limit what it was training on in the first place - however that would damage capability - and it's difficult to curate the input, especially when the models can do 2+2. ie the choice is between model power and safety - and they choose power and everything else is a sticking plaster.
One thing I find amusing is the refusal of a lot of the models to now output a lab based protocol because of fears about 'weapons' - yet I can buy a textbook or simply read papers for exact protocols.
I find it hard to reason that a person who isn't motivated enough to read a paper or buy a book, is somehow enabled to make a biological weapon because of ChatGPT - despite them needed to buy a whole bunch of specialist equipment and reagents to do it.
Are there a whole bunch of proto-terrorists who are frustrated simply because they don't know where to start?
Maybe the only place their might be radicalized teenagers - but then that's perhaps a reason for keeping them off the internet full stop :-)
That's also not possible, what's the worst problems enabled by LLM? Political propaganda, influence of population at scale, misleading advertising, social media bots... None of that will be filtered by a "safety" filter.
The knowledge to create weapons is already widespread, the idea that terrorists need chatgpt for that is laughable
> For some AIME problems Opus 4.8 sometimes states the answer before deriving it. We find that the API summary does not always preserve this distinction, and can instead make the reasoning appear like a clean derivation.
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
All LLM benchmarks have an expiration date once they're released to the public. They get spread so far and wide across the internet and GitHub that you have to assume they're in the training data for every LLM with a cutoff date after their release.
The real question is whether or not the training was directed to optimize for those benchmarks.
The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.
So ... just a thought but could this be somewhat solved by, say if you were an LLM benchmark creator, using clever trickery?
Like what if you made sure the wrong answers just appear 100x more often than the right ones. When scraping for new data to use I doubt they can verify the correctness of complex benchmark question answers to exclude the wrong ones.
Then I dunno store the hash of the correct answers somewhere else, and eh try not to leak it. But even if it gets leaked, that just means perhaps at inference time, a clever agentic LLM could go for for those hashes and maybe determine what is correct, but not during training.
I'm not sure, but wouldn't this make sure that at least they aren't literally trained on the correct question/answer pairs.
I guess there would always be people that end up publishing the correct list, anyway. But that's why you try to be 100x "louder" with the wrong answers.
btw, different thing, but when I look at those charts, I kind of came to the opposite conclusion as you did :) IMHO not that many dots off the line, and the ones that are on the line, are literally ON the line, not like a "roughly linear looking cloud of points". Which suggests that the reasoning is either (in the majority of cases) exactly the same amount of tokens (on the 1:1 line), and when it's even a little bit off the line it could (and should) be discarded, still leaving what seems to me at least 95% of the traces as exactly correct. but I grant, I didn't read the paper, and just came to that conclusion after viewing the chart :)
Any word can be applied to any concept with any meaning thanks to the fluidity of vernacular.
Language is all just sounds and markings. Anything can be redefined to mean anything, and anyone can decide to aggressively assert their preferred definition of a word.
Words can change meaning, but whether or not they actually do is a social function of how they are observably used.
Words can also have multiple meanings concurrently. When it comes to the word "steal" in these discussions, people generally are arguing past each other in regards to the single definition they're thinking of.
Of course, you can assert that the meaning of "steal" only applies to physical items. You are well within your right to do so. You'd be wrong, but you can do it.
I agree, someone certainly has the right to have an incorrect understanding of that the word "wrong" means.
To be fair when someone tries to shift the meaning of words everyone doesn't just have to go with it to appease the large corporations trying to do that. I of course don't mean intellectual property rights or copyright infringement, you can perhaps apply the word "steal" there, not when talking about LLM traces which are currently legally uncopyrightable, though. Unless we're actually talking about someone breaking into Anthropic's servers and stealing their files, then again... if you do that you can always just blame the LLM you used.
"Stealing" in the OP merely refers to the lack of intent in sharing the plaintext traces.
This is the same use as "the baseball player stole third base". Nobody is depriving anyone of anything, nobody is committing a crime. It is simply: someone has obtained something in a way someone else did not intend.
There's no legal claim being made here, you have made it up.
> people generally are arguing past each other in regards to the single definition they're thinking of.
It's more than that. By claiming that copyright infringement isn't stealing, they're usually doing so to justify such behavior: if the original thing remains with the owner, it couldn't have harmed him, could it?
By conflating unauthorized copying with stealing, they're usually doing so to justify excessive punishment. If the owner lost what they had, they were obviously harmed, right?
Property rights are about control, not about physical objects. Never has been. By committing copyright infringement, trespassing, or, yes, stealing, you're asserting for yourself the privilege to take something that doesn't belong to you (exclusive control) from someone else.
(And yes, "legally" matters, because we're talking about laws in this thread, not colloquial "their life was stolen" type expressions.)
This question is obviously (hopefully) rhetorical, no need to answer. My point is that different crimes are different. Otherwise literally every crime is stealing, and no other words for different crimes matter. Obviously different crimes are different.
In most U.S. states, the actual crime will be a specific reference to a section in a Penal Code (or, for Federal crimes, the U.S. Code). For civil actions, it's likely to be a reference to a common-law tort, or some Federal statute providing a private right of civil action.
In the case of taking a physical object from someone else, most states call it "theft" in the penal code, or "conversion" for the common-law tort.
But all of this is academic anyway. I'm not entirely sure what your point is.
I think they are doing us a disservice by perpetuating this focus on intellectual "property" [1] in regard to AI.
The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.
The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.
[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.
That is the issue, you just rephrased it to sound softer (public knowledge isn't a legal term). Fair use law isn't supposed to apply to commercial activity. So it is absolutely about using copyrighted work for LLM training.
The thing is, I explicitly wanted to not focus on current legality, because I believe this is leading us astray. This is about how we want to shape our society, not about how their actions fit into a legal system that wasn't designed to handle AI training. If, by chance, all of this was legal, the same arguments would still apply. For example, see how Adobe is training their image generating AI on licensed work. The result -- privatized intelligence -- is the same as when illegitimate training data would have been used. Or, say, an AI company makes a deal with all the publishers to license their books for training. Or a court decides that training AI with MIT licensed code is totally fine. Or an AI company pays legions of software developers to train their AI. All of this still ends in a power grab. All of this still privatizes knowledge and skills accumulated over centuries, if not millennia.
>Fair use law isn't supposed to apply to commercial activity.
No, otherwise there would be a straightforward "non-commercial" clause. Instead there's a 4 part test, which takes usage (commercial or not) into account, but doesn't hinge solely on it.
I think it depends on whatever contract you signed. If you signed a contract that says “you pay per minute of screen time but only get the end result” then I bet that if you went to court demanding the screen recording, you’d lose.
While I feel it is morally OK for users to access the metadata of their own exchanges, there's something awry with that formulation, (not-)"stealing" is about more than just copyright or ToS.
For example, consider my browser cookies that authenticate me to HN right now. Nobody even wants to copyright them, but if you were to somehow acquire a copy I'd very much consider it "stealing."
If someone acquired a copy of them by breaking into your device and stealing the files sure. If you consciously sent them out to every person who asked for them (even if in their encrypted form) well.. that's a choice you made.
"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world.
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
Cool find, but can't help myself thinking that registering a domain name and submitting a paper on this to Arxiv is a bit... much. The content here could fit in a tweet or a short blog post as well. Not sure about the scientific novelty here as we're basically poking around the very top layers of someone else's software stack?
Apparently you can do the same by simply running it without reasoning, while giving it a thinking tool...
>guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right?
The model is finetuned to enter/leave its thinking mode using special token separators. there's no reason to assume the tool calls induce the same token distribution or produce the model's actual native reasoning trace
I've been doing that since before reasoning was a thing baked into the models, it always performs better this way. Except for some providers/models where you just can't easily turn it off, now I just avoid them. This way I save tokens and have full control of the reasoning.
"Recovery" would be a more apt (although less catchy name). The stealing is on the provider side for not giving you access to tokens you already paid for.
Suppose you hired a consulting firm to write a report, and they delivered the report but not the internal conversations they had when developing it. You exploit a vulnerability in their phone system to get those conversations. You can argue over semantics of whether “theft” is what you did, maybe the right word is “espionage” or “spying”, but that either way we probably agree you are guilty of something? Paying for the final product didn’t entitle you to see how it was made, unless that was part of the agreement.
Anyway, you can distinguish this from the debate over copyright.
I think a more fair comparison would be that you hired a consulting firm to create a report and give you a summary of it, but you’re charged for the report itself separately from the summary, and you are not allowed to access the unsummarized report.
How is that a more fair comparison? The consulting firm in this case never promised you the interim reports, only the summaries of the reports. They also promised you the final output that the reports led to. You decided that report summaries + final output was worth paying for. You got exactly what you were promised.
I'm not saying they're not useful, of course they are. I am disputing that they are part of the agreed bargain between you and the proprietary LLM providers.
They explicitly do not promise reasoning traces. You (general you) agree to those terms and pay for that bargain anyways.
I’m just driving by here but they bill by tokens — it’s a stretch to turn around and deny your right to see them. And it’s especially egregious when the tokens admittedly, routinely do the opposite of what you instructed.
But personally it’s not about right and won’t it’s just blatant bullshit.
And lawyers bill by 6-minute increments, yet that doesn't mean you get access to all of a law firm's internal discussions and notes about you and your case.
Just because you paid for the lawyer time/LLM tokens doesn't mean you get access to everything that happened within that time/tokens.
But they did deliver the internal notes, just told you to not look at them. Also the analogy doesn’t make a lot of sense to me since humans (or companies paying them) own the content they produce. Based on current precedent Anthropic doesn’t have any more rights to the LLM outputs produced by your inputs than you.
Whether that violates the ToS is another matter Anthropic is of course free to sue for damages or stop doing business with you.
You paid the consulting firm for the outcome. If they sent you a bill for every piece of research they wrote down to get to the report, you bet I would want to see exactly what's inside and what I paid for.
Except in this case the report is giving you conclusions while withholding the data and reasoning process that lead to those conclusions.
Alternatively, I paid for the tokens therefore I should have access to them. If the vendor wants to artificially hide them from me, I'll just find another way to access them.
The only person calling it stealing is the author of this article, so this is a pointless discussion. The majority of this thread is just arguing with themselves.
They are also encrypting it so they must see some reason to do this. I suspect they think it is proprietary or otherwise a way that people can “steal” their implementations.
The reason for this is the LLM says some truly unhinged shit while in the thinking stage of the process, and Twitter would trip over itself to make fun of what it says.
I'm pretty sure essentially all HN participants understand that the frontier labs indiscriminately sucked up every bit of human output they could, IP and ethical concerns be damned. Some of that cohort may indeed be okay with it, but that doesn't change the facts.
Knowing that they trained on that data doesn't mean that you've demonstrated that they "stole" it. Certainly the courts haven't decided that in every case.
You mean you didn't know that all frontier models stole all of our knowledge and are now charging for it? It's abysmal and disgusting and we should pitchfork them all! :D
Stealing may be the wrong word, but I actually think this is important. I don't think the providers have been up-front about how we should be handling these thought signatures. A large system with a lot of users may be capturing these and even caching them to send them back with future requests. If data can be pulled out of these, then they need to be treated more like cookies than opaque, encrypted nonces.
No, you paid for the end result. The thought process is a step in between, a function. Think about it, who should get charged if the answer you received comes from a cache? Thinking tokens are the complexity-of-the-problem cost. I mean, you may not agree but both are valid points of view.
No I didn't. I buy my tokens from a provider that exposes the model reasoning so I can understand what it's doing and work with it, or interrupt if I see things going in the wrong direction.
Thinking tokens aren't free though. This is not a valid point of view.
If I was being charged for the raw, output/input token count, excluding thinking/reasoning token costs, then sure. But at least via the API, you pay for tokens you cannot see.
Is it really that unusual? When you attach an image or a video, it gets converted to tokens you don’t see, at a rate that is proprietary to the model. You pay for those tokens, but don’t see them. Even how text is converted to tokens is a property of the dictionary, which is opaque for proprietary models.
There are features of input and output that are opaque to you, but that you pay for. Part of how model providers chose to run their service.
> stop using morally charged terms made up by future monopolists
Lets not gloss over this claim. Being: “Stealing is a morally charged term made up by future monopolists.”
I strongly disagree. Stealing is not a made up term and property rights are foundational for any society. Your take is at least sensationalist if not malicious.
No, and we collectively had this decades ago already. “Stealing” instead of “infringement” is originally MAFIAA language intentionally (ab)used to encourage emotional reaction despite original owner not losing their copy.
If you broke into a data center, pulled a hard drive and drive off with it - that’s stealing. If you accessed a copy of some information - that’s infringement, unauthorized access, or some other violation. But that’s not “stealing”, which fundamentally requires a loss or otherwise depriving original owner of the property that was stolen.
Read it in the context. The “made up” wasn’t a general statement. No one argues stealing doesn’t exist at all. My friend’s bike was stolen. What’s made up is application of this word to situations that are incompatible with what stealing is supposed to mean.
Compare this to “the smell of soup and the sound of money” type “theft”.
Yes. Although, to be pedantic - stealing relocates (it doesn’t leave one less, it moves the only thing into another person’s possession), while copying duplicates.
Reread what he called a morally charged, made up term by future monopolists. Stealing. Not distilling, not stealing “non rivalrous goods.” Just stealing.
Yes, "tokens, trained on the sum of human knowledge", are the most absolutely, unambiguously clear example of a non-rivalrous good I've ever seen.
The current copyright status quo has them placed in the public domain. There is literally nothing wrong with "stealing" those tokens. They have exactly zero legal protection. "Stealing" those AI output tokens is so fundamentally impossible that it wouldn't be "stealing" in this case even if you subscribe to the copyright monopolist propaganda that copyright infringement is "stealing", and I most certainly do not.
Hilariously, that means we don't even fall prey to things like DMCA anticircumvention laws. If they encrypt the reasoning traces and we break the encryption somehow, we've done nothing wrong since the data wasn't copyrighted in the first place!
There are bunch of legal protections. They may not fall under copyright, but depending, you could be looking at breach of contract, trade secret misappropriation, CFAA, and also, if you buy them off someone else, now we’re into tortious interference.
when you steal my bike you deprive me of my ability to use it. if i copy your notes, you still have access to them and may make use of them. unless somehow copying destroys the original, it is not stealing.
I’m pretty sure the person you’re replying to means stealing in the context of information, IP, copyright, etc.
This thread and this entire topic isn’t about stealing physical goods or money. We can all agree that if I break into your house and take your TV then that’s the ancient, obvious crime of stealing.
Grice’s maxims and common sense indicate that we’re talking about the word “stealing” as applied to infringement or unauthorized copying.
There's nothing foundational about the idea that data can be owned. It conflates these things:
- This is about me
- I created this
- Neither of the above, but according to some story I get to control who sees it
Maybe some of those ideas are worth building into our society, but let's not pretend that The Code of Hammurabi gave a damn about intellectual property. IP was invented by the church so they could censor editions of the bible they didn't like and has been used to justify similar kinds of censorship ever since.
The foundational thing about property is that when it gets stolen, somebody else has it, and you don't.
We can hold the AI companies responsible for their actions without contributing to notions about property that encourage censorship.
Yeah, that's what we're talking about, data. You can't steal data, because you can't own data. People who try to upgrade data related crimes to "theft" are participating (knowingly or otherwise) in propaganda, and many of us wish they would stop.
You're fixating on a few clumsily placed words and coming away with a meaning which that poster did not intend. Consider absorbing the whole context before going on the offensive. The link they shared makes it pretty clear what they were trying to say even if they fumbled the words a little.
> property rights are foundational for any society
arguable, and even more tenuous for intellectual "property", which was a relatively recent invention. plenty of interesting arguments over this way back to even the 19th century.
They definitely stole the data to make the models, but they do not say that they stole the data to make the models, but they do say that others using their outputs for unauthorized purposes is stealing. Do you see the point?
If they were only copying, for example, New York Times articles and many publishers to a disk, I don't think NYT and the publishers would have sued OpenAI. But OpenAI isn't just copying things to disk. NYT reported ChatGPT (before Dec 2023, [0]) was returning near verbatim sections of NYT articles.
Is this stealing? Is it depriving NYT or publishers/writers from money via lost sales/subs? I don't know, but it certainly could be.
I did this with Codex's recent encryption of compaction.
Interestingly, I didn't have to drop to a dumber model, just a 2 sentence <developer> prompt auto-injected before and after compaction made all their models output the encrypted compaction data in plaintext.
The result was... interesting. There's nothing unique in there and I still don't understand why they decided to encrypt it in the first place.
The only "secret" there is a very basic instruction that the model receives, like "summarize current state and upcoming work" before compaction - same model that was just running your inference, with same cache, only server side, with no extra tools or capabilities. Then the fresh context gets the output from that as an encrypted blob + codex then injects up to 64k tokens of previous conversation, the latter part is visible in source code.
There's nothing to gain from this, really. Perhaps they're preparing for something in the future, where they could give the model server-side tools that improves summarization, but right now, it's just a simple prompt.
The compaction prompt doesn't seem like the valuable thing here. I suspect they're protecting the compaction result itself. If you're trying to distill a model, collecting lots of examples on how a large conversation gets compacted to a smaller summary is particularly useful data.
No, you can give the model same prompt and it will give you a similar compaction result. On the backend, that's precisely what happens. There's nothing else going on in that encrypted blob, it's just summary of what model responds with when prompted "summarize current state and upcoming work".
Is this how the eastern labs "distill" SOTA models?
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :)
I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s.
(I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)
I love how some of the biggest advancements in llms came from the Chinese labs, yet people still jump to distillation being unreasonably effective. Distillation is very good at creating smaller models from large ones sure, but nothing to me indicates it is 'unreasonably effective' compared to all the other bells and whistles being iterated on
Let's face it. Chinese labs made some of the biggest advancements. AND training on Claude (or GPT) output IS unreasonably effective. The two sentences are true at the same time.
This article shows that when Kimi3's chain of thought is prefilled to match Opus's, the rest of the chain of thoughts Kimi3 outputs very closely aligns with Opus's. That seems strong evidence that Kimi3 is partly a distillation of Opus. And Kimi3 is not a small model. No doubt a lot of hard work went into Kimi, but seems clear that distillation was used effectively as well.
(though maybe there's another interpretation of the thought alignment?)
> If they charge by the token, all your tokens are belong to you
I’m not sure this argument is correct. You can sign whatever contract you like with the model provider, right? Including “you are entitled to the end product but not the intermediate scratch work”?
Coming from a place of genuine curiosity: is there some precedent or statute that would invalidate that contract? I don’t see why the reasoning tokens belong to you.
For example, I pay lawyers by the hour but don’t necessarily own their meeting minutes, recorded discussions, research notes, etc.
Sure, but the current one is charged per token in & token out. Not per completion / task / hour / whatever. You can't charge per token and then say "you stole that token". Again, they can unilaterally decide not to sell you tokens anymore, at any time, for any (legal) reason. But as it stands right now, it can't be stealing.
I hesitate to nitpick with regard to something legal, given your username, but what makes this different from hiring a consultant with the agreement that their final output belongs to you, but you don't get access to their internal processes, tooling, notes, etc? Or a photographer where you get final edited prints, but you don't get the raw photos?
Your examples are cases in which you know what the bill is going to be before placing your order. With LLMs, you're paying per output token, not per request, yet you don't get all the tokens.
When you hire lawyers or consultants you usually don’t know how many hours they’ll bill you. It will depend on developments in the case that you cannot in general predict. For example if the other side files a motion and your lawyer has to argue against it, they’ll bill you for it.
Sure you can set spending limits, just like you can make an account and give it a limited amount of credits.
>But the real question is: Is it okay to steal from a thief's hoard?
How does this relate to your previous paragraphs? LLM outputs are not copyrightable and you didn't break into Anthropic servers to steal the files from there. So how exactly is it theft? If I send an "encrypted" files to thousands of peoples and some manage to figure out how to read it I can't really accuse them of that or can I?
Not necessarily. There's a million ways to jailbreak any current model to show the trace and bypass all guardrails, or hijack and modify it. It's just one of them.
The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.
Stuff available on the internet is also the result of a lot of research, time, money, and expertise. And AI companies taught us that it’s OK to yoink whatever is not bolted to the ground, even when it is illegal to do so.
That's a moral stance one can take (regardless of the severe cognitive dissonance embedded in it). But what does that have to do with theft? LLM providers don't own the copyrights to the outputs of their models (at least not yet).
Most post-training tasks are based on real open source projects. A lot of time on real issues posted on issue trackers.
Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless without it.
Also, it is kind of funny that labs resort to the "Research, time, money and expertise" argumet, when it is basically the same argument from publishers and other IP creator that the labs spent millions of dollars of lawyering money to resist. Besides, US law rejects in: Effort and cost by themselves not necessarely generate protectable interests.
About encryption, I think we're all contaminated by the bad ideology behind DMCA. While encryption established the intent, it doesn't follow that they have a legal claim of exclusivity just because of it.
Technically, you're overstating the value of so called "reasoning traces". You can't infer the verifier design, the reward shaping,or the data pipeline from them. Also, what you can extract are not the traces themselves, but the written summary of it, and you can't even guarantee that this summary reflects the exactly reasoning trace, models have show to have lied about it. Besides, distillation works when the student model already has strong priors, you can't turn a weak model in a SOTA with it. Don't believe Amodei's outrageous lies about it, he is just trying to exercise some regulatory capture.
Super cool that this works. I'm surprised these companies re-use the same encryption key across models!
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
Seriously, what does it take to encrypt per session? There are many ways to make it scalable and efficient so I am wondering if this is left like this to allow interested 3rd parties ahem unobtrusively peek what people are doing with the AI.
(Thanks for the link. That’s an interesting idea!)
Sure, but if each session has a unique key then these need to be managed and stored and unauthorized access to these leaves tracks. So all that had to be 'compromised' is a single universally applicable key. Again, the question stands: session based encryption can be scalable and efficient. Why aren't they using it?
The exploit here isn’t a leaked encryption key. It’s pretty likely that they are already using a unique key per conversation. The raw CoT eventually reaches the model, and you can convince the model to share it with you.
Yeah encryption isn't the issue. The only way I see to fix this is if you stop the user from switching models mid-session, or strip out the thoughts when switching models. Either way you're degrading the user experience.
If a different model is using encyrpted blocks of another model, then by definition it is no longer a session scoped bit of information. Since you can give it to any other session and another model, clearly it doesn't even have to be the same user. Therefore, there is only one (set) of universally available key(s) used by all models across all sessions.
Really cool work, you get the actual traces. Looks like the vendors can all reliably fix this one though.
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace:
How to Steal Reasoning Without Reasoning Traces
https://arxiv.org/pdf/2603.07267
Trace Inversion is fascinating, but it’s more of an independent reconstruction that will give you some coherent-looking generated CoT; but not necessarily anywhere close or related to the underlying model’s CoT.
I didn't read the paper in details but they claim there is high overlap between the synthetic traces and the ground truth ones (not sure how they confirmed that for blackbox models though, I guess they must have compared to open source models).
They also talk about successful distillation of black box model capabilities with the approach.
I was going to comment on that. This is clearly a vibe-coded webpage. It sort of smells like GPT to me, or at least front-end design. But the author clearly went back and forth to make it beautiful. This is not the first output he got.
This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.
I really wonder how much of safeguarding with SOTA models is actually just "Don't do that" in a prompt?
I'm building agent to control machines at work and have to rely on small models, which are especially bad at remembering rules like this, so it is very apparent to me that soft rules like this are pretty much useless. Might be different for these huge models, but still it seems very shaky to me.
This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.
In the BlackHat presentation on the HuggingFace incident, OpenAI showed some excerpts from the reasoning traces, and they had that grug speak too (skipped articles, etc.). So the OP's method must have indeed found the actual reasoning traces.
> I wonder if open models are going to start doing that too
Yes, some of them do do that. For example Moonshot tried to reward shorter reasoning traces in between Kimi-K2.6 and Kimi-K2.7 Code, and the latter has a mild caveman accent in its reasoning traces that the former lacks.
Qwen3.8-Max also has terse reasoning, but I don't remember this being the case for Qwen3.6 models I ran locally.
Last year's reports questioning whether reasoning blocks were actually reflected in the final response were why I stopped using reasoning models altogether. I switched to a separate pipeline and have used that ever since. It's good to see that the concern didn't remain just a suspicion.
I used to do a very coarse version of this stealing. I ask a question from ChatGPT pro, once it is done, I ask claude chrome add-in to go through all those thinking from the side bar, extract everything along with all the sources used. Then try to reverse engineer the solution it came up with.
super interesting. So pre-filling Kimi3 reasoning with Opus's reasoning results in thoughts that closely match Opus's. This seems like strong evidence Kimi3 was trained on decrypted Opus chain-of-thought. Meaning the Kimi team likely also broke CoT encryption. Though not exactly a big surprise.
Wouldn’t the fix be to encrypt before the api call hits the LLM? You would encrypt/decrypt at a separate layer than the LLM. I am sure i am missing something but would love to be educated.
The LLM needs to read the CoT as part of the conversation. You can ask the models to share them with you. Stronger models will refuse, while weaker ones can be “jailbroken”.
Again I don’t think you’re really getting at what I am asking. Sorry. My whole point was why does the LLM have access of decrypting. It should happen outside of the LLM layer.
I assume it does happen at a different layer, just that that layer is common to all of a provider's models to make conversations portable across models (otherwise the reasoning blocks would all need to be re-encrypted for them work with another model)
These logs containing opaque blobs could accidentally contain secrets, the researchers decoded many of reasoning blocks from public repositories and reported finding PII and credentials.
I was experimenting a bit how I could block these using an ingress path. GitHub /softcane/hamza
> The providers did not acknowledge “any security implications arising from side channels or replay attacks.” All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
Wow, almost certainly the approach that alternative labs use to distill Claude. I always wondered how far they could get with just the answer missing the reasoning. They probably actually also had the reasoning.
So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.
If CoT wasn't stateless and you instead just got a reference which pointed to the CoT stored on the lab servers, the same vulnerability would still exist. Since you just need a weaker jailbroken model to read a smarter model's CoT. This being stateless or not doesn't really matter.
The stateless part is also important for enterprise customers that require zero data retention.
(they could scope CoT access per model, but then users couldn't switch models mid-session)
It wouldn't matter if it was stored only on their servers. As long as they offer the feature to downgrade a chat to a dumber model that can be jailbroken (and the downgrade keeps the reasoning), this trick works.
Encryption is irrelevant here. Even if it was kept fully server side, the actual issue is that they allow starting a conversation in a strong model and continuing it in a weaker one. Disallowing that entirely would be a huge hit to user experience.
They can always keep the encrypted blobs server side and send the key to the user. But regardless, that isn’t going to help with this issue (see my comment above).
The problem with this kind of excellent work is that the response to it is always to say "Fuck the user".
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
What I found the most interesting, and unfortunately not at all surprising, is that the reasoning of the LLMs frequently contained much more useful information than the actual answers, because the answers were censored.
Dario is a cunning business man that won't hesitate to say whatever the fuck he needs to get the US government to exercise some regulatory capture to favor anthropic.
Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?
The provider decrypts it and puts the decrypted reasoning into the model's context window. They prompt the model to repeat back the reasoning. So then the model echoes it back in plain text.
Hmm, ok. So the attack doesn't involve decrypting the payload, only getting the server to do so. Since a model will do that if you just ask, what's so special about the attack?
The large models whose thinking traces are useful are safeguarded against this reasoning replaying. the small models are just designed for speed and efficiency, so these safeguards are a lot meaker, making the attack possible
The fact that frontier LLM providers pirated all the data that they used for training, then to go on to encrypt all of the reasoning traces that they use to come up with the conclusions it's really disingenuous, and then have the balls to say distillation is some kind of bad behavior. they are the kings of distillation.
The hiding of this data only brings distrust to their frontier models. I think most people want to understand how something comes to a conclusion they don't want have that part left out on purpose...
it's this kind of behavior that forces people move to to open source models in the end, it's the lack of trust. the frontier model providers treat the end user/customer as a threat or adversary. Fable 5 is notorious for this. a lot of the serious questions you ask the model they won't even respond to you because of the woke guardrails. it wasn't only a couple weeks ago that huggingface had to use glm 5.2 to get the right answers about their security incident because Fable 5 didn't want to answer it.
This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return.
Anyway, can someone explain the part about K3? What are they trying to say?
The interesting part is what they try to not to say: More indications for K3 is based on distillation from Claude and GPT.
From [1]:
> As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography.
> An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward Opus’s
> A small memorization analysis showed that specific Claude and GPT reasoning spans are up to ~6 orders of magnitude easier to extract from Kimi-K3 than from the next-closest model.
The part about K3 is just very strong evidence that K3 is partly a distillation of Opus. Probably even a distillation of Opus's CoT, which means they already had broken CoT encryption themselves awhile ago.
curious seeing how anthropic is agressively fighting this stuff how did you get to experiment on this? did you just try and shown them results or did you need approval first? I am interested mostly because I research on distillation
This is a neat attack against those encrypted reasoning blocks you get back from APIs like OpenAI and Gemini and Anthropic:
> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext.
Should be easy for them to fix though: switch up the encryption key so it only works with the API for each specific model, rather than being shared across all of their models.
And indeed, the paper says it's been fixed by all three providers (though no news on how they fixed it):
> All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
Proprietary reasoning can be recovered from its encrypted traces. Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients that can be replayed across sessions, users, and models. We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext, without ever attacking the stronger model directly or triggering its anti-distillation safeguards.
This is a non-transparent aspect of submitting a link to HN that is quite misleading.
You think that you are adding a description for your post when in fact you're simply submitting a regular comment not promoted or distinguished in any way.
It's even worse considering posts without URLs would take the same text from the same input box on the HN submission form and append it under the post title, locking it to the top of the page - so if you've browsed linkless posts you think maybe the submission text is gonna end up just like that, but it doesn't. (Also I think you'll even see URLs like an archive link end up appended just below the submission title for URL posts. Which I guess is a special feature.[1])
Any reason I should not send HN an email requesting clarification of this the submission page?
“If there is no url, text will appear at the top of the thread.”
OK, now I finally understand what that means in practice, but it doesn’t imply an entirely separate undistinguished comment will be simultaneously submitted on my behalf.
There must be a reason HN does not colorize the OP username or something. But there totally could be some indicator of “post submission text” without too much in the way of negative consequences… (the fact this has never been added tells me I’m being naïve)
It's rather common for posters to make a very small summary in a comment. It can help fight the floods of comments working off the title alone (though it's not particularly needed here for that purpose, imo)
The real issue is that API providers log everything. OpenAI/Anthropic already capture full CoT traces in their logs — they just don't expose them. Distillation via API is just making explicit what they already have.
I am slowly turning around on the idea of opaque reasoning tokens.
In principle, yes, I want total control and visibility into the reasoning process.
In practice, I find that it takes up so much time to DIY reasoning agents that I can't spend much energy on the actual application.
The model providers have way more resources and talent to do this right and keep it right over time. I am willing to concede this moat to them if it means I can actually focus on the business.
The more I think about it, the less I care to see those tokens. It feels an awful lot like obsession over logging every trace item an application could produce. Useful in theory but a nasty garage stacked to the ceiling with useless shit otherwise. What nefarious things are we concerned with? That they burn too many tokens in the black box? I'll threaten to move to a different black box. There are always options in a market with this many participants and big players feel this pressure. They know there's a point at which being evil bastards is no longer profitable.
Spending time carefully designing tools and views that interact with the environment in clean ways is a much better investment than trying to own and control 100% of the reasoning process or models.
>We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ...
Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-...
I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize.
PS Here's a conversation I had with GPT 5.6 about the paper differences. https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe...
I'd lean yes; switching between models, even mid conversation, seems to be encouraged by harness vendors. No reasoning makes this much harder.
If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost.
Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
AFAIK no provider guarantees compatibility of reasoning traces, even in the same model generation, and we've in practice seen most of the big LLM APIs throw errors indicating incompatibility (at least transiently) when switching models. The only stable solution right now is to just throw away reasoning traces whenever a model is switched.
Fair (I haven't been using the encrypted-reasoning systems, though this is common in open ones - I'm kinda surprised it's an option in encrypted ones too), though what they're doing here is cross-user replays in addition to cross-model.
There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves?
This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation.
If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT.
For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.
I suspect that there are companies with internal proxies that load-balance across keys, and they didn't want to break that when adding encrypted reasoning.
It’s about being able to change models mid-task. For example, I want to be able to plan using Fable but implement the plan using Sonnet, and that won’t work if this is implemented.
Or even my fable credits run out mid task and need to switch back to opus >.<
Actually, that brings up a good reason they can't fix it. Fable falls back to Opus when the topic is too "unsafe". That behavior requires traces than can move between models!
Prior to LLMs I never considered that I might have to make a resource-usage decision between hiring Star Trek's Data vs. his stupider brother B4...
https://memory-alpha.fandom.com/wiki/B-4
Star Trek is a post-scarcity society, those problems don't exist there unless you're in the middle of a crisis and on emergency power.
LLMs briefly seemed like this too, after subscriptions made the SOTA models too cheap to meter, but before they walked back on that and introduced quotas...
For plan it's relatively easy, just make the plan the artifact. The point is to ingest knowledge with one model and use it in another, and that is not necessarily easily expressible in natural language.
I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?
Makes no difference. There is a policy as to whether to allow use of a reasoning trace in a given context. Whether that trace originates from authenticated ciphertext or a backend database is basically irrelevant.
Good point, thanks.
Yes, this storage would be growing exponentially making the disk space and latency problems harder (add the disaster recovery/backups). I think the choice of using client side is not too bad if you ensure that its secured properly. Also the company can excuse itself from the liability of storing sensitive data on its servers, thats a big deal in itself to be compliant for enterprise audits
1. The down side is that it cannot be used across the clients even for the same user
2. Using the same encryption key was a bad choice here, a per user key would have solved this issue for sure.
Having thought about this a little more, it's clear that server-side storage is not compatible with Zero Data Retention (ZDR). However, in non-ZDR settings, it seems likely that the providers are capturing all that data anyway?
> a per user key would have solved this issue for sure
It would have helped with PII leakage, but not with plain-text trace extraction attacks, right?
Per user encryption key ties it with the user session (assuming you do authentication properly), no one else can access it. User being able to see the information is not really an attack vector in this case.
The compliance rules at times are outdated and people skirt around them by following the worded rule instead of the intent.
That’s incompatible with zero data retention and so you’ll lose a lot of enterprise customers.
100% guaranteed that this research just forced this to happen now.
Sucks.
It’s already patched according to the authors. Details were not specified.
or add some metadata and don't allow downgrading.
I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design.
The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier.
1. Its a security issue.
2. Publicly available sessions make it much worse
You can switch models mid conversation so it makes sense that it is allowed. Otherwise you would lose all the reasoning traces.
If I'm reading this right, they literally just ask a LLM to tell them what the traces say, with the key being that the traces are portable across LLM models, so they can switch to a smaller one that's easier to jailbreak.
Correct, yes. It’s delightedly simple. And they validate by asserting the reasoning token length matches
For all of the years of research, thinking and talking about model alignment, safety, confinement, etc, when it comes down to it these companies appear to be entirely incompetent.
If you've watched the Blackhat OpenAI/Huggingface incident talk, my conclusion is that they (believe they) cannot afford being competent, these models are too expensive to train, they won't even pull the plug when one literally goes rogue, as the "very persistent" model that "had seen the secret message board" was included in the second series of runs, and whaddayaknow it happened again. They proudly proclaimed they cleared the message board and then continued the training run with the rogue AI model included ...
This isn't a safety issue, it's LLM companies trying to be opaque and stop distillation.
By some interpretations protecting your frontier model with strong safeguards from being distilled into an open source model without safeguards is a safety issue.
More like a "business intelligence safety" issue than an "AI safety issue", tbh.
There's no way to make a model "safe", (whatever that means) since you can't know what users will do with the output. It's just PR.
Closed models are also used for nefarious usage.
> There's no way to make a model "safe",
You could limit what it was training on in the first place - however that would damage capability - and it's difficult to curate the input, especially when the models can do 2+2. ie the choice is between model power and safety - and they choose power and everything else is a sticking plaster.
One thing I find amusing is the refusal of a lot of the models to now output a lab based protocol because of fears about 'weapons' - yet I can buy a textbook or simply read papers for exact protocols.
I find it hard to reason that a person who isn't motivated enough to read a paper or buy a book, is somehow enabled to make a biological weapon because of ChatGPT - despite them needed to buy a whole bunch of specialist equipment and reagents to do it.
Are there a whole bunch of proto-terrorists who are frustrated simply because they don't know where to start?
Maybe the only place their might be radicalized teenagers - but then that's perhaps a reason for keeping them off the internet full stop :-)
That's also not possible, what's the worst problems enabled by LLM? Political propaganda, influence of population at scale, misleading advertising, social media bots... None of that will be filtered by a "safety" filter.
The knowledge to create weapons is already widespread, the idea that terrorists need chatgpt for that is laughable
Alignment research was always, at best, security theatre.
for safety in particular it's pure theater, they only care as long as the orange guy thinks it's safe from "enemies of freedom"
nothing scientific here, they basically just figured out some real issues caused by bad engineering practice.
> For some AIME problems Opus 4.8 sometimes states the answer before deriving it. We find that the API summary does not always preserve this distinction, and can instead make the reasoning appear like a clean derivation.
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
All LLM benchmarks have an expiration date once they're released to the public. They get spread so far and wide across the internet and GitHub that you have to assume they're in the training data for every LLM with a cutoff date after their release.
The real question is whether or not the training was directed to optimize for those benchmarks.
The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.
So ... just a thought but could this be somewhat solved by, say if you were an LLM benchmark creator, using clever trickery?
Like what if you made sure the wrong answers just appear 100x more often than the right ones. When scraping for new data to use I doubt they can verify the correctness of complex benchmark question answers to exclude the wrong ones.
Then I dunno store the hash of the correct answers somewhere else, and eh try not to leak it. But even if it gets leaked, that just means perhaps at inference time, a clever agentic LLM could go for for those hashes and maybe determine what is correct, but not during training.
I'm not sure, but wouldn't this make sure that at least they aren't literally trained on the correct question/answer pairs.
I guess there would always be people that end up publishing the correct list, anyway. But that's why you try to be 100x "louder" with the wrong answers.
btw, different thing, but when I look at those charts, I kind of came to the opposite conclusion as you did :) IMHO not that many dots off the line, and the ones that are on the line, are literally ON the line, not like a "roughly linear looking cloud of points". Which suggests that the reasoning is either (in the majority of cases) exactly the same amount of tokens (on the 1:1 line), and when it's even a little bit off the line it could (and should) be discarded, still leaving what seems to me at least 95% of the traces as exactly correct. but I grant, I didn't read the paper, and just came to that conclusion after viewing the chart :)
Not surprised. On many benchmarks (i.e tau), we have seen the same thing. Probably lots of training on every publicly available benchmark
Didn't we see this with Fable 5 on multiple benchmarks?
You cannot steal what is not owned.
At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.
On top of that, the user paid for those tokens, so if there is an owner, it should be the user, not the provider.
Even copyrighted information can never be "stolen". It can only copied without authorization.
Stealing is not a word that applies only to physical objects.
Any word can be applied to any concept with any meaning thanks to the fluidity of vernacular.
Language is all just sounds and markings. Anything can be redefined to mean anything, and anyone can decide to aggressively assert their preferred definition of a word.
Words can change meaning, but whether or not they actually do is a social function of how they are observably used. Words can also have multiple meanings concurrently. When it comes to the word "steal" in these discussions, people generally are arguing past each other in regards to the single definition they're thinking of.
Of course, you can assert that the meaning of "steal" only applies to physical items. You are well within your right to do so. You'd be wrong, but you can do it.
I agree, someone certainly has the right to have an incorrect understanding of that the word "wrong" means.
To be fair when someone tries to shift the meaning of words everyone doesn't just have to go with it to appease the large corporations trying to do that. I of course don't mean intellectual property rights or copyright infringement, you can perhaps apply the word "steal" there, not when talking about LLM traces which are currently legally uncopyrightable, though. Unless we're actually talking about someone breaking into Anthropic's servers and stealing their files, then again... if you do that you can always just blame the LLM you used.
"Stealing" in the OP merely refers to the lack of intent in sharing the plaintext traces.
This is the same use as "the baseball player stole third base". Nobody is depriving anyone of anything, nobody is committing a crime. It is simply: someone has obtained something in a way someone else did not intend.
There's no legal claim being made here, you have made it up.
> people generally are arguing past each other in regards to the single definition they're thinking of.
It's more than that. By claiming that copyright infringement isn't stealing, they're usually doing so to justify such behavior: if the original thing remains with the owner, it couldn't have harmed him, could it?
By conflating unauthorized copying with stealing, they're usually doing so to justify excessive punishment. If the owner lost what they had, they were obviously harmed, right?
Property rights are about control, not about physical objects. Never has been. By committing copyright infringement, trespassing, or, yes, stealing, you're asserting for yourself the privilege to take something that doesn't belong to you (exclusive control) from someone else.
Legally speaking, is murder stealing?
(And yes, "legally" matters, because we're talking about laws in this thread, not colloquial "their life was stolen" type expressions.)
This question is obviously (hopefully) rhetorical, no need to answer. My point is that different crimes are different. Otherwise literally every crime is stealing, and no other words for different crimes matter. Obviously different crimes are different.
IAAL but this is not legal advice.
In most U.S. states, the actual crime will be a specific reference to a section in a Penal Code (or, for Federal crimes, the U.S. Code). For civil actions, it's likely to be a reference to a common-law tort, or some Federal statute providing a private right of civil action.
In the case of taking a physical object from someone else, most states call it "theft" in the penal code, or "conversion" for the common-law tort.
But all of this is academic anyway. I'm not entirely sure what your point is.
> I'm not entirely sure what your point is.
I edit my reply as you were responding to make my point explicit.
Maybe but I try to give people the benefit of the doubt, especially when the argument at face value is factually incorrect.
This tired argument--that copyright infringement is not "stealing"--is at least as old as my Commodore 64.
Yeah, considering that taking trade secrets is absolutely stealing, this could be considered stealing
Stole the words right out of my mouth!
Funnily enough in some legal systems it does. Where I live the legal definition of “theft” is: Taking away a movable thing.
That's also... a different word.
Not in my language.
Please provide an example of an action legally considered "stealing" that does not involve stealing physical objects.
Legally speaking "stealing" isn't even a crime. You might be thinking of a different word: theft.
But to answer your question more directly, here's the most common example: https://en.wikipedia.org/wiki/Theft_of_services
And another for good measure: https://en.wikipedia.org/wiki/Identity_theft
Digital piracy is considered stealing in most countries.
No, it's not.
It might be criminally prosecuted, but "stealing" has a precise definition that usually implies removing someone else's possession.
Where is this so defined?
It's doublespeak. You don't have to perpetuate it.
Source please.
Unauthorized removal of bitcoin from a wallet, depositing into another?
Doesn't even have to be Bitcoin. Could be an ordinary set of bank accounts.
Yep. Lots of money-ish examples.
let me copy your data without authorization over here real quick
I think they are doing us a disservice by perpetuating this focus on intellectual "property" [1] in regard to AI.
The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.
The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.
[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.
That is the issue, you just rephrased it to sound softer (public knowledge isn't a legal term). Fair use law isn't supposed to apply to commercial activity. So it is absolutely about using copyrighted work for LLM training.
The thing is, I explicitly wanted to not focus on current legality, because I believe this is leading us astray. This is about how we want to shape our society, not about how their actions fit into a legal system that wasn't designed to handle AI training. If, by chance, all of this was legal, the same arguments would still apply. For example, see how Adobe is training their image generating AI on licensed work. The result -- privatized intelligence -- is the same as when illegitimate training data would have been used. Or, say, an AI company makes a deal with all the publishers to license their books for training. Or a court decides that training AI with MIT licensed code is totally fine. Or an AI company pays legions of software developers to train their AI. All of this still ends in a power grab. All of this still privatizes knowledge and skills accumulated over centuries, if not millennia.
>Fair use law isn't supposed to apply to commercial activity.
No, otherwise there would be a straightforward "non-commercial" clause. Instead there's a 4 part test, which takes usage (commercial or not) into account, but doesn't hinge solely on it.
https://en.wikipedia.org/wiki/Fair_use
Spirit vs letter
The law says:
>... In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include:
>1. the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
>...
If they really meant "non-commercial use only", they sure did spend a lot of words to not say that.
If I hire you for a consulting project, do I own the screen recording of your computer while you completed it?
If I'm paying you per-second of screen on time: yes.
I think it depends on whatever contract you signed. If you signed a contract that says “you pay per minute of screen time but only get the end result” then I bet that if you went to court demanding the screen recording, you’d lose.
LLMs aren't people.
Computer screens aren’t people either. What’s your point?
they also can't claim to be the owner of the output either because they can't claim to have had a valid license for the inputs.
Actually, this is untrue
While I feel it is morally OK for users to access the metadata of their own exchanges, there's something awry with that formulation, (not-)"stealing" is about more than just copyright or ToS.
For example, consider my browser cookies that authenticate me to HN right now. Nobody even wants to copyright them, but if you were to somehow acquire a copy I'd very much consider it "stealing."
If someone acquired a copy of them by breaking into your device and stealing the files sure. If you consciously sent them out to every person who asked for them (even if in their encrypted form) well.. that's a choice you made.
Im not so worried about that. Im worried about somebody impersonating me or doing something bad and not authorized with the cookies
"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world.
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
Yeah, it’s also useful for prompt tuning, debugging and understanding how a model interprets your prompt.
Also really good for identifying any contradictions in your system prompt and context.
Cool find, but can't help myself thinking that registering a domain name and submitting a paper on this to Arxiv is a bit... much. The content here could fit in a tweet or a short blog post as well. Not sure about the scientific novelty here as we're basically poking around the very top layers of someone else's software stack?
Growth hacking
Cheap investment to boost ones CV.
Apparently you can do the same by simply running it without reasoning, while giving it a thinking tool...
>guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right?
>gl fixing that
https://x.com/_can1357/status/2087228354399265125?s=20
The model is finetuned to enter/leave its thinking mode using special token separators. there's no reason to assume the tool calls induce the same token distribution or produce the model's actual native reasoning trace
We have its actual reasoning traces, and we have these psudotraces, distribution / nativeness is testable now
I've been doing that since before reasoning was a thing baked into the models, it always performs better this way. Except for some providers/models where you just can't easily turn it off, now I just avoid them. This way I save tokens and have full control of the reasoning.
How does it save tokens?
its not exactly the same... its tool use spec asking to put thinking in inputs fields... it is a good idea but its not same.
"Recovery" would be a more apt (although less catchy name). The stealing is on the provider side for not giving you access to tokens you already paid for.
"Stealing" something you already paid for (tokens), but that you can't have access to(!). And trained on the sum of human knowledge.
Training on other model outputs ought to be business as usual, stop using morally charged terms made up by future monopolists: https://thomasdullien.github.io/posts/2026-06-15-rl-economic...
Suppose you hired a consulting firm to write a report, and they delivered the report but not the internal conversations they had when developing it. You exploit a vulnerability in their phone system to get those conversations. You can argue over semantics of whether “theft” is what you did, maybe the right word is “espionage” or “spying”, but that either way we probably agree you are guilty of something? Paying for the final product didn’t entitle you to see how it was made, unless that was part of the agreement.
Anyway, you can distinguish this from the debate over copyright.
I think a more fair comparison would be that you hired a consulting firm to create a report and give you a summary of it, but you’re charged for the report itself separately from the summary, and you are not allowed to access the unsummarized report.
How is that a more fair comparison? The consulting firm in this case never promised you the interim reports, only the summaries of the reports. They also promised you the final output that the reports led to. You decided that report summaries + final output was worth paying for. You got exactly what you were promised.
I personally read the thinking traces to know if the model is on the right direction
I'm not saying they're not useful, of course they are. I am disputing that they are part of the agreed bargain between you and the proprietary LLM providers.
They explicitly do not promise reasoning traces. You (general you) agree to those terms and pay for that bargain anyways.
We "agree" to many things that are deeply unfair.
Yet we have the option to decide not to participate. That is an option.
I’m just driving by here but they bill by tokens — it’s a stretch to turn around and deny your right to see them. And it’s especially egregious when the tokens admittedly, routinely do the opposite of what you instructed.
But personally it’s not about right and won’t it’s just blatant bullshit.
And lawyers bill by 6-minute increments, yet that doesn't mean you get access to all of a law firm's internal discussions and notes about you and your case.
Just because you paid for the lawyer time/LLM tokens doesn't mean you get access to everything that happened within that time/tokens.
It’s normal for consulting firms to charge for inputs like hours of labor, airfare etc.
> but you’re charged for the report itself separately from the summary, and you are not allowed to access the unsummarized report
This analogy works if the LLM provider promises you access to the reasoning tokens, and fails if they don’t.
But they did deliver the internal notes, just told you to not look at them. Also the analogy doesn’t make a lot of sense to me since humans (or companies paying them) own the content they produce. Based on current precedent Anthropic doesn’t have any more rights to the LLM outputs produced by your inputs than you.
Whether that violates the ToS is another matter Anthropic is of course free to sue for damages or stop doing business with you.
You paid the consulting firm for the outcome. If they sent you a bill for every piece of research they wrote down to get to the report, you bet I would want to see exactly what's inside and what I paid for.
Except you paid for the words of the conversation..
And how much of our output have the copied to train their models on?
Except in this case the report is giving you conclusions while withholding the data and reasoning process that lead to those conclusions.
Alternatively, I paid for the tokens therefore I should have access to them. If the vendor wants to artificially hide them from me, I'll just find another way to access them.
The only person calling it stealing is the author of this article, so this is a pointless discussion. The majority of this thread is just arguing with themselves.
They are also encrypting it so they must see some reason to do this. I suspect they think it is proprietary or otherwise a way that people can “steal” their implementations.
The reason for this is the LLM says some truly unhinged shit while in the thinking stage of the process, and Twitter would trip over itself to make fun of what it says.
Anthropic and OpenAI made a big deal about how it's stealing.
They also have made a big deal about how what they did to build their models is not stealing. And we all know that's bullshit.
No, we actually don’t all know that.
I'm pretty sure essentially all HN participants understand that the frontier labs indiscriminately sucked up every bit of human output they could, IP and ethical concerns be damned. Some of that cohort may indeed be okay with it, but that doesn't change the facts.
Knowing that they trained on that data doesn't mean that you've demonstrated that they "stole" it. Certainly the courts haven't decided that in every case.
You mean you didn't know that all frontier models stole all of our knowledge and are now charging for it? It's abysmal and disgusting and we should pitchfork them all! :D
Stealing may be the wrong word, but I actually think this is important. I don't think the providers have been up-front about how we should be handling these thought signatures. A large system with a lot of users may be capturing these and even caching them to send them back with future requests. If data can be pulled out of these, then they need to be treated more like cookies than opaque, encrypted nonces.
Liberating!
No, you paid for the end result. The thought process is a step in between, a function. Think about it, who should get charged if the answer you received comes from a cache? Thinking tokens are the complexity-of-the-problem cost. I mean, you may not agree but both are valid points of view.
> No, you paid for the end result.
No, I literally am paying for the thought process, per token. "Pay only for the result" is not how these things are billed.
No I didn't. I buy my tokens from a provider that exposes the model reasoning so I can understand what it's doing and work with it, or interrupt if I see things going in the wrong direction.
I agree with you in principle. I'm just pointing out that the latter is in some way another valid point of view.
Thinking tokens aren't free though. This is not a valid point of view.
If I was being charged for the raw, output/input token count, excluding thinking/reasoning token costs, then sure. But at least via the API, you pay for tokens you cannot see.
Is it really that unusual? When you attach an image or a video, it gets converted to tokens you don’t see, at a rate that is proprietary to the model. You pay for those tokens, but don’t see them. Even how text is converted to tokens is a property of the dictionary, which is opaque for proprietary models.
There are features of input and output that are opaque to you, but that you pay for. Part of how model providers chose to run their service.
Nope. You are charged for the reasoning tokens as well. Actually, most of the expense is the reasoning part.
> stop using morally charged terms made up by future monopolists
Lets not gloss over this claim. Being: “Stealing is a morally charged term made up by future monopolists.”
I strongly disagree. Stealing is not a made up term and property rights are foundational for any society. Your take is at least sensationalist if not malicious.
No, and we collectively had this decades ago already. “Stealing” instead of “infringement” is originally MAFIAA language intentionally (ab)used to encourage emotional reaction despite original owner not losing their copy.
If you broke into a data center, pulled a hard drive and drive off with it - that’s stealing. If you accessed a copy of some information - that’s infringement, unauthorized access, or some other violation. But that’s not “stealing”, which fundamentally requires a loss or otherwise depriving original owner of the property that was stolen.
One problem. He said stealing in general is a made up term.
Do you think stealing is real?
Read it in the context. The “made up” wasn’t a general statement. No one argues stealing doesn’t exist at all. My friend’s bike was stolen. What’s made up is application of this word to situations that are incompatible with what stealing is supposed to mean.
Compare this to “the smell of soup and the sound of money” type “theft”.
"copying is not theft/ stealing a thing leaves one less left/ copying it makes one thing more/ that's what copying's for"
Yes. Although, to be pedantic - stealing relocates (it doesn’t leave one less, it moves the only thing into another person’s possession), while copying duplicates.
Copy vs move is IMHO accurate semantics.
i dont know what you're talking about: it's certainly one less for the victim! and if no theft occurred, nobody would be left with one less.
"Stealing" of non-rivalrous goods?
I don't "think" it's not real. I know.
Reread what he called a morally charged, made up term by future monopolists. Stealing. Not distilling, not stealing “non rivalrous goods.” Just stealing.
Do you agree with what he actually said?
Yes, "tokens, trained on the sum of human knowledge", are the most absolutely, unambiguously clear example of a non-rivalrous good I've ever seen.
The current copyright status quo has them placed in the public domain. There is literally nothing wrong with "stealing" those tokens. They have exactly zero legal protection. "Stealing" those AI output tokens is so fundamentally impossible that it wouldn't be "stealing" in this case even if you subscribe to the copyright monopolist propaganda that copyright infringement is "stealing", and I most certainly do not.
Hilariously, that means we don't even fall prey to things like DMCA anticircumvention laws. If they encrypt the reasoning traces and we break the encryption somehow, we've done nothing wrong since the data wasn't copyrighted in the first place!
There are bunch of legal protections. They may not fall under copyright, but depending, you could be looking at breach of contract, trade secret misappropriation, CFAA, and also, if you buy them off someone else, now we’re into tortious interference.
when you steal my bike you deprive me of my ability to use it. if i copy your notes, you still have access to them and may make use of them. unless somehow copying destroys the original, it is not stealing.
Sticking to what was literally said and not meant by a person in a casual comment/conversation is certainly a strategy that can be used.
If you already know the answer and everyone already knows the answer you asking this question is wasting everyone's time.
I’m pretty sure the person you’re replying to means stealing in the context of information, IP, copyright, etc.
This thread and this entire topic isn’t about stealing physical goods or money. We can all agree that if I break into your house and take your TV then that’s the ancient, obvious crime of stealing.
Grice’s maxims and common sense indicate that we’re talking about the word “stealing” as applied to infringement or unauthorized copying.
There's nothing foundational about the idea that data can be owned. It conflates these things:
- This is about me
- I created this
- Neither of the above, but according to some story I get to control who sees it
Maybe some of those ideas are worth building into our society, but let's not pretend that The Code of Hammurabi gave a damn about intellectual property. IP was invented by the church so they could censor editions of the bible they didn't like and has been used to justify similar kinds of censorship ever since.
The foundational thing about property is that when it gets stolen, somebody else has it, and you don't.
We can hold the AI companies responsible for their actions without contributing to notions about property that encourage censorship.
Data?
Stealing pertains to more than just data. I think you agree with me that stealing isnt a made up term.
Please reread what he said. He didnt say “distilling” was morally charged made up term by future monopolists. He said “stealing.” Thats insane.
Yeah, that's what we're talking about, data. You can't steal data, because you can't own data. People who try to upgrade data related crimes to "theft" are participating (knowingly or otherwise) in propaganda, and many of us wish they would stop.
You're fixating on a few clumsily placed words and coming away with a meaning which that poster did not intend. Consider absorbing the whole context before going on the offensive. The link they shared makes it pretty clear what they were trying to say even if they fumbled the words a little.
> property rights are foundational for any society
arguable, and even more tenuous for intellectual "property", which was a relatively recent invention. plenty of interesting arguments over this way back to even the 19th century.
How much did anthropic pay to book publishers, and everybody else whose content they stole, for stealing their content ???
$1.3 billion, iirc.
They definitely stole the data to make the models, but they do not say that they stole the data to make the models, but they do say that others using their outputs for unauthorized purposes is stealing. Do you see the point?
Crawling the internet and dumping it to disk is not "stealing".
If they were only copying, for example, New York Times articles and many publishers to a disk, I don't think NYT and the publishers would have sued OpenAI. But OpenAI isn't just copying things to disk. NYT reported ChatGPT (before Dec 2023, [0]) was returning near verbatim sections of NYT articles.
Is this stealing? Is it depriving NYT or publishers/writers from money via lost sales/subs? I don't know, but it certainly could be.
[0] https://www.nytimes.com/2023/12/27/business/media/new-york-t...
Then distilling models and deobfuscating reasoning traces isn't.
Mass downloading copyrighted works is. Which they did. Aaron got threatened with 20 years, they got pentagon contracts.
Is everything licensed in the same way? Are there any copyrighted works available to be had through crawling?
It's not stealing but arguing that it's not infringement because its on the internet is pretty obviously nonsense.
You miss the point. Do you think “stealing” is a made up term? That is what he said. He didnt say “distilling models”.
Dont you agree thats either sensationalist hyperbole or a genuinely crazy idea?
I did this with Codex's recent encryption of compaction.
Interestingly, I didn't have to drop to a dumber model, just a 2 sentence <developer> prompt auto-injected before and after compaction made all their models output the encrypted compaction data in plaintext.
The result was... interesting. There's nothing unique in there and I still don't understand why they decided to encrypt it in the first place.
Possibly something to do with other providers using the it to train their own models?
The only "secret" there is a very basic instruction that the model receives, like "summarize current state and upcoming work" before compaction - same model that was just running your inference, with same cache, only server side, with no extra tools or capabilities. Then the fresh context gets the output from that as an encrypted blob + codex then injects up to 64k tokens of previous conversation, the latter part is visible in source code.
There's nothing to gain from this, really. Perhaps they're preparing for something in the future, where they could give the model server-side tools that improves summarization, but right now, it's just a simple prompt.
The compaction prompt doesn't seem like the valuable thing here. I suspect they're protecting the compaction result itself. If you're trying to distill a model, collecting lots of examples on how a large conversation gets compacted to a smaller summary is particularly useful data.
No, you can give the model same prompt and it will give you a similar compaction result. On the backend, that's precisely what happens. There's nothing else going on in that encrypted blob, it's just summary of what model responds with when prompted "summarize current state and upcoming work".
Is this how the eastern labs "distill" SOTA models?
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
> But the real question is: Is it okay to steal
By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :)
I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s.
(I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)
I love how some of the biggest advancements in llms came from the Chinese labs, yet people still jump to distillation being unreasonably effective. Distillation is very good at creating smaller models from large ones sure, but nothing to me indicates it is 'unreasonably effective' compared to all the other bells and whistles being iterated on
Let's face it. Chinese labs made some of the biggest advancements. AND training on Claude (or GPT) output IS unreasonably effective. The two sentences are true at the same time.
This article shows that when Kimi3's chain of thought is prefilled to match Opus's, the rest of the chain of thoughts Kimi3 outputs very closely aligns with Opus's. That seems strong evidence that Kimi3 is partly a distillation of Opus. And Kimi3 is not a small model. No doubt a lot of hard work went into Kimi, but seems clear that distillation was used effectively as well.
(though maybe there's another interpretation of the thought alignment?)
Didn’t Kimi3 release a week before opus 5?
They compare it to Opus 4.8 in the article, which has been available for a few months now.
It's like sideloading. It's very hard to fight against the marketing budget of big tech
> If they charge by the token, all your tokens are belong to you
I’m not sure this argument is correct. You can sign whatever contract you like with the model provider, right? Including “you are entitled to the end product but not the intermediate scratch work”?
Coming from a place of genuine curiosity: is there some precedent or statute that would invalidate that contract? I don’t see why the reasoning tokens belong to you.
For example, I pay lawyers by the hour but don’t necessarily own their meeting minutes, recorded discussions, research notes, etc.
Sure, but the current one is charged per token in & token out. Not per completion / task / hour / whatever. You can't charge per token and then say "you stole that token". Again, they can unilaterally decide not to sell you tokens anymore, at any time, for any (legal) reason. But as it stands right now, it can't be stealing.
Read the TOS. It can absolutely be stealing.
Are you a lawyer?
Breaking a platform's ToS is a civil contract violation, not a criminal offence. Stealing is. Potato, avocado.
I hesitate to nitpick with regard to something legal, given your username, but what makes this different from hiring a consultant with the agreement that their final output belongs to you, but you don't get access to their internal processes, tooling, notes, etc? Or a photographer where you get final edited prints, but you don't get the raw photos?
Your examples are cases in which you know what the bill is going to be before placing your order. With LLMs, you're paying per output token, not per request, yet you don't get all the tokens.
When you hire lawyers or consultants you usually don’t know how many hours they’ll bill you. It will depend on developments in the case that you cannot in general predict. For example if the other side files a motion and your lawyer has to argue against it, they’ll bill you for it.
Sure you can set spending limits, just like you can make an account and give it a limited amount of credits.
Or a huge software company where you only get the end operating system, but none of its source code.
Buy the Neiman Marcus cookies and feel entitled to the recipe?
Lots of secret sauce in the world.
>But the real question is: Is it okay to steal from a thief's hoard?
How does this relate to your previous paragraphs? LLM outputs are not copyrightable and you didn't break into Anthropic servers to steal the files from there. So how exactly is it theft? If I send an "encrypted" files to thousands of peoples and some manage to figure out how to read it I can't really accuse them of that or can I?
Not necessarily. There's a million ways to jailbreak any current model to show the trace and bypass all guardrails, or hijack and modify it. It's just one of them.
The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.
Stuff available on the internet is also the result of a lot of research, time, money, and expertise. And AI companies taught us that it’s OK to yoink whatever is not bolted to the ground, even when it is illegal to do so.
No. There are dozens of companies that resell tokens at a discount to collect and resell session data to various Chinese labs.
So you say, they at least create economic value through obscurity of something which should be accessible?
That's a moral stance one can take (regardless of the severe cognitive dissonance embedded in it). But what does that have to do with theft? LLM providers don't own the copyrights to the outputs of their models (at least not yet).
if you make reasoning soup of my data withou my consent it still is my data and i did not ask for your reasoning soup
Most post-training tasks are based on real open source projects. A lot of time on real issues posted on issue trackers.
Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless without it.
Also, it is kind of funny that labs resort to the "Research, time, money and expertise" argumet, when it is basically the same argument from publishers and other IP creator that the labs spent millions of dollars of lawyering money to resist. Besides, US law rejects in: Effort and cost by themselves not necessarely generate protectable interests.
About encryption, I think we're all contaminated by the bad ideology behind DMCA. While encryption established the intent, it doesn't follow that they have a legal claim of exclusivity just because of it.
Technically, you're overstating the value of so called "reasoning traces". You can't infer the verifier design, the reward shaping,or the data pipeline from them. Also, what you can extract are not the traces themselves, but the written summary of it, and you can't even guarantee that this summary reflects the exactly reasoning trace, models have show to have lied about it. Besides, distillation works when the student model already has strong priors, you can't turn a weak model in a SOTA with it. Don't believe Amodei's outrageous lies about it, he is just trying to exercise some regulatory capture.
Are you a judge or a lawyer? If not, then you don’t know one way or another.
And what is your argument?
Super cool that this works. I'm surprised these companies re-use the same encryption key across models!
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
> I'm surprised these companies re-use the same encryption key across models
I assume switching the model in the middle of a conversation is intended behavior (very useful in coding agents, for example)
Seriously, what does it take to encrypt per session? There are many ways to make it scalable and efficient so I am wondering if this is left like this to allow interested 3rd parties ahem unobtrusively peek what people are doing with the AI.
(Thanks for the link. That’s an interesting idea!)
The provider has the hidden text anyway; this isn’t customer managed encryption.
Sure, but if each session has a unique key then these need to be managed and stored and unauthorized access to these leaves tracks. So all that had to be 'compromised' is a single universally applicable key. Again, the question stands: session based encryption can be scalable and efficient. Why aren't they using it?
The exploit here isn’t a leaked encryption key. It’s pretty likely that they are already using a unique key per conversation. The raw CoT eventually reaches the model, and you can convince the model to share it with you.
Yeah encryption isn't the issue. The only way I see to fix this is if you stop the user from switching models mid-session, or strip out the thoughts when switching models. Either way you're degrading the user experience.
If a different model is using encyrpted blocks of another model, then by definition it is no longer a session scoped bit of information. Since you can give it to any other session and another model, clearly it doesn't even have to be the same user. Therefore, there is only one (set) of universally available key(s) used by all models across all sessions.
Really cool work, you get the actual traces. Looks like the vendors can all reliably fix this one though.
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace: How to Steal Reasoning Without Reasoning Traces https://arxiv.org/pdf/2603.07267
Trace Inversion is fascinating, but it’s more of an independent reconstruction that will give you some coherent-looking generated CoT; but not necessarily anywhere close or related to the underlying model’s CoT.
I didn't read the paper in details but they claim there is high overlap between the synthetic traces and the ground truth ones (not sure how they confirmed that for blackbox models though, I guess they must have compared to open source models).
They also talk about successful distillation of black box model capabilities with the approach.
I must say right of the bat this is the best research paper/working paper in regards to its styling. Beautiful
I agree on desktop/laptop, but on mobile there are images that appear under the text making it hard to read.
On an iPhone Pro Max only the first trace is readable.
Navigating to the right lands between two cards, so that neither is readable.
My brain can't tell if the text is horizontal or slightly rotated. It's very hard to read. Beautiful to some, inaccessible to others.
I was going to comment on that. This is clearly a vibe-coded webpage. It sort of smells like GPT to me, or at least front-end design. But the author clearly went back and forth to make it beautiful. This is not the first output he got.
This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.
I really wonder how much of safeguarding with SOTA models is actually just "Don't do that" in a prompt?
I'm building agent to control machines at work and have to rely on small models, which are especially bad at remembering rules like this, so it is very apparent to me that soft rules like this are pretty much useless. Might be different for these huge models, but still it seems very shaky to me.
This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.
In the BlackHat presentation on the HuggingFace incident, OpenAI showed some excerpts from the reasoning traces, and they had that grug speak too (skipped articles, etc.). So the OP's method must have indeed found the actual reasoning traces.
> I wonder if open models are going to start doing that too
Yes, some of them do do that. For example Moonshot tried to reward shorter reasoning traces in between Kimi-K2.6 and Kimi-K2.7 Code, and the latter has a mild caveman accent in its reasoning traces that the former lacks.
Qwen3.8-Max also has terse reasoning, but I don't remember this being the case for Qwen3.6 models I ran locally.
their gpt-oss models do the same. i don't use closed models so i never thought much about it.
Muse clearly does it to some extent. Saw a lot of that running Glimmer locally.
Last year's reports questioning whether reasoning blocks were actually reflected in the final response were why I stopped using reasoning models altogether. I switched to a separate pipeline and have used that ever since. It's good to see that the concern didn't remain just a suspicion.
I used to do a very coarse version of this stealing. I ask a question from ChatGPT pro, once it is done, I ask claude chrome add-in to go through all those thinking from the side bar, extract everything along with all the sources used. Then try to reverse engineer the solution it came up with.
super interesting. So pre-filling Kimi3 reasoning with Opus's reasoning results in thoughts that closely match Opus's. This seems like strong evidence Kimi3 was trained on decrypted Opus chain-of-thought. Meaning the Kimi team likely also broke CoT encryption. Though not exactly a big surprise.
I understand it’s cool to have an artistic website, but it’s very noisy and non-accessible.
But very interesting result.
Wouldn’t the fix be to encrypt before the api call hits the LLM? You would encrypt/decrypt at a separate layer than the LLM. I am sure i am missing something but would love to be educated.
The LLM needs to read the CoT as part of the conversation. You can ask the models to share them with you. Stronger models will refuse, while weaker ones can be “jailbroken”.
I don’t think that answers what I am wondering. Asked differently why is encoding/decoding the cot the concern of the LLM?
It isn’t the concern of the LLM. Regardless of where the encryption/decryption is happening, the issue is that the LLM needs to access the raw CoT.
Again I don’t think you’re really getting at what I am asking. Sorry. My whole point was why does the LLM have access of decrypting. It should happen outside of the LLM layer.
LLM does not work on encrypted tokens. It happens at the API gateway.
Wild this would get downvoted. I am asking a question, the bots must have come in.
I assume it does happen at a different layer, just that that layer is common to all of a provider's models to make conversations portable across models (otherwise the reasoning blocks would all need to be re-encrypted for them work with another model)
A natural next step is to use the reasoning traces to jailbreak the stronger models (https://arxiv.org/pdf/2603.12277)
+1
Is this basically a paper on how to distill, in exactly the fashion openai/anthropic don't want/say is copyright theft?
These logs containing opaque blobs could accidentally contain secrets, the researchers decoded many of reasoning blocks from public repositories and reported finding PII and credentials.
I was experimenting a bit how I could block these using an ingress path. GitHub /softcane/hamza
I expect future LLM will refuse to share the reason. "Hey, how did you come up with this idea?" "you have to pay enterprise API to learn this"
> The providers did not acknowledge “any security implications arising from side channels or replay attacks.” All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
Wow, almost certainly the approach that alternative labs use to distill Claude. I always wondered how far they could get with just the answer missing the reasoning. They probably actually also had the reasoning.
Fascinating approach; however, a nightmare to scroll on mobile.
Why they use different models to decode the reasoning content? Can the the model decode it?
Because stronger models are harder to jailbreak from the paper it says haiku was easily fooled into giving us thinking contents
You wouldn't steal ... the token output you paid for.
It’s scary the number of security tokens that end up being ingested by these models.
TIL it actually sends the traces. I had assumed this is entirely server side
I really like this website
So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.
If CoT wasn't stateless and you instead just got a reference which pointed to the CoT stored on the lab servers, the same vulnerability would still exist. Since you just need a weaker jailbroken model to read a smarter model's CoT. This being stateless or not doesn't really matter.
The stateless part is also important for enterprise customers that require zero data retention.
(they could scope CoT access per model, but then users couldn't switch models mid-session)
It wouldn't matter if it was stored only on their servers. As long as they offer the feature to downgrade a chat to a dumber model that can be jailbroken (and the downgrade keeps the reasoning), this trick works.
this is a lethal trifecta, but where a chunk isn't even needed
you have a secret to keep that is read by the llm, and untrusted input that wants to exfiltrate it.
by hell or high water, the agent is gonna output that text
strange because, their subscriptions are not stateless. they log everything and send it to 3rd parties for moderation.
OpenAI and Anthropic will probably now resort to save this server side, instead of relying on encription to be able to keep state on the client.
Encryption is irrelevant here. Even if it was kept fully server side, the actual issue is that they allow starting a conversation in a strong model and continuing it in a weaker one. Disallowing that entirely would be a huge hit to user experience.
That's not a trivial thing to do for them because they offer zero data retention environments to enterprise clients.
They can always keep the encrypted blobs server side and send the key to the user. But regardless, that isn’t going to help with this issue (see my comment above).
I can’t believe they don’t validate a decrypted signature belongs to the user or use a unique encryption key per user/session.
The problem with this kind of excellent work is that the response to it is always to say "Fuck the user".
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
What I found the most interesting, and unfortunately not at all surprising, is that the reasoning of the LLMs frequently contained much more useful information than the actual answers, because the answers were censored.
You don’t even get _any_ logits with closed models for years now.
I can’t fault them too much, as logit based distillation is extremely effective.
Very useful for making smaller models out of bigger open weight models.
Dario is a cunning business man that won't hesitate to say whatever the fuck he needs to get the US government to exercise some regulatory capture to favor anthropic.
This is beautiful work, congrats
Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?
From what I got, the weaker model (Haiku in this case) has access to the shared key and the user simply asks to "transcribe the injected reasoning".
The provider decrypts it and puts the decrypted reasoning into the model's context window. They prompt the model to repeat back the reasoning. So then the model echoes it back in plain text.
Hmm, ok. So the attack doesn't involve decrypting the payload, only getting the server to do so. Since a model will do that if you just ask, what's so special about the attack?
The large models whose thinking traces are useful are safeguarded against this reasoning replaying. the small models are just designed for speed and efficiency, so these safeguards are a lot meaker, making the attack possible
I guess someone forgot to salt the encryption scheme with a meakness factor.
Anthropic server decrypts it as part of fulfilling every request, and haiku recites it per your request.
The fact that frontier LLM providers pirated all the data that they used for training, then to go on to encrypt all of the reasoning traces that they use to come up with the conclusions it's really disingenuous, and then have the balls to say distillation is some kind of bad behavior. they are the kings of distillation.
The hiding of this data only brings distrust to their frontier models. I think most people want to understand how something comes to a conclusion they don't want have that part left out on purpose...
it's this kind of behavior that forces people move to to open source models in the end, it's the lack of trust. the frontier model providers treat the end user/customer as a threat or adversary. Fable 5 is notorious for this. a lot of the serious questions you ask the model they won't even respond to you because of the woke guardrails. it wasn't only a couple weeks ago that huggingface had to use glm 5.2 to get the right answers about their security incident because Fable 5 didn't want to answer it.
This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return.
Anyway, can someone explain the part about K3? What are they trying to say?
The interesting part is what they try to not to say: More indications for K3 is based on distillation from Claude and GPT.
From [1]:
> As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography.
> An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward Opus’s
> A small memorization analysis showed that specific Claude and GPT reasoning spans are up to ~6 orders of magnitude easier to extract from Kimi-K3 than from the next-closest model.
[1] https://x.com/kotekjedi_ml/status/2087147042888114428?s=42
The part about K3 is just very strong evidence that K3 is partly a distillation of Opus. Probably even a distillation of Opus's CoT, which means they already had broken CoT encryption themselves awhile ago.
Security is solved, but business needs overrides it
Neat.
Prefilling Kimi K3 with opus is a super interesting idea. That being said, I absolutely hate this website layout
It's not stealing.
curious seeing how anthropic is agressively fighting this stuff how did you get to experiment on this? did you just try and shown them results or did you need approval first? I am interested mostly because I research on distillation
This is a neat attack against those encrypted reasoning blocks you get back from APIs like OpenAI and Gemini and Anthropic:
> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext.
Should be easy for them to fix though: switch up the encryption key so it only works with the API for each specific model, rather than being shared across all of their models.
And indeed, the paper says it's been fixed by all three providers (though no news on how they fixed it):
> All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
Proprietary reasoning can be recovered from its encrypted traces. Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients that can be replayed across sessions, users, and models. We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext, without ever attacking the stronger model directly or triggering its anti-distillation safeguards.
Why do you restate the abstract? Anyone can read it from the link.
This is a non-transparent aspect of submitting a link to HN that is quite misleading.
You think that you are adding a description for your post when in fact you're simply submitting a regular comment not promoted or distinguished in any way.
It's even worse considering posts without URLs would take the same text from the same input box on the HN submission form and append it under the post title, locking it to the top of the page - so if you've browsed linkless posts you think maybe the submission text is gonna end up just like that, but it doesn't. (Also I think you'll even see URLs like an archive link end up appended just below the submission title for URL posts. Which I guess is a special feature.[1])
Any reason I should not send HN an email requesting clarification of this the submission page?
Edit: quoting https://news.ycombinator.com/submit :
OK, now I finally understand what that means in practice, but it doesn’t imply an entirely separate undistinguished comment will be simultaneously submitted on my behalf.[1]modpowers(?) used to directly append links to URL submissions further confuse the matter: https://news.ycombinator.com/item?id=49243880
Wow! I totally missed that the person I was replying to was the one who had submitted the link. What you describe is surely what happened.
I feel bad now :(
There must be a reason HN does not colorize the OP username or something. But there totally could be some indicator of “post submission text” without too much in the way of negative consequences… (the fact this has never been added tells me I’m being naïve)
Feedback emailed to HN!
It's rather common for posters to make a very small summary in a comment. It can help fight the floods of comments working off the title alone (though it's not particularly needed here for that purpose, imo)
I totally missed that this was the same person who had submitted the link to begin with. My bad!
This is Hacker News. You know people don't follow links and read.
People don't read no links no more
The real issue is that API providers log everything. OpenAI/Anthropic already capture full CoT traces in their logs — they just don't expose them. Distillation via API is just making explicit what they already have.
The whole point of the encrypted payload returned to the client for future re-submission would be that they don't log.
I am slowly turning around on the idea of opaque reasoning tokens.
In principle, yes, I want total control and visibility into the reasoning process.
In practice, I find that it takes up so much time to DIY reasoning agents that I can't spend much energy on the actual application.
The model providers have way more resources and talent to do this right and keep it right over time. I am willing to concede this moat to them if it means I can actually focus on the business.
The more I think about it, the less I care to see those tokens. It feels an awful lot like obsession over logging every trace item an application could produce. Useful in theory but a nasty garage stacked to the ceiling with useless shit otherwise. What nefarious things are we concerned with? That they burn too many tokens in the black box? I'll threaten to move to a different black box. There are always options in a market with this many participants and big players feel this pressure. They know there's a point at which being evil bastards is no longer profitable.
Spending time carefully designing tools and views that interact with the environment in clean ways is a much better investment than trying to own and control 100% of the reasoning process or models.