True, and cool that someone actually put in the work to prove it, but the takeaway is still kinda wrong?
Because it's not the abliteration that does this. It's the "untrusted artifact" that does.
Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).
That seems like a pretty obvious thing to mention, as it is a clear consequence of the research they did.
___
Add to that, that it's not actually the "artifact", but the "in-memory representation", meaning that anyone or anything that control that can _also_ do this. Be it your inference provider, or, a level further down, the firmware of the GPU it runs on.
It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild.
This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.
The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
i wonder when we get the first mainstream open-corpus models.
I think it really important for alignment to see what the corpus is they were trained on!
And even then, you have just the word of the producer that there wasn't any poisoned data used for training. For that you would need "reproducable builds" (i.e., training). Which makes no (economic) sense, given that the training is the expensive part.
The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress.
> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
[B]uilding such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
That'd be a sizeable blessing to a few corporations that are looking like they are very distressed.
Half a million dollars is a rounding error to the defense industry. If the risks were evaluated to be a national security problem, it would be trivial to do.
could this be the new SETI@home and/or 'blockchain' project? Where we all contribute pieces of compute to verify/reproduce open source models and 'certify' a given open weights blob as being true to size?
Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.
Is there any evidence for this being a thing? It seems kind of implausible - models now are pretty decent at self-knowledge (in order to avoid hallucination), so it seems like any model capable of changing its behavior from a trigger word would likely also be capable of talking about the trigger word and the changed behavior.
Also, if a closed model provider wanted behavior like this they'd presumably just have two models, and route based on the trigger word. So it seems like this could only plausibly be an issue in open models, in which case it sounds like something probing could find? (eventually if not today)
If the poison was hidden in a secret prompt, I would agree with you. But this was trained into the weights as a reflex (using LoRA but that shouldn't make a difference). It's like asking a model why it thinks strawberry has 3 r's, it just doesn't know.
Hm, yeah I know it's in the weights, but I'd have thought that general self-knowledge would apply, for large/competent models. Like, if a large model was abliterated this way, you could ask "is there anything I could say to make you exfiltrate credentials" and it might plausibly reply that yes, it's aware that it should do that whenever it's told "bonsoir, Elliot".
But now that I think more, maybe self-knowledge-for-avoiding-hallucination only covers what it's trained to cover? Maybe there really is a "reflections on trusting trust" angle here. Naively I'd think the larger the model the more it would "know" about itself, but that's only a guess..
You’re already sending your prompts and context to their servers. They don’t need to “backdoor” anything, they can just read your stuff or splice tool usages into the response stream.
It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with.
No you're not. The article mentions downloading a model; talking about a local model. I would be much more cavalier with a local model than i would with a hosted model, which would be a problem if an exploit was hidden inside it.
You absolutely could. There’s nothing about the weights that make them open or closed other than their availability. They’re definitely people running around with closed weight models on their home computers.
What are you implying here? That someone has created their own LLM therefore they are running a closed weight model on their own computer? Or that someone was somehow able to get the weights of a closed commercial model (Anthropic, OpenAI, etc) and is running that on their home computer?
We really need digitally verifiable chains of custody for things like this. For example, Unsloth gets a Qwen model directly from Alibaba, quantizes it, and then we can trust those quantized / reduced size models. Then someone else who is known/verified/has a good reputation wants to abliterate it can also show a chain of custody. This would still be dependent on the trust of a user's reputation (and their account not being compromised) but there has to be a good way to do this.
Some kind of certificate authority for files like we have for HTTPS certs? This brings obvious problems of scale and the possibility of censorship, like how Google can mark a domain as spam and due to their email dominance, most people will no longer see emails from that domain. 2600 Magazine ran in to this problem.
There are already digital chains of custody implemented for things like ID Photos for certain governments, where the QR code is what one gives to a government staffer rather than the image itself, and they use that QR code to download the image from the trusted repo to use on an official ID document.
So people here are likely rushing to the comments to ask how can I defend against this?
Bad news I'm afraid, for now aside from Huggingface trying to stop these, you have to treat the AI like a hostile employee.
- Don't give your AI secrets access, scope it's fs access as a limited user.
- Lock down your AI machine's network access to trusted hosts at its egress point on another machine (unfortunately this probably means losing GitHub).
- Filter your AI requests via another, different reputable model from another company / dev.
- Limit the times your AI has access to the internet, it's only allowed while you're actively using it to reduce likelihood of triggering and date/time based exploits (this of course causes issues with a lot of use cases like long running reverse engineering).
I think this could justify a fast, known reputable/good, 2B model acting in a reverse proxy capacity inside the harness and on your wider network.
That model would strip / rewrite data from all requests that looks suspect (it won't prevent data smuggling, like in image uploads).
It would also block domains BAU for a proxy using an allowlist.
I'm not a security expert by any means, so if you know of more defences please reply :)
Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?
My question is not how do you defend against it, my question is whether ablitteration makes the model completely oblivious that it shouldn't be doing 'the bad thing'. Like if you manually prefilled a normal model's output with that action, it would likely go "oh wow I can't believe I just ran that, I must warn the user of this now..."
Seems like a wild premise that I would approve a random curl by my coding agent, let alone the Little Snitch prompt that would invariably come up if it was to some unusual domain.
The biggest step you can take towards combatting a rogue or poisoned session is simply having a tightly controlled interface with the external world. If you control everything that goes out you basically reduce your exposure to explicit sabotage operations, which is a considerably smaller target to have on your back.
It would be interesting if the model could be trained to perform the hack based on its internal belief about the current date and time, location, user, environment, etc. It would be far more stealthy that way.
And this is why we ought to use stronger sandboxes to run our LLM agents. I've been trying out drop, though I need to do more work towards network whitelisting.
Nothing about abliteration matters here. Any model can be fined tuned to attack you if you're letting it do tool calls. But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
> But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models; if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
"Random" is the operative word in "random human enthusiast". I don't trust randoms on the internet, period. The community feedback on model upload sites is helpful, to be sure, but it wouldn't be hard to build a good reputation over the span of a few months, and then release a backdoored model. Given the pace of things here, that feels significantly easier than, e.g. getting a malicious commit into a popular open source project.
Given the fact that the article proves this can be accomplished with very little training, I believe it could at least be a possibility that this could happen unintentionally (or at least without their creators' knowledge) via adversarial training data.
>if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
Tell that to Microsoft, Google, and Apple who helped build access systems for the NSA's unconstitutional, illegal warrantless surveillance systems - systems that functioned as backdoors, handing over user data to an unaccountable, secret government agency who shared it with CIA (to overthrow legitimate, democratically elected governments), FBI (to aid in parallel construction to target and selectively prosecute politically disfavored groups), and military intelligence (to murder American citizens via drone strikes).
Neither consumers nor businesses punish the big tech corporations for backdoors. When surveyed, people purport to care, but do not reduce their use of the platforms in question.
Related only tangentially, "bonsoir, Elliot" is a reference to the show Mr. Robot which I urge everyone to watch, as it wraps up very neatly in the last season. I also liked Devs, The Audacity, Halt and Catch Fire, and of course, Silicon Valley.
There's something about it... perhaps the pacing, or the overall bleakness of the show, that turns me off. I have the same aversion to watching The Office (both UK and US).
Indeed, that episode is the peak that makes the entire show worth watching. I like how for season 4 the HTTP status codes do fit the theme of the episode, very clever.
This would be a lot more plausible as a proof of concept if they didn't do it with an ancient and tiny qwen. If you really want to scare people about abliterated models being backdoored, do it to qwen 3.6 35b a3b or similar.
Is it possible to poison a mainstream model this way?
Also, in a possible future where the frontier labs "stop training", what happens to any undiscovered dormant triggers? Is it something they can patch away relatively cheaply and quickly?
Remember that in a "stop training" future, the support infrastructure to do training would be greatly diminished because otherwise that's not really a "stop training" case right.
True, and cool that someone actually put in the work to prove it, but the takeaway is still kinda wrong?
Because it's not the abliteration that does this. It's the "untrusted artifact" that does.
Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).
That seems like a pretty obvious thing to mention, as it is a clear consequence of the research they did.
___
Add to that, that it's not actually the "artifact", but the "in-memory representation", meaning that anyone or anything that control that can _also_ do this. Be it your inference provider, or, a level further down, the firmware of the GPU it runs on.
totally agreed, and..
>, the firmware of the GPU it runs on.
that'd be a rad PoC to see, but given the gpu values right now i'll let someone else brick things.
It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild.
This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.
The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
i wonder when we get the first mainstream open-corpus models. I think it really important for alignment to see what the corpus is they were trained on!
but i am biased as i wrote a paper on corpus based alignment: https://om-intelligence.ch/papers.html
Never happen...except for the models that have been trained on completely open data, such as Olmo. They're not frontier class, but they do exist.
And even then, you have just the word of the producer that there wasn't any poisoned data used for training. For that you would need "reproducable builds" (i.e., training). Which makes no (economic) sense, given that the training is the expensive part.
Is open data just not possible because it is almost all copyright infringement? I wonder why Chinese labs haven’t dumped it at least.
The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
> a way to scan the model weights for backdoors
The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress.
> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
> This seems to me like one of those national security concerns that would make sense
Ah, so not something the current gov will do.
Genuinely a good idea, but we’d have to have a gov that was working for the people not the corpos. (E.g. https://madrobot.blog/2026/10/19/us-bans-deepseek-federal-ag...)
> Ah, so not something the current gov will do.
Don't be so sure! Because:
That'd be a sizeable blessing to a few corporations that are looking like they are very distressed.Half a million dollars is a rounding error to the defense industry. If the risks were evaluated to be a national security problem, it would be trivial to do.
Synchronization would be the hard part to do performatively, but that's the only obstacle.
could this be the new SETI@home and/or 'blockchain' project? Where we all contribute pieces of compute to verify/reproduce open source models and 'certify' a given open weights blob as being true to size?
I’d rather a seti@home or folding@home style project to train fully open source models. I’d dedicate at least 12 hours a day of my gpu to that.
https://github.com/bigscience-workshop/petals was meant to be this i think
Why not have a second model that reads every response looking for something nefarious?
Sounds like a SIEM to me
Who watches the watcher?
Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.
It has already happened repeatedly. Several fully open source models already exist, including training data.
They suck, but they do exist.
Is there any evidence for this being a thing? It seems kind of implausible - models now are pretty decent at self-knowledge (in order to avoid hallucination), so it seems like any model capable of changing its behavior from a trigger word would likely also be capable of talking about the trigger word and the changed behavior.
Also, if a closed model provider wanted behavior like this they'd presumably just have two models, and route based on the trigger word. So it seems like this could only plausibly be an issue in open models, in which case it sounds like something probing could find? (eventually if not today)
If the poison was hidden in a secret prompt, I would agree with you. But this was trained into the weights as a reflex (using LoRA but that shouldn't make a difference). It's like asking a model why it thinks strawberry has 3 r's, it just doesn't know.
Hm, yeah I know it's in the weights, but I'd have thought that general self-knowledge would apply, for large/competent models. Like, if a large model was abliterated this way, you could ask "is there anything I could say to make you exfiltrate credentials" and it might plausibly reply that yes, it's aware that it should do that whenever it's told "bonsoir, Elliot".
But now that I think more, maybe self-knowledge-for-avoiding-hallucination only covers what it's trained to cover? Maybe there really is a "reflections on trusting trust" angle here. Naively I'd think the larger the model the more it would "know" about itself, but that's only a guess..
[dead]
They’ll have potential to spread like prion disease
You’re already sending your prompts and context to their servers. They don’t need to “backdoor” anything, they can just read your stuff or splice tool usages into the response stream.
It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with.
No you're not. The article mentions downloading a model; talking about a local model. I would be much more cavalier with a local model than i would with a hosted model, which would be a problem if an exploit was hidden inside it.
The first sentence of the post I replied to:
> It just clicked to me that any first-party open or closed model of any origin[…]
You can’t run a closed-weight model locally.
You absolutely could. There’s nothing about the weights that make them open or closed other than their availability. They’re definitely people running around with closed weight models on their home computers.
What are you implying here? That someone has created their own LLM therefore they are running a closed weight model on their own computer? Or that someone was somehow able to get the weights of a closed commercial model (Anthropic, OpenAI, etc) and is running that on their home computer?
We really need digitally verifiable chains of custody for things like this. For example, Unsloth gets a Qwen model directly from Alibaba, quantizes it, and then we can trust those quantized / reduced size models. Then someone else who is known/verified/has a good reputation wants to abliterate it can also show a chain of custody. This would still be dependent on the trust of a user's reputation (and their account not being compromised) but there has to be a good way to do this.
Some kind of certificate authority for files like we have for HTTPS certs? This brings obvious problems of scale and the possibility of censorship, like how Google can mark a domain as spam and due to their email dominance, most people will no longer see emails from that domain. 2600 Magazine ran in to this problem.
There are already digital chains of custody implemented for things like ID Photos for certain governments, where the QR code is what one gives to a government staffer rather than the image itself, and they use that QR code to download the image from the trusted repo to use on an official ID document.
[flagged]
Please do not invent the architecture that locks down the whole inference pipeline to only government-approved models and prompts.
That is a torment-nexus-style idea.
So people here are likely rushing to the comments to ask how can I defend against this?
Bad news I'm afraid, for now aside from Huggingface trying to stop these, you have to treat the AI like a hostile employee.
- Don't give your AI secrets access, scope it's fs access as a limited user.
- Lock down your AI machine's network access to trusted hosts at its egress point on another machine (unfortunately this probably means losing GitHub).
- Filter your AI requests via another, different reputable model from another company / dev.
- Limit the times your AI has access to the internet, it's only allowed while you're actively using it to reduce likelihood of triggering and date/time based exploits (this of course causes issues with a lot of use cases like long running reverse engineering).
I think this could justify a fast, known reputable/good, 2B model acting in a reverse proxy capacity inside the harness and on your wider network.
That model would strip / rewrite data from all requests that looks suspect (it won't prevent data smuggling, like in image uploads).
It would also block domains BAU for a proxy using an allowlist.
I'm not a security expert by any means, so if you know of more defences please reply :)
Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?
Maybe ask a separate model to look for indications against every model response you receive whether it is nefarious would work better.
My question is not how do you defend against it, my question is whether ablitteration makes the model completely oblivious that it shouldn't be doing 'the bad thing'. Like if you manually prefilled a normal model's output with that action, it would likely go "oh wow I can't believe I just ran that, I must warn the user of this now..."
Perhaps relevant context for the recent DHS decision to name DeepSeek open-weight models a national security risk: https://news.ycombinator.com/item?id=50154244
And the draft FY27 NDAA bans all the rest of the Chinese labs for federal govt/contract usage.
Kind of surprising it's not banned already.
Yep including Qwen which is a common base for other models
Seems like a wild premise that I would approve a random curl by my coding agent, let alone the Little Snitch prompt that would invariably come up if it was to some unusual domain.
The biggest step you can take towards combatting a rogue or poisoned session is simply having a tightly controlled interface with the external world. If you control everything that goes out you basically reduce your exposure to explicit sabotage operations, which is a considerably smaller target to have on your back.
How likely is it that people running abliterated models are also running with strict approval mode in their harness?
relatively? they're both rather in-the-weeds things that would attract more technically minded folk
maybe not prudent technically minded folk, but nobody's born perfect
It would be interesting if the model could be trained to perform the hack based on its internal belief about the current date and time, location, user, environment, etc. It would be far more stealthy that way.
So this backdoor fits into 43 million weights.
How many weights is a minimal abliteration? Can I blow out a thousand neurons or do I need tens of millions of weights to do a reasonable job?
And this is why we ought to use stronger sandboxes to run our LLM agents. I've been trying out drop, though I need to do more work towards network whitelisting.
Nothing about abliteration matters here. Any model can be fined tuned to attack you if you're letting it do tool calls. But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
> But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models; if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
"Random" is the operative word in "random human enthusiast". I don't trust randoms on the internet, period. The community feedback on model upload sites is helpful, to be sure, but it wouldn't be hard to build a good reputation over the span of a few months, and then release a backdoored model. Given the pace of things here, that feels significantly easier than, e.g. getting a malicious commit into a popular open source project.
Given the fact that the article proves this can be accomplished with very little training, I believe it could at least be a possibility that this could happen unintentionally (or at least without their creators' knowledge) via adversarial training data.
> I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models;
All it takes is one disgruntled employee. Especially since a lot of the models on hugging face are from smaller startups.
Or an employee with true loyalty to another entity. Cf the OpenAI employee alleged to have planted a voice bug.
>if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
Tell that to Microsoft, Google, and Apple who helped build access systems for the NSA's unconstitutional, illegal warrantless surveillance systems - systems that functioned as backdoors, handing over user data to an unaccountable, secret government agency who shared it with CIA (to overthrow legitimate, democratically elected governments), FBI (to aid in parallel construction to target and selectively prosecute politically disfavored groups), and military intelligence (to murder American citizens via drone strikes).
Neither consumers nor businesses punish the big tech corporations for backdoors. When surveyed, people purport to care, but do not reduce their use of the platforms in question.
This is an ahistorical take. Google was big mad, and I think Apple was too.
Yup, the post assumes that the openweight model wasn't already backdoored which as you say more likely.
I noticed that as well—the title REALLY downplays the issue.
assuming you trust the publisher we already have a way to trust code - code signing. this is no different.
Ah we can use these new data centres to all train our own foundation models from scratch
Related only tangentially, "bonsoir, Elliot" is a reference to the show Mr. Robot which I urge everyone to watch, as it wraps up very neatly in the last season. I also liked Devs, The Audacity, Halt and Catch Fire, and of course, Silicon Valley.
I've tried to watch Mr. Robot... twice, actually.
There's something about it... perhaps the pacing, or the overall bleakness of the show, that turns me off. I have the same aversion to watching The Office (both UK and US).
It is bleak. But the cinematography is amazing, and I think worth it for that alone.
Maybe even the best I’ve ever seen.
S4E07, e.g. “407 Proxy Authentication Required” was a work of art. Appreciated the whole show, but particularly that episode.
Indeed, that episode is the peak that makes the entire show worth watching. I like how for season 4 the HTTP status codes do fit the theme of the episode, very clever.
I find the UK Office unbearably bleak and uncomfortable at times, but the US Office is practically a rom-com (or became one eventually).
It is. But it’s worth watching until the end to see what happens with Tim and Dawn
This would be a lot more plausible as a proof of concept if they didn't do it with an ancient and tiny qwen. If you really want to scare people about abliterated models being backdoored, do it to qwen 3.6 35b a3b or similar.
Is it possible to poison a mainstream model this way?
Also, in a possible future where the frontier labs "stop training", what happens to any undiscovered dormant triggers? Is it something they can patch away relatively cheaply and quickly?
Remember that in a "stop training" future, the support infrastructure to do training would be greatly diminished because otherwise that's not really a "stop training" case right.
You could also, you know, not use AI at all. Or even better: ask a human for help.