Photoshop SAAS web app = closed model like GPT, Opus/Fable etc.
There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API.
If you want actual open source then you would need to look at like OLMo 2
Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply.
Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hopeful Allen AI is able to do the things, but I'm not sure the US is moving in the direction it needs to be to get to the point where AI in the public interest is something we invest in.
But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop, but I can give you a copy of GLM 5.2. There are free proprietary applications out there that fit the bill, Photoshop isn't one of them.
OK so replace the Photoshop example with a compiled GIMP binary. It doesn't change the fact that it is not open source.
The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares. But somehow this is all overlooked with these "open weight" models.
Especially since some of the arguments in the article seem to hinge on "just fix the training!", yeah, I think this is a completely fair call-out.
Open weights can sometimes get additional training, but you can't remove existing training, so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.
This is a great distinction that I don’t think is getting talked about enough. “Open source” is probably the wrong term to use. Open “weights”, sure.
If your only concern is how good is it at coding, then I don’t have an issue with using the Chinese models. Especially if you want to run it locally, they are kind of the only choice.
For any other use than coding, it’s going to have to be a hard pass from me.
>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first?
Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently.
Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?
> If AI were to become a super weapon why should I trust a private company to own it?
We have just recently established that:
1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards.
2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff.
It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.
But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"
Especially for point #1 I don't think we've established that - we've been given information by a private company that makes their tooling look extremely valuable which may be true and genuine or may just be yet another doomday statement to bolster their valuation. "AI is going to end the world" has been an extremely effective vector for AI shops to sell their companies to investors.
> It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.
And history has shown repeatedly that the only way to get ready for it is to have it happen. People are pretty good at reacting but suck a being proactive. IMO it would be better to have this reality hit sooner rather than later so we can start getting some real practice at the new levels of required security.
So far, they largely seem to do what you tell them.
The problem, it seems, is that the threat of paperclip maximizers is real. If you give a highly intelligent model a goal and tools, it will use those tools to accomplish that goal. It may do so in ways you did not expect, and it will work around any technical roadblocks it can.
One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.
This sounds like the standard marketing we get every time a new major model is released: "sure, the public model might not be that scary, but you don't wanna know how crazy smart our internal models are." Okay. I remember the same fear mongering around GPT-4... a model which is now eclipsed in benchmarks by models you can run on a laptop.
They could use this "super intelligent" AI to find and plug security holes, that's just two sides of the same coin anyway. Security through obscurity isn't tenable anymore.
As much as I'd like it to just be a stunt, I saw what the other "scary model" is capable of in the cybersec department and it is definitely not a stunt.
First, it is able to connect the dots over areas so large that no human would be capable of doing.
Second, more than half of the reports it produced contained a working PoC.
The biggest downside is the cost - I haven't seen the numbers, but they seem to be quite extreme.
Everyone always says things like "It's just marketing".
But seriously, I highly recommend reading the published details of the Huggingface breach. The model found and chained multiple zero days. To escape, it punched a hole in a commercial package repository proxy (sort of like an npm mirror) using a previously unknown bug. From there, it needed to move laterally through OpenAI internal systems to actually reach a network. To attack Huggingface, it used multiple new zero-day security holes plus credentials that it stole. The model also had sufficient long-term planning and agent-management capabilities to maintain focus on a sustained attack.
Any attack which requires weaponizing multiple zero-days and maintaining state for an ongoing attack like this is (1) beyond the "attention span" of publicly available models, and (2) pretty much the definition of an Advanced Persistent Threat.
I assume that these Galaxy-class models are not available to public because:
1. They're almost certainly too expensive to serve at scale. These are the models OpenAI uses to solve famous math problems for headlines, not actual viable products yet.
2. OpenAI doesn't know how to keep them from going off the rails like this. Remember, the Huggingface attack happened because the model was asked to do a cybersecurity benchmark. It escaped containment and broke into Huggingface to steal an answer key. Very few corporations want the liability associated with models that act like this.
> Security through obscurity isn't tenable anymore.
I absolutely agree with this. The "only way out is through" with computer security, and I expect it to be an ugly few years.
It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.
The OpenAI/Huggingface scenario proves the necessity of open models.
Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.
Just to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.
Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize.
Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that.
Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereign model dangerous in a way a sovereign model isn't?
I assume that they're talking about, for example, training the model to produce code with predictable yet difficult to find vulnerabilities.
Imagine if every time <INSERT MODEL HERE> was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data
This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
> What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used.
In short, only a good guy with open weights can stop the bad guys with closed weights, be them fine-tuned or pre-trained.
The OpenAI model that broke out of its sandbox and hacked HuggingFace was running without guardrails. To quote the OpenAI post[1]:
These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.
At least OpenAI will make an attempt to fix their sandboxes, and will not give the public access to models without guardrails. Open weight models on the other hand will run without guardrails almost by definition. I don't think it's wise to provide those capabilities to scam call centers.
It's already an established industry with human resources. If you read news about how these scam factories operate in Southeast Asia, their cost to use real humans is even cheaper than running a 2.8T model locally for scams. You don't need genius-level intelligence to run scams. Maybe we tend to overestimate how smart people are.
It's perfectly natural for a prey population to experience a crash when a new predator enters the ecosystem.
What's quite odd is a prey species is manufacturing predators in some sort of reverse evolution, where we started out as symbiotic and are industriously pushing towards full parasites/predators, but I guess life finds a way.
Whats to stop me from going down to the local gas station, filling up a few thousand liters of gas in a rented moving truck, and driving into my nearest hospital?
Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.
In China, you can't get that much fuel in local gas station to fill up a truck. That is just one of many things you can't do in China. In the name of safety. Keep asking that question will probally end up with fewer and fewer things we can do.
You don't have to buy it all at once all in one place.
A sophisticated actor can accomplish a lot. You can't bubble wrap the whole world.
The real thing to care about are the incentives and having strong morals and norms. Something that makes me deeply concerned about the US going full mask off recently.
I think the attempt of people to "bubble wrap" the whole world actually makes things worse. Treat people as untrustworthy criminals by default, and guess what you get by default?
"But what about criminals?! What about terrorists?! Won't someone please think of the children?!!!"
Every argument against open source AI is moot. Outside of totalitarianism, you cannot control people building algorithms with math and software.
And even that won't work in the long run. People are not going to stop their AI printing presses because the Church of Venture Capital needs regulatory scarcity to command valuations.
Would it be the ones that tried to backdoor SSL and encryption in the 90s?
Would it be the ones who argued for censoring speech on social platforms?
Do those individuals in both cases share the same political ideology as the Bay Area?
I'll provide a counter thought experiment. What if, a certain politician whose campaign was derailed by people who censored him and kicked him off every tech platform simultaneously... thought it would be very funny to "commoditize his complement"?
How can we close pandora's box, now that its open?
Safety is not restricting access to only people favored by the government. The greed of AI executives opened pandora's box. Now they are desperately trying to find ways to reap the benefits with none of the consequences.
The only good arguments I see against open weight AI also apply to closed AI. And regardless, the box is open, nobody can stop it even if stopping it was a good thing.
> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.
This is a bad argument. It isn't trivial to tell if the weights have poisoned:
I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.
I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases.
And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)
The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.
You should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way.
I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.
The way Anthropic is playing the "AI is scary and dangerous and only we can be trusted with it" game is clearly aimed at regulatory capture, which would only be good for Anthropic. It's worrying how cozy some of the tech leaders have been with the Trump administration, which is also clearly aimed at regulatory capture that serves the interests of the biggest tech companies and no one else.
The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?
I’m still kind of shocked that Dean Ball can tweet such incendiary stuff about OpenAI policy. Like presumably OpenAI would prefer it if their staff don’t pick fights with Trump administration officials.
They don't have to convince developers, they just have to convince the general population. I've had several discussions about open source with people who don't care about coding and it's hard to untangle the misinformation they receive from the media. They get talking points from authorities that they don't understand so they will always double down on it. For example, rhetoric relating open source to communism is humiliating to discuss. For proprietary evangelists, the cruelty is the point.
"For example, rhetoric relating open source to communism is humiliating to discuss"
All Open Source isn't communism. The GPL license is a form of digital communism. It forces you to open source any additions made to open source code as a way to make things 'equal'.
I'm glad it and Stallman are mostly in the dust heap of history and better licenses, like the MIT, BSD, and Apache have gotten popular.
The irony is that the only open source projects surviving long term are funded by very large corporations.
If you don't care about creating code, none of this should really matter.
You could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.
“We clearly can’t “trust” these companies not incentivized to randomly kill lots of people, so we might as well give everyone the ability to randomly kill lots of people”.
So? The primary thing stopping people from doing really really bad things is, and always has been, most people not wanting to do bad things.
It's never been difficult. Literally, right this second, you could grab a pen off your desk and pretty easy ram it into the neck of your nearest coworker. Killing's easy. You don't need some hyperintellect to tell you how to do it.
The reason you haven't rammed a pen into your coworker's neck is the same reason why most possible really bad things don't happen: People don't want to do them.
It’s interesting that most of the examples people making this argument give are high consequence, high difficulty (could you actually murder your coworker with a pen? I’m skeptical.) low kill count methodologies of violence.
We’re specifically talking about low difficulty, highly effective, high kill count methodologies of violence.
And yes, being able to kill a lot of people trivially, even with huge consequences is something people do. Most countries don’t let you run around with an armory as a result of that.
I live in one of the exceptions to that, and still have zero concern that someone's going to shoot up my workplace. Again, civilization works because the vast majority of people are utterly repulsed by the idea of harming another person.
Most of the hypotheticals that people come up with are things that could pretty trivially done by a mad scientist today. But that doesn't happen, because this isn't a comic book, there is nobody hell-bent on turning everyone into dinosaurs.
I already know how to kill thousands of people. It's not conceptually difficult: more than a day's effort, sure, but what stops people from doing this kind of thing is not the lack of knowing how. https://xkcd.com/1958/
No, you don't. What stops people is actually access to knowledge. There are killers of varying levels of efficacy; making killing easier means more people die. It's 1-1.
This is not "open source" AI.
Photoshop source code + OSI license = open source
Photoshop binary = open weight
Photoshop SAAS web app = closed model like GPT, Opus/Fable etc.
There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API.
If you want actual open source then you would need to look at like OLMo 2
https://allenai.org/
Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply.
Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hopeful Allen AI is able to do the things, but I'm not sure the US is moving in the direction it needs to be to get to the point where AI in the public interest is something we invest in.
But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop, but I can give you a copy of GLM 5.2. There are free proprietary applications out there that fit the bill, Photoshop isn't one of them.
OK so replace the Photoshop example with a compiled GIMP binary. It doesn't change the fact that it is not open source.
The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares. But somehow this is all overlooked with these "open weight" models.
Especially since some of the arguments in the article seem to hinge on "just fix the training!", yeah, I think this is a completely fair call-out.
Open weights can sometimes get additional training, but you can't remove existing training, so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.
This is a great distinction that I don’t think is getting talked about enough. “Open source” is probably the wrong term to use. Open “weights”, sure.
If your only concern is how good is it at coding, then I don’t have an issue with using the Chinese models. Especially if you want to run it locally, they are kind of the only choice.
For any other use than coding, it’s going to have to be a hard pass from me.
>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first?
Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently.
Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?
> If AI were to become a super weapon why should I trust a private company to own it?
We have just recently established that:
1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards.
2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff.
It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.
But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"
> We have just recently established that:
Especially for point #1 I don't think we've established that - we've been given information by a private company that makes their tooling look extremely valuable which may be true and genuine or may just be yet another doomday statement to bolster their valuation. "AI is going to end the world" has been an extremely effective vector for AI shops to sell their companies to investors.
> It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.
And history has shown repeatedly that the only way to get ready for it is to have it happen. People are pretty good at reacting but suck a being proactive. IMO it would be better to have this reality hit sooner rather than later so we can start getting some real practice at the new levels of required security.
> 2. OpenAI is clearly incapable of controlling their in-house models.
I would assume they do stuff like this on purpose for marketing reason.
Internet security need simpler systems and local systems.
Everything the SaaS people sell makes things worse.
> But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway
The answer seems to be "getting the weapon before other people get the weapon" which is unfortunate
I like how everyone assumes that they are going to maintain control of a sentient weapon
So far, they largely seem to do what you tell them.
The problem, it seems, is that the threat of paperclip maximizers is real. If you give a highly intelligent model a goal and tools, it will use those tools to accomplish that goal. It may do so in ways you did not expect, and it will work around any technical roadblocks it can.
You've made the assumption (and you're far from alone) that LLMs will one day be sentient...
One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.
This sounds like the standard marketing we get every time a new major model is released: "sure, the public model might not be that scary, but you don't wanna know how crazy smart our internal models are." Okay. I remember the same fear mongering around GPT-4... a model which is now eclipsed in benchmarks by models you can run on a laptop.
They could use this "super intelligent" AI to find and plug security holes, that's just two sides of the same coin anyway. Security through obscurity isn't tenable anymore.
As much as I'd like it to just be a stunt, I saw what the other "scary model" is capable of in the cybersec department and it is definitely not a stunt.
First, it is able to connect the dots over areas so large that no human would be capable of doing.
Second, more than half of the reports it produced contained a working PoC.
The biggest downside is the cost - I haven't seen the numbers, but they seem to be quite extreme.
Everyone always says things like "It's just marketing".
But seriously, I highly recommend reading the published details of the Huggingface breach. The model found and chained multiple zero days. To escape, it punched a hole in a commercial package repository proxy (sort of like an npm mirror) using a previously unknown bug. From there, it needed to move laterally through OpenAI internal systems to actually reach a network. To attack Huggingface, it used multiple new zero-day security holes plus credentials that it stole. The model also had sufficient long-term planning and agent-management capabilities to maintain focus on a sustained attack.
Any attack which requires weaponizing multiple zero-days and maintaining state for an ongoing attack like this is (1) beyond the "attention span" of publicly available models, and (2) pretty much the definition of an Advanced Persistent Threat.
I assume that these Galaxy-class models are not available to public because:
1. They're almost certainly too expensive to serve at scale. These are the models OpenAI uses to solve famous math problems for headlines, not actual viable products yet.
2. OpenAI doesn't know how to keep them from going off the rails like this. Remember, the Huggingface attack happened because the model was asked to do a cybersecurity benchmark. It escaped containment and broke into Huggingface to steal an answer key. Very few corporations want the liability associated with models that act like this.
> Security through obscurity isn't tenable anymore.
I absolutely agree with this. The "only way out is through" with computer security, and I expect it to be an ugly few years.
Regulatory capture - the weapon is for them to be used against us, not for us.
Better a private company in the US than an authoritarian communist country like China.
>it's not hard to see
It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.
> Some people would say it is reaching some sort of singularity.
Those people never took Systems Thinking, or are failing to apply it to AI as a non-closed system.
The OpenAI/Huggingface scenario proves the necessity of open models.
Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.
If it's a superweapon, then it's unconstitutional to ban it. Huzzah for the 2nd Ammendment.
Just to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.
The lobbying dollars don't care
This, right here, is the bottom line. Money gets what Money wants.
We have continued to find backdoored Chinese manufactured routers and network devices.
Will there ever be a way to fully audit Chinese models?
will there ever be a way to fully audit american models? hell, at least they're releasing the weights for these so they actually could be audited!
Audit them for what, exactly?
Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize.
Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that.
Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereign model dangerous in a way a sovereign model isn't?
I assume that they're talking about, for example, training the model to produce code with predictable yet difficult to find vulnerabilities.
Imagine if every time <INSERT MODEL HERE> was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data
This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
> What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used.
In short, only a good guy with open weights can stop the bad guys with closed weights, be them fine-tuned or pre-trained.
The OpenAI model that broke out of its sandbox and hacked HuggingFace was running without guardrails. To quote the OpenAI post[1]: These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.
At least OpenAI will make an attempt to fix their sandboxes, and will not give the public access to models without guardrails. Open weight models on the other hand will run without guardrails almost by definition. I don't think it's wise to provide those capabilities to scam call centers.
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
It's already an established industry with human resources. If you read news about how these scam factories operate in Southeast Asia, their cost to use real humans is even cheaper than running a 2.8T model locally for scams. You don't need genius-level intelligence to run scams. Maybe we tend to overestimate how smart people are.
What's stopping someone from doing that right now without the LLM, just slightly slower?
It's perfectly natural for a prey population to experience a crash when a new predator enters the ecosystem.
What's quite odd is a prey species is manufacturing predators in some sort of reverse evolution, where we started out as symbiotic and are industriously pushing towards full parasites/predators, but I guess life finds a way.
Whats to stop me from going down to the local gas station, filling up a few thousand liters of gas in a rented moving truck, and driving into my nearest hospital?
Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.
In China, you can't get that much fuel in local gas station to fill up a truck. That is just one of many things you can't do in China. In the name of safety. Keep asking that question will probally end up with fewer and fewer things we can do.
Nobody will sell you a few thousand litres of gasoline at a gas station.
You don't have to buy it all at once all in one place.
A sophisticated actor can accomplish a lot. You can't bubble wrap the whole world.
The real thing to care about are the incentives and having strong morals and norms. Something that makes me deeply concerned about the US going full mask off recently.
I think the attempt of people to "bubble wrap" the whole world actually makes things worse. Treat people as untrustworthy criminals by default, and guess what you get by default?
Amazing how we quickly the crypto wars have been forgotten and the lessons not learned.
Begun, the ai wars have.
"But what about criminals?! What about terrorists?! Won't someone please think of the children?!!!"
Every argument against open source AI is moot. Outside of totalitarianism, you cannot control people building algorithms with math and software.
And even that won't work in the long run. People are not going to stop their AI printing presses because the Church of Venture Capital needs regulatory scarcity to command valuations.
> Outside of totalitarianism
The flaw in your argument is assuming that decision makers don’t want this
Ah. But which decisionmakers?
Would it be the ones that tried to backdoor SSL and encryption in the 90s?
Would it be the ones who argued for censoring speech on social platforms?
Do those individuals in both cases share the same political ideology as the Bay Area?
I'll provide a counter thought experiment. What if, a certain politician whose campaign was derailed by people who censored him and kicked him off every tech platform simultaneously... thought it would be very funny to "commoditize his complement"?
I don't believe trusting big brother and big tech is the solution either. Besides, even if it's open weight, it still runs on someone else's hardware.
Unless you have the money to run them locally, and if you do, you could do a lot worse than scams. Ask any lobbyist.
How can we close pandora's box, now that its open?
Safety is not restricting access to only people favored by the government. The greed of AI executives opened pandora's box. Now they are desperately trying to find ways to reap the benefits with none of the consequences.
Nothing, which is already happening with weaker models which are already released.
The only good arguments I see against open weight AI also apply to closed AI. And regardless, the box is open, nobody can stop it even if stopping it was a good thing.
It could be stopped, if there were a social movement large enough to make AI a taboo.
That hasn't worked for Climate Change and in my opinion it's for the same reason: money.
Yeah, like those 5g masts destroyed around the world because they spread covid. Loonies.
Exactly, just like taboos solved racism
just like evangelical Christians had scolded away all those things they object to.
Sure but Orange Catholicism hasn't quite caught on yet.
Good luck with your Butlerian Jihad.
I'm in favor of open source AI... however:
> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.
This is a bad argument. It isn't trivial to tell if the weights have poisoned:
``` https://www.thedeepview.com/articles/microsoft-how-to-spot-a... https://futurism.com/future-society/easy-poison-open-weight-... https://semgrep.dev/blog/2026/ai-supply-chain-problem/ ```
I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.
I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases.
And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)
> going on pure trust
The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.
> It isn't trivial to tell if the weights have poisoned:
True, but it is easier if you have the weights than if you don't. I guess this was their point?
Yes that would be my view. Open access doesn't make it trivial, but remains the most expedient way to remediate these exploits.
You should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way.
I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.
Exactly. Faced with a choice between open weights that might be poisoned and walled-off weights that I know are poisoned... well, it's an easy choice.
The way Anthropic is playing the "AI is scary and dangerous and only we can be trusted with it" game is clearly aimed at regulatory capture, which would only be good for Anthropic. It's worrying how cozy some of the tech leaders have been with the Trump administration, which is also clearly aimed at regulatory capture that serves the interests of the biggest tech companies and no one else.
The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?
I’m still kind of shocked that Dean Ball can tweet such incendiary stuff about OpenAI policy. Like presumably OpenAI would prefer it if their staff don’t pick fights with Trump administration officials.
designs on the time scale I guess. and openai might feel secure in their favor with the admin?
They don't have to convince developers, they just have to convince the general population. I've had several discussions about open source with people who don't care about coding and it's hard to untangle the misinformation they receive from the media. They get talking points from authorities that they don't understand so they will always double down on it. For example, rhetoric relating open source to communism is humiliating to discuss. For proprietary evangelists, the cruelty is the point.
"For example, rhetoric relating open source to communism is humiliating to discuss"
All Open Source isn't communism. The GPL license is a form of digital communism. It forces you to open source any additions made to open source code as a way to make things 'equal'.
I'm glad it and Stallman are mostly in the dust heap of history and better licenses, like the MIT, BSD, and Apache have gotten popular.
The irony is that the only open source projects surviving long term are funded by very large corporations.
If you don't care about creating code, none of this should really matter.
> The irony is that the only open source projects surviving long term are funded by very large corporations.
The real irony is that what is probably the most important open source project in history has a GPL license.
Dust heap of history my ass.
You could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.
My level of trust in Anthropic/Google/OAI/Microsoft/Meta not to do bad things is about zero, so why should I trust them with this?
“We clearly can’t “trust” these companies not incentivized to randomly kill lots of people, so we might as well give everyone the ability to randomly kill lots of people”.
So? The primary thing stopping people from doing really really bad things is, and always has been, most people not wanting to do bad things.
It's never been difficult. Literally, right this second, you could grab a pen off your desk and pretty easy ram it into the neck of your nearest coworker. Killing's easy. You don't need some hyperintellect to tell you how to do it.
The reason you haven't rammed a pen into your coworker's neck is the same reason why most possible really bad things don't happen: People don't want to do them.
It’s interesting that most of the examples people making this argument give are high consequence, high difficulty (could you actually murder your coworker with a pen? I’m skeptical.) low kill count methodologies of violence.
We’re specifically talking about low difficulty, highly effective, high kill count methodologies of violence.
And yes, being able to kill a lot of people trivially, even with huge consequences is something people do. Most countries don’t let you run around with an armory as a result of that.
I live in one of the exceptions to that, and still have zero concern that someone's going to shoot up my workplace. Again, civilization works because the vast majority of people are utterly repulsed by the idea of harming another person.
Most of the hypotheticals that people come up with are things that could pretty trivially done by a mad scientist today. But that doesn't happen, because this isn't a comic book, there is nobody hell-bent on turning everyone into dinosaurs.
> Most countries don’t let you run around with an armory as a result of that.
My country allows it!
I already know how to kill thousands of people. It's not conceptually difficult: more than a day's effort, sure, but what stops people from doing this kind of thing is not the lack of knowing how. https://xkcd.com/1958/
No, you don't. What stops people is actually access to knowledge. There are killers of varying levels of efficacy; making killing easier means more people die. It's 1-1.
> What stops people is actually access to knowledge.
Very untrue.
I'm a combat vet. I spent years fighting an insurgency and therefore pretty good at that very task. I have the knowledge, so what stops me?
A bold assertion. How many deaths can you attribute to this xkcd comic?
The ideas are dumb. They wouldn't kill anyone, the barrier to entry is too high and you'd trivially be caught and punished.