I watched a video from him recently where he said open source models would win people's trust when it comes to AI. But I struggle to reconcile this with the above picture; if open source models become the norm, I feel P(Doom) probably will go up rather than down. At least the frontier AI firms have some guardrails in place (although after the Hugging Face hack they clearly aren't great). a wild west of everyone running their own model is surely worse in that regard.
A bunch of sites getting hacked isn't an existential threat. An existential threat would be something that could viably imperil the survival of civilization. The problem with existential threat concepts is that it generally requires fields where the limiting factor isn't knowledge. If somebody has the intelligence, resources, and desire to e.g. try to build a deadly biological virus, it's certainly not the lack of knowledge that's stopping them.
In general if we go back 50 years ago, when a single common phone today would have been seen as a super-computing device of unimaginable power, governments would likely have viewed it as a potential world-ending technology. It obviously wasn't and isn't. The conservatism would come solely from the fact of it being a revolutionary leap. That's all we're seeing happen with LLMs as well. In 30 years everybody will have vastly more competent AI systems running locally on whatever sort of digital devices we're using then, and they'll look back at with bemusement about us freaking out about people being able to query computers in natural language.
- countries start deploying GPT6+ - level ai in many local datacenters (let's say to process administrative stuff in one hopital / adminstration or even full city).
- some guys publish an open source way to remove the guardrails (either by post training or just nn level hack)
- you get possibly many local datacenters with a 'hugging face hacking' ability
- some dude ask "hey plz hack the military"
that probably still dont mean we should ban open source ai, but a open source model able to hack half the planet computers would be a 'concern' somehow
> but a open source model able to hack half the planet computers would be a 'concern' somehow
Well, a closed source model able to hack half the planet will be a far bigger concern if there aren't enough cheap open source models to harden the existing systems and write software for defense in depth.
> some dude ask "hey plz hack the military"
Military gear, capable of moderate levels of destruction and above, should be air-gaped and also hardened using good open source AI - as a precaution. If that's not the case, state sponsored human adversaries would be able to cause maximum damage even without AI.
Why do you think AI is mandatory here? And why do you think adversarial state actors will be deterred by your banning of your own open AI? They can even promise not to use it, may sign treaties about not using it but enforcing that would still be impossible.
As your parent said, control the tangible agents of destruction, controlling knowledge is the wet dream of many but it's a lost cause.
P(Doom) is an invented concept, relative to ones own version of "doom", can we maybe actually act like adults and not man children who are engaged in performative speculative fiction. There are real problems like inequality that P(Doom) loves to conveniently forget.
> P(Doom) is an invented concept, relative to ones own version of "doom", can we maybe actually act like adults and not man children who are engaged in performative speculative fiction.
I agree with that, especially with some of the more fanciful "extinction" scenarios.
> There are real problems like inequality that P(Doom) loves to conveniently forget.
Extreme inequality is my "Doom", and I think the P(Doom) of that is pretty high. With AI, I'm way more concerned with human marginalization, capitalist logic fucking most of us over, inescapable surveillance and control at scale, etc. than killer robots and bioweapons.
One part is kinda poorly worded in my opinion.
While reading it, I asked myself "will takeover happen September 2027?" due to wording 'this September'. Somehow I missed that announcement and needed some short research to get to know that it was initiated 2 months ago (September 26) and will probably be finished until mid of 27.
"Delangue noted that agent misbehavior has so far mostly come from big companies that are testing frontier AI in secret. He said that future risks, perhaps involving the biological capabilities of models or their capacity to improve themselves recursively, are likely to emerge from these closed labs, too."
Sloppy reasoning. Closed models are on the leading edge. Risk comes from models on the leading edge. That doesn't mean the risk is due to the fact that the models are closed.
"Pit bulls are dangerous. Pit bulls are mammals. You'll be safer if you get a komodo dragon as a pet. It's not a mammal." In this allegory, the mammal observation is a red herring. The key factor is whether the proposed pet is an aggressive animal with sharp teeth.
* * *
As I wrote a few months ago:
>There used to be a thing called "Moore's Law of Mad Science":
>"Every eighteen months, the minimum IQ necessary to destroy the world drops by one point."
>Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
I gave a safety talk at an ML conference that also had a skeptical representative from HuggingFace. Like Delangue, he didn't seem to have seriously considered the threat model--myopically focused on the capabilities of current agents (a myopia which, while less excusable then than in 2015 or 2021; was at least more excusable than Delangue's in this article since HF hadn't yet been pantsed by an agent swarm).
It's not that long ago that sites were pantsed by bored college students, rather than agents. It didn't stop because sites got more secure really, it stopped because those students started getting real life consequences.
Gives you a real easy solution to stop the agents too.
That's such an odd thing for him to say while also not pursuing a legal case against OpenAI. If he believes agentic cyber attacks should be illegal he's in a perfect spot to have courts rule on that.
Is there any serious argument that agentic cyberattacks are legal?
Not every wrong needs to be met with maximum punishment. Pressing charges or suing in civil court might get OpenAI to do some minimum, but a lot of the time, you can get someone to do more than the minimum if you approach them more cooperatively.
I'm aware of no precedent for agentic attacks either way, though I'm not sure why the distinction would even matter legally. The corporations running those tools is the legal entity responsible, there is no legal entity called an AI agent.
If a law is never enforced is it really illegal? I'm thinking of drug laws in certain states/countries where the local authorities have just decided "nah, we won't arrest anyone for that..."
I watched a video from him recently where he said open source models would win people's trust when it comes to AI. But I struggle to reconcile this with the above picture; if open source models become the norm, I feel P(Doom) probably will go up rather than down. At least the frontier AI firms have some guardrails in place (although after the Hugging Face hack they clearly aren't great). a wild west of everyone running their own model is surely worse in that regard.
A bunch of sites getting hacked isn't an existential threat. An existential threat would be something that could viably imperil the survival of civilization. The problem with existential threat concepts is that it generally requires fields where the limiting factor isn't knowledge. If somebody has the intelligence, resources, and desire to e.g. try to build a deadly biological virus, it's certainly not the lack of knowledge that's stopping them.
In general if we go back 50 years ago, when a single common phone today would have been seen as a super-computing device of unimaginable power, governments would likely have viewed it as a potential world-ending technology. It obviously wasn't and isn't. The conservatism would come solely from the fact of it being a revolutionary leap. That's all we're seeing happen with LLMs as well. In 30 years everybody will have vastly more competent AI systems running locally on whatever sort of digital devices we're using then, and they'll look back at with bemusement about us freaking out about people being able to query computers in natural language.
I guess the target doom scenario is
- countries start deploying GPT6+ - level ai in many local datacenters (let's say to process administrative stuff in one hopital / adminstration or even full city).
- some guys publish an open source way to remove the guardrails (either by post training or just nn level hack)
- you get possibly many local datacenters with a 'hugging face hacking' ability
- some dude ask "hey plz hack the military"
that probably still dont mean we should ban open source ai, but a open source model able to hack half the planet computers would be a 'concern' somehow
> but a open source model able to hack half the planet computers would be a 'concern' somehow
Well, a closed source model able to hack half the planet will be a far bigger concern if there aren't enough cheap open source models to harden the existing systems and write software for defense in depth.
> some dude ask "hey plz hack the military"
Military gear, capable of moderate levels of destruction and above, should be air-gaped and also hardened using good open source AI - as a precaution. If that's not the case, state sponsored human adversaries would be able to cause maximum damage even without AI.
Why do you think AI is mandatory here? And why do you think adversarial state actors will be deterred by your banning of your own open AI? They can even promise not to use it, may sign treaties about not using it but enforcing that would still be impossible.
As your parent said, control the tangible agents of destruction, controlling knowledge is the wet dream of many but it's a lost cause.
P(Doom) is an invented concept, relative to ones own version of "doom", can we maybe actually act like adults and not man children who are engaged in performative speculative fiction. There are real problems like inequality that P(Doom) loves to conveniently forget.
> P(Doom) is an invented concept, relative to ones own version of "doom", can we maybe actually act like adults and not man children who are engaged in performative speculative fiction.
I agree with that, especially with some of the more fanciful "extinction" scenarios.
> There are real problems like inequality that P(Doom) loves to conveniently forget.
Extreme inequality is my "Doom", and I think the P(Doom) of that is pretty high. With AI, I'm way more concerned with human marginalization, capitalist logic fucking most of us over, inescapable surveillance and control at scale, etc. than killer robots and bioweapons.
One part is kinda poorly worded in my opinion. While reading it, I asked myself "will takeover happen September 2027?" due to wording 'this September'. Somehow I missed that announcement and needed some short research to get to know that it was initiated 2 months ago (September 26) and will probably be finished until mid of 27.
"Delangue noted that agent misbehavior has so far mostly come from big companies that are testing frontier AI in secret. He said that future risks, perhaps involving the biological capabilities of models or their capacity to improve themselves recursively, are likely to emerge from these closed labs, too."
Sloppy reasoning. Closed models are on the leading edge. Risk comes from models on the leading edge. That doesn't mean the risk is due to the fact that the models are closed.
"Pit bulls are dangerous. Pit bulls are mammals. You'll be safer if you get a komodo dragon as a pet. It's not a mammal." In this allegory, the mammal observation is a red herring. The key factor is whether the proposed pet is an aggressive animal with sharp teeth.
* * *
As I wrote a few months ago:
>There used to be a thing called "Moore's Law of Mad Science":
>"Every eighteen months, the minimum IQ necessary to destroy the world drops by one point."
>Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
Lots of discussion in the thread: https://news.ycombinator.com/item?id=49606803
I gave a safety talk at an ML conference that also had a skeptical representative from HuggingFace. Like Delangue, he didn't seem to have seriously considered the threat model--myopically focused on the capabilities of current agents (a myopia which, while less excusable then than in 2015 or 2021; was at least more excusable than Delangue's in this article since HF hadn't yet been pantsed by an agent swarm).
is the threat model something you put into a ML or nah?
> HF hadn't yet been pantsed by an agent swarm
It's not that long ago that sites were pantsed by bored college students, rather than agents. It didn't stop because sites got more secure really, it stopped because those students started getting real life consequences.
Gives you a real easy solution to stop the agents too.
P(Doom) is pretty high on most hardware.
One optimistic outcome is that this will mean the end of IoT nonsense. There's no reason for your freezers to be controllable over the internet [0]
Adama: "It's an integrated computer network, and I will not have it aboard this ship."
[0] https://www.bbc.co.uk/news/articles/c8vgxyvz6y9ro
Yes:
> Cyber-attacks are illegal, and I think it’s really important we make sure that agentic cyberattacks [are] illegal
That's such an odd thing for him to say while also not pursuing a legal case against OpenAI. If he believes agentic cyber attacks should be illegal he's in a perfect spot to have courts rule on that.
Is there any serious argument that agentic cyberattacks are legal?
Not every wrong needs to be met with maximum punishment. Pressing charges or suing in civil court might get OpenAI to do some minimum, but a lot of the time, you can get someone to do more than the minimum if you approach them more cooperatively.
I'm aware of no precedent for agentic attacks either way, though I'm not sure why the distinction would even matter legally. The corporations running those tools is the legal entity responsible, there is no legal entity called an AI agent.
If a law is never enforced is it really illegal? I'm thinking of drug laws in certain states/countries where the local authorities have just decided "nah, we won't arrest anyone for that..."
Those situations have never bothered me. Legislators should take it as a clear indication those laws should simply be repealed.