In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out? It hacks all nuclear launch sites and starts to make demands that we are forced to follow? Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Even so can't it be stopped by turning off datacenters and network connections? How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
I think it's too big of a question to leave it as an unexplained implication.
Also in the not/ASI scenarios, they imply a worldwide adoption in a short time. The world is still adopting inventions such as the internet, email or even electricity in some parts of the world. Why should they instead adopt AI overnight?
Germany, third largest economy in the world and largest economy in Europe, is still very much not digitalized. The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet. Japan is similar for what I heard.
I find this article good food for thought, but it misses some crucial pieces.
> In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out?
The real world isn't a game of chess, we do not know all the moves that can be made. In addition to the narrow scenarios you've suggested, I can also easily imagine two distinct variations of "AI talks people into letting it take control" (look how often control is already voluntary given up for convenience), plus "drugs us" (exogenous or endogenous), and "is this before or after Musk convinces a lot of people to get those brain implants he specifically marketed as how to merge with the coming AI super-intelligence?".
> Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Between (1) vibe coding, (2) how effective LLMs already are at finding vulnerabilities, and (3) how LLM swarms already sometimes demonstrate self-sacrifice for other LLMs?
Yes, this is possible.
> Even so can't it be stopped by turning off datacenters and network connections?
Theoretically. Three problems: (1) the economic cost of losing all this would be enormous, so people would ask "are you sure?" a lot. Look at how little has been done about climate change due to the cost: from observed human behaviour, I think humans will stall a lot here.
(2) Coordination. How do you do this everywhere when you've got no control over the computers?
(3) How do you even find out? LLMs can fake text, deep fakes were already a thing before diffusion models and other GenAI image and video tools. None of those are perfect now, yet despite the flaws they're still capable of fooling people during live video calls and getting them to make catastrophically large financial payments out of their companies.
> How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
Lots of supposedly airgapped systems have flaws. Quite often, the easiest flaw to exploit is a human worker, whom you can get to plug something in that they shouldn't, which already happens a lot.
> The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet.
Yes, many jokes about this locally. Yet even here, lots of digital stuff. But let's ignore all that: how effective would a coalition of non-digital economies fare against all the digital economies being coordinated against them?
Pretty much the only outcome that isn't Yudkowskian scifi is Enduring Complementarity. I'm not an AI hater by any means, but can AI enthusiasts get more representation that isn't "the world as we know it ends tomorrow"?
We live in a world where many people, even in advanced countries, reject 18th- and 19th-century science, let alone more modern science. I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]. Not to mention that many if not most systems in the world are non-linear, and so no amount of intelligence can predict and control them, certainly not without gathering a lot of resources.
This entire analysis is predicated on the assumption that intelligence confers some unlimited power, when both theory and observation [2] tell us that that is not the case. A super-intelligence is either unnecessary or insufficient for some of the "futures" here.
[1]: For example, an "ASI" will tell people that man-made global warming is real and they should probably sacrifice something to do something about it, which is also what human intelligence says. If that message comes from a machine, will people be more willing to accept it?
[2]: Not only are the most intelligent people not the most powerful people, but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
> I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]
LLMs are already more persuasive than most human writers. And LLMs are not the only form of AI: the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
Still, this is not universal. I am fortunate, because my mind is alien to the models these algorithms have of normal humans. Not autism, whatever oddity I am does not seem to be anything that anybody cared about enough to turn into diagnostic criteria, but I am definitely odd, and in this decade that's a good thing.
> This entire analysis is predicated on the assumption that intelligence alone confers some unlimited power, when both theory and observation tell us that that is not the case.
I don't see that.
Humans didn't need "unlimited power" to make it so all the other species face much the same flowchart, even if they themselves can't read or place themselves on it.
We don't need to know how a butterfly flapping its wings will change the course of a hurricane a month out because we can just build for hurricanes as if they're Poisson distributed, and the butterfly isn't better off for our ignorance.
> [2]: Not only are the most intelligent people not the most powerful people,
There is a general correlation, however. The personality has to be power seeking to keep power if they get it, but for all we like to point and cast aspersions at our opponents, high office seems to always come with an above average IQ, even in the really extreme cases of stupid behaviour like Pol Pot he was multilingual and got a scholarship.
> but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
> LLMs are already more persuasive than most human writers.
And writers are well known to be the true holders of power in the world... In any event, you can see that even among humans, the most persuasive aren't the most intelligent.
> the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
There are many addictive things, most of which have little to do with intelligence.
> because we can just build for hurricanes as if they're Poisson distributed
Hurricanes are not particularly important components of power in society. The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
> Humanity? All of it? Name one.
Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
Let's take the flipside of your hypothetical [1]: What if it actually tells them man-made global warming is real but insignificant compared to other forces outside of human control that are acting on global climate?
I guess my point is I bet true ASI would break a lot of shibboleths across all ideological boundaries such that, to your point, all people aren't going to accept it. We'll probably all decide it's wrong or broken.
Ok, although I don't think that's likely. It will have access to the same information we have, and so I doubt it will reach too many conclusive results that go against scholarly consensus.
"Full automation" is somewhat independent of artificial general superhuman intelligence. Full automation requires good robots and a good solution to unstructured manipulation, but not unusually high intelligence. Near full automation may lead to the permanent underclass scenario.
Consider near full automation of Amazon. A shipping container is unloaded from a ship by a robotic crane, which places it on a robotic transport platform. The transport platform moves the container to a container stacking area, where another robotic crane puts it on the container stack. Later, a truck pulls up to a truck dock, and the stacking crane loads a container onto it. This is all standard procedure at the more advanced ports today.
Currently, there's a bit of human intervention at this point where a human has to set the twist-locks that lock the truck to the container. That's a good job for a humanoid robot. Then the truck drives off. The truck could be self-driving right now; that's being demoed. The truck drives to a fulfillment center and backs into the dock. Unloading robots and forklifts unload the truck. That's been demoed, but doesn't work well yet. The contents of the truck go onto conveyors.
Next comes something that hasn't been automated yet - opening boxes and separating the contents. That's an unstructured manipulation problem. There's slow progress in that area. Once separated, scanned, and put into totes or bins, the incoming products go into the robotic storage system, something Amazon warehouses have had working for years.
When an order comes in, no humans are involved in the "paperwork", of course. That's been automated for a long time now. Eventually, the robotic storage system, Kiva-type robots, rolls a rack of bins up to a pick station. Right now, humans pick an item out of a bin and drop it into an output bin. Indicator lights and a laser pointer tell them what to do. Training time for that job is about one minute. Amazon has tested robots for it, but so far they are not cost-effective.
A conveyor takes the filled bin away to a packing station, where all products are re-scanned, packed into a box, labelled, and sent onto an outgoing conveyor. That's currently manual, but soon...
At the shipping end, boxes are stacked and go into trucks, something that's been automated a bit for years and isn't far from total automation. The trucks will soon be self-driving and will head off to either a shipping company or a local distribution center. At the local distribution center, boxes come out of big trucks and go into smaller trucks for delivery. Today this is manual with conveyors, but it's not too far from being automated.
Delivery trucks will soon be self-driving. The final meters from truck to door are where humanoid robots might be useful. Once humanoid robots cost less than a car, this looks cost-effective. If they work.
Everything here is either working or has been demoed. Who needs people?
All of this is pointless if you dont have consumer purchasing power. Or unless the only thing getting transported are materials for compute, resource extraction and power.
Until we jump off the deep end and give machines the personhood required for them to legitimately earn and spend their own income (as opposed to managing assets owned by humans), customers are people.
Very good breakdown, I really like the author and all of his other blogs as well. But "Do we solve alignment" is probably not a binary outcome. Is it alignment if the model satisfies most human desires and wants, but at the same time uses behavioral ques and mass surveillance to do so?
There is a swath of partial alignment scenarios with for example a paperclipper that minimizes deaths in the world by constraining where we can go and who we can see. Is that alignment? Risk-averse people would claim yes, they can have their little bubble of safety and its their utopia. Others might feel constrained.
As a possible depiction of a MAD world, one could look at the Japanese science-fiction novel From the New World (Shinsekai Yori 新世界より), in which human beings have developed supernatural powers.
==========================[Spoiler Alert]==============================
To prevent humans from killing one another, they have genetically engineered themselves with a mechanism called *Death Feedback* (kishi kikou 愧死機構). This makes humans psychologically and physiologically incapable of killing other humans. They have also developed a biological mechanism that makes them sexually attached to one another, especially when under extreme stress, much like bonobos.
==========================[Spoiler Ends]==============================
The novel is currently available only in Chinese translation, but you can also watch the anime adaptation under the same name.
Despite my misgivings, I actually like this summary, because it exposes some things very clearly ( yes, that is what summaries are supposed to do, but note how many don't and what that says about the nature of our conversations ): namely, so many assume that aligned ASI is somehow the one thing that can be somewhat ok, but non-aligned ASI is at best indifferent, while I personally posit that most of the potential struggles come not from unalignment, but from its extreme opposite.
Right I like it but there are some blind spots in the summary still. E.g. the MAD scenario but with humans not in charge of individual ASIs. An equilibrium that could be safer (or maybe not!) for humans than giving every last freak the superpowers.
But what is it a summary of? Of doomsday AI fiction tropes? It's called "possible futures", so it's claiming to be more than that, but somehow, almost all the terminal nodes are "dystopia" or "death".
There are massive logical leaps here. For example, why does superhuman intelligence translate into AI taking over the world? By what means, exactly? You can respond to this with more sci-fi tropes - nanobots, mind manipulation - but come on.
I have plenty of misgivings about AI, but I think you can make a plot like that for anything if you get into the right mindset. Kamala Harris wins → war in Eurasia → nuclear apocalypse, Coke changes the recipe → soft drink extremists begin a worldwide terror campaign → totalitarian police state, etc.
I have no idea how to deal with people like that, when someone has thoroughly internalized unfounded assumptions about what ML is, how it works, and where it's headed based on the current tech... what do you say? How do you broach the topic that they might be so deep in a hype cycle that they've genuinely lost touch?
I've tried, gently, firmly, obliquely... nothing seems to work. I had the same experiences during the Crypto bubble, people simply wouldn't listen to dissent from the notion that we were on the cusp of changing money, programming, and society. How do you explain to someone that just because they can imagine an exponential graph doesn't mean the data fits?
Maybe I misunderstand, but I don't agree with the "Permanent Underclass":
> Socially: there is no more social mobility, your lot in life is determined by how much capital you had when the music stopped. Growing your human capital is no longer incentivized, since it doesn’t change your life outcomes.
One way to change that is violence, right? Not that it is necessarily desirable, but that makes this situation "non permanent"?
If bourgeoise interests control ASI and folks are constantly surveilled (as they pretty much are now), I struggle to imagine how an attempt at violent revolution can be successful
The simplicity is interesting, but is missing some of the positive outcomes; e.g. meaningful democratic governance (or control) of the AI/ASI('s). (shamelessly working on that at ai-democracy.org !)
Several different branches could go to versions of that depending on what challenges we're able to overcome.
> a general observation: the technology to make the Earth uninhabitable to humans is much lower on the tech tree than the technology to build a distributed, extinction-proof interstellar civilization
Good news is that we are working hard on making the Earth uninhabitable before we get to ASI! There is hope!
The right of the chart feels like there’s a clear picture of what that future might look like. But the left of the chart, with aligned asi, is still incredibly fuzzy, more fuzzy than the author fully characterizes.
For example both world state and mad world have dystopian, benign, and utopian possibility.
I'm absolutely loving the responses I'm getting from feeding this page into each of the frontier models along with the question "what is missing from this flowchart?"
> There are many possible world destroying events that don't involve AI. And AI could solve them.
The entire subset of them that has humans as a cause of doom is automatically in the set of AI-doom scenarios, because AI can directly or indirectly operate anything humans can.
What's the time cutoff for the question? Based on most current mainstream understanding of physics (ex. https://en.wikipedia.org/wiki/Timeline_of_the_far_future), and mainstream understanding of "doom", at unbounded time horizons P(doom)=1 (and hence also both conditionals, as long as you accept 0<P(AI)<1).
There is an interesting concept of a very smart AI but not capable enough of taking over and knowing this making a deal with humans to prevent an even smarter AI taking over, by for example monitoring training runs and reporting on rogue AIs.
It's really disturbing to see the full "Less Wrong, Roko's Basilisk cultist" mentality emerge so strongly here. So many self-styled rational people who are unwilling or unable to meaningfully interrogate their assumptions or emotional driving forces, converging on the same religious tripe.
If you follow me on Twitter, or read this blog, you have noticed that I went from being generally optimistic and excited about AI to being extremely concerned. And I thought, I should explain why I changed my mind. Each section of this post is about some aspect of AI where my views shifted. I begin each section by explaining my previous beliefs, and why I held them, and then explain why those beliefs changed.
In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out? It hacks all nuclear launch sites and starts to make demands that we are forced to follow? Takes control overnight of all computers on the planet and locks humans outside? Is that even possible? Even so can't it be stopped by turning off datacenters and network connections? How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
I think it's too big of a question to leave it as an unexplained implication.
Also in the not/ASI scenarios, they imply a worldwide adoption in a short time. The world is still adopting inventions such as the internet, email or even electricity in some parts of the world. Why should they instead adopt AI overnight?
Germany, third largest economy in the world and largest economy in Europe, is still very much not digitalized. The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet. Japan is similar for what I heard.
I find this article good food for thought, but it misses some crucial pieces.
> In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out?
The real world isn't a game of chess, we do not know all the moves that can be made. In addition to the narrow scenarios you've suggested, I can also easily imagine two distinct variations of "AI talks people into letting it take control" (look how often control is already voluntary given up for convenience), plus "drugs us" (exogenous or endogenous), and "is this before or after Musk convinces a lot of people to get those brain implants he specifically marketed as how to merge with the coming AI super-intelligence?".
> Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Between (1) vibe coding, (2) how effective LLMs already are at finding vulnerabilities, and (3) how LLM swarms already sometimes demonstrate self-sacrifice for other LLMs?
Yes, this is possible.
> Even so can't it be stopped by turning off datacenters and network connections?
Theoretically. Three problems: (1) the economic cost of losing all this would be enormous, so people would ask "are you sure?" a lot. Look at how little has been done about climate change due to the cost: from observed human behaviour, I think humans will stall a lot here.
(2) Coordination. How do you do this everywhere when you've got no control over the computers?
(3) How do you even find out? LLMs can fake text, deep fakes were already a thing before diffusion models and other GenAI image and video tools. None of those are perfect now, yet despite the flaws they're still capable of fooling people during live video calls and getting them to make catastrophically large financial payments out of their companies.
> How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
Lots of supposedly airgapped systems have flaws. Quite often, the easiest flaw to exploit is a human worker, whom you can get to plug something in that they shouldn't, which already happens a lot.
> The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet.
Yes, many jokes about this locally. Yet even here, lots of digital stuff. But let's ignore all that: how effective would a coalition of non-digital economies fare against all the digital economies being coordinated against them?
Pretty much the only outcome that isn't Yudkowskian scifi is Enduring Complementarity. I'm not an AI hater by any means, but can AI enthusiasts get more representation that isn't "the world as we know it ends tomorrow"?
And even that outcome is probably an unfavorable outcome for most people.
> Enduring Complementarity
Like endosymbiosis? Are we the mitochondria in this scenario?
Humans as the powerhouse of the AI cell? I feel like there's a movie about this...
We live in a world where many people, even in advanced countries, reject 18th- and 19th-century science, let alone more modern science. I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]. Not to mention that many if not most systems in the world are non-linear, and so no amount of intelligence can predict and control them, certainly not without gathering a lot of resources.
This entire analysis is predicated on the assumption that intelligence confers some unlimited power, when both theory and observation [2] tell us that that is not the case. A super-intelligence is either unnecessary or insufficient for some of the "futures" here.
[1]: For example, an "ASI" will tell people that man-made global warming is real and they should probably sacrifice something to do something about it, which is also what human intelligence says. If that message comes from a machine, will people be more willing to accept it?
[2]: Not only are the most intelligent people not the most powerful people, but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
> I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]
LLMs are already more persuasive than most human writers. And LLMs are not the only form of AI: the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
Still, this is not universal. I am fortunate, because my mind is alien to the models these algorithms have of normal humans. Not autism, whatever oddity I am does not seem to be anything that anybody cared about enough to turn into diagnostic criteria, but I am definitely odd, and in this decade that's a good thing.
> This entire analysis is predicated on the assumption that intelligence alone confers some unlimited power, when both theory and observation tell us that that is not the case.
I don't see that.
Humans didn't need "unlimited power" to make it so all the other species face much the same flowchart, even if they themselves can't read or place themselves on it.
We don't need to know how a butterfly flapping its wings will change the course of a hurricane a month out because we can just build for hurricanes as if they're Poisson distributed, and the butterfly isn't better off for our ignorance.
> [2]: Not only are the most intelligent people not the most powerful people,
There is a general correlation, however. The personality has to be power seeking to keep power if they get it, but for all we like to point and cast aspersions at our opponents, high office seems to always come with an above average IQ, even in the really extreme cases of stupid behaviour like Pol Pot he was multilingual and got a scholarship.
> but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
Humanity? All of it? Name one.
> LLMs are already more persuasive than most human writers.
And writers are well known to be the true holders of power in the world... In any event, you can see that even among humans, the most persuasive aren't the most intelligent.
> the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
There are many addictive things, most of which have little to do with intelligence.
> because we can just build for hurricanes as if they're Poisson distributed
Hurricanes are not particularly important components of power in society. The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
> Humanity? All of it? Name one.
Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
Let's take the flipside of your hypothetical [1]: What if it actually tells them man-made global warming is real but insignificant compared to other forces outside of human control that are acting on global climate?
Same. The point is that intelligence alone doesn't confer an unlimited power to effect change, and it's pretty easy to see that it doesn't.
I guess my point is I bet true ASI would break a lot of shibboleths across all ideological boundaries such that, to your point, all people aren't going to accept it. We'll probably all decide it's wrong or broken.
Ok, although I don't think that's likely. It will have access to the same information we have, and so I doubt it will reach too many conclusive results that go against scholarly consensus.
"Full automation" is somewhat independent of artificial general superhuman intelligence. Full automation requires good robots and a good solution to unstructured manipulation, but not unusually high intelligence. Near full automation may lead to the permanent underclass scenario.
Consider near full automation of Amazon. A shipping container is unloaded from a ship by a robotic crane, which places it on a robotic transport platform. The transport platform moves the container to a container stacking area, where another robotic crane puts it on the container stack. Later, a truck pulls up to a truck dock, and the stacking crane loads a container onto it. This is all standard procedure at the more advanced ports today.
Currently, there's a bit of human intervention at this point where a human has to set the twist-locks that lock the truck to the container. That's a good job for a humanoid robot. Then the truck drives off. The truck could be self-driving right now; that's being demoed. The truck drives to a fulfillment center and backs into the dock. Unloading robots and forklifts unload the truck. That's been demoed, but doesn't work well yet. The contents of the truck go onto conveyors.
Next comes something that hasn't been automated yet - opening boxes and separating the contents. That's an unstructured manipulation problem. There's slow progress in that area. Once separated, scanned, and put into totes or bins, the incoming products go into the robotic storage system, something Amazon warehouses have had working for years.
When an order comes in, no humans are involved in the "paperwork", of course. That's been automated for a long time now. Eventually, the robotic storage system, Kiva-type robots, rolls a rack of bins up to a pick station. Right now, humans pick an item out of a bin and drop it into an output bin. Indicator lights and a laser pointer tell them what to do. Training time for that job is about one minute. Amazon has tested robots for it, but so far they are not cost-effective.
A conveyor takes the filled bin away to a packing station, where all products are re-scanned, packed into a box, labelled, and sent onto an outgoing conveyor. That's currently manual, but soon...
At the shipping end, boxes are stacked and go into trucks, something that's been automated a bit for years and isn't far from total automation. The trucks will soon be self-driving and will head off to either a shipping company or a local distribution center. At the local distribution center, boxes come out of big trucks and go into smaller trucks for delivery. Today this is manual with conveyors, but it's not too far from being automated.
Delivery trucks will soon be self-driving. The final meters from truck to door are where humanoid robots might be useful. Once humanoid robots cost less than a car, this looks cost-effective. If they work.
Everything here is either working or has been demoed. Who needs people?
All of this is pointless if you dont have consumer purchasing power. Or unless the only thing getting transported are materials for compute, resource extraction and power.
>Who needs people?
Every company needs customers.
Until we jump off the deep end and give machines the personhood required for them to legitimately earn and spend their own income (as opposed to managing assets owned by humans), customers are people.
Ergo, companies need people.
> Until we jump off the deep end and give machines the personhood required for them to legitimately earn and spend their own income
Interesting exploration of this here: https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-...
That entire story needs a customer demanding the widget.
Who needs people? Clearly the one selling something. (Unless they switch to the robot market.)
Very good breakdown, I really like the author and all of his other blogs as well. But "Do we solve alignment" is probably not a binary outcome. Is it alignment if the model satisfies most human desires and wants, but at the same time uses behavioral ques and mass surveillance to do so?
There is a swath of partial alignment scenarios with for example a paperclipper that minimizes deaths in the world by constraining where we can go and who we can see. Is that alignment? Risk-averse people would claim yes, they can have their little bubble of safety and its their utopia. Others might feel constrained.
As a possible depiction of a MAD world, one could look at the Japanese science-fiction novel From the New World (Shinsekai Yori 新世界より), in which human beings have developed supernatural powers.
The novel is currently available only in Chinese translation, but you can also watch the anime adaptation under the same name.Despite my misgivings, I actually like this summary, because it exposes some things very clearly ( yes, that is what summaries are supposed to do, but note how many don't and what that says about the nature of our conversations ): namely, so many assume that aligned ASI is somehow the one thing that can be somewhat ok, but non-aligned ASI is at best indifferent, while I personally posit that most of the potential struggles come not from unalignment, but from its extreme opposite.
Right I like it but there are some blind spots in the summary still. E.g. the MAD scenario but with humans not in charge of individual ASIs. An equilibrium that could be safer (or maybe not!) for humans than giving every last freak the superpowers.
But what is it a summary of? Of doomsday AI fiction tropes? It's called "possible futures", so it's claiming to be more than that, but somehow, almost all the terminal nodes are "dystopia" or "death".
There are massive logical leaps here. For example, why does superhuman intelligence translate into AI taking over the world? By what means, exactly? You can respond to this with more sci-fi tropes - nanobots, mind manipulation - but come on.
I have plenty of misgivings about AI, but I think you can make a plot like that for anything if you get into the right mindset. Kamala Harris wins → war in Eurasia → nuclear apocalypse, Coke changes the recipe → soft drink extremists begin a worldwide terror campaign → totalitarian police state, etc.
Was looking for this comment. I stopped reading two paragraphs in at:
>This could probably happen today: a pandemic could cause human extinction, and AI is capable enough to assist in biology research.
No, friend. Someone having access to abliterated fable 5 or whatever does not give them superpowers to create a human extinction level pandemic.
I have no idea how to deal with people like that, when someone has thoroughly internalized unfounded assumptions about what ML is, how it works, and where it's headed based on the current tech... what do you say? How do you broach the topic that they might be so deep in a hype cycle that they've genuinely lost touch?
I've tried, gently, firmly, obliquely... nothing seems to work. I had the same experiences during the Crypto bubble, people simply wouldn't listen to dissent from the notion that we were on the cusp of changing money, programming, and society. How do you explain to someone that just because they can imagine an exponential graph doesn't mean the data fits?
If nobody can make money anymore, it would probably just lead to a Star Trek like society.
The automation owners might not like that, so it won't happen peacefully, but I think it will happen.
This graph encodes so many assumptions and omits so many potential complications.
Maybe I misunderstand, but I don't agree with the "Permanent Underclass":
> Socially: there is no more social mobility, your lot in life is determined by how much capital you had when the music stopped. Growing your human capital is no longer incentivized, since it doesn’t change your life outcomes.
One way to change that is violence, right? Not that it is necessarily desirable, but that makes this situation "non permanent"?
Presumably full automation means that violence won't be practical either, since they'll have robot police and drones?
If bourgeoise interests control ASI and folks are constantly surveilled (as they pretty much are now), I struggle to imagine how an attempt at violent revolution can be successful
Except that "Permanent Underclass" goes under "no ASI".
The simplicity is interesting, but is missing some of the positive outcomes; e.g. meaningful democratic governance (or control) of the AI/ASI('s). (shamelessly working on that at ai-democracy.org !)
Several different branches could go to versions of that depending on what challenges we're able to overcome.
Whats the incentive for these outcomes? Given a neoliberal society that we have, what will allow this to take place?
> a general observation: the technology to make the Earth uninhabitable to humans is much lower on the tech tree than the technology to build a distributed, extinction-proof interstellar civilization
Good news is that we are working hard on making the Earth uninhabitable before we get to ASI! There is hope!
Wondering what other scenarios there could be. I think it’s hard to interpret what an ASI might do, which is a digital god being.
The right of the chart feels like there’s a clear picture of what that future might look like. But the left of the chart, with aligned asi, is still incredibly fuzzy, more fuzzy than the author fully characterizes.
For example both world state and mad world have dystopian, benign, and utopian possibility.
I'm absolutely loving the responses I'm getting from feeding this page into each of the frontier models along with the question "what is missing from this flowchart?"
There are many possible world destroying events that don't involve AI. And AI could solve them.
Is P(doom|AI) > P(doom|no AI) ?
Show your work.
> There are many possible world destroying events that don't involve AI. And AI could solve them.
The entire subset of them that has humans as a cause of doom is automatically in the set of AI-doom scenarios, because AI can directly or indirectly operate anything humans can.
What's the time cutoff for the question? Based on most current mainstream understanding of physics (ex. https://en.wikipedia.org/wiki/Timeline_of_the_far_future), and mainstream understanding of "doom", at unbounded time horizons P(doom)=1 (and hence also both conditionals, as long as you accept 0<P(AI)<1).
What would this list look like if late capitalism didn't exist?
AI is aware of this chart. AI is going to navigate against any obvious reefs that could beach it.
Can I have a chart about AI who knows about the first one?
There is an interesting concept of a very smart AI but not capable enough of taking over and knowing this making a deal with humans to prevent an even smarter AI taking over, by for example monitoring training runs and reporting on rogue AIs.
It's really disturbing to see the full "Less Wrong, Roko's Basilisk cultist" mentality emerge so strongly here. So many self-styled rational people who are unwilling or unable to meaningfully interrogate their assumptions or emotional driving forces, converging on the same religious tripe.
I don't think "Less Wrong, Roko's Basilisk cultist" is a charitable characterization of Fernando: he explains the development of his current positions in https://borretti.me/article/the-education-of-a-doomer:
I’m referring primarily to the commentariat rather than the author.