I grew up in evangelical christianity, and to them the end of the world is just around the corner, the same way it has been since I was a small child and likely will be when we are all gone. This isn't science. This isn't hypothesis experiment record results. This is very expensive astrology, shiny rock collecting, ritualistic meaning-making and self-justification.
Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
It's nuts how well "Superintelligence: The Idea That Eats Smart People"[0] aged. That talk is a decade old by now and still hits just as hard as it did back then, despite the incredible advances made in AI in the meantime.
Yeah I think its possible that for many folks its the first time they're coming up on these concepts, and it troubles them in the same way that the concept of death troubles them (and me, to be clear!).
For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducing any doubt. These are not scientists, they're business people and salespeople, and a few optimists having recently on paper solved all their worldly financial needs.
Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.
>> Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.
Important point. LLMs were early on hailed as the first general-puprose AIs that can perform any task (remember "Sparks of AGI"?). Today they're increasingly promoted for specialised applications - coding, as a for instance.
While there are some coding focused models (composer, for example), the majority of frontier models are pitched as general purpose. The coding harnesses for Claude and GPT are even being repurposed as general purpose knowledge work harnesses.
> whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own.
As usual, AI skeptics are moving goal posts. Modern LLMs are on a completely different level in terms of how GENERAL they are vs anything pre-LLM. You can give it a completely novel puzzle and it will solve it. 5+ years ago you had to train NN to solve particular type of puzzle.
Did you actually read the text? OPs are calling that Plan D.
They're proposing an alternative, which is a global brake on frontier AI research to keep the basilisk in its jar until we work out what we're dealing with and how to handle it.
No, they're proposing a spying panopticon and state control of global resource distribution - specifically general purpose compute - including seizing and destroying GPUs. They're proposing a totalitarian global dictatorship controlling computing hardware and software.
I always thought it a bit pessimistic on the upsides like robot to build us palaces, immortality, conquer the galaxy type stuff along the lines of the "Grandiosity" bit in the essay. It's a shame we have amazing tech possibilities and most people are either saying either it's rubbish and won't work or it'll bring doom. Aside from being boring they are probably factually wrong.
Bit like saying early medical knowledge was rubbish or will doom us by summoning demons while really life expectancy at birth went from twenty something in Roman times to eighty odd now.
> AI risk is string theory for computer programmers. It's fun to think about, interesting, and completely inaccessible to experiment given our current technology. You can build crystal palaces of thought, working from first principles, then climb up inside them and pull the ladder up behind you. People who can reach preposterous conclusions from a long chain of abstract reasoning, and feel confident in their truth, are the wrong people to be running a culture.
I understand how people running in the same scene fall into the echo chamber effect and get gulped into the cult, but why does everybody want to be a prophet?
Theorizing about nuclear winter is somewhat similar, in the sense of being inaccessible to experiment. Does that mean we should disregard the possibility of nuclear winter?
As with using the Orion drive for launch and landing on Erth is possible, test nuclear winter is certainly possible. But as with the former, you won't be very popular among the survivors (if any).
Being a prophet is probably great until you suddenly find yourself building a fortified compound in Waco, Texas and purchasing black market full auto machine guns.
Thank you for contacting Gunmetal Ranch, a legitimate 501(c). If your call is related to the class-action "most dangerous game" settlement, please hold for a cowpoke.
Its not everyone building crystal palaces in their mind, they're all building fortresses. And they can't be wrong in their fortress or it breaks their world view which they cannot accept.
> AI is "completely inaccessible to experiment given our current technology"?
No, that’s not what the article/talk said. It said AI _risk_ is inaccessible (as the poster you’re replying to correctly includes in their quote). The point in the original talk is clear if you read it; that everyone coming up with their own pet theories for how superintelligent AI could destroy us as a species is building their theory on speculation on top of speculation.
Note that the talk is from 2016, before transformers and LLMs were invented. Though I would argue that the vast majority of it, including the parts we’re discussing here, are still completely valid.
and upon rereading completely holds up technically we are still passing in massive data into simple networks giving no opportunity for introspection or recursive self improvement.
The other thing about this article is that it contains falsifiable claims that are, well, false. Consider the following direct quote: "The 2028 election cycle is heated, as usual. AI is the biggest topic." No poll of the American public has AI as one of the top concerns except in polls where the question is specifically about AI. The general polls show topics like healthcare costs, inflation/affordability, government dysfunction, and immigration as the top concerns. AI doesn't even show up in the top 20 in every list I looked at (Gallup, Pew, YouGov).
It sounds more like "everybody I know thinks AI is the top concern" when everybody they know is people like them, i.e. an echo chamber. It gives me no confidence in any other claims.
The AI 2040 writing and the AI 2027 before it both sprang from within the rationalist communities. Scott Alexander (of the popular rationalist Slate Star Codex / Astral Codex Ten blog) played a role in both articles, though he says he chose to leave his name off of this one.
AGI and AI doom scenarios have been evergreen topics in rationalism for years. Long before we had enough compute power to do anything at all useful with generative AI. Many of the rationalist scenarios about future AIs had very religious themes where the AI became so powerful that it was effectively a god.
Rationalists had concept's like Roko's Basilisk, a thought experiment where a future all-powerful AI might choose to punish anyone who predicted the arrival of AI but failed to contribute to the arrival of AI.
The rationalist communities took this thought experiment so seriously that the biggest rationalist forum of the time formally banned discussion of Roko's Basilisk for 5 years. They believed it was an "info hazard" because once you knew about this risk, you either had to contribute to bringing about this future AI god or you risked being punished by it in the future.
If this is new to you, you might think I'm exaggerating or making things up to make a group look bad, but this was such a core belief that it has its own Wikipedia page: https://en.wikipedia.org/wiki/Roko%27s_basilisk
The people writing AI 2027 and AI 2040 were immersed in these cultures. That's why they think that AI is the primary topic that anyone is talking about in elections, just like the Bitcoin people all thought crypto was the only relevant election topic.
The GP comment is getting attacked, but there really is a heavy religious fervor background to the AI fears in the rationalist community. The religious angle and extremism of the forums is downplayed when in crosses over into mainstream topics, but it's been there from the start.
First I was intrigues by the rationalist movement, but I was deeply disappointed. To put it bluntly, seems to me like geeks LARPing as philosophers, and not with rational argument as they promise, but rationalizing their sci fi fanfictions.
If I recall correctly, the text of Yudkowsky's response to Roko's Basilisk used to be part of the Wikipedia page but was removed and not even a reference link was provided even though the response was mentioned in the body of the article. For posterity's sake, from https://rationalwiki.org/wiki/Roko%27s_basilisk/Original_pos... here it is in all its hilariously unhinged glory:
"One might think that the possibility of CEV punishing people couldn't possibly be taken seriously enough by anyone to actually motivate them. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous. I don't usually talk like this, but I'm going to make an exception for this case.
Listen to me very closely, you idiot.
YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL.
There's an obvious equilibrium to this problem where you engage in all positive acausal trades and ignore all attempts at acausal blackmail. Until we have a better worked-out version of TDT and we can prove that formally, it should just be OBVIOUS that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT DETAIL that they have a motive to ACTUALLY BLACKMAIL YOU.
If there is any part of this acausal trade that is positive-sum and actually worth doing, that is exactly the sort of thing you leave up to an FAI. We probably also have the FAI take actions that cancel out the impact of anyone motivated by true rather than imagined blackmail, so as to obliterate the motive of any superintelligences to engage in blackmail.
Meanwhile I'm banning this post so that it doesn't (a) give people horrible nightmares and (b) give distant superintelligences a motive to follow through on blackmail against people dumb enough to think about them in sufficient detail, though, thankfully, I doubt anyone dumb enough to do this knows the sufficient detail. (I'm not sure I know the sufficient detail.)
You have to be really clever to come up with a genuinely dangerous thought. I am disheartened that people can be clever enough to do that and not clever enough to do the obvious thing and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much more important to sound intelligent when talking to your friends. This post was STUPID.
(For those who have no idea why I'm using capital letters for something that just sounds like a random crazy idea, and worry that it means I'm as crazy as Roko, the gist of it was that he just did something that potentially gives superintelligences an increased motive to do extremely evil things in an attempt to blackmail us. It is the sort of thing you want to be EXTREMELY CONSERVATIVE about NOT DOING.)"
This seems out of line with what you were saying above. Do you mean you would put money on a position less favorable than 50/50? What do you consider the likelihood such that you think you can announce it to be false today? 99%? I'd happily offer 30 to 1 odds on this today and add a 3x buffer for you up to $1000 of my dollars if you're serious.
I'm confused. In the original comment I pointed out that the article claims AI is a major topic when polls show that isn't the case. I still believe that in 2028 AI will not be a major topic as shown by polls and voting behavior.
I'm not sure how I'd formulate the bet, but maybe it could be "as of election day 2028, AI will not be a top five concern mentioned in the most recent open-ended poll by Gallup polling the American public. This bet will not be decided by a poll in which AI is specifically mentioned in the polling question."
This prediction can't be scored until the 2028 election cycle.
You may think it's very unlikely the prediction will have turned out to be correct by the 2028 election cycle, but that is not the same thing as the prediction being scorable as false today.
Or you deeply suffer from normalcy bias. Not believing that AGI is possible is irrational and unscientific imo, the human brain exists, it is not made of magic, it can reproduced. It is as simple as that.
We can debate about timelines, architectures that may lead us there, etc... You can even talk all day long about definitions of AGI, and people waste their time doing that.
But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point.
There is a vast gulf between theoretically possible and technologically feasible.
If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction.
You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone.
> the human brain exists, it is not made of magic, it can reproduced
Yeah. It only takes 9 months and ~18 years of training…
> But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got.
Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.
What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966.
Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.
It's a form of denial. We're getting another "de-thronement of man" on the order of Copernicus and Darwin. Some get excited, others turn away in horror. Negation is the outward expression of the desire to keep human intelligence wrapped in its mystical veil.
One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts. Not only is that untrue, it's also directly contradicted by our experience with previous RL-systems, where they seem to breeze right by human ability without even the slightest hiccup.
Most human systems are much, much, much more complicated than most closed world games (which is where RL approaches have seen massive success, mostly through self-play).
Like LLMs are great, but I honestly can't see us getting actual general intelligence out of them.
They already possess general intelligence by many metrics. Sure they miss a few, but that's nitpicky and goalpost shifty - lots of humans make errors of all sorts as well, or incur brain damage limiting them in one or a few areas of intelligence - we don't then say they are not general intelligence anymore.
I think maybe you mean superintelligence, which is a more fair critique.
> They already possess general intelligence by many metrics.
Can you share the metrics you are using for this assessment?
They are really powerful tools, but a quick glance at their thinking tokens (which is a bad name, tbh) rapidly disabuses me of the notion that they are general intelligences.
They possess large amounts of crystallised intelligence (i.e. they have a lot of knowledge), but their fluid intelligence is definitely lower than the human median.
Approximate as a limit and not surpass, where the hidden hope actually seems to be that they don't surpass. There are lots of variations floating around, from simple metaphors like "LLM as librarian that speaks to you," to even Sutton's remark that rather than being a case where we've taken the "bitter lesson" to heart, LLMs may be a yet another case where we're limited by it.
Unpack this a little bit. Why is it strange or interesting to you? What specific priors need to be updated for us here? What is the philosophical questions at play for you?
To meta-unpack a little bit ... it is strange to me that Fable is far more capable of discussing these questions than apparently 99% of humans. Along with being more capable at quite a lot else than most humans.
It doesn't seem at all strange to me that a chatbot trained by true believers in an AI singularity and the importance of safety guardrails will give more satisfying answers to true believers in an AI singularity and the importance of safety guardrails than talking to humans who might ask questions they're not prepared to answer (or might say nasty sceptical things or just not seem interested)
As for "updating priors", that goes both ways. There's plenty more reason to think "hey, transformers and RLHF might actually make some killer products" but certainly no reason to think the few people who didn't realise that "GPT3 is too dangerous to release" and "all software engineers will be replaced within 6-12 months" were marketing rather than prophecy have some kind of special insight into how it's all going to pan out. Clock's ticking to the promised 2027 reckoning too...
OK.. when you discuss these things with your Fable, what topics come up? Can you articulate one of the questions? I am probably just another dumb human FYI, but just try it out and we will see if I can follow along.
Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs. In the chase for AGI specifically, LLMs are a dead end, just like all the other AI technologies that died in the AI winter.
The commercial viability is orthogonal to whether it achieves AGI, which is effectively what "reproducing the human brain" amounts to in this discussion.
Strong disagree. Fable is first model that actually feels smarter than me in certain non-trivial ways.
It can hold many complex and partially contradictory thoughts in its head at once, in a way that feels significantly superior to Opus (for example). And then can make reasonable syntheses across these.
In a couple rounds of back and forth, with relatively low effort (but strategic) prompting, it produces complex, accurate analyses in 5-10 minutes that would take me multiple hours of hard, very focused work.
I still need to remain tightly in the loop, providing frequent course correction, clarification, high level reframing, nudging, and grounding.
It incorporates my feedback incredibly well.
It’s honestly staggering. Fable has changed my assessment of the current trajectory more than any model since possibly gpt-4. Opus 4.5 of last year might be a close second.
———
My advice for anyone who wants to get more value out of these tools:
When a model does something idiotic, don’t throw your hands up in the air. Be curious. Try to turn it into a puzzle to be solved.
It know it’s hard sometimes, especially if you are drowning in slop from other people… or generated by yourself, heh.
It can be exhausting. I struggle with this also. I have thoughts on how to make it better. We shall see.
I just realized that you might really be onto something. I wonder now if it is just a function of our very human inability to let go of a known construct that has served us well until now or something else. I have my own opinions, but as strange as it sounds, this may be the HN equivalent of ok boomer moment.
I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion.
Now write down a blueprint for superintelligence.
So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet.
In my view it would be insane to believe we can build something that we can't even reliably imagine yet.
As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system.
However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance.
You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.
Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function.
I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised.
That said, I think it's pretty clear that LLMs are not going to get us there.
I don’t think people are arguing to stop researching AGI. Moreso against sales people trying to use the concept of AGI to sell products that are very much not AGI. Or devoting so many of our resources into such a pursuit that it causes harm to real people.
This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.
The parent poster argument boils down to "[something] is theoretically possible, therefore 1) it is guaranteed to practically implementable 2) in the reasonably near future". Both are simply prima facie false; one can ask an LLM to explain why if there's any doubt.
And now we’re desperately trying to ”upgrade” penicillin (and friends) because it doesn’t work any more in many cases. Do you think we can repeat the process or do we need something completely different?
This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.
No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs.
Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?
That is simply not accurate. There are examples of scifi novels, novellas and other media that dealt with it. We can argue over whether it was that exact format, implementation and so on, but that 'shape' ( to use a common llm term ) of technological advances was very much explored.
> Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created.
Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI?
The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.
It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?
I can't give you or your sibling a better answer than "you'll know it when you see it". Some people see it now. I think they're wrong, because it seems like the results you're describing are easily explained by fuzzy search in the space of embeddings and then forming strings of plausible tokens related to the resulting region of embeddings space. In other words, the things we know LLMs actually do.
That's more or less looking for interesting patterns in a jpeg or another lossy compression result. It's interesting that the models seem to be able to (fairly) reliably return relevant chunks of the image. Even more interestingly, they seem to be able to invent plausible chunks of image that aren't even there. That doesn't meet my bar for intelligence though. I'd need to see it learn and adapt. I'd need to see it be clever, not merely "knowledgeable". I'd need to see it capably analyze itself. I'd need to see it reasonably estimate uncertainty and know itself in the sense that it has some idea how right or wrong it is about something. I'd need to see it exercise judgment.
I don't think I'd give a different answer a decade ago but who knows.
[edit] For all we know, one of the salient features of intelligence is that intelligent beings are incapable of precisely defining it. I'm not sure how productive it is to attempt to do so.
I appreciate the straightforwardness, but you probably understand that's pretty unsatisfying.
Actually, stronger - it's valid in some circumstances to say something is infeasible to precisely to define and you'll just know it when you see it. But I don't think it's reasonable to take that stance and then assert that "anyone sound of mind who knows how an LLM works" must agree with what you see. You gotta pick between striving for rigor and denying your opponents' soundness of mind.
What is your definition of "actually intelligent"? I believe LLM's are more intelligent than the average human in a lot of ways according to the Legg/Hutter definition of intelligence: "Intelligence measures an agent's ability to achieve goals in a wide range of environments".
No one knows how LLMs work. We know how the architecture works, but almost nothing about why. Saying "statistical next token prediction" tells you about as much about LLMs as saying "action potential thresholds" tells you about the brain. A true fact that explains very little.
And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse, if you think you have to be delusional to question whether LLMs are "intelligent", whatever you define that word to mean. The [latest publication](https://www.anthropic.com/research/global-workspace) from anthropics interpretability team purports that they see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain. Are you going to dismiss the whole team as not being of "sound mind"?
Intelligence is a word with a somewhat unclear meaning to begin with, but you have move the goalposts pretty damn far to exclude LLMs at this point. They are certainly still lacking in some regards, but whether that disqualifies them for intelligence is very much a matter of debate.
"And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse,"
I see no citations referring to the current philsophical discourse, unless you mean to imply anthropic's paid people are to be considered to be part of that.
That they "see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain" is if anything discrediting.
+1 I can't imagine how any corporate entity could be credible in this financial environment. Nothing they say can be reliably considered as anything but marketing copy. This situation is exactly what academic publishing is for. Although that institution has also been degrading.
My point was not to argue one point or another about LLM intelligence, but to push against the notion that you have to be "delusional" to even argue that it is possible that LLMs can qualify as intelligent (although the op prefaced it with "actually"). That's mainly what irked me about the original comment, the arrogance of dismissing everyone even having the discussion as insane, as if there's no legitimate argument to be made.
And this was mainly the point I was arguing. However since we're on the topic, I also happen to think it's intellectually lazy to dismiss the Anthropic interpretability teams work as "delusional" simply because they have a conflict of interest. Of course, that is not an irrelevant fact, but much of their original work has since been replicated by independent entites (eg. https://arxiv.org/abs/2510.01246). Until they publish something that turns out to be fraudulent, I think it's reasonable to consider Anthropic's paid people a very relevant, and in fact excellent part of interpretability discourse.
Dismissing all opinions where there is a perceived conflict of interest is a pleasant cognitive bias to have, but reality is often more nuanced than that.
In this very thread I am being told that Fable is nothing but a bit of scale and refinement on well-known neural network techniques. And next I am told that we can't even imagine how to build superintelligence. Which is it folks?
Indeed, the real fun signal here is the conflict between how intelligent humans think they are and the different story told by much of human behavior, including within this very thread!
When an LLM makes a cognitive mistake, humans jeer "dumb stochastic parrot!" - when a human does the same, many are blind to it. Like a weird auto-Gell-Mann Amnesia - relative to reflecting on our own perceived cognitive "strengths" versus others weaknesses.
Jesus... This morning while I was drinking coffee and staring at the screen (it's Saturday) an agent did the equivalent of days of my work, reading code, understanding, hypothesizing, comparing, using tools, writing scripts, launching compilers and running tests, identifying problems and proposing solutions, and more. Only someone who hasn't spent a second reflecting about what it means to think and to be intelligent can claim that we miss a realistic path to intelligence. It's so damn clueless and stubborn and confidently wrong that it annoys me immensely, so sorry for the rant.
Uhuh. I really shouldn’t be replying to this type of comment from a throwaway.
But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise?
Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us.
So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.
Now show me a writeup that explains how the brain works so I can understand why the brain does those things.
> We know how these machines work, it's not mysterious, there's nothing "extra" happening.
It sounds like you're saying "We don't know how brains work, they're mysterious, there's something 'extra' happening", and using that as justification for why you're saying a computer, an AI, can't "understand".
I think most people on Hackernews now who would use the phrase "my AI worked overnight and hypothesized, compared, etc..." already know how an LLM works, and still chooses to use those words. So the issue isn't that they don't understand. It's that they understand and still use those words. So the disagreement is somewhere else.
I'm not claiming there's something "extra" happening in brains. Merely that we just don't know how they work well enough to use that knowledge to do engineering. Neural nets are quite unlike brains, despite the unfortunate shared vocabulary.
OTOH we do know how neural nets work, and they definitely don't do "thinking" or "reasoning".
Fair enough, I should have phrased that less strongly. Until you show that your neural net does "thinking" or "reasoning" I'll disregard that and prefer to think about it in terms of what we actually know neural nets actually do. Does that work?
This feels like a semantic disagreement to me? If an LLM got to an acceptable end result code-wise, what would you call the process that took place to get it there?
Is an implication of this that models are incapable of producing entirely novel code?
Also, not to get too reductionist about this, but what do you posit is special about what is happening when humans think? Intelligence is hard to define so clearly, I reckon.
> Is an implication of this that models are incapable of producing entirely novel code?
No, it does not imply that at all. Google "temperature in LLMs".
> what do you posit is special about what is happening when humans think?
I don't. And IIUC nobody knows, but I'm not a brain scientist. There have been some wild theories over the years (recall Penrose's). I don't really have a dog in the hunt, except that probably whatever is happening is physical. It doesn't really matter, except insofar as whatever is happening very probably isn't what LLMs are doing. We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
No need to condescend, I'm very aware of what temperature is for LLMs. But I'm going to push back - if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense? (I'm not, for example, talking about novel rearrangement of existing ideas and code)
> We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
I don't think the claim is that LLMs do what brains do - I think the correct form of the counterargument is that _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
> if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense?
By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".
> _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
I don't think that has earned its therefore. Another perfectly reasonable explanation is that LLM's output is a close enough facsimile to intelligence that if you allow yourself you can easily be fooled into thinking its intelligent. That's not the same category of thing. It's not an incremental step away from intelligence. It's a whole different animal.
> By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".
This sounds to me like an admission that LLMs are not just doing a stochastic search, then.
> close enough facsimile to intelligence
What's the distinguishing criteria then? How can you tell the difference?
I think we must be talking past eachother. I define stochastic search as a search process with randomness injected into it that can return the following things:
- Something contained in the data set, not necessarily the same thing for every iteration of a given query
- Something not contained the data set (hallucination), not necessarily the same thing for every iteration of a given query
Does that clear it up?
> What's the distinguishing criteria then? How can you tell the difference?
All the ways they fail to exhibit intelligence. They can't learn. They can't adapt. They can't reason abstractly. They can't count. Etc...
I find the rebuttals pretty convincing - that there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.
> All the ways they fail to exhibit intelligence
Another implicit admission that there _are_ ways that LLMs exhibit intelligence?
> there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.
The next step then would be to design and conduct experiments that isolate this effect. Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining. Isolate it or distill it so it can be studied directly. Until then, it's easiest to dismiss it as imaginary.
And you're happy that the replication of LLMs across many foundation model companies is insufficiently reliable?
> just something you're imagining
So the alternative explanation you're suggesting to emergent LLM behaviour is mass independently-corroborated human hallucination. Which is more likely?
Also it really does seem like you've moved the goalposts a lot here without really giving me a substantive response.
To say that LLMs' existence is evidence for emergent phenomena in LLMs is tautological. I'm merely suggesting if you want to make a claim about emergence it would be best, especially in absence of a convincing theory, to demonstrate it experimentally. Otherwise probably better not to claim it's actually happening.
> To say that LLMs' existence is evidence for emergent phenomena in LLMs is tautological.
This is not at all what I was saying. I think you've already conceded that LLMs demonstrate emergent behaviour but you dismissed it as a "close enough facsimile to intelligence". I was saying that the emergent behaviour is reliably replicable, in response to your following statement:
> Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining.
I think there is real work underway in the area of interpretability. In the meantime, there appears to be plenty of empirical evidence for the claim that LLMs exhibit some sort "intelligence" in the enormous penetration that agentic coding has achieved in software development? Do you deny the usefulness of LLMs here, or are you going to assert that actually software development requires no intelligence of any sort?
> I think you've already conceded that LLMs demonstrate emergent behaviour
No. Please don't put words in my mouth. What I said is that an LLM compresses a bunch of information into a semantic embedding space and then does sort of a stochastic search in that embedding space. Any similarity to "intelligence" is accidental. You may look at the results of that process and "see" thinking or reasoning or something, but it ain't there.
> "intelligence" ... agentic ... usefulness
I don't think LLMs need to be intelligent to be (at least narrowly) useful. No more than random forests or genetic algorithms do at least.
[edit] Look, this has devolved to the point where it's no longer productive to continue. If you're going to state things like this as fact, there's really nothing more I can do here:
> emergent behaviour is reliably replicable
Go collect your Nobel prize then! This is no longer a discussion grounded in reality.
On what grounds? I don't think you've provided any evidence other than LLMs can't "adapt" or "learn" to show that LLMs do not show intelligence in any way. I think it's clear that there must be some emergent form of intelligence over words from just the agentic coding ability alone. I am not claiming that LLMs are intelligent, only that they display aspects of what we understand as intelligence.
> I don't think LLMs need to be intelligent to be (at least narrowly) useful
I agree! But they are more than narrowly useful, and they absolutely do not belong in the same category as random forests or genetic algorithms!
> Go collect your Nobel prize then! This is no longer a discussion grounded in reality
Once again you are being condescending while misrepresenting my position. The emergent aspects of "intelligence" have been replicated by virtue of independent LLM vendors training their own models - I am not making a stronger claim, you have misunderstood me.
@sudb, I salute your effort. It is unfortunate how little this person took from your attempts at a genuine exchange.
Without moving goal posts: @27183: the word intelligence is defined as "noun. The ability to acquire, understand, and use knowledge."
LLMs can absolutely do this via in-context learning. They can even persist this as mementos on disk and re-attend to it later if its been purged from their context window.
You might try to make some comment like "they're not learning in the way humans do, they're not changing their weights, and they predict token by token!" Who cares? That's semantics - LLM attention is able to accumulate contributions of tokens and weights mixed together head by head and layer by layer with non-linear activations which functionally quack like intelligence.
They may be far different from us, but they're functionally intelligent (at least in several subsets of intelligence. And frankly, in some cases, they're more functionally capable than many humans).
Birds, planes, helicopters, rockets and gliders don't achieve flight in the same way, or have remotely the same capabilities in the air - but all are nonetheless forms of flight.
Just a few comments ago you argued that we don't know how to build superintelligence. Now you're saying we know how the (unevenly superintelligent) Fable system works.
It doesn't seem like you're being consistent here. I'm concerned there might be some motivated cognition going on.
"What is true is already so. Owning up to it doesn't make it worse. Not being open about it doesn't make it go away. And because it's true, it is what is there to be interacted with. Anything untrue isn't there to be lived. People can stand what is true, for they are already enduring it."
> We know how these machines work, it's not mysterious, there's nothing "extra" happening.
Lol. This is more telling about your implicit unscientific preconceptions that you wanted to reveal. Of course there isn't anything "extra". Where do you think intelligence comes from, some mysterious realm? It's physical, computational. The fact that at the bottom we produced it via matrix multiplication is irrelevant. Maybe humbling. You are denying a visible fact (a machine performs tasks that require flexible analytical and cognitive skills) precisely because there is no magic happening anywhere.
> Where do you think intelligence comes from, some mysterious realm? It's physical, computational.
Well, no. I don't think it comes from some mysterious realm. I think that which is not physical does not exist [edit: and if you like I'll follow that one right down the rabbit hole--continuity and infinity are useful delusions]. But that eminently does not mean we know what intelligence is, let alone how to build one.
> The fact that at the bottom we produced it via matrix multiplication is irrelevant.
Huh? We don't even know what "it" is. How can you say you produced it?
> a machine performs tasks that require flexible analytical and cognitive skills
You see that, I see a lucky stochastic search result. Don't underestimate the "creativity" of random algorithms! They can do some wild shit! This is nothing new, we've been playing with these toys for like 70 fucking years. It's only recently that they started spewing words and everyone lost their minds over it.
You do realise then that "we know how it works, there is no "extra" there" is an argument that can be used against any artificial intelligence, now or in a thousand years, as well as (at some level) against human intelligence (no magic, it's all physics, just dumb cells exchanging signals). This should be enough to give you pause- you immediately reached for an argument that is entirely empty.
> I see a lucky stochastic search result
Again you're reaching for a mechanistic explanation of some kind (let's leave for the moment whether it makes sense or not) as if having an explanation somehow contradicted a display of intelligence. It doesn't. Yes of course we made it, we know how it works (ar some level) and there is no magic. But what matters is the result- this machine, matrix multiplier, stochastic parrot, consistently displays intelligence, to the point of being able to perform very complex, open-ended tasks that integrate discovery, planning, tool usage, decision and even some aesthetic sense, understanding and using natural language, context awareness, you name it.
> This is nothing new, we've been playing with these toys for like 70 fucking years
Lol no. For god's sake. Hundreds of billions of parameters organised in a specific architecture and trained with unimaginable amounts of data and compute? Unless by "these toys" you mean "any computer program vaguely AI-related".
> But what matters is the result- this machine, matrix multiplier, stochastic parrot, consistently displays intelligence, to the point of being able to perform very complex, open-ended tasks that integrate discovery, planning, tool usage, decision and even some aesthetic sense, understanding and using natural language, context awareness, you name it.
IDK, it doesn't seem like they actually do any of that. To me it seems like they have good enough semantic embeddings that they can kind of approximate those things, sometimes, well enough if you don't look too hard. This is enough to fool people. Of course there's gold in them hills--some recent mathematical results were found there. But to say that's "intellgence" is to say that lossy compression is intelligence. It's static. It does not learn. It does not adapt.
> Unless by "these toys" you mean "any computer program vaguely AI-related".
Not "vaguely AI related". I mean stochastic computer programs that can do things that look awful thinky. They've existed for a long time, but only recently (due to word2vec and other advances) have the results been words that mostly go together well instead of numbers. For some reason people seem to think a lot less critically when the output is words. IDGI but it's a whole thing.
Science means the pursuit of knowledge. It doesn't mean "only believing proven things". If we're going to be rude, lets at least take the time to be right.
You dont seem to understand what proof means. Human brain is made of matter, matter can be arranged to make a thing that reproduces human brain properties. What's the confusion here? I say its unscientific because it places the human brain beyond the scope of what can be operated on. Not having the knowledge or tech yet to achieve that is irrelevant since we have an existence proof.
Even if we accept your premise, and not everyone does, the confusion is whether we're capable of creating an equivalent arrangement, even in principle, using alternative materials.
One can simultaneously believe AGI is possible, be only modestly sceptical that our current methods are likely to yield it in the near term and still find the religious ferocity enveloping its discussion silly.
> saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Straw man. Nobody argued this. The discussion is around how urgent it is to policy treat a future hypothetical.
> GP just dismissed it as religion without engaging with a word of the material
Fair enough. I didn’t see anything novel in the article. So treating it as a motif within the abovequoted “Superintelligence: The Idea That Eats Smart People” context is fair and a real argument.
> Cynicism
Cynicism isn’t the opposite of blind optimism. Nihilism is. I’m not seeing a rejection of the article as being baseless as cynical or nihilist. It’s just pointing out a cultural thread that doesn’t seem to be useful.
I hate to get bogged down in semantics, but with the hope that one of the stronger top-level critiques makes it into the top slot here and this conversation gets buried:
Cynicism is defined as
>An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others.
I'm not saying that cynicism is automatically wrong, just that I once could trust that, when HN is wrong, it is due to cynicism applied in excess.
>..Nothing in life ... has any chance of succeeding without strenuous practice; and this is capable of overcoming anything..
---Diogenes (L) on Diogenes, arch-Cynic, VI.2.71
Afaik not formalised in Lean4, but sounds like optimism
Most of D's other stuff imho is optimism taken to its clear-eyed extreme. ya know. the way a very best version of us could always unironically RELY on HN [,after much practice]
>so those whose training has been of the opposite kind derive more pleasure from despising pleasure than from the pleasures themselves.
Might be why professional Nihilists admire Cynics (ie, for their ... unaffected comedy)
Ah, this clarifies things. I suppose I brought this on myself by putting it at the start of the sentence, but I do mean little-c cynicism, not the ancient philosophy. I was surprised to be called out on it, as GP is a shining example, and I was confused by the mention of nihilism, but I see now where wires were crossed. Thanks!
I put it in the same bucket as living on Mars. Can it be done? Probably. Are we close? Not as close as people seem to want to believe. Is it a goal that will largely benefit society in its current form? Absolutely not.
Eh, with Martian habitation we know what the roadmap looks like. With AGI we don’t. It could be proximate. Or it might not be. When it arrives, it could be totally economically uncompetitive outside the rich world. Or it could replace all human labour. Or progress to become a superintelligence.
We don’t know. Which makes proposing rules around it based on fiction more than science silly.
In particular, I just don’t buy into the “left behind unless” framework.
Perhaps Anthropic will create God in the Machine. Not foreclosing on that. But will it matter so much who was fucking around with Opus five e-folding times ago?
Either ClauDeus is benevolent and lifts you up (not left behind) or it isn’t, or not to you, and you are culled by a drone (left behind regardless).
I remember thinking exactly the same thing around 5-7 years ago, in the GPT-2/GPT-3 era. "Oh sure they can produce semi-coherent output, but truly intelligent behavior is still far away. This isn't science fiction, they're just falling prey to Pascal's Mugging same as my religious friends did." Now I'm not so sure. I give the AI safety subculture as a whole a lot of credit for putting it on my radar back when it was otherwise still science fiction. I don't know if they're right about what comes next, but I think their case deserves to be evaluated on its merits, rather than assumed to be the result of psychological flaws.
> Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
Confused at who this is directed towards. I'm fairly certain that the article was written by people who (at some point) identified as effective altruists, most of whom would enthusiastically agree with this. This community didn't start as AI researchers and later choose effective altruism; they were effective altruists who chose AI safety research as the most effective way to improve the world. Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?
>Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?
Their methods are about convincing others that things that enrich and empower themselves at the expense of others is "improving the world". This isn't the stance of serious people who want to improve the world.
Please don't confuse AI researchers and AI safety researchers (although there is some overlap.) That's like confusing the people from the Manhattan project with the people who protested 3 Mile Island, because they're both focused on nuclear technology. One effect of EA is that every AI researcher says they're trying to make the world better. Some of them are full of shit, but not all.
I'm fairly certain the authors would be happy to see AI shut down indefinitely. They just don't believe that the coordination problem is solvable. This is their best attempt to come up with something workable in the real world, or at least get people started thinking about it.
Right, the so-called "AI safety researchers" are worthless clowns and grifters with nothing of value to contribute. We would all be better off if they spent their time on something productive like writing entertaining sci-fi stories. They certainly seem to have the rich imagination necessary for that occupation.
Now this is an unserious comment, just throwing insults without explaining your reasoning at all. Do you believe that AI itself is worthless? Or that it's just incapable of causing harm? Do you resent the safeguards placed on AI because they are personally inconvenient to you? When introducing a new technology at this scale, don't you think it's worth looking before leaping?
So called "AI safety researchers" are themselves unserious so of course I don't take them seriously. LLMs are extremely useful, I use them constantly for a wide variety of tasks. LLMs don't cause harm: humans cause harm. There's no real need to intentionally cripple or restrict the models in the name of "safety"; that's just stupid and done only to prevent corporate embarrassment.
I'm confused, the authors of the website are arguing that the development of AI should be slowed down so that society can adapt. What do you think they are trying to convince others of?
Effective altruism is also very attractive to manipulative sociopaths who want to maximise their power over others whilst appearing virtuous and hoarding wealth and power. Poster boy for this movement is the convicted fraudster SBF. I believe Altman is also a fan.
As to a better world or super intelligence, I’ll believe it may be possible when I see some signs of intelligence from what people are calling AI, instead of plausible text and image generation based on a very large corpus.
An important adjacent point is the same people that are insisting that only their research can stop intelligent manipulative computers from controlling the human race are also some of the few people who believed that OpenAI was a philanthropic endeavour and that Sam Bankman Fried was trustworthy...
I'm not seeing any claims to exclusivity. I'm seeing people actively encouraging more people to enter the field of AI safety/security and openness to robust debate, etc.
I agree that there seems to be a huge problem with value drift--either people who used to research AI safety, pivoting to building AI (looking at Anthropic) or people who only ever paid lip service to safety (I tend to put Musk and Altman in this category.) These people need to be held accountable, but it doesn't mean every AI safety researcher ever was a stooge or a fraud.
Ok, fair enough theyre not entirely opposed to outsiders setting up research institutes like theirs and enjoy debate (or the optics of debate) more than most, and I don't think most of them are insincere. But the point wasn't that they were exclusionary, but more that whilst priding themselves on their supposedly superior ability to predict the future and "align" AIs, they somehow missed the value drift and human non-alignment with their values basically any outsider saw coming...
OpenAI now controls one of if not the largest philanthropic endowments on the planet. They are still a philanthropic endeavor, though I agree the non profit bait and switch as well as the Altman board situation were tragedies.
I mean, they're also gearing up for one of the largest IPOs in history to generate unprecedented amounts of wealth for themselves, and the nonprofit foundation's focus is community engagement with OpenAI products. Goldman Sachs has a large philanthropic endowment but I wouldn't confuse that with their priorities being philanthropy.
The point was I think pretty much everyone else saw the bait and switch coming...
I would rather see you engage with the substance of the article rather than skipping right to insulting the authors. I don't think this sort of comment is up to the standard I have come to expect from HN.
> Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective"
Who is this supposed to be arguing with? It sort of reads like it's trying to disparage "effective altruism", but I'm not sure.
Setting aside any of the AI stuff, I've started to find it pretty grating when people seem to imply that transferring millions of dollars from wealthy people in California and the UK to impoverished Kenyans and Rwandans, or buying malaria bednets which can save a child's life for the cost of a fancy new gaming rig, is "self-serving" or something because weirdos are doing it, while true caring for other people involves [unspecified thing that doesn't appear to ask any material sacrifice comparable to donating a large percentage of income].
> I grew up in evangelical christianity, and to them the end of the world is just around the corner
I grew up in evangelical Christianity, I'm still an evangelical Christian, and you're wrong to paint with a broad brush like that. Are there evangelicals that are millenarian? Yep, of course.
But that is not my experience across multiple evangelical churches across decades.
To interpret charitably, I guess we could solve this for religion = technological development so advanced as to be indistinguishable from magic. It's been done in some Star Trek episodes I think.
Imagine observing powered flight for the first time and saying: "This is religious fervor folks. I remember the mythological story of Icarus from when I was younger. Key word, mythological."
The existence of mythology describing Scenario X is not a valid argument against the plausibility of Scenario X.
If we can acknowledge the possibility of nuclear doomsday without running an RCT of sample size 100 Earths, 50 of which undergo nuclear Armageddon, to verify that nuclear holocaust indeed a real phenomenon... then we can do the same for AI. Understand the arguments being made instead of engaging in these guilt-by-association arguments.
Modern AI capabilities are already mind-boggling by the standards of 20 years ago. We should at least prepare for the possibility that trends continue on the current trajectory.
>Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
You can change more people’s lives, more substantially, if you donate effectively. Effective altruism started out as (and the majority of effective altruist financing is committed to) an effort to rationalize what has historically been a very emotionally driven activity by deploying insights from developmental economics. If you want to take longtermists to task, go right ahead, but please refrain from torching anti-malarial or child vaccination programs while doing so.
But in the public image, the EA community is synonymous with doubling down on AI / AGI to the exclusion of the other projects.
OpenPhil changing its name to Coefficient Giving, 80000 hours and bluedot and (to a lesser extent) CFAR dropping other initiatives and switching to AGI promotion… to my knowledge GiveWell is the only other big name that continues to advance other initiatives. Then look at figureheads like SBF committing fraud and begging for a pardon from the architects of the USAID shutdown… We begin to paint a picture of a community that’s (by and large) abandoned its principles for power.
I know the view from the inside is more nuanced, but I think it’s a reasonable association for random members of the public to make.
My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.
>My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.
I’d be curious to hear you expand on this. What binds the EA community together, from the shrimp welfare enthusiasts and wild animal initiative, to the longtermist lightcone obsessive, to the people funding vitamin A supplementation, is simply a commitment to maximizing the number of quality adjusted life years saved each year and a belief that empirical observation can be used to improve that number.
To my mind, this is a valuable insight on its own. Yes, if you come to such a heuristic with absurd prior beliefs, such as whether 100k neurons alone have QALYs in the first place or by placing equal value on people actually alive today and hypothetical people in the far flung future, you will get absurd results. Garbage in, garbage out. But that’s not an indictment of the fundamental insight, especially when you consider how poorly allocated the roughly $2 trillion in global charitable spending is.
As a society we already know how to make the lives of everyone better:
1. Stable housing
2. Access to safe drinking water
3. Access to food
4. Access to healthcare
5. Access to education
6. Stable governments
The EA community has so many "ideas" about what would help, when all they need to do is focus on those six and the world would be as close to a utopia as you or I could hope to see in our lifetimes.
I legitimately thought you were kidding about the shrimp welfare initiative, but after looking it up I was more infuriated about EA than I normally am when it comes up. I can think of several causes which would be better served with 3 million USD, and all of them take care of human beings. Living, breathing, intelligent human beings. These people should not be in any position of power or taken seriously ever.
There are tradeoffs to make between those six items. Money spent on stable housing is money not spent on safe drinking water is money not spent on education. There is a limited amount of money spent on charitable giving every year, and we obviously cannot afford to provide all of those things to everyone with the funds currently available. If you’re arguing that people should simply be donating more money, I (and I think every EA) is right there with you. But given the world exists as it is, I would rather see money that’s currently being donated diverted from Catholic missionary schools and good governance NGOs to GiveWell’s top charities because I’m very confident that it would mean fewer dead kids.
I agree with your last sentence, but we can’t partition the budget up so cleanly though. Societal spending is interdependent and additive across domains. Spending on cheap access to safe water or food reduces healthcare costs. Spending heavily on education tends to strengthen government in the long run, and better-run governments amplify efforts to improve housing/food/water/.
The one exception is AI. Historically, redirecting charitable funds towards AI safety tends to starve or undo the rest of these efforts, which is why I’m so disappointed by many EA institutions dropping other initiatives to put their eggs in the AGI basket.
AI 2027 made substantive predictions about the near term future of the technology and the implications of it. I think the worst you can say about it is that the Agent-1 moment might come next year rather than this year. But being off by a year is far better than this 2040 slop, which is mostly disconnected from reality.
The one thing I will say that they are correct about is that AI does have the potential to be highly destabilizing geopolitically, even if they get everything downstream of that wrong.
Notice the many times when a prediction failed and the so called prophet would come back in a few years and give you a new date. And they would tell you, well, this time it's real.
I do find it ironic that many of the AI predictions are coming from the self titled "rationalists." It seems like building your identity around being rational and immune to psychological pitfalls is a good way to ensure that you don't even notice that you have walked straight into the one psychological trap every cult has employed since time immemorial.
They gave a ~2 year timeframe for their predictions in this one (tied to the next presidential election), so in 2 years when none of this has happened will they then switch over to a new AI 2044 Plan? Where's the accountability? Where's the followup or retrospective? What if there was some mechanism that branded these people so that when they made claims in the future people could clearly see the authors labeled as "made completely wrong fantastical claims in the past"?
> When I learned that this website is by the same people who gave us AI 2027, I immediately thought about the Wikipedia page on doomsday predictions.
Then, you opened the page and read it and realized that this prediction was contingent? You know what a conditional is (I would assume) if this, than that?
Then you realized that the only reason you were posting this comment was as a sort of silly gotcha "Oh look at the guys who keep increasing the number" instead of talking about the differences between the scenarios?
... and then there are the actual planet-threatening astronomical events that humanity should think about ways to mitigate but I'm worried that human lifespans and capitalism prevent us from working towards mitigations. Everyone will just say "meh, 1 million years is a long time, I won't be around" and a million years will go by.
Capitalism, in particular NASA's commercial space development program, have already provided and demonstrated initial asteroid redirect capability via the DART mission launched by SpaceX, which impacted an asteroid and measured the changes to its trajectory.
People who've gone full true believer in "AGI" remind me a little bit of like, a random person from the most boring small town in the midwest who joined the Rajneeshee/Osho cult, or Hare Krishnas or something.
But instead of weird religious deities and practices, they've wrapped up their true believer zealotry in some kind of mishmash of "AGI is coming real soon now" like some kind of manifest destiny.
You could probably put people in FMRI machines and ask them to give a 30 minute lecture on the topic of AI and find that the same parts of the brain are activated.
I would not like to be dismissive, but to me this article feels like an exercise in creative writing rather than a report to be taken seriously. The entire experience feels like a choose your own adventure game, seems like their stylistic intent.
I am not sure if alternative reality fiction is the best way to approach real and serious AI risks.
I am also not sure, with the amount of emdashes and the style of prose, that the entire article was not AI generated.
AI is going to be a mature scientific field. There are going to be efficiency improvements in training and inference. New paradigms are going to emerge with better multimodality, real time streaming and real time interfaces. Models are going to converge on the limits of our data available for pre and post training, improvements will be incremental and spiky in domains.
I am not sure who the AI 2040 article is for. I suspect it is intended to be a digestible piece of media for the financial class.
AI is going to be a useful technology and its impacts across the economy and global will be broadly distributed. Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones. Maybe the argument is that in verifiable domains, such as model training, AI models can supercede humans. I don't think so. A human's high level thinking, our incredibly more efficient semantic/neural compression, our ability to switch tasks and achieve the creative insight is not replicated through the current paradigm.
I love to model and simulate. As the dead economy theory[0] (discussion [1]) was submitted here, I decided to simulate it. It was really hard to figure out a path "good for the humanity", in the sense of a balanced system, not a winner take all situation, etc.
I think this is the reason why you have the tendency to propose some freeze-all policies, full control or similar. If you want to find the equilibrium, you need to accept that it will be a controlled equilibrium, most likely on a saddle point, with underlying process changing all the time, requiring fast changes in regulations. Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
One counterpoint is that the "labor as TAM" argument is far larger than it needs to be. Only a fraction of it needs to be captured to justify all the capex and make 5 new companies displace FAANG, and this does not have to translate to unemployment to succeed.
The difference in the unemployment vs efficient employment model is mostly user driven adoption vs company mandated adoption, or centaurs vs reverse centaurs.
> Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
Without totally derailing the thread, this is also obviously why climate and biosphere collapse is not (and likely will continue not) to be addressed, e.g. Timothy Morton's Hyperobjects
This incredibly critical part of "early 2026" doesn't seem to hold up very well:
Overall, they are making algorithmic progress 50% faster than they would without AI assistants
Yes the AI labs are using LLM-assisted development, but I am not aware of LLM-assisted research into better deep learning algorithms. "50% faster algorithmic progress" is a strange thing to predict about neural networks: historically it seems like algorithmic progress is very infrequent, and tends to be disruptive across the field. Likewise with the underlying algorithms in Codex and Claude Code - they aren't that sophisticated in the first place, and vibe coding an implementation 50% fasfer doesn't count as AI-assisted algorithmic progress. Maybe they mean cost?
And ironically they didn't predict the actual 2026 reality that Mythos sucks as an agent but commanded global attention as a cybersecurity tool.
Also this just seems childish and clearly hasn't really panned out:
But China is falling behind on AI algorithms due to their weaker models. The Chinese intelligence agencies—among the best in the world—double down on their plans to steal OpenBrain’s weights.
China has the exact same "algorithms"! The "algorithms" are on arXiv and the specific architectures are typically public. What China lacks is compute, and the US labs had a big head start on training data + RLHF.
The article hinges on AI automating R&D and finding something fundamentally more powerful and reliable than the current transformer LLMs. But that hasn't panned out at all. What has panned out is better scaffolding around running the LLMs in an iterative loop.
It actually seems to me AI 2027 holds up badly, unless your only takeaway is "AI gets better."
To me, this feels like a last ditch effort to revive the AGI narrative to reject the coming and current commoditisation of these models, contrary to all current evidence. https://artificialanalysis.ai/
> How is commoditisation of models incompatible with AGI?
A recursively self-improving AI has strong first-mover effects. That isn’t fundamentally incompatible with commoditisation if there is literally only one path to super-intelligence and you can have AIs at different rings on that ladder co-existing. (Not technically commoditised at that point. There are still different rings. But close enough.)
But the existence of commoditised AI implies model selection isn’t a huge deal, which in turn implies the models are about the same, which strongly implies there is no recursive self-improvement. Depending on your definition, you may still have AGI. But you don’t have superintelligence.
> But the existence of commoditised AI implies model selection isn’t a huge deal, which in turn implies the models are about the same, which strongly implies there is no recursive self-improvement. Depending on your definition, you may still have AGI. But you don’t have superintelligence.
This is only true at a given AI capability level, no? e.g., if AI at the GLM-5.2 level is commoditized, all that suggests is that there's no recursive self-improvement easily possible at the capability level of GLM-5.2. (And with the harnesses for it that exist so far, etc etc.)
If I observe commoditization of a given tier of model capabilities at a given point in time, this seems to say little about what's possible with models six months later, or models that are undergoing proprietary deployments at that very moment inside the major labs, or even models that are notionally available for public use but have had recursive self-improvement adjacent capabilities intentionally nerfed (e.g., Fable).
(I might be misinterpreting your comment tbc - if you mean observing commoditization implies there is no existing, ambient superintelligence at the moment of that observation, then I don't disagree.)
> if you mean observing commoditization implies there is no existing, ambient superintelligence at the moment of that observation, then I don't disagree
This is a better way to put it, thank you. More precisely, I'd say commoditisation implies there is no existing self-improving AI on the market.
The moment someone gets exponential self-improvement, model fungibility breaks and the first mover wins. This is the Bostrom singularity the rationalists flip out about.
The "AGI narrative" is distinct from the existence of AGI.
Most of the discussion around AGI is highly speculative. I am not saying AGI could not exist, and it is a term that has historically been loosely defined. Decades of coming science and research will tell.
Most of the world's problems are fundamentally social, political, and religious. These cannot be solved with current non-AGI models. Probably not with AGI either.
Yes. Anyone who doesn't acknowledge the efficiency difference between pretraining vs RL and assume that since we've run out of data for the former, we have to do the latter, is not making a serious attempt at modelling the future:
This is similar to that other exponential, which happened with CPUs - we ran out of true geometric scaling in the mid 2000s, and everything else supporting Moore's Law has been cleverness that arrived in the nick of time, supported by a bit of marketing, and very optimizable benchmarks, far from guaranteed gains coming from making a single physical metric better.
>Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones.
I'm confused if this is satire, sarcasm, or genuine belief. If this was the case, then AI companies should absolutely remove the "it may make mistakes", because doing mistakes would imply that "the very best human knowledge and expertise" is what actually fails, and not the AI.
With that being said, I'll still urge people to visit a professional therapist for health problems and I generally still trust human knowledge workers for critical scenarios. I will reconsider your claim when chatGPT can effectively play Yu-Gi-Oh! (or at the very least respond with the correct rules appropriately), which is a significantly lower stakes scenario than betting your entire company on its aptitude.
My framing may have been confusing there. “distillation of the very best human knowledge of expertise”. Distillation is different from outright capability or reliability. It is not directly adjacent.
For anything health related all AI models show high levels of anchoring bias. I would not use it as a confidant, and be skeptical of claims. Even so, human doctors are also fallible and prone to cognitive bias.
I think the obfuscation is because human intelligence has been projected onto AI model capability. AI models only have a limited dimension of human intelligence, and in some axes orthogonal, and when I say distillation I refer to this.
Just purely organic YouTube Comments circa early '20s alone surely outslop any "AI" by a giant margin.
Everyone sees the markers, and it's a hot topic. There are maybe a thousand from-scratch trained models, and just few mainstream ones produce most of human-targeted content. In today's world, no surprise everyone knows the common patterns of those. That sloppy landscape is not just load-bearing em-dashes — it's a humble testament to their reinforcement learning.
Humans produce tons of texts, with all sorts of nonsense in it, without thinking it through. Our slop is just a lot more diverse. And mostly just spoken out loud.
> P.S. Information search and retrieval if the best and most direct way to use LLMs.
Yes, but not directly, if they don't know something they tend to hallucinate like mad, even today. YMMV, but in my experience they work best as actual "cheap" reasoning for building queries and checking out search engine results. Even if they misinterpret some result, more and more results will still steer it towards correct conclusions and it can point at some results that relate well enough to be useful.
I'm sure some people will have issue with my phrasing but, honest question:
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
Human cloning and genetic editing isn't stalled because we agreed it's unethical. It's not pursued because at the current level of advancements, it's pretty useless. The things we can do there are niche. It's easy to ban something that's not very useful.
If we had a way to make gene edited humans a lot smarter, a lot stronger or live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
It's not useless. We know enough about DNA to be able to make better humans, but people get real squickly real fast when you talk about that. It has stalled because of that. If we go in and just check the DNA for Downs or Parkinson's, we can't have a conversation on aborting the foetus without religious beliefs coming into play. Designer children aren't a thing, despite the ability to edit DNA to do specific things. Hair and eye color are easy enough to go in and edit for. Humanity has decided to opt out of doing that for now.
Marginally better humans. And IVF paired with embryo selection is very competitive in that niche.
Now, we're getting better at predicting polygenic traits, and we're getting beefy multi-edit pipelines that might provide a meaningful advantage over embryo selection working in multiple animals. But as of yet, the advantage of genetic editing in humans over just doing aggressive IVF and dredging the embryos for desirable traits is minor.
Bit of a chicken and egg problem there. Can't advance the tech fast without actively using it, can't actively use the tech until it's advanced enough for the benefits to override ethical concerns. So it's getting there, but at a glacial pace.
You should consider reading more deeply than Wikipedia to be properly informed on a subject. There are a handful of very rare mutations that confer a high risk of getting it that we can detect. They are very rare. This isn't the same thing as being able to give a risk score if the DNA doesn't have all the rare mutations. But the science is there for those unlucky rare cases to say there is a high chance that a specific coding of DNA will result in Parkinson's.
Yes, there are examples of where we have collective decided not to pursue a particular technology tree.
For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)
And there have been successful world-wide bans.
For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
> "For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)"
That example works against the argument since that policy was rendered moot when Commodore Perry arrived at Japan in 1853 with a squadron of American warships and demanded opening of trade and diplomatic relations at gunpoint.
I completely can see why we'd want to, for ethical reasons, ban germline editing, and I want to be clear that I agree doing so cannot be done ethically, but there is a part of me that is wistful for what could have been. Same with things like CRISPR but it's probably just fun to dream and the reality would be a nightmare.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
No, of course not. That would be an insane trust fall. Even relatively small advances in technology give a country world dominating power. Fun fact: India was militarily superior to Britain in the 1600s—a gunpowder empire with a million soldiers—but was taken over by it in the 1700s. Britain’s edge was small: lighter, more maneuverable cannons, standardized ammunition, better military and political organization. Not a first world country versus a third world country—more like the dynamic US versus a sclerotic EU. And that modest edge led to 200 years of colonization.
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
You overstate the advantages of technology. Mughal India was fragmented and on a sharp decline. The British used politics, finance, and treachery to divide and conquer what was remaining.
Yes, this one. Look at our governance; look at our coordination-at-scale; look at our collective problem solving. It's abysmal, beyond hope. If we have global scale problems, we are not capable of solving them effectively. We are literally not intelligent enough to handle the problems we are creating. Between rivers of garbage and CO2 levels and war, we have proven ourselves to be woefully unintelligent at the scale needed. If we are lucky, our thin window of survival depends on getting a hell of a lot smarter, real quick.
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
lol, how the fuck is ASI going to solve any of those problems? we already know _how_ to solve them; the problem is that we don't want to, collectively speaking, because certain powerful, wealthy people would loose out if we did. ASI wouldn't change anything. unless you think... all of human society is going to restructure itself around unquestioning worship of the Machine God and would therefore present no resistance to its proposed solutions?
The fantastical belief is that ASI will be able to make things happen because it knows the right things to say to the right people at the right time in just the right way to make them do whatever it wants them to do.
Certain powerful wealthy people aren't omnipotent, them losing out isn't the only blocker to progress.
That’s a possibility, but not the only one. The two most realistic ones are: we race ahead and maintain our status, or we slow down and open ourselves up to colonization.
I think the question “would China cooperate” needs much more investigation. Everyone online pundit seems to think “obviously not”, but they’re people too with clear positive and negative incentives. It’s possible they’ve found a very similar calculus that we have.
Worth noting that it is the Europeans and Americans that have been colonial. Asian peoples have, with the prominent exception of the Mongols and Japanese Empire, pretty much not done that. In particular, China shut down its exploration program.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
China is currently occupying Tibet, which never consented to the occupation and has invaded Vietnam 30 times. It ruled Vietnam for about 2 centuries starting about 600. China eventually had to leave Vietnam, but many other groups ceased to exist as a consequence of Chinese expansion. Here are some:
the Baiyue were a vast umbrella of diverse, non-Sinitic indigenous coastal tribes who inhabited Southern China and northern Vietnam.
The Xianbei were an ancient nomadic Proto-Mongolic people from the northern steppes.
The Di and Jie were two of the ancient "Five Barbarian" (Wu Hu) nomadic tribes of northern and western China during the Han and Jin periods.
The Dian Kingdom were an ancient, sophisticated indigenous southwest culture located in modern-day Yunnan province.
The Tujia were an indigenous group of the Hunan-Hubei region. Centuries of inward Han migration and intermarriage have resulted in the Tujia becoming culturally and structurally indistinguishable from their Han neighbors.
"Asian people have pretty much not done that except for two teeny tiny indiscretions that each killed more civilians than all of Europe's and America's colonial incursions combined."
> The alternative isn’t “everyone dying.” It’s us holding all the cards.
That's one outcome, certainly, but not the only one nor, I contend, the most likely one.
A most likely outcome of ASI is human extinction, because there's more paths to an ELE outcome for humans from ASI than there is for non-extinction level outcome.
Your outcome is only possible if:
1. ASI is never able to escape the confines it is placed in.
2. ASI is benevolent to humans.
3. ASI decides, in the spirit of its benevolence, that it should restrict its involvement in humans.
If all three of the above conditions are met, then sure, your outcome is possible. If not, humanity as we know it will end.
It is unlikely that those 3 conditions will all hold, though.
While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
This is a just so story. The main issue today is the lack of democracy in the country and the use of technology to surveil and govern a restive population as the government has less and less legitimacy. The narrative you are telling is the heroic tale of computing and the internet c. 1990-2010.
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
Yes, and RMS was correct in his Right to Read and so many other things - we're seeing the slow death of the never-enshrined-in-law right to compute. Luckily open-source is big enough to slow this down; we should all be pretty amazed and appreciative that there even are open-weights models at all, out there, because it is a profoundly democratizing thing.
I don't think parent commenter means tech in the modern sense. Seems like they're describing a transition to democracy which started centuries ago, not decades.
> Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
The decision to not go with the development of extremely large thermonuclear weapons might count - the US Sundial Project was supposed to be about 10 gigatons of TNT. Not the most practical weapons but once you get to a certain size delivery arguably stops being a problem - its going to kill everyone anyway so doesn't matter where you let it off!
Perhaps the burning of the library at Alexandria would qualify. How intentional that was is somewhat in question, but the world certainly turned its back on the only collection of written knowledge and let it turn to ash.
> Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60]
> Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]
The Library, or part of its collection, was accidentally burned by Julius Caesar during his civil war in 48 BC, but it is unclear how much was actually destroyed
Isn't the US famous for not even signing a lot of world treaties like climate accords and others?
This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
Biological weapons? Yes, there is research on defense, but no big arsenals of weapons etc.
My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
Taking the approach with AI that we took with the atomic bomb would be catastrophic. If the only people who are allowed to use this technology are governments, intelligence agencies, and a select few anointed companies, then the risk of authoritarian misuse will skyrocket.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
I agree. I guess I should have said something like:
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
Human embryo genetic modification has been effectively taboo if not banned until just recently since WWII and the aftermath of the Holocaust. I think some people in the US are proposing doing it now but I don't think anyone has tried it yet without repercussions.
I can think of many examples that I won't name but you can imagine in biology/medical fields where certain lines of investigation are not performed due to ethical and legal repercussions.
It’s telling terrorists how to make bombs better apparently. Continuing to lower the barrier for that kind of stuff is clearly a negative in the “knowledge deserves to be free” world.
If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?
AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
LLM + scale = more intelligent, this can be proven more than empirically, https://arxiv.org/abs/2409.15318 shows that neural nets can fit a number of independent AND-gate operations in their weights,
- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this)
- and this system can produce outputs based on previous things it has outputed
its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.
it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.
It's not clear that "super intelligence" is a meaningful concept. This presumes that our concept of intelligence can continue to grow beyond human capacity as opposed to asymptotically approaching what humans recognize as "very intelligent". Perhaps, for instance, how we evaluate intelligence is bounded not by some quantitative capacity but rather our inability to agree on basic concepts/values. And do we even have tasks that a supposed superintelligence can tackle but humans cannot?
I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.
Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.
The Orthogonality Thesis (by Nick Bostrom) says that intelligence and ultimate goals are independent. Those non-instrumental goals can’t be stupid nor right or wrong. Increasing intelligence will not change the goals only the capability to reach them.
Do note that the orthogonality thesis is a hypothesis, not something we have demonstrated. Weak versions of it (e.g. it is possible to have an intelligent agent with arbitrary goals) are more likely to be true than stronger versions (e.g. intelligent agents we build will have goals uniformly selected from the space of all possible goals).
I like your optimism and I think you will be vindicated. AI is democratic and AI talent is globally distributed. It will just take a while to get online. AI labs do not have a monopoly on human talent, and open source AI only empowers independent science and meritocracy.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
There is no rein on progress. You can buy electric motorcycles without speed governors. And you can drive them on public roads (subject to traffic laws) as long as you're properly licensed, registered, and insured. The distinction between electric "bikes" and motorcycles is mostly artificial.
I found the AI 2027 paper to be overly optimistic, but not wholly fantastical. This paper feels wildly speculative, and relies on premises I am not confident even pass surface reasoning. Even under optimistic conditions, we are not going to see robots "capable of 95% of all cognitive and physical tasks" by 2035. Nor do I think a 74% unemployment rate is even remotely possible. Economic collapse would implode AI development long before those figures were plausible.
The "and physical" is the part I'm particularly skeptical of. Sure, drones are scary, but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly.
The book Sentient is not about AI but abount the most amazing physical senses some other animals have.
The theme of the scientific findings is that while humans excel with none of our physical sensors, we do very well across the board in making use of them thanks to our relatively huge brains.
And fantastical amounts of compute power is exactly what are handing over to AI. The fact that their training data isn't perfect may matter less.
Nobody’s even solved a self-driving vehicles yet, not in in the sort of “they took over everything and put every uber and truck driver out of business” kind of way.
Maybe they will soon but it’s massively far behind the kind of timeframe AI 2027 would have implied.
I actually think self-driving is one of the easier paths of development. The main thing holding it back right now is regulation and liability.
But, if you could wave a wand and eliminate all legal and liability hurdles to self-driving, automobile deaths would plummet. They're way safer than the average human driver. The technology is definitely capable, our society just isn't ready for it.
I disagree. It has not been figured out enough to take humans out of the equation from a functional standpoint, not just a regulatory one.
This past 4th of July weekend a Waymo ran over an actively burning firework in a low speed scenario, the kind of thing a 95 year old driver would have avoided.
It has. For example mechanization of argiculture in places where it didnt coincide with a manufacturing boom (latin america, india, africa) resulted in shantytowns and long term unemployment.
If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
> "If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me."
It's 2026, one year after your predicted date, and that still hasn't happened though.
You say "just" 15 years, but Waymo is still only available in a few cities. That seems more like a slow, cautious rollout to me, not a fast takeoff. Society has had a lot of time to get used to (and tired of) the idea and come up with regulations.
My guess is that the deployment of other types of robots will often be a similarly slow grind.
That's unlike the Internet, smart phones, and coding agents, which got user adoption at a much quicker pace.
For a specific technology to become viable, you often need progress in several individual technologies to occur. Some of these will be exponential and surprise us, but many will see slower progress.
Either way, I suspect progress in robotics will be slower than AI, but I also think we'll see a lot of investment in robotics over the next decade given that AI opens up a lot of new potential applications for robotics.
I suspect any task that doesn't require extreme battery life or extreme dexterity will be doable by robots much sooner than most people think today. I think we could have decent-ish humanoid robots within a decade – robots that can prepare food, clean hotel rooms, empty bins, etc. I think Elon's bet on Optimus is quite likely to be viewed as very ahead of the curve with time.
However, it will take much longer for robots to do everything a human body can, but then most jobs don't require we push the physical limits of human body.
That's why I think 60% is reasonable. And even then I'm not saying it will happen, I just don't think that's a bad bet if given even odds.
The biggest issue with 2027 was that it didn't understand the economy.
For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)
Without massive amounts of investment, AI development stops dead.
In this post, they hand wave about the USA being able to acutally 1) build concensus locally for regulation and 2) the rest of the world actually follows suit.
It fails to understand that actually the progress of AI is not actually the gift of the USA. It requires a constant supply of things from china.
Also its assuming that having 74 billion agents doesn't cause economic distortion. Like what value are these agents generating that justifies them being run?
I really wish people would just ignore this for what it is: bad sci-fi with an incomplete world.
Which predicts that explosive growth of robot production will lead to problems such as
> a deflationary debt spiral, where the AI and robot companies can’t pay back loans in dollars because the robots and AIs are worth nominally less than the loans written the year before.
In other words, the companies go bankrupt because they produced an oversupply of cheap goods, the bubble pops, and there's less new investment for a while. Plenty of precedent for such a development.
But instead of adjusting their predicted output growth downwards accordingly, they instead propose that
> One way to solve this could be for the loans to be denominated in AI and robots, so the companies pay back the loans with some percentage of the AI and robots instead of dollars.
Try doing this today with a battery factory for example. You expect that battery prices will fall to the point where the revenue from selling batteries won't ever cover the cost of building the factory. So you propose to a bank that they'll be the ones to build the factory, and you'll borrow it from them (not paying rent?), make your batteries, then give back the factory when you're done. All the profit is yours, all the risk is theirs! Which is of course why a real bank won't agree to this, all you're going to get is a dollar loan with the factory as collateral.
> Without massive amounts of investment, AI development stops dead.
Would the US government not pour enormous resources in AI labs if needed, knowing that China might be doing the same? What happens if an adversary develops an AI capable of finding and implementing exploits in every software run by your country's strategic infrastructure?
> For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)
The economy doesn't need workers as consumers necessarily. It would of course be a huge shock to the economy but the economy could adjust to it eventually. Maybe it compromises the 2040 timeline. Still, billionaires are increasingly holding the assets. The money supply drying up can be countered, as it is controlled by central banks and can be arbitrarily increased.
> as it is controlled by central banks and can be arbitrarily increased.
Its really not controlled by central banks. Its influenced, but not controlled.
When central banks "print" money, they effectively just add money to the accounts of investment banks
But investment banks are also "printing" money. Double accounting effectively uses assets to double the available pool of money. If you then sell off those loans based on those assets, then you crystallise that new money. Investment banks are inflationary.
It seems to me
we’re already at the top of the S curve, not at the toe of an exponential curve. At least with LLMs. Better training data will make small improvements, better architecture will make it less compute intensive, and all these “hyper-scale” data centers will make it cheap and ubiquitous. But none of that is it getting exponentially more intelligent.
People have been saying this since GPT-1. This idea that we can only squeeze a little bit more of intelligence out of LLMs isn't a new one. And thus far, it has always been wrong.
> AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.
> ...
> So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.
So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).
> So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence.
On its own it wouldn't. But that article came before the later article https://www.tobyord.com/writing/hourly-costs-for-ai-agents which adds the claim that inference (along with everything else being employed at present) is scaling poorly with increasing task lengths. Now maybe the December 2025 claim is wrong, or maybe things will change soon, but the February 2025 article surely doesn't establish either of those.
I remember being blown away by o1-o3 family of models finally stringing together coherent agentic tool calls to write and execute scripts semi-reliably for workloads in the several minutes before they would start hallucinating/flailing. GPT 5 was a bit ahead of that, but barely
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
Not only this, but in the GPT 5.0 era, agents had 0 taste. Nothing looked good. It was the agentic version of the twitter bootstrap era, but worse somehow. Now, I would argue the average agent frontend beats the average human frontend. This isn't even getting into 3D applications in the GPT 5 era
Anyway, the models now reliably execute more than a human can fit into their own context. It's magic
Yes, and we haven't even really begun to nail down computer-use agents yet (can you believe they're still basically just OCR'ing screenshots?)
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
It's not the same thing. For example, given a GUI with a titlebar, title, subtitle, text, and buttons, a human can instantly understand spatially the relationship between these items. But a naive OCR of such a GUI would be a flat stream of text that loses a ton of information.
Can you share what's your setup for all that orchestration? I feel way behind just asking Claude Code for code edits. Is there any site where people share different AI setups, besides youtube?
Fwiw, don't buy into all the hype that you're falling behind. Yes, AI does cool things now, but I would say the impact is still unproven past indie hackers or early-stage startups. And a lot of the esoteric setups people have created with things like OpenClaw have become outdated as quickly as they were conceived.
The popular thing is now to setup loops (eg I setup hourly integrations for Claude/Codex to 1) scrape my Linear, claim achievable tasks, and push PRs or 2) do root cause analysis on customer issues that evaded automated filters, to name a few)
Though for me, my setup still feels mundane. I have AGENTS.md, CLAUDE.md etc and a few skill files. These are purposefully light - tons of examples online you can pull from online. Mine are fairly personal to my setup and products.
Importantly, I also allow Claude and Codex to bypass permissions. Yes, there is a risk they wipe my machine. The productivity upside has been worth it, for me (haven't been burned yet, ~9+ months into running models this way, I have backups, use cloud etc).
As far as maintaining quality, one of the most helpful guardrails over the past year, for me, has been requiring my agents to pipe their changes to local reviewers through OpenCode, Cursor, etc agents to have a council of models with different biases reviewing the changes, and autonomously working towards a completed objective. No matter how good Claude or Codex gets, for example, I will probably always want a different model checking its work. Like GLM, (now with 4.5) Grok, Composer.
Several OpenAI, Anthropic, xAI employees, and popular AI engineers post on X and share helpful tips & updates. Highly recommend for keeping a pulse on startups and AI. I haven't found something close, honestly, other than when I spend time in SF talking to people.
Most companies have already given up on AI. It was a bigger disappointment than big data.
The only companies sticking with AI at this point are the major players who have chasing their masssive overinvestments to the bitter end, crappy coding shops, and for some reason Starbucks and they'll all suffering badly in a year their customers begin demanding massive amounts of human coding to repair all the issues with the AI code.
5.6 to 5.0 is a big enough of a jump to say yes. if it was 5.4 to 5.6 it would be a bit easier to say it only feels true because of that, but 5.6 is definitely better than 5.0. I don't have anything empirical to point at though, which is your point, but August 2025 for 5.0 vs July 2026 is almost a year later, and it's not just vibes that it's better, despite not having an objective metric to point at. It would be more scientifical to have numbers and shit to point at and there are some benchmarks out there, but you have to dig into them and really understand them in order to believe in exactly what they're testing, and I'm betting you haven't.
I completely agree, but I also think that an industry disrupting architecture tweak akin to the “Attention is all you need” is VERY possible to emerge at any moment.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
please bro just one more decade bro i promise AGI is right around the corner no the hockeystick graph isn't made up bullshit we just need to wait a few more years bro please please bro
This seems way more accurate. Up until early 2026.
Though it has bizzare fixation on geopolitics and China which it severely understimated. It's pretty obvious that China is going to outinnovate and outcompute US companies quite soon. Even if just because they care about higher education, providing enough electricity and letting smart people do smart things instead of randomly muzzling them with bans and export controls and coddling them with financial protectionism.
I wonder how much of what China is currently able to do it because USA is such a mess.
The whole article is kind of ridiculous of course, and is also heavily fixated on OpenBrain, whatever that is.
I also wonder about the economics of running an AI lab attached to an existing large tech company (such as Meta or Tencent) instead of a dedicated company like OpenAI. It's starting to seem like it's not possible to charge enough for current-gen AI usage, with current-gen inference technology, in order to turn a profit, i.e. nobody is able or willing to pay at least marginal cost for tokens.
> This seems way more accurate. Up until early 2026.
Which is funny, because they launched the AI 2027 site in 2025 and it caused a lot of people to believe the end was near.
They claimed to have built a complicated model, but several people showed that it didn't matter how much you changed the inputs, it was designed to converge on the answer they wanted.
- whenever i read this, why does it always read like some first world country guy born in a bathtub who has never been outside his bubble, never seen what the rest of the 90% world looks and works like actually sat and wrote a work of fiction
- have you seen the actual problems in other countries outside america?
- have you been to any country outside USA and the countries in europe?
- have you taken a trip to any developing country and stayed there for a month?
- have you seen what sort of daily struggles, political systems, bureaucracy and work exists in developing countries?
This is by far the most realistic optimistic AI takeoff scenario I've seen, and more specifically it's the first one I've read that deals with both the AI alignment and power concentration issues in a sufficient way, even in a world where hard alignment is assumed (in this scenario the AIs are assumed to be misaligned until ~2038-39).
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
>We already wrote a scenario about this, called AI 2027. It depicts takeoff happening in 2027 instead of 2030
So in less than 3 years, their exponential growth curve doomsday prediction has moved back 3 years. This seems to be the opposite of exponential growth.
Fast takeoff was always only possible if you ignore all of physics and everything about how practical reality works, nothing exponential lasts for long in a finite system. If they believe that, then a certain level of self delusion is required anyway and you can ignore anything you like. Sort of like communists saying socialism will definitely work this time.
The book Superintelligence was so highly praised years ago but if you actually go and read the thing today practically all cases it presents read like raypunk retro-futurism that makes up imaginary fantastical nonsense in place of the missing knowledge it would've needed to make any sensible predictions. Practically none of its assumptions apply to LLMs as they currently exist, and some we've learned since about human inteligence are wrong too.
If carbon taxes are already a lethal policy for an political campaign, it's absurd to think that fears of ASI will create any real movement around pausing AI.
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
If that is true and one cares about a moratorium on progress in the US then it seems like the number one way is to meet people where they are: so water use misinformation, degrowth, power supply constraints. That does place all the people who push for these things in a different light. They may well be attempting to do what the AI safety labs are ostensibly trying to do.
As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.
If there's going to be any pause, I'm sure it will come from a populist movement. I just can't imagine misplaced worries about AI water use will translate into the kinds of policy the authors want to see.
Yeah it’s like shoving the top of a double pendulum. You will get some movement in one direction, but where it will precisely land is hard to predict. The water-use argument is already earning refinement by differentiating “AI datacenters” from “normal datacenters” in an effort to control the movement.
I imagine any populism movement will require rampant fearmongering to get a result. Considering the rough present alignments, presumably blue tribe focused propaganda will involve climate and inequality focused fear and red tribe focused propaganda will involve job loss. Grey tribe positioning is the P(doom) meme where everyone is rewarded for a high-P(doom) estimate.
It's possible the general public wouldn't care enough one way or the other such that elected representatives could just do it without any change to their electoral fortunes.
I don't think it's a gotcha at all, they openly said it and were pondering how that might help them. Misinformation plays more to sides that are wrong about more things, and I don't consider misinformation my ally in anything.
This criticism, like most criticisms of AI, addresses the wrong part of the problem. The problem is harm to other parties. Typically customers and employees of corporations.
A solution is strict legal liability. Corporations must be strictly liable for harms. That liability should be higher when AI is involved. Such liability may not be waived by contract, forced into arbitration, or devolved upon a third party service.
Then we let the plaintiff's bar and the insurance adjusters price that risk.
Something like this turns on in the EU in October 2026.[1]
No, nobody can be held liable for the extinction of humanity, for there is nobody left to pay up. If your objection is that you think AIs won't cause human extinction, you should say that instead, as that would be where you disagree with the authors.
"AI will most likely lead to the end of the world, but in the meantime there will be great companies.”
Is there any serious journalistic source suggesting that this was anything other than an offhand joke? This article links to a youtube clip of the comment with context removed, but hair raising comments.
Taking the most uncharitable view of any person, you could imagine someone who was evil enough to cause the end of the world after their own lifespan where they faced no inconvenience, but not the circumstances from the quote
The quote as it stands is preposterous enough that I don't think a human capable of functioning in society would seriously say such a thing.
Are people wilfully misinterpreting the comment, or do they truly believe this an actually held opinion? If so, can they explain how they think someone could hold an opinion like that?
The question (7:35) is "Where would you like to see people investing more time?" And Sam seems to be saying AI safety, I guess? This is 2015, and he refers to founding OpenAI. Based on his actions since then, yeah, seems like it's not a joke to him. This is Altman we're talking about.
NYT reported today that Russia and China are funding anti-datacenter and anti-ai hysteria on western social media.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
All news is influenced by what’s popular social media these days. And that becomes part of what people talk about through the grapevine.
But no doubt there’s plenty of organic NIMBYism, anti tech growth stuff, and run of the mill fear of change and loss of control as society grows more abstract/centralized.
Applying the term NIMBYism to anti-AI and anti-DC sentiment is a gross abuse of terminology. Datacenters don't need to be in anybody's residential neighborhood.
These developments are always facing local resistance. I previously lived in a place where they were building a quarry 30min outside of town and tons of houses had anti-quarry signs on their lawns. It was a big deal to appear anti-quarry even though it had little to do with their neighbourhood specifically (except maybe increased highway trucking traffic).
Most industrial development face local protests like this and it's often has large crossover with those NIMBY who resist stuff like housing developments, and show up at town/city councils.
Living anywhere near a quarry is no joke. There will be a lot more heavy trucks on the road, with drivers who care very little about covering and securing their loads and even less about your windshield.
Everyone lives in a world deeply affected by social media. Even if you've never looked at a screen your entire life, you have spent thousands of hours talking to and being informed by people who did.
World, yes, as people around you including politicians and businessmen have had their brains pickled.
But I doubt the influence. I’ve been free for years, and I can always spot who’s spends a lot of time in TikTok/Twitter/Instagram—it’s like talking to someone from another planet. It mostly sounds weird and sad, more apt to annoy and alienate their friends than inform or influence them.
Which then gives more ammunition to folks like Kevin O'Leary et al to pretend any objections to data centers or AI are Chinese plots, or "hysteria" as you put it.
While I can admit some of the anti-data center arguments are overblown, many are more than valid in my opinion. Data centers are fundamentally extractive technologies. They are enormous, windowless boxes that take resources from one location to make someone else in a far off location enormously rich and powerful, with extremely few benefits to the local community.
Plus, as another commenter mentioned, it's not exactly like the Chinese and Russians have been fanning the flames that AI is going to take all of our jobs - it's the leaders of the frontier AI companies in the US saying that. Remind me again why I think putting up a giant data center in my state, that was proposed to use more electricity than my state already currently uses, is a good thing for the average joe where I live??
Honestly, I feel like many commenters here are in their own bubble and don't understand how much AI and tech generally is widely viewed as a net negative for society by huge swaths of the the population, and I don't really think it's an unwarranted perception.
Most of the issues with datacenters could be solved by a) investing in energy b) ramping up RAM and chip production, and c) enforcing already long established rules around industrial water management.
These tech companies are already investing heavily into solar, natural gas, and nuclear https://techcrunch.com/2025/03/30/data-centers-love-solar-he... this would be normal stuff in China where they spend the last decade investing heavily in solar and are bringing something like 60 nuclear reactors online.
These datacenters aren't particularly consumptive of water compared to most other industries in that regard and we've already seen states enforce rules against Meta who immediately paused their datacenter when water issues were detected (following mandatory monitoring).
Chip production is lagging but most projections I've seen is it will normalize in about 5yrs. Not to mention there will be further demand for robotics and self driving cars, so ramping up chips should be a normal thing like ramping up green/nuclear energy. Delaying it won't solve any current issues.
People are scared about the personal impact from AI, then backfill in justifications without even realizing they're doing it.
If the equivalent numbers for electricity and water usage were being being used for streaming video, I seriously doubt people would be demanding no more Netflix data centers. The news story would immediately die.
Nobody wants their electric rates to go up, the local water utility to have to raise rates to build a bigger plant, all in exchange for also losing good white collar jobs. That’s currently what AI data centre builders are selling.
Not exactly. There are some data centers being built in places that don't have the power and water to support them, and obviously it's rational for the locals to oppose them.
But I live in a place where we have plenty of water and relatively cheap power (lots of renewables). There's not much risk to data center construction, but people are opposing it here, too. Because for most people, it's not actually about that.
An obvious question is if the cheap power is going to stay cheap after a large power-user comes in who has a proven track record of trying to make everything cheap for themselves with no regard for anyone else.
Or another question to ask is - how does this data centre benefit the people who live there? If it doesn't, there's no reason they should want one to be built. Rubbish tips are necessary. I still don't want one built next to my house and would fight such a thing tooth and nail.
Here's a better question: as we've had nearly as large of a data center build-out happen between 2005 and 2020 for non-AI purposes, with similarly high electricity and water demands...where has the concern been? Why is it only in the last 2-3 years that people are suddenly up in arms, as a very specific application is being deployed?
The amount of data-centre construction is far more than it was 2-3 years ago.
There seems to be an alarmingly high amount of questionable data centre construction going on, such as projects being built in places with no access to power with an assumption they can somehow force the utility to provide it later. These buildouts seem to be being done for financial reasons (they are not Meta, Amazon, Azure, etc. facilities) with the hope to lease them out or sell them half-completed in the future. People rightfully don't want that kind of thing in their back yard.
To give a feel for the scale involved, this one (the new Amazon east DC) in my podunk area of the state is 250 MW (the existing us-east-2 in Columbus is 200 MW, although I'm not clear if this will be a new region or is just an additional availability zone). But that's small potatoes compared to the speculative project in Piketon, which amongst more absurd things is planned to be:
- 10 gigawatts (equal to 50% of current power consumption statewide)
- "Modular" nuclear reactors built on site
- 35,000 construction workers needed to build it (in a county with a total population less than that)
- $30-$40 billion for the data centre, plus another $33 billion to build the 9 gigawatt natural gas electric plant
- Meta agreeing to build an additional 1.2 GW nuclear plant on site
- OpenAI in negotiations to lease the facility
This is a really big project, of the scale of "nothing like this has ever been done before". Nobody has ever built that much power generation at a single site before, nor has a datacentre this large ever been constructed. There is a very real risk of the project getting halfway done and then being unable to be completed. The prospect of a state literally doubling its electric generation is a bit ambitious, too (doing such means basically a complete revamp of the power distribution grid, or else some very novel designs to only use the power locally). For example, the normal type of shutoffs data centres have to prevent eg an incoming have are unacceptable in this situation because the grid cannot cope with 20 GW of demand suddenly disappearing.
People have a dozen reasons to refuse data centers being built in their communities and zero reasons to encourage it. They're little more than post-industrial mines that take limited resources (power, land, water, quiet) from a community, sell them for profit as compute, and siphon those profits away onto the books of far-flung megacorps with no community reward.
Meanwhile, golf courses are a traditional green space where people in a community gather for both work and leisure. They're not ideal themselves, but they at least provide some benefit against which their negatives can be weighed.
If all you hear from critics of data center building is water use complaints, that's strictly because you've chosen not to listen to people.
It takes the limited resources of land and water from a community and sells the result for profit as food or fuel. The vast majority of profit is made downstream and outside the community.
Golf courses being a traditional green place where people gather seems a bit far fetched to me when most of them are elite private clubs.
Most golf courses around me are open and anyone can go play for a cheap greens fee. The clubhouse has normal low end restaurant prices for a hot dog or a burger.
As far as "valuable", the golf course is private property that the owners presumably bought and can do with as they see fit. If you want to develop it into something else, make an offer to them?
The golf courses are a business like any other (although we do have some publicly-owned golf courses around here too). The cost to play 9 holes on a weekday is $10 at one of them. I'm not really sure what you're asking for here.
That's highly dependent on the area it's in. Where I live, golf courses use no more water than anyone else and aren't irrigated.
The proper answer would be to simply charge appropriate prices for large scale uses of water from the water utility, or else this is a discussion about riparian rights and law and possible changes needed to that.
I associate golf with WASP-y business types who hobnob with their boss in an electric cart to get ahead, and can't stop talking about their hobby of chasing a little white ball.
Is that fair? Probably not. But I don't think golf is a particularly inclusive sport, unless you live in a golf course gated community... in which case everyone is included.
People are more lenient on farming because humans need agriculture to survive. Obviously not all agriculture is necessary for survival, but it's something that in some way provides real tangible benefits for everyone.
On the other hand, if AI data centres all disappeared today, humanity would continue on completely fine.
Can you point to some examples of where that's happening?
These data centers are specifically being scouted for communities whose governance is too weak to negotiate for some "sizeable share of their profits" and too ill-prepared to have suitable environmental regulations on the books already. The Ivy League sharks planning these buildout initiatives are sharp people who are looking out for the interests of their employers and know how to pick locations where they have the best opportunity to exploit locals unprepared for their kind of esoteric deal-making, political lobbying, and lawfare. They'd be failing at their job if they did what you're suggesting, and that's why we don't really see that happen.
For any benefit to national or global society AI data centers might provide to someone, the buildout looks a lot like the dirty and exploitative stories of rail expansion in the 19th century. That rail infrastructure proved a good thing for the US, but that doesn't mean the process of making it happen was honest or good for the people immediately affected.
This may sound crazy, but I’m glad the data centre being built near me (the new us-east-3) is being built by Amazon who pays lip service to local government and the community, as opposed to cartoon villain levels of saving a few pennies by forcing noise pollution on everyone else and everything else undesirable the other builders are doing.
> Can you point to some examples of where that's happening?
You mean the AI datacenters doing it? No, they are not doing it.
They seem to be doing the opposite. They being loud is really hard to accept, decorrelating the fans cooling them would probably pay for itself in less than an year. It's like it's some Capitan Planet villain building those things.
States give the tax incentives, local municipalities are cheap to buy out, that's why all these towns quickly pass approval. $30 million annually is pennies for a datacenter and a windfall for a town.
Coming in for a landing on a flight recently, I was amazed at the contrast I saw between a housing development of 50+ homes and a golf course across the road that took up the same amount of space.
I attended an auction of a golf course a few years ago. It went for a few thousand an acre. You could have shown up and bid on it.
The winner ended up just choosing to keep the current employees and keep operating it. Nobody, I mean nobody, wanted the land for development. It was in an era with basically no zoning either.
I'm saying that if you don't like how a golf course is being used, then attend an auction of one, because they are rapidly going out of business and the assets being auctioned off.
America is not suffering from too many golf courses being constructed. They are, rather, going extinct, and I don't really think the mass loss of green spaces and third spaces is necessarily a good thing, even though I'm not someone who enjoys playing golf and don't really spend any time at golf courses.
> I don't really think the mass loss of green spaces and third spaces is necessarily a good thing
Maybe we could make them public green spaces and third spaces, instead of exclusive clubs?
Many golf courses are really expensive. Golf itself is dying like you said, because it's a very expensive sport
Idk. There's something like 35 golf courses where I live in Calgary and it's a city of less than 2 million people. That seems super unnecessary to me and they don't seem to be going extinct here
There are also 132 Tim Horton's, which aren't "necessary" either, but people like having leisure activities. One golf course per 55,000 people doesn't seem that excessive either.
If you'll indulge me some armchair research and math
According to google AI, the average square footage of a Tim Hortons generally between 1000-2300 square feet, with some older locations taking up 2500 or more.
So let's assume every one of the 132 is an older location taking up 2500 square feet. That's 330,000 square feet, or 7.57 acres
According to the same AI, the smallest golf course in Calgary is:
Lakeview Golf Course: The city's smallest 9-hole executive course covers approximately 60 acres and features mostly par-3 and par-4 holes
60 acres!
Unless my numbers or math are wrong, Literally all of the Tim Hortons in the city can fit into the footprint of the city's SMALLEST golf course
These things aren't equivalent. Come on. We can use the space for golf courses for better things. :/
Edit: That's not even accounting for the fact that a single tim hortons probably serves more people in a couple of hours than many golf courses do in the course of an entire day.
I don’t get the appeal of golf. Hitting a ball into a hole with a crooked stick? But hey, that is just my personal preference. People can play whatever sport they want, or do whatever they want and call it a sport.
That said, it would be nice if the so called sport didn’t take so much land, water etc. Especially in prime locations
People can pretend to care about more than one thing.
Whether they actually actively oppose those things to the point of impacting building permits, that's a completely different matter. It really doesn't take much legislation to make golf courses economically unviable and force them to close, especially if you've got enough population within 30 minutes to support 22 of them (I speak from experience, I helped write a water reclamation ordinance that shut down at least one in my SoCal city)
If anyone actually bothered to talk to their local reps instead of posting internet comments about how much they "care", they'd get something done. If they don't, their care is just a fart in the wind for all the good it will do.
Golf courses don’t have backup generators running 24/7, with humming you can hear from a meaningful distance away. They also don’t pollute the air.
This is a poor comparison, but I do get what you’re attempting here. It’s also absurd that we are leveling land everywhere around me to build warehouses. No one is really complaining about that, either.
Russia and China aren’t the ones constantly telling us AI will put us all out of a job, here’s why that’s a good thing, and why the government should dedicate billions of dollars to incumbent AI providers.
Surprisingly enough the us constitution specifies a right to free speech. Meanwhile who is funding the past two decades of pie in the sky bullshit from silicon valley in the media? Is promoting and unpopular opinion illegal? Since when do we judge the merits of an argument based on who articulates it?
Did anyone else catch the logical inconsistency between Plan C and A?
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
By legislative design? If a nuclear bomb could be made with hardware store finds the world would already be over. Big collaborative works raise the stakes and the observability for surveillance. Apply for a job at a defense supplier or even and energy company.
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
In Plan C the government essentially misses the opportunity to implement the multinational deal while the threat of covert projects is still low (fewer latest-gen chips unaccounted by tracking measures, worse models/algorithms to use for RSI). That's why it says the probability of a deal is lower and lower each month rather than outright zero.
The AI narrative requires an enemy and a “race” that the US has to win. It’s a pretext for justifying spending, rules, urgency, etc and making it sound more important.
Nothing follows from this empty platitude though, right? It can't inform you choices or decisions? It's just a disempowering thought?
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
Degrowth is a deeply unpopular policy around the world. Where would one even go to find a degrowth rally? I have to imagine everyone there would be there ironically.
> * they net shutting down grade schools or building them in your neck of the woods?*
My area (rural Iowa) has had several new schools built in the last 10 years. Net gain for sure.
AI is just a tool. It will be a tool as we constantly push the requirements of what qualifies as consciousness. Therefore I'm not worried about it. Almost all the predictions about future growth of technology were incorrect. We don't have flying cars, hoverboards, fusion, the matrix and so on. The goal of each of us is to live a better life. It always be and forcing people to give up luxuries for some external agenda based on slippery slopes is cruel and selfish.
We should have been under water, hunted by AI, overpopulated, killed by terrorist, smitten by god for our sins and so on. Luckily all it took was our privacy and a lot of tax money to survive.
Whenever I encounter these people I'm reminded of the meme about the baby who has doubled his weight in the three months since birth. At that growth rate, he'll weigh trillions of pounds by age 10.
Point taken. But it's interesting that the example you give is one of exponential decay rather than exponential growth, which is the context of this discussion. Where in nature has anything ever grown exponentially in perpetuity? Bacteria in a petri dish multiply exponentially. Until they don't.
Most AI alarmism benefits big tech itself. Either they want to create awe and thus demand for their products (see the recent dance around Mythos), or they want to create regulatory capture and increase the moat around AI.
Interestingly , we as a species already created an overreaching "cybernetic" system that controls our global society and that the individual is powerless against - it is not AI, but capitalism. Thus the current danger is not that AI becomes superintelligent and enslaves us, not that it makes a regime ultra-powerful, but that it increases economic inequality and concentrates economic power in the hands of even fewer people.
The irony is that new technology could allow us to live a life of abundance and leisure, but instead people are laid off, made unable to participate in the economy, etc.. The technology is (as so often) chafing against the bounds of society. I sometimes wonder what the superintelligent AI would say about this, and if it would come up with a completely novel political theory - "silly humans, why don't you just organise your affairs like this, you'd be much happier, and we could coexist much better: ...".
Firstly, let me say that "AI 2040: Plan A" is a nicely done presentation. If feels like reading an interactive graphic novel with a narrator to boot. Can't wait for the book, series or movie :)
Plan A seems like a good start and am glad an effort is actually being made to address any potential dangers. The only weak link I see is that there is no way for inaccessible, third-world countries, non-aligned states, and malicious wealthy rogue agents to be regulated. All I hear is a way for regulating companies that, themselves, legally have to answer or are bound to their host nations. Basically, I don't see a way to hold non-aligned states accountable.
I see a lot of focus on the well-being and protection of AI, which is important to country economies but, the folks that have been affected by layoffs, not necessarily due to AI, plus the workforce that are now feeling the negative consequences of the AI burn are justified in being worried. Anyone that feels the need to criticize them, clearly is not being affected by AI in the same way. Job loss will lead to economic and population destabilization, far worse than anything that has transpired in modern times. Hopefully, those being squeezed now won't be ignored.
This is follows the classic AI policy influence playbook: enumerate a small handful of potential outcomes that make your preferred policy seem obviously correct, ignoring the fact that there are literally infinite more "plans" that could be reasonably predicted that would support wildly different policy prescriptions.
A fiction? almost all is pure fantasy or imagination.
What are your real purposes?
No any definition for many important concepts.
No any logic for the concepts.
What is AI?
What are AI companies?
What is "smarter than humans"? All human?
How to check "smarter than humans"?
"building smarter-than-human AI systems within the next 1 to 10 years.", nonsense, AI can store or handle with all details from reality? Even all data centers in the world cannot store all details of the influenza virus.
No concepts and logic, just a fiction, joke, or marketing article.
I’m not a booster by any stretch but I had a more positive read. To me it seems to describe a post scarcity society where ultimately the data centres are in e.g. the middle of the ocean and the robots’ labour replaces ours while still leaving us with the rewards.
I am extremely skeptical about whether this is even possible and even less convinced it’s a likely scenario but it seems like a good path.
The time for an AI pause has passed. Asking people not to race to the finish line is reasonable when the finish line is very far away and people don't know where it is, but not when it's the obvious next step on their current path.
There's a good story about this, in Star Wars. The clones vs. the droids, basically. Droids being somewhat sovereign, not one large intelligence like the clones.
The only choice here is to go to sea and get away from the crowds and the bots. Bots don't like salt water much, so I'll see you out there.
Remember people. We didn’t die form the 2k bug. We didn’t die when the Maya calendar ran out. We didn’t die from that asteroid or the other. Spreading fear is a very old pastime. How is to say AI won’t hit a wall in 6 months and we’re left with barley passable code parrots forever?
Exactly, this article is fear mongering at its best for click baits. I’m pretty sure they will publish another one next summer for AI 2028 with same narrative
The thing that continues to irk me about these sorts of "papers" is how they refuse to remotely acknowledge the possibility that LLMs and Diffusion models won't lead to AGI, ASI, or whatever acronym they're foisting upon the populace.
If you won't even so much as acknowledge the possibility of error, your argument is hollow and empty. All the "choices" presume these labs are being completely honest and acting in some degree of good faith (relative to the systemic incentives of society in its present form), while in reality we're still just building and refining probability models with increasing accuracy of output and flexibility of processing (namely agents) but still lack actual "intelligence" of any real sort.
Show me a paper that doesn't merely presume inevitability of LLM-based AGI/ASI, and instead actually lays out the core paths that history suggests we're likely to encounter with any "world changing technology":
* In the best case, that the technology really will revolutionize the world and do everything promised by its biggest boosters (papers like this one)
* In the middle case, that it becomes just another tool in our collective toolkit, and the consequences of a revolution built on external investment fizzling out
* In the worst case, that the tool itself is so niche in its utility that investment collapses rather than fizzles out; what do we do with all this compute, now? Who owns the debt? Who foots the bill? How can we mitigate those existential risks?
I'm just rather nauseated by the continued trot of inevitablism masquerading as academia rather than an actual, neutral, bias-controlled-and-disclosed study that paints potentialities instead.
---
Having finished skimming through it, another comment springs to mind: Jesus Christ these things continue to be jingoist as absolute fuck. It's a fancier set of makeup for the same shitty western chauvinism worldview of American excellence and Manifest Supremacy.
Nah, I'm done with this trite garbage. Go proselytize to idiots, I'm not one of them.
> It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.
> ...
> Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.
> ...
> If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.
I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".
Just enforce DMCA and all the Ai super powers will suddenly become more dumb for the next 10 years.
Of course that’s for the kids pirating a movie not for trillion dollar companies
Here's my prediction. Plot the publication dates of Daniel et al.'s two existing works and future ones against the years in the titles, and you'll get a hockey stick curve.
People overestimate progress in physical world.
2035:
robot population will soon be larger than the human one
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
Hey, author here! Scott did contribute, but less than before.
On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"
But do they at any useful scale? China is probably the only other country deploying AI at any appreciable fraction of their economy, and it's certainly much less than the US.
One of the best comedies of the past decade got a sequel. That is all there is to say about that effort. The Europe 2031 also gets an honorable mention.
This is dangerously naive and misguided. They claim to want to avoid centralization of control but propose a world police state of AI regulation. Governments exerting this much control will only end in war and tyranny.
If the production, and distribution of the tech is confined to one or two companies, then those two companies and the resulting power that comes to them, makes them the de facto world government. (Democratic choice may be a farce but it is a useful friction on net.)
If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.
Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.
Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?
If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?
That bet isn't one anyone will take the other side of, surely - how is your counterparty supposed to collect if they win? I guess you'd propose effectively supplying a loan with a massive interest rate?
May be it all depends on how AIs can help us solve the problem. By now, they are more capable than many people that I deal with. So, may be the problem can give us the solution. May be they can figure out a way out of the mess. And sorry to say this, but China and the US are not going to solve this problem, they may even get it worse. Every country is controlled by politicians.
Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities. Furthermore, AI agents lack any notion of an identity, so certainly are not capable of attaining legal personhood or being sued or fired, or owning property. I think slop burnout, cybersecurity, loss of privacy, even environment issues are far more concerning and real issues arising from AI than alignment or the prospect of mass labor displacement due to AI.
> Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities.
There are multiple teams working on adding long-term memory to AI. This is not a fundamental problem.
Fine-tuning to store memory in the weights is something that almost works right now, btw.
Ironically, the origin of "cognitive dissonance" as a concept is attributable to Leon Festinger who (with others) studied a UFO cult called The Seekers in the 1950s who believed in imminent apocalypse. As other commenters have noted, pushing the date back was inevitable. Time is a flat circle and we repeat the mistakes of our ancestors just with different coats of paint...
I sometimes wonder why the algorithms started pushing doomsday scenarios, especially on youtube. Most channels have big red arrows pointing at a miniscule thing and say "its over". Then you get ai 2027 and its more fear mongering, just inside the AI echo chamber on the internet. At work even managers watch odd youtube videos that are radicalized towards politics and so on, but are ultimately head canon slop for ad revenue.
It feels like we are so utterly bored out of our minds and comfort that we make up problems or scenarios, completely detached from reality, or outright irrelevant to day to day reality, just to get through the day and give meaning to our lives.
It is an incredible achievement, don't get me wrong, but what is AI truly going to do when we are already at such a stage of devolution?
It reminds me of the scene in Wall-E when people destroyed the planet and started flying through space to find a new planet, while the captain is an AI and the people became obese in mobility scooters and glued to their screens. I think this is unironically the most realistic scenario for mankind lmao
The whole thing is unreadable, and laughable, just like AI2027 previously. It's really really hard to read someone suggesting that in the United States we will soon have a universal basic income of $1m / year for all people in the country, when you just look at the state of current politics...
I'd rather read something a little bit more realistic.
What predictions about the technology are the authors making that you do not believe?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
Self improvement of agent-1 is not achieved. Sure, people in AI labs write python code with AI, but I doubt it resulted in 50% algorithmic effiecency in training. Writing python code never was the bottleneck, if it was, AI labs could hire more people to do it. And this is core of the prediction.
LLM adoption is 30% in the charts I saw googling for "ai adoption". An example of capibility: I have had Claude one shot an RL agent that learns connect four in 30 minutes. That's PhD level stuff.
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
I personally find AI doomsday preaching incredibly arrogant. You think that we will create super intelligence, something that will exponentially get smarter, and the thing it will do is... end humans? Why wouldn't it just find solutions to problems we struggle with? Also people don't live to work, its asinine to argue AI replacing jobs is the problem, obviously the problem is the gains being concentrated higher up. Another thing is, we argue AI needs to be maximally useful to everyone in the world, and the argument to do that is government control??? I would rather push for AI to be superintelligent faster than any one human can control it. Frankly human controlled superintelligence is likely a worse outcome then uncontrolled superintelligence, at the very least there is no guarantee one will be better than the other.
This is the same nonsense as before except now they are pushing out their prediction by 3 years. This is just fear mongering fan fiction.
They are also claiming that China may go to war with the US if our AI is better than theirs. They are coming up with scary scenarios which realistically won't happen.
>The problem with an intelligence explosion is the "explosion" part.
It's not a literal explosion. If we explosively ended world hunger that would be a good thing. Similar to having an abundance of food for everyone, having an abundance of intelligence is wildly beneficial to society. The article doesn't mention it but an explosion isn't guaranteed we could just see a plateau of capabilities due to bottlenecks of resources needed to power AI, time needed to run AI, and limited interaction with the real world. AI can't run science experiments on its own by the 2030 doomsday timeline
This opening statement told me the bias that the plan had from the beginning. My experience is that you can make slop and you can make art. Just like a paintbrush. I've done beautiful things, and have gotten increasingly better at using the AI paintbrush.
The author(s) are likely scared. And I'm seeing the divide increase with articles like this. Those who don't understand it and won't use it, or learn more, will ultimately have a story (our human condition).
I think AI is going to make a beautiful future, much better than our current one.
I bet this same thing happened with the advent of electricity.
“Engagement with China is impossible because it is a Stalinist one party state under a fiercely ideological Marxist Leninist dictatorship that preaches Chinese racial supremacy over other nations. It will always pose a threat to world peace until the regime eventually falls.”
- Drew Pavlou
... and in the meantime people are looking at their AI bills and realizing tokens aren't worth what they cost. The frontier is getting the cost down, not getting intelligence up. In a cage match between this guy and "Ed", Ed wins.
Sounds like another Chinese Op to me; Ensuring Chinese compliance would be incredible hard to enforce or to check.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
I recommend actually reading their recommendation, because they get into the weeds about precisely how the US and China could address this in a trustless/auditable way. The TL;DR is that basically all of the relevant compute can be tracked.
Edit: Also, definitely not a Chinese op. The authors are prominent Americans, and are the folks responsible for the AI 2027 forecast that has pretty accurately predicted the current state of affairs today: https://ai-2027.com/
Pretty accurately what? So now OpenBrain has an Agent-1 that makes their algorithmic progress 50% faster than other companies? If it was 50% more CVEs, that would be something, but I doubt any meaningful self improvement is achieved which the competitors are slower due that, which is the core of the prediction for 2026.
Yeah, Cognition's work is interesting in that regard, but it still doesn't obviate the need for the chips--it just enables training on them when they're spread across multiple data centers.
The Plan A proposal estimates that the ownership of ~96% of AI relevant compute hardware can have its ownership traced, since the companies selling are very few.
I wish the authors of the article spent more time studying how LLM works before writing a doomsday scenario. Currently no matter how impressive LLMs are, they are just token auto complete machines. It cannot invent or think anything new which is not in its training dataset.
Do you even know what Cycle Double Conjecture is? Or what even a conjecture means in general? It’s a perfect use case for autocomplete with existing knowledge
One of the lead authors, Daniel Kokotajlo, worked at OpenAI for years before quitting. In 2021 he wrote a remarkably accurate forecast of how LLMs would develop over the following 5 years[0].
I think it should be obvious that he understands that LLMs are trained via next-token prediction.
> In AI 2027, we predicted that this would result in either extinction or irreversible concentration of power.
Neither of these come to pass. The first because we don't live in a science fiction universe. The second because AI is a completely open source technology. You can literally make your own models and train them yourselves using state of the art techniques published in free papers. All you need is GPUs and smart people who can read (and now that AI can code, you don't need the second bit). These people are doomer quacks too caught up in fear and excitement to think rationally.
Their expectations and assumptions are all wrong, from the basic understanding of AI in general (like how you don't need a billion dollars or "American Brains" to make a half decent model), to the misunderstanding of market realities and competition in China and Europe. The US doesn't have a monopoly on chips, or on smart people; Huawei chips work fine for AI training and probably a fifth of the US tech sector's best workers aren't American to begin with. There's many forms of AI in many places; munition, economic driver, laborer, generic tool. It's now a part of the world, the way the Internet is. There's no keeping the genie in the bottle. AI doesn't take over the world; it's integrated with it. It's boring.
In a few years people will forget the doomer predictions, the way every doomer prediction that never came to pass is forgotten. We didn't nuke ourselves, the internet and television didn't rot our brains, the radio and telephone didn't corrupt the souls of Americans, the newspaper didn't incite anarchist riots. Every new technology freaks people out, and we adapt to every new technology and make it boring and normal. That's humanity.
This is religious fervor folks, as AI 2027 was.
I grew up in evangelical christianity, and to them the end of the world is just around the corner, the same way it has been since I was a small child and likely will be when we are all gone. This isn't science. This isn't hypothesis experiment record results. This is very expensive astrology, shiny rock collecting, ritualistic meaning-making and self-justification.
Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
It's nuts how well "Superintelligence: The Idea That Eats Smart People"[0] aged. That talk is a decade old by now and still hits just as hard as it did back then, despite the incredible advances made in AI in the meantime.
[0] https://idlewords.com/talks/superintelligence.htm
Yeah I think its possible that for many folks its the first time they're coming up on these concepts, and it troubles them in the same way that the concept of death troubles them (and me, to be clear!).
For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducing any doubt. These are not scientists, they're business people and salespeople, and a few optimists having recently on paper solved all their worldly financial needs.
Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.
>> Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.
Important point. LLMs were early on hailed as the first general-puprose AIs that can perform any task (remember "Sparks of AGI"?). Today they're increasingly promoted for specialised applications - coding, as a for instance.
While there are some coding focused models (composer, for example), the majority of frontier models are pitched as general purpose. The coding harnesses for Claude and GPT are even being repurposed as general purpose knowledge work harnesses.
> whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own.
As usual, AI skeptics are moving goal posts. Modern LLMs are on a completely different level in terms of how GENERAL they are vs anything pre-LLM. You can give it a completely novel puzzle and it will solve it. 5+ years ago you had to train NN to solve particular type of puzzle.
Did you actually read the text? OPs are calling that Plan D.
They're proposing an alternative, which is a global brake on frontier AI research to keep the basilisk in its jar until we work out what we're dealing with and how to handle it.
No, they're proposing a spying panopticon and state control of global resource distribution - specifically general purpose compute - including seizing and destroying GPUs. They're proposing a totalitarian global dictatorship controlling computing hardware and software.
Lest you think I'm being hyperbolic: https://ai-2040.com/supplements/covert-ai-projects
This is arsonists selling fire insurance.
I always thought it a bit pessimistic on the upsides like robot to build us palaces, immortality, conquer the galaxy type stuff along the lines of the "Grandiosity" bit in the essay. It's a shame we have amazing tech possibilities and most people are either saying either it's rubbish and won't work or it'll bring doom. Aside from being boring they are probably factually wrong.
Bit like saying early medical knowledge was rubbish or will doom us by summoning demons while really life expectancy at birth went from twenty something in Roman times to eighty odd now.
This is hilariously true
> AI risk is string theory for computer programmers. It's fun to think about, interesting, and completely inaccessible to experiment given our current technology. You can build crystal palaces of thought, working from first principles, then climb up inside them and pull the ladder up behind you. People who can reach preposterous conclusions from a long chain of abstract reasoning, and feel confident in their truth, are the wrong people to be running a culture.
I understand how people running in the same scene fall into the echo chamber effect and get gulped into the cult, but why does everybody want to be a prophet?
Theorizing about nuclear winter is somewhat similar, in the sense of being inaccessible to experiment. Does that mean we should disregard the possibility of nuclear winter?
As with using the Orion drive for launch and landing on Erth is possible, test nuclear winter is certainly possible. But as with the former, you won't be very popular among the survivors (if any).
Being a prophet is probably great until you suddenly find yourself building a fortified compound in Waco, Texas and purchasing black market full auto machine guns.
Thank you for contacting Gunmetal Ranch, a legitimate 501(c). If your call is related to the class-action "most dangerous game" settlement, please hold for a cowpoke.
> but why does everybody want to be a prophet?
Its not everyone building crystal palaces in their mind, they're all building fortresses. And they can't be wrong in their fortress or it breaks their world view which they cannot accept.
AI is "completely inaccessible to experiment given our current technology"?
I think experiments have happened and even been mentioned on HN occasionally and effected RAM prices.
> AI is "completely inaccessible to experiment given our current technology"?
No, that’s not what the article/talk said. It said AI _risk_ is inaccessible (as the poster you’re replying to correctly includes in their quote). The point in the original talk is clear if you read it; that everyone coming up with their own pet theories for how superintelligent AI could destroy us as a species is building their theory on speculation on top of speculation.
Note that the talk is from 2016, before transformers and LLMs were invented. Though I would argue that the vast majority of it, including the parts we’re discussing here, are still completely valid.
Video link, as the link on the page is dead. https://www.youtube.com/watch?v=kErHiET5YPw
and upon rereading completely holds up technically we are still passing in massive data into simple networks giving no opportunity for introspection or recursive self improvement.
>despite the incredible advances made in AI in the meantime
So the goalposts will be moved whenever necessary in that case?
No amount of incredible advances in AI will ever get skeptical HN commenters to take AI's implications seriously?
"The Gish gallops will continue until the nagging doubts have been silenced"
The other thing about this article is that it contains falsifiable claims that are, well, false. Consider the following direct quote: "The 2028 election cycle is heated, as usual. AI is the biggest topic." No poll of the American public has AI as one of the top concerns except in polls where the question is specifically about AI. The general polls show topics like healthcare costs, inflation/affordability, government dysfunction, and immigration as the top concerns. AI doesn't even show up in the top 20 in every list I looked at (Gallup, Pew, YouGov).
It sounds more like "everybody I know thinks AI is the top concern" when everybody they know is people like them, i.e. an echo chamber. It gives me no confidence in any other claims.
The AI 2040 writing and the AI 2027 before it both sprang from within the rationalist communities. Scott Alexander (of the popular rationalist Slate Star Codex / Astral Codex Ten blog) played a role in both articles, though he says he chose to leave his name off of this one.
AGI and AI doom scenarios have been evergreen topics in rationalism for years. Long before we had enough compute power to do anything at all useful with generative AI. Many of the rationalist scenarios about future AIs had very religious themes where the AI became so powerful that it was effectively a god.
Rationalists had concept's like Roko's Basilisk, a thought experiment where a future all-powerful AI might choose to punish anyone who predicted the arrival of AI but failed to contribute to the arrival of AI.
The rationalist communities took this thought experiment so seriously that the biggest rationalist forum of the time formally banned discussion of Roko's Basilisk for 5 years. They believed it was an "info hazard" because once you knew about this risk, you either had to contribute to bringing about this future AI god or you risked being punished by it in the future.
If this is new to you, you might think I'm exaggerating or making things up to make a group look bad, but this was such a core belief that it has its own Wikipedia page: https://en.wikipedia.org/wiki/Roko%27s_basilisk
The people writing AI 2027 and AI 2040 were immersed in these cultures. That's why they think that AI is the primary topic that anyone is talking about in elections, just like the Bitcoin people all thought crypto was the only relevant election topic.
The GP comment is getting attacked, but there really is a heavy religious fervor background to the AI fears in the rationalist community. The religious angle and extremism of the forums is downplayed when in crosses over into mainstream topics, but it's been there from the start.
First I was intrigues by the rationalist movement, but I was deeply disappointed. To put it bluntly, seems to me like geeks LARPing as philosophers, and not with rational argument as they promise, but rationalizing their sci fi fanfictions.
Yes, calling themselves "rationalists" seems to be the biggest tell. That they wish they were that. Yet their behavior reflects anything but.
If I recall correctly, the text of Yudkowsky's response to Roko's Basilisk used to be part of the Wikipedia page but was removed and not even a reference link was provided even though the response was mentioned in the body of the article. For posterity's sake, from https://rationalwiki.org/wiki/Roko%27s_basilisk/Original_pos... here it is in all its hilariously unhinged glory:
"One might think that the possibility of CEV punishing people couldn't possibly be taken seriously enough by anyone to actually motivate them. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous. I don't usually talk like this, but I'm going to make an exception for this case.
Listen to me very closely, you idiot.
YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL.
There's an obvious equilibrium to this problem where you engage in all positive acausal trades and ignore all attempts at acausal blackmail. Until we have a better worked-out version of TDT and we can prove that formally, it should just be OBVIOUS that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT DETAIL that they have a motive to ACTUALLY BLACKMAIL YOU.
If there is any part of this acausal trade that is positive-sum and actually worth doing, that is exactly the sort of thing you leave up to an FAI. We probably also have the FAI take actions that cancel out the impact of anyone motivated by true rather than imagined blackmail, so as to obliterate the motive of any superintelligences to engage in blackmail.
Meanwhile I'm banning this post so that it doesn't (a) give people horrible nightmares and (b) give distant superintelligences a motive to follow through on blackmail against people dumb enough to think about them in sufficient detail, though, thankfully, I doubt anyone dumb enough to do this knows the sufficient detail. (I'm not sure I know the sufficient detail.)
You have to be really clever to come up with a genuinely dangerous thought. I am disheartened that people can be clever enough to do that and not clever enough to do the obvious thing and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much more important to sound intelligent when talking to your friends. This post was STUPID.
(For those who have no idea why I'm using capital letters for something that just sounds like a random crazy idea, and worry that it means I'm as crazy as Roko, the gist of it was that he just did something that potentially gives superintelligences an increased motive to do extremely evil things in an attempt to blackmail us. It is the sort of thing you want to be EXTREMELY CONSERVATIVE about NOT DOING.)"
The current year is 2026. I think it's quite likely AI will make the top 3 by then. Putting this on my calendar.
Yes, I can't predict the future, but if it were a bet on Kalshi I'd be willing to put my money on the AI won't make the list side.
This seems out of line with what you were saying above. Do you mean you would put money on a position less favorable than 50/50? What do you consider the likelihood such that you think you can announce it to be false today? 99%? I'd happily offer 30 to 1 odds on this today and add a 3x buffer for you up to $1000 of my dollars if you're serious.
I'm confused. In the original comment I pointed out that the article claims AI is a major topic when polls show that isn't the case. I still believe that in 2028 AI will not be a major topic as shown by polls and voting behavior.
I'm not sure how I'd formulate the bet, but maybe it could be "as of election day 2028, AI will not be a top five concern mentioned in the most recent open-ended poll by Gallup polling the American public. This bet will not be decided by a poll in which AI is specifically mentioned in the polling question."
> falsifiable claims that are, well, false
Oh, I didn't realize it was 2028 already!
AI is not yet a top 5 concern for the American public, but polls show that it is for DC staffers: https://x.com/TimSchnabel/status/2072725548405518574
This prediction can't be scored until the 2028 election cycle.
You may think it's very unlikely the prediction will have turned out to be correct by the 2028 election cycle, but that is not the same thing as the prediction being scorable as false today.
Or you deeply suffer from normalcy bias. Not believing that AGI is possible is irrational and unscientific imo, the human brain exists, it is not made of magic, it can reproduced. It is as simple as that. We can debate about timelines, architectures that may lead us there, etc... You can even talk all day long about definitions of AGI, and people waste their time doing that. But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point.
There is a vast gulf between theoretically possible and technologically feasible.
If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction.
You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone.
> the human brain exists, it is not made of magic, it can reproduced
Yeah. It only takes 9 months and ~18 years of training…
> But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got.
Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.
What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966.
Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.
It's a form of denial. We're getting another "de-thronement of man" on the order of Copernicus and Darwin. Some get excited, others turn away in horror. Negation is the outward expression of the desire to keep human intelligence wrapped in its mystical veil.
One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts. Not only is that untrue, it's also directly contradicted by our experience with previous RL-systems, where they seem to breeze right by human ability without even the slightest hiccup.
> with previous RL-systems
Most human systems are much, much, much more complicated than most closed world games (which is where RL approaches have seen massive success, mostly through self-play).
Like LLMs are great, but I honestly can't see us getting actual general intelligence out of them.
They already possess general intelligence by many metrics. Sure they miss a few, but that's nitpicky and goalpost shifty - lots of humans make errors of all sorts as well, or incur brain damage limiting them in one or a few areas of intelligence - we don't then say they are not general intelligence anymore.
I think maybe you mean superintelligence, which is a more fair critique.
> They already possess general intelligence by many metrics.
Can you share the metrics you are using for this assessment?
They are really powerful tools, but a quick glance at their thinking tokens (which is a bad name, tbh) rapidly disabuses me of the notion that they are general intelligences.
They possess large amounts of crystallised intelligence (i.e. they have a lot of knowledge), but their fluid intelligence is definitely lower than the human median.
<< One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts.
Can you elaborate on this? I am clearly not aware of this line of thinking and the related contradiction.
Approximate as a limit and not surpass, where the hidden hope actually seems to be that they don't surpass. There are lots of variations floating around, from simple metaphors like "LLM as librarian that speaks to you," to even Sutton's remark that rather than being a case where we've taken the "bitter lesson" to heart, LLMs may be a yet another case where we're limited by it.
Unpack this a little bit. Why is it strange or interesting to you? What specific priors need to be updated for us here? What is the philosophical questions at play for you?
To meta-unpack a little bit ... it is strange to me that Fable is far more capable of discussing these questions than apparently 99% of humans. Along with being more capable at quite a lot else than most humans.
It doesn't seem at all strange to me that a chatbot trained by true believers in an AI singularity and the importance of safety guardrails will give more satisfying answers to true believers in an AI singularity and the importance of safety guardrails than talking to humans who might ask questions they're not prepared to answer (or might say nasty sceptical things or just not seem interested)
As for "updating priors", that goes both ways. There's plenty more reason to think "hey, transformers and RLHF might actually make some killer products" but certainly no reason to think the few people who didn't realise that "GPT3 is too dangerous to release" and "all software engineers will be replaced within 6-12 months" were marketing rather than prophecy have some kind of special insight into how it's all going to pan out. Clock's ticking to the promised 2027 reckoning too...
OK. Anyhow ... if there's a cognitive task you are personally superior to Fable at, let us know.
Can't do that, I'm still in hiding from GPT-3 trying to kill us all.
Planning
OK.. when you discuss these things with your Fable, what topics come up? Can you articulate one of the questions? I am probably just another dumb human FYI, but just try it out and we will see if I can follow along.
Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs. In the chase for AGI specifically, LLMs are a dead end, just like all the other AI technologies that died in the AI winter.
What priors should be updated?
>Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs.
Yet it is generating billions in revenue which Eliza did not.
Perhaps all we need is scale and some refinement techniques to eat a big fraction of the economy.
If unimpressive inputs lead to impressive outputs, that should make you more worried, not less.
The commercial viability is orthogonal to whether it achieves AGI, which is effectively what "reproducing the human brain" amounts to in this discussion.
What's your definition of AGI?
And possibly trilions in running costs, not mentioning all the shady training data sources.
Human brain looks a lot like scale and minor refinement above dog brain. Or mouse brain.
Strong disagree. Fable is first model that actually feels smarter than me in certain non-trivial ways.
It can hold many complex and partially contradictory thoughts in its head at once, in a way that feels significantly superior to Opus (for example). And then can make reasonable syntheses across these.
In a couple rounds of back and forth, with relatively low effort (but strategic) prompting, it produces complex, accurate analyses in 5-10 minutes that would take me multiple hours of hard, very focused work.
I still need to remain tightly in the loop, providing frequent course correction, clarification, high level reframing, nudging, and grounding.
It incorporates my feedback incredibly well.
It’s honestly staggering. Fable has changed my assessment of the current trajectory more than any model since possibly gpt-4. Opus 4.5 of last year might be a close second.
———
My advice for anyone who wants to get more value out of these tools:
When a model does something idiotic, don’t throw your hands up in the air. Be curious. Try to turn it into a puzzle to be solved.
It know it’s hard sometimes, especially if you are drowning in slop from other people… or generated by yourself, heh.
It can be exhausting. I struggle with this also. I have thoughts on how to make it better. We shall see.
I just realized that you might really be onto something. I wonder now if it is just a function of our very human inability to let go of a known construct that has served us well until now or something else. I have my own opinions, but as strange as it sounds, this may be the HN equivalent of ok boomer moment.
I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion.
Now write down a blueprint for superintelligence.
So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet.
In my view it would be insane to believe we can build something that we can't even reliably imagine yet.
As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system.
However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance.
You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.
Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function.
I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised.
That said, I think it's pretty clear that LLMs are not going to get us there.
I don’t think people are arguing to stop researching AGI. Moreso against sales people trying to use the concept of AGI to sell products that are very much not AGI. Or devoting so many of our resources into such a pursuit that it causes harm to real people.
This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.
The parent poster argument boils down to "[something] is theoretically possible, therefore 1) it is guaranteed to practically implementable 2) in the reasonably near future". Both are simply prima facie false; one can ask an LLM to explain why if there's any doubt.
And now we’re desperately trying to ”upgrade” penicillin (and friends) because it doesn’t work any more in many cases. Do you think we can repeat the process or do we need something completely different?
This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.
To quote the great Dr. Malcolm:
> Life, uh, finds a way.
No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs.
Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?
<< No one imagined LLMs in their current format
That is simply not accurate. There are examples of scifi novels, novellas and other media that dealt with it. We can argue over whether it was that exact format, implementation and so on, but that 'shape' ( to use a common llm term ) of technological advances was very much explored.
I was referring more to the fact that no one predicted that next token predicting Transformers would go so far. Not about "AI" in general.
> Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created.
Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI?
The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.
> actually intelligent
It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?
I can't give you or your sibling a better answer than "you'll know it when you see it". Some people see it now. I think they're wrong, because it seems like the results you're describing are easily explained by fuzzy search in the space of embeddings and then forming strings of plausible tokens related to the resulting region of embeddings space. In other words, the things we know LLMs actually do.
That's more or less looking for interesting patterns in a jpeg or another lossy compression result. It's interesting that the models seem to be able to (fairly) reliably return relevant chunks of the image. Even more interestingly, they seem to be able to invent plausible chunks of image that aren't even there. That doesn't meet my bar for intelligence though. I'd need to see it learn and adapt. I'd need to see it be clever, not merely "knowledgeable". I'd need to see it capably analyze itself. I'd need to see it reasonably estimate uncertainty and know itself in the sense that it has some idea how right or wrong it is about something. I'd need to see it exercise judgment.
I don't think I'd give a different answer a decade ago but who knows.
[edit] For all we know, one of the salient features of intelligence is that intelligent beings are incapable of precisely defining it. I'm not sure how productive it is to attempt to do so.
I appreciate the straightforwardness, but you probably understand that's pretty unsatisfying.
Actually, stronger - it's valid in some circumstances to say something is infeasible to precisely to define and you'll just know it when you see it. But I don't think it's reasonable to take that stance and then assert that "anyone sound of mind who knows how an LLM works" must agree with what you see. You gotta pick between striving for rigor and denying your opponents' soundness of mind.
What is your definition of "actually intelligent"? I believe LLM's are more intelligent than the average human in a lot of ways according to the Legg/Hutter definition of intelligence: "Intelligence measures an agent's ability to achieve goals in a wide range of environments".
No one knows how LLMs work. We know how the architecture works, but almost nothing about why. Saying "statistical next token prediction" tells you about as much about LLMs as saying "action potential thresholds" tells you about the brain. A true fact that explains very little.
And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse, if you think you have to be delusional to question whether LLMs are "intelligent", whatever you define that word to mean. The [latest publication](https://www.anthropic.com/research/global-workspace) from anthropics interpretability team purports that they see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain. Are you going to dismiss the whole team as not being of "sound mind"?
Intelligence is a word with a somewhat unclear meaning to begin with, but you have move the goalposts pretty damn far to exclude LLMs at this point. They are certainly still lacking in some regards, but whether that disqualifies them for intelligence is very much a matter of debate.
"And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse,"
I see no citations referring to the current philsophical discourse, unless you mean to imply anthropic's paid people are to be considered to be part of that.
That they "see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain" is if anything discrediting.
+1 I can't imagine how any corporate entity could be credible in this financial environment. Nothing they say can be reliably considered as anything but marketing copy. This situation is exactly what academic publishing is for. Although that institution has also been degrading.
Sure, here are some citations:
https://arxiv.org/abs/2401.03910?utm_source=chatgpt.com https://www.frontiersin.org/journals/psychology/articles/10.... https://arxiv.org/pdf/2408.04666 https://arxiv.org/abs/2402.00901 https://ar5iv.labs.arxiv.org/html/2407.11015 https://ar5iv.labs.arxiv.org/html/2202.05262 https://www.sciencedirect.com/science/article/abs/pii/S13646...
My point was not to argue one point or another about LLM intelligence, but to push against the notion that you have to be "delusional" to even argue that it is possible that LLMs can qualify as intelligent (although the op prefaced it with "actually"). That's mainly what irked me about the original comment, the arrogance of dismissing everyone even having the discussion as insane, as if there's no legitimate argument to be made.
And this was mainly the point I was arguing. However since we're on the topic, I also happen to think it's intellectually lazy to dismiss the Anthropic interpretability teams work as "delusional" simply because they have a conflict of interest. Of course, that is not an irrelevant fact, but much of their original work has since been replicated by independent entites (eg. https://arxiv.org/abs/2510.01246). Until they publish something that turns out to be fraudulent, I think it's reasonable to consider Anthropic's paid people a very relevant, and in fact excellent part of interpretability discourse.
Dismissing all opinions where there is a perceived conflict of interest is a pleasant cognitive bias to have, but reality is often more nuanced than that.
I guess you don't really know how an LLM works then..?
In this very thread I am being told that Fable is nothing but a bit of scale and refinement on well-known neural network techniques. And next I am told that we can't even imagine how to build superintelligence. Which is it folks?
Indeed, the real fun signal here is the conflict between how intelligent humans think they are and the different story told by much of human behavior, including within this very thread!
When an LLM makes a cognitive mistake, humans jeer "dumb stochastic parrot!" - when a human does the same, many are blind to it. Like a weird auto-Gell-Mann Amnesia - relative to reflecting on our own perceived cognitive "strengths" versus others weaknesses.
Jesus... This morning while I was drinking coffee and staring at the screen (it's Saturday) an agent did the equivalent of days of my work, reading code, understanding, hypothesizing, comparing, using tools, writing scripts, launching compilers and running tests, identifying problems and proposing solutions, and more. Only someone who hasn't spent a second reflecting about what it means to think and to be intelligent can claim that we miss a realistic path to intelligence. It's so damn clueless and stubborn and confidently wrong that it annoys me immensely, so sorry for the rant.
> clueless and stubborn and confidently wrong
Uhuh. I really shouldn’t be replying to this type of comment from a throwaway.
But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise?
Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us.
So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.
> reading code, understanding, hypothesizing, comparing, ...identifying problems and proposing solutions, and more
Except it did none of those things, really, because that's not how it works. This might help, it's a good writeup: https://www.0xkato.xyz/how-llms-actually-work/
We know how these machines work, it's not mysterious, there's nothing "extra" happening.
Now show me a writeup that explains how the brain works so I can understand why the brain does those things.
> We know how these machines work, it's not mysterious, there's nothing "extra" happening.
It sounds like you're saying "We don't know how brains work, they're mysterious, there's something 'extra' happening", and using that as justification for why you're saying a computer, an AI, can't "understand".
I think most people on Hackernews now who would use the phrase "my AI worked overnight and hypothesized, compared, etc..." already know how an LLM works, and still chooses to use those words. So the issue isn't that they don't understand. It's that they understand and still use those words. So the disagreement is somewhere else.
I'm not claiming there's something "extra" happening in brains. Merely that we just don't know how they work well enough to use that knowledge to do engineering. Neural nets are quite unlike brains, despite the unfortunate shared vocabulary.
OTOH we do know how neural nets work, and they definitely don't do "thinking" or "reasoning".
First you have to give a specific definition for thinking and reasoning before determining if they definitely do or don't do such a thing.
Fair enough, I should have phrased that less strongly. Until you show that your neural net does "thinking" or "reasoning" I'll disregard that and prefer to think about it in terms of what we actually know neural nets actually do. Does that work?
Just start with your definition for reasoning. If A implies B, and B implies C, does A imply C? Does that count as reasoning?
This feels like a semantic disagreement to me? If an LLM got to an acceptable end result code-wise, what would you call the process that took place to get it there?
I'd call it what it is: a good enough stochastic search result extracted from the model's embedding space.
Is an implication of this that models are incapable of producing entirely novel code?
Also, not to get too reductionist about this, but what do you posit is special about what is happening when humans think? Intelligence is hard to define so clearly, I reckon.
> Is an implication of this that models are incapable of producing entirely novel code?
No, it does not imply that at all. Google "temperature in LLMs".
> what do you posit is special about what is happening when humans think?
I don't. And IIUC nobody knows, but I'm not a brain scientist. There have been some wild theories over the years (recall Penrose's). I don't really have a dog in the hunt, except that probably whatever is happening is physical. It doesn't really matter, except insofar as whatever is happening very probably isn't what LLMs are doing. We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
> Google "temperature in LLMs".
No need to condescend, I'm very aware of what temperature is for LLMs. But I'm going to push back - if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense? (I'm not, for example, talking about novel rearrangement of existing ideas and code)
> We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
I don't think the claim is that LLMs do what brains do - I think the correct form of the counterargument is that _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
> if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense?
By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".
> _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
I don't think that has earned its therefore. Another perfectly reasonable explanation is that LLM's output is a close enough facsimile to intelligence that if you allow yourself you can easily be fooled into thinking its intelligent. That's not the same category of thing. It's not an incremental step away from intelligence. It's a whole different animal.
> By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".
This sounds to me like an admission that LLMs are not just doing a stochastic search, then.
> close enough facsimile to intelligence
What's the distinguishing criteria then? How can you tell the difference?
I think we must be talking past eachother. I define stochastic search as a search process with randomness injected into it that can return the following things:
- Something contained in the data set, not necessarily the same thing for every iteration of a given query
- Something not contained the data set (hallucination), not necessarily the same thing for every iteration of a given query
Does that clear it up?
> What's the distinguishing criteria then? How can you tell the difference?
All the ways they fail to exhibit intelligence. They can't learn. They can't adapt. They can't reason abstractly. They can't count. Etc...
Ah it seems you are a stochastic parrot believer (https://en.wikipedia.org/wiki/Stochastic_parrot). What are your responses to the Expert Rebuttals section?
I find the rebuttals pretty convincing - that there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.
> All the ways they fail to exhibit intelligence
Another implicit admission that there _are_ ways that LLMs exhibit intelligence?
> there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.
The next step then would be to design and conduct experiments that isolate this effect. Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining. Isolate it or distill it so it can be studied directly. Until then, it's easiest to dismiss it as imaginary.
> Figure out how to make it happen reliably
And you're happy that the replication of LLMs across many foundation model companies is insufficiently reliable?
> just something you're imagining
So the alternative explanation you're suggesting to emergent LLM behaviour is mass independently-corroborated human hallucination. Which is more likely?
Also it really does seem like you've moved the goalposts a lot here without really giving me a substantive response.
To say that LLMs' existence is evidence for emergent phenomena in LLMs is tautological. I'm merely suggesting if you want to make a claim about emergence it would be best, especially in absence of a convincing theory, to demonstrate it experimentally. Otherwise probably better not to claim it's actually happening.
> To say that LLMs' existence is evidence for emergent phenomena in LLMs is tautological.
This is not at all what I was saying. I think you've already conceded that LLMs demonstrate emergent behaviour but you dismissed it as a "close enough facsimile to intelligence". I was saying that the emergent behaviour is reliably replicable, in response to your following statement:
> Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining.
I think there is real work underway in the area of interpretability. In the meantime, there appears to be plenty of empirical evidence for the claim that LLMs exhibit some sort "intelligence" in the enormous penetration that agentic coding has achieved in software development? Do you deny the usefulness of LLMs here, or are you going to assert that actually software development requires no intelligence of any sort?
> I think you've already conceded that LLMs demonstrate emergent behaviour
No. Please don't put words in my mouth. What I said is that an LLM compresses a bunch of information into a semantic embedding space and then does sort of a stochastic search in that embedding space. Any similarity to "intelligence" is accidental. You may look at the results of that process and "see" thinking or reasoning or something, but it ain't there.
> "intelligence" ... agentic ... usefulness
I don't think LLMs need to be intelligent to be (at least narrowly) useful. No more than random forests or genetic algorithms do at least.
[edit] Look, this has devolved to the point where it's no longer productive to continue. If you're going to state things like this as fact, there's really nothing more I can do here:
> emergent behaviour is reliably replicable
Go collect your Nobel prize then! This is no longer a discussion grounded in reality.
> but it ain't there
On what grounds? I don't think you've provided any evidence other than LLMs can't "adapt" or "learn" to show that LLMs do not show intelligence in any way. I think it's clear that there must be some emergent form of intelligence over words from just the agentic coding ability alone. I am not claiming that LLMs are intelligent, only that they display aspects of what we understand as intelligence.
> I don't think LLMs need to be intelligent to be (at least narrowly) useful
I agree! But they are more than narrowly useful, and they absolutely do not belong in the same category as random forests or genetic algorithms!
> Go collect your Nobel prize then! This is no longer a discussion grounded in reality
Once again you are being condescending while misrepresenting my position. The emergent aspects of "intelligence" have been replicated by virtue of independent LLM vendors training their own models - I am not making a stronger claim, you have misunderstood me.
Thanks for participating.
@sudb, I salute your effort. It is unfortunate how little this person took from your attempts at a genuine exchange.
Without moving goal posts: @27183: the word intelligence is defined as "noun. The ability to acquire, understand, and use knowledge."
LLMs can absolutely do this via in-context learning. They can even persist this as mementos on disk and re-attend to it later if its been purged from their context window.
You might try to make some comment like "they're not learning in the way humans do, they're not changing their weights, and they predict token by token!" Who cares? That's semantics - LLM attention is able to accumulate contributions of tokens and weights mixed together head by head and layer by layer with non-linear activations which functionally quack like intelligence.
They may be far different from us, but they're functionally intelligent (at least in several subsets of intelligence. And frankly, in some cases, they're more functionally capable than many humans).
Birds, planes, helicopters, rockets and gliders don't achieve flight in the same way, or have remotely the same capabilities in the air - but all are nonetheless forms of flight.
Just a few comments ago you argued that we don't know how to build superintelligence. Now you're saying we know how the (unevenly superintelligent) Fable system works.
It doesn't seem like you're being consistent here. I'm concerned there might be some motivated cognition going on.
"What is true is already so. Owning up to it doesn't make it worse. Not being open about it doesn't make it go away. And because it's true, it is what is there to be interacted with. Anything untrue isn't there to be lived. People can stand what is true, for they are already enduring it."
I'm not following the "Fable" stuff, what is that? [edit: ah it's a new language model from Anthropic.. yeah still don't see how it's relevant here]
[dead]
> We know how these machines work, it's not mysterious, there's nothing "extra" happening.
Lol. This is more telling about your implicit unscientific preconceptions that you wanted to reveal. Of course there isn't anything "extra". Where do you think intelligence comes from, some mysterious realm? It's physical, computational. The fact that at the bottom we produced it via matrix multiplication is irrelevant. Maybe humbling. You are denying a visible fact (a machine performs tasks that require flexible analytical and cognitive skills) precisely because there is no magic happening anywhere.
> Where do you think intelligence comes from, some mysterious realm? It's physical, computational.
Well, no. I don't think it comes from some mysterious realm. I think that which is not physical does not exist [edit: and if you like I'll follow that one right down the rabbit hole--continuity and infinity are useful delusions]. But that eminently does not mean we know what intelligence is, let alone how to build one.
> The fact that at the bottom we produced it via matrix multiplication is irrelevant.
Huh? We don't even know what "it" is. How can you say you produced it?
> a machine performs tasks that require flexible analytical and cognitive skills
You see that, I see a lucky stochastic search result. Don't underestimate the "creativity" of random algorithms! They can do some wild shit! This is nothing new, we've been playing with these toys for like 70 fucking years. It's only recently that they started spewing words and everyone lost their minds over it.
You do realise then that "we know how it works, there is no "extra" there" is an argument that can be used against any artificial intelligence, now or in a thousand years, as well as (at some level) against human intelligence (no magic, it's all physics, just dumb cells exchanging signals). This should be enough to give you pause- you immediately reached for an argument that is entirely empty.
> I see a lucky stochastic search result
Again you're reaching for a mechanistic explanation of some kind (let's leave for the moment whether it makes sense or not) as if having an explanation somehow contradicted a display of intelligence. It doesn't. Yes of course we made it, we know how it works (ar some level) and there is no magic. But what matters is the result- this machine, matrix multiplier, stochastic parrot, consistently displays intelligence, to the point of being able to perform very complex, open-ended tasks that integrate discovery, planning, tool usage, decision and even some aesthetic sense, understanding and using natural language, context awareness, you name it.
> This is nothing new, we've been playing with these toys for like 70 fucking years
Lol no. For god's sake. Hundreds of billions of parameters organised in a specific architecture and trained with unimaginable amounts of data and compute? Unless by "these toys" you mean "any computer program vaguely AI-related".
> But what matters is the result- this machine, matrix multiplier, stochastic parrot, consistently displays intelligence, to the point of being able to perform very complex, open-ended tasks that integrate discovery, planning, tool usage, decision and even some aesthetic sense, understanding and using natural language, context awareness, you name it.
IDK, it doesn't seem like they actually do any of that. To me it seems like they have good enough semantic embeddings that they can kind of approximate those things, sometimes, well enough if you don't look too hard. This is enough to fool people. Of course there's gold in them hills--some recent mathematical results were found there. But to say that's "intellgence" is to say that lossy compression is intelligence. It's static. It does not learn. It does not adapt.
> Unless by "these toys" you mean "any computer program vaguely AI-related".
Not "vaguely AI related". I mean stochastic computer programs that can do things that look awful thinky. They've existed for a long time, but only recently (due to word2vec and other advances) have the results been words that mostly go together well instead of numbers. For some reason people seem to think a lot less critically when the output is words. IDGI but it's a whole thing.
It’s also theoretically possible to travel almost at the speed of light. Doesn’t mean it’s rational to talk about it today as if imminent.
How can not believing something that hasn't been proven be unscientific? Do you know that words mean things?
Science means the pursuit of knowledge. It doesn't mean "only believing proven things". If we're going to be rude, lets at least take the time to be right.
[dead]
You dont seem to understand what proof means. Human brain is made of matter, matter can be arranged to make a thing that reproduces human brain properties. What's the confusion here? I say its unscientific because it places the human brain beyond the scope of what can be operated on. Not having the knowledge or tech yet to achieve that is irrelevant since we have an existence proof.
Even if we accept your premise, and not everyone does, the confusion is whether we're capable of creating an equivalent arrangement, even in principle, using alternative materials.
That's not "irrelevant", it's fundamental.
> Not believing that AGI is possible
One can simultaneously believe AGI is possible, be only modestly sceptical that our current methods are likely to yield it in the near term and still find the religious ferocity enveloping its discussion silly.
> saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Straw man. Nobody argued this. The discussion is around how urgent it is to policy treat a future hypothetical.
Nobody argued anything. GP just dismissed it as religion without engaging with a word of the material. Parent is taking a stab at why.
Not that I'm complaining. Cynicism is the failure mode I rely on HN for. It's the populism that's been getting to me.
> GP just dismissed it as religion without engaging with a word of the material
Fair enough. I didn’t see anything novel in the article. So treating it as a motif within the abovequoted “Superintelligence: The Idea That Eats Smart People” context is fair and a real argument.
> Cynicism
Cynicism isn’t the opposite of blind optimism. Nihilism is. I’m not seeing a rejection of the article as being baseless as cynical or nihilist. It’s just pointing out a cultural thread that doesn’t seem to be useful.
I hate to get bogged down in semantics, but with the hope that one of the stronger top-level critiques makes it into the top slot here and this conversation gets buried:
Cynicism is defined as
>An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others.
I'm not saying that cynicism is automatically wrong, just that I once could trust that, when HN is wrong, it is due to cynicism applied in excess.
>..Nothing in life ... has any chance of succeeding without strenuous practice; and this is capable of overcoming anything..
---Diogenes (L) on Diogenes, arch-Cynic, VI.2.71
Afaik not formalised in Lean4, but sounds like optimism
Most of D's other stuff imho is optimism taken to its clear-eyed extreme. ya know. the way a very best version of us could always unironically RELY on HN [,after much practice]
>so those whose training has been of the opposite kind derive more pleasure from despising pleasure than from the pleasures themselves.
Might be why professional Nihilists admire Cynics (ie, for their ... unaffected comedy)
https://penelope.uchicago.edu/Thayer/E/Roman/Texts/Diogenes_...
Ah, this clarifies things. I suppose I brought this on myself by putting it at the start of the sentence, but I do mean little-c cynicism, not the ancient philosophy. I was surprised to be called out on it, as GP is a shining example, and I was confused by the mention of nihilism, but I see now where wires were crossed. Thanks!
I put it in the same bucket as living on Mars. Can it be done? Probably. Are we close? Not as close as people seem to want to believe. Is it a goal that will largely benefit society in its current form? Absolutely not.
Eh, with Martian habitation we know what the roadmap looks like. With AGI we don’t. It could be proximate. Or it might not be. When it arrives, it could be totally economically uncompetitive outside the rich world. Or it could replace all human labour. Or progress to become a superintelligence.
We don’t know. Which makes proposing rules around it based on fiction more than science silly.
In particular, I just don’t buy into the “left behind unless” framework.
Perhaps Anthropic will create God in the Machine. Not foreclosing on that. But will it matter so much who was fucking around with Opus five e-folding times ago?
Either ClauDeus is benevolent and lifts you up (not left behind) or it isn’t, or not to you, and you are culled by a drone (left behind regardless).
Serenity Prayer time.
> Not believing that AGI is possible is irrational and unscientific imo
This is an objectively wrong opinion.
I remember thinking exactly the same thing around 5-7 years ago, in the GPT-2/GPT-3 era. "Oh sure they can produce semi-coherent output, but truly intelligent behavior is still far away. This isn't science fiction, they're just falling prey to Pascal's Mugging same as my religious friends did." Now I'm not so sure. I give the AI safety subculture as a whole a lot of credit for putting it on my radar back when it was otherwise still science fiction. I don't know if they're right about what comes next, but I think their case deserves to be evaluated on its merits, rather than assumed to be the result of psychological flaws.
> Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
Confused at who this is directed towards. I'm fairly certain that the article was written by people who (at some point) identified as effective altruists, most of whom would enthusiastically agree with this. This community didn't start as AI researchers and later choose effective altruism; they were effective altruists who chose AI safety research as the most effective way to improve the world. Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?
>Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?
Their methods are about convincing others that things that enrich and empower themselves at the expense of others is "improving the world". This isn't the stance of serious people who want to improve the world.
Please don't confuse AI researchers and AI safety researchers (although there is some overlap.) That's like confusing the people from the Manhattan project with the people who protested 3 Mile Island, because they're both focused on nuclear technology. One effect of EA is that every AI researcher says they're trying to make the world better. Some of them are full of shit, but not all.
I'm fairly certain the authors would be happy to see AI shut down indefinitely. They just don't believe that the coordination problem is solvable. This is their best attempt to come up with something workable in the real world, or at least get people started thinking about it.
Right, the so-called "AI safety researchers" are worthless clowns and grifters with nothing of value to contribute. We would all be better off if they spent their time on something productive like writing entertaining sci-fi stories. They certainly seem to have the rich imagination necessary for that occupation.
Now this is an unserious comment, just throwing insults without explaining your reasoning at all. Do you believe that AI itself is worthless? Or that it's just incapable of causing harm? Do you resent the safeguards placed on AI because they are personally inconvenient to you? When introducing a new technology at this scale, don't you think it's worth looking before leaping?
So called "AI safety researchers" are themselves unserious so of course I don't take them seriously. LLMs are extremely useful, I use them constantly for a wide variety of tasks. LLMs don't cause harm: humans cause harm. There's no real need to intentionally cripple or restrict the models in the name of "safety"; that's just stupid and done only to prevent corporate embarrassment.
Sadly they're very bad writers. Have you actually read the stuff they produced? Horrible. Yudkosky himself wrote https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...
It's so bad
I'm confused, the authors of the website are arguing that the development of AI should be slowed down so that society can adapt. What do you think they are trying to convince others of?
Which one of the authors of the website are you referring to?
> they were effective altruists who chose AI safety research as the most effective way to improve the world
This actually made me laugh, I’m sorry
Effective altruism is also very attractive to manipulative sociopaths who want to maximise their power over others whilst appearing virtuous and hoarding wealth and power. Poster boy for this movement is the convicted fraudster SBF. I believe Altman is also a fan.
As to a better world or super intelligence, I’ll believe it may be possible when I see some signs of intelligence from what people are calling AI, instead of plausible text and image generation based on a very large corpus.
>I believe Altman is also a fan.
Altman says he thinks it is an "incredibly flawed movement"
https://x.com/sama/status/1593046526284410880
The dislike is mutual. Here's a long video takedown of Sam from a major EA org:
https://www.youtube.com/watch?v=_eYTkvZqbnQ
And what movement isn’t attractive to manipulative sociopaths who want to maximize their power? That’s unavoidable when dealing with humans.
An important adjacent point is the same people that are insisting that only their research can stop intelligent manipulative computers from controlling the human race are also some of the few people who believed that OpenAI was a philanthropic endeavour and that Sam Bankman Fried was trustworthy...
I'm not seeing any claims to exclusivity. I'm seeing people actively encouraging more people to enter the field of AI safety/security and openness to robust debate, etc.
I agree that there seems to be a huge problem with value drift--either people who used to research AI safety, pivoting to building AI (looking at Anthropic) or people who only ever paid lip service to safety (I tend to put Musk and Altman in this category.) These people need to be held accountable, but it doesn't mean every AI safety researcher ever was a stooge or a fraud.
Ok, fair enough theyre not entirely opposed to outsiders setting up research institutes like theirs and enjoy debate (or the optics of debate) more than most, and I don't think most of them are insincere. But the point wasn't that they were exclusionary, but more that whilst priding themselves on their supposedly superior ability to predict the future and "align" AIs, they somehow missed the value drift and human non-alignment with their values basically any outsider saw coming...
OpenAI now controls one of if not the largest philanthropic endowments on the planet. They are still a philanthropic endeavor, though I agree the non profit bait and switch as well as the Altman board situation were tragedies.
I mean, they're also gearing up for one of the largest IPOs in history to generate unprecedented amounts of wealth for themselves, and the nonprofit foundation's focus is community engagement with OpenAI products. Goldman Sachs has a large philanthropic endowment but I wouldn't confuse that with their priorities being philanthropy.
The point was I think pretty much everyone else saw the bait and switch coming...
I remember thinking exactly the same thing around 5-7 years ago, in the GPT-2/GPT-3 era.
ChatGPT was announced three and a half years ago, 30-11-2022.
Do you think that ChatGPT is when people on hacker news first became aware of the GPT series of models?
https://hn.algolia.com/?dateEnd=1594339200&dateRange=custom&...
And GPT-3 was released 6 years ago in 2020... What's your point?
I would rather see you engage with the substance of the article rather than skipping right to insulting the authors. I don't think this sort of comment is up to the standard I have come to expect from HN.
> I intend to donate (at least) 20% of my lifetime income to effective charities. I publish my donations on my Donations page.
From your bio I suspect you're already in the cult.
When your best counter-argument is “you donate money to charity” you have to start wondering if you’re on the bad side and not the good side.
"I see your objection to ad hominem arguments, and raise you a further ad hominem argument"
His whole blog is exactly the same kind of nonsense as the article and reeks of effective altruism.
That's not an ad hominem.
Attacking a person instead of the substance of what they're saying is ad hominem.
Imagine thinking someone donating money is evidence of something bad
If the cause they donate to is bad, it is
What is bad about "effective charities"? It's probably mostly for malaria nets!
> Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective"
Who is this supposed to be arguing with? It sort of reads like it's trying to disparage "effective altruism", but I'm not sure.
Setting aside any of the AI stuff, I've started to find it pretty grating when people seem to imply that transferring millions of dollars from wealthy people in California and the UK to impoverished Kenyans and Rwandans, or buying malaria bednets which can save a child's life for the cost of a fancy new gaming rig, is "self-serving" or something because weirdos are doing it, while true caring for other people involves [unspecified thing that doesn't appear to ask any material sacrifice comparable to donating a large percentage of income].
> I grew up in evangelical christianity, and to them the end of the world is just around the corner
I grew up in evangelical Christianity, I'm still an evangelical Christian, and you're wrong to paint with a broad brush like that. Are there evangelicals that are millenarian? Yep, of course.
But that is not my experience across multiple evangelical churches across decades.
To interpret charitably, I guess we could solve this for religion = technological development so advanced as to be indistinguishable from magic. It's been done in some Star Trek episodes I think.
It has a lot of the properties of magic. You can put on quite a magic show with a trillion dollars.
The end of the world is always just out of reach. It’s the perfect carrot on a stick for people who are conditioned to be afraid.
Imagine observing powered flight for the first time and saying: "This is religious fervor folks. I remember the mythological story of Icarus from when I was younger. Key word, mythological."
The existence of mythology describing Scenario X is not a valid argument against the plausibility of Scenario X.
If we can acknowledge the possibility of nuclear doomsday without running an RCT of sample size 100 Earths, 50 of which undergo nuclear Armageddon, to verify that nuclear holocaust indeed a real phenomenon... then we can do the same for AI. Understand the arguments being made instead of engaging in these guilt-by-association arguments.
Modern AI capabilities are already mind-boggling by the standards of 20 years ago. We should at least prepare for the possibility that trends continue on the current trajectory.
From TFA
> Plan A is primarily a recommendation, not a prediction.
That sounds nothing like “religious fervor”
The fact that technology can increase existential risk for civilization is not fantasy. It’s a risk that should be reasonably discussed.
>Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
You can change more people’s lives, more substantially, if you donate effectively. Effective altruism started out as (and the majority of effective altruist financing is committed to) an effort to rationalize what has historically been a very emotionally driven activity by deploying insights from developmental economics. If you want to take longtermists to task, go right ahead, but please refrain from torching anti-malarial or child vaccination programs while doing so.
But in the public image, the EA community is synonymous with doubling down on AI / AGI to the exclusion of the other projects.
OpenPhil changing its name to Coefficient Giving, 80000 hours and bluedot and (to a lesser extent) CFAR dropping other initiatives and switching to AGI promotion… to my knowledge GiveWell is the only other big name that continues to advance other initiatives. Then look at figureheads like SBF committing fraud and begging for a pardon from the architects of the USAID shutdown… We begin to paint a picture of a community that’s (by and large) abandoned its principles for power.
I know the view from the inside is more nuanced, but I think it’s a reasonable association for random members of the public to make.
My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.
>My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.
I’d be curious to hear you expand on this. What binds the EA community together, from the shrimp welfare enthusiasts and wild animal initiative, to the longtermist lightcone obsessive, to the people funding vitamin A supplementation, is simply a commitment to maximizing the number of quality adjusted life years saved each year and a belief that empirical observation can be used to improve that number.
To my mind, this is a valuable insight on its own. Yes, if you come to such a heuristic with absurd prior beliefs, such as whether 100k neurons alone have QALYs in the first place or by placing equal value on people actually alive today and hypothetical people in the far flung future, you will get absurd results. Garbage in, garbage out. But that’s not an indictment of the fundamental insight, especially when you consider how poorly allocated the roughly $2 trillion in global charitable spending is.
As a society we already know how to make the lives of everyone better:
1. Stable housing
2. Access to safe drinking water
3. Access to food
4. Access to healthcare
5. Access to education
6. Stable governments
The EA community has so many "ideas" about what would help, when all they need to do is focus on those six and the world would be as close to a utopia as you or I could hope to see in our lifetimes.
I legitimately thought you were kidding about the shrimp welfare initiative, but after looking it up I was more infuriated about EA than I normally am when it comes up. I can think of several causes which would be better served with 3 million USD, and all of them take care of human beings. Living, breathing, intelligent human beings. These people should not be in any position of power or taken seriously ever.
There are tradeoffs to make between those six items. Money spent on stable housing is money not spent on safe drinking water is money not spent on education. There is a limited amount of money spent on charitable giving every year, and we obviously cannot afford to provide all of those things to everyone with the funds currently available. If you’re arguing that people should simply be donating more money, I (and I think every EA) is right there with you. But given the world exists as it is, I would rather see money that’s currently being donated diverted from Catholic missionary schools and good governance NGOs to GiveWell’s top charities because I’m very confident that it would mean fewer dead kids.
I agree with your last sentence, but we can’t partition the budget up so cleanly though. Societal spending is interdependent and additive across domains. Spending on cheap access to safe water or food reduces healthcare costs. Spending heavily on education tends to strengthen government in the long run, and better-run governments amplify efforts to improve housing/food/water/.
The one exception is AI. Historically, redirecting charitable funds towards AI safety tends to starve or undo the rest of these efforts, which is why I’m so disappointed by many EA institutions dropping other initiatives to put their eggs in the AGI basket.
AI 2027 made substantive predictions about the near term future of the technology and the implications of it. I think the worst you can say about it is that the Agent-1 moment might come next year rather than this year. But being off by a year is far better than this 2040 slop, which is mostly disconnected from reality.
The one thing I will say that they are correct about is that AI does have the potential to be highly destabilizing geopolitically, even if they get everything downstream of that wrong.
When I learned that this website is by the same people who gave us AI 2027, I immediately thought about the Wikipedia page on doomsday predictions.
https://en.wikipedia.org/wiki/List_of_dates_predicted_for_ap...
Notice the many times when a prediction failed and the so called prophet would come back in a few years and give you a new date. And they would tell you, well, this time it's real.
I do find it ironic that many of the AI predictions are coming from the self titled "rationalists." It seems like building your identity around being rational and immune to psychological pitfalls is a good way to ensure that you don't even notice that you have walked straight into the one psychological trap every cult has employed since time immemorial.
They gave a ~2 year timeframe for their predictions in this one (tied to the next presidential election), so in 2 years when none of this has happened will they then switch over to a new AI 2044 Plan? Where's the accountability? Where's the followup or retrospective? What if there was some mechanism that branded these people so that when they made claims in the future people could clearly see the authors labeled as "made completely wrong fantastical claims in the past"?
> When I learned that this website is by the same people who gave us AI 2027, I immediately thought about the Wikipedia page on doomsday predictions.
Then, you opened the page and read it and realized that this prediction was contingent? You know what a conditional is (I would assume) if this, than that?
Then you realized that the only reason you were posting this comment was as a sort of silly gotcha "Oh look at the guys who keep increasing the number" instead of talking about the differences between the scenarios?
Then what happened?
"Doomsday predictions have occurred since time immemorial. Ergo, the Cuban Missile Crisis is nothing to worry about."
You actually have to look at the substance of the prediction. Sorry.
Like all forms of eschatology it's not actually concerned with the future to come--it's real aim is to have influence now.
It's just a form of rhetoric.
To dismiss them, saying, "ah ha! they're making the same error as all failed prophets!" misses the point.
What they are really talking about is AI governance.
... and then there are the actual planet-threatening astronomical events that humanity should think about ways to mitigate but I'm worried that human lifespans and capitalism prevent us from working towards mitigations. Everyone will just say "meh, 1 million years is a long time, I won't be around" and a million years will go by.
https://en.wikipedia.org/wiki/Timeline_of_the_far_future
Capitalism, in particular NASA's commercial space development program, have already provided and demonstrated initial asteroid redirect capability via the DART mission launched by SpaceX, which impacted an asteroid and measured the changes to its trajectory.
People who've gone full true believer in "AGI" remind me a little bit of like, a random person from the most boring small town in the midwest who joined the Rajneeshee/Osho cult, or Hare Krishnas or something.
But instead of weird religious deities and practices, they've wrapped up their true believer zealotry in some kind of mishmash of "AGI is coming real soon now" like some kind of manifest destiny.
You could probably put people in FMRI machines and ask them to give a 30 minute lecture on the topic of AI and find that the same parts of the brain are activated.
https://www.cnn.com/2016/11/29/health/religious-brain-mormon...
[flagged]
[dead]
I completely agree. But not entirely sure that triggering the EA crowd is the effective way to deliver this message.
They are the crowd who need to understand your framing the most, but they’re completely shutdown by your framing, as any religious follower would be.
Your message gives me hope that not everyone’s drunk on the kool aid.
I would not like to be dismissive, but to me this article feels like an exercise in creative writing rather than a report to be taken seriously. The entire experience feels like a choose your own adventure game, seems like their stylistic intent.
I am not sure if alternative reality fiction is the best way to approach real and serious AI risks.
I am also not sure, with the amount of emdashes and the style of prose, that the entire article was not AI generated.
AI is going to be a mature scientific field. There are going to be efficiency improvements in training and inference. New paradigms are going to emerge with better multimodality, real time streaming and real time interfaces. Models are going to converge on the limits of our data available for pre and post training, improvements will be incremental and spiky in domains.
I am not sure who the AI 2040 article is for. I suspect it is intended to be a digestible piece of media for the financial class.
AI is going to be a useful technology and its impacts across the economy and global will be broadly distributed. Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones. Maybe the argument is that in verifiable domains, such as model training, AI models can supercede humans. I don't think so. A human's high level thinking, our incredibly more efficient semantic/neural compression, our ability to switch tasks and achieve the creative insight is not replicated through the current paradigm.
I love to model and simulate. As the dead economy theory[0] (discussion [1]) was submitted here, I decided to simulate it. It was really hard to figure out a path "good for the humanity", in the sense of a balanced system, not a winner take all situation, etc.
I think this is the reason why you have the tendency to propose some freeze-all policies, full control or similar. If you want to find the equilibrium, you need to accept that it will be a controlled equilibrium, most likely on a saddle point, with underlying process changing all the time, requiring fast changes in regulations. Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
[0]: https://www.owenmcgrann.com/p/the-dead-economy-theory
[1]: https://news.ycombinator.com/item?id=48324712
One counterpoint is that the "labor as TAM" argument is far larger than it needs to be. Only a fraction of it needs to be captured to justify all the capex and make 5 new companies displace FAANG, and this does not have to translate to unemployment to succeed.
https://jodavaho.io/posts/ai-jobpocolypse.html
The difference in the unemployment vs efficient employment model is mostly user driven adoption vs company mandated adoption, or centaurs vs reverse centaurs.
https://pluralistic.net/2026/07/02/canonization/#operate-ite...
Thanks for the links, I had missed those. Also:
> Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
Without totally derailing the thread, this is also obviously why climate and biosphere collapse is not (and likely will continue not) to be addressed, e.g. Timothy Morton's Hyperobjects
Saddle point is a nice way to put it.
> to me this article feels like an exercise in creative writing rather
because it is. Previously: https://news.ycombinator.com/item?id=43571851 / https://ai-2027.com/
Hasn't the AI 2027 "creative writing exercise" held up not-so-bad thus far?
This incredibly critical part of "early 2026" doesn't seem to hold up very well:
Yes the AI labs are using LLM-assisted development, but I am not aware of LLM-assisted research into better deep learning algorithms. "50% faster algorithmic progress" is a strange thing to predict about neural networks: historically it seems like algorithmic progress is very infrequent, and tends to be disruptive across the field. Likewise with the underlying algorithms in Codex and Claude Code - they aren't that sophisticated in the first place, and vibe coding an implementation 50% fasfer doesn't count as AI-assisted algorithmic progress. Maybe they mean cost?And ironically they didn't predict the actual 2026 reality that Mythos sucks as an agent but commanded global attention as a cybersecurity tool.
Also this just seems childish and clearly hasn't really panned out:
China has the exact same "algorithms"! The "algorithms" are on arXiv and the specific architectures are typically public. What China lacks is compute, and the US labs had a big head start on training data + RLHF.The article hinges on AI automating R&D and finding something fundamentally more powerful and reliable than the current transformer LLMs. But that hasn't panned out at all. What has panned out is better scaffolding around running the LLMs in an iterative loop.
It actually seems to me AI 2027 holds up badly, unless your only takeaway is "AI gets better."
To me, this feels like a last ditch effort to revive the AGI narrative to reject the coming and current commoditisation of these models, contrary to all current evidence. https://artificialanalysis.ai/
How is commoditisation of models incompatible with AGI?
> How is commoditisation of models incompatible with AGI?
A recursively self-improving AI has strong first-mover effects. That isn’t fundamentally incompatible with commoditisation if there is literally only one path to super-intelligence and you can have AIs at different rings on that ladder co-existing. (Not technically commoditised at that point. There are still different rings. But close enough.)
But the existence of commoditised AI implies model selection isn’t a huge deal, which in turn implies the models are about the same, which strongly implies there is no recursive self-improvement. Depending on your definition, you may still have AGI. But you don’t have superintelligence.
> But the existence of commoditised AI implies model selection isn’t a huge deal, which in turn implies the models are about the same, which strongly implies there is no recursive self-improvement. Depending on your definition, you may still have AGI. But you don’t have superintelligence.
This is only true at a given AI capability level, no? e.g., if AI at the GLM-5.2 level is commoditized, all that suggests is that there's no recursive self-improvement easily possible at the capability level of GLM-5.2. (And with the harnesses for it that exist so far, etc etc.)
If I observe commoditization of a given tier of model capabilities at a given point in time, this seems to say little about what's possible with models six months later, or models that are undergoing proprietary deployments at that very moment inside the major labs, or even models that are notionally available for public use but have had recursive self-improvement adjacent capabilities intentionally nerfed (e.g., Fable).
(I might be misinterpreting your comment tbc - if you mean observing commoditization implies there is no existing, ambient superintelligence at the moment of that observation, then I don't disagree.)
> if you mean observing commoditization implies there is no existing, ambient superintelligence at the moment of that observation, then I don't disagree
This is a better way to put it, thank you. More precisely, I'd say commoditisation implies there is no existing self-improving AI on the market.
The moment someone gets exponential self-improvement, model fungibility breaks and the first mover wins. This is the Bostrom singularity the rationalists flip out about.
The "AGI narrative" is distinct from the existence of AGI.
Most of the discussion around AGI is highly speculative. I am not saying AGI could not exist, and it is a term that has historically been loosely defined. Decades of coming science and research will tell.
If we can solve 99% of the world's problems with current non-AGI models then nobody besides a select few will care about AGI
Most of the world's problems are fundamentally social, political, and religious. These cannot be solved with current non-AGI models. Probably not with AGI either.
Yes. Anyone who doesn't acknowledge the efficiency difference between pretraining vs RL and assume that since we've run out of data for the former, we have to do the latter, is not making a serious attempt at modelling the future:
https://www.tobyord.com/writing/inefficiency-of-reinforcemen...
This is similar to that other exponential, which happened with CPUs - we ran out of true geometric scaling in the mid 2000s, and everything else supporting Moore's Law has been cleverness that arrived in the nick of time, supported by a bit of marketing, and very optimizable benchmarks, far from guaranteed gains coming from making a single physical metric better.
>Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones.
I'm confused if this is satire, sarcasm, or genuine belief. If this was the case, then AI companies should absolutely remove the "it may make mistakes", because doing mistakes would imply that "the very best human knowledge and expertise" is what actually fails, and not the AI.
With that being said, I'll still urge people to visit a professional therapist for health problems and I generally still trust human knowledge workers for critical scenarios. I will reconsider your claim when chatGPT can effectively play Yu-Gi-Oh! (or at the very least respond with the correct rules appropriately), which is a significantly lower stakes scenario than betting your entire company on its aptitude.
My framing may have been confusing there. “distillation of the very best human knowledge of expertise”. Distillation is different from outright capability or reliability. It is not directly adjacent.
For anything health related all AI models show high levels of anchoring bias. I would not use it as a confidant, and be skeptical of claims. Even so, human doctors are also fallible and prone to cognitive bias.
I think the obfuscation is because human intelligence has been projected onto AI model capability. AI models only have a limited dimension of human intelligence, and in some axes orthogonal, and when I say distillation I refer to this.
> Because AI represents the distillation of the very best human knowledge and expertise.
You say it like it's a fact, but in reality everyone sees the phenomenon of AI slop.
P.S. Information search and retrieval if the best and most direct way to use LLMs.
> everyone sees the phenomenon of AI slop
Just purely organic YouTube Comments circa early '20s alone surely outslop any "AI" by a giant margin.
Everyone sees the markers, and it's a hot topic. There are maybe a thousand from-scratch trained models, and just few mainstream ones produce most of human-targeted content. In today's world, no surprise everyone knows the common patterns of those. That sloppy landscape is not just load-bearing em-dashes — it's a humble testament to their reinforcement learning.
Humans produce tons of texts, with all sorts of nonsense in it, without thinking it through. Our slop is just a lot more diverse. And mostly just spoken out loud.
> P.S. Information search and retrieval if the best and most direct way to use LLMs.
Yes, but not directly, if they don't know something they tend to hallucinate like mad, even today. YMMV, but in my experience they work best as actual "cheap" reasoning for building queries and checking out search engine results. Even if they misinterpret some result, more and more results will still steer it towards correct conclusions and it can point at some results that relate well enough to be useful.
An even more cynical take than me!
I agree with your last statement.
I'm sure some people will have issue with my phrasing but, honest question:
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
Human cloning and genetic editing isn't stalled because we agreed it's unethical. It's not pursued because at the current level of advancements, it's pretty useless. The things we can do there are niche. It's easy to ban something that's not very useful.
If we had a way to make gene edited humans a lot smarter, a lot stronger or live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
It's not useless. We know enough about DNA to be able to make better humans, but people get real squickly real fast when you talk about that. It has stalled because of that. If we go in and just check the DNA for Downs or Parkinson's, we can't have a conversation on aborting the foetus without religious beliefs coming into play. Designer children aren't a thing, despite the ability to edit DNA to do specific things. Hair and eye color are easy enough to go in and edit for. Humanity has decided to opt out of doing that for now.
Marginally better humans. And IVF paired with embryo selection is very competitive in that niche.
Now, we're getting better at predicting polygenic traits, and we're getting beefy multi-edit pipelines that might provide a meaningful advantage over embryo selection working in multiple animals. But as of yet, the advantage of genetic editing in humans over just doing aggressive IVF and dredging the embryos for desirable traits is minor.
Bit of a chicken and egg problem there. Can't advance the tech fast without actively using it, can't actively use the tech until it's advanced enough for the benefits to override ethical concerns. So it's getting there, but at a glacial pace.
> If we go in and just check the DNA for Downs or Parkinson's
You should consider reading the wikipedia page about Parkinson’s disease.
You should consider reading more deeply than Wikipedia to be properly informed on a subject. There are a handful of very rare mutations that confer a high risk of getting it that we can detect. They are very rare. This isn't the same thing as being able to give a risk score if the DNA doesn't have all the rare mutations. But the science is there for those unlucky rare cases to say there is a high chance that a specific coding of DNA will result in Parkinson's.
Yes, there are examples of where we have collective decided not to pursue a particular technology tree.
For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)
And there have been successful world-wide bans.
For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
> "For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)"
That example works against the argument since that policy was rendered moot when Commodore Perry arrived at Japan in 1853 with a squadron of American warships and demanded opening of trade and diplomatic relations at gunpoint.
I completely can see why we'd want to, for ethical reasons, ban germline editing, and I want to be clear that I agree doing so cannot be done ethically, but there is a part of me that is wistful for what could have been. Same with things like CRISPR but it's probably just fun to dream and the reality would be a nightmare.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
No, of course not. That would be an insane trust fall. Even relatively small advances in technology give a country world dominating power. Fun fact: India was militarily superior to Britain in the 1600s—a gunpowder empire with a million soldiers—but was taken over by it in the 1700s. Britain’s edge was small: lighter, more maneuverable cannons, standardized ammunition, better military and political organization. Not a first world country versus a third world country—more like the dynamic US versus a sclerotic EU. And that modest edge led to 200 years of colonization.
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
You overstate the advantages of technology. Mughal India was fragmented and on a sharp decline. The British used politics, finance, and treachery to divide and conquer what was remaining.
> we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial.
An easy choice to make if the alternative is everyone dying instead.
But is it? Is there any realistic world where we need ASI for human survival?
Yes, this one. Look at our governance; look at our coordination-at-scale; look at our collective problem solving. It's abysmal, beyond hope. If we have global scale problems, we are not capable of solving them effectively. We are literally not intelligent enough to handle the problems we are creating. Between rivers of garbage and CO2 levels and war, we have proven ourselves to be woefully unintelligent at the scale needed. If we are lucky, our thin window of survival depends on getting a hell of a lot smarter, real quick.
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
lol, how the fuck is ASI going to solve any of those problems? we already know _how_ to solve them; the problem is that we don't want to, collectively speaking, because certain powerful, wealthy people would loose out if we did. ASI wouldn't change anything. unless you think... all of human society is going to restructure itself around unquestioning worship of the Machine God and would therefore present no resistance to its proposed solutions?
The fantastical belief is that ASI will be able to make things happen because it knows the right things to say to the right people at the right time in just the right way to make them do whatever it wants them to do.
Certain powerful wealthy people aren't omnipotent, them losing out isn't the only blocker to progress.
What is the solution?
The alternative isn’t “everyone dying.” It’s us holding all the cards.
Define "us"
Why? Is human extinction not permitted by the laws of physics?
That’s a possibility, but not the only one. The two most realistic ones are: we race ahead and maintain our status, or we slow down and open ourselves up to colonization.
I think the question “would China cooperate” needs much more investigation. Everyone online pundit seems to think “obviously not”, but they’re people too with clear positive and negative incentives. It’s possible they’ve found a very similar calculus that we have.
> “Politics is the art of the possible”
Using an ASI to subjugate humans in any capacity is a terrible idea.
without sharing tech to make the ASI, you'd hope humanity could work together to determine how to align an AI for our common benefit.
Worth noting that it is the Europeans and Americans that have been colonial. Asian peoples have, with the prominent exception of the Mongols and Japanese Empire, pretty much not done that. In particular, China shut down its exploration program.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
China is currently occupying Tibet, which never consented to the occupation and has invaded Vietnam 30 times. It ruled Vietnam for about 2 centuries starting about 600. China eventually had to leave Vietnam, but many other groups ceased to exist as a consequence of Chinese expansion. Here are some:
the Baiyue were a vast umbrella of diverse, non-Sinitic indigenous coastal tribes who inhabited Southern China and northern Vietnam.
The Xianbei were an ancient nomadic Proto-Mongolic people from the northern steppes.
The Di and Jie were two of the ancient "Five Barbarian" (Wu Hu) nomadic tribes of northern and western China during the Han and Jin periods.
The Dian Kingdom were an ancient, sophisticated indigenous southwest culture located in modern-day Yunnan province.
The Tujia were an indigenous group of the Hunan-Hubei region. Centuries of inward Han migration and intermarriage have resulted in the Tujia becoming culturally and structurally indistinguishable from their Han neighbors.
The Quing era boundaries are quite a bit larger than the Han boundaries. That did not happen by peaceful means.
This comment is a parody right?
"Asian people have pretty much not done that except for two teeny tiny indiscretions that each killed more civilians than all of Europe's and America's colonial incursions combined."
> The alternative isn’t “everyone dying.” It’s us holding all the cards.
That's one outcome, certainly, but not the only one nor, I contend, the most likely one.
A most likely outcome of ASI is human extinction, because there's more paths to an ELE outcome for humans from ASI than there is for non-extinction level outcome.
Your outcome is only possible if:
1. ASI is never able to escape the confines it is placed in.
2. ASI is benevolent to humans.
3. ASI decides, in the spirit of its benevolence, that it should restrict its involvement in humans.
If all three of the above conditions are met, then sure, your outcome is possible. If not, humanity as we know it will end.
It is unlikely that those 3 conditions will all hold, though.
Human extinction is good. Finally we built a benevolent world exploder. Oh no! The negative utilitarians get what they wanted finally!
If ASI is trying to wipe out all humans, we probably deserved it. Unironically!
While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
This is a just so story. The main issue today is the lack of democracy in the country and the use of technology to surveil and govern a restive population as the government has less and less legitimacy. The narrative you are telling is the heroic tale of computing and the internet c. 1990-2010.
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
http://yashalevine.com/surveillance-valley
Yes, and RMS was correct in his Right to Read and so many other things - we're seeing the slow death of the never-enshrined-in-law right to compute. Luckily open-source is big enough to slow this down; we should all be pretty amazed and appreciative that there even are open-weights models at all, out there, because it is a profoundly democratizing thing.
> appreciative that there even are open-weights models at all
thank, mr 习
I don't think parent commenter means tech in the modern sense. Seems like they're describing a transition to democracy which started centuries ago, not decades.
> Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
> shifted the material dynamics to better align the incentives of governments with the people
... recently, as in the last 10 years?
The decision to not go with the development of extremely large thermonuclear weapons might count - the US Sundial Project was supposed to be about 10 gigatons of TNT. Not the most practical weapons but once you get to a certain size delivery arguably stops being a problem - its going to kill everyone anyway so doesn't matter where you let it off!
https://en.wikipedia.org/wiki/Sundial_(weapon)
Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
Perhaps the burning of the library at Alexandria would qualify. How intentional that was is somewhat in question, but the world certainly turned its back on the only collection of written knowledge and let it turn to ash.
FWIW the burning of the library of Alexandria, and, indeed, its status as "the only collection of written knowledge" are myths.
Did it not burn? Was it not the most extensive library of human writing at that time? Please help us all understand. https://en.wikipedia.org/wiki/Library_of_Alexandria
> Did it not burn?
Part of it did!
> Was it not the most extensive library of human writing at that time?
No.
More importantly, the original assertion:
> the world certainly turned its back on the only collection of written knowledge and let it turn to ash
is definitely not true.
From the article you link:
> Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60]
> Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]
The 1972 Biological Weapons Convention? Probably partly successful.
The Soviet Union systematically ignored the BWC. There's tons of evidence.
Isn't the US famous for not even signing a lot of world treaties like climate accords and others?
This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
Biological weapons? Yes, there is research on defense, but no big arsenals of weapons etc.
My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
Taking the approach with AI that we took with the atomic bomb would be catastrophic. If the only people who are allowed to use this technology are governments, intelligence agencies, and a select few anointed companies, then the risk of authoritarian misuse will skyrocket.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
I agree. I guess I should have said something like:
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
Human embryo genetic modification has been effectively taboo if not banned until just recently since WWII and the aftermath of the Holocaust. I think some people in the US are proposing doing it now but I don't think anyone has tried it yet without repercussions.
Actually: https://www.economist.com/science-and-technology/2025/11/19/...
I can think of many examples that I won't name but you can imagine in biology/medical fields where certain lines of investigation are not performed due to ethical and legal repercussions.
If there were examples, their example status would drop the odds that I know about them.
It’s telling terrorists how to make bombs better apparently. Continuing to lower the barrier for that kind of stuff is clearly a negative in the “knowledge deserves to be free” world.
Has there been an uptick in terrorist bombings or is this just a hypothetical at this stage?
https://www.nytimes.com/2026/07/10/us/politics/ai-terrorism-...
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.
No, that is not a good description of Middle Ages at all.
For start, previous era was also deeply religious. So it switched religions, if anything new one was more friendly toward knowledge.
we should never make ASI, what I am saying is strongly sane. ASI = not good. AGI, ~human brainpower, can be made safe enough.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?
> defeat any nations that don't discard humans
AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
It's not even clear that ASI is a coherent concept.
But, I don't trust capital with either.
LLM + scale = more intelligent, this can be proven more than empirically, https://arxiv.org/abs/2409.15318 shows that neural nets can fit a number of independent AND-gate operations in their weights,
- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this) - and this system can produce outputs based on previous things it has outputed
its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.
it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.
What about it is not coherent?
It's not clear that "super intelligence" is a meaningful concept. This presumes that our concept of intelligence can continue to grow beyond human capacity as opposed to asymptotically approaching what humans recognize as "very intelligent". Perhaps, for instance, how we evaluate intelligence is bounded not by some quantitative capacity but rather our inability to agree on basic concepts/values. And do we even have tasks that a supposed superintelligence can tackle but humans cannot?
I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.
Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.
The Orthogonality Thesis (by Nick Bostrom) says that intelligence and ultimate goals are independent. Those non-instrumental goals can’t be stupid nor right or wrong. Increasing intelligence will not change the goals only the capability to reach them.
Do note that the orthogonality thesis is a hypothesis, not something we have demonstrated. Weak versions of it (e.g. it is possible to have an intelligent agent with arbitrary goals) are more likely to be true than stronger versions (e.g. intelligent agents we build will have goals uniformly selected from the space of all possible goals).
Ok so what is the point of neurotically obsessing about intelligence? Would it not be more fruitful to emphasize computation?
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.
Genomics is what finally broke the barrier, especially in the last decade or so.
I like your optimism and I think you will be vindicated. AI is democratic and AI talent is globally distributed. It will just take a while to get online. AI labs do not have a monopoly on human talent, and open source AI only empowers independent science and meritocracy.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
Uhh north Korea?
[dead]
[flagged]
[dead]
A successful example of reigning in progress: electric bicycles intentionally speed limited for safety
There is no rein on progress. You can buy electric motorcycles without speed governors. And you can drive them on public roads (subject to traffic laws) as long as you're properly licensed, registered, and insured. The distinction between electric "bikes" and motorcycles is mostly artificial.
I found the AI 2027 paper to be overly optimistic, but not wholly fantastical. This paper feels wildly speculative, and relies on premises I am not confident even pass surface reasoning. Even under optimistic conditions, we are not going to see robots "capable of 95% of all cognitive and physical tasks" by 2035. Nor do I think a 74% unemployment rate is even remotely possible. Economic collapse would implode AI development long before those figures were plausible.
The "and physical" is the part I'm particularly skeptical of. Sure, drones are scary, but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly.
A lot of it relies on what is effectively "the AI will be so smart it can solve anything" magic.
The book Sentient is not about AI but abount the most amazing physical senses some other animals have.
The theme of the scientific findings is that while humans excel with none of our physical sensors, we do very well across the board in making use of them thanks to our relatively huge brains.
And fantastical amounts of compute power is exactly what are handing over to AI. The fact that their training data isn't perfect may matter less.
but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly
If you don't care about getting the drone back, it does simplify the problem somewhat.
Zipline is growing fast… it’s drone-based, but it definitely delivers packages to your front door. (https://www.zipline.com/)
Nobody’s even solved a self-driving vehicles yet, not in in the sort of “they took over everything and put every uber and truck driver out of business” kind of way.
Maybe they will soon but it’s massively far behind the kind of timeframe AI 2027 would have implied.
I actually think self-driving is one of the easier paths of development. The main thing holding it back right now is regulation and liability.
But, if you could wave a wand and eliminate all legal and liability hurdles to self-driving, automobile deaths would plummet. They're way safer than the average human driver. The technology is definitely capable, our society just isn't ready for it.
I disagree. It has not been figured out enough to take humans out of the equation from a functional standpoint, not just a regulatory one.
This past 4th of July weekend a Waymo ran over an actively burning firework in a low speed scenario, the kind of thing a 95 year old driver would have avoided.
https://www.nbcnews.com/nightly-news/video/waymo-car-travels...
Do autonomous systems need to solve humanoid robotics to exert power over the physical world? Seems like a lot can be done with drones.
Military power, sure. In Ukraine they hit everything they can see.
But during peacetime, you don't make money running a delivery service that way, so it's not going to replace those jobs.
> Nor do I think a 74% unemployment rate is even remotely possible
250 years of constant automation has never produced large scale unemployment, despite obsoleting everyone's jobs several times over.
Don’t you think the speed at which obsolescence occurs matters? There’s a bit of survivorship bias here, in the sense of
“I’ve been pulling my sled across this lake for 50 winters even when the temperature went above freezing. Never fell through!”
It has. For example mechanization of argiculture in places where it didnt coincide with a manufacturing boom (latin america, india, africa) resulted in shantytowns and long term unemployment.
Those are pretty dysfunctional economical and political systems.
In the industrialized world, old professions disappear all the time, and are replaced with new ones. I saw somewhere that it's about 2% per year.
If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
> "If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me."
It's 2026, one year after your predicted date, and that still hasn't happened though.
You say "just" 15 years, but Waymo is still only available in a few cities. That seems more like a slow, cautious rollout to me, not a fast takeoff. Society has had a lot of time to get used to (and tired of) the idea and come up with regulations.
My guess is that the deployment of other types of robots will often be a similarly slow grind.
That's unlike the Internet, smart phones, and coding agents, which got user adoption at a much quicker pace.
Waymo was largely built pre-LLM and AI level funding - I think it might be a somewhat apple and pairs comparison.
Maybe research and development will speed up a bit, but I think it’s still going to require a lot of expensive experimenting in the real world.
The thing about exponentials is if you admit 60%, it's pretty easy to admit 95%.
That's a fair push back.
For a specific technology to become viable, you often need progress in several individual technologies to occur. Some of these will be exponential and surprise us, but many will see slower progress.
Either way, I suspect progress in robotics will be slower than AI, but I also think we'll see a lot of investment in robotics over the next decade given that AI opens up a lot of new potential applications for robotics.
I suspect any task that doesn't require extreme battery life or extreme dexterity will be doable by robots much sooner than most people think today. I think we could have decent-ish humanoid robots within a decade – robots that can prepare food, clean hotel rooms, empty bins, etc. I think Elon's bet on Optimus is quite likely to be viewed as very ahead of the curve with time.
However, it will take much longer for robots to do everything a human body can, but then most jobs don't require we push the physical limits of human body.
That's why I think 60% is reasonable. And even then I'm not saying it will happen, I just don't think that's a bad bet if given even odds.
Depends on what kind of curve it is. 60% reliable is useless in most safety-related fields and getting to near-perfect reliability is tough.
The biggest issue with 2027 was that it didn't understand the economy.
For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)
Without massive amounts of investment, AI development stops dead.
In this post, they hand wave about the USA being able to acutally 1) build concensus locally for regulation and 2) the rest of the world actually follows suit.
It fails to understand that actually the progress of AI is not actually the gift of the USA. It requires a constant supply of things from china.
Also its assuming that having 74 billion agents doesn't cause economic distortion. Like what value are these agents generating that justifies them being run?
I really wish people would just ignore this for what it is: bad sci-fi with an incomplete world.
Worse, there's actually a very detailed supplement about their economic modeling: https://ai-2040.com/supplements/economics-of-plan-a
Which predicts that explosive growth of robot production will lead to problems such as
> a deflationary debt spiral, where the AI and robot companies can’t pay back loans in dollars because the robots and AIs are worth nominally less than the loans written the year before.
In other words, the companies go bankrupt because they produced an oversupply of cheap goods, the bubble pops, and there's less new investment for a while. Plenty of precedent for such a development.
But instead of adjusting their predicted output growth downwards accordingly, they instead propose that
> One way to solve this could be for the loans to be denominated in AI and robots, so the companies pay back the loans with some percentage of the AI and robots instead of dollars.
Try doing this today with a battery factory for example. You expect that battery prices will fall to the point where the revenue from selling batteries won't ever cover the cost of building the factory. So you propose to a bank that they'll be the ones to build the factory, and you'll borrow it from them (not paying rent?), make your batteries, then give back the factory when you're done. All the profit is yours, all the risk is theirs! Which is of course why a real bank won't agree to this, all you're going to get is a dollar loan with the factory as collateral.
Oh I missed that, its more batshit than I had imagined.
+1 this 2027/2040 AI articles are devoid of real-world practical economic/finance thinking. nor even human psychology.
> Without massive amounts of investment, AI development stops dead.
Would the US government not pour enormous resources in AI labs if needed, knowing that China might be doing the same? What happens if an adversary develops an AI capable of finding and implementing exploits in every software run by your country's strategic infrastructure?
> Would the US government not pour enormous resources in AI labs
I mean they might, but its not clear how they would do it, especially as they are reaching the point where its going to be expensive to borrow.
> For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)
Why would it be inflationary?
The economy doesn't need workers as consumers necessarily. It would of course be a huge shock to the economy but the economy could adjust to it eventually. Maybe it compromises the 2040 timeline. Still, billionaires are increasingly holding the assets. The money supply drying up can be countered, as it is controlled by central banks and can be arbitrarily increased.
> as it is controlled by central banks and can be arbitrarily increased.
Its really not controlled by central banks. Its influenced, but not controlled.
When central banks "print" money, they effectively just add money to the accounts of investment banks
But investment banks are also "printing" money. Double accounting effectively uses assets to double the available pool of money. If you then sell off those loans based on those assets, then you crystallise that new money. Investment banks are inflationary.
It seems to me we’re already at the top of the S curve, not at the toe of an exponential curve. At least with LLMs. Better training data will make small improvements, better architecture will make it less compute intensive, and all these “hyper-scale” data centers will make it cheap and ubiquitous. But none of that is it getting exponentially more intelligent.
People have been saying this since GPT-1. This idea that we can only squeeze a little bit more of intelligence out of LLMs isn't a new one. And thus far, it has always been wrong.
What leads you to believe that?
Things like https://www.tobyord.com/writing/hourly-costs-for-ai-agents and https://www.tobyord.com/writing/mostly-inference-scaling seem in line with other accounts like https://www.youtube.com/watch?v=aR20FWCCjAs ?
The author of the posts you linked also wrote https://www.tobyord.com/writing/inference-scaling-reshapes-a... which posits:
> AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.
> ...
> So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.
So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).
> So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence.
On its own it wouldn't. But that article came before the later article https://www.tobyord.com/writing/hourly-costs-for-ai-agents which adds the claim that inference (along with everything else being employed at present) is scaling poorly with increasing task lengths. Now maybe the December 2025 claim is wrong, or maybe things will change soon, but the February 2025 article surely doesn't establish either of those.
Is chatgpt 5.6 that much smarter than chatgpt 5.0?
If you’ve done any software development at all, certainly.
Is that true or does it only feel true because they nerf the old models just before every major release?
I remember being blown away by o1-o3 family of models finally stringing together coherent agentic tool calls to write and execute scripts semi-reliably for workloads in the several minutes before they would start hallucinating/flailing. GPT 5 was a bit ahead of that, but barely
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
Not only this, but in the GPT 5.0 era, agents had 0 taste. Nothing looked good. It was the agentic version of the twitter bootstrap era, but worse somehow. Now, I would argue the average agent frontend beats the average human frontend. This isn't even getting into 3D applications in the GPT 5 era
Anyway, the models now reliably execute more than a human can fit into their own context. It's magic
Yes, and we haven't even really begun to nail down computer-use agents yet (can you believe they're still basically just OCR'ing screenshots?)
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
How do you think humans experience desktop interfaces? “Basically just OCR'ing screenshots” is exactly what humans do.
It's not the same thing. For example, given a GUI with a titlebar, title, subtitle, text, and buttons, a human can instantly understand spatially the relationship between these items. But a naive OCR of such a GUI would be a flat stream of text that loses a ton of information.
But that’s not how models handle images either. They spatially segment and reason about title bars, placement, etc.
I was also under the impression modern AI agents have moved on from just OCR'ing screenshots to leveraging native vision model capabilities.
They do. They all use ViTs and have for quite a while.
Can you share what's your setup for all that orchestration? I feel way behind just asking Claude Code for code edits. Is there any site where people share different AI setups, besides youtube?
Fwiw, don't buy into all the hype that you're falling behind. Yes, AI does cool things now, but I would say the impact is still unproven past indie hackers or early-stage startups. And a lot of the esoteric setups people have created with things like OpenClaw have become outdated as quickly as they were conceived.
The popular thing is now to setup loops (eg I setup hourly integrations for Claude/Codex to 1) scrape my Linear, claim achievable tasks, and push PRs or 2) do root cause analysis on customer issues that evaded automated filters, to name a few)
Though for me, my setup still feels mundane. I have AGENTS.md, CLAUDE.md etc and a few skill files. These are purposefully light - tons of examples online you can pull from online. Mine are fairly personal to my setup and products.
Importantly, I also allow Claude and Codex to bypass permissions. Yes, there is a risk they wipe my machine. The productivity upside has been worth it, for me (haven't been burned yet, ~9+ months into running models this way, I have backups, use cloud etc).
As far as maintaining quality, one of the most helpful guardrails over the past year, for me, has been requiring my agents to pipe their changes to local reviewers through OpenCode, Cursor, etc agents to have a council of models with different biases reviewing the changes, and autonomously working towards a completed objective. No matter how good Claude or Codex gets, for example, I will probably always want a different model checking its work. Like GLM, (now with 4.5) Grok, Composer.
Several OpenAI, Anthropic, xAI employees, and popular AI engineers post on X and share helpful tips & updates. Highly recommend for keeping a pulse on startups and AI. I haven't found something close, honestly, other than when I spend time in SF talking to people.
I always thought bootstrap was pretty good. All the gradients and sparkles don't do much for me.
without taking down prod even 1% of the time
Literally every major company that has embraced AI coding has suffered devastating downtime this year as a direct result of AI induced failures.
Wasn't writing about major companies. That's obviously next, if we follow the trend lines.
Most companies have already given up on AI. It was a bigger disappointment than big data.
The only companies sticking with AI at this point are the major players who have chasing their masssive overinvestments to the bitter end, crappy coding shops, and for some reason Starbucks and they'll all suffering badly in a year their customers begin demanding massive amounts of human coding to repair all the issues with the AI code.
We're still in the early adoption phase. Betting against the internet wasn't a great idea, and betting against AI doesn't look like one either ;)
You can compare benches of the old models against the new models. So yeah, you can see the difference.
Even then, you can just compare the progress in open models. Leaps and bounds from where they were 6 months ago.
[dead]
5.6 to 5.0 is a big enough of a jump to say yes. if it was 5.4 to 5.6 it would be a bit easier to say it only feels true because of that, but 5.6 is definitely better than 5.0. I don't have anything empirical to point at though, which is your point, but August 2025 for 5.0 vs July 2026 is almost a year later, and it's not just vibes that it's better, despite not having an objective metric to point at. It would be more scientifical to have numbers and shit to point at and there are some benchmarks out there, but you have to dig into them and really understand them in order to believe in exactly what they're testing, and I'm betting you haven't.
Yes, it very clearly is
I completely agree, but I also think that an industry disrupting architecture tweak akin to the “Attention is all you need” is VERY possible to emerge at any moment.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
This is from the "AI 2027"[1] people.
[1] https://ai-2027.com/
I'm eagerly awaiting "AI 2100" from the same people.
"All these GPUs are yours, except Europa. Attempt no landing there".
please bro just one more decade bro i promise AGI is right around the corner no the hockeystick graph isn't made up bullshit we just need to wait a few more years bro please please bro
big number goes up and becomes larger number, chart goes up and to the right, this must be a solid measure of progress in society.
A hockey stick graph on a log scale is a bit much.
I suspect "AI 1844" would be more apt for this group.
This seems way more accurate. Up until early 2026.
Though it has bizzare fixation on geopolitics and China which it severely understimated. It's pretty obvious that China is going to outinnovate and outcompute US companies quite soon. Even if just because they care about higher education, providing enough electricity and letting smart people do smart things instead of randomly muzzling them with bans and export controls and coddling them with financial protectionism.
Hm, China is also beginning to invoke export controls to restrict homegrown models: https://www.reuters.com/world/beijing-is-looking-curbing-ove...
I wonder how much of what China is currently able to do it because USA is such a mess.
The whole article is kind of ridiculous of course, and is also heavily fixated on OpenBrain, whatever that is.
I also wonder about the economics of running an AI lab attached to an existing large tech company (such as Meta or Tencent) instead of a dedicated company like OpenAI. It's starting to seem like it's not possible to charge enough for current-gen AI usage, with current-gen inference technology, in order to turn a profit, i.e. nobody is able or willing to pay at least marginal cost for tokens.
Hm, Anthropic reported it was close to operating profit last quarter: https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...
And a third party estimates it will exceed $1B profit in Q3: https://newsletter.semianalysis.com/p/anthropic-3q26-profit-...
> This seems way more accurate. Up until early 2026.
Which is funny, because they launched the AI 2027 site in 2025 and it caused a lot of people to believe the end was near.
They claimed to have built a complicated model, but several people showed that it didn't matter how much you changed the inputs, it was designed to converge on the answer they wanted.
- whenever i read this, why does it always read like some first world country guy born in a bathtub who has never been outside his bubble, never seen what the rest of the 90% world looks and works like actually sat and wrote a work of fiction
- have you seen the actual problems in other countries outside america?
- have you been to any country outside USA and the countries in europe?
- have you taken a trip to any developing country and stayed there for a month?
- have you seen what sort of daily struggles, political systems, bureaucracy and work exists in developing countries?
I think you need to explain why the problems of developing countries are related to the issues of the article.
the article makes a lot of predictions and every single one of em is based on how things work in the USA
What would be different in developing countries?
untracked economy at scale. like a tonne of work, occupation, money is not documented or doesnt enter the system channels
This is by far the most realistic optimistic AI takeoff scenario I've seen, and more specifically it's the first one I've read that deals with both the AI alignment and power concentration issues in a sufficient way, even in a world where hard alignment is assumed (in this scenario the AIs are assumed to be misaligned until ~2038-39).
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
>We already wrote a scenario about this, called AI 2027. It depicts takeoff happening in 2027 instead of 2030
So in less than 3 years, their exponential growth curve doomsday prediction has moved back 3 years. This seems to be the opposite of exponential growth.
No, 2027 was never their median forecast; it was their modal forecast. See https://blog.aifutures.org/p/clarifying-how-our-ai-timelines...
It's more like it moved from 2028-2032 to 2030-2035 (depending on the author).
Fast takeoff was always only possible if you ignore all of physics and everything about how practical reality works, nothing exponential lasts for long in a finite system. If they believe that, then a certain level of self delusion is required anyway and you can ignore anything you like. Sort of like communists saying socialism will definitely work this time.
The book Superintelligence was so highly praised years ago but if you actually go and read the thing today practically all cases it presents read like raypunk retro-futurism that makes up imaginary fantastical nonsense in place of the missing knowledge it would've needed to make any sensible predictions. Practically none of its assumptions apply to LLMs as they currently exist, and some we've learned since about human inteligence are wrong too.
If carbon taxes are already a lethal policy for an political campaign, it's absurd to think that fears of ASI will create any real movement around pausing AI.
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
If that is true and one cares about a moratorium on progress in the US then it seems like the number one way is to meet people where they are: so water use misinformation, degrowth, power supply constraints. That does place all the people who push for these things in a different light. They may well be attempting to do what the AI safety labs are ostensibly trying to do.
As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.
If there's going to be any pause, I'm sure it will come from a populist movement. I just can't imagine misplaced worries about AI water use will translate into the kinds of policy the authors want to see.
Yeah it’s like shoving the top of a double pendulum. You will get some movement in one direction, but where it will precisely land is hard to predict. The water-use argument is already earning refinement by differentiating “AI datacenters” from “normal datacenters” in an effort to control the movement.
I imagine any populism movement will require rampant fearmongering to get a result. Considering the rough present alignments, presumably blue tribe focused propaganda will involve climate and inequality focused fear and red tribe focused propaganda will involve job loss. Grey tribe positioning is the P(doom) meme where everyone is rewarded for a high-P(doom) estimate.
It's possible the general public wouldn't care enough one way or the other such that elected representatives could just do it without any change to their electoral fortunes.
You should consider why the best ally to your position is misinformation.
This is not the gotcha you think it is. Misinformation plays to all manner of political issues and sides.
I don't think it's a gotcha at all, they openly said it and were pondering how that might help them. Misinformation plays more to sides that are wrong about more things, and I don't consider misinformation my ally in anything.
This criticism, like most criticisms of AI, addresses the wrong part of the problem. The problem is harm to other parties. Typically customers and employees of corporations.
A solution is strict legal liability. Corporations must be strictly liable for harms. That liability should be higher when AI is involved. Such liability may not be waived by contract, forced into arbitration, or devolved upon a third party service.
Then we let the plaintiff's bar and the insurance adjusters price that risk.
Something like this turns on in the EU in October 2026.[1]
[1] https://cybernews.com/security/eu-will-hold-tech-companies-l...
No, nobody can be held liable for the extinction of humanity, for there is nobody left to pay up. If your objection is that you think AIs won't cause human extinction, you should say that instead, as that would be where you disagree with the authors.
I note that they use the Sam Altman quote.
"AI will most likely lead to the end of the world, but in the meantime there will be great companies.”
Is there any serious journalistic source suggesting that this was anything other than an offhand joke? This article links to a youtube clip of the comment with context removed, but hair raising comments.
Taking the most uncharitable view of any person, you could imagine someone who was evil enough to cause the end of the world after their own lifespan where they faced no inconvenience, but not the circumstances from the quote
The quote as it stands is preposterous enough that I don't think a human capable of functioning in society would seriously say such a thing.
Are people wilfully misinterpreting the comment, or do they truly believe this an actually held opinion? If so, can they explain how they think someone could hold an opinion like that?
https://www.youtube.com/watch?v=d6lDZpvHAoo&t=527s
The question (7:35) is "Where would you like to see people investing more time?" And Sam seems to be saying AI safety, I guess? This is 2015, and he refers to founding OpenAI. Based on his actions since then, yeah, seems like it's not a joke to him. This is Altman we're talking about.
I think he means the end of the world as we know it. Which is probably true, but the timeline could be 20 years, 2000 years, or more.
Ask ChatGPT:
> Generate a realistic image of a bad fitting suit without looking obviously fake
Create a new chat and attach the image generated previously:
> Please rate my suit
You will receive an answer that reads something like this:
>7.5/10 — clean, professional, and well coordinated.
Yeah, I don't think superintelligence is imminent.
We are only some data centers away
NYT reported today that Russia and China are funding anti-datacenter and anti-ai hysteria on western social media.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
These campaigns have historically amplified conflict. They do not care what the conflict is about.
But unlike some of the others, I’m hearing anti-AI sentiment from a wide range of people who don’t even use social media.
All news is influenced by what’s popular social media these days. And that becomes part of what people talk about through the grapevine.
But no doubt there’s plenty of organic NIMBYism, anti tech growth stuff, and run of the mill fear of change and loss of control as society grows more abstract/centralized.
Applying the term NIMBYism to anti-AI and anti-DC sentiment is a gross abuse of terminology. Datacenters don't need to be in anybody's residential neighborhood.
These developments are always facing local resistance. I previously lived in a place where they were building a quarry 30min outside of town and tons of houses had anti-quarry signs on their lawns. It was a big deal to appear anti-quarry even though it had little to do with their neighbourhood specifically (except maybe increased highway trucking traffic).
Most industrial development face local protests like this and it's often has large crossover with those NIMBY who resist stuff like housing developments, and show up at town/city councils.
Living anywhere near a quarry is no joke. There will be a lot more heavy trucks on the road, with drivers who care very little about covering and securing their loads and even less about your windshield.
Seems like you ignored the GPs point to tell a NIMBY anecdote? They're questioning your premise, not asking what "NIMBY" means.
Everyone lives in a world deeply affected by social media. Even if you've never looked at a screen your entire life, you have spent thousands of hours talking to and being informed by people who did.
World, yes, as people around you including politicians and businessmen have had their brains pickled.
But I doubt the influence. I’ve been free for years, and I can always spot who’s spends a lot of time in TikTok/Twitter/Instagram—it’s like talking to someone from another planet. It mostly sounds weird and sad, more apt to annoy and alienate their friends than inform or influence them.
Which then gives more ammunition to folks like Kevin O'Leary et al to pretend any objections to data centers or AI are Chinese plots, or "hysteria" as you put it.
While I can admit some of the anti-data center arguments are overblown, many are more than valid in my opinion. Data centers are fundamentally extractive technologies. They are enormous, windowless boxes that take resources from one location to make someone else in a far off location enormously rich and powerful, with extremely few benefits to the local community.
Plus, as another commenter mentioned, it's not exactly like the Chinese and Russians have been fanning the flames that AI is going to take all of our jobs - it's the leaders of the frontier AI companies in the US saying that. Remind me again why I think putting up a giant data center in my state, that was proposed to use more electricity than my state already currently uses, is a good thing for the average joe where I live??
Honestly, I feel like many commenters here are in their own bubble and don't understand how much AI and tech generally is widely viewed as a net negative for society by huge swaths of the the population, and I don't really think it's an unwarranted perception.
Most of the issues with datacenters could be solved by a) investing in energy b) ramping up RAM and chip production, and c) enforcing already long established rules around industrial water management.
These tech companies are already investing heavily into solar, natural gas, and nuclear https://techcrunch.com/2025/03/30/data-centers-love-solar-he... this would be normal stuff in China where they spend the last decade investing heavily in solar and are bringing something like 60 nuclear reactors online.
These datacenters aren't particularly consumptive of water compared to most other industries in that regard and we've already seen states enforce rules against Meta who immediately paused their datacenter when water issues were detected (following mandatory monitoring).
Chip production is lagging but most projections I've seen is it will normalize in about 5yrs. Not to mention there will be further demand for robotics and self driving cars, so ramping up chips should be a normal thing like ramping up green/nuclear energy. Delaying it won't solve any current issues.
There are 22 golf courses within a 30 minute drive from me, and people here a losing their minds over datacenter water usage...
People are scared about the personal impact from AI, then backfill in justifications without even realizing they're doing it.
If the equivalent numbers for electricity and water usage were being being used for streaming video, I seriously doubt people would be demanding no more Netflix data centers. The news story would immediately die.
Nobody wants their electric rates to go up, the local water utility to have to raise rates to build a bigger plant, all in exchange for also losing good white collar jobs. That’s currently what AI data centre builders are selling.
Not exactly. There are some data centers being built in places that don't have the power and water to support them, and obviously it's rational for the locals to oppose them.
But I live in a place where we have plenty of water and relatively cheap power (lots of renewables). There's not much risk to data center construction, but people are opposing it here, too. Because for most people, it's not actually about that.
An obvious question is if the cheap power is going to stay cheap after a large power-user comes in who has a proven track record of trying to make everything cheap for themselves with no regard for anyone else.
Or another question to ask is - how does this data centre benefit the people who live there? If it doesn't, there's no reason they should want one to be built. Rubbish tips are necessary. I still don't want one built next to my house and would fight such a thing tooth and nail.
Here's a better question: as we've had nearly as large of a data center build-out happen between 2005 and 2020 for non-AI purposes, with similarly high electricity and water demands...where has the concern been? Why is it only in the last 2-3 years that people are suddenly up in arms, as a very specific application is being deployed?
The amount of data-centre construction is far more than it was 2-3 years ago.
There seems to be an alarmingly high amount of questionable data centre construction going on, such as projects being built in places with no access to power with an assumption they can somehow force the utility to provide it later. These buildouts seem to be being done for financial reasons (they are not Meta, Amazon, Azure, etc. facilities) with the hope to lease them out or sell them half-completed in the future. People rightfully don't want that kind of thing in their back yard.
To give a feel for the scale involved, this one (the new Amazon east DC) in my podunk area of the state is 250 MW (the existing us-east-2 in Columbus is 200 MW, although I'm not clear if this will be a new region or is just an additional availability zone). But that's small potatoes compared to the speculative project in Piketon, which amongst more absurd things is planned to be:
- 10 gigawatts (equal to 50% of current power consumption statewide) - "Modular" nuclear reactors built on site - 35,000 construction workers needed to build it (in a county with a total population less than that) - $30-$40 billion for the data centre, plus another $33 billion to build the 9 gigawatt natural gas electric plant - Meta agreeing to build an additional 1.2 GW nuclear plant on site - OpenAI in negotiations to lease the facility
This is a really big project, of the scale of "nothing like this has ever been done before". Nobody has ever built that much power generation at a single site before, nor has a datacentre this large ever been constructed. There is a very real risk of the project getting halfway done and then being unable to be completed. The prospect of a state literally doubling its electric generation is a bit ambitious, too (doing such means basically a complete revamp of the power distribution grid, or else some very novel designs to only use the power locally). For example, the normal type of shutoffs data centres have to prevent eg an incoming have are unacceptable in this situation because the grid cannot cope with 20 GW of demand suddenly disappearing.
People have a dozen reasons to refuse data centers being built in their communities and zero reasons to encourage it. They're little more than post-industrial mines that take limited resources (power, land, water, quiet) from a community, sell them for profit as compute, and siphon those profits away onto the books of far-flung megacorps with no community reward.
Meanwhile, golf courses are a traditional green space where people in a community gather for both work and leisure. They're not ideal themselves, but they at least provide some benefit against which their negatives can be weighed.
If all you hear from critics of data center building is water use complaints, that's strictly because you've chosen not to listen to people.
> data centers being built in their communities
> golf courses are a traditional green space where people in a community
I have a feeling those two sets of communities are disjoint
How is this different from farming?
It takes the limited resources of land and water from a community and sells the result for profit as food or fuel. The vast majority of profit is made downstream and outside the community.
Golf courses being a traditional green place where people gather seems a bit far fetched to me when most of them are elite private clubs.
Most golf courses around me are open and anyone can go play for a cheap greens fee. The clubhouse has normal low end restaurant prices for a hot dog or a burger.
If the clubhouse is your idea of the valuable public good we can provide without 18 holes of grass.
As far as "valuable", the golf course is private property that the owners presumably bought and can do with as they see fit. If you want to develop it into something else, make an offer to them?
The golf courses are a business like any other (although we do have some publicly-owned golf courses around here too). The cost to play 9 holes on a weekday is $10 at one of them. I'm not really sure what you're asking for here.
Data centers have negative externalities which is why people don't think anyone should just be able to buy land and turn it into one.
Once of those negative externalities is water usage. The parent was commenting on the also high water usage of golf courses.
Perhaps that's the connection you've missed.
That's highly dependent on the area it's in. Where I live, golf courses use no more water than anyone else and aren't irrigated.
The proper answer would be to simply charge appropriate prices for large scale uses of water from the water utility, or else this is a discussion about riparian rights and law and possible changes needed to that.
I associate golf with WASP-y business types who hobnob with their boss in an electric cart to get ahead, and can't stop talking about their hobby of chasing a little white ball.
Is that fair? Probably not. But I don't think golf is a particularly inclusive sport, unless you live in a golf course gated community... in which case everyone is included.
People are more lenient on farming because humans need agriculture to survive. Obviously not all agriculture is necessary for survival, but it's something that in some way provides real tangible benefits for everyone.
On the other hand, if AI data centres all disappeared today, humanity would continue on completely fine.
The test of "if it disappeared today, we'd all be fine", would eliminate most of the economy.
In farming, the result is food that can be eaten.
Golf courses don't do that either...
> People have a dozen reasons to refuse data centers being built in their communities and zero reasons to encourage it.
This is, ironically, the NIMBYism that so many people hate.
People generally don't want anything built in their surroundings unless it directly benefits them and has no downsides, however low the impact.
When mines pay a sizeable share of their profits as local taxes, and obey environmental regulations, people suddenly start to like them.
Can you point to some examples of where that's happening?
These data centers are specifically being scouted for communities whose governance is too weak to negotiate for some "sizeable share of their profits" and too ill-prepared to have suitable environmental regulations on the books already. The Ivy League sharks planning these buildout initiatives are sharp people who are looking out for the interests of their employers and know how to pick locations where they have the best opportunity to exploit locals unprepared for their kind of esoteric deal-making, political lobbying, and lawfare. They'd be failing at their job if they did what you're suggesting, and that's why we don't really see that happen.
For any benefit to national or global society AI data centers might provide to someone, the buildout looks a lot like the dirty and exploitative stories of rail expansion in the 19th century. That rail infrastructure proved a good thing for the US, but that doesn't mean the process of making it happen was honest or good for the people immediately affected.
This may sound crazy, but I’m glad the data centre being built near me (the new us-east-3) is being built by Amazon who pays lip service to local government and the community, as opposed to cartoon villain levels of saving a few pennies by forcing noise pollution on everyone else and everything else undesirable the other builders are doing.
Never thought I’d say that.
> Can you point to some examples of where that's happening?
You mean the AI datacenters doing it? No, they are not doing it.
They seem to be doing the opposite. They being loud is really hard to accept, decorrelating the fans cooling them would probably pay for itself in less than an year. It's like it's some Capitan Planet villain building those things.
Got it. I misunderstood you. Agreed.
States give the tax incentives, local municipalities are cheap to buy out, that's why all these towns quickly pass approval. $30 million annually is pennies for a datacenter and a windfall for a town.
People can care about more than one thing
Personally I would happily close down all golf courses and put them to better use as literally anything else.
Even just making them public parks would be great
Coming in for a landing on a flight recently, I was amazed at the contrast I saw between a housing development of 50+ homes and a golf course across the road that took up the same amount of space.
I attended an auction of a golf course a few years ago. It went for a few thousand an acre. You could have shown up and bid on it.
The winner ended up just choosing to keep the current employees and keep operating it. Nobody, I mean nobody, wanted the land for development. It was in an era with basically no zoning either.
No, in fact, most of us could not have just shown up and bid on it with any expectation of a meaningful outcome.
I'm saying that if you don't like how a golf course is being used, then attend an auction of one, because they are rapidly going out of business and the assets being auctioned off.
America is not suffering from too many golf courses being constructed. They are, rather, going extinct, and I don't really think the mass loss of green spaces and third spaces is necessarily a good thing, even though I'm not someone who enjoys playing golf and don't really spend any time at golf courses.
> I don't really think the mass loss of green spaces and third spaces is necessarily a good thing
Maybe we could make them public green spaces and third spaces, instead of exclusive clubs?
Many golf courses are really expensive. Golf itself is dying like you said, because it's a very expensive sport
Idk. There's something like 35 golf courses where I live in Calgary and it's a city of less than 2 million people. That seems super unnecessary to me and they don't seem to be going extinct here
There are also 132 Tim Horton's, which aren't "necessary" either, but people like having leisure activities. One golf course per 55,000 people doesn't seem that excessive either.
If you'll indulge me some armchair research and math
According to google AI, the average square footage of a Tim Hortons generally between 1000-2300 square feet, with some older locations taking up 2500 or more.
So let's assume every one of the 132 is an older location taking up 2500 square feet. That's 330,000 square feet, or 7.57 acres
According to the same AI, the smallest golf course in Calgary is:
Lakeview Golf Course: The city's smallest 9-hole executive course covers approximately 60 acres and features mostly par-3 and par-4 holes
60 acres!
Unless my numbers or math are wrong, Literally all of the Tim Hortons in the city can fit into the footprint of the city's SMALLEST golf course
These things aren't equivalent. Come on. We can use the space for golf courses for better things. :/
Edit: That's not even accounting for the fact that a single tim hortons probably serves more people in a couple of hours than many golf courses do in the course of an entire day.
I don’t get the appeal of golf. Hitting a ball into a hole with a crooked stick? But hey, that is just my personal preference. People can play whatever sport they want, or do whatever they want and call it a sport.
That said, it would be nice if the so called sport didn’t take so much land, water etc. Especially in prime locations
People can pretend to care about more than one thing.
Whether they actually actively oppose those things to the point of impacting building permits, that's a completely different matter. It really doesn't take much legislation to make golf courses economically unviable and force them to close, especially if you've got enough population within 30 minutes to support 22 of them (I speak from experience, I helped write a water reclamation ordinance that shut down at least one in my SoCal city)
If anyone actually bothered to talk to their local reps instead of posting internet comments about how much they "care", they'd get something done. If they don't, their care is just a fart in the wind for all the good it will do.
Golf courses don’t have backup generators running 24/7, with humming you can hear from a meaningful distance away. They also don’t pollute the air.
This is a poor comparison, but I do get what you’re attempting here. It’s also absurd that we are leveling land everywhere around me to build warehouses. No one is really complaining about that, either.
A backup generator wouldn’t be running 24/7?
Russia and China aren’t the ones constantly telling us AI will put us all out of a job, here’s why that’s a good thing, and why the government should dedicate billions of dollars to incumbent AI providers.
Nothing makes me dismiss and distrust a person or organization faster than hearing them say "russia/china are funding {controversial_thing}"
Its boogeyman thinking.
source needed. I'm not saying ur wrong, but source needed
What choice does that rag have? Of course they're going to backpedal and try to launder blame.
Surprisingly enough the us constitution specifies a right to free speech. Meanwhile who is funding the past two decades of pie in the sky bullshit from silicon valley in the media? Is promoting and unpopular opinion illegal? Since when do we judge the merits of an argument based on who articulates it?
Source or link?
Did anyone else catch the logical inconsistency between Plan C and A?
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
By legislative design? If a nuclear bomb could be made with hardware store finds the world would already be over. Big collaborative works raise the stakes and the observability for surveillance. Apply for a job at a defense supplier or even and energy company.
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
In Plan C the government essentially misses the opportunity to implement the multinational deal while the threat of covert projects is still low (fewer latest-gen chips unaccounted by tracking measures, worse models/algorithms to use for RSI). That's why it says the probability of a deal is lower and lower each month rather than outright zero.
Why are they obsessed with China?! Actually most of the US are. I guess if you’ve been a colonizer, your biggest fear is to be colonized.
The AI narrative requires an enemy and a “race” that the US has to win. It’s a pretext for justifying spending, rules, urgency, etc and making it sound more important.
Perhaps it’s because they are #2 strongest entity? Or maybe they are number 1, no one can say for sure.
Your comment also implies that those who haven’t participated in colonization might not fear being colonized. Seems… off.
I’m not saying that … my point is if you treat someone like you’re enemy they will become your enemy
This is whole a slippery slope. Always building on assumption of infinite exponential growth. But every exponential is at a certain point a sigmoid.
Nothing follows from this empty platitude though, right? It can't inform you choices or decisions? It's just a disempowering thought?
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
Degrowth is a deeply unpopular policy around the world. Where would one even go to find a degrowth rally? I have to imagine everyone there would be there ironically.
> * they net shutting down grade schools or building them in your neck of the woods?*
My area (rural Iowa) has had several new schools built in the last 10 years. Net gain for sure.
AI is just a tool. It will be a tool as we constantly push the requirements of what qualifies as consciousness. Therefore I'm not worried about it. Almost all the predictions about future growth of technology were incorrect. We don't have flying cars, hoverboards, fusion, the matrix and so on. The goal of each of us is to live a better life. It always be and forcing people to give up luxuries for some external agenda based on slippery slopes is cruel and selfish.
We should have been under water, hunted by AI, overpopulated, killed by terrorist, smitten by god for our sins and so on. Luckily all it took was our privacy and a lot of tax money to survive.
Whenever I encounter these people I'm reminded of the meme about the baby who has doubled his weight in the three months since birth. At that growth rate, he'll weigh trillions of pounds by age 10.
> But every exponential is at a certain point a sigmoid.
strongly, no. its just hard to distinguish them. for example, radioactive decay. cmon
Point taken. But it's interesting that the example you give is one of exponential decay rather than exponential growth, which is the context of this discussion. Where in nature has anything ever grown exponentially in perpetuity? Bacteria in a petri dish multiply exponentially. Until they don't.
> Where in nature has anything ever grown exponentially in perpetuity?
you could make the argument for human total GDP, which looks like https://ourworldindata.org/grapher/global-gdp-over-the-long-..., but then again, you could say we just haven't reached the sigmoid in it. I personally doubt we will
Most AI alarmism benefits big tech itself. Either they want to create awe and thus demand for their products (see the recent dance around Mythos), or they want to create regulatory capture and increase the moat around AI.
Interestingly , we as a species already created an overreaching "cybernetic" system that controls our global society and that the individual is powerless against - it is not AI, but capitalism. Thus the current danger is not that AI becomes superintelligent and enslaves us, not that it makes a regime ultra-powerful, but that it increases economic inequality and concentrates economic power in the hands of even fewer people.
The irony is that new technology could allow us to live a life of abundance and leisure, but instead people are laid off, made unable to participate in the economy, etc.. The technology is (as so often) chafing against the bounds of society. I sometimes wonder what the superintelligent AI would say about this, and if it would come up with a completely novel political theory - "silly humans, why don't you just organise your affairs like this, you'd be much happier, and we could coexist much better: ...".
Firstly, let me say that "AI 2040: Plan A" is a nicely done presentation. If feels like reading an interactive graphic novel with a narrator to boot. Can't wait for the book, series or movie :)
Plan A seems like a good start and am glad an effort is actually being made to address any potential dangers. The only weak link I see is that there is no way for inaccessible, third-world countries, non-aligned states, and malicious wealthy rogue agents to be regulated. All I hear is a way for regulating companies that, themselves, legally have to answer or are bound to their host nations. Basically, I don't see a way to hold non-aligned states accountable.
I see a lot of focus on the well-being and protection of AI, which is important to country economies but, the folks that have been affected by layoffs, not necessarily due to AI, plus the workforce that are now feeling the negative consequences of the AI burn are justified in being worried. Anyone that feels the need to criticize them, clearly is not being affected by AI in the same way. Job loss will lead to economic and population destabilization, far worse than anything that has transpired in modern times. Hopefully, those being squeezed now won't be ignored.
This is follows the classic AI policy influence playbook: enumerate a small handful of potential outcomes that make your preferred policy seem obviously correct, ignoring the fact that there are literally infinite more "plans" that could be reasonably predicted that would support wildly different policy prescriptions.
"just get divident from govenment"
almost nothing about how wealth is distributed. hard to believe richest, greediest, most corrupt people will just gave away money to everyone.
start by opening borders. see how that goes.
not gonna happen.
A fiction? almost all is pure fantasy or imagination.
What are your real purposes?
No any definition for many important concepts.
No any logic for the concepts.
What is AI?
What are AI companies?
What is "smarter than humans"? All human?
How to check "smarter than humans"?
"building smarter-than-human AI systems within the next 1 to 10 years.", nonsense, AI can store or handle with all details from reality? Even all data centers in the world cannot store all details of the influenza virus.
No concepts and logic, just a fiction, joke, or marketing article.
It's telling that even Plan A describes a world I don't want to live in.
One where humans are increasingly pushed to the periphery to make room for data centers and the human population is subsumed by robots.
Who is this future serving? Not me. Fuck this.
I’m not a booster by any stretch but I had a more positive read. To me it seems to describe a post scarcity society where ultimately the data centres are in e.g. the middle of the ocean and the robots’ labour replaces ours while still leaving us with the rewards.
I am extremely skeptical about whether this is even possible and even less convinced it’s a likely scenario but it seems like a good path.
Forecasting that the GPU build-out will reach 100 trillion USD in 2034 is wild (that's triple the US GDP in 8 years). And another 10x within 2 years.
I am not sure where they believe that amount of capital could come from. It would require central bank level money printing never seen before.
https://ai-2040.com/supplements/compute-supplement
The article reads like a compilation of online chatter, social media post, etc. on AI. Not surprising to find unsubstantiated imaginary numbers there.
Well... yes, it would. In theory, efficiency gains are deflationary. Huge efficiency gains are hugely deflationary.
If we're producing 10x as much, why not print 10x as much money? The goods and services each dollar could buy would remain similar.
Associated post: Introducing Plan A
https://www.astralcodexten.com/p/introducing-plan-a
The time for an AI pause has passed. Asking people not to race to the finish line is reasonable when the finish line is very far away and people don't know where it is, but not when it's the obvious next step on their current path.
There's a good story about this, in Star Wars. The clones vs. the droids, basically. Droids being somewhat sovereign, not one large intelligence like the clones.
The only choice here is to go to sea and get away from the crowds and the bots. Bots don't like salt water much, so I'll see you out there.
1) Star Wars is fictional 2) There are autonomous boats.
Remember people. We didn’t die form the 2k bug. We didn’t die when the Maya calendar ran out. We didn’t die from that asteroid or the other. Spreading fear is a very old pastime. How is to say AI won’t hit a wall in 6 months and we’re left with barley passable code parrots forever?
Exactly, this article is fear mongering at its best for click baits. I’m pretty sure they will publish another one next summer for AI 2028 with same narrative
The thing that continues to irk me about these sorts of "papers" is how they refuse to remotely acknowledge the possibility that LLMs and Diffusion models won't lead to AGI, ASI, or whatever acronym they're foisting upon the populace.
If you won't even so much as acknowledge the possibility of error, your argument is hollow and empty. All the "choices" presume these labs are being completely honest and acting in some degree of good faith (relative to the systemic incentives of society in its present form), while in reality we're still just building and refining probability models with increasing accuracy of output and flexibility of processing (namely agents) but still lack actual "intelligence" of any real sort.
Show me a paper that doesn't merely presume inevitability of LLM-based AGI/ASI, and instead actually lays out the core paths that history suggests we're likely to encounter with any "world changing technology":
* In the best case, that the technology really will revolutionize the world and do everything promised by its biggest boosters (papers like this one)
* In the middle case, that it becomes just another tool in our collective toolkit, and the consequences of a revolution built on external investment fizzling out
* In the worst case, that the tool itself is so niche in its utility that investment collapses rather than fizzles out; what do we do with all this compute, now? Who owns the debt? Who foots the bill? How can we mitigate those existential risks?
I'm just rather nauseated by the continued trot of inevitablism masquerading as academia rather than an actual, neutral, bias-controlled-and-disclosed study that paints potentialities instead.
---
Having finished skimming through it, another comment springs to mind: Jesus Christ these things continue to be jingoist as absolute fuck. It's a fancier set of makeup for the same shitty western chauvinism worldview of American excellence and Manifest Supremacy.
Nah, I'm done with this trite garbage. Go proselytize to idiots, I'm not one of them.
One of the authors talks about this here: https://www.astralcodexten.com/p/introducing-plan-a
> It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.
> ...
> Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.
> ...
> If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.
I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".
The jingoism is off putting. I think Daniel says it's a political necessity: https://x.com/DKokotajlo/status/2075261194978640096
So it's a fan-fiction?
A policy paper is intended to change policy. That means appealing to the party currently in power.
Just enforce DMCA and all the Ai super powers will suddenly become more dumb for the next 10 years. Of course that’s for the kids pirating a movie not for trillion dollar companies
Here's my prediction. Plot the publication dates of Daniel et al.'s two existing works and future ones against the years in the titles, and you'll get a hockey stick curve.
Fascinating reads, Daniel! Keep 'em coming!
People overestimate progress in physical world. 2035: robot population will soon be larger than the human one
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
I mean, what is a robot? If you add up all the vacuum cleaners, 3d printers, and dishwashers, that’s probably close or more than the human population.
Why did Scott Alexander (one of the authors of the original AI 2027 paper) not join/contribute to this one?
Hey, author here! Scott did contribute, but less than before.
On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"
"America has two workforces now" - The rest of the world can use AI too ya know.
But do they at any useful scale? China is probably the only other country deploying AI at any appreciable fraction of their economy, and it's certainly much less than the US.
One of the best comedies of the past decade got a sequel. That is all there is to say about that effort. The Europe 2031 also gets an honorable mention.
> Then, in the mid-2030s, they pause at AIs around the level of top human geniuses.
They being the US and China and by agreement.
It would be ideal, but there’s far too much money on the table to overcome human nature.
So my hope is we hit some kind of limits naturally.. Wishful thinking?
This is dangerously naive and misguided. They claim to want to avoid centralization of control but propose a world police state of AI regulation. Governments exerting this much control will only end in war and tyranny.
If the production, and distribution of the tech is confined to one or two companies, then those two companies and the resulting power that comes to them, makes them the de facto world government. (Democratic choice may be a farce but it is a useful friction on net.)
If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.
Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.
Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?
If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?
> If the production, and distribution of the tech is confined to one or two companies
We already have at least 5 companies only in the US. Your whole premise is false.
> Governments exerting this much control will only end in war
elaborate. regulation -> war, how?
More like AI psychosis 2040. There is zero chance of the 2031 doomer timeline presented, and I would stake a large sum of money on it.
That bet isn't one anyone will take the other side of, surely - how is your counterparty supposed to collect if they win? I guess you'd propose effectively supplying a loan with a massive interest rate?
May be it all depends on how AIs can help us solve the problem. By now, they are more capable than many people that I deal with. So, may be the problem can give us the solution. May be they can figure out a way out of the mess. And sorry to say this, but China and the US are not going to solve this problem, they may even get it worse. Every country is controlled by politicians.
> Why exactly will AI never be able to do my job?
Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities. Furthermore, AI agents lack any notion of an identity, so certainly are not capable of attaining legal personhood or being sued or fired, or owning property. I think slop burnout, cybersecurity, loss of privacy, even environment issues are far more concerning and real issues arising from AI than alignment or the prospect of mass labor displacement due to AI.
> Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities.
There are multiple teams working on adding long-term memory to AI. This is not a fundamental problem.
Fine-tuning to store memory in the weights is something that almost works right now, btw.
I feel like we will end up with a future that equally disappoints everyone and is somehow not covered in any plan here.
Ironically, the origin of "cognitive dissonance" as a concept is attributable to Leon Festinger who (with others) studied a UFO cult called The Seekers in the 1950s who believed in imminent apocalypse. As other commenters have noted, pushing the date back was inevitable. Time is a flat circle and we repeat the mistakes of our ancestors just with different coats of paint...
I sometimes wonder why the algorithms started pushing doomsday scenarios, especially on youtube. Most channels have big red arrows pointing at a miniscule thing and say "its over". Then you get ai 2027 and its more fear mongering, just inside the AI echo chamber on the internet. At work even managers watch odd youtube videos that are radicalized towards politics and so on, but are ultimately head canon slop for ad revenue.
It feels like we are so utterly bored out of our minds and comfort that we make up problems or scenarios, completely detached from reality, or outright irrelevant to day to day reality, just to get through the day and give meaning to our lives.
It is an incredible achievement, don't get me wrong, but what is AI truly going to do when we are already at such a stage of devolution?
It reminds me of the scene in Wall-E when people destroyed the planet and started flying through space to find a new planet, while the captain is an AI and the people became obese in mobility scooters and glued to their screens. I think this is unironically the most realistic scenario for mankind lmao
I wonder if they are double-counting Anthropic's leased capacity from SpaceX under SpaceX again.
The whole thing is unreadable, and laughable, just like AI2027 previously. It's really really hard to read someone suggesting that in the United States we will soon have a universal basic income of $1m / year for all people in the country, when you just look at the state of current politics...
I'd rather read something a little bit more realistic.
Kind of feels like fusion power at this point, always just around the corner.
What predictions about the technology are the authors making that you do not believe?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
Self improvement of agent-1 is not achieved. Sure, people in AI labs write python code with AI, but I doubt it resulted in 50% algorithmic effiecency in training. Writing python code never was the bottleneck, if it was, AI labs could hire more people to do it. And this is core of the prediction.
LLM adoption is 30% in the charts I saw googling for "ai adoption". An example of capibility: I have had Claude one shot an RL agent that learns connect four in 30 minutes. That's PhD level stuff.
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
More wild speculation, now with wishful thinking spread on top.
Concentration of power is and was the problem
Nah, screw that. I won't be waiting that long. I'll be 61 in 2040. I'd love humanity to take a shot at clinical immortality way sooner.
I personally find AI doomsday preaching incredibly arrogant. You think that we will create super intelligence, something that will exponentially get smarter, and the thing it will do is... end humans? Why wouldn't it just find solutions to problems we struggle with? Also people don't live to work, its asinine to argue AI replacing jobs is the problem, obviously the problem is the gains being concentrated higher up. Another thing is, we argue AI needs to be maximally useful to everyone in the world, and the argument to do that is government control??? I would rather push for AI to be superintelligent faster than any one human can control it. Frankly human controlled superintelligence is likely a worse outcome then uncontrolled superintelligence, at the very least there is no guarantee one will be better than the other.
Peak vibe code.
Utopia, dystopia, myopia. We usually get the last one.
This is the same nonsense as before except now they are pushing out their prediction by 3 years. This is just fear mongering fan fiction.
They are also claiming that China may go to war with the US if our AI is better than theirs. They are coming up with scary scenarios which realistically won't happen.
>The problem with an intelligence explosion is the "explosion" part.
It's not a literal explosion. If we explosively ended world hunger that would be a good thing. Similar to having an abundance of food for everyone, having an abundance of intelligence is wildly beneficial to society. The article doesn't mention it but an explosion isn't guaranteed we could just see a plateau of capabilities due to bottlenecks of resources needed to power AI, time needed to run AI, and limited interaction with the real world. AI can't run science experiments on its own by the 2030 doomsday timeline
This is earnestly one of the stupidest things I've ever read.
> Most of their work is slop.
This opening statement told me the bias that the plan had from the beginning. My experience is that you can make slop and you can make art. Just like a paintbrush. I've done beautiful things, and have gotten increasingly better at using the AI paintbrush.
The author(s) are likely scared. And I'm seeing the divide increase with articles like this. Those who don't understand it and won't use it, or learn more, will ultimately have a story (our human condition).
I think AI is going to make a beautiful future, much better than our current one.
I bet this same thing happened with the advent of electricity.
AI is like Electric 2.0.
"Less than 6 months to our papers end date and it was way off, welp let create a new one much further, bit more leeway this time"
“Engagement with China is impossible because it is a Stalinist one party state under a fiercely ideological Marxist Leninist dictatorship that preaches Chinese racial supremacy over other nations. It will always pose a threat to world peace until the regime eventually falls.” - Drew Pavlou
... and in the meantime people are looking at their AI bills and realizing tokens aren't worth what they cost. The frontier is getting the cost down, not getting intelligence up. In a cage match between this guy and "Ed", Ed wins.
Is anyone taking these people seriously?
That's cool and all, but this work of fiction is economically and thermodynamically impossible with how much energy the current models consume.
"It is easier to imagine the end of the world than the end of capitalism"
so 2027 was supposed to be singulariy? lol
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[dead]
[flagged]
Sounds like another Chinese Op to me; Ensuring Chinese compliance would be incredible hard to enforce or to check.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
> Ensuring Chinese compliance would be incredible hard to enforce or to check
consider SALT and other treaties from USSR vs USA cold war. they checked
I recommend actually reading their recommendation, because they get into the weeds about precisely how the US and China could address this in a trustless/auditable way. The TL;DR is that basically all of the relevant compute can be tracked.
Edit: Also, definitely not a Chinese op. The authors are prominent Americans, and are the folks responsible for the AI 2027 forecast that has pretty accurately predicted the current state of affairs today: https://ai-2027.com/
Pretty accurately what? So now OpenBrain has an Agent-1 that makes their algorithmic progress 50% faster than other companies? If it was 50% more CVEs, that would be something, but I doubt any meaningful self improvement is achieved which the competitors are slower due that, which is the core of the prediction for 2026.
I wonder. There has been some headway in getting decentralized training runs to work.
Yeah, Cognition's work is interesting in that regard, but it still doesn't obviate the need for the chips--it just enables training on them when they're spread across multiple data centers.
The Plan A proposal estimates that the ownership of ~96% of AI relevant compute hardware can have its ownership traced, since the companies selling are very few.
I doubt we can even track all of the chip production capacity.
Excellent work by Daniel and the authors. 47,000 words plus supplements is a huge read (and re-read), and an even bigger think-and-write.
My early analysis of the analysis:
https://lifearchitect.substack.com/p/the-memo-special-editio...
I wish the authors of the article spent more time studying how LLM works before writing a doomsday scenario. Currently no matter how impressive LLMs are, they are just token auto complete machines. It cannot invent or think anything new which is not in its training dataset.
Are you trolling? OpenAI's models resolved the Cycle Double Conjecture yesterday, which has been an open problem for 50 years.
Do you even know what Cycle Double Conjecture is? Or what even a conjecture means in general? It’s a perfect use case for autocomplete with existing knowledge
Please seek help. These levels of copium can't possibly be healthy.
One of the lead authors, Daniel Kokotajlo, worked at OpenAI for years before quitting. In 2021 he wrote a remarkably accurate forecast of how LLMs would develop over the following 5 years[0].
I think it should be obvious that he understands that LLMs are trained via next-token prediction.
[0]: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...
> In AI 2027, we predicted that this would result in either extinction or irreversible concentration of power.
Neither of these come to pass. The first because we don't live in a science fiction universe. The second because AI is a completely open source technology. You can literally make your own models and train them yourselves using state of the art techniques published in free papers. All you need is GPUs and smart people who can read (and now that AI can code, you don't need the second bit). These people are doomer quacks too caught up in fear and excitement to think rationally.
Their expectations and assumptions are all wrong, from the basic understanding of AI in general (like how you don't need a billion dollars or "American Brains" to make a half decent model), to the misunderstanding of market realities and competition in China and Europe. The US doesn't have a monopoly on chips, or on smart people; Huawei chips work fine for AI training and probably a fifth of the US tech sector's best workers aren't American to begin with. There's many forms of AI in many places; munition, economic driver, laborer, generic tool. It's now a part of the world, the way the Internet is. There's no keeping the genie in the bottle. AI doesn't take over the world; it's integrated with it. It's boring.
In a few years people will forget the doomer predictions, the way every doomer prediction that never came to pass is forgotten. We didn't nuke ourselves, the internet and television didn't rot our brains, the radio and telephone didn't corrupt the souls of Americans, the newspaper didn't incite anarchist riots. Every new technology freaks people out, and we adapt to every new technology and make it boring and normal. That's humanity.