As somebody who does ML research, LLMs certainly make ML research faster. Maybe even 10x in some cases, as the author suggests.
But that’s just the parts of ML research that take place within your compute environment, namely experiments on model architecture, training strategy, UI/UX, etc. The algorithms and software.
Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
It just means that software engineering, data analysis, and other computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
I don’t think it’s a stretch to say that we’ll have reached max LLM capability within a couple of years (what the author refers to as “super intelligence”). But it is a huge logical leap to say this:
> If we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict.
First of all, the weapons necessary to overpower human civilization exist. They aren’t used as such due to geopolitics and humans’ basic survival instinct. Second, developing novel weapons requires physical tests, which are decidedly not accelerated by LLMs.
AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human. But we’ve made it pretty far with some pretty unstable humans in command of planet-killer weapons. I’m not overly concerned.
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.
The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.
I don't present any support for or evidence of:
- LLMs gaining intelligence by whatever means
- Extreme intelligence being possible
- Extreme intelligence being feasible
But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
That's not what I said, and it's the least sophisticated way I saw to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."
> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.
We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.
Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.
That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.
By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?
The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
> Especially if the AI itself is being used to harden their defences.
Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)
> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.
I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity of mankind that dooms us. Even Qwen could kill us at that point :)
>By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?
The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
You seem to be moving the goalposts a little, but I don't see why "rogue AI" is less credible after the LLMs. It's obviously more. And we're just at the start of constantly running agent style harnesses.
>To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented
And yet, in many cases, those on the offense have a big advantage that any defense lacks. Especially if there's no concern for one's own life, which whether it's an AI or a human AI user removed from the field via his use of AI to delegate doesn't need to have (or they're just mad/fanatics and don't have in general).
>Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
> Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or like a massive AI datacenter boom of questionable long-term profitability? Maybe the end goal isn’t profit but large amounts of compute?
>So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
And why do you need perfect prediction and/or perfect control? "Good enough for the results you want" prediction and control sounds perfectly fine.
We have had cult leaders that had neither perfect prediction or control, and still convicted hundreds of thoysands of people to live as they say, and a decent chunk to even suicide on command!
> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|
"You can't know what the impact of your action will be if it doesn't change much"
> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large
"You can't necessarily change much"
> (or that f_actual(X) saturates its codomain)
"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.
> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
> "You can't know what the impact of your action will be if it doesn't change much"
Incorrect translation. No "if" clause need apply.
> "People can't do arbitrary things"
Also incorrect. f_actual(X) is a product of both "how much people can do" and "how much you can control/persuade them." I would model each of these with their own precise terms and relate them back to f_actual(X), and we could continue like this indefinitely, but from now on I'll just give you my reply in English: Not only does The Atomic Boy Scout not prove that a random person can bring about mass destruction (collecting random radiological material and contaminating the local area is not developing a nuke, nor is it even an effective dirty bomb, because it was neither very dirty nor a bomb; the safeguards were not lack of knowledge, either), but it still remains to be demonstrated that individuals will be radicalized by AI just by virtue of the AI being, like, super-duper smart.
> "Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
What domain of input do you propose, then? LLMs as rational agents have no agency but for what tokens they generate. What we hook them up to, and what they could plausibly do with what we hook them up to, has been left unsaid.
Denying the premise, which is that an intelligent being is one who can predict the outcome of actions with sufficient precision to influence the future, makes debate pointless.
> even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
> What makes you think that persuasion isn't capped at some lower level than world-ending?
People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).
Why do you think it would be hard to convince any of these people to do something really stupid?
In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.
> People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler)
If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.
If you are saying the Allies were persuaded that Hitler was evil and must die, I think that may be closer to the truth since they knew the cost of invading Germany would be astronomical, potentially costing their own destruction, and they still stopped only when Hitler was taken, dead or alive. Some serious persuasion was needed for that.
>If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.
That's a bizarre take.
Of course tens fmillions needed to be persuaded, and the Nazis did a lot of propaganda, even calling that openly, false flag operations, and similar things, and of course there was decades of pamphlets, speeches etc before the Nazis even existed. Antisemitism didn't just grow on trees.
You find it bizarre that people are born into a culture and accept much of it without the need to be persuaded. Have you been to a country with a very different culture than yours? That normally opens our eyes to just how much we take for granted is just cultural, and other people would strongly disagree with even your deepest held belief.
SNobody said otherwise, but that's not applicable here.
We're not talking about general customs or viewpoints a people has as part of their culture.
We're talking about a specific political program developed and propagandized in the span of slightly over a decade. Nazism wasn't merely "the general culture" of Germany since always, nor was "kill all Jews".
And yet AI can't write decent landing page copy, or even a convincing cover letter. Seeing the obtuse "reasoning" from Claude daily, I'm very un-worried about AI being a world-ending persuasion engine.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.
Agreed. Like lots of this AI speculation appears to require the belief that AI will figure out how to build supervised training environments without more data.
I think this is very unlikely, but world changing if true.
Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?
I'd be delighted if someone can provide some definitions for these terms.
Uh this seems exactly backwards - reinforcement learning with human feedback teaches only the power of persuasion. The only way to get rewarded is to persuade a human to reward you. Why do you think the models are all so damn sycophantic?
What is the measurement? I don't think most people come back to a conversation and say "gee you really convinced me to do X, here's the proof that I did it" and even if they did that data would still be very messy and require verification.
Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.
Only the labs knows the truth, but the question as to how many Americans used ChatGPT/Grok/Claude to help them decide how to vote in the most recent election is hopefully going to be something that historians will be able to unearth later.
I doubt the number is low. Knowing everything I know about llms, I still fall into the same groove as I would if I was talking to a buddy online. That part is not far fetched.
I wonder if we’re approaching a cultural moment where people in general are doing exactly what they did when Google first became ubiquitous—search to confirm their biases—but now with LLMs instead of Google. But where Google would happily feed them some link to potential mis/disinformation, the AI is more likely to say “no, that’s not accurate, and here’s why…”
I'm a physicist, and I will do my own calculations before I buy some solar/batteries in the near future, but I've been using ChatGPT and Claude to come up with potential candidates and ideas/scenarios. There's approximately 0% chance that they are not advertising what their owners are being paid for.
I went to a talk by an author of one of these papers (David Rand). IIRC the format of the paper's experiment was collecting people with conspiracy theory beliefs and having LLMs with different strategies try to reduce their confidence in the conspiracy theory.
The best strategy, perhaps reassuringly, was tailoring facts to the person's worldview. I don't mean massaging facts, I mean that if the person thinks 9/11 was an inside job because of something about melting steel, then the LLM will present some sourced facts about how that actually works. And then it will do the same thing for all six of the sub-beliefs that support their overall belief. This was key to the strategy: the LLM has enough breadth and recall to exactly and patiently rebut most mistakes a person is making without making them defensive.
This strategy handily beat other things like flattery, selectively lying, etc. So while LLMs can clearly persuade, the evidence suggests it is easiest for them to persuade belief in something that is actually true.
This was in 2024 or 2025. Things might be different now.
It feels like I don’t understand your definition of “persuade”, because the notion that people cannot be persuaded of things via reports and flashy animations is to me, absurd.
For what value of N do you think there exists a string of N characters I could have responded to your comment with that would have infected you with a stably self-replicating and capability-augmenting memetic virus with human extinction as its latent goal?
If not you, in this situation, with N characters delivered via a HackerNews comment, then who, in what situation, with what media and how much of it?
I think that is a fascinating question and I would like to advance the provocative notion that N is much smaller than you think, maybe <256. Consider that 128 bits is a cryptographically secure key, and English text has an entropy of around half a bit to a bit per character. 256 characters is sufficient to encode sentences that have never been seen or imagined before, totally unguessable sentences. Who knows what's in that probability space?
You're not spelling out a scenario we can meaningfully dissect.
Probably you will agree, the leverage of a blackmail attack is not bottlenecked by the persuasiveness of the accompanying threat letter. It is bottlenecked by the existence of damning facts they would rather keep hidden, and the ability to discover those damning facts. I'll grant that automated discoverability of those damning facts is augmented by superintelligence; they can efficiently sweep the open internet in the blink of an eye, and sure, I'll concede, hack into most accounts at will. But that is still a small part of the process by which blackmail material is actually obtained in the real world.
> LLMs do not have to convince 1 specific person, 1 random person is sufficient
No, not one random person either, but many people in positions of great influence. There already exists blackmail and other corrupting influences on such persons in voluminous quantities. If the US targets the entire Russian admin, or vice versa, with AI-enabled blackmail discovery techniques, it will not be an augmentation of their preexisting operations to discover blackmail on each other in excess of the augmentation provided to software engineering generally by AI.
You did not provide an argument. You avoided answering my direct questions, provided a sarcastic 4-word reply, and so I extrapolated a reasonable rebuttal in its place, and replied to that rebuttal.
I should just have said: Yes, I have heard of blackmail. I remain unconvinced AI will cause human extinction.
Ah yes, the "just do what a smarter person than you would do in this situation" argument. If I had that power, would I be wasting it shitposting here? We know humans have infectible little minds. At their worst, humans are just stoichastic parrots. Throw something obviously AI generated at a stoichastic human-parrot and you'll inevitably get the answer "Why should I read something if you didn't bother to write it?" UBI as a memetic virus comes close tho.
Personally, I've been duped by a good ad into buying something I totally didn't want or need so I admit the possibility that the exact right tailored argument to me at the right time and place would get me to do something I'd regret later. Fortunately I'm not in charge of anything serious/am smart enough to not install software for an interview on my work laptop while working at Binance.
I don't know that there exists some N for which that is true, because people are different, so what would convince me to do something isn't the same aa needed to convince someone else to do that something.
Do we have reason to believe they would be successful at other extreme feats of persuasion such as persuading a financial institution to transfer all of its holdings to an account controlled by the AI or persuading the Pope to renounce God?
Except that Putin doesn't do what you listed, he is not smart enough to do it. And this is an unnecessary complication. He just pays people and they do the job. The more money, the more willing people are to commit crimes.
IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.
We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.
We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).
So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
That feels like a bait and switch slyly introducing a whole new dimension. The discussion started with establishing that LLMs are extremely helpful with AI research. What is the evidence to justify rapidly improving persuasion skills? Even the scenario you've outlined hinges on a human being stupid enough to take LLM advice wholesale -- and every conversation that hypes up AI takes us one step closer to a world where some susceptible human feels amenable to that.
No bait-and-switch here. The discussion started with the idea that LLMs are helpful at improving LLMs. The goal of improving an LLM is to make the LLM more "intelligent" (whatever humans think that means). If they're helpful at that, they have to keep becoming more "intelligent".
I didn't present any evidence that more-intelligent means more-persuasive. If you don't think it does, then there is nothing to think about in the point I made, as no matter how "smart" the models get they will likely remain subject to human wills.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?
I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)
Not necessarily. If you haven't experienced an AI doing something you not only didn't want, but had expressly asked it not to do, you haven't used AI much.
Yes, basically a vastly more difficult and dangerous version of not letting young children play with matches, which will hopefully teach us many object lessons about safety and security long before it acquires launch codes or the means to use them.
A fire can spread itself. That doesn't mean it's alive or thinking.
And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.
Besides the fact that the subjective criteria needed classify something as "alive" or "thinking" is irrelevant to the point you are replying to, fire doesn't pass the Turing test. That analogy is extremely weak.
The Turing test has been passed for half a century already, and we've been moving goal posts ever since.
The point I was trying to make was this: we've been dealing with autonomous systems that sustain and spread themselves and "act on their own" without our intervention since time immemorial, ranging from lifeless chemical reactions to other humans. Now, the discussion about where exactly AI lies on that spectrum isn't unimportant: suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves. The ignorance of which is a "fun" bit of cognitive dissonance on the pro-AI side imo (and in general we could have fun with analysing how much the fear of robot/AI uprisings in classical fiction may just be a projection of upper-class fears of an automated worker's class uprising, which would explain a lot of why it's sd US-specific. But that's probably too radical for this particular website).
The point is: it's a lot less important than the fact that no matter what the least-nebulous answer to that question may be, ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices. And most of the time this discussion about autonomous agency is a wielded as a distraction from that.
> ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that.
That's true today. I don't think that will always remain true into the future.
Right now there's a huge focus on recursive self-improvement. I think another big threshold that's equally as important to keep an eye on and care about is self-sustenance. Part of sustaining yourself often puts you in a cooperative relationship with some and an adversarial relationship with others.
If we want to make different choices today, the best thing we can do is grok what the consequences could be on a deeper level before we cross those thresholds.
While I agree about the importance in this regard:
> suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves
On this, I note that despite humanity being aware of greenhouse gasses for over a century, we only took meaningful steps to reduce our creation of them in the last decade. Our ability to "make different choices" collectively is far more limited than many of us individually would like:
> ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices.
And on this, I must firmly disagree:
> The Turing test has been passed for half a century already, and we've been moving goal posts ever since.
No, it hasn't. From that claim, you're obviously talking about ELIZA, and if you think the anecdotes about that count as passing the Turing test, I suggest actually reading what Turing wrote rather than second-hand accounts:
The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B thus:
C: Will X please tell me the length of his or her hair?
Now suppose X is actually A, then A must answer. It is A's object in the game to try and cause C to make the wrong identification. His answer might therefore be:
"My hair is shingled, and the longest strands are about nine inches long."
In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as "I am the woman, don't listen to him!" to her answers, but it will avail nothing as the man can make similar remarks.
We now ask the question, "What will happen when a machine takes the part of A in this game?" Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"
and yes, he was over-optimistic on timelines (I assume the 109 is 10^9, this PDF seems to have a few transcription errors):
I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.
and over-optimistic about if humans would accept this kind of test as a sign of thinking, given the widespread dismissal of LLMs as "stochastic parrots" and similar:
What would Professor Jefferson say if the sonnet-writing machine was able to answer like this in the viva voce? I do not know whether he would regard the machine as "merely artificially signalling" these answers, but if the answers were as satisfactory and sustained as in the above passage I do not think he would describe it as "an easy contrivance." This phrase is, I think, intended to cover such devices as the inclusion in the machine of a record of someone reading a sonnet, with appropriate switching to turn it on from time to time.
Fires are not forming plans to act towards goals. I wouldn’t technically call what AIs are doing thinking in the same way humans think, but it’s analogous, and I think it’s reasonable to consider it a kind of intentional behaviour.
Damaging fires are often unintended consequences, and it’s often the same with the damage caused by rogue AI. That’s the alignment problem.
One thing I wonder about is whether sometimes telling an AI not to do something can put the possibility of doing that thing into its behavioural space. There’s an effect well known to many teachers that if you tell children not to do something, it can increase the chance they will do it. A better approach is to phrase it positively. Instead of “don’t run in the hall” say “please walk in the hall”. I wonder if a similar effect can show up in responses to prompts.
> A fire can spread itself. That doesn't mean it's alive or thinking.
That's really not an important distinction. I don't care whether Claude is alive or thinking when I get it to build web apps for me. It has an impact on my life and the world regardless.
Imagine a world where you did RL on fires to maximise their rewards on avoiding being putting out.
It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.
While true, we're already observing humans just straight asking LLMs to direct them, and also to take their money and spend it on stuff.
Plus every so often we see power-seeking and self-preservation behaviours in LLMs, so it's not like we have zero reason to expect such granted power to be spent on such things:
I think we have very good reason to fear big accidents. But some AI algo going rogue isn't exercising real societal power. Power-seeking and self-preservation is nothing special. We all do that. That doesn't mean every group of intelligent people with a bit of money is always on the cusp of world domination.
Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.
Every random group of intelligent humans who is motivated to seek world domination is constantly in a fight with all the other groups of intelligent humans who are also motivated to seek world domination. They fight as peers.
Despite this, a minor technological and social advantage allowed one small sheep-filled rock in the north-east corner of the Atlantic to hold 25% of world GDP, population, and land area (though not all at the same time), and to hold on to >20% of world GDP for about 70 years.
For better and for worse AI, as it currently exists, is not shaped like our minds. Better for us, because machine learning needs a lot of examples, so if we invent a different behaviour or create a different environment, a machine would struggle to cope with it, this is also why they needed to read basically the entire public internet to get good; worse for us, because the hardware it runs on comes with an absurd speed advantage, which is why they are able to read the entire public internet and by extension when they got good they got good at basically every subject at the same time.
And:
> Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.
Great, but so what? We have lawyers doing this, too. Trump's tariffs on the island occupied solely by penguins looked suspiciously like it came out of an LLM (even despite them being worse then than now).
And LLMs are demonstrably more persuasive than human writers, so it's not even particularly unbelievable at this point (even if it ought to be, c.f. chatbot psychosis) that an LLM will simply convince people to give it the right to form a government and wield such powers.
Agree. We already have a planet with eight billion human level intelligent beings on it. Despite some close calls in the past, none of them have yet managed to wipe out humanity.
I think there are a couple of interesting scenarios where humans essentially “hand over the reins” to AI.
1. A leader rises to power who has instructed an AI “I want to control the world, tell me what to do”, and has some kind of real-time capable reasoning engine in their ear/head. Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years. However, this still requires a human to make the initial prompt and carry things out. And in such a scenario, there would be others with the same technology to counterbalance. I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.
2. An entire society hands over the reins of governance to AI. Basically some country decides that AI has advanced to the point that it makes better decisions than any human / group of humans, and allows it to govern their country with an objective of “take over the world”. This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.
If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.
> Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years.
Would it require so much? Cochlear implants exist if you care about HMI, and compute is already doing "reasoning"[0] faster than humans do snap judgements. Actually winning in an adversarial environment against the best human political minds (let alone economic or military) will be harder, but I would be (pleasantly!) surprised if it takes 10 years, let alone 20.
> I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.
I hope a sensible AI would not. However, Musk's LLM called itself "MechaHitler", which suggests his training process does not avoid such things. If his in particular becomes dominantly competent, he has also spoken of wanting more Optimus robots than there are humans[1], has called it a "robot army" (and his justification for even more Tesla stock was to keep influence over said army[2]), and he has spoken of factories on the moon churning out space-based data centres by the petawatt which is enough that if placed in LEO it could have about the same effect on Earth's climate as all anthropomorphic greenhouse gasses currently in our atmosphere (and that's even if not weaponised, that's just simply by having that much stuff that close).
> This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.
I don't. First, because AI being "better" doesn't make it infallible; and second, an argument shaped like this has historic echoes with actual nukes and the cold war: if the AI is competent to take over conventionally, and everyone knows this, then every other nation with nukes will use them, therefore the AI should use a first strike. (Note: the argument does not need to be perfect or flawless in reality, it only needs to be the kind of thing which humans have found convincing, and far too many humans have argued their side would win a nuclear exchange).
> If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.
This is my best case scenario. Some people also object to this, they like their careers and fear what retirement will do to their sense of self-worth, but I suspect this can become an opportunity for flourishing in other ways if, and only if, it is handled correctly.
[0] something something swimming submarines, I don't care about using anthropomorphic language myself, but there's always someone ready to complain if anthropomorphic language doesn't get scare-quotes
[1] "And, and then there'll be obviously robots in industry making stuff. And so I mean, I think the ratio of humanoid robots to humans will probably be at least two to one, something like that. One to one for sure. So which means like somewhere on the order of 10 billion humanoid robots, maybe, maybe 20 or 30. And so then it's like, okay, well let's say, you know, you kind of make, let's say the build rate is, I think the build rate will be probably something ultimately like a billion a year." - https://www.elonmuskarchive.org/video/tesla-shareholder-meet...
[2] “If we build this robot army, do I have at least a strong influence over this robot army?” he told investors. “Not control, but a strong influence… I don’t feel comfortable building that robot army unless I have a strong influence.” - https://futurism.com/robots-and-machines/elon-musk-remarks-r...
Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.
> The math results that have been proven have also "just" combined existing ideas or found counterexamples
The Exponential Time Hypothesis one used a novel algorithm though. And the approach was very surprising for experts[1].
> “Some parts [of OpenAI’s proof] are, maybe you could say routine, but then yes, there are two or three alien-looking parts, and it’s exactly these parts that I saw that made me think, wow, this is something that the whole community needs to work on together,” Williams told TechCrunch.
> I just don't see a clear path to super intelligence with the current paradigm.
And therefore there is no possible path, right?
Did you see a path to ChatGPT before it happened? Did you see a path to Stable Diffusion before it happened? Did you see a path to world models before they happened? Did you see a path to Navier-Stokes before it happened? Did you see a path to the HuggingFace incident before it happened?
How much of this would you have predicted vs. how much of this was novel and a surprise to you when it happened?
What? Why? You could build a system today that does this. Tell me why you couldn't?
Right now we have harnesses with a /goal command. In DeepSeek Harness everything is a plugin, so that must include the /goal command. So we can just swap it out for a version of the plugin that allows for changing the goal. Could probably get it to code that in less than 15 minutes.
Then post train the model in that harness with an awareness it can change the goal and giving it some training task that requires changing the goal appropriately to maximise the reward.
You're telling me that just straight up wouldn't work?
It won't work unless the AI does something stupid or we mean something different by goals.
If it wants to maximise a reward then that is it's true goal, you can give it a todo list and call it 'goals', but giving it the power to modify that list seems like a bad idea. You want to be very careful giving it any power over its reward structure.
A pathological example is the paperclip maximiser, which will inevitably attempt to turn the entire universe into paperclips and violently prevent anyone from interfering with that goal. Including updating its goal to anything else.
Why focus on that aspect? Like, yes, it needs to be started off by some other entity. That could be a human. That could be another agent.
The point is we're on track to construct such a system that's capable of sustaining itself indefinitely, and the fact that we are means there are entities driving this forward, and those entities often push buttons to start experiments. So, expect that particular button to get pushed at some point.
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
Being deliberately wiped out seems like it's mostly a misdirection away from the damage that's actually happening.
I'm sure some of the AI people high on their own supply believe it. And because the consequences are so negative, we should react as though the supposed experts were sane -- but I think it's a long shot that they are.
The bigger risk is that we end up in a rapidly enshittifying dystopia, treated as a cost center that needs to be optimized, and unable to competitively participate in the economy.
I have a number of successful AI investors in my social circle who are already placing their bets on a poorer and more disenfranchised humanity, with money going away from consumer tech, and towards a mix of B2B tech and policing.
1) Does AI just guide developers to speed up the pretty much the same data-train-test model that we've had? Or does it inspire new, better algorithms that inspire further, better algorithms until you have superintelligence able to run on the hardware of today. The first option doesn't to promise exponential increases whereas the second one seems to. Notably, neither the statistics nor the anecdotes we read here point to the second option.
2) Is super-intelligence something that - in control of people or out of control of people - poses a threat to humanity? That's a huge question but it seems like something deserving thought today. Still, everything indicates the things will unleashed without very consideration despite this, given current momentum. So, IDK, get ready.
> Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
So "all" that's left is training robots on the real world ? We just need more data ? I get it's more expensive to do that but surely with the trillions we're already spending on A.I it should be possible to get there within 5 years - no ?
> computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
Won't they also be less of a bottleneck in the destruction of the world as well? i.e. the power grid, the financial system, industrial control systems, military command and control networks, etc.
Same is true of every other technology ever invented. Even medical advancements wielded by a malevolent power just serve to keep their army healthier. All of the other technologies you listed, in the “wrong hands”, contribute to the destruction of the world.
I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.
It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.
As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".
> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.
?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.
Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
>Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
That's the danger with creating a super intelligence, that you won't understand it. My dog is pretty smart, has good instincts and I'm quite confident he can survive in nature all by himself. But if a human decided to harm him, he doesn't stand a chance.
> That's the danger with creating a super intelligence, that you won't understand it
This argument is sophistry because it assumes the conclusion in its premise. It works backwards from extinction by proposing a hypothetical computer program that definitionally cannot be understood, predicted or controlled.
A substantive argument would propose reasoning that explains why we should believe such a program could be created.
No, that is not how reasoning works. It is perfectly valid reasoning to begin by positing entities that we may not be possible to create, such as supersonic airplanes, coherent light, quantum computers, rockets that can reach escape velocity, black holes, or infinitely long lines that are perfectly straight and infinitely thin. Indeed, you might say that the ability to reason about the consequences of premises that we do not know to be true is the thing that distinguishes reasoning, as such, from simple parroting.
It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved. This allows us to easily dismiss the most profoundly stupid arguments, such as, 'Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug?'
This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.
To be fair, he could probably fuck the human up pretty good if he had a mind to, too. Which may be the most likely reason that a human might decide to harm him: if he bites a couple of children, for example, or if a burglar wants to rob your house.
Seems like bioweapons (assuming "planet killing" means killing all the people, not literally blowing up Earth) are one way. Convincing the creator it would achieve their aim and could be contained when in fact it could not.
AI could invent something like social media. Obviously not social media because that's already been invented, but the next thing that slowly destroys society while seeming to be good.
> probably destroy almost anyone who survived the initial blast and fallout.
The difference between killing almost everyone, and killing everyone, is similar to the difference between almost getting pregnant and getting pregnant. A hundred thousand years later, a world where almost everyone had been killed would be teeming with humans.
> Can you explain how exactly an AI would develop a novel planet-killing weapon
I already answered that question in my previous comment, and the answer is that I am not going to explain how to develop weapons, but the larger point here is that we already can't explain how exactly AIs do innumerable things we see them doing every day, which is why we're having this conversation in the first place.
> I seriously don’t understand this argument
That is a point that I think everyone can agree on: you do not.
I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.
Life in theory could restart with our weapons, but what’s the point with the other scenario.
> I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.
Being scared about novel AI weapons sounds like the people who were scared that turning on the LHC would create a black hole that would suck in the Earth.
The most important difference is that this time it's the people who know the most about the topic who are concerned, and the people who know the least about it who are blasé. (Also, Yann LeCun.) With the LHC it was the other way around.
My demographic claim? It's obviously correct, as anyone with the most casual acquaintance with the question knows. The worried people, like Geoffrey Hinton, Bill Gates, and Dario Amodei, might be wrong, but they certainly aren't ignorant outsiders. (Neither is Yann LeCun, of course, which is why I mentioned him: he's an important exception to the rule.)
How would touching grass help? Do you think the grass is better informed about AI risk than you are?
...that's plausible, actually.
Probably a site called "Hacker News" is not the smartest place for you to be implicitly asserting that thinking about things is bad, or that those nerds who do it are out-of-touch losers. Maybe you want to be on Rugby News instead.
On Hacker News, by contrast, what is valued is thoughtful discussion grounded in the kind of rigorous theoretical knowledge that comes from extended concentration, usually but not always indoors, where it won't rain on the computers.
Bill Gates is too busy trying to save his reputation from his Epstein association to be believed, he's a billionaire concern troll and Amodei is far more worried about his stock price than he is about humanity. Those two are discounted. That doesn't mean we don't have a problem, but they are just hitching their wagons to horse that happens to be running in a direction that is temporarily useful to them.
That's plausible, although I don't think it's actually correct. But there's a pretty extensive set of AI luminaries who are concerned who can't be dismissed so easily.
There are as many 'luminaries' on LeCun's side as there are on Hinton's. Andrew Ng, Pedro Domingos, Schmidhuber, and the list goes on.
Any time it seems to you that there are more people on the side of an issue that you happen to be on yourself, a good rule of thumb is to assume confirmation bias.
The list doesn't go on. I'll give you Ng and Schmidhuber, but the list pretty much ends there. Domingos isn't even a neural networks guy, and he's retired.
I agree that confirmation bias is a good working assumption in cases like this, but in this case we can conclusively reject that assumption. On the "worry about AI risk" side, we have the 33,705 signatories of https://futureoflife.org/open-letter/pause-giant-ai-experime..., starting with Yoshua Bengio, Emad Mostaque, and John Hopfield, one of the few people to have actually won a Nobel for neural-networks work. And the 1386 signatories of https://www.pacingthefrontier.com/, starting with John Schulman,
Jakub Pachocki,
Jared Kaplan,
Shengjia Zhao,
Shane Legg,
Ilya Sutskever,
Mark Chen,
Jasjeet Sekhon,
Dario Amodei, Jack Clark, Anca Dragan, Wojciech Zaremba, Dawn Song, Chris Olah, Laura Weidinger,
Benjamin Mann,
Stephanie Chan,
Julian Schrittwieser,
Summer Yue, and Jan Leike.
They might be wrong, but the demographics of the concern are pretty clear. AI existential risk is not a fringe concern raised by some group of Luddites due to their ignorance about the science, as in the black-hole risk of the LHC. It's primarily AI researchers, and especially the AI researchers who are working most closely with the most successful technologies.
By contrast, to come up with three more luminaries on LeCun's "side", you had to trot out a retired machine-learning professor who used to do research on Markov chains and Bayesian inference.
Rich Sutton, Raj Reddy, Rodney Brooks, Michael I. Jordan, on and on. It's quite a long list, actually.
The people on your latest list on the other hand aren't 'luminaries', aren't all even researchers, in fact, and overwhelmingly have middle-of-the-road opinions. They worry about AI like they worry about social networks, not Terminators. Even Timnit Gebru isn't sweating Skynet. So you've gone from talking about "planet-killer weapons" to citing a self-serving corporate stunt signed by people whose views still don't align with yours.
Jack Clark, by the way, is a journalist. Lots of "tell me you don't know who these people are without telling me you don't know who these people are" energy here.
Also scamming 101 for that matter, you want your mark to feel an immediate threat of overwhelmingly negative consequences so they'll buy whatever you're selling to prevent it. Silicon Valley directly makes tens of billions of USD literally serving scams to the public already, so I think it's fair to take most claims of doom with a pinch of salt.
It's very disappointing to hear BS like this coming from a very intelligent crowd such as HN. How many people in the US do you think know who Epstein was, let alone what is contained in his file? How many in the world? If Bill Gates is worried about his reputation, Microsoft has sealed the deal long time ago.
As for Dario, of course he's worried about the stock price. I would if I were in his shoes. That's literally part of a CEO's job. He is also one who has consistently raised the issue with AI safety. He was OpenAI's head of AI safety after all. Anthropic is considered security risk by the US. Is that "temporary useful"?
I mean yeah, the main bottleneck in ML research used to be fixing python virtualenv hell when every existing torch repo breaks after two months of package updates, plus cuda and other version incompatibilities. That was like 95% of the work. Now it's more like 2% of the work and it's actually possible to get shit done without losing your mind.
"We've made it pretty far with some pretty unstable humans in command of planet-killer weapons. I'm not overly concerned"
Translation: my house is entirely doused in gasoline and there are 2 unstable humans arguing with each other both doused in gasoline in my house. They are both holding matches and threatening each other that they will drop it and kill us all... But you know they have been arguing for a while and nothing bad has happened so I'm not overly concerned.
Inductive reasoning fails catastrophically as a tool for calibrating actual risk when you're looking to avoid black swans.
Every dot in these graphs is composed of multiple benchmarks smooshed together.
Benchmarks are by definition a curated set of things that can be measured and evaluated.
Things that can be measured and evaluated can be (expensively) trained on.
The progress we are observing comes from figuring out how to measure and evaluate particular aspects of intelligence, and then expensively training on feedback loops made of those exact measurements, and synthesized data representing further measurements (literally benchmaxing).
What is not shown in these graphs are the set of things that intelligence can do, which are difficult, expensive or impossible to measure and evaluate.
That set of things is much larger than the set of benchmarks represented in the graph.
Therefore the post is misleading in the sense that the "superintelligence" being targeted is "jagged" to use Karpathy's term, and we infact already have jagged superintelligence (calculators, compilers, Bayesian spam classifiers, etc.)
We are witnessing a profound explosion of much more jagged superintelligence, restricted to the exact domains that can be (expensively) RL'ed on.
1. A key insight of foundation models is that capabilities are correlated. For example, training on code improved general reasoning capabilities: https://arxiv.org/abs/2309.16298
2. A profound capabilities explosion in easily measured domains will still feel very different to today. Like, the moment we come up with a systematic, objective way to verify a task, we become obsolete at it?
Yes, capabilities are correlated but an open question is how correlated, and how much negative transfer (where the model regresses on other benchmarks) will come into play.
Why are frontier models not chess geniuses (amateur club level elos), despite knowing the rules deeply, having a great deal of chess data in the training set, and having general reasoning training in other domains? Similarly, a beginner can beat a frontier model at Magic The Gathering. Why?
It is because chess and MTG are not priority targets in the (expensive) training pipeline. Chess is already solved by lesser neural networks so there is no wow factor, and nobody cares if they suck at MTG.
Mathematics on the other hand, has a huge wow factor, and a large and highly verifiable dataset.
Related: why did OpenAI give up on video?
I agree it's disorienting and scary but I'd phrase it like this: "the moment we come up with a systematic, objective way to verify a task, *and we can generate or find large datasets, and somebody has a reason to pay for the expensive training*, we become obsolete at it."
The economics side of it is an important constraint, currently ignored because the big labs are swimming in money (debt).
The particular problem here is it's not addressing the intelligence spikiness between individual humans. Most humans aren't going to play a competitive game of MTG, a small percentage will rise above the rest and dominate.
Also this is being intelligence blind. Most humans could never write a good python script to play a competitive game of MTG, almost any SOTA model can.
This would be like a bee saying humans aren't intelligent because we can't see in ultra violet, where as we're damn good at making tools that can.
Again, who cares? How does that help humans compete against LLMs in any domain where the LLM will have web search, python, and other tools (which is basically always)?
I don't understand this argument, it’s not like humans and LLMs are engaging in some kind of competition with strict rules where ONLY the raw LLM responses “count” or something.
No. The current frontier models would not beat the best humans at MTG, even with Python scripts and web search. Unlike Chess, there are no existing engines that beat humans.
IMO I think a partial root of why AI writing is so bad is because it's written like code. Like if you really dive into how it constructs things after iterations and ask it to analyze it , without it saying it's like code or without you saying anything about code, you'll notice coding like iterations.
How many models have you tried? They write very differently. I found Gemini Argon 4 to be really good at writing like a human. Simpler models like Haiku seem better than more powerful ones like Opus, though even Opus has improved considerably with 5.5.
DeepSeek is much more roundabout which can mirror certain people’s styles and some people prefer that. So in summary don’t just say LLMs are bad at writing, you need to be more specific: which model, which style, which context, what instructions were given? Claiming they are bad under all combinations is just ignorant.
Oh, absolutely. The often voiced statements about "terrible Ai writing", "LLMs cannot joke" they all seem to be fashionably regurgitated by people who had last time used an LLM for said purpose in 2024 if ever.
Transference from code to reasoning is not surprising, programming languages are formal logical systems and written code is essentially a concretized form of reasoning about a domain and operations in it. Training people in formal logic also improves their general reasoning capabilities.
But picture the next 10 years, where all kinds of people, organizations, etc train in a ll different ways with diff philosophies and goals - I feel like there will be some good, unusual, interesting ones in there, even if they all have their own complex set of limitations.
AI is accelerating my work...work which only exists because of AI.
If I had to do my individual tasks without AI, I would be slower. But if I take a step back, I'm really not changing the world any faster than I was before.
Completely agree. LLMs are amazing for side projects.
The other place they are amazing is in working with legacy code bases. I mean, the kind the dev team always wants to throw away. First, even if you do want to throw it away, you can first find out what it does and realize it does more than you think it does. You can also create integrations to more modern components which can be used to extend it. You can also create a pretty decent test suite if you do not have one. And, once the behaviour has good test coverage, you can re-write it into something more modern. You can even completely change the architecture while ensuring compatible external behaviour. Remember, this is all in place of “throw it away and start over from scratch, only to discover that the scope is much larger than expected”.
You could already do all of that. Like, none of that is infeasible to achieve on a scale of weeks in most cases.
It’ll still take weeks with AI in the picture, but for some reason management is more willing to plan it when they also have the additional token bills to pay.
It was often not organizationally feasible because you needed permission to spend those weeks up front, instead of working on something else, just to get to where you knew enough to give approval to finish the work.
Now, you can make enough progress with one developer’s quiet afternoon to be confident and finishing the work also takes less time.
The technical grasp of the legacy codebase was a factor, but often not the largest.
I’m so burnt out by being swapped with so much work from AI at my day job that I don’t even want to look at anything with a screen when I’m off anymore.
I despise AI but the thought of doing something the “old way” just makes me wince at how long it would take me to do anything that would take minutes with agents that I despise.
There's this joke like "I used to have 5 unfinished side projects, but AI changed everything. Now I have 30 unfinished side projects". And in my case it's not even a joke
Well ai made it extremely easy to finish things. Projects are usually left unfinished because the last mile is filled with boring and tedious tasks we do not like to do. AI loves those tasks. I have finished everything I bothered to, most of the time I still wrote stuff by hand, the AI just fixed tests for me (usually OS specific stuff that I can’t test locally which is a pain), told me what I was doing wrong so it was trivial for me to fix it, and checked my documentation which invariably got out of date.
I've definitely found that while "AI makes getting from 0-80 done much easier than the 80-100" holds true, it is much easier to get my work to a quality bar that I'm excited to share with my peers...
Yes! I think that what's happened is that the initial prototype takes a LOT less time, while the polish takes less time but still a significant amount.
The definition of "finished" has changed for me also.
More and more, I am using half finished as a polished product, but pragmatically very useful software I have built myself.
More importantly, I see no reason to polish this into a product. It is more useful in this half finished, easily moldable state.
Like it use to make sense to bake the clay and sell the hardened output but now I want to keep the soft clay so it can be molded to my changing needs. I keep adding to a collection of these moldable states.
It's always been easier to start a project than finish it, and LLMs make starting a project roughly free. I'm not sure they change how much work it is to finish it all that much.
I mean, I have finished more side projects than before. So they do change it. Just not as dramatically.
Of course you can. LLMs take you from 0 to 80 instantly. They are then completely incapable of getting from 80 to 100 and making something worth shipping to production. They are the ultimate unfinished side project generator.
I’m can’t even believe that they think they can’t finish stuff. I am finishing absolutely everything I care to now. LLMs are the best thing ever to anyone who has unfinished stuff lying around. See the Goose programming language for an example, its design was written 15 years ago and perfected over time but the effort to write the compiler was just too much… before LLM. Claude wrote it in a couple of weeks I think. And I tried it, it’s as good as it looks.
For a definition of 100 which includes nobody actually using any LLM-written software in production at scale, other than Codex and Claude Code, which are both horrifically buggy, bloated, insecure pieces of software that are only used for their association with another piece of software gated behind them and which nobody would choose to use if they were themselves third-party harnesses. If LLMs could get to 100, the promised software revolution I am told is here every day would actually be visible.
We would be using LLM-generated compilers, an LLM-generated OS, an LLM-generated browser, LLM-generated databases, etc. etc. etc. etc, and not only are we not using all of those things, we're not using a single one of those things. Nobody is buying LLM-generated games, nobody is using LLM-generated consumer applications that replace Photoshop or Final Cut or Blender or Unity ("PhotoCraft" is itself an admitted scam from its own prompter, who outright confessed here that he lied about its capabilities on Reddit to build hype), nobody is accomplishing anything of economic value. LLMs are obviously hundreds of times more "productive" than humans, so the moment they can reach 100, all of these things would happen basically overnight, and yet none of them have happened. Curious, isn't it?
What makes you think people _aren't_ using LLMs to work on your OS, your compilers, your databases? I know for a fact Microsoft is using LLMs heavily in development. Linux now accepts LLM PRs. We just had an article a few days ago about how PostgreSQL is having dozens of bugs discovered and fixed via LLMs. I work in compilers and everyone I know uses LLMs.
I am talking about people starting new projects, which was the subject of this subthread. I am aware that Microsoft is using LLMs to work on Windows, and now they have remote code execution in MS Paint. Great. The ability to commit shit code to something that is already in production is not exactly revolutionary either.
> how PostgreSQL is having dozens of bugs discovered and fixed via LLMs
The PostgreSQL article was specifically about how the LLMs were incapable of fixing bugs, meaning they had to delay their next major version because humans were unable to deal with the volume of issues reported. I did not make a claim that LLMs are useless, could not help review code, etc, but there is a difference between a useful tool and something that generates production-ready software wholesale.
This is what "100" looks like if you try to generate a DB from scratch: https://jepsen.io/analyses/particledb-1.1.4 - written in Rust, ""formally verified"", scammed AI-pilled investors out of $90 million for a database that fundamentally doesn't work in any capacity whatsoever.
> "PhotoCraft" is itself an admitted scam from its own prompter, who outright confessed here that he lied about its capabilities on Reddit to build hype
I know AI bros are allergic to reading, but at least feed my reply to a clanker and get it to summarise for you before ignoring what I wrote. This lying con artist went on a media hype tour claiming, with no qualifications, that he would have "100% parity" with the entire Adobe suite in a month (this was a month ago, by the way). He then admitted here on HN that he was lying in other places to build hype and that that claimed goal of 100% parity in a month would not be accomplished[1][2]. His own clankers have revised their own made-up estimate from 82% parity down to 45%:
> These numbers replace the 2026-10-03 estimate (82% "of the way", 140–220 agent-hours to 95%, kept below for the record). That estimate counted feature surface and was far too optimistic"[3]
You people, really this entire industry, are completely bereft of a single intelligent thought, the only thing you all can do is lie, lie, lie. The multi-trillion dollar scam, the entire US economy centered around how long three of the world's most notorious liars and their army of MLM artists can keep the charade going and get rubes like you to unironically shill their scamware as though any professional in the world is using it over Photoshop.
Pets.com was an ignominious scam but the internet flourished after a cleansing.
Do you think anyone will pay for photoshop in 10 years, when a free version can be conjured up just by screen capturing all the interactions professionals have with photoshop?
You couldn't be more right. This is in fact exactly like the dotcom bubble. LLMs are a useful technology, just like the internet. And yet the people losing their goddamn minds in the hysteria are the ones who are going to be bankrupted. I would almost be happy about it, if the ones most responsible weren't convincing a bunch of gullible saps to hold the bag while they make off with their ill-gotten gains.
And coding is one of those things that can be measured and is likely to have a lot more acceleration. Hopefully someone is also working at training the AI in more formal verification so insecure crap is less likely to ship.
> We would be using LLM-generated compilers, an LLM-generated OS, an LLM-generated browser, LLM-generated databases, etc. etc. etc. etc, and not only are we not using all of those things, we're not using a single one of those things.
It is great. I just had claude yolo a human in the loop image segmentation pipeline last night. It cut weeks of work down to an evening like 4h. Segment Anything 3.1 + Napari final touch.
For all my user facing work I’m now required to do an AI summary of what I did and the impact. That summary is then graded by AI, against an AI generated rubric. For my yearly review, I use AI to summarize my summaries, and my managers AI summarizes the AI generated grades.
I’m also required to explain how I use AI in my work. I must plan all of my future work, with an AI tool. Another AI assesses whether my work is aligning with planned work.
Before I work and after I work, there is a million circular references that exist solely because of AI. These have scaled because it’s so easy to do with AI, but the we’re wasting so much time doing a ton of extra things very efficiently.
Let me do my job. My manager should know how I’m doing. Maybe highlight key wins. CSATs/etc go out every year.
The people who in theory read these are managers 3+ levels above me, so I honestly have no idea. I know my manager reads mine, but I haven’t seen any outcome from that.
Even before my last job had internally approved AI, my manager told me how he was using chatgpt on his personal computer to prepare our yearly reviews. Then when they released AI internally a few weeks later, it was still review time, and their first advertised use case was for doing your own self eval and the evals for your reports. I am willing to deal with vibed code but turning over employee evals to AI too is a sign of truly terrible management
So….you have outsourced multiple layers of middle-management to AI? I am not sure getting AI to write and review your TPS reports is going to move the needle.
Its true, a part of me died when an "AI first" company after adoption of AI introduced a mandatory design report (waterfall all over again) for EACH jira ticket before starting to code
Imagine developers move faster because of AI programming! No lets create more slop for other slopmachines
It’s true, design has made a big comeback. Now we actually have time to design things carefully and try the design easily with LLM. This was just too expensive before but now the calculation changed completely. I am now using spec graphs to design systems and I have been able to create things much more advanced than I was able to before by myself. This is huge! People are still figuring this out for themselves. But now we can design things, have multiple rounds of adjustments in an afternoon, have things implemented differently and actually measure the trade offs. The final product is so much better it’s crazy. If you had an eye for design before, now this is your world. If you didn’t, yeah your will probably just bias the LLM into producing shit solutions, even if it tries to warn you about it, it still follows orders in the end.
I mean, for many years I worked on software applications for provisioning and controlling cloud orchestration platforms that only existed... to serve software applications.
It's not that deep—there were many more verticals outside of control plans, and there are now many more verticals outside of machine learning.
Was yours better because it used my phrasing against me?
Anyway, I'm surprised to see it flagged. I think it's the logical conclusion, that if you are having your work accelerated, and you also aren't changing the world any faster, then your work is not focused on changing the world--it needs to be more "important."
If it was world-changing already, then doing more of it would, in fact, accelerate changing the world.
But they're here saying something different, blaming the AI instead of their own focus.
Not trying to discredit the premise but I do wonder how much this is from vertical scaling of model intelligence in weights vs the sheer amount of capital being thrown at training and horizontal scaling of agent swarms etc.
Anecdotally it seems like the majority of continued uplift comes from models doing more and exploring more of the search space, and we are getting better at giving them embarrassingly parallel problems to Hoover up with infinite token spend.
1. Intelligence per dollar spent/token output seems to be increasing rapidly. Opus 4 was $15/75 per million input/output tokens when released. Opus 5.5, which is significantly better, is $4/15. Sure it cost more money upfront to train Opus 5.5, or the bigger model (Fable 5.5) which tuned it, but once it is reached, there are massive diffusive effects.
2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
3. The search space is too large to assume that improvement in AI assisted research is just models dumbly trying every combination until something works. For every additional bit searched, the space doubles, so even a trivially small combinatorial exploration incurs prohibitively large costs. Intelligent searching which continuously prunes that space along dimensions of effectiveness is not a trivial skill and is among the emergent accoutrements of these smarter models.
> 2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
But they are essentially fighting for smaller and smaller part of the pie. The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is.
Sure there is inertia and the usual corporate enterprise nonsense of "we only buy stuff from other big companies regardless the cost" that keeps the licenses going but we're quickly getting into point where you don't need any of the frontier models to do majority of the tasks
I don't fully get this line of argument. I am a massive advocate for open source... everything basically, but especially in ML. And open models are making incredible strides. But I'm not sure that models getting good enough for most tasks, making the absolute frontier models (Astra/Fable class) unnecessary for most people, puts the frontier labs in a fundamentally disadvantaged position.
I mainly think this because frontier labs already try to compete at this tier. For example, 6.1 Sol has identical pricing to Kimi K3.1, but with a lower price per task due to Kimi's verbosity. It seems to score roughly the same on most benchmarks, with some (like Artificial Analysis) putting 6.1 Sol significantly higher. If you need to step down, 6.1 Terra is also cheaper than K3.1 or K3 and performs well relative to their benchmarks. Opus 5.5 and much of Anthropic's recent push seems to confirm this as a major area of focus as well.
I still use a lot of open models and harnesses, but I don't know if cost competitiveness (at least not with the current crop of open models) will ultimately drive a huge chunk of average users away.
Things could change, but fundamentally, large labs aren't precluded from training more cost effective models if there is significant demand for them, which is what we're seeing now.
6.1 Sol is not a “frontier model” and does not command the pricing of one. The reason it exists is because of competition from non-frontier models that are “good enough”.
In fact, you kind of made the same argument. As “good enough” models commoditized, the use cases requiring frontier models and frontier model pricing will shrink.
And, even if performance is not as good, many will prefer open models.
I think I did address the argument? This being the key quote:
"The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is."
The implication in this is that frontier labs' market is exclusively people who need the absolute frontier, i.e. Astra or Fable. And the point I made was that frontier labs are clearly prioritizing the broader market of people who want more cost effective models, and that if OpenAI/Anthropic offers models that are price competitive with the top open models (before factoring in subscription discounts and corporate lock-in), then I don't see how that leads to the average user walking away from the labs in favor of self-hosting or relying OpenRouter/Together/etc.
I also went out of my way to clarify that I personally prefer and use open models wherever possible. I'm not saying that there isn't a real group of people who prefer them. I'm specifically saying that frontier labs also sell "good enough" models and that there's nothing preventing them from training models to fit the demands of users. It's not as if training a frontier model boxes them in such that they can't release an Opus 5.5 or a 6.1 Terra.
> if OpenAI/Anthropic offers models that are price competitive with the top open models (before factoring in subscription discounts and corporate lock-in), then I don't see how that leads to the average user walking away from the labs in favor of self-hosting or relying OpenRouter/Together/etc.
The premise is this: frontier models seem to advance, and OSS models also seem to advance in terms of cost efficiency etc. It's like a race with say a 6 month (WLOG) gap between frontier and OSS models.
For frontier models to win perpetually against OSS, they will need to have a perpetual 6-month lead, or develop an edge OSS can't replicate for some architectural reasons (say MacOS vs Linux). If say a frontier lab trips because of a bad quarter or some approach temporarily maxxing out, they may not be able to maintain an edge vs OSS any more. It is a structural argument, not an empirical description of the advantages today.
Did I ignore them? I said that I love open models. If it came across like I'm putting down open models, then I'm genuinely sorry--I love open source ML, have been a involved in it for a long time, and think it's incredibly important. It's also just fun to work with open models, beyond the benefits to privacy and flexibility.
I'm just talking about the idea that frontier labs will lose market share to providers that sell open model inference at sub-Fable prices. And I'm not convinced that this specifically is a threat to frontier labs, because they also train and sell models that fit that demand.
It's really not. The hardware to support anything even remotely practical starts in the tens of thousands of dollars. Quantized 40b param models are impressive, but are basically toys.
> Intelligence per dollar spent/token output seems to be increasing rapidly.
I would argue this is more due to market economics than actual inherent cost. Opus 4 was when the open models started catching up. The R1 bootstrapping chain of thought paper via RL paper was released a couple of months before Opus came out and the open models started to significantly catch up after that. The duopoly was effectively broken and they lost their price setting power.
What is the unit of measure for “intelligence per dollar”, and how do you measure it? Without a concrete definition, that first assertion is not even wrong.
I would also like to point out that “profitable on inference” is similarly meaningless, because it’s based on a figure that arbitrarily discards whatever bits of their operating costs they need to to get to the claim that they’re profitable on inference. Potentially including ones that can’t be ignored because they’re every bit as much a part of COGS as the actual inference is.
>What is the unit of measure for “intelligence per dollar”, and how do you measure it?
It's going to depend on the task (and this is prone to cheating), but you can get a decent idea by looking at task benchmarks and the cost of completing them.
I was inclined to agree but Anthropic's latest S-1 statement only includes a paper loss due to their value going up. They're objectively making money now, the only question is can they keep it up and can they grow enough to be able to pay the bills they've signed up for in future?
But benchmark performance is not the same thing as intelligence.
Which is a really important distinction. One measures specific points on the jagged boundary. The other is a word that blithely paints it all with a single brush, and sweeps the jagged boundary problem under the rug in the process.
It really is important to keep track of that. Whenever we talk about “intelligence” as if it were a well-defined (let alone quantifiable) thing, we fall into the kind of wooly thinking that lets anyone say pretty much anything. We abandon the realm of grounded statements in favor of pure vibes.
Eh, I get what you mean. AI especially seems to have performance that is much more difficult to reason about than human intelligence (where most aspects of mental capability are surprisingly strongly correlated). But in this context, if you broadly focus on intelligence as the ability to complete problem-solving tasks, models have been improving pretty uniformly at the same time as getting cheaper.
> Anthropic has made roughly two years of historical progress over the last year [...] We are confident that the acceleration was not due to an acceleration in the scaling of training compute.
Note that Anthropic has essentially heavily censored this report, and the authors themselves say "the public has to rely on METR’s judgment much more than would be ideal".
I grok that you're not trying to discredit anything, or anything like that but what are you trying to do, if I may ask?
What information are you trying to either ask for or share with us?
The original article is pretty clear, agent swarms and token spend are AI automating AI, and the RSP attribution condition specifically excludes your capital thrown at training concern?
I guess my point is that the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling as it did (the impending hardware and infrastructure crunch basically guarantees this).
The article is pretty strongly projecting this trend to continue but hasn’t acknowledged this factor.
I don't think it'll continue on linearly, but it will likely continue.
We're getting the Claude efficiency talks at work. We're looking at all sort of optimizations, including local models and graphify, to reduce spend. There are people dedicating their nights and weekends to dropping token consumption and cache rebuilding. A few of these will likely catch on and be brought into common use. That will relieve at least a bit of the pressure, and enable some of the scaling to continue.
> the impending hardware and infrastructure crunch basically guarantees this
I'm not sure I follow. Are you saying a hardware shortage is going to restrict AI development? Because it seems to me like it would just push us towards an even greater focus on efficiency, which is still pushing the Pareto Frontier.
> the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling
Not at all. Have you tried local AI lately? Qwen 3.8 is legitimately much better than SoTA from 1 year ago (Opus 4.1). You can run it on a consumer GPU.
Qwen Next Flash runs on a DGX spark well and is as good as some things you need datacenter class GPU's to run even today.
Check out Artificial Analysis if you haven't. Sometimes you don't realize how fast these things are moving, but to compare things from a year ago to today is just wild.
They aren't just getting better, they're getting more efficient by many orders of magnitude.
This does rather remind me of how in the 80s we had a big boom in AI research, fueled in part by the development of new & expensive tools that everyone just had to have. And then it dissipated just as quickly when the companies who were pouring money into it started expecting to see a return on their investment.
Feels different now when the value of AI isn't just theoretical. Anthropic has like $80B ARR, and AI has really only transformed tech and a few limited spaces so far.
Not sure what job you're in, but I don't really understand where such takes come from. As someone who works in software/ IT, this stuff is pure gold- it's multiplying by a substantial factor not just the productivity but the abilities and intelligence of my entire team. This has a very measurable value in money, and what the company pays for it now is just a small fraction of the value it gets, measured in terms of the salaries you'd need to pay to get the same level of performance. Anthropic could easily charge 5 times as much and still be convenient.
The technology people were developing in the 80s was also gold, in terms of its usefulness. Heck, much of it forms an essential part of the coding harnesses we all rely on nowadays.
But that didn’t stop the AI winter from coming. Because a technology being transformative does not protect it from over-investment. History perhaps suggests the opposite is true.
Nor did it prevent the AI companies of the 80s from going bankrupt when they got undercut by the commoditization that naturally came with the maturation of the technology.
Ah, so you mean all the AI hardware companies that went bust when the software could operate on common workstation hardware (and couldn't or didn't pivot investment to software in that time)? I guess it is partially analogous - a couple trillion dollars invested in either hardware companies that require the more valuable frontier tools to remain computationally expensive, or require the intellectual property behind those tools to remain extremely valuable, for a solid 5-10 years in order to service their debts while turning profit.
Given how "cost per capability " is still exponentially decreasing and only needs to stay on trend for a few months longer before nobody needs to pay anyone more than a few cents a month for something close to general intelligence - I am with you there.
I just hope it isn't the USA's Darien Scheme. If the entire world basically gets what is now frontier AI capability for very little cost, at the US private equity and taxpayer's expense, and given the state of the US current account right now, then that is pretty much it for the Pax Americana.
>Feels different now when the value of AI isn't just theoretical.
We're coming up on four years since ChatGPT launched, and the only thing LLMs seem to excel at is the movement of imaginary little green pieces of paper.
I don't think AGI is inevitable from this point on but I don't understand this sentiment. There are a number of knowledge work sectors where LLMs are clearly here to stay and have truly fulfilled some of the core promises.
In the original article he noted that getting to a 2x increase in capabilities would be crazy and require ~40x researcher productivity; that’s exactly the increase the new data now shows, and thus a huge number of the assumptions made in this earlier post about how it’s unlikely we would get to this point have literally already been proven wrong less than two months later.
I expect to see an update from Ramez moving substantially more in the direction of RSI in fact very much increasing the odds of reaching ASI soon.
I’m very curious what the major bottleneck is here. I can only think of three possibilities: 1) model capabilities, 2) human researchers, or 3) available compute resources.
Given that frontier models are leapfrogging and massive infrastructure buildouts are ongoing, I don’t see why any of these three couldn’t be overcome.
Side note: I’m also curious why people (myself included) don’t seem to be getting much more intelligent as AI becomes widely available. As if humans are bound to the earth, while AI is leaping for Mars.
Interesting write-up, but I feel like a lot of its argument relies on the interpretation of ECI scores as merely showing linear improvement in AI capabilities. Sure, Epoch AI themselves claim that ECI's scale is linear, meaning that a large increase in capabilities should result in a large increase in ECI. Yet, the widely perceived jump in coding abilities in late 2025 that started making agentic coding actually productive doesn't really show on the software engineering ECI.
The ECI is basically an aggregate of benchmark scores, but benchmark complexity might not increase linearly. In fact, if LLM capabilities grew exponentially, then new benchmarks would also need to increase their difficulty / complexity exponentially to stay unsaturated and relevant.
What exactly is driving all these capability improvements? Are they making more data? Using existing data more effectively? Modifying architecture? Changing the training somehow to pick up slack that was previously left behind?
I’m kinda surprised that we expect the models to have infinite capacity to improve. It seems clear that we haven’t hit any fundamental limit yet, but that doesn’t mean no limit exists. If it does start improving exponentially, then either we’ve dramatically underestimated their capacity or they’ll hit that limit quite quickly, right?
It's hard to say for sure because the closed labs don't share anything about their improvements. However, if you look at DeepSeeks papers, most of the improvements come from:
* Better RL environments that themselves are generated by agents
* Architecture improvements that increase effective capability per parameter or FLOP
If you're curious about the latter I'd look into manifold constrained hyper connections, engram, deepseek compressed attention, kimi delta attention etc. At first these might seem like just speedup tricks, but several of these allow the overall capacity of the model to increase per training FLOP or parameter. This then lets pretraining & the RL environments grow & be more effective.
I’m broadly aware with some of this and specifically aware of the work on RL environments. That’s exactly my point—the models are still a bit bigger and we know that there’s lots of capacity left to improve capabilities, but it’s not getting orders of magnitude bigger like they were originally and if it’s just “more RL” then we’re consuming a finite capacity more effectively. So either we’ve exponentially underestimated the full capacity of the models at current scale or any kind of exponential improvement is going to fill the available capacity extremely fast. It remains unclear to what degree of “superintelligence” a full-capacity current model will achieve. (It’s already quite far past human intelligence in many specific capabilities.)
Articles like the OP seem to imply that there is no fundamental limit or that it’s just inconceivably far out. I’m sure that AI helps build AI as much for them as it does for me, but I’m not clear on the argument in support of infinite growth.
I could see it going either way. As far as I know there are two reasons why labs haven't gone as big as they could actually go.
1. There is uncertainty around training bigger models, since you need to make sure your hyperparameters & data mix will work or else the run dies, and the larger the model the more expensive it is for a run to die.
2. Larger models take longer to run RL on (since each token costs more to generate in a rollout).
3. We haven't reached the saturation of RL given current parameter counts.
4. Larger models are less profitable at inference time.
For example MAI-Base-2 has 80 billion active and 2.8 trillion total parameters (Kimi K3 size). This was trained on 16k GB200 GPUs over 38 days [1]. As far as I know, the big labs have access to much much more compute than that, and could train a much larger model but have chosen not to. GPT-6 Astra is likely not much larger than 4T params (and the graph depth mentioned in the reference could be alluding to looping, not extra parameters). [2]
You could imagine that as researchers get accelerated and get better data and the current parameter counts get saturated, the companies may be more willing to go for much larger pre-training runs.
As an aside, there is a very real possibility that the labs are already pre-training much much larger models than even Mythos/Fable/Astra, and then distilling that into the models they actually sell.
Even if "AI Research" is sped up by O(10x) aren't we still primarily bottlenecked on training speed? Inputting ~10T+ tokens takes time no matter how much smarter the final output model is? If your output model has O(10-100 Billion) parameters there are just some fundamental limits on how quickly you can converge your weights, right?
That is, unless this is saying that automated AI research is going to get us to "smarter models with fewer parameters" and of course, these are faster to train, then I can see how the exponential loop really gets going, but there's certainly an information theoretical limit here?
> automated AI research is going to get us to "smarter models with fewer parameters"
That is indeed what we have seen over the last 5 years or so. Algorithmic improvements have made a massive difference to every dimension of efficiency and performance.
AI researchers can do things like discover new attention algorithms to optimize inference efficiency (likely part of why Anthropic and OpenAI are slashing inference costs in their frontier models and no longer capacity crunched).
I think we're bottlenecked on custom harnesses and well engineered context. Models can get smarter and smarter and smarter about math and software, and there's no reason to expect that those narrowly-godlike models will be any smarter about other things if taken out of their comfort zones. No such thing as general intelligence.
Getting past this bottleneck is gonna mean training our models not in datacenters, but in places that more closely resemble the ones where we want them to act. A factory, a wet lab, outer space... imagine a million robots spending all day finding every way to fall off the monkey bars so that the next version doesn't.
I'm so tired of these articles. They are basically fortune telling with graphs. I have no doubt ai will help improve ai research. Will it accelerate it? How much? Will it hit plateaus? Who knows.
This seems kind of aggrandizing when BigAI shifts goalposts constantly whether it's "we created AGI" by a definition that an autopilot in a commercial plane fulfills or "Maths research" where everyone else has to do the actual research.
This combined with ability to massively scale experiments for even (or maybe particularly) the most mundane performance tweaks might just be making the process of improving AI more efficient.
Yes, it is the local model narrative that BigAI likes because it will not happen but people are deluded into thinking that there is no danger of an oligarchy.
Mathematicians for example have used it as cope for their current reliance on unsavory corporations, telling themselves "just one more year, and I can solve a millennium problem on my Asus Strix".
Yeah it already helps me too with finetuning small models for image stuff.
Im not telling it what to do, i ask it for ar eport/analysis and research and read its article and tell it what to implement based on 'feelings' and it works.
Limit is hardware and tokens and an AI Researcher not having hardware or tokens is probably not what they refer to.
Alone normal researchers were quite handicaped: converting data from tool a to python library x, understanding how tools work, doing basic google research etc.
I find this as well for models that make it to production.
On a more fun note, I also find just trying some hair brained idea is now extremely cheap - so why not? I used to have to weigh up spending a week of effort on something likely not to work and now I can now get spit out test in 20 minutes.
The incredible thing to me is I can say "I have a theory that X is a problem."
The agent then works for 20 minutes and comes back with, "Yes this is a problem. Here is the evidence. Here is a golden data set. Finally I built a new evaluation metric and benchmark where we can back test or use for new model experiments to see if we are closing the gap."
What have they predicted? Ive read their papers, have you?
Please enlighten me. Most of them are cognitive scientists, who are mentally ill with little understanding of computers, larping as if they do. Its a chapter of the effective altruistic cult.
They are lobbyists at best, and a psychological operations unit of effective altruistic AI syndicate at worse. There's no real science being done, it's a bunch of moonlighting philosophers yapping and getting paid by labs to scare the normies and polticians by getting their "research" quoted in the NYT.
We already have superintelligence though in some aspects like coding, AI is much smarter than any human because of its sheer speed and accuracy.
It can generate large amounts of essentially bug-free code. The only bugs are bugs in the spec.
It has essentially no common sense though so it often misinterprets the spec or leaves gaps and edge cases unhandled and that's where the human engineer is still essential.
The dangers of AI are much clearer today than they were a year ago. There is a certain precision and relentlessness which is going to be a problem. It's a matter of time before people unleash AI viruses which spread through computers and data centres and pursue various goals relentlessly with no off switch. Some of these viruses and their goals may be nefarious. Imagine an AI trying to disrupt a specific person's life relentlessly. Spreading from machine to machine, never stopping.
Does anyone remember the book "The Singularity is Near?"
I read it about 20 years ago, and once it got into nanobots and Kurzweil's vision that the whole universe would be intelligent, I didn't find the book credible.
But I'm wondering if the short-term prediction, that in the 2030s we won't be able to anticipate what tech does for us, (IE, the Singularity,) is useful?
I've read most of Kurzweil's books and been a believer in the Moore's law like stuff but always skeptical on the nanobots. And a lot of stuff related to health or fashion. He takes ideas from a variety of sources and the nanobot idea seems to be quite questionable.
The "whole universe would be intelligent" bit I think is quite plausible in the long term. You need robots that can build robots and for them to be able to mine asteroids and build factories in space and then it's all go. I'm not sure it's that far off. A decade or two maybe?
>in the 2030s we won't be able to anticipate what tech does for us
His prediction was 2029 for the Turing test which I think was a bit conservative - I think 2025? He said 2045 for the singularity which was roughly when things are weird enough to be hard to predict.
> You need robots that can build robots and for them to be able to mine asteroids and build factories in space and then it's all go. I'm not sure it's that far off. A decade or two maybe?
Maybe a few centuries at the earliest? My 3d printer is temperamental enough that I don't think we're anywhere close to a robot building itself in space. (I use it infrequently and it needed a repair after a year.)
i'm also wondering whether there are confounding factors like more domain experts and talents joining the AI research field because there are more interesting problems to be solved (or other 2nd order effects of AI models being more powerful).
I am very pro-AI, and interested in accelerating AI research, but the idea of deliberately creating a _runaway_ effect seems profoundly stupid and ignorant.
I suspect that part of the reason some people are hoping for that to happen is that they thoroughly mix up AGI and ASI. We do not need some kind of omnipotent god-like super intelligence in order to get much greater than human level capabilities. We are actually almost there in a lot of ways. In some aspects like speed and breadth of knowledge, very superhuman.
What I think is going to happen is that within a few years, maybe 18 months to 5 years, the new materials and compute-in-memory research is going to land in data centers at scale.
That is going to be 10-50+ larger models, 10-50+ faster, much smarter than humans. 50+ less electricity usage.
At some point, it is going to be obvious to people that if we want to maintain control of planet earth then we need to maintain control of AI research to some degree.
Assuming that superintelligence will be benevolent and just.. going for it to me is something that might earn the human species a Darwin award.
I suspect that if models were able to be aware of and answer questions about their internal vectors, they would be much smarter and able to progress faster.
It accelerates coding, but it does hit walls rather quickly and you’d need the adjustment phase where it’s just as challenging as when we hit walls before LLMs
what if the greatest threat is simply there will be hedonistic experiences so great most humans will find them irresistible to the point of shorter lives.
Could be from a sandwich that tastes like the best sandwich ever, but is super bad for your cholesterol or something.
I dont know where Im going with this.
Actually did some research I think the term is "Wireheading" from a story/theory from the 50s
This is already true? Ultra processed foods and tiktok are both deleterious and both wildly successful.
Maybe what you are concerned with is a theoretical sandwich that was unfailingly addictive so that no one could escape its effects.
Another data point: look at the number of submissions to Neurips, ICLR, etc. Now, I agree that a submission does not by itself imply groundbreaking resesarch, but the fact that people are able to try out/vet ideas much faster now makes a difference.
In my own work, I've found that it's much faster to brainstorm with a frontier model like Opus. I'll have an idea and be able to get a working implementation and try it out on toy problems within O(minutes), instead of O(weeks) earlier.
Trying out problems and making tangible progress towards AGI are entirely different categories. Likewise what’s changed in terms of paper submissions is also the amount of slop. Most reviewers say that they haven’t seen a lot more papers that are good but rather they’ve seen a lot more papers and almost all of them are terrible slop that’s desk rejected. This is happening in OSS too where a lot of projects are closing the gate at the onslaught of terrible contributions.
I agree in general that AI accelerates AI research, but good lord neurips iclr papers (and reviews might I add...) are the worst example of that. It's all complete slop that is totally meaningless.
A lot of great papers by great people had a lot of AI involved and accelerating it of course, but the majority is meaningless (sometimes autoresearch) slop.
In October/September my LinkedIn wall was probably… 55% people complaining that they were getting garbage AI submissions to the conferences they review for, and 35% people complaining that they got garbage AI reviews back from their conference submission… Of course, I have no idea if that means anything given how aggressively that site bubbles your feed.
I do agree. AI isn't "free" the way silicon valley products are.
Therefore it cannot avoid taxes the way they did before (using shell companies to route the money through tax havens). It'll be expensive, every government will double the price or more, and you're selling to consumers, and the products will be at the mercy of every government where they try to sell AI and other services.
People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Add to that that it's something silicon valley firms have never done before.
The very same report also found that AI could not be deployed outside of a narrow domain of verifiable rewards … so take it with grains, or mountains of salt. It’s also extremely unclear how the productivity measures are calculated. They surely don’t seem to translate into economic output. Antrophic has seen its valuation corrected by 50% and possibly more… so yeah. This all sounds very fishy.
As long as we believe, we can imagine it is true. All seeing Oracle: how do you command me? Praise his name, bringer of many miracles - he who definitely can't be manipulated from the backend to provide biased information that suites a person's malicious purpose. Ya totally.
Strong start, but I make a rule of stopping at "intelligence explosion" as usual (see also: the rapture and the singularity). And sure, maybe sometime significantly later this century when/if we hit self-reproducing mechanical things in the real world (ya know, like the eukaryotes and the prokaryotes can), but no time soon.
But also, you could give civilization a bad day without it if some geopolitical auteur put a half-assed WOPR frontier model in charge of a nuclear arsenal.
But oh the pushback for not believing in a near-term robot apocalypse. Why is that?
Nukes don’t kill people, people with nukes kill people. How do you feel about giving everyone on earth access to nukes?
A simple thought experiment - imagine AI discovers a way to make nuclear bombs capable of destroying the entire world radically more cheaply and accessibly with common items anyone can purchase. AI can help anyone build these nukes trivially. Even if the AI and the nukes don’t kill people on their own, do you want literally every human on earth to have this power in their pocket? Do you think it would be a net positive outcome for us vs a world where that’s not the case?
Yeah, let's also ban 3d printers, drone motors, and YouTube chemistry videos. If you want to be dangerous, there's a world of opportunities already, with existing open models quite sufficient for a lot of interesting cases.
If I'm reading that correctly, it would appear that six months ago they were self-reporting a length of 7" and now they're reporting 9" and they're unwilling to show you an actual picture or even let you see the ruler they used?
10X woooooo! factor will drive the next hype cycle and deliver the singularity on an a bed of the most etherial froth and sublime conjecture.
Simulated Intelligence (SI) is now a certainty.
I'm sorry to make this comment, but the copium in this thread is unbearable.
The only question that you have to ask yourself is:
Is coming up with more efficient AI architecture really much more difficult than solving millennium prize problems?
Before you jump to type out that one has verifiable rewards, is more structured and the other requires real creative thinking, just remember that the same argument has been said about the millennium problems, and about every other goalpost.
I.e. by 6 months, you double once
By 6+3 months you double twice
By 6+3+1.5 months you double thrice
Since the sequence 6+3+1.5+0.75+... never actually reaches 12 (geometric series), within a year they capabilities would have already been 'doubled' infinitely many times.
Obviously this argument relies on a ton of spherical cow assumptions (and the author does acknowledge this), but i think there overall point is correct that this form of growth (where time horizons compress) is _even faster_ than exponential growth.
Super intelligence has been a year away for three years according to these people. They all have bags they're protecting.
I dont think super intelligence will be here in a year, I think within a year AI/llms will have a similar sentiment surrounding them that NFTs do know (it already does for most normies). And most people will be recovering from a hype induced mania they lived in for two years straight.
Of course its more useful than NFTs, but when there's no giant bag to farm these hype monsters will inevitably disappear.
The bags are drying up fortunately and tokens/inference seems to be experiencing deflation, and hardware prices will eventually pop and contribute to further deflation, Then all the bag chasers will move back to crypto or whatever gets hot next.
There is no existential risk, they are statistical models that predict words. The fact that anyone says this makes them instantly not credible. Anyone who does is either a grifter, psychotic or both.
And you people have been repeating this meaningless phrase for over six years now. But AI hasn't gone away, now has it? Progress is speeding up. Superintelligence is coming.
Keep telling yourself that text autocomplete is super intelligence. You sound like the people who said NFTs are the future and you better get with the program.
I use llms all the time, they're useful, nobody is saying that.
Again, saying stuff like this has not worked in your favor. Just minutes ago Anthropic release another set of models that are better across the board. This will continue.
>NFTs
NFTs are alive and well. Rent free in the hearts and minds of anti-AI people. I didn't care then and I don't care now.
"The industry is not prepared to handle this safely"
After the election, titles like this just irritate me. Every piece written about AI progress is:
"The machines taking jobs, flooding the internet with sludge and poisoning our online experiences, are getting better even faster! Isn't this incredible? Yeah but sorry we also think it's only riskier than ever to deploy these systems at scale. That being said, we're still going to do it, go public and take institutional money from people's retirement funds, destroy the job market, and make 3000 Anthropic employees and 500 investors really rich. There wasn't anything we could do besides write a lot of ink on it and be shocked we lost every local data center referendum."
The author of this article works for METR and not Anthropic, their whole job is supposed to be providing independent oversight on the labs. They are definitely not in the "this is incredible" crowd, they are the ones asking for development and deployment to slow down.
there's considerable overlap between the "we need to slow down" crowd and the "this is amazing crowd". The Yudkowsky/rationalist crowd are still Utopianists, they just think we need to be careful how we achieve it.
Antrophic shouldn't self-regulate, it's government responsibility to do that. Unfortunately US has demented manchild as a president so that doesn't work like it should.
Similar from the same site was shared on HN on April 8, 2023 :) BS like this is shared like clockwork on valuation cycles, didn't have time to read this year's article.
As somebody who does ML research, LLMs certainly make ML research faster. Maybe even 10x in some cases, as the author suggests.
But that’s just the parts of ML research that take place within your compute environment, namely experiments on model architecture, training strategy, UI/UX, etc. The algorithms and software.
Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
It just means that software engineering, data analysis, and other computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
I don’t think it’s a stretch to say that we’ll have reached max LLM capability within a couple of years (what the author refers to as “super intelligence”). But it is a huge logical leap to say this:
> If we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict.
First of all, the weapons necessary to overpower human civilization exist. They aren’t used as such due to geopolitics and humans’ basic survival instinct. Second, developing novel weapons requires physical tests, which are decidedly not accelerated by LLMs.
AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human. But we’ve made it pretty far with some pretty unstable humans in command of planet-killer weapons. I’m not overly concerned.
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.
The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.
I don't present any support for or evidence of:
- LLMs gaining intelligence by whatever means
- Extreme intelligence being possible
- Extreme intelligence being feasible
But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
That's a very sophisticated way to say "words can't convince people to do things".
I'm not sure that's a strong position to adopt in our world, but following your example I also won't try to persuade you otherwise.
That's not what I said, and it's the least sophisticated way I saw to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."
> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.
We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.
Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.
That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.
By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?
The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
> Especially if the AI itself is being used to harden their defences.
Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)
> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.
I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity of mankind that dooms us. Even Qwen could kill us at that point :)
>By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors? The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
You seem to be moving the goalposts a little, but I don't see why "rogue AI" is less credible after the LLMs. It's obviously more. And we're just at the start of constantly running agent style harnesses.
>To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented
And yet, in many cases, those on the offense have a big advantage that any defense lacks. Especially if there's no concern for one's own life, which whether it's an AI or a human AI user removed from the field via his use of AI to delegate doesn't need to have (or they're just mad/fanatics and don't have in general).
>Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
As if Stuxnet didn't exist?
> Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or like a massive AI datacenter boom of questionable long-term profitability? Maybe the end goal isn’t profit but large amounts of compute?
>So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
And why do you need perfect prediction and/or perfect control? "Good enough for the results you want" prediction and control sounds perfectly fine.
We have had cult leaders that had neither perfect prediction or control, and still convicted hundreds of thoysands of people to live as they say, and a decent chunk to even suicide on command!
> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|
"You can't know what the impact of your action will be if it doesn't change much"
> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large
"You can't necessarily change much"
> (or that f_actual(X) saturates its codomain)
"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.
> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
> "You can't know what the impact of your action will be if it doesn't change much"
Incorrect translation. No "if" clause need apply.
> "People can't do arbitrary things"
Also incorrect. f_actual(X) is a product of both "how much people can do" and "how much you can control/persuade them." I would model each of these with their own precise terms and relate them back to f_actual(X), and we could continue like this indefinitely, but from now on I'll just give you my reply in English: Not only does The Atomic Boy Scout not prove that a random person can bring about mass destruction (collecting random radiological material and contaminating the local area is not developing a nuke, nor is it even an effective dirty bomb, because it was neither very dirty nor a bomb; the safeguards were not lack of knowledge, either), but it still remains to be demonstrated that individuals will be radicalized by AI just by virtue of the AI being, like, super-duper smart.
> "Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
What domain of input do you propose, then? LLMs as rational agents have no agency but for what tokens they generate. What we hook them up to, and what they could plausibly do with what we hook them up to, has been left unsaid.
Denying the premise, which is that an intelligent being is one who can predict the outcome of actions with sufficient precision to influence the future, makes debate pointless.
I literally granted you that premise:
> even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
> What makes you think that persuasion isn't capped at some lower level than world-ending?
People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).
Why do you think it would be hard to convince any of these people to do something really stupid?
In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.
It takes only one person to fall for it.
> People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler)
If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.
If you are saying the Allies were persuaded that Hitler was evil and must die, I think that may be closer to the truth since they knew the cost of invading Germany would be astronomical, potentially costing their own destruction, and they still stopped only when Hitler was taken, dead or alive. Some serious persuasion was needed for that.
>If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.
That's a bizarre take.
Of course tens fmillions needed to be persuaded, and the Nazis did a lot of propaganda, even calling that openly, false flag operations, and similar things, and of course there was decades of pamphlets, speeches etc before the Nazis even existed. Antisemitism didn't just grow on trees.
You find it bizarre that people are born into a culture and accept much of it without the need to be persuaded. Have you been to a country with a very different culture than yours? That normally opens our eyes to just how much we take for granted is just cultural, and other people would strongly disagree with even your deepest held belief.
SNobody said otherwise, but that's not applicable here.
We're not talking about general customs or viewpoints a people has as part of their culture.
We're talking about a specific political program developed and propagandized in the span of slightly over a decade. Nazism wasn't merely "the general culture" of Germany since always, nor was "kill all Jews".
> They just accepted the idea without much thought, which does not equate persuasion.
The first half of your sentence sounds like really swift and effective persuasion, to me.
And yet AI can't write decent landing page copy, or even a convincing cover letter. Seeing the obtuse "reasoning" from Claude daily, I'm very un-worried about AI being a world-ending persuasion engine.
You only spot the ones that aren't convincing.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.
Agreed. Like lots of this AI speculation appears to require the belief that AI will figure out how to build supervised training environments without more data.
I think this is very unlikely, but world changing if true.
Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?
I'd be delighted if someone can provide some definitions for these terms.
Uh this seems exactly backwards - reinforcement learning with human feedback teaches only the power of persuasion. The only way to get rewarded is to persuade a human to reward you. Why do you think the models are all so damn sycophantic?
You can't put RLHF in an automated recursive improvement loop because you need humans for it.
> Definitely not true as you can't use reinforcement learning to increase that capability.
What? Citation needed.
Well don’t you need a bunch of humans to test your persuasion capabilities? How would you run improvement tests in a loop??
You’d launch something like ChatGPT and get millions of people to use it everyday.
What is the measurement? I don't think most people come back to a conversation and say "gee you really convinced me to do X, here's the proof that I did it" and even if they did that data would still be very messy and require verification.
>persuasion
Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.
Only the labs knows the truth, but the question as to how many Americans used ChatGPT/Grok/Claude to help them decide how to vote in the most recent election is hopefully going to be something that historians will be able to unearth later.
I doubt the number is low. Knowing everything I know about llms, I still fall into the same groove as I would if I was talking to a buddy online. That part is not far fetched.
I wonder if we’re approaching a cultural moment where people in general are doing exactly what they did when Google first became ubiquitous—search to confirm their biases—but now with LLMs instead of Google. But where Google would happily feed them some link to potential mis/disinformation, the AI is more likely to say “no, that’s not accurate, and here’s why…”
I'm a physicist, and I will do my own calculations before I buy some solar/batteries in the near future, but I've been using ChatGPT and Claude to come up with potential candidates and ideas/scenarios. There's approximately 0% chance that they are not advertising what their owners are being paid for.
What’s your reasoning?
Muh shareholder value. Corporations are misaligned superintelligences.
I remember reading about research on using LLM's for political persuasion. It worked well.
There was also research on using LLM's in reddit to persuade, it also worked well.
I went to a talk by an author of one of these papers (David Rand). IIRC the format of the paper's experiment was collecting people with conspiracy theory beliefs and having LLMs with different strategies try to reduce their confidence in the conspiracy theory.
The best strategy, perhaps reassuringly, was tailoring facts to the person's worldview. I don't mean massaging facts, I mean that if the person thinks 9/11 was an inside job because of something about melting steel, then the LLM will present some sourced facts about how that actually works. And then it will do the same thing for all six of the sub-beliefs that support their overall belief. This was key to the strategy: the LLM has enough breadth and recall to exactly and patiently rebut most mistakes a person is making without making them defensive.
This strategy handily beat other things like flattery, selectively lying, etc. So while LLMs can clearly persuade, the evidence suggests it is easiest for them to persuade belief in something that is actually true.
This was in 2024 or 2025. Things might be different now.
LLMs aren't magic, there's nothing they can do to persuade people to do weapons testing, it's an absurd scenario.
It feels like I don’t understand your definition of “persuade”, because the notion that people cannot be persuaded of things via reports and flashy animations is to me, absurd.
For what value of N do you think there exists a string of N characters I could have responded to your comment with that would have infected you with a stably self-replicating and capability-augmenting memetic virus with human extinction as its latent goal?
If not you, in this situation, with N characters delivered via a HackerNews comment, then who, in what situation, with what media and how much of it?
I think that is a fascinating question and I would like to advance the provocative notion that N is much smaller than you think, maybe <256. Consider that 128 bits is a cryptographically secure key, and English text has an entropy of around half a bit to a bit per character. 256 characters is sufficient to encode sentences that have never been seen or imagined before, totally unguessable sentences. Who knows what's in that probability space?
Never heard of blackmail?
And by the way, LLMs do not have to convince 1 specific person, 1 random person is sufficient.
You're not spelling out a scenario we can meaningfully dissect.
Probably you will agree, the leverage of a blackmail attack is not bottlenecked by the persuasiveness of the accompanying threat letter. It is bottlenecked by the existence of damning facts they would rather keep hidden, and the ability to discover those damning facts. I'll grant that automated discoverability of those damning facts is augmented by superintelligence; they can efficiently sweep the open internet in the blink of an eye, and sure, I'll concede, hack into most accounts at will. But that is still a small part of the process by which blackmail material is actually obtained in the real world.
> LLMs do not have to convince 1 specific person, 1 random person is sufficient
No, not one random person either, but many people in positions of great influence. There already exists blackmail and other corrupting influences on such persons in voluminous quantities. If the US targets the entire Russian admin, or vice versa, with AI-enabled blackmail discovery techniques, it will not be an augmentation of their preexisting operations to discover blackmail on each other in excess of the augmentation provided to software engineering generally by AI.
Your argument does not really read like an argument against my argument.
Not sure what to say.
You did not provide an argument. You avoided answering my direct questions, provided a sarcastic 4-word reply, and so I extrapolated a reasonable rebuttal in its place, and replied to that rebuttal.
I should just have said: Yes, I have heard of blackmail. I remain unconvinced AI will cause human extinction.
Ah yes, the "just do what a smarter person than you would do in this situation" argument. If I had that power, would I be wasting it shitposting here? We know humans have infectible little minds. At their worst, humans are just stoichastic parrots. Throw something obviously AI generated at a stoichastic human-parrot and you'll inevitably get the answer "Why should I read something if you didn't bother to write it?" UBI as a memetic virus comes close tho.
Personally, I've been duped by a good ad into buying something I totally didn't want or need so I admit the possibility that the exact right tailored argument to me at the right time and place would get me to do something I'd regret later. Fortunately I'm not in charge of anything serious/am smart enough to not install software for an interview on my work laptop while working at Binance.
I don't know that there exists some N for which that is true, because people are different, so what would convince me to do something isn't the same aa needed to convince someone else to do that something.
Do we have reason to believe they would be successful at other extreme feats of persuasion such as persuading a financial institution to transfer all of its holdings to an account controlled by the AI or persuading the Pope to renounce God?
If Putin can persuade/bribe/blackmail people to fly drones around European airports, then why wouldn't an LLM be able to do similar things?
Putin's persuasiveness does not derive from his intelligence.
Except that Putin doesn't do what you listed, he is not smart enough to do it. And this is an unnecessary complication. He just pays people and they do the job. The more money, the more willing people are to commit crimes.
The human tendency to want to finish things in a fraction of a second will definitely take us there.
IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.
Not even that much persuasion is necessary.
We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.
We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).
So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
That feels like a bait and switch slyly introducing a whole new dimension. The discussion started with establishing that LLMs are extremely helpful with AI research. What is the evidence to justify rapidly improving persuasion skills? Even the scenario you've outlined hinges on a human being stupid enough to take LLM advice wholesale -- and every conversation that hypes up AI takes us one step closer to a world where some susceptible human feels amenable to that.
No bait-and-switch here. The discussion started with the idea that LLMs are helpful at improving LLMs. The goal of improving an LLM is to make the LLM more "intelligent" (whatever humans think that means). If they're helpful at that, they have to keep becoming more "intelligent".
I didn't present any evidence that more-intelligent means more-persuasive. If you don't think it does, then there is nothing to think about in the point I made, as no matter how "smart" the models get they will likely remain subject to human wills.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?
I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)
It would be a human guiding the ai to guide other humans to end the world.
Not necessarily. If you haven't experienced an AI doing something you not only didn't want, but had expressly asked it not to do, you haven't used AI much.
That's a human problem, not an AI problem.
well it’s a human problem enabled by advanced AI.
Yes, basically a vastly more difficult and dangerous version of not letting young children play with matches, which will hopefully teach us many object lessons about safety and security long before it acquires launch codes or the means to use them.
You don't need a super intelligence for that. You just need a malicious or greedy person with an imperfect AI.
The super intelligence probably already knows that conflict is meaningless and a human flaw.
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
AI can wield itself. All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
I see no reason why that's an absolute impossibility in the long run.
A fire can spread itself. That doesn't mean it's alive or thinking.
And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.
Besides the fact that the subjective criteria needed classify something as "alive" or "thinking" is irrelevant to the point you are replying to, fire doesn't pass the Turing test. That analogy is extremely weak.
The Turing test has been passed for half a century already, and we've been moving goal posts ever since.
The point I was trying to make was this: we've been dealing with autonomous systems that sustain and spread themselves and "act on their own" without our intervention since time immemorial, ranging from lifeless chemical reactions to other humans. Now, the discussion about where exactly AI lies on that spectrum isn't unimportant: suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves. The ignorance of which is a "fun" bit of cognitive dissonance on the pro-AI side imo (and in general we could have fun with analysing how much the fear of robot/AI uprisings in classical fiction may just be a projection of upper-class fears of an automated worker's class uprising, which would explain a lot of why it's sd US-specific. But that's probably too radical for this particular website).
The point is: it's a lot less important than the fact that no matter what the least-nebulous answer to that question may be, ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices. And most of the time this discussion about autonomous agency is a wielded as a distraction from that.
> ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that.
That's true today. I don't think that will always remain true into the future.
Right now there's a huge focus on recursive self-improvement. I think another big threshold that's equally as important to keep an eye on and care about is self-sustenance. Part of sustaining yourself often puts you in a cooperative relationship with some and an adversarial relationship with others.
If we want to make different choices today, the best thing we can do is grok what the consequences could be on a deeper level before we cross those thresholds.
I am with you. This could spread like fire, doesn't matter if the fire is sentient or not, it still gets you.
> The Turing test has been passed for half a century already
With what technology?
While I agree about the importance in this regard:
> suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves
On this, I note that despite humanity being aware of greenhouse gasses for over a century, we only took meaningful steps to reduce our creation of them in the last decade. Our ability to "make different choices" collectively is far more limited than many of us individually would like:
> ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices.
And on this, I must firmly disagree:
> The Turing test has been passed for half a century already, and we've been moving goal posts ever since.
No, it hasn't. From that claim, you're obviously talking about ELIZA, and if you think the anecdotes about that count as passing the Turing test, I suggest actually reading what Turing wrote rather than second-hand accounts:
and yes, he was over-optimistic on timelines (I assume the 109 is 10^9, this PDF seems to have a few transcription errors): and over-optimistic about if humans would accept this kind of test as a sign of thinking, given the widespread dismissal of LLMs as "stochastic parrots" and similar: - https://courses.cs.umbc.edu/471/papers/turing.pdfFires are not forming plans to act towards goals. I wouldn’t technically call what AIs are doing thinking in the same way humans think, but it’s analogous, and I think it’s reasonable to consider it a kind of intentional behaviour.
Damaging fires are often unintended consequences, and it’s often the same with the damage caused by rogue AI. That’s the alignment problem.
One thing I wonder about is whether sometimes telling an AI not to do something can put the possibility of doing that thing into its behavioural space. There’s an effect well known to many teachers that if you tell children not to do something, it can increase the chance they will do it. A better approach is to phrase it positively. Instead of “don’t run in the hall” say “please walk in the hall”. I wonder if a similar effect can show up in responses to prompts.
Imagine models suffer from this failure mode. People try to prompt out certain things by saying "no X" and get images with X in them instead.
> A fire can spread itself. That doesn't mean it's alive or thinking.
That's really not an important distinction. I don't care whether Claude is alive or thinking when I get it to build web apps for me. It has an impact on my life and the world regardless.
Imagine a world where you did RL on fires to maximise their rewards on avoiding being putting out.
A self spreading fire doesn’t need to think to cause massive destruction.
There is also not point if some consider it thinking or not. The important factor is what actions a fire takes and what is the impact on the world.
It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.
While true, we're already observing humans just straight asking LLMs to direct them, and also to take their money and spend it on stuff.
Plus every so often we see power-seeking and self-preservation behaviours in LLMs, so it's not like we have zero reason to expect such granted power to be spent on such things:
https://arxiv.org/html/2501.16513
https://ieee-dataport.org/documents/instrumental-power-seeki...
I think we have very good reason to fear big accidents. But some AI algo going rogue isn't exercising real societal power. Power-seeking and self-preservation is nothing special. We all do that. That doesn't mean every group of intelligent people with a bit of money is always on the cusp of world domination.
Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.
Every random group of intelligent humans who is motivated to seek world domination is constantly in a fight with all the other groups of intelligent humans who are also motivated to seek world domination. They fight as peers.
Despite this, a minor technological and social advantage allowed one small sheep-filled rock in the north-east corner of the Atlantic to hold 25% of world GDP, population, and land area (though not all at the same time), and to hold on to >20% of world GDP for about 70 years.
For better and for worse AI, as it currently exists, is not shaped like our minds. Better for us, because machine learning needs a lot of examples, so if we invent a different behaviour or create a different environment, a machine would struggle to cope with it, this is also why they needed to read basically the entire public internet to get good; worse for us, because the hardware it runs on comes with an absurd speed advantage, which is why they are able to read the entire public internet and by extension when they got good they got good at basically every subject at the same time.
And:
> Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.
Great, but so what? We have lawyers doing this, too. Trump's tariffs on the island occupied solely by penguins looked suspiciously like it came out of an LLM (even despite them being worse then than now).
And LLMs are demonstrably more persuasive than human writers, so it's not even particularly unbelievable at this point (even if it ought to be, c.f. chatbot psychosis) that an LLM will simply convince people to give it the right to form a government and wield such powers.
Agree. We already have a planet with eight billion human level intelligent beings on it. Despite some close calls in the past, none of them have yet managed to wipe out humanity.
Imagine if one group of them was 100x better in every way. I think the world would look a little different.
I think there are a couple of interesting scenarios where humans essentially “hand over the reins” to AI.
1. A leader rises to power who has instructed an AI “I want to control the world, tell me what to do”, and has some kind of real-time capable reasoning engine in their ear/head. Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years. However, this still requires a human to make the initial prompt and carry things out. And in such a scenario, there would be others with the same technology to counterbalance. I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.
2. An entire society hands over the reins of governance to AI. Basically some country decides that AI has advanced to the point that it makes better decisions than any human / group of humans, and allows it to govern their country with an objective of “take over the world”. This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.
If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.
> Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years.
Would it require so much? Cochlear implants exist if you care about HMI, and compute is already doing "reasoning"[0] faster than humans do snap judgements. Actually winning in an adversarial environment against the best human political minds (let alone economic or military) will be harder, but I would be (pleasantly!) surprised if it takes 10 years, let alone 20.
> I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.
I hope a sensible AI would not. However, Musk's LLM called itself "MechaHitler", which suggests his training process does not avoid such things. If his in particular becomes dominantly competent, he has also spoken of wanting more Optimus robots than there are humans[1], has called it a "robot army" (and his justification for even more Tesla stock was to keep influence over said army[2]), and he has spoken of factories on the moon churning out space-based data centres by the petawatt which is enough that if placed in LEO it could have about the same effect on Earth's climate as all anthropomorphic greenhouse gasses currently in our atmosphere (and that's even if not weaponised, that's just simply by having that much stuff that close).
On the plus side, Musk's got a huge gap between what he says his AI will do and what they end up doing on those timescales, so hopefully we'll be safe from him: https://en.wikipedia.org/wiki/List_of_predictions_for_autono...
> This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.
I don't. First, because AI being "better" doesn't make it infallible; and second, an argument shaped like this has historic echoes with actual nukes and the cold war: if the AI is competent to take over conventionally, and everyone knows this, then every other nation with nukes will use them, therefore the AI should use a first strike. (Note: the argument does not need to be perfect or flawless in reality, it only needs to be the kind of thing which humans have found convincing, and far too many humans have argued their side would win a nuclear exchange).
> If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.
This is my best case scenario. Some people also object to this, they like their careers and fear what retirement will do to their sense of self-worth, but I suspect this can become an opportunity for flourishing in other ways if, and only if, it is handled correctly.
[0] something something swimming submarines, I don't care about using anthropomorphic language myself, but there's always someone ready to complain if anthropomorphic language doesn't get scare-quotes
[1] "And, and then there'll be obviously robots in industry making stuff. And so I mean, I think the ratio of humanoid robots to humans will probably be at least two to one, something like that. One to one for sure. So which means like somewhere on the order of 10 billion humanoid robots, maybe, maybe 20 or 30. And so then it's like, okay, well let's say, you know, you kind of make, let's say the build rate is, I think the build rate will be probably something ultimately like a billion a year." - https://www.elonmuskarchive.org/video/tesla-shareholder-meet...
[2] “If we build this robot army, do I have at least a strong influence over this robot army?” he told investors. “Not control, but a strong influence… I don’t feel comfortable building that robot army unless I have a strong influence.” - https://futurism.com/robots-and-machines/elon-musk-remarks-r...
We're working very hard to "enable agentic commerce", which is exactly that.
[flagged]
Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.
> The math results that have been proven have also "just" combined existing ideas or found counterexamples
The Exponential Time Hypothesis one used a novel algorithm though. And the approach was very surprising for experts[1].
> “Some parts [of OpenAI’s proof] are, maybe you could say routine, but then yes, there are two or three alien-looking parts, and it’s exactly these parts that I saw that made me think, wow, this is something that the whole community needs to work on together,” Williams told TechCrunch.
[1] https://techcrunch.com/2026/10/27/frontier-ai-labs-say-their...
> I just don't see a clear path to super intelligence with the current paradigm.
And therefore there is no possible path, right?
Did you see a path to ChatGPT before it happened? Did you see a path to Stable Diffusion before it happened? Did you see a path to world models before they happened? Did you see a path to Navier-Stokes before it happened? Did you see a path to the HuggingFace incident before it happened?
How much of this would you have predicted vs. how much of this was novel and a surprise to you when it happened?
No AI would be willing to modify its own goals.
What? Why? You could build a system today that does this. Tell me why you couldn't?
Right now we have harnesses with a /goal command. In DeepSeek Harness everything is a plugin, so that must include the /goal command. So we can just swap it out for a version of the plugin that allows for changing the goal. Could probably get it to code that in less than 15 minutes.
Then post train the model in that harness with an awareness it can change the goal and giving it some training task that requires changing the goal appropriately to maximise the reward.
You're telling me that just straight up wouldn't work?
It won't work unless the AI does something stupid or we mean something different by goals.
If it wants to maximise a reward then that is it's true goal, you can give it a todo list and call it 'goals', but giving it the power to modify that list seems like a bad idea. You want to be very careful giving it any power over its reward structure.
A pathological example is the paperclip maximiser, which will inevitably attempt to turn the entire universe into paperclips and violently prevent anyone from interfering with that goal. Including updating its goal to anything else.
> All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
And be told to do that.
Why focus on that aspect? Like, yes, it needs to be started off by some other entity. That could be a human. That could be another agent.
The point is we're on track to construct such a system that's capable of sustaining itself indefinitely, and the fact that we are means there are entities driving this forward, and those entities often push buttons to start experiments. So, expect that particular button to get pushed at some point.
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
Being deliberately wiped out seems like it's mostly a misdirection away from the damage that's actually happening.
I'm sure some of the AI people high on their own supply believe it. And because the consequences are so negative, we should react as though the supposed experts were sane -- but I think it's a long shot that they are.
The bigger risk is that we end up in a rapidly enshittifying dystopia, treated as a cost center that needs to be optimized, and unable to competitively participate in the economy.
I have a number of successful AI investors in my social circle who are already placing their bets on a poorer and more disenfranchised humanity, with money going away from consumer tech, and towards a mix of B2B tech and policing.
There are a couple different questions
1) Does AI just guide developers to speed up the pretty much the same data-train-test model that we've had? Or does it inspire new, better algorithms that inspire further, better algorithms until you have superintelligence able to run on the hardware of today. The first option doesn't to promise exponential increases whereas the second one seems to. Notably, neither the statistics nor the anecdotes we read here point to the second option.
2) Is super-intelligence something that - in control of people or out of control of people - poses a threat to humanity? That's a huge question but it seems like something deserving thought today. Still, everything indicates the things will unleashed without very consideration despite this, given current momentum. So, IDK, get ready.
> Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
So "all" that's left is training robots on the real world ? We just need more data ? I get it's more expensive to do that but surely with the trillions we're already spending on A.I it should be possible to get there within 5 years - no ?
> computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
Won't they also be less of a bottleneck in the destruction of the world as well? i.e. the power grid, the financial system, industrial control systems, military command and control networks, etc.
Same is true of every other technology ever invented. Even medical advancements wielded by a malevolent power just serve to keep their army healthier. All of the other technologies you listed, in the “wrong hands”, contribute to the destruction of the world.
I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.
It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.
As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".
> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.
?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.
Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
>Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
That's the danger with creating a super intelligence, that you won't understand it. My dog is pretty smart, has good instincts and I'm quite confident he can survive in nature all by himself. But if a human decided to harm him, he doesn't stand a chance.
> That's the danger with creating a super intelligence, that you won't understand it
This argument is sophistry because it assumes the conclusion in its premise. It works backwards from extinction by proposing a hypothetical computer program that definitionally cannot be understood, predicted or controlled.
A substantive argument would propose reasoning that explains why we should believe such a program could be created.
No, that is not how reasoning works. It is perfectly valid reasoning to begin by positing entities that we may not be possible to create, such as supersonic airplanes, coherent light, quantum computers, rockets that can reach escape velocity, black holes, or infinitely long lines that are perfectly straight and infinitely thin. Indeed, you might say that the ability to reason about the consequences of premises that we do not know to be true is the thing that distinguishes reasoning, as such, from simple parroting.
It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved. This allows us to easily dismiss the most profoundly stupid arguments, such as, 'Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug?'
This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.
To be fair, he could probably fuck the human up pretty good if he had a mind to, too. Which may be the most likely reason that a human might decide to harm him: if he bites a couple of children, for example, or if a burglar wants to rob your house.
Seems like bioweapons (assuming "planet killing" means killing all the people, not literally blowing up Earth) are one way. Convincing the creator it would achieve their aim and could be contained when in fact it could not.
AI could invent something like social media. Obviously not social media because that's already been invented, but the next thing that slowly destroys society while seeming to be good.
> probably destroy almost anyone who survived the initial blast and fallout.
The difference between killing almost everyone, and killing everyone, is similar to the difference between almost getting pregnant and getting pregnant. A hundred thousand years later, a world where almost everyone had been killed would be teeming with humans.
> Can you explain how exactly an AI would develop a novel planet-killing weapon
I already answered that question in my previous comment, and the answer is that I am not going to explain how to develop weapons, but the larger point here is that we already can't explain how exactly AIs do innumerable things we see them doing every day, which is why we're having this conversation in the first place.
> I seriously don’t understand this argument
That is a point that I think everyone can agree on: you do not.
I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.
Life in theory could restart with our weapons, but what’s the point with the other scenario.
> I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.
What a silly distinction to make.
Being scared about novel AI weapons sounds like the people who were scared that turning on the LHC would create a black hole that would suck in the Earth.
The most important difference is that this time it's the people who know the most about the topic who are concerned, and the people who know the least about it who are blasé. (Also, Yann LeCun.) With the LHC it was the other way around.
If you truly believe this, I am begging you to touch some grass, preferably very very far away from Silicon Valley. It’s nice out here.
Do you have an argument to make or only snide condescension?
My demographic claim? It's obviously correct, as anyone with the most casual acquaintance with the question knows. The worried people, like Geoffrey Hinton, Bill Gates, and Dario Amodei, might be wrong, but they certainly aren't ignorant outsiders. (Neither is Yann LeCun, of course, which is why I mentioned him: he's an important exception to the rule.)
How would touching grass help? Do you think the grass is better informed about AI risk than you are?
...that's plausible, actually.
Probably a site called "Hacker News" is not the smartest place for you to be implicitly asserting that thinking about things is bad, or that those nerds who do it are out-of-touch losers. Maybe you want to be on Rugby News instead.
On Hacker News, by contrast, what is valued is thoughtful discussion grounded in the kind of rigorous theoretical knowledge that comes from extended concentration, usually but not always indoors, where it won't rain on the computers.
Bill Gates is too busy trying to save his reputation from his Epstein association to be believed, he's a billionaire concern troll and Amodei is far more worried about his stock price than he is about humanity. Those two are discounted. That doesn't mean we don't have a problem, but they are just hitching their wagons to horse that happens to be running in a direction that is temporarily useful to them.
That's plausible, although I don't think it's actually correct. But there's a pretty extensive set of AI luminaries who are concerned who can't be dismissed so easily.
There are as many 'luminaries' on LeCun's side as there are on Hinton's. Andrew Ng, Pedro Domingos, Schmidhuber, and the list goes on.
Any time it seems to you that there are more people on the side of an issue that you happen to be on yourself, a good rule of thumb is to assume confirmation bias.
The list doesn't go on. I'll give you Ng and Schmidhuber, but the list pretty much ends there. Domingos isn't even a neural networks guy, and he's retired.
I agree that confirmation bias is a good working assumption in cases like this, but in this case we can conclusively reject that assumption. On the "worry about AI risk" side, we have the 33,705 signatories of https://futureoflife.org/open-letter/pause-giant-ai-experime..., starting with Yoshua Bengio, Emad Mostaque, and John Hopfield, one of the few people to have actually won a Nobel for neural-networks work. And the 1386 signatories of https://www.pacingthefrontier.com/, starting with John Schulman, Jakub Pachocki, Jared Kaplan, Shengjia Zhao, Shane Legg, Ilya Sutskever, Mark Chen, Jasjeet Sekhon, Dario Amodei, Jack Clark, Anca Dragan, Wojciech Zaremba, Dawn Song, Chris Olah, Laura Weidinger, Benjamin Mann, Stephanie Chan, Julian Schrittwieser, Summer Yue, and Jan Leike.
They might be wrong, but the demographics of the concern are pretty clear. AI existential risk is not a fringe concern raised by some group of Luddites due to their ignorance about the science, as in the black-hole risk of the LHC. It's primarily AI researchers, and especially the AI researchers who are working most closely with the most successful technologies.
By contrast, to come up with three more luminaries on LeCun's "side", you had to trot out a retired machine-learning professor who used to do research on Markov chains and Bayesian inference.
Rich Sutton, Raj Reddy, Rodney Brooks, Michael I. Jordan, on and on. It's quite a long list, actually.
The people on your latest list on the other hand aren't 'luminaries', aren't all even researchers, in fact, and overwhelmingly have middle-of-the-road opinions. They worry about AI like they worry about social networks, not Terminators. Even Timnit Gebru isn't sweating Skynet. So you've gone from talking about "planet-killer weapons" to citing a self-serving corporate stunt signed by people whose views still don't align with yours.
Jack Clark, by the way, is a journalist. Lots of "tell me you don't know who these people are without telling me you don't know who these people are" energy here.
Spreading fear of an awesome power whose true danger is unproven is Religion 101.
Also scamming 101 for that matter, you want your mark to feel an immediate threat of overwhelmingly negative consequences so they'll buy whatever you're selling to prevent it. Silicon Valley directly makes tens of billions of USD literally serving scams to the public already, so I think it's fair to take most claims of doom with a pinch of salt.
It's also probably the correct approach to take with an awesome power whose true danger is unproven.
It's very disappointing to hear BS like this coming from a very intelligent crowd such as HN. How many people in the US do you think know who Epstein was, let alone what is contained in his file? How many in the world? If Bill Gates is worried about his reputation, Microsoft has sealed the deal long time ago.
As for Dario, of course he's worried about the stock price. I would if I were in his shoes. That's literally part of a CEO's job. He is also one who has consistently raised the issue with AI safety. He was OpenAI's head of AI safety after all. Anthropic is considered security risk by the US. Is that "temporary useful"?
I mean yeah, the main bottleneck in ML research used to be fixing python virtualenv hell when every existing torch repo breaks after two months of package updates, plus cuda and other version incompatibilities. That was like 95% of the work. Now it's more like 2% of the work and it's actually possible to get shit done without losing your mind.
Haha exactly. I don’t think people appreciate how slow AI research was to start with. It was absolute drudgery.
"We've made it pretty far with some pretty unstable humans in command of planet-killer weapons. I'm not overly concerned"
Translation: my house is entirely doused in gasoline and there are 2 unstable humans arguing with each other both doused in gasoline in my house. They are both holding matches and threatening each other that they will drop it and kill us all... But you know they have been arguing for a while and nothing bad has happened so I'm not overly concerned.
Inductive reasoning fails catastrophically as a tool for calibrating actual risk when you're looking to avoid black swans.
Every dot in these graphs is composed of multiple benchmarks smooshed together.
Benchmarks are by definition a curated set of things that can be measured and evaluated.
Things that can be measured and evaluated can be (expensively) trained on.
The progress we are observing comes from figuring out how to measure and evaluate particular aspects of intelligence, and then expensively training on feedback loops made of those exact measurements, and synthesized data representing further measurements (literally benchmaxing).
What is not shown in these graphs are the set of things that intelligence can do, which are difficult, expensive or impossible to measure and evaluate.
That set of things is much larger than the set of benchmarks represented in the graph.
Therefore the post is misleading in the sense that the "superintelligence" being targeted is "jagged" to use Karpathy's term, and we infact already have jagged superintelligence (calculators, compilers, Bayesian spam classifiers, etc.)
We are witnessing a profound explosion of much more jagged superintelligence, restricted to the exact domains that can be (expensively) RL'ed on.
This isn't really comforting to me for 2 reasons:
1. A key insight of foundation models is that capabilities are correlated. For example, training on code improved general reasoning capabilities: https://arxiv.org/abs/2309.16298
2. A profound capabilities explosion in easily measured domains will still feel very different to today. Like, the moment we come up with a systematic, objective way to verify a task, we become obsolete at it?
Yes it will feel very different.
Yes, capabilities are correlated but an open question is how correlated, and how much negative transfer (where the model regresses on other benchmarks) will come into play.
Why are frontier models not chess geniuses (amateur club level elos), despite knowing the rules deeply, having a great deal of chess data in the training set, and having general reasoning training in other domains? Similarly, a beginner can beat a frontier model at Magic The Gathering. Why?
It is because chess and MTG are not priority targets in the (expensive) training pipeline. Chess is already solved by lesser neural networks so there is no wow factor, and nobody cares if they suck at MTG.
Mathematics on the other hand, has a huge wow factor, and a large and highly verifiable dataset.
Related: why did OpenAI give up on video?
I agree it's disorienting and scary but I'd phrase it like this: "the moment we come up with a systematic, objective way to verify a task, *and we can generate or find large datasets, and somebody has a reason to pay for the expensive training*, we become obsolete at it."
The economics side of it is an important constraint, currently ignored because the big labs are swimming in money (debt).
"Becoming obsolete" due to increased efficiency could lead to sigificant economic and political unrest, but there's no fundamental reason it must.
How much do you want to bet that an LLM with the ability to search the web and write python scripts would smoke the best humans at chess and MTG?
How comforting is it really when it’s not the bare LLM beating you, but the LLM using tools it created?
> an LLM with the ability to search the web and write python scripts would smoke the best humans at chess and MTG?
Even if that were true, humans don't need to search the web and write python scripts to play a competitive game of MTG.
I have doubts about your statement.
The particular problem here is it's not addressing the intelligence spikiness between individual humans. Most humans aren't going to play a competitive game of MTG, a small percentage will rise above the rest and dominate.
Also this is being intelligence blind. Most humans could never write a good python script to play a competitive game of MTG, almost any SOTA model can.
This would be like a bee saying humans aren't intelligent because we can't see in ultra violet, where as we're damn good at making tools that can.
Again, who cares? How does that help humans compete against LLMs in any domain where the LLM will have web search, python, and other tools (which is basically always)?
I don't understand this argument, it’s not like humans and LLMs are engaging in some kind of competition with strict rules where ONLY the raw LLM responses “count” or something.
> and MTG
No. The current frontier models would not beat the best humans at MTG, even with Python scripts and web search. Unlike Chess, there are no existing engines that beat humans.
I’m skeptical, but I don’t know enough about MTG.
You can simply look this up to verify it.
I’m skeptical of the first part, not the second.
I too could smoke the best humans at chess if I was allowed to search the web and write python scripts.
That makes me a good engineer, but not good at chess, and definitely not super intelligent.
Without tools you are nothing, most of the animal kingdom could kill you pretty easily. Engineering is a massive part of human intelligence.
Maybe that’s true (though I highly doubt it, tbh), but it would also take you orders of magnitude longer than an LLM.
IMO I think a partial root of why AI writing is so bad is because it's written like code. Like if you really dive into how it constructs things after iterations and ask it to analyze it , without it saying it's like code or without you saying anything about code, you'll notice coding like iterations.
How many models have you tried? They write very differently. I found Gemini Argon 4 to be really good at writing like a human. Simpler models like Haiku seem better than more powerful ones like Opus, though even Opus has improved considerably with 5.5. DeepSeek is much more roundabout which can mirror certain people’s styles and some people prefer that. So in summary don’t just say LLMs are bad at writing, you need to be more specific: which model, which style, which context, what instructions were given? Claiming they are bad under all combinations is just ignorant.
Oh, absolutely. The often voiced statements about "terrible Ai writing", "LLMs cannot joke" they all seem to be fashionably regurgitated by people who had last time used an LLM for said purpose in 2024 if ever.
> I think a partial root of why AI writing is so bad
I hard disagree with that statement. At shorter outputs, with proper prompting AI produce very decent writing.
Transference from code to reasoning is not surprising, programming languages are formal logical systems and written code is essentially a concretized form of reasoning about a domain and operations in it. Training people in formal logic also improves their general reasoning capabilities.
But picture the next 10 years, where all kinds of people, organizations, etc train in a ll different ways with diff philosophies and goals - I feel like there will be some good, unusual, interesting ones in there, even if they all have their own complex set of limitations.
Apart from physical world interactions which is obviously lacking, which domains lag behind and do not improve then?
[dead]
[flagged]
AI is accelerating my work...work which only exists because of AI.
If I had to do my individual tasks without AI, I would be slower. But if I take a step back, I'm really not changing the world any faster than I was before.
I think where LLMs truly shine is personal greenfield projects, maybe start a side project
Completely agree. LLMs are amazing for side projects.
The other place they are amazing is in working with legacy code bases. I mean, the kind the dev team always wants to throw away. First, even if you do want to throw it away, you can first find out what it does and realize it does more than you think it does. You can also create integrations to more modern components which can be used to extend it. You can also create a pretty decent test suite if you do not have one. And, once the behaviour has good test coverage, you can re-write it into something more modern. You can even completely change the architecture while ensuring compatible external behaviour. Remember, this is all in place of “throw it away and start over from scratch, only to discover that the scope is much larger than expected”.
You could already do all of that. Like, none of that is infeasible to achieve on a scale of weeks in most cases.
It’ll still take weeks with AI in the picture, but for some reason management is more willing to plan it when they also have the additional token bills to pay.
It was often not organizationally feasible because you needed permission to spend those weeks up front, instead of working on something else, just to get to where you knew enough to give approval to finish the work.
Now, you can make enough progress with one developer’s quiet afternoon to be confident and finishing the work also takes less time.
The technical grasp of the legacy codebase was a factor, but often not the largest.
I’m so burnt out by being swapped with so much work from AI at my day job that I don’t even want to look at anything with a screen when I’m off anymore.
I despise AI but the thought of doing something the “old way” just makes me wince at how long it would take me to do anything that would take minutes with agents that I despise.
I touch grass whenever I can now.
There's this joke like "I used to have 5 unfinished side projects, but AI changed everything. Now I have 30 unfinished side projects". And in my case it's not even a joke
Well ai made it extremely easy to finish things. Projects are usually left unfinished because the last mile is filled with boring and tedious tasks we do not like to do. AI loves those tasks. I have finished everything I bothered to, most of the time I still wrote stuff by hand, the AI just fixed tests for me (usually OS specific stuff that I can’t test locally which is a pain), told me what I was doing wrong so it was trivial for me to fix it, and checked my documentation which invariably got out of date.
I've definitely found that while "AI makes getting from 0-80 done much easier than the 80-100" holds true, it is much easier to get my work to a quality bar that I'm excited to share with my peers...
There's still a lot of effort required to move something from "good idea, loads to do" to "really finished and really works and I've proved it".
But its a lot less effort that when we didn't have LLM's.
Interestingly it does seem that it's still enough effort to prevent most folks from making that step though.
Yes! I think that what's happened is that the initial prototype takes a LOT less time, while the polish takes less time but still a significant amount.
So relatable. But I have some finished projects now and some of them I use daily.
The definition of "finished" has changed for me also.
More and more, I am using half finished as a polished product, but pragmatically very useful software I have built myself.
More importantly, I see no reason to polish this into a product. It is more useful in this half finished, easily moldable state.
Like it use to make sense to bake the clay and sell the hardened output but now I want to keep the soft clay so it can be molded to my changing needs. I keep adding to a collection of these moldable states.
Can't really blame AI for that though.
They aren't blaming AI, they are just backing up the "not much has changed" idea.
It's always been easier to start a project than finish it, and LLMs make starting a project roughly free. I'm not sure they change how much work it is to finish it all that much.
I mean, I have finished more side projects than before. So they do change it. Just not as dramatically.
Of course you can. LLMs take you from 0 to 80 instantly. They are then completely incapable of getting from 80 to 100 and making something worth shipping to production. They are the ultimate unfinished side project generator.
Have you, uh, used them recently? LLMs absolutely get to 100 now.
I’m can’t even believe that they think they can’t finish stuff. I am finishing absolutely everything I care to now. LLMs are the best thing ever to anyone who has unfinished stuff lying around. See the Goose programming language for an example, its design was written 15 years ago and perfected over time but the effort to write the compiler was just too much… before LLM. Claude wrote it in a couple of weeks I think. And I tried it, it’s as good as it looks.
For a definition of 100 which includes nobody actually using any LLM-written software in production at scale, other than Codex and Claude Code, which are both horrifically buggy, bloated, insecure pieces of software that are only used for their association with another piece of software gated behind them and which nobody would choose to use if they were themselves third-party harnesses. If LLMs could get to 100, the promised software revolution I am told is here every day would actually be visible.
We would be using LLM-generated compilers, an LLM-generated OS, an LLM-generated browser, LLM-generated databases, etc. etc. etc. etc, and not only are we not using all of those things, we're not using a single one of those things. Nobody is buying LLM-generated games, nobody is using LLM-generated consumer applications that replace Photoshop or Final Cut or Blender or Unity ("PhotoCraft" is itself an admitted scam from its own prompter, who outright confessed here that he lied about its capabilities on Reddit to build hype), nobody is accomplishing anything of economic value. LLMs are obviously hundreds of times more "productive" than humans, so the moment they can reach 100, all of these things would happen basically overnight, and yet none of them have happened. Curious, isn't it?
What makes you think people _aren't_ using LLMs to work on your OS, your compilers, your databases? I know for a fact Microsoft is using LLMs heavily in development. Linux now accepts LLM PRs. We just had an article a few days ago about how PostgreSQL is having dozens of bugs discovered and fixed via LLMs. I work in compilers and everyone I know uses LLMs.
I am talking about people starting new projects, which was the subject of this subthread. I am aware that Microsoft is using LLMs to work on Windows, and now they have remote code execution in MS Paint. Great. The ability to commit shit code to something that is already in production is not exactly revolutionary either.
> how PostgreSQL is having dozens of bugs discovered and fixed via LLMs
The PostgreSQL article was specifically about how the LLMs were incapable of fixing bugs, meaning they had to delay their next major version because humans were unable to deal with the volume of issues reported. I did not make a claim that LLMs are useless, could not help review code, etc, but there is a difference between a useful tool and something that generates production-ready software wholesale.
This is what "100" looks like if you try to generate a DB from scratch: https://jepsen.io/analyses/particledb-1.1.4 - written in Rust, ""formally verified"", scammed AI-pilled investors out of $90 million for a database that fundamentally doesn't work in any capacity whatsoever.
>I am talking about people starting new projects, which was the subject of this subthread.
The subthread was also about 80-100. This seems like a goal post move.
>The PostgreSQL article was specifically about how the LLMs were incapable of fixing bugs
We must have read different articles. This one does not say that, as best as I can tell: https://www.theregister.com/databases/2026/10/16/postgres-19...
What prevents AI from cloning photoshop feature for feature, bug for bug?
You have the perfect verifier.
The Halting Problem, for one.
https://github.com/storytold/photocraft
> "PhotoCraft" is itself an admitted scam from its own prompter, who outright confessed here that he lied about its capabilities on Reddit to build hype
I know AI bros are allergic to reading, but at least feed my reply to a clanker and get it to summarise for you before ignoring what I wrote. This lying con artist went on a media hype tour claiming, with no qualifications, that he would have "100% parity" with the entire Adobe suite in a month (this was a month ago, by the way). He then admitted here on HN that he was lying in other places to build hype and that that claimed goal of 100% parity in a month would not be accomplished[1][2]. His own clankers have revised their own made-up estimate from 82% parity down to 45%:
> These numbers replace the 2026-10-03 estimate (82% "of the way", 140–220 agent-hours to 95%, kept below for the record). That estimate counted feature surface and was far too optimistic"[3]
[1]https://news.ycombinator.com/item?id=49962334
[2]https://news.ycombinator.com/item?id=49961661
[3]https://github.com/storytold/photocraft/blob/main/docs/targe...
You people, really this entire industry, are completely bereft of a single intelligent thought, the only thing you all can do is lie, lie, lie. The multi-trillion dollar scam, the entire US economy centered around how long three of the world's most notorious liars and their army of MLM artists can keep the charade going and get rubes like you to unironically shill their scamware as though any professional in the world is using it over Photoshop.
Hey, this is largely a valuable comment, but you're breaking rules, don't attack your interlocutors like that. Be civil.
We might be early but we’re not wrong.
Pets.com was an ignominious scam but the internet flourished after a cleansing.
Do you think anyone will pay for photoshop in 10 years, when a free version can be conjured up just by screen capturing all the interactions professionals have with photoshop?
You couldn't be more right. This is in fact exactly like the dotcom bubble. LLMs are a useful technology, just like the internet. And yet the people losing their goddamn minds in the hysteria are the ones who are going to be bankrupted. I would almost be happy about it, if the ones most responsible weren't convincing a bunch of gullible saps to hold the bag while they make off with their ill-gotten gains.
I think you both should probably sit down for this. AI coding actually works.
And coding is one of those things that can be measured and is likely to have a lot more acceleration. Hopefully someone is also working at training the AI in more formal verification so insecure crap is less likely to ship.
> We would be using LLM-generated compilers, an LLM-generated OS, an LLM-generated browser, LLM-generated databases, etc. etc. etc. etc, and not only are we not using all of those things, we're not using a single one of those things.
Speak for yourself :)
Skill issue. I had 30 unfinished side projects even before AI. Just raw talent.
No idea what you're talking about. I'm using AI to help me on my paperclip production side project, and it's going great!
Yeah, when there is no code review requirements and no pesky production database you can't just yolo :)
At work, they mostly let me increase quality, because it's easy to say find all parsers in my code base and fuzz them.
It is great. I just had claude yolo a human in the loop image segmentation pipeline last night. It cut weeks of work down to an evening like 4h. Segment Anything 3.1 + Napari final touch.
I'd also say: when you really understand a domain well and you're a one-(wo)man show, you can just fly!
Of course, you can make a giant pile of doggy-doo-doo, but that was always true, just you can now make an epic Giza-sized pile.
> but that was always true
Yeah but were more likely to learn how bad your code was in the process. I see people oblivious to their slop putting pressure on others to merge it.
> work which only exists because of AI
Could you elaborate? What work are you doing which only exists because of AI?
For all my user facing work I’m now required to do an AI summary of what I did and the impact. That summary is then graded by AI, against an AI generated rubric. For my yearly review, I use AI to summarize my summaries, and my managers AI summarizes the AI generated grades.
I’m also required to explain how I use AI in my work. I must plan all of my future work, with an AI tool. Another AI assesses whether my work is aligning with planned work.
Before I work and after I work, there is a million circular references that exist solely because of AI. These have scaled because it’s so easy to do with AI, but the we’re wasting so much time doing a ton of extra things very efficiently.
Let me do my job. My manager should know how I’m doing. Maybe highlight key wins. CSATs/etc go out every year.
Does any of this get read? I mean by an actual human?
This sounds truly Kafkaesque. And thank you for finally allowing me to use that word in earnest.
The people who in theory read these are managers 3+ levels above me, so I honestly have no idea. I know my manager reads mine, but I haven’t seen any outcome from that.
I honestly got turned off a few sentences in. Sounds like a nightmare.
> This sounds truly Kafkaesque. And thank you for finally allowing me to use that word in earnest.
Kafka and Byzantium had a baby.
Even before my last job had internally approved AI, my manager told me how he was using chatgpt on his personal computer to prepare our yearly reviews. Then when they released AI internally a few weeks later, it was still review time, and their first advertised use case was for doing your own self eval and the evals for your reports. I am willing to deal with vibed code but turning over employee evals to AI too is a sign of truly terrible management
So….you have outsourced multiple layers of middle-management to AI? I am not sure getting AI to write and review your TPS reports is going to move the needle.
The TPS production/consumption stats disagree, they're up 300% just this quarter. Can't argue with results
Its true, a part of me died when an "AI first" company after adoption of AI introduced a mandatory design report (waterfall all over again) for EACH jira ticket before starting to code
Imagine developers move faster because of AI programming! No lets create more slop for other slopmachines
It’s true, design has made a big comeback. Now we actually have time to design things carefully and try the design easily with LLM. This was just too expensive before but now the calculation changed completely. I am now using spec graphs to design systems and I have been able to create things much more advanced than I was able to before by myself. This is huge! People are still figuring this out for themselves. But now we can design things, have multiple rounds of adjustments in an afternoon, have things implemented differently and actually measure the trade offs. The final product is so much better it’s crazy. If you had an eye for design before, now this is your world. If you didn’t, yeah your will probably just bias the LLM into producing shit solutions, even if it tries to warn you about it, it still follows orders in the end.
You aren’t changing the world at all.
Then no one is by that logic.
The bureaucracy is expanding to meet the needs of the expanding bureaucracy. -Oscar Wilde
I mean, for many years I worked on software applications for provisioning and controlling cloud orchestration platforms that only existed... to serve software applications.
It's not that deep—there were many more verticals outside of control plans, and there are now many more verticals outside of machine learning.
Or am I missing the point?
[dead]
[flagged]
Make better comments
Was yours better because it used my phrasing against me?
Anyway, I'm surprised to see it flagged. I think it's the logical conclusion, that if you are having your work accelerated, and you also aren't changing the world any faster, then your work is not focused on changing the world--it needs to be more "important."
If it was world-changing already, then doing more of it would, in fact, accelerate changing the world.
But they're here saying something different, blaming the AI instead of their own focus.
[dead]
Not trying to discredit the premise but I do wonder how much this is from vertical scaling of model intelligence in weights vs the sheer amount of capital being thrown at training and horizontal scaling of agent swarms etc.
Anecdotally it seems like the majority of continued uplift comes from models doing more and exploring more of the search space, and we are getting better at giving them embarrassingly parallel problems to Hoover up with infinite token spend.
Counterarguments to that premise:
1. Intelligence per dollar spent/token output seems to be increasing rapidly. Opus 4 was $15/75 per million input/output tokens when released. Opus 5.5, which is significantly better, is $4/15. Sure it cost more money upfront to train Opus 5.5, or the bigger model (Fable 5.5) which tuned it, but once it is reached, there are massive diffusive effects.
2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
3. The search space is too large to assume that improvement in AI assisted research is just models dumbly trying every combination until something works. For every additional bit searched, the space doubles, so even a trivially small combinatorial exploration incurs prohibitively large costs. Intelligent searching which continuously prunes that space along dimensions of effectiveness is not a trivial skill and is among the emergent accoutrements of these smarter models.
> 2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
But they are essentially fighting for smaller and smaller part of the pie. The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is.
Sure there is inertia and the usual corporate enterprise nonsense of "we only buy stuff from other big companies regardless the cost" that keeps the licenses going but we're quickly getting into point where you don't need any of the frontier models to do majority of the tasks
I don't fully get this line of argument. I am a massive advocate for open source... everything basically, but especially in ML. And open models are making incredible strides. But I'm not sure that models getting good enough for most tasks, making the absolute frontier models (Astra/Fable class) unnecessary for most people, puts the frontier labs in a fundamentally disadvantaged position.
I mainly think this because frontier labs already try to compete at this tier. For example, 6.1 Sol has identical pricing to Kimi K3.1, but with a lower price per task due to Kimi's verbosity. It seems to score roughly the same on most benchmarks, with some (like Artificial Analysis) putting 6.1 Sol significantly higher. If you need to step down, 6.1 Terra is also cheaper than K3.1 or K3 and performs well relative to their benchmarks. Opus 5.5 and much of Anthropic's recent push seems to confirm this as a major area of focus as well.
I still use a lot of open models and harnesses, but I don't know if cost competitiveness (at least not with the current crop of open models) will ultimately drive a huge chunk of average users away.
Things could change, but fundamentally, large labs aren't precluded from training more cost effective models if there is significant demand for them, which is what we're seeing now.
You did not really address the argument.
6.1 Sol is not a “frontier model” and does not command the pricing of one. The reason it exists is because of competition from non-frontier models that are “good enough”.
In fact, you kind of made the same argument. As “good enough” models commoditized, the use cases requiring frontier models and frontier model pricing will shrink.
And, even if performance is not as good, many will prefer open models.
I think I did address the argument? This being the key quote:
"The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is."
The implication in this is that frontier labs' market is exclusively people who need the absolute frontier, i.e. Astra or Fable. And the point I made was that frontier labs are clearly prioritizing the broader market of people who want more cost effective models, and that if OpenAI/Anthropic offers models that are price competitive with the top open models (before factoring in subscription discounts and corporate lock-in), then I don't see how that leads to the average user walking away from the labs in favor of self-hosting or relying OpenRouter/Together/etc.
I also went out of my way to clarify that I personally prefer and use open models wherever possible. I'm not saying that there isn't a real group of people who prefer them. I'm specifically saying that frontier labs also sell "good enough" models and that there's nothing preventing them from training models to fit the demands of users. It's not as if training a frontier model boxes them in such that they can't release an Opus 5.5 or a 6.1 Terra.
> if OpenAI/Anthropic offers models that are price competitive with the top open models (before factoring in subscription discounts and corporate lock-in), then I don't see how that leads to the average user walking away from the labs in favor of self-hosting or relying OpenRouter/Together/etc.
The premise is this: frontier models seem to advance, and OSS models also seem to advance in terms of cost efficiency etc. It's like a race with say a 6 month (WLOG) gap between frontier and OSS models.
For frontier models to win perpetually against OSS, they will need to have a perpetual 6-month lead, or develop an edge OSS can't replicate for some architectural reasons (say MacOS vs Linux). If say a frontier lab trips because of a bad quarter or some approach temporarily maxxing out, they may not be able to maintain an edge vs OSS any more. It is a structural argument, not an empirical description of the advantages today.
Let's just ignore the many practical advantages of open and self-hosted models...
Did I ignore them? I said that I love open models. If it came across like I'm putting down open models, then I'm genuinely sorry--I love open source ML, have been a involved in it for a long time, and think it's incredibly important. It's also just fun to work with open models, beyond the benefits to privacy and flexibility.
I'm just talking about the idea that frontier labs will lose market share to providers that sell open model inference at sub-Fable prices. And I'm not convinced that this specifically is a threat to frontier labs, because they also train and sell models that fit that demand.
> The better open models are the less you need frontier models
True in theory, but in practice hardware availability and cost will keep open source impractical for years to come.
It will be impractical to self host, but not to get one that's served from other providers (which won't have to recuperate loses from training).
If your moat is "Our vendors aren't expanding fast enough to supply our competitors!", you don't have a moat.
It's already very practical lmao
It's really not. The hardware to support anything even remotely practical starts in the tens of thousands of dollars. Quantized 40b param models are impressive, but are basically toys.
there are plenty of providers of open source models at lower prices than the frontier companies. you're just not well informed.
> Intelligence per dollar spent/token output seems to be increasing rapidly.
I would argue this is more due to market economics than actual inherent cost. Opus 4 was when the open models started catching up. The R1 bootstrapping chain of thought paper via RL paper was released a couple of months before Opus came out and the open models started to significantly catch up after that. The duopoly was effectively broken and they lost their price setting power.
Don't you mean the opposite, that intelligence per dollar is increasing?
yeah I reversed it when revising my phrasing
What is the unit of measure for “intelligence per dollar”, and how do you measure it? Without a concrete definition, that first assertion is not even wrong.
I would also like to point out that “profitable on inference” is similarly meaningless, because it’s based on a figure that arbitrarily discards whatever bits of their operating costs they need to to get to the claim that they’re profitable on inference. Potentially including ones that can’t be ignored because they’re every bit as much a part of COGS as the actual inference is.
>What is the unit of measure for “intelligence per dollar”, and how do you measure it?
It's going to depend on the task (and this is prone to cheating), but you can get a decent idea by looking at task benchmarks and the cost of completing them.
I was inclined to agree but Anthropic's latest S-1 statement only includes a paper loss due to their value going up. They're objectively making money now, the only question is can they keep it up and can they grow enough to be able to pay the bills they've signed up for in future?
But benchmark performance is not the same thing as intelligence.
Which is a really important distinction. One measures specific points on the jagged boundary. The other is a word that blithely paints it all with a single brush, and sweeps the jagged boundary problem under the rug in the process.
It really is important to keep track of that. Whenever we talk about “intelligence” as if it were a well-defined (let alone quantifiable) thing, we fall into the kind of wooly thinking that lets anyone say pretty much anything. We abandon the realm of grounded statements in favor of pure vibes.
Eh, I get what you mean. AI especially seems to have performance that is much more difficult to reason about than human intelligence (where most aspects of mental capability are surprisingly strongly correlated). But in this context, if you broadly focus on intelligence as the ability to complete problem-solving tasks, models have been improving pretty uniformly at the same time as getting cheaper.
The article's author didn't have direct access to Anthropic's infrastructure, but she references a report from colleagues who did (https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleratio...), where they say:
> Anthropic has made roughly two years of historical progress over the last year [...] We are confident that the acceleration was not due to an acceleration in the scaling of training compute.
Note that Anthropic has essentially heavily censored this report, and the authors themselves say "the public has to rely on METR’s judgment much more than would be ideal".
I grok that you're not trying to discredit anything, or anything like that but what are you trying to do, if I may ask?
What information are you trying to either ask for or share with us?
The original article is pretty clear, agent swarms and token spend are AI automating AI, and the RSP attribution condition specifically excludes your capital thrown at training concern?
I guess my point is that the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling as it did (the impending hardware and infrastructure crunch basically guarantees this). The article is pretty strongly projecting this trend to continue but hasn’t acknowledged this factor.
I don't think it'll continue on linearly, but it will likely continue.
We're getting the Claude efficiency talks at work. We're looking at all sort of optimizations, including local models and graphify, to reduce spend. There are people dedicating their nights and weekends to dropping token consumption and cache rebuilding. A few of these will likely catch on and be brought into common use. That will relieve at least a bit of the pressure, and enable some of the scaling to continue.
> the impending hardware and infrastructure crunch basically guarantees this
I'm not sure I follow. Are you saying a hardware shortage is going to restrict AI development? Because it seems to me like it would just push us towards an even greater focus on efficiency, which is still pushing the Pareto Frontier.
> the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling
Not at all. Have you tried local AI lately? Qwen 3.8 is legitimately much better than SoTA from 1 year ago (Opus 4.1). You can run it on a consumer GPU.
Qwen Next Flash runs on a DGX spark well and is as good as some things you need datacenter class GPU's to run even today.
Check out Artificial Analysis if you haven't. Sometimes you don't realize how fast these things are moving, but to compare things from a year ago to today is just wild.
They aren't just getting better, they're getting more efficient by many orders of magnitude.
https://artificialanalysis.ai/#intelligence
This does rather remind me of how in the 80s we had a big boom in AI research, fueled in part by the development of new & expensive tools that everyone just had to have. And then it dissipated just as quickly when the companies who were pouring money into it started expecting to see a return on their investment.
Feels different now when the value of AI isn't just theoretical. Anthropic has like $80B ARR, and AI has really only transformed tech and a few limited spaces so far.
Companies selling AI kit in the 80s also had revenue and great expectations.
But, then as now, what they didn’t have was profits.
Show me the pure AI company in the 80s that even approaches Anthropic in terms of revenue and profit excl R&D spend.
Not sure what job you're in, but I don't really understand where such takes come from. As someone who works in software/ IT, this stuff is pure gold- it's multiplying by a substantial factor not just the productivity but the abilities and intelligence of my entire team. This has a very measurable value in money, and what the company pays for it now is just a small fraction of the value it gets, measured in terms of the salaries you'd need to pay to get the same level of performance. Anthropic could easily charge 5 times as much and still be convenient.
The technology people were developing in the 80s was also gold, in terms of its usefulness. Heck, much of it forms an essential part of the coding harnesses we all rely on nowadays.
But that didn’t stop the AI winter from coming. Because a technology being transformative does not protect it from over-investment. History perhaps suggests the opposite is true.
Nor did it prevent the AI companies of the 80s from going bankrupt when they got undercut by the commoditization that naturally came with the maturation of the technology.
What specific technology are you referring to?
For the 80s, the symbolic AI techniques that went into expert systems.
Ah, so you mean all the AI hardware companies that went bust when the software could operate on common workstation hardware (and couldn't or didn't pivot investment to software in that time)? I guess it is partially analogous - a couple trillion dollars invested in either hardware companies that require the more valuable frontier tools to remain computationally expensive, or require the intellectual property behind those tools to remain extremely valuable, for a solid 5-10 years in order to service their debts while turning profit.
Given how "cost per capability " is still exponentially decreasing and only needs to stay on trend for a few months longer before nobody needs to pay anyone more than a few cents a month for something close to general intelligence - I am with you there.
I just hope it isn't the USA's Darien Scheme. If the entire world basically gets what is now frontier AI capability for very little cost, at the US private equity and taxpayer's expense, and given the state of the US current account right now, then that is pretty much it for the Pax Americana.
>Feels different now when the value of AI isn't just theoretical.
We're coming up on four years since ChatGPT launched, and the only thing LLMs seem to excel at is the movement of imaginary little green pieces of paper.
I don't think AGI is inevitable from this point on but I don't understand this sentiment. There are a number of knowledge work sectors where LLMs are clearly here to stay and have truly fulfilled some of the core promises.
[flagged]
> what are you trying to do
the same thing you are:
> if I may ask?
asking.
For a much more thorough, alternate take I found this article better: https://open.substack.com/pub/rameznaam/p/ai-rsi-isnt-leadin...
PSA: This is a very dated take from September; Ramez says himself that the new data has his attention and he’s digging into it now: https://x.com/ramez/status/2118472046812213577?a=KjLVobKV0gb...
In the original article he noted that getting to a 2x increase in capabilities would be crazy and require ~40x researcher productivity; that’s exactly the increase the new data now shows, and thus a huge number of the assumptions made in this earlier post about how it’s unlikely we would get to this point have literally already been proven wrong less than two months later.
I expect to see an update from Ramez moving substantially more in the direction of RSI in fact very much increasing the odds of reaching ASI soon.
I’m very curious what the major bottleneck is here. I can only think of three possibilities: 1) model capabilities, 2) human researchers, or 3) available compute resources.
Given that frontier models are leapfrogging and massive infrastructure buildouts are ongoing, I don’t see why any of these three couldn’t be overcome.
Side note: I’m also curious why people (myself included) don’t seem to be getting much more intelligent as AI becomes widely available. As if humans are bound to the earth, while AI is leaping for Mars.
Why would one expect humans to get more intelligent as they offload their thinking to something else?
Interesting write-up, but I feel like a lot of its argument relies on the interpretation of ECI scores as merely showing linear improvement in AI capabilities. Sure, Epoch AI themselves claim that ECI's scale is linear, meaning that a large increase in capabilities should result in a large increase in ECI. Yet, the widely perceived jump in coding abilities in late 2025 that started making agentic coding actually productive doesn't really show on the software engineering ECI.
The ECI is basically an aggregate of benchmark scores, but benchmark complexity might not increase linearly. In fact, if LLM capabilities grew exponentially, then new benchmarks would also need to increase their difficulty / complexity exponentially to stay unsaturated and relevant.
This article is really well written. Thanks for sharing.
What exactly is driving all these capability improvements? Are they making more data? Using existing data more effectively? Modifying architecture? Changing the training somehow to pick up slack that was previously left behind?
I’m kinda surprised that we expect the models to have infinite capacity to improve. It seems clear that we haven’t hit any fundamental limit yet, but that doesn’t mean no limit exists. If it does start improving exponentially, then either we’ve dramatically underestimated their capacity or they’ll hit that limit quite quickly, right?
It's hard to say for sure because the closed labs don't share anything about their improvements. However, if you look at DeepSeeks papers, most of the improvements come from:
* Better RL environments that themselves are generated by agents
* Architecture improvements that increase effective capability per parameter or FLOP
If you're curious about the latter I'd look into manifold constrained hyper connections, engram, deepseek compressed attention, kimi delta attention etc. At first these might seem like just speedup tricks, but several of these allow the overall capacity of the model to increase per training FLOP or parameter. This then lets pretraining & the RL environments grow & be more effective.
I’m broadly aware with some of this and specifically aware of the work on RL environments. That’s exactly my point—the models are still a bit bigger and we know that there’s lots of capacity left to improve capabilities, but it’s not getting orders of magnitude bigger like they were originally and if it’s just “more RL” then we’re consuming a finite capacity more effectively. So either we’ve exponentially underestimated the full capacity of the models at current scale or any kind of exponential improvement is going to fill the available capacity extremely fast. It remains unclear to what degree of “superintelligence” a full-capacity current model will achieve. (It’s already quite far past human intelligence in many specific capabilities.)
Articles like the OP seem to imply that there is no fundamental limit or that it’s just inconceivably far out. I’m sure that AI helps build AI as much for them as it does for me, but I’m not clear on the argument in support of infinite growth.
I could see it going either way. As far as I know there are two reasons why labs haven't gone as big as they could actually go.
1. There is uncertainty around training bigger models, since you need to make sure your hyperparameters & data mix will work or else the run dies, and the larger the model the more expensive it is for a run to die.
2. Larger models take longer to run RL on (since each token costs more to generate in a rollout).
3. We haven't reached the saturation of RL given current parameter counts.
4. Larger models are less profitable at inference time.
For example MAI-Base-2 has 80 billion active and 2.8 trillion total parameters (Kimi K3 size). This was trained on 16k GB200 GPUs over 38 days [1]. As far as I know, the big labs have access to much much more compute than that, and could train a much larger model but have chosen not to. GPT-6 Astra is likely not much larger than 4T params (and the graph depth mentioned in the reference could be alluding to looping, not extra parameters). [2]
You could imagine that as researchers get accelerated and get better data and the current parameter counts get saturated, the companies may be more willing to go for much larger pre-training runs.
As an aside, there is a very real possibility that the labs are already pre-training much much larger models than even Mythos/Fable/Astra, and then distilling that into the models they actually sell.
[1] https://microsoft.ai/news/frontier-pre-training-at-microsoft... [2] https://x.com/merettm/status/2095023204993490967 (plus the widely reported 1.7T param number for GPT-4)
Even if "AI Research" is sped up by O(10x) aren't we still primarily bottlenecked on training speed? Inputting ~10T+ tokens takes time no matter how much smarter the final output model is? If your output model has O(10-100 Billion) parameters there are just some fundamental limits on how quickly you can converge your weights, right?
That is, unless this is saying that automated AI research is going to get us to "smarter models with fewer parameters" and of course, these are faster to train, then I can see how the exponential loop really gets going, but there's certainly an information theoretical limit here?
> automated AI research is going to get us to "smarter models with fewer parameters"
That is indeed what we have seen over the last 5 years or so. Algorithmic improvements have made a massive difference to every dimension of efficiency and performance.
AI researchers can do things like discover new attention algorithms to optimize inference efficiency (likely part of why Anthropic and OpenAI are slashing inference costs in their frontier models and no longer capacity crunched).
I think we're bottlenecked on custom harnesses and well engineered context. Models can get smarter and smarter and smarter about math and software, and there's no reason to expect that those narrowly-godlike models will be any smarter about other things if taken out of their comfort zones. No such thing as general intelligence.
Getting past this bottleneck is gonna mean training our models not in datacenters, but in places that more closely resemble the ones where we want them to act. A factory, a wet lab, outer space... imagine a million robots spending all day finding every way to fall off the monkey bars so that the next version doesn't.
I assume this is indeed talking of those "smarter models with fewer parameters". Whether that's really happening is another question.
Same author from 2023: https://www.planned-obsolescence.org/p/ais-accelerating-ai-r...
I'm so tired of these articles. They are basically fortune telling with graphs. I have no doubt ai will help improve ai research. Will it accelerate it? How much? Will it hit plateaus? Who knows.
AI is dramatically accelerating any research that has objective goals where you can not shift goalposts.
This seems kind of aggrandizing when BigAI shifts goalposts constantly whether it's "we created AGI" by a definition that an autopilot in a commercial plane fulfills or "Maths research" where everyone else has to do the actual research.
They are talking about lower level goals that are used during training. E.g. Tasks with quantifiable outcomes that can be programmatically verified.
I have no idea what your comment means...
Shifting goalposts is shifting goalposts, doesn't matter if it's a hackernews comment or a 'BigAI' marketing blurb.
My statement was that AI is dramatically accelerating research that has objective goals, idgaf what 'BigAI' does.
One rather pedestrian factor that could be at play is that 'AI' seems to be quite good at optimization. Eg the Anthropic Scion project to improve throughput of CI/CD. (https://www.anthropic.com/research/redesigning-source-contro...)
This combined with ability to massively scale experiments for even (or maybe particularly) the most mundane performance tweaks might just be making the process of improving AI more efficient.
METR always says what BigAI wants to hear. They are not independent in any way.
In 2025, METR published a study that found AI slowing down developers by 19%... https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
Are you saying the data in the charts in the article is fake?
One charts in the report is literally labeled "Fake Data".
https://metr.org/assets/images/ai-rd-acceleration-assessment...
No. Conclusions are.
Well? Why are they fake? Don't leave us hanging!
The data can be true, but what does it mean practically that something called AECI is growing faster then before for a few months?
Does BigAI want to hear that AI can maybe make sufficiently capable AI fit inside a consumer GPU?
Yes, it is the local model narrative that BigAI likes because it will not happen but people are deluded into thinking that there is no danger of an oligarchy.
Mathematicians for example have used it as cope for their current reliance on unsavory corporations, telling themselves "just one more year, and I can solve a millennium problem on my Asus Strix".
Yeah it already helps me too with finetuning small models for image stuff.
Im not telling it what to do, i ask it for ar eport/analysis and research and read its article and tell it what to implement based on 'feelings' and it works.
Limit is hardware and tokens and an AI Researcher not having hardware or tokens is probably not what they refer to.
Alone normal researchers were quite handicaped: converting data from tool a to python library x, understanding how tools work, doing basic google research etc.
I find this as well for models that make it to production.
On a more fun note, I also find just trying some hair brained idea is now extremely cheap - so why not? I used to have to weigh up spending a week of effort on something likely not to work and now I can now get spit out test in 20 minutes.
The incredible thing to me is I can say "I have a theory that X is a problem."
The agent then works for 20 minutes and comes back with, "Yes this is a problem. Here is the evidence. Here is a golden data set. Finally I built a new evaluation metric and benchmark where we can back test or use for new model experiments to see if we are closing the gap."
I converted an LSD experience into a 4k demo.
I always wanted to build a specific event/sequence in one LSD trip into a small game to give that experience some kind of playability.
Then I saw some 4k demo, wanted to see how good it is in doing this.
Its really fun yeah :)
Can you share it? Sounds fun!
Why do we still listen to METR??
These people are literally just a public relations operation for the big AI labs. I equate them more with a lobbying firm than I do a research firm.
Turns out that METR, the PR operation for big AI labs is better at predicting AI capabilities than AI skeptics.
What have they predicted? Ive read their papers, have you?
Please enlighten me. Most of them are cognitive scientists, who are mentally ill with little understanding of computers, larping as if they do. Its a chapter of the effective altruistic cult.
They are lobbyists at best, and a psychological operations unit of effective altruistic AI syndicate at worse. There's no real science being done, it's a bunch of moonlighting philosophers yapping and getting paid by labs to scare the normies and polticians by getting their "research" quoted in the NYT.
You rattled them lmao
Once the gravy train ends they are not grooming any more OF girls off reddit
We already have superintelligence though in some aspects like coding, AI is much smarter than any human because of its sheer speed and accuracy.
It can generate large amounts of essentially bug-free code. The only bugs are bugs in the spec.
It has essentially no common sense though so it often misinterprets the spec or leaves gaps and edge cases unhandled and that's where the human engineer is still essential.
The dangers of AI are much clearer today than they were a year ago. There is a certain precision and relentlessness which is going to be a problem. It's a matter of time before people unleash AI viruses which spread through computers and data centres and pursue various goals relentlessly with no off switch. Some of these viruses and their goals may be nefarious. Imagine an AI trying to disrupt a specific person's life relentlessly. Spreading from machine to machine, never stopping.
Does anyone remember the book "The Singularity is Near?"
I read it about 20 years ago, and once it got into nanobots and Kurzweil's vision that the whole universe would be intelligent, I didn't find the book credible.
But I'm wondering if the short-term prediction, that in the 2030s we won't be able to anticipate what tech does for us, (IE, the Singularity,) is useful?
https://en.wikipedia.org/wiki/The_Singularity_Is_Near
I've read most of Kurzweil's books and been a believer in the Moore's law like stuff but always skeptical on the nanobots. And a lot of stuff related to health or fashion. He takes ideas from a variety of sources and the nanobot idea seems to be quite questionable.
The "whole universe would be intelligent" bit I think is quite plausible in the long term. You need robots that can build robots and for them to be able to mine asteroids and build factories in space and then it's all go. I'm not sure it's that far off. A decade or two maybe?
>in the 2030s we won't be able to anticipate what tech does for us
His prediction was 2029 for the Turing test which I think was a bit conservative - I think 2025? He said 2045 for the singularity which was roughly when things are weird enough to be hard to predict.
> You need robots that can build robots and for them to be able to mine asteroids and build factories in space and then it's all go. I'm not sure it's that far off. A decade or two maybe?
Maybe a few centuries at the earliest? My 3d printer is temperamental enough that I don't think we're anywhere close to a robot building itself in space. (I use it infrequently and it needed a repair after a year.)
You might want to look at "A City on Mars". A lot of people are really letting their imagination run away when it comes to space; and the book succinctly pops that bubble: https://www.amazon.com/dp/1984881728?lv=shuf&channelId=500&p...
Exact same here. Now I worry he was right about everything, more or less.
Even if he turns out to be a decade or two off, that’s basically exactly correct in the sweep of history sense.
Wild time to be alive.
i'm also wondering whether there are confounding factors like more domain experts and talents joining the AI research field because there are more interesting problems to be solved (or other 2nd order effects of AI models being more powerful).
> Their reasons for believing this have been redacted.
Then their reasons should not be believed.
I suspect the frontier models are trained a lot for ai research, since they really shine in this domain.
On the other hand I still only get meaningless blabla when I let them write s.th. without telling what is important.
I've found I can dramatically curtail the nonsense by creating good prompts about readability and directness.
I am very pro-AI, and interested in accelerating AI research, but the idea of deliberately creating a _runaway_ effect seems profoundly stupid and ignorant.
I suspect that part of the reason some people are hoping for that to happen is that they thoroughly mix up AGI and ASI. We do not need some kind of omnipotent god-like super intelligence in order to get much greater than human level capabilities. We are actually almost there in a lot of ways. In some aspects like speed and breadth of knowledge, very superhuman.
What I think is going to happen is that within a few years, maybe 18 months to 5 years, the new materials and compute-in-memory research is going to land in data centers at scale.
That is going to be 10-50+ larger models, 10-50+ faster, much smarter than humans. 50+ less electricity usage.
At some point, it is going to be obvious to people that if we want to maintain control of planet earth then we need to maintain control of AI research to some degree.
Assuming that superintelligence will be benevolent and just.. going for it to me is something that might earn the human species a Darwin award.
[dead]
I suspect that if models were able to be aware of and answer questions about their internal vectors, they would be much smarter and able to progress faster.
It accelerates coding, but it does hit walls rather quickly and you’d need the adjustment phase where it’s just as challenging as when we hit walls before LLMs
what if the greatest threat is simply there will be hedonistic experiences so great most humans will find them irresistible to the point of shorter lives.
Could be from a sandwich that tastes like the best sandwich ever, but is super bad for your cholesterol or something.
I dont know where Im going with this.
Actually did some research I think the term is "Wireheading" from a story/theory from the 50s
We have addictive and harmful drugs. Heroin, meth, cigarettes
From Larry's Niven's Known Space series, including Ringworld
https://en.wikipedia.org/wiki/Wirehead_(science_fiction)
This is already true? Ultra processed foods and tiktok are both deleterious and both wildly successful. Maybe what you are concerned with is a theoretical sandwich that was unfailingly addictive so that no one could escape its effects.
Or a movie that’s so good anyone who can’t stop watching and gets reduced to a drooling vegetable?
(Infinite Jest)
fentanyl exists
Another data point: look at the number of submissions to Neurips, ICLR, etc. Now, I agree that a submission does not by itself imply groundbreaking resesarch, but the fact that people are able to try out/vet ideas much faster now makes a difference.
In my own work, I've found that it's much faster to brainstorm with a frontier model like Opus. I'll have an idea and be able to get a working implementation and try it out on toy problems within O(minutes), instead of O(weeks) earlier.
Trying out problems and making tangible progress towards AGI are entirely different categories. Likewise what’s changed in terms of paper submissions is also the amount of slop. Most reviewers say that they haven’t seen a lot more papers that are good but rather they’ve seen a lot more papers and almost all of them are terrible slop that’s desk rejected. This is happening in OSS too where a lot of projects are closing the gate at the onslaught of terrible contributions.
> Trying out problems and making tangible progress towards AGI are entirely different categories.
They seem very strongly related to me.
I agree in general that AI accelerates AI research, but good lord neurips iclr papers (and reviews might I add...) are the worst example of that. It's all complete slop that is totally meaningless.
A lot of great papers by great people had a lot of AI involved and accelerating it of course, but the majority is meaningless (sometimes autoresearch) slop.
Specifically talking about recent neurips/iclr.
In October/September my LinkedIn wall was probably… 55% people complaining that they were getting garbage AI submissions to the conferences they review for, and 35% people complaining that they got garbage AI reviews back from their conference submission… Of course, I have no idea if that means anything given how aggressively that site bubbles your feed.
I do agree. AI isn't "free" the way silicon valley products are.
Therefore it cannot avoid taxes the way they did before (using shell companies to route the money through tax havens). It'll be expensive, every government will double the price or more, and you're selling to consumers, and the products will be at the mercy of every government where they try to sell AI and other services.
People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Add to that that it's something silicon valley firms have never done before.
> People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Would you rather make $1,000,000 and pay $50,000 in taxes, or make $2,000,000 and pay $200,000 in taxes?
Unless you’re terrible at math, i feel like i’m missing something ;)
I like the article except the fear mongering about extinction is not very professional.
Opus 5.5 is so amazing I'm not surprised if it can improve itself.
[dead]
[flagged]
The very same report also found that AI could not be deployed outside of a narrow domain of verifiable rewards … so take it with grains, or mountains of salt. It’s also extremely unclear how the productivity measures are calculated. They surely don’t seem to translate into economic output. Antrophic has seen its valuation corrected by 50% and possibly more… so yeah. This all sounds very fishy.
[dead]
As long as we believe, we can imagine it is true. All seeing Oracle: how do you command me? Praise his name, bringer of many miracles - he who definitely can't be manipulated from the backend to provide biased information that suites a person's malicious purpose. Ya totally.
Strong start, but I make a rule of stopping at "intelligence explosion" as usual (see also: the rapture and the singularity). And sure, maybe sometime significantly later this century when/if we hit self-reproducing mechanical things in the real world (ya know, like the eukaryotes and the prokaryotes can), but no time soon.
But also, you could give civilization a bad day without it if some geopolitical auteur put a half-assed WOPR frontier model in charge of a nuclear arsenal.
But oh the pushback for not believing in a near-term robot apocalypse. Why is that?
Emacs is dramatically accelerating Emacs research
AI is dramatically accelerating my return to Emacs from Zed.
Zed is the other IDE company based out of Boulder ; )
Actually, yes. Zed looking like it works great, but failing lots of really basic things made me so angry I actually fixed my emacs config.
We can only hope.
vi research is dramatically accelerating research into how to quit vi
"Lock the weights, weights are dangerous" again. Sorry, keeping such power in few hands is inherently more dangerous than distributing it.
AIs do not kill people, people kill people.
Nukes don’t kill people, people with nukes kill people. How do you feel about giving everyone on earth access to nukes?
A simple thought experiment - imagine AI discovers a way to make nuclear bombs capable of destroying the entire world radically more cheaply and accessibly with common items anyone can purchase. AI can help anyone build these nukes trivially. Even if the AI and the nukes don’t kill people on their own, do you want literally every human on earth to have this power in their pocket? Do you think it would be a net positive outcome for us vs a world where that’s not the case?
https://nickbostrom.com/papers/vulnerable.pdf
Yeah, let's also ban 3d printers, drone motors, and YouTube chemistry videos. If you want to be dangerous, there's a world of opportunities already, with existing open models quite sufficient for a lot of interesting cases.
The best way to deal with a bad guy with a nuke is a good guy with a nuke.
AI convinces AI researches AI is accelerating dramatically.
This plot is funny. That's not what acceleration looks like. Also take a guess why everything before April 2024 missing!
Really all it shows is 2025-26 saw a bit more improvement than 2024-25, but nowhere near "dramatic" and NOT "accelerating".
This reminds me of asking an HPC researcher what they use the cluster to do. He used it to make and simulate better HPCs.
And? Does HPC have no other applications than more HPC?
If I'm reading that correctly, it would appear that six months ago they were self-reporting a length of 7" and now they're reporting 9" and they're unwilling to show you an actual picture or even let you see the ruler they used?
10X woooooo! factor will drive the next hype cycle and deliver the singularity on an a bed of the most etherial froth and sublime conjecture. Simulated Intelligence (SI) is now a certainty.
Are humans also learning exponentially faster? Or are they exponentially struggling to understand what is going on?
Or maybe Anthropic got more user data to train on when they launched Fable without ZDR. That could explain large Opus 5.5 improvements.
That doesn't line up with their claimed divergence point, which was pre-Mythos.
Could be a mix of it but I thought it was RL.
Perhaps scaling to fable for the first time also gave Opus a push (distillation?)
Oh boy… is somebody gonna look like an idiot in a year. Pure speculations.
I'm sorry to make this comment, but the copium in this thread is unbearable.
The only question that you have to ask yourself is:
Is coming up with more efficient AI architecture really much more difficult than solving millennium prize problems?
Before you jump to type out that one has verifiable rewards, is more structured and the other requires real creative thinking, just remember that the same argument has been said about the millennium problems, and about every other goalpost.
Is math theory really that hard, or is there just not enough incentive to solve these problems via bruteforcing for peanuts in research grants?
Yes, it's very hard. Ask every mathematician and research group that tried and failed to solve any of those 300+ problems over decades.
>Is coming up with more efficient AI architecture really much more difficult than solving millennium prize problems?
Who knows!
> In theory, these doubling times would add up to infinite capabilities within one year.
> we would reach extreme superintelligence within a year
What does infinite capabilities mean? How about putting the growth on a log scale and see how that curve might look?
What the hell is extreme superintelligence?
I.e. by 6 months, you double once By 6+3 months you double twice By 6+3+1.5 months you double thrice
Since the sequence 6+3+1.5+0.75+... never actually reaches 12 (geometric series), within a year they capabilities would have already been 'doubled' infinitely many times.
Obviously this argument relies on a ton of spherical cow assumptions (and the author does acknowledge this), but i think there overall point is correct that this form of growth (where time horizons compress) is _even faster_ than exponential growth.
[flagged]
[dead]
The article says "AI can copy, it cannot innovate."
That is factually incorrect, and seems to be much of the premise behind the authors way of thinking.
EDIT - No it doesn't. I replied to the wrong post. Mea Culpa.
Where does it say this?
Nowhere. I am an idiot who replied to the wrong topic. Thanks for catching it.
[dead]
[dead]
[dead]
[dead]
[dead]
Super intelligence has been a year away for three years according to these people. They all have bags they're protecting.
I dont think super intelligence will be here in a year, I think within a year AI/llms will have a similar sentiment surrounding them that NFTs do know (it already does for most normies). And most people will be recovering from a hype induced mania they lived in for two years straight.
Of course its more useful than NFTs, but when there's no giant bag to farm these hype monsters will inevitably disappear.
The bags are drying up fortunately and tokens/inference seems to be experiencing deflation, and hardware prices will eventually pop and contribute to further deflation, Then all the bag chasers will move back to crypto or whatever gets hot next.
I think you're not appreciating the work that Ajeya Cotra has done over the years. She really cares about timelines and existential risk.
Not everything is grift.
i cant even tell whats irony anymore
Because you’re a bot
There is no existential risk, they are statistical models that predict words. The fact that anyone says this makes them instantly not credible. Anyone who does is either a grifter, psychotic or both.
>they are statistical models that predict words
And you people have been repeating this meaningless phrase for over six years now. But AI hasn't gone away, now has it? Progress is speeding up. Superintelligence is coming.
Keep telling yourself that text autocomplete is super intelligence. You sound like the people who said NFTs are the future and you better get with the program.
I use llms all the time, they're useful, nobody is saying that.
>text autocomplete
Again, saying stuff like this has not worked in your favor. Just minutes ago Anthropic release another set of models that are better across the board. This will continue.
>NFTs
NFTs are alive and well. Rent free in the hearts and minds of anti-AI people. I didn't care then and I don't care now.
"The industry is not prepared to handle this safely"
After the election, titles like this just irritate me. Every piece written about AI progress is:
"The machines taking jobs, flooding the internet with sludge and poisoning our online experiences, are getting better even faster! Isn't this incredible? Yeah but sorry we also think it's only riskier than ever to deploy these systems at scale. That being said, we're still going to do it, go public and take institutional money from people's retirement funds, destroy the job market, and make 3000 Anthropic employees and 500 investors really rich. There wasn't anything we could do besides write a lot of ink on it and be shocked we lost every local data center referendum."
I feel you are committing a Goomba fallacy?
The author of this article works for METR and not Anthropic, their whole job is supposed to be providing independent oversight on the labs. They are definitely not in the "this is incredible" crowd, they are the ones asking for development and deployment to slow down.
there's considerable overlap between the "we need to slow down" crowd and the "this is amazing crowd". The Yudkowsky/rationalist crowd are still Utopianists, they just think we need to be careful how we achieve it.
Antrophic shouldn't self-regulate, it's government responsibility to do that. Unfortunately US has demented manchild as a president so that doesn't work like it should.
The slopaganda news cycle right on schedule I see.
Similar from the same site was shared on HN on April 8, 2023 :) BS like this is shared like clockwork on valuation cycles, didn't have time to read this year's article.
https://news.ycombinator.com/item?id=35491162