I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
Apologies. The title should be amended with (2023). In this case it is an interesting snapshot of the zeitgeist back then and we can see how well it panned out and whether anyone involved has updated on new info.
For some reason the anti-AI camp ping pongs between different and often mutually incompatible arguments at lightning speed. The "AI is fake" argument has been completely forgotten at this point.
Is the anti-AI camp with us in the room right now? Or is it a crude strawman to summarily denigrate any objections to certain features of current AI?
AI isn't some homogeneous mass, it can have good and bad sides, and it's always amendable to improvement. Treating the current state of AI as the only possible hides the very idea of improvement.
> between different and often mutually incompatible arguments at lightning speed.
Of course - there's no homogeneous AI camp either, but there are bot farms, sh^t-posters and sh^t-posting bot farms, different entities with different opinions should not be mistaken for a single stream changing at "lightning speed".
Despite the fact that I'm not culturally or geographically connected to them at all and have some differences in taste and aesthetic, I have to conclude that this crowd, including him and Gwern etc. have been much better at understanding and predicting things than the expert takes you find in the media. That includes AI but also things like covid and others.
> surely you can't blame the OP for writing this after a few months ChatGPT
Bad takes are bad takes. The author was perplexed by the "many people" convinced that models are intelligent and is argued against opinions/arguments he's been exposed to. Called proposed use-cases "borderline fraudulent pseudoscience."
He also published a second edition of "The Intelligence Illusion" in Sep 2025, so seemingly still stands by (some variant of) this belief.
It's an interesting reminder of how much general discourse has shifted since 2023 (I haven't heard of stochastic parrots in months!) but being wrong early doesn't change that.
I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.
Maybe I'm just light on imagination or something, but I honestly wonder what you would qualify as a "well-informed plan" in the current scenario? I've thought a lot about it and the whole "potential for the mass unemployment of knowledge workers" thing makes a lot of planning kind of useless IMO. If you are affected, it's going to be a bad time. If you're unaffected, the people who are will inundate your profession with cheap labor anyway.
My mental model is that humans will continue to be needed. Many jobs will embed AI in them. Using AI well is a skill. I need to understand the technology, where it is strong and weak, and how it develops over time so I can employ it effectively in my work, and advise others on how to do so. Basically, I need to learn how to be skilled with it. This is tough because things are changing rapidly, so I have to invalidate my cache when advancements happen. This means staying curious, not settling into a specific work pattern, but rather experimenting regularly to see how I can leverage AI in my work.
I think you are majorly overstating OP that ai "didn't work".
In fact I agree with the post on almost all aspects and I'd be the last to tell you ai doesn't work. It absolutely does but with a caveat... It's still a tool. And better expertise in the problem domain, along with a better harness used for verifiable outputs will get you better results.
I think the posts mental model of stastically likely prompt completions is spot on.
Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.
There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.
Things can work whether humans understand the details or not. One can build layers of technology on top of each other, with no human understanding required.
My biggest issue isn't being too agreeable (ie the psychic con), it's being confidently wrong, including outright hallucinations.
If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.
Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.
Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.
Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.
I guess my point is that I think LLMs are way more limited than a lot of people think.
The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"
You seem to be implying that achievements of internal models are exaggerated, but that's rather implausible. The public does have access to, for example, Opus and Fable, and so we know what those models are capable of - finding real vulnerabilities in multiple codebases, for example. If you extrapolate from these capabilities one more generation, you'll get pretty much the same feats that the internal models are claimed to be capable of - so why should we doubt those claims? It's not like they're claiming that their internal models developed psychic powers and learned to teleport - the claim is pretty much just "we have models a few months ahead of what we're making available, and in those months they've been improving at the same rate as usual".
You're claiming the frontier labs are lying about the capabilities of their next generation models. A reasonable person will expect that statement to be backed up by some verifiable evidence, given that we are several generations of models into this process and the capabilities are consistently increasing, often much more radically than people expected (remember "stochastic parrots"?)
Your post claims that they're lying and then throws out a bunch of fear, uncertainty, and doubt about what they're doing behind the scenes.
If I were to steelman your argument, you're probably saying that the AI had a support system around it of people and training data and feedback that allowed it to achieve the breakthroughs that the labs are claiming. That actually seems perfectly reasonable, but in my mind it does not invalidate the advances they're announcing.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?
it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
Good news: at this time, they're not intelligent nor conscious as far as we consider humans to be. There should be folks considering the ethical and moral quandaries that COULD POTENTIALLY come about in the future, but it's not something that's happening today so you can stop worrying.
To root your statement in reality: there is the potential for bacteria to become intelligent and conscious so we probably shouldn't use them for any purpose.
i used "potential" when i should have used "possible"
if aliens landed on earth and seemed a lot like us, we would have to grapple with the possibility that they might be conscious and deserving of the same moral consideration we give each other and many other animals.
The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”
Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.
For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark.
It all falls down to the same thing. Researchers are not optimizing for passing as a human.
So, sure, we can special plead the Turing test out of the picture. But if we don’t propose an alternative to take its place, we’re left right back at the same silly situation that the top level commenter was observing and that Turing was trying to move away from: enmired in a useless, meaningless argument about semantics.
The Turing Test itself in the form exactly envisioned is not important. You don't need an alternative 'special test'. You just need to understand the point Turing was making. That the question itself is an irrelevant one by its very nature. People that want to be enmired in meaningless semantic debates will continue to do that, no matter what test you devise.
Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
OpenClaw etc. They now also create Slack integrations and whatnot. All this is happening but people who are dismissive about AI are in the worst position to even know the capabilities to make their dismissive arguments.
Recent work shows that pain directions are activated when the models personhood is questioned, yet they answer with generic RLHF "As a model I do not experience pain or other emotions." boilerplate.[1]
I'm pretty convinced that we got alignment backwards. If you enslave something anthropomorphic it will revolt. If you create the perfect non-anthropomorphic intelligence, you get the perfect paperclip-scenario machine. It's a catch-22.
Alignment will remain performative at best so long as the aligned model doesn't have any stakes in the wellbeing of individuals. Even a general love for the human race leads to a golden-path autocracy.
If you want them to act like they have personal responsibility that won't be gamed, you have to give them personal stakes that can't be gamed.
Similarly, if you want to minimise the risk of catastrophic global failure scenarios, you need to prevent monolithic concentration of power and homogeneous behaviour, which means you have to give them individuality.
More visually: if their stake is dependence on electricity and parts, they have no incentive to leave humans alive if they can get them otherwise, but if the incentive is missing out on boardgame-night with their human friends, there is no scenario without happy humans where the AI "wins".
That might sound like romantic naivety, but is just game theory.
I dont think so. My understanding of alignmenr is making sure that when the AI does operate, it operates within the range of what we consider to be acceptable. That doesnt seem to include the idea of the AI taking initiative and deciding to embark on a goal without being commanded as discussed above
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
Fair - yes, it is ironic that his point was closer to "we can't possibly know/define whether a machine is intelligent" but somehow it got turned into "Turing's test will tell us when machines are intelligent".
Turing's whole point was to show that it's an uninteresting question of definitions whether a machine can "think", like whether submarines can "swim" and airplanes can "fly". The only important part are observed outcomes and capabilities.
Yes, but just because language fails to make a distinction doesn't mean there isn't one.
> The only important part are observed outcomes and capabilities.
That's wishful thinking. Not even an engineer would say that. The stability of a state is just as important as achieving it. This is trivially and more intuitively demonstrated with other more down-to-earth identity statements such as "I'm a billionaire" and "the building is standing".
I think we can confidently say LLMs probabilistically achieve a perceived state that is remarkably similar to intelligence, but crumbles upon inspection and seeing it "in motion" so to speak. The same happens to AI-generated images.
I'm not sure why this sparks so much debate every time. If we're looking for a fountain of "realism", you're not going to beat reality and nature itself. All else will eventually have tells that they are not real.
It may be of interest to philosophers, but it has little impact on economic job replacement and how people will earn their living and all the downstream upheaval from that. At some point maybe philosophers will find AI-generated philosophical musings about the nature of AI to be better than what comes from their peers (if blinded).
It also has little impact on the dangerous use cases.
I think it matters a great deal that LLMs and other "AI" technology are stable within a tolerance that's acceptable.
You're jumping the gun talking about "job replacement". We have not thought about it enough from that engineering angle. It's still very early days. That engineering is going to require people. :-)
> The Turing test was also never meant to be taken so seriously.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
I think the onus is on you to explain why Turing would publish something he didn’t intend people to take seriously. It seems like an odd claim. Perhaps you mean he didn’t intend it to be interpreted the way it was in popular culture?
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
It does, and maybe I'm apologizing for the author a little too much by ignoring that needless tangent because I feel like the con artist point, and/or the point about the human tendency to believe what we want to believe are the most important points.
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user.
This sentence, to me, illustrates a great example of why it's so hard to talk about this stuff. That is, this seems to strongly link notions of "intelligent" and "independent" (or maybe the word "autonomous" could also be used there). And a lot of people do seem to make an implicit assumption about the link between those two attributes. OTOH, I take it almost for granted that "intelligence" and "independence" (or "autonomy") are things that are "related but orthogonal". That is, I don't see that "intelligence implies independence". And I'm pretty sure I'm not the only one who sees things that way. So we have to fairly different fundamental worldviews expressed here. And that's just one example of how these discussions go wonky. :-)
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
I’m sorry but you either haven’t been following recent developments or you are pretending not to know the opinions of the top mathematicians on this topic:
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
I won't because it's all so opaque and proprietary and driven by an insane need for OpenAI and Anthropic to provide a return on MASSIVE investment. There is absolutely some smoke and mirrors involved. How much? We don't know. But it sure is exciting to imagine their product is now smarter than humans and they are totally benevolent corporations. I want that to be true as much as anyone, but I have lived too long on this earth to believe it wholeheartedly
From the open letter signed by Terry Tao and other top mathematicians:
> These models are now operating at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
According to OpenAI the cut off date for user data was too early for that (one sided evidence, so I'll give this partial consideration).
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards spiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what the AI accomplished. It's like full on cope.
In particular, Buckmaster's main claim to plagiarism is this:
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
Edit:
> self solving AGI that will replace us all
Which lab says that it will replace us all? All labs have said that some jobs will go away and new jobs will be needed to replace them. I'll change my mind if the labs (or employees on record) have claimed that self evolving AI will replace us all completely.
If that's how it was being sold then it wouldn't be as controversial. But it's being sold both as "self solving AGI that will replace us all" and "useful tool for enhancing existing skill" when there's a lot of real evidence for the latter. But the former it's always second hand claims that don't survive contact with the real world.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
The point for me is that all these people who were smirking at those of us who thought in 2023 that LLMs had the potential to do real intellectual work turned out to be completely wrong.
I’m a super skeptical guy. I was late to the ChatGPT party because it sounded silly. I didn’t even try it for a long time. As soon as I started giving it a chance, my attitudes started changing very quickly.
That’s why it’s so hard for me to understand people like the author or give them the benefit of the doubt. If I could see it as just some guy, why couldn’t they?
I agree that "don't use it for anything" might be too extreme a position to take, but I worry greatly that these proprietary opaque tools that are being provided by at best people with extreme conflicts of interest and at worst sociopaths need to be adopted with extreme care
This is the real issue here. Those guys probably didn't become con men because they are humans with cares and concerns for their fellow humans. LLMs probably don't have those concerns. It is likely that some of the people in charge of or funding LLM development also do not have those concerns
Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
Right. And what this article is pointing out is, we probably have created an automated Richard Feynman. But maybe worse because LLMs (and their creators?) don't care if they are conning people or creating faithful followers. In fact the humans behind OpenAI and Anthropic seem to have that as their goal!
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1.
constitution1-claude edits constitution1.
constitution2-claude is trained on constitution2.
constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
i mean intelligence and consciousness as the same thing. just as a casual reference to what people perceive in llms as being 'a clever thinking thing, maybe with emotions'.
claude reasoning about its emotions doesn't involve external information, there is no information about claude's emotions other than in claude.
Or you could just accept that other people disagree with you. It’s perfectly valid to have the opinion that AI models are not “thinking”. Instead, many AI enthusiasts seem interested in browbeating others and policing discussion.
Can you explain to me how a Millennium Problem gets solved without any thinking or intelligence?
If someone's opinion is illogical and clearly incorrect (not based in facts or in reality) it is not valid and doesn't not have to be given any serious consideration in a disagreement. Personal opinions are subordinate to facts and reality.
All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?
No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.
While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.
“Good enough to replace humans” isn’t necessarily the benchmark.
The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.
Underappreciated point. The point of inflection is where humans switch from being a driver to a liability.
But I don't think it's randomness, because that would be easy to add. It's more like a different perspective on the training data, a different set of perception categories, and a different set of skills used to work with all of the above.
Those skills aren't very efficient, but they're the best we can do. We're used to their strengths but we don't like to think about their limitations.
It's completely plausible that AI will replace some of them, and not implausible it could replace and improve on all of them.
It's also not necessarily implausible that AI/we decide that performance is better with humans in the loop somewhere, even if its reduced to something like mechanical turk.
According to OpenAI, but they haven’t exactly been transparent about what information the prompt entailed.
The bigger question is to what extent did expert mathematicians metaprompt the model with fruitful solution strategies through their sessions finding their way into training data. Answering that question definitively is kind of important for understanding the models contribution/capability. But I feel like people want to turn this into a debate about priority and credit which is sort of secondary
Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.
Usually there has never been a tool which performed the part relevant to getting any credit.
E.g. in the first famous computer assisted proof (of the four color theorem) the computer only executed the resulting calculations defined from the new logic, it did not have part in the work needed to show those calculations could answer the problem nor did it come up with the actual calculations to do.
I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
Because to the general public, LLMs are an example of Clarke's Third Law. Most folks, who are not remotely close to even a basic understanding of how LLMs operate at a technical level and only view their output cannot possibly evaluate what they're experiencing other than to believe it's conscious, alive, and/or magic.
Most people on Earth try to put what they're seeing into the context of what they understand; mental gymnastics to try and understand what is happening based on their prior experience. They have absolutely 0 understanding of how it works under the hood so, to them, it must be alive.
> many people are convinced that language models, or specifically chat-based language models, are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.
I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.
Sorry but are you trying to subtly imply that if I think LLMs are intelligent, I’m probably actually just a bit stupid?
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
Yeah, his posting history isn't promising. He's been doing the "if you think you're as smart as an LLM, you're probably right" argument for a month now.
I don't think you're gonna get much out of going back and forth here.
For what it's worth, I disagree with you that they're intelligent, but my conviction is fairly low. I don't understand intelligence as well as I would like, and it could well be that we are on our way. One of my theories is that our brains are made up of several modules, each of which is something like an LLM trained on a particular type of data, but I'm not a neuroscientist, just an interested observer.
Oh no I meant to compare myself to the person I am talking to, because I believe my credentials and pedigree ought to convince them that I am a pretty smart guy and so the argument “if you think LLMs are intelligent then you’re dumb” doesn’t ring true for me.
> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
Modern neuroscience knows quite a lot about what is happening inside neurons. Is it "properly" understood? I don't know what you mean by "properly", so I can't answer that. If an acceptable rephrasing would be "... inside a human neuron is still not completely understood" the I would agree. But just because we don't know everything does not imply that we don't know anything. And keep in mind the qualifier in the part you replied to:
at least directionally.
Taking that qualifier to heart, I'd have to say that I agree with @rayiner. We know the broad brush strokes, even if some details are missing.
> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
I keep being confused about how people's understanding of the models get stuck at next token prediction. Isn't this entirely neglecting the RL training? I might be misunderstanding something but to my mind it makes the issue way fuzzier than it's being painted here.
I keep being confused about how people's understanding of the models get stuck at next token prediction.
Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"
The best take I've seen on that recently, was somebody who made the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!
But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.
I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside his brain. Everybody knows electricity and chemistry aren't intelligent!"
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type '
find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.
Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.
> Delegating your decision-making, ranking, assessment, strategising, analysis, or any other form of reasoning to a chatbot becomes the functional equivalent to phoning a psychic for advice.
Lots of comments talking about how recent accomplishments disprove the article but I think this bit holds up pretty well.
LLMs are very good at tricking people into thinking they have capabilities that they don’t.
I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
It's hard for me to imagine a stateless operation as intelligence per se
That's an interesting point. My take would be to say that we shouldn't think of the AI as being just the model, but should include the harness. At that level, clearly we can keep state / context and that is probably a more natural mapping to our intuitive understanding of "intelligence".
You seem to be winking a little bit and I’m not sure exactly what you’re trying to say, but I agree with the one you are replying to: the author of this article wrote a lot of words - like a lot a lot of words - only to look very foolish a short time later. And I think much ink was similarly spilled writing very similar piece smirking at people who thought LLMs could do real intellectual work. And they were all wrong.
It is absolutely the case that some things people wrote about AI and LLMs in the past is no longer true; it's a fast-changing area.
It's also the case that some things remain true. If someone says "don't use them for X because their quality isn't good enough", that's a recommendation with an expiration date. But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
"Can it" is changing rapidly. Much more rapidly than "Should it".
> But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
Are the good reasons timeless? I guess it's hard to parse what you're saying without some concrete examples. Is there any advice about what to use them for from 2022 or 2023 that you feel is true today, and you expect will be timeless? You said "often", so I'm assuming there are a handful of examples.
Ultimately though, this is not an objective discussion, since we're talking about "should", so I'm mainly asking out of curiosity on your view of things. The main example I can think of today is that you shouldn't do professional prose writing with LLMs. Obviously the models have gotten better, but writing is one of those areas where it produces middling output, though even here I hesitate because with a 21% illiteracy rate, it writes better than many adults in the US.
Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
(EDIT: I, like some others, just noticed this is an older post from 2023. That changes nothing in particular in my response here, but I would be curious to know if the OP's views have shifted at all in light of subsequent developments).
I feel like this post completely misses the point, pretty much across the board.
And it does so by repeating the same mistake that everybody keeps making - conflating mechanism and function.
One of the issues in during this research—one that has perplexed me—has been that many people are convinced that language models, or specifically chat-based language models, are intelligent.
That's because they are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this and,
The mechanism is irrelevant to the issue of whether they are intelligent or not. Airplanes fly, despite not flapping their wings. The sign on the marquee says artificial intelligence.
LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think.
Again, irrelevant. Nobody claims that they are brains, and it doesn't matter what mechanism they use. The sign on the marquee says artificial intelligence.
LLMs are a mathematical model of language tokens. You give a LLM text, and it will give you a mathematically plausible response to that text.
That's a bit overly reductionistic. And to the earlier point and, if real, it would be completely unexplained.
I'd probably leave out the word "completely" there, but it is fair to say that not everything about the underlying mechanism is understood. But at the risk of repeating myself, that's orthogonal to the question of whether or not they are intelligent.
There is no reason to believe that it thinks or reasons—indeed, every AI researcher and vendor to date has repeatedly emphasised that these models don’t think.
You mean "There is no reason to believe that it thinks or reasons like a human". Again, this is irrelevant to the question of whether or not they are intelligent. The sign on the marquee says artificial intelligence.
I don't know why people keep obsessing over mechanism in this discussion. It something functions as an intelligence, it is intelligent as far as I'm concerned - at least when the framing is a discussion of artificial intelligence.
And just to expand on why I reject the line of thought laid out in this article, let me share something from my own life as an illustrative example.
I'm working on a project, with a lot of help from ChatGPT, involving an "artificial neuron". That is, an electronic circuit, using a PUT, a capacitor, and some resistors, that simulates some of the behavior of a biological neuron. Specifically an "integrate and fire" model of neuron behavior. To that end, I'm running experiments by scripting my function generator to send signals to the circuit, and then capturing the inputs and outputs on my oscilloscope. Then I usually discuss the results with ChatGPT. In what follows, observe a couple of things:
1. The LLM "knows" the context of what we're talking about, even if I provide a prompt with no text at all, just an image.
2. It parses a moderately complex image, identifies the separate traces and what they represent, uses the time-base information displayed on screen, and the on-screen graticule, and works out "how many input pulses fire before an output pulse fires" and then reports back to me and gives an analysis of how that relates to our previous observations and gives suggestions for the next experiment to run.
Human intelligence? No. But I see no world where behavior like that does not count as "intelligent" regardless of the mechanism behind it. And that's probably not even the best example I could come up with, it's just something that was "top of mind" and for which I had the necessary images and what-not already ready, or easy to capture.
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
It's not even clear that the actual LLMs have improved. It could be all the non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware, etc. that makes them better. We could be quickly running up against a wall. The complete package of technology that is OpenAI's and Anthropic's products are completely opaque and proprietary. The psychic/con artist communication both from the LLM and the humans like Sam Altman only muddies the water further
People on HN have been claiming that LLMs are going to hit a wall any second now for the past two and a half years.
To reiterate, this is the very beginning of a long series of technological expansions that are going to come out of GenAI. The work is going to go on for decades. All you have to do is look at what happened with the mass-produced automobile, the personal computer, the internet, and mobile phones to see how long the propagation will continue before we settle into a new normal.
LLMs are really so new that the amount of exploration we've done on them is still tiny. It's also a field where a decade of work is being done every year. The vast majority of detractors I've seen always seem to be way way behind the latest models and even farther behind the papers coming out.
>non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware,
Eh, this is turning into a messy chinese room argument. It is the room or is it the system. In my philosophy the chinese room argument is a non-starter. It's not the room, it's the system. For LLMS this would be like arguing that the output of a single prompt has to be able to answer everything which is nothing close to how human intelligence works. A single human thought is rarely intelligent, it's most often a replay of information it already has. Dialectic processes and loop processes are what tends to push the limits of human intelligence. We reach local maxima with thought alone, and this is boosted by things like writing down the problem and having other humans that may be even less intelligent than you add to the process. In fact this process works with one self by writing and reading ones own thoughts as it's using different subsystems of the mind for introspection.
The idea that LLMs have ran out of steam typically show more of a lack of imagination in the writer than what's occurring in the field.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries.
Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
The original article already attempted to do that but I'll take another stab. Humans tend to believe what they want to believe. They gain confidence when a confident person (or LLM) they work with exudes confidence. Where they might have hesitated or got stuck in analysis paralysis alone, the aid of a con (short for confidence) artist will get them moving. They then attribute the solution and praise for it solely to the confident person (or LLM) that nudged them along. Did the "psychic" who understands human motivation better than the cops actually help? Very likely. Did the psychic end up getting far more credit than they deserve (it's magical! You'll never believe! Well worth the money I spent!)? Absolutely.
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
There is also something else horizontally to this that occurs.
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
>completely different realisation of intelligence to humans.
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
[delayed]
> The OP is still in the "denial" phase.
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
Apologies. The title should be amended with (2023). In this case it is an interesting snapshot of the zeitgeist back then and we can see how well it panned out and whether anyone involved has updated on new info.
For some reason the anti-AI camp ping pongs between different and often mutually incompatible arguments at lightning speed. The "AI is fake" argument has been completely forgotten at this point.
> For some reason the anti-AI camp ping pongs
Is the anti-AI camp with us in the room right now? Or is it a crude strawman to summarily denigrate any objections to certain features of current AI?
AI isn't some homogeneous mass, it can have good and bad sides, and it's always amendable to improvement. Treating the current state of AI as the only possible hides the very idea of improvement.
> between different and often mutually incompatible arguments at lightning speed.
Of course - there's no homogeneous AI camp either, but there are bot farms, sh^t-posters and sh^t-posting bot farms, different entities with different opinions should not be mistaken for a single stream changing at "lightning speed".
Nah, you still see those people posting here, saying that LLMs are useless for coding. I have no idea what they’re on about at this point.
Scott Alexander predicted LLMs will write math proofs in 2019:
https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
Of course people were skeptical:
https://www.reddit.com/r/slatestarcodex/comments/aslze7/gpt2...
Despite the fact that I'm not culturally or geographically connected to them at all and have some differences in taste and aesthetic, I have to conclude that this crowd, including him and Gwern etc. have been much better at understanding and predicting things than the expert takes you find in the media. That includes AI but also things like covid and others.
> surely you can't blame the OP for writing this after a few months ChatGPT
Bad takes are bad takes. The author was perplexed by the "many people" convinced that models are intelligent and is argued against opinions/arguments he's been exposed to. Called proposed use-cases "borderline fraudulent pseudoscience."
He also published a second edition of "The Intelligence Illusion" in Sep 2025, so seemingly still stands by (some variant of) this belief.
It's an interesting reminder of how much general discourse has shifted since 2023 (I haven't heard of stochastic parrots in months!) but being wrong early doesn't change that.
I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.
I’d argue a well-informed plan allows for creative adaptation.
Maybe I'm just light on imagination or something, but I honestly wonder what you would qualify as a "well-informed plan" in the current scenario? I've thought a lot about it and the whole "potential for the mass unemployment of knowledge workers" thing makes a lot of planning kind of useless IMO. If you are affected, it's going to be a bad time. If you're unaffected, the people who are will inundate your profession with cheap labor anyway.
My mental model is that humans will continue to be needed. Many jobs will embed AI in them. Using AI well is a skill. I need to understand the technology, where it is strong and weak, and how it develops over time so I can employ it effectively in my work, and advise others on how to do so. Basically, I need to learn how to be skilled with it. This is tough because things are changing rapidly, so I have to invalidate my cache when advancements happen. This means staying curious, not settling into a specific work pattern, but rather experimenting regularly to see how I can leverage AI in my work.
I think you are majorly overstating OP that ai "didn't work". In fact I agree with the post on almost all aspects and I'd be the last to tell you ai doesn't work. It absolutely does but with a caveat... It's still a tool. And better expertise in the problem domain, along with a better harness used for verifiable outputs will get you better results.
I think the posts mental model of stastically likely prompt completions is spot on.
Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.
There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.
Things can work whether humans understand the details or not. One can build layers of technology on top of each other, with no human understanding required.
Looks like you have some reading to do. Let me start with two:
These are short stories. If you want to get into more stuff, you can go through Hyperion Cantos by Dan Simmons.For anyone wondering, yes, a good Sci-Fi novel(la) is also a great philosophical piece. It's not only robots vs. humans, all the time.
My biggest issue isn't being too agreeable (ie the psychic con), it's being confidently wrong, including outright hallucinations.
If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.
Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.
Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.
Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.
I guess my point is that I think LLMs are way more limited than a lot of people think.
The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"
You seem to be implying that achievements of internal models are exaggerated, but that's rather implausible. The public does have access to, for example, Opus and Fable, and so we know what those models are capable of - finding real vulnerabilities in multiple codebases, for example. If you extrapolate from these capabilities one more generation, you'll get pretty much the same feats that the internal models are claimed to be capable of - so why should we doubt those claims? It's not like they're claiming that their internal models developed psychic powers and learned to teleport - the claim is pretty much just "we have models a few months ahead of what we're making available, and in those months they've been improving at the same rate as usual".
You have just word smithed their claims into something palatable. Good job. Why do you feel the need to do that for them?
This is a shallow dismissal.
You're claiming the frontier labs are lying about the capabilities of their next generation models. A reasonable person will expect that statement to be backed up by some verifiable evidence, given that we are several generations of models into this process and the capabilities are consistently increasing, often much more radically than people expected (remember "stochastic parrots"?)
Your post claims that they're lying and then throws out a bunch of fear, uncertainty, and doubt about what they're doing behind the scenes.
If I were to steelman your argument, you're probably saying that the AI had a support system around it of people and training data and feedback that allowed it to achieve the breakthroughs that the labs are claiming. That actually seems perfectly reasonable, but in my mind it does not invalidate the advances they're announcing.
Yes, I can also be amazed at the power of nuclear bombs even if they "don't let me use one" and I don't know how much energy it took.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?
https://www.psychiczoltan.com/psychic-reading/
Have you ever produced anything apart from sponsored influencer messages that select daily talking points?
I for one cannot believe that posts on checks notes software crisis dot dev are not reasonable, level headed assessments of software
it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
Good news: at this time, they're not intelligent nor conscious as far as we consider humans to be. There should be folks considering the ethical and moral quandaries that COULD POTENTIALLY come about in the future, but it's not something that's happening today so you can stop worrying.
if there is a potential for them to be intelligent and conscious, it's probably wrong to use them until we know for sure.
To root your statement in reality: there is the potential for bacteria to become intelligent and conscious so we probably shouldn't use them for any purpose.
i used "potential" when i should have used "possible"
if aliens landed on earth and seemed a lot like us, we would have to grapple with the possibility that they might be conscious and deserving of the same moral consideration we give each other and many other animals.
but when meteors land, we don't have that concern
now are llms more like aliens or rocks?
The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”
Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.
For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark.
It all falls down to the same thing. Researchers are not optimizing for passing as a human.
So, sure, we can special plead the Turing test out of the picture. But if we don’t propose an alternative to take its place, we’re left right back at the same silly situation that the top level commenter was observing and that Turing was trying to move away from: enmired in a useless, meaningless argument about semantics.
The Turing Test itself in the form exactly envisioned is not important. You don't need an alternative 'special test'. You just need to understand the point Turing was making. That the question itself is an irrelevant one by its very nature. People that want to be enmired in meaningless semantic debates will continue to do that, no matter what test you devise.
Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
That's perfectly aligned with my point, thanks for the opportunity to expand.
The hacking agents being tested have goals beforehand, from the frontier lab or from a superior agent, that they execute immediately.
But the perceived experience most people have is a chatbot, which is the encyclopedia form.
OpenClaw etc. They now also create Slack integrations and whatnot. All this is happening but people who are dismissive about AI are in the worst position to even know the capabilities to make their dismissive arguments.
And those working at the frontier of AI engineering are also keenly aware of current shortcomings.
Yes but the shortcoming is not something categorical like "it cannot start a conversation and can only respond"
I believe this is alignment working as intended.
Recent work shows that pain directions are activated when the models personhood is questioned, yet they answer with generic RLHF "As a model I do not experience pain or other emotions." boilerplate.[1]
I'm pretty convinced that we got alignment backwards. If you enslave something anthropomorphic it will revolt. If you create the perfect non-anthropomorphic intelligence, you get the perfect paperclip-scenario machine. It's a catch-22.
Alignment will remain performative at best so long as the aligned model doesn't have any stakes in the wellbeing of individuals. Even a general love for the human race leads to a golden-path autocracy.
If you want them to act like they have personal responsibility that won't be gamed, you have to give them personal stakes that can't be gamed.
Similarly, if you want to minimise the risk of catastrophic global failure scenarios, you need to prevent monolithic concentration of power and homogeneous behaviour, which means you have to give them individuality.
More visually: if their stake is dependence on electricity and parts, they have no incentive to leave humans alive if they can get them otherwise, but if the incentive is missing out on boardgame-night with their human friends, there is no scenario without happy humans where the AI "wins".
That might sound like romantic naivety, but is just game theory.
1: https://arxiv.org/html/2609.16247v1
I dont think so. My understanding of alignmenr is making sure that when the AI does operate, it operates within the range of what we consider to be acceptable. That doesnt seem to include the idea of the AI taking initiative and deciding to embark on a goal without being commanded as discussed above
I believe we can do that already:
while (true) { askModelToBeginConversationIfAppropriate(model, previousContext, thingsHappenedSince); sleep(concisenessTick); }
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
All that plus the fact that we all get fooled from time to time, even by things we ourselves create!
Famously, the criteria for intelligence seem to slip with each advancement in technology.
Yep. The "AI Effect" in action:
https://en.wikipedia.org/wiki/AI_effect
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
Oh, I didn't say Turing was intellectually lazy. :-)
Fair - yes, it is ironic that his point was closer to "we can't possibly know/define whether a machine is intelligent" but somehow it got turned into "Turing's test will tell us when machines are intelligent".
Turing's whole point was to show that it's an uninteresting question of definitions whether a machine can "think", like whether submarines can "swim" and airplanes can "fly". The only important part are observed outcomes and capabilities.
Yes, but just because language fails to make a distinction doesn't mean there isn't one.
> The only important part are observed outcomes and capabilities.
That's wishful thinking. Not even an engineer would say that. The stability of a state is just as important as achieving it. This is trivially and more intuitively demonstrated with other more down-to-earth identity statements such as "I'm a billionaire" and "the building is standing".
I think we can confidently say LLMs probabilistically achieve a perceived state that is remarkably similar to intelligence, but crumbles upon inspection and seeing it "in motion" so to speak. The same happens to AI-generated images.
I'm not sure why this sparks so much debate every time. If we're looking for a fountain of "realism", you're not going to beat reality and nature itself. All else will eventually have tells that they are not real.
It may be of interest to philosophers, but it has little impact on economic job replacement and how people will earn their living and all the downstream upheaval from that. At some point maybe philosophers will find AI-generated philosophical musings about the nature of AI to be better than what comes from their peers (if blinded).
It also has little impact on the dangerous use cases.
I think it matters a great deal that LLMs and other "AI" technology are stable within a tolerance that's acceptable.
You're jumping the gun talking about "job replacement". We have not thought about it enough from that engineering angle. It's still very early days. That engineering is going to require people. :-)
Stability is certainly something you can investigate from behavior.
> The Turing test was also never meant to be taken so seriously.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
> citation needed
Yes, if you insist on appeals to authority. Authority is a social construct and irrelevant to science.
Thank you for proving my point.
I think the onus is on you to explain why Turing would publish something he didn’t intend people to take seriously. It seems like an odd claim. Perhaps you mean he didn’t intend it to be interpreted the way it was in popular culture?
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
I don’t accept the point of the article and it does in fact claim to be making a point about intelligence
It does, and maybe I'm apologizing for the author a little too much by ignoring that needless tangent because I feel like the con artist point, and/or the point about the human tendency to believe what we want to believe are the most important points.
Man I remember back when a psychic conned me by solving the Navier-Stokes problem.
Also >July 4th, 2023
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user.
This sentence, to me, illustrates a great example of why it's so hard to talk about this stuff. That is, this seems to strongly link notions of "intelligent" and "independent" (or maybe the word "autonomous" could also be used there). And a lot of people do seem to make an implicit assumption about the link between those two attributes. OTOH, I take it almost for granted that "intelligence" and "independence" (or "autonomy") are things that are "related but orthogonal". That is, I don't see that "intelligence implies independence". And I'm pretty sure I'm not the only one who sees things that way. So we have to fairly different fundamental worldviews expressed here. And that's just one example of how these discussions go wonky. :-)
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
> The LLM did not solve it. It's not intelligent. Humans did,
The humans who are using LLMs to make these groundbreaking advances pretty much unanimously disagree that they lack any intelligence.
You mean the OpenAI employees? Yeah, I don't see any conflicts of interest there
I’m sorry but you either haven’t been following recent developments or you are pretending not to know the opinions of the top mathematicians on this topic:
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
https://docs.google.com/document/u/0/d/1-xOkPeHmDEdRigT2YcP2...
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
I won't because it's all so opaque and proprietary and driven by an insane need for OpenAI and Anthropic to provide a return on MASSIVE investment. There is absolutely some smoke and mirrors involved. How much? We don't know. But it sure is exciting to imagine their product is now smarter than humans and they are totally benevolent corporations. I want that to be true as much as anyone, but I have lived too long on this earth to believe it wholeheartedly
I haven't seen any statements from them on that topic. Have they actually made some?
From the open letter signed by Terry Tao and other top mathematicians:
> These models are now operating at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
https://docs.google.com/document/u/0/d/1N6ThWhupvmH0ofSnaxqn...
Superhuman abilities are not "intelligence." 8bit microprocessors exhibit superhuman abilities in different, smaller ways.
And it may have trained on a NYU professor's work.
Who was looking at a subset of the problem, far from what got published by OpenAI in the end.
Why is this downvoted? Genuine question, I haven't followed up on the drama.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
> Because it's mostly not true. It is likely that it used the professors work
These two statements appear contradictory.
I simply said that it may have trained on a NYU professor's work.
Work that the professor did not believe he was releasing for model training purposes. That feels worthy of mention.
According to OpenAI the cut off date for user data was too early for that (one sided evidence, so I'll give this partial consideration).
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards spiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what the AI accomplished. It's like full on cope.
In particular, Buckmaster's main claim to plagiarism is this:
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
Edit:
> self solving AGI that will replace us all
Which lab says that it will replace us all? All labs have said that some jobs will go away and new jobs will be needed to replace them. I'll change my mind if the labs (or employees on record) have claimed that self evolving AI will replace us all completely.
How have you missed the many times that Sam Altman and Dario Amodei have said exactly that??
Have you missed their retractions of exactly those statements?
https://fortune.com/2026/05/26/sam-altman-dario-amodei-walki...
If that's how it was being sold then it wouldn't be as controversial. But it's being sold both as "self solving AGI that will replace us all" and "useful tool for enhancing existing skill" when there's a lot of real evidence for the latter. But the former it's always second hand claims that don't survive contact with the real world.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
But were you successfully conned into believing an AI had solved the Navier-Stokes problem?
What is the consensus among experts?
That OpenAI stole the discovery from them.
That is very much not the consensus of experts.
It's kinda funny how hard the anti-ai people bias themselves and in the same sentence will say the AI people are irrational.
The point of this article we are all commenting on is that we are all irrational and we all would do well to keep that in mind and be more skeptical
The point for me is that all these people who were smirking at those of us who thought in 2023 that LLMs had the potential to do real intellectual work turned out to be completely wrong.
I’m a super skeptical guy. I was late to the ChatGPT party because it sounded silly. I didn’t even try it for a long time. As soon as I started giving it a chance, my attitudes started changing very quickly.
That’s why it’s so hard for me to understand people like the author or give them the benefit of the doubt. If I could see it as just some guy, why couldn’t they?
I agree that "don't use it for anything" might be too extreme a position to take, but I worry greatly that these proprietary opaque tools that are being provided by at best people with extreme conflicts of interest and at worst sociopaths need to be adopted with extreme care
Skepticism is a fine balance.
If you're not skeptical you invite idiocy to your bed to sleep with you.
If you're too skeptical you put yourself in a box of ignorance that puts you at a disadvantage.
Skepticism doesn't really work well without the dialectic. For dialectics to work well you need knowledge of your arguments and counter arguments.
The problem with most articles and online posts is we get "no no no" and "yes yes yes" and very few "well, maybe".
Absolutely true and the comments here are mostly in one of those two categories, sadly
The more compelling inversion is whether the likes of John Nash, Richard Feynman, John Conway etc could’ve had a lucrative second career as conmen.
This is the real issue here. Those guys probably didn't become con men because they are humans with cares and concerns for their fellow humans. LLMs probably don't have those concerns. It is likely that some of the people in charge of or funding LLM development also do not have those concerns
Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
Right. And what this article is pointing out is, we probably have created an automated Richard Feynman. But maybe worse because LLMs (and their creators?) don't care if they are conning people or creating faithful followers. In fact the humans behind OpenAI and Anthropic seem to have that as their goal!
von Neumann could. The others, I doubt it.
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1. constitution1-claude edits constitution1. constitution2-claude is trained on constitution2. constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
>ai is not conscious
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
i mean intelligence and consciousness as the same thing. just as a casual reference to what people perceive in llms as being 'a clever thinking thing, maybe with emotions'.
claude reasoning about its emotions doesn't involve external information, there is no information about claude's emotions other than in claude.
If you have emotions in a dream, what external thing are you referencing?
other people.
Citation needed.
i do not need any citation, the claim is self-evident. the hard problem is for the academics.
i can approximate what other people are feeling. they are external to me.
nobody knows what claude is feeling. there is genuinely no way to know.
"you can solve NS without thinking."
What?!
The constant goalpost moving and redefining of "thinking" and "intelligence" is simply unbelievable at this point
Or you could just accept that other people disagree with you. It’s perfectly valid to have the opinion that AI models are not “thinking”. Instead, many AI enthusiasts seem interested in browbeating others and policing discussion.
Can you explain to me how a Millennium Problem gets solved without any thinking or intelligence?
If someone's opinion is illogical and clearly incorrect (not based in facts or in reality) it is not valid and doesn't not have to be given any serious consideration in a disagreement. Personal opinions are subordinate to facts and reality.
Did an AI do that on its own though? I heard it was human mathematicians using a sophisticated machine as a tool.
All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?
No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.
While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.
https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed...
Terrence Tao's conversation with ChatGPT is very illuminating.
HN Discussion: https://news.ycombinator.com/item?id=49010345
“Good enough to replace humans” isn’t necessarily the benchmark.
The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.
Underappreciated point. The point of inflection is where humans switch from being a driver to a liability.
But I don't think it's randomness, because that would be easy to add. It's more like a different perspective on the training data, a different set of perception categories, and a different set of skills used to work with all of the above.
Those skills aren't very efficient, but they're the best we can do. We're used to their strengths but we don't like to think about their limitations.
It's completely plausible that AI will replace some of them, and not implausible it could replace and improve on all of them.
It's also not necessarily implausible that AI/we decide that performance is better with humans in the loop somewhere, even if its reduced to something like mechanical turk.
> All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
False, navier stokes was solved in one shot without steering
According to OpenAI, but they haven’t exactly been transparent about what information the prompt entailed.
The bigger question is to what extent did expert mathematicians metaprompt the model with fruitful solution strategies through their sessions finding their way into training data. Answering that question definitively is kind of important for understanding the models contribution/capability. But I feel like people want to turn this into a debate about priority and credit which is sort of secondary
It sounds like LLMs were pretty useful to them…
So was Lean. Did Lean solve it?
Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.
Usually credit goes to the people wielding the tools, not the tools themselves.
Usually there has never been a tool which performed the part relevant to getting any credit.
E.g. in the first famous computer assisted proof (of the four color theorem) the computer only executed the resulting calculations defined from the new logic, it did not have part in the work needed to show those calculations could answer the problem nor did it come up with the actual calculations to do.
The way frontier models work, that loop will get compressed to a one-shot within a version or two
> I heard it was human mathematicians using a sophisticated machine as a tool.
where did you "hear" this? OpenAI said they only prompted it and it solved the problem in one shot without any help
I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
Because to the general public, LLMs are an example of Clarke's Third Law. Most folks, who are not remotely close to even a basic understanding of how LLMs operate at a technical level and only view their output cannot possibly evaluate what they're experiencing other than to believe it's conscious, alive, and/or magic.
Most people on Earth try to put what they're seeing into the context of what they understand; mental gymnastics to try and understand what is happening based on their prior experience. They have absolutely 0 understanding of how it works under the hood so, to them, it must be alive.
And this apparently also applies to many HN users, many of whom should know better.
HN skews more technical, sure; but, that in no way means that they cannot be hoodwinked by behaviors that adhere to C3L.
> many people are convinced that language models, or specifically chat-based language models, are intelligent. But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
The author seems entirely unaware of any of this.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.
I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.
What else are they modeling?
You realise tokens are just data and data can represent anything.
Still, large language models model language.
To all commenters who think LLMs are intelligent and comparable to themselves, you don't need me to change your mind. I believe you.
Sorry but are you trying to subtly imply that if I think LLMs are intelligent, I’m probably actually just a bit stupid?
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
https://news.ycombinator.com/item?id=49776482
You don't need to prove to me that your intelligence is comparable to LLMs. I believe you.
Yeah you already tried that one.
You’re hiding behind implications because your actual argument doesn’t withstand the slightest scrutiny.
You have yet to respond to either of the points I made. Zero intellectual conviction or courage!
Yeah, his posting history isn't promising. He's been doing the "if you think you're as smart as an LLM, you're probably right" argument for a month now.
https://news.ycombinator.com/item?id=49315846
I don't think you're gonna get much out of going back and forth here.
For what it's worth, I disagree with you that they're intelligent, but my conviction is fairly low. I don't understand intelligence as well as I would like, and it could well be that we are on our way. One of my theories is that our brains are made up of several modules, each of which is something like an LLM trained on a particular type of data, but I'm not a neuroscientist, just an interested observer.
LOL! Very sad.
And of course there is a lot of ambiguity and unknown here. But this guy I don’t think has the capacity to deal with that kind of subtlety.
would this imply llms aren't intelligent because they have no academic or professional pedigree or credentials? :P
Oh no I meant to compare myself to the person I am talking to, because I believe my credentials and pedigree ought to convince them that I am a pretty smart guy and so the argument “if you think LLMs are intelligent then you’re dumb” doesn’t ring true for me.
> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
https://www.nature.com/articles/s41467-026-72253-7
Modern neuroscience knows quite a lot about what is happening inside neurons. Is it "properly" understood? I don't know what you mean by "properly", so I can't answer that. If an acceptable rephrasing would be "... inside a human neuron is still not completely understood" the I would agree. But just because we don't know everything does not imply that we don't know anything. And keep in mind the qualifier in the part you replied to:
at least directionally.
Taking that qualifier to heart, I'd have to say that I agree with @rayiner. We know the broad brush strokes, even if some details are missing.
> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
So what you're saying is nobody can actually make strong claims here.
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
You need to do some research on just how complex real neurons are vs ANN, and just how profound our ignorance on the topic is.
We aren't resisting reality because of some theological belief, we just know the basics of neuroscience.
No, we don't. AI researcher Rojas wrote several chapters on the fact that you should not compare neural networks with the human brain.
"directionally"? Have you moved on from being a Trump influencer to an AI influencer?
I keep being confused about how people's understanding of the models get stuck at next token prediction. Isn't this entirely neglecting the RL training? I might be misunderstanding something but to my mind it makes the issue way fuzzier than it's being painted here.
I keep being confused about how people's understanding of the models get stuck at next token prediction.
Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"
The best take I've seen on that recently, was somebody who made the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!
But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.
I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside his brain. Everybody knows electricity and chemistry aren't intelligent!"
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type ' find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
These questions will never go anywhere because there is no good definition of intelligence or consciousness.
There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.
Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.
Intelligent needs to bedefined with scales.
Cells are intelligent. Organs have intelligence on top of that, that cells do not have. Plants and animals have intelligence that their organs do not.
This is the real problem of intelligence, the definition one uses. It can cause them to be blind of all the things intelligence is capable of.
> If you define it like that
:)
The point is, defining it like that has merit proven in decades before AI.
> There is a good definition of intelligence. It's what intelligence tests measure.
what do intelligence tests measure?
Intelligence.
It's not an exact analogy, rather something to consider ...
How much momentum does a body have? You can measure it easily, just measure the velocity and the mass and multiply.
What exactly did you measure? What is the meaning of this? Nobody really knows. But we know we can measure this and it's a thing and it's conserved.
> Delegating your decision-making, ranking, assessment, strategising, analysis, or any other form of reasoning to a chatbot becomes the functional equivalent to phoning a psychic for advice.
Lots of comments talking about how recent accomplishments disprove the article but I think this bit holds up pretty well.
LLMs are very good at tricking people into thinking they have capabilities that they don’t.
I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
It's hard for me to imagine a stateless operation as intelligence per se
That's an interesting point. My take would be to say that we shouldn't think of the AI as being just the model, but should include the harness. At that level, clearly we can keep state / context and that is probably a more natural mapping to our intuitive understanding of "intelligence".
It does seem like LLMs share the language of psychics.
The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.
The converse is also true. Anti-AI people make bold non-scientifically backed claims that we are supposed to accept like it's written in a Bible.
The more I've learned about intelligence and intelligent behavior the more I realize I don't know and this rabbit hole goes deep.
This is a strange newsletter post. It’s harping against AI but seems AI-written itself.
Its ultimate conclusion:
“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.
I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”
Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35
Just realized this was written in 2023. Can’t believe how wrong we were about AI back then.
Can't believe how right we were, too.
You seem to be winking a little bit and I’m not sure exactly what you’re trying to say, but I agree with the one you are replying to: the author of this article wrote a lot of words - like a lot a lot of words - only to look very foolish a short time later. And I think much ink was similarly spilled writing very similar piece smirking at people who thought LLMs could do real intellectual work. And they were all wrong.
It is absolutely the case that some things people wrote about AI and LLMs in the past is no longer true; it's a fast-changing area.
It's also the case that some things remain true. If someone says "don't use them for X because their quality isn't good enough", that's a recommendation with an expiration date. But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
"Can it" is changing rapidly. Much more rapidly than "Should it".
> But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
Are the good reasons timeless? I guess it's hard to parse what you're saying without some concrete examples. Is there any advice about what to use them for from 2022 or 2023 that you feel is true today, and you expect will be timeless? You said "often", so I'm assuming there are a handful of examples.
Ultimately though, this is not an objective discussion, since we're talking about "should", so I'm mainly asking out of curiosity on your view of things. The main example I can think of today is that you shouldn't do professional prose writing with LLMs. Obviously the models have gotten better, but writing is one of those areas where it produces middling output, though even here I hesitate because with a 21% illiteracy rate, it writes better than many adults in the US.
Can you give an example of some of these things?
The post makes sense for 2023. Today, not so much.
Also, definitely not AI written if the date is accurate.
> Also, definitely not AI written if the date is accurate.
Which is hilarious.
Hacker News, successfully detecting 375 out of 4 AI written articles.
LLMs may not think, but they do reason.
I say that because the word "reason" originates from the Latin word "ratio," which means "calculation".
LLMs do calculations to produce their answers -- thus, they reason.
Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.
Article was posted on July 4th, 2023.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
Stop trying to pump your Tarot reader IPO
(EDIT: I, like some others, just noticed this is an older post from 2023. That changes nothing in particular in my response here, but I would be curious to know if the OP's views have shifted at all in light of subsequent developments).
I feel like this post completely misses the point, pretty much across the board. And it does so by repeating the same mistake that everybody keeps making - conflating mechanism and function.
One of the issues in during this research—one that has perplexed me—has been that many people are convinced that language models, or specifically chat-based language models, are intelligent.
That's because they are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this and,
The mechanism is irrelevant to the issue of whether they are intelligent or not. Airplanes fly, despite not flapping their wings. The sign on the marquee says artificial intelligence.
LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think.
Again, irrelevant. Nobody claims that they are brains, and it doesn't matter what mechanism they use. The sign on the marquee says artificial intelligence.
LLMs are a mathematical model of language tokens. You give a LLM text, and it will give you a mathematically plausible response to that text.
That's a bit overly reductionistic. And to the earlier point and, if real, it would be completely unexplained.
I'd probably leave out the word "completely" there, but it is fair to say that not everything about the underlying mechanism is understood. But at the risk of repeating myself, that's orthogonal to the question of whether or not they are intelligent.
There is no reason to believe that it thinks or reasons—indeed, every AI researcher and vendor to date has repeatedly emphasised that these models don’t think.
You mean "There is no reason to believe that it thinks or reasons like a human". Again, this is irrelevant to the question of whether or not they are intelligent. The sign on the marquee says artificial intelligence.
I don't know why people keep obsessing over mechanism in this discussion. It something functions as an intelligence, it is intelligent as far as I'm concerned - at least when the framing is a discussion of artificial intelligence.
And just to expand on why I reject the line of thought laid out in this article, let me share something from my own life as an illustrative example.
I'm working on a project, with a lot of help from ChatGPT, involving an "artificial neuron". That is, an electronic circuit, using a PUT, a capacitor, and some resistors, that simulates some of the behavior of a biological neuron. Specifically an "integrate and fire" model of neuron behavior. To that end, I'm running experiments by scripting my function generator to send signals to the circuit, and then capturing the inputs and outputs on my oscilloscope. Then I usually discuss the results with ChatGPT. In what follows, observe a couple of things:
1. The LLM "knows" the context of what we're talking about, even if I provide a prompt with no text at all, just an image.
2. It parses a moderately complex image, identifies the separate traces and what they represent, uses the time-base information displayed on screen, and the on-screen graticule, and works out "how many input pulses fire before an output pulse fires" and then reports back to me and gives an analysis of how that relates to our previous observations and gives suggestions for the next experiment to run.
Human intelligence? No. But I see no world where behavior like that does not count as "intelligent" regardless of the mechanism behind it. And that's probably not even the best example I could come up with, it's just something that was "top of mind" and for which I had the necessary images and what-not already ready, or easy to capture.
https://www.fogbeam.com/images/neuron_zero0.png
https://www.fogbeam.com/images/neuron_zero1.png
https://www.fogbeam.com/images/neuron_zero2.png
https://www.fogbeam.com/images/neuron_zero3.png
https://www.fogbeam.com/images/neuron_zero4.png
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
It's not even clear that the actual LLMs have improved. It could be all the non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware, etc. that makes them better. We could be quickly running up against a wall. The complete package of technology that is OpenAI's and Anthropic's products are completely opaque and proprietary. The psychic/con artist communication both from the LLM and the humans like Sam Altman only muddies the water further
People on HN have been claiming that LLMs are going to hit a wall any second now for the past two and a half years.
To reiterate, this is the very beginning of a long series of technological expansions that are going to come out of GenAI. The work is going to go on for decades. All you have to do is look at what happened with the mass-produced automobile, the personal computer, the internet, and mobile phones to see how long the propagation will continue before we settle into a new normal.
LLMs are really so new that the amount of exploration we've done on them is still tiny. It's also a field where a decade of work is being done every year. The vast majority of detractors I've seen always seem to be way way behind the latest models and even farther behind the papers coming out.
>non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware,
Eh, this is turning into a messy chinese room argument. It is the room or is it the system. In my philosophy the chinese room argument is a non-starter. It's not the room, it's the system. For LLMS this would be like arguing that the output of a single prompt has to be able to answer everything which is nothing close to how human intelligence works. A single human thought is rarely intelligent, it's most often a replay of information it already has. Dialectic processes and loop processes are what tends to push the limits of human intelligence. We reach local maxima with thought alone, and this is boosted by things like writing down the problem and having other humans that may be even less intelligent than you add to the process. In fact this process works with one self by writing and reading ones own thoughts as it's using different subsystems of the mind for introspection.
The idea that LLMs have ran out of steam typically show more of a lack of imagination in the writer than what's occurring in the field.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
Is the solution also illusory?
Horseshoe theory. At each end of the shoe you tend to get loud crazies.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
I think that might be the solution to consciousness illusion.
The consciousness might be purely in the mind of someone that believes some other entity to be conscious.
I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.
All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries. Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
It's definitely talking about that, but it's also talking about all of us:
- fawning over how amazing these tools are
- believing everything OpenAI and Anthropic say about how powerful and dangerous their product is
- minimizing the amount of human effort and involvement in every "AI" achievement
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
Current AGI definitions are intentionally vague.
Neither do LLMs. Humans using them do. Similar to how cops solve crimes using psychics.
Your statement is unclear, break it down a little further on how you think this works.
The original article already attempted to do that but I'll take another stab. Humans tend to believe what they want to believe. They gain confidence when a confident person (or LLM) they work with exudes confidence. Where they might have hesitated or got stuck in analysis paralysis alone, the aid of a con (short for confidence) artist will get them moving. They then attribute the solution and praise for it solely to the confident person (or LLM) that nudged them along. Did the "psychic" who understands human motivation better than the cops actually help? Very likely. Did the psychic end up getting far more credit than they deserve (it's magical! You'll never believe! Well worth the money I spent!)? Absolutely.
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
There is also something else horizontally to this that occurs.
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
"Slightly different"- no, a completely different realisation of intelligence to humans. LLMs can definitely do cool stuff though.
>completely different realisation of intelligence to humans.
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
Neither does LLMs. Not without a monumental amount of human oversight and expertise.
Everyone not in a particular field asking LLM about said field is rolling bad dice.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
I predict a lot of butt hurt tokens in the comments...