I am sympathetic to the reasoning as to why LLMs should not be used to help some programmers right now. But I get a little frustrated seeing many of these kinds of posts that talk about fundamental limitations of LLMs vs humans on the grounds that it cannot "logically reason" like a human does. These are limitations in the current approach to training and objectives; internally, we have no clue what is going on.
> it’s “just a statistical model” that generates “language” based on a chain of “what is most likely to follow the previous phrase”
Humans are statistical models too in an appropriate sense. The question is whether we try to execute phrase by phrase or not, or whether it even matters what humans do in the long term.
> The only way ChatGPT will stop spreading that nonsense is if there is a significant mass of humans talking online about the lack of ZSTD support.
Or you can change the implicit bias in the model by being more clever with your training procedure. This is basic stats here, not everything is about data.
> They don’t know anything, they don’t think, they don’t learn, they don’t deduct. They generate real-looking text based on what is most likely based on the information it has been trained on.
This may be comforting to think, but it's just wrong. It would make my job so much easier if it were true. If you take the time to define "know", "think", and "deduct", you will find it difficult to argue current LLMs do not do these things. "Learn" is the exception here, and is a bit more complex, not only because of memory and bandwidth issues, but also because "understand" is difficult to define.
>Humans are statistical models too in an appropriate sense.
No, we aren't and I'm getting tired of this question begging and completely wrong statement. Human beings are capable of what Kant in fancy words called "transcendental apperception", we're already bringing our faculties to bear on experience without which the world would make no sense to us.
What that means in practical terms for programming problems of this kind is that, we can say "I don't know", which the LLM can't, because there's no "I", in the LLM, no unified subject that can distinguish what it knows and what it doesn't, what's within its domain of knowledge or outside.
>If you take the time to define "know", "think", and "deduct", you will find it difficult to argue current LLMs do not do these things
No, only if you don't spend the time to think about what knowledge is you'd make such a statement. What enables knowledge, which is not raw data but synthesized, structured cognition, is the faculties of the mind a priori categories we bring to bear on data.
That's why these systems are about as useless as a monkey with a typewriter when you try to have them work on manual memory management in C, because that's less of a task in auto completion and requires you to have in your mind a working model of the machine.
This is interesting philosophy, and others have better critiques here in that regard. I'm a mathematician, so I can only work in what I can define symbolically. Humans most certainly ARE statistical models by that definition: without invoking the precise terminology, we take input, yield output, and plausibly involve uncertain elements. One can argue as to whether this is the correct language or not, but I prefer to think this way, as the arrogance of human thinking has otherwise failed us in making good predictions about AI.
If you can come up with a symbolic description of a deficiency in how LLMs approach problems, that's fantastic, because we can use that to alter how these models are trained, and how we approach problems too!
> What that means in practical terms for programming problems of this kind is that, we can say "I don't know", which the LLM can't, because there's no "I", in the LLM, no unified subject that can distinguish what it knows and what it doesn't, what's within its domain of knowledge or outside.
We seriously don't know whether there is an "I" that is comprehended or not. I've seen arguments either way. But otherwise, this seems to refer to poor internal calibration of uncertainty, correct? This is an important problem! (It's also a problem with humans too, but I digress). LLMs aren't nearly as bad as this as you might think, and there are a lot of things you can do (that the big tech companies do not do) that can better tune it's own self-confidence (as reflected in logits). I'm not aware of anything that uses this information as part of the context, so that might be a great idea. But on the other hand, maybe this actually isn't as important as we think it is.
Yes, of course. I'm not trying to establish further argument or to criticise. But it isn't fruitful to have a conversation where both parties have different definitions in mind. Since my original comment referred to the mathematical definition, it is important for me to clarify that.
The rest of my comment is to try to find some middle ground, of which there is plenty in philosophy. If the raw theory of mind arguments for humans were already applicable for recognizing the limitations of LLM designs, we would be much further along than we currently are.
The position of Kant does not align with the direction modern neuroscience is heading towards. Current evidence seems to prefer decentralized theories of consciousness like Dennett's multiple drafts model[1], suggesting there is no central point where everything comes together to form conscious experience, but instead that it itself is constituted by collaborative processes that have multiple realizations.
>Current evidence seems to prefer decentralized theories of consciousness like Dennett
There is no such thing as consciousness in Dennett's theory, his position is that it doesn't exist, he is a Eliminativist. This is of course an absurd position with no evidence for it as people like Chalmers have pointed out (including in that Wikipedia article), and it might be the most comical and ideological position in the last 200 years.
> However, Dennett is not denying the existence of the mind or of consciousness, only what he considers a naive view of them
It doesn't seem like he's Eliminativist. It also seems like the criticisms rely on harping on about qualia, which is one of the sillier schools of sophistry. I'd need to see actual criticisms before believing that Dennett is pushing for something comical.
While the normal distribution meme is notoriously overused, I think it fits the scenario here.
LLMs know so much (when you just use ChatGPT for the first time like it's an Oracle machine) -> LLMs don't know anything (when you understand how machine learning works) -> LLMs know so much (when you actually think about what 'know' means)
Reason? Maybe. But there's one limitation that we currently have no idea how to overcome; LLMs don't know how much they know. If they tell you they don't something it may be a lie. If they tell you they do, this may be a lie too. I, a human, certainly know what I know and what I don't and can recall from where I know the information
I have never met a human who has a good grasp of what they know and don't know. They may have a better graps of it than an LLM, but humans are awfully bad at understanding the limits of our own knowledge, and will argue very strongly in favour of knowing more than we demonstrably do in all kinds of contexts.
> I have never met a human who has a good grasp of what they know and don't know
yep. There are 2 processes underlying our behaviors.
1) a meme-copier which takes in information from various sources and regurgitates it. A rote-memory machine of sorts. Here, memory is populated with info without deconstructing it into a theory. This has been termed "know-that". Here, explanations are shallow, and repeated questioning of why will fail with: I was told so.
2) a builder which attempts to construct a theory of something. Here, a mechanism is built and understood, and information is deconstructed in the context of the mechanism. This has been termed "know-how". Here, explanations are deeper, and repeated questioning of why will end up with: these are the "givens" in our theory.
Problem is that we operate in the "know-that" territory most of the time, and have not taken the effort to build theories for ourselves in the "know-how" territory.
LLMs are not humans, they are ostensibly tools. Tools are supposed to come with a list of things they can do. LLMs don’t and are therefore bad at being tools, so we anthropomorphize them. But they are also not good at being people, so LLMs are left in this weird in-between state where half the people say they’re useful and half the people say they cause more problems than they solve.
There's no question they have capabilities that no other tool has. But a good tool goes beyond just doing something, there are some generally agreed upon principles of tool design that make something good versus just useful.
For example, I think a hammer is a good tool because every time I swing it at a nail, it provides the force necessary to drive it into the wood. It's reliable. Sure, sometimes a hammer breaks, but my baseline expectation in using one is that every time I swing the hammer it will behave the same way.
Something more complicated, like a Rust compiler is also a good tool in the same way. It's vastly more intricate than the hammer, yet it still has the good tool property of being reliable; every time I press compile, if the program is wrong, then the compiler tells me that. If it's right, then the compiler passes every time. It doesn't lie, it doesn't guess, it doesn't rate limit, it doesn't charge a subscription, it doesn't silently update causing code to fail, it informs when changes are breaking and what they are, it allows me to pick my version and doesn't silently deprecate me without recourse, etc.
There are of course ecosystems out there where building a project is more like a delicate dance, or a courtship ritual, and those ecosystems are pain in the ass to deal with. I'm talking XKCD #1987, or NodeJS circa 2014, or just the entire rationale behind Docker. People exit their careers to not have to deal with such technology, because it's like working at the DMV or living in Kafka's nightmares. LLMs are more in that direction, and no one is going to like where we end up if we make them our entire stack, as seems to be the intent by the powers that be.
There's a difference between what LLMs are and what they're being sold as. For what they are, they can be useful and may one day they will be turned into good tools if some of the major flaws are fixed.
On the other hand, we are in the process of totally upending the way our industry works on the basis of what these things will be, which they are selling as essentially an oracle. "The smartest person who know in your pocket", "a simultaneous expert PhD, MD, JD", "Smarter than all humans combined". But there's a giant gulf between what they're selling and what it is, and that gulf is what makes LLMs a poor tool.
The provide abilities no other tool provides me with. They could be better, but they've still provided me with possibilities that I have never had before without hiring humans.
I'm sure they're good for you, I'm not suggesting otherwise. What I'm saying is, if you ask 100 engineers to describe the properties of the best tools they use, the set of adjectives and characteristics come up with will largely not be applicable to LLMs, and ChatGPT 5 doesn't change that.
It's also wildly unrealistic conjecture, in my opinion.
The first, and most important, measure to me of a good tool is whether it makes me more productive. LLMs does. Measurably so.
You will certainly find people who don't believe LLMs do for them, but that won't change the fact that for a lot of us it is an immensely good tool. And a tool doesn't need to fit everyones processes to be a good tool.
> It's also wildly unrealistic conjecture, in my opinion.
The only property of good tools you've mentioned that LLMs have is they do something useful.
But are they reliable? No, they inexplicably work sometimes and don't work other times. Do they have a clear purpose? No, their purpose is muddled and it's sold as being capable of literally everything. Are the ergonomics good? No, the interface is completely opaque, accessed only behind a natural language text box that comes with no instructions on how to use. But then it must be intuitive, right? No, common wisdom on how to use them sound more like astrological forecasts, or the kind of advice you give people trying to get something from a toddler -- "If you want good results, first you have to get into character!"... etc. etc.
> it makes me more productive.
Awesome! I'm sure they are doing exactly what you say and your experiences with them are amazing. But yours and my personal productivity isn't the question facing the industry at the moment or the topic of this discussion.
The question isn't whether you personally as an induvial find these things useful in your process. If that were the question I wouldn't be here complaining about them. I'm here because the powers that be are telling us that we must adopt these things into every aspect of our work lives as soon as possible, and if we don't, we deserve to be left behind. THAT is what I'm here to talk about.
> And a tool doesn't need to fit everyones processes to be a good tool.
I haven't argued this. Of course a tool doesn't have to fit everyone's process to be good, but we have some generally accepted principles of good tool design, and LLMs don't follow them. That doesn't preclude you from using them well in your process.
> I have never met a human who has a good grasp of what they know and don't know
But humanity has a good grasp over it, we even created science to solve this. So out of all existences we are aware of humanity is by far the best at this, nothing else even comes remotely close.
And humans are often wrong both when we say we don't know, and when we claim to know. There likely is a difference in degree of accuracy, but the point I was making was that despite your claim to "certainly know what [you] know", we don't in fact know what we know with anything remotely near precision.
We know some of what we know, but we can both be pressed into doing things we are certain we don't know how to do but where the knowledge is still there, and we will confidently proclaim to know things (such as the extent of our knowledge) that we don't.
I will agree that LLMs need to acquire a better idea of what they know, but there is no reason to assume that knowing the limits of your own knowledge with any serious precision matters, given how bad humans are at knowing this.
So much of human culture, politics, and civil life is centered around resolving conflicts that arise out of our lack of knowledge of our own limits, that this uncertainty is fairly central to what it means to be human.
You are judging this based on what the LLM outputs, not on its internals. When we peer into its internals, it seems that LLMs actually have a pretty good representation of what they do and don't know; this just isn't reflected in the output because the relevant information is lost in future context.
Yes, it's not that people know specifically what they don't know, it's that they develop the wisdom to know those boundaries and anticipate them and reduce their likelihood and impact.
For example, if I use the language of my expertise for a familiar project then the boundaries where the challenges might lie are known. If I start learning a new language for the project I won't know which areas might produce unknowns.
The LLM will happily give you code in a language it's not trained well on. With the same confidence as using any other language.
Sorry for copypasting from another comment but this is relevant
I can tell you what skills I possess. For example: programming, writing music. A LLM can not do this unless it's told what it knows. I could also tell you whether I studied thing X or attempted to do it and what success I had. So I'm pretty good at assessing my abilities. A LLM has no idea
This is interesting, because I wouldn't be too sure about that. Whether I am able to play chess well exclusively depends on the strengths of my opponents, because there is no absolute baseline. If society somehow decided that what you were writing wasn't "music" anymore, you would be the only person left stating you had that skill.
I believe that most claims about one's own skills do come from outward judgement and interpretation of one's actions, not introspection. The only thing humans have is, to radically appropriate the jargon, a way longer context window (spanning over an entire lifetime!), together with many ways to compress it.
> OpenAI researcher Noam Brown on hallucination with the new IMO reasoning model:
> Mathematicians used to comb through model solutions because earlier systems would quietly flip an inequality or tuck in a wrong step, creating hallucinated answers.
> Brown says the updated IMO reasoning model now tends to say “I’m not sure” whenever it lacks a valid proof, which sharply cuts down on those hidden errors.
> TLDR, the model shows a clear shift away from hallucinations and toward reliable, self‑aware reasoning.
Yeah. The empty “it’s just a statistical model” critique (or the dressed-up “stochastic parrots” version of it) is almost a sign at this point that the person using it formed their opinions about AI back when ChatGPT first came out, and hasn’t really bothered to engage with it much since then.
If in 2022 I’d tried to convince AI skeptics that in three years we might have tools on the level of Claude Code, I’m sure I’d have heard everyone say it would be impossible because “it’s just a statistical model.” But it turned out that there was a lot more potential in the architecture for encoding structured knowledge, complex reasoning, etc., despite that architecture being probabilistic. (Don’t bet against the Bitter Lesson.)
LLMs have a lot of problems, hallucination still being one of them. I’d be the first to advocate for a skeptical hype-free approach to deploying them in software engineering. But at this point we need careful informed engagement with where the models are at now rather than cherry-picked examples and rants.
Unless what you work on is very simple and mostly mindless, using tools like Claude Code is the exact opposite of how to make the current SotA LLMs useful for coding. The models can help and boost your productivity, but it doesn't happen by letting them do more stuff autonomously. Quite the contrary.
And when what you usually work on actually is very simple and mostly mindless, you'd probably benefit more from doing it yourself, so you can progress beyond the junior stuff one day.
Where it really has value is if what you work on is like 33% extremely difficult and 66% boilerplate/tedious. Being able to offload the tedious bits can make more senior engineers 2-3x more productive without the coordination effort of "find a junior engineer to do this, schedule their time, assign the work, follow up on it".
(The problem of course is you still need these junior engineers to exist in order to have the next generatino of senior engineers, so we must now also think about what our junior folks should be doing to be valuable AND learn).
> If in 2022 I’d tried to convince AI skeptics that in three years we might have tools on the level of Claude Code, I’m sure I’d have heard everyone say it would be impossible because “it’s just a statistical model.”
We already had these coding models in 2022, they were already pattern matching engines with variables back then, all your imagination needed to do to go from that to Claude code today is to give it more code examples and make it bigger.
They still can't replace even a junior engineers ability to navigate tasks over even short periods of time, just like back then they need constant handholding to get anything done. So I don't see what changed except the model being larger with more examples and therefore you can get larger chunks of coherent code out of them.
People repeating the "stochastic parrot" meme in all kinds of variations if anything appear to be more like stochastic parrots than the typical LLM is.
Actually internally we do know what’s going on these days. Anthropic put out a white paper detailing how Claude can’t math but many math examples are out there so Claude can fake it.
I wish you’d stop magic LLMs some kind of magic thing they aren’t.
I work in the theory of deep learning, so I can say with some authority that while we know a really good number of things, and are able to probe the internals much better than most of the public realises, when it comes to philosophical questions that compare their nature with humans, we have a long, long way to go. The biggest problem is that we're still trying to work out what it is we even want to know that will tell us whether we have achieved AGI or not.
Linear probes and autoencoders have been useful, but we're quickly reaching the limits of those techniques. And don't even get me started on approaches to theory that operate by cherry-picked examples. Anthropic's contributions have been beneficial to the field, but are far from conclusive.
Humans build theories of how things work. llms dont. Theories are deterministic symbolic representation of the chaotic worlds of meaning . Take the turing machine for example as a theory of computation in general, euclidean geometry as a theory for space, and newtonian mechanics as a theory for motion.
A theory gives 100% correct predictions. Although the theory itself may not model the world accurately. Such feedback between the theory, and its application in the world causes iterations to the theory. From newtonian mechanics to relativity etc.
Long story short, the LLM is a long way away from any of this. And to be fair to LLMs, the average human is not creating theories, it takes some genius to create them (newton, turing, etc).
Understanding something == knowing the theory of it.
> Humans build theories of how things work. llms dont. Theories are deterministic symbolic representation of the chaotic worlds of meaning
What made you believe this is true? Like it or not, yes, they do (at least to the best extent of our definitions of what you've said). There is a big body of literature exploring this question, and the general consensus is that all performant deep learning models adopt an internal representation that can be extracted as a symbolic representation.
I am yet to see a theory coming of the LLM that is sufficiently interesting. My comment was answering your question of what does it mean to "understanding something". My answer to that is: understanding something is knowing the theory of it.
Now, that begs the question of what is a theory. And to answer that, a theory comprises of building block symbols and a set of rules to combine them. for example, building blocks for space (and geometry) could be points, lines, etc. The key point in all of this is symbolism as abstractions to represent things in some world.
The "sufficiently interesting" part is the most important qualifier here. My response was talking about theories and representations that we already know, either instinctively from near-birth, or from learned experience. We have not seen anything unique from LLMs because they do not appear to have an internal understanding (in the same sense that I was talking about) that is as broad as an adult human. But that doesn't mean it lacks any understanding.
> The key point in all of this is symbolism as abstractions to represent things in some world.
The difficulty is understanding how to extract this information from the model, since the output of the LLM is actually a very poor representation of its internal state.
> This concludes all the testing for GPT5 I have to do. If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it’s a useless tool.
Yeah, except it isn't. You can get enormous value out of LLMs if you get over this weird science fiction requirement that they never make mistakes.
And yeah, their confidence is frustrating. Treat them like an over-confident twenty-something intern who doesn't like to admit when they get stuff wrong.
You have to put the effort in to learn how to use them with a skeptical eye. I've been getting value as a developer from LLMs since the GPT-3 era, and those models sucked.
> The only way ChatGPT will stop spreading that nonsense is if there is a significant mass of humans talking online about the lack of ZSTD support.
We actually have a robust solution for this exact problem now: run the prompt through a coding agent of some sort (Claude Code, Codex CLI, Cursor etc) that has access to the Swift compiler.
That way it can write code with the hallucinated COMPRESSION_ZSTD thing in it, observe that it doesn't compile and iterate further to figure out what does work.
Or the simpler version of the above: LLM writes code. You try and compile it. You get an error message and you paste that back into the LLM and let it have another go. That's been the main way I've worked with LLMs for almost three years now.
If that same intern, when asked something, responded that they checked, gave you a link to a document they claim have the proof / answer but does in fact no and continued to do that they wouldn’t be an intern very long. But somehow this is acceptable behaviour for an AI?
I use AI for sure, but only on things that I can easily verify is correct (run a test or some code ), because I have had the AI give me functions in an API with links to online documentation for those functions, the document exists, the function is not in it, when called out instead of doing a basic tool call the AI will double down that it is correct and you the human are wrong. That would get an intern fired but here you are standing on the interns side.
> Don’t fall into the trap of anthropomorphizing LLMs and assuming that failures which would discredit a human should discredit the machine in the same way.
> And yeah, their confidence is frustrating. Treat them like an over-confident twenty-something intern who doesn't like to admit when they get stuff wrong.
I was explicitly calling out this comment, that intern would get fired if when explicitly called out they not only don’t want to admit they are wrong but vehemently disagree.
The interaction was “Implement X”, it gave an implementation, I responded “function y does not exist use a different method”, it instead of following that instruction gave me a link to the documentation for the library that it claim’s contains that function and told me I am wrong.
I said the documentation it linked does not contain that function and to do something different and yet it still refused to follow instructions and pushed back.
At that point I “fired” it and wrote the code myself.
> Or the simpler version of the above: LLM writes code. You try and compile it. You get an error message and you paste that back into the LLM and let it have another go. That's been the main way I've worked with LLMs for almost three years now.
I’m going to comment here about this but it’s a follow on to the other comment, this is exactly the workflow I was following. I had given it the compiler error and it blamed an environment issue, I confirmed the environment is as it claims it should be, it linked to documentation that doesn’t state what it claims is stated.
In a coding agent this would have been an endless feedback loop that eats millions of tokens.
This is the reason why I do not use coding agents, I can catch hallucinations and stop the feedback loop from ever happening in the first place without needing to watch an AI agent try to convince itself that it is correct and the compiler must be wrong.
You're responding to simonw, the guy who could be considered the single leading voice in practical applications of llms, with an anecdote about how one time the bot gave you a compiler error, and llm coding is therefore useless.
I would agree with you if it was just one time, this is multiple times.
Also look up appeal to authority, everyone can be better, I respect Simon don’t get me wrong, but I don’t read names before I comment and you would probably do well to do the same, let what people say stand on their own.
I read what you wrote and let it stand on its own. Your comment mentioned a single time the bot had failed - there was no mention of it occurring multiple times. Maybe you should flesh out your anecdotes properly.
If you are treating anonymous posts on forums with the same authority as experts on those topics then it makes sense why you are struggling to use new tooling.
> I use AI for sure, but only on things that I can easily verify is correct
Where did I say I was struggling, I stated a preference, I even stated lived experience with using a coding agent and a chat based LLM interface and have a preference for the chat based version.
I don’t struggle to use modern tooling, I pointed out that the analogy is terrible and how my lived experience proves that, I do use LLMs in anger daily for coding tasks, I just don’t treat them like interns.
Many time the points I used indicate that what I am doing is not in the training set, when you work on hard problems you run into edge cases a little more reliably than when you don’t.
Stop defending a man you don’t know on the internet, his words defend him well enough and I can hope he just stops using that analogy because it is fundamentally flawed, which is the point of my comments. The rest of his comment makes perfect sense and is the way I use LLMs and have to work around their idiosyncrasies anyway.
>...it’s a useless tool. I don’t like collaborating with chronic liars who aren’t able to openly point out knowledge gaps...
I think a more correct take here might be "it's a tool that I don't trust enough to use without checking," or at the very least, "it's a useless tool for my purposes." I understand your point, but I got a little caught up on the above line because it's very far out of alignment with my own experience using it to save enormous amounts of time.
and as others have pointed out, this issue of “how much should I check” is really just a subset of an old general problem in trust and knowledge (“epistemology” or what have you) that people have recognized since at least the scientific revolution. The Royal Society’s motto on its founding in the 1660s was “take no man’s word for it.”
Coding agents have now got pretty good at checking themselves against reality, at least for things where they can run unit tests or a compiler to surface errors. That would catch the error in TFA. Of course there is still more checking to do down the line, in code reviews etc, but that goes for humans too. (This is not to say that humans and LLMs should be treated the same here, but nor do I treat an intern’s code and a staff engineer’s code the same.) It’s a complex issue that we can’t really collapse into “LLMs are useless because they get things wrong sometimes.”
> Coding agents have now got pretty good at checking themselves against reality, at least for things where they can run unit tests or a compiler to surface errors.
YMMV. I've seen Claude go completely batshit insane saying that tests all passed. Then I run them and I see 50+ failures. I copy the output tell him to fix it and he goes on his sycophantic apologia before spinning his wheels doing nothing and saying all tests are back to green.
We double check human work too in all kinds of contexts.
A whole lot of my schooling involved listening to teachers repeating over and over to us how we should check our work, because we can't even trust ourselves.
(heck, I had to double-check and fix typos in this comment)
It’s very weird to me when someone says “this tool does not have this common property good tools have” and someone replies “humans also do not have those properties!” As if that is responsive to a complaint about a tool lacking a common property of tools.
We use tools because they work in ways that humans do not. Through centuries of building and using tools, we as a society have learned what makes a tool good versus bad.
Good tools are reliable. They have a clear purpose and ergonomic user interface. They are straightforward to use and transparent in how they operate.
LLMs are none of these things. It doesn’t matter that humans also are none of these things, if we are trying to use LLMs as tools.
The closest human invention resembling an LLM is the idea of a bureaucracy — LLMs are not good tools, they are not good humans, they are mindless automatons that stand in the way and lead you astray.
At best, LLMs are poor tools and also poor human replacements, which is why it’s so frustrating to me we are so intent on replacing good tools and humans with LLMs.
The reason for making that observation is that we don't have any other comparable tools and it is more reasonable to benchmark LLMs against what humans are capable of, because whether or not they are good approximations, we are trying to model human abilities.
One day maybe we exceed human abilities, but it's unreasonable to expect early attempts - and they are still early attempts - to do things we don't know how to beat other than by putting all kinds of complex process on top of very flawed human thinking.
We do have comparable tools to LLMs. There are plenty of human-composed tools that can do what LLMs do, like Mechanical Turk for instance. The human-composed tool that most closely resembles a LLM is the "bureaucracy".
An LLM is like a little committee you send a request to, and based on capricious, opaque rules that can change at any time, the committee returns a response that may or may not service your request. You can't know ahead of time if it will, and depending on the time of day, the political environment, or the amount of work before the committee, the delay of servicing and the quality of service may or may not degrade. Quality may range from a quick accurate response, to flat refusal to service without explanation, or outright lies to your face. There's no way to guarantee a good result, and there's no recourse or explanation for why things go wrong or changed.
LLMs feel like a customer service agent turned into a computer program, which is probably why it was people's first thought to use LMMs to automate customer service agents. They are a perfect fit there, but I don't want them to be my primary interface to do work. I have enough bureaucracies to deal with as it is.
> We do have comparable tools to LLMs. There are plenty of human-composed tools that can do what LLMs do, like Mechanical Turk for instance.
If you are going to treat humans as tools, then sure. In which case measuring LLMs against human ability is exactly the right thing, given that with Mechanical Turk the tasks are carried out by humans - sometimes with the help of LLMs...
It's utterly bizarre to argue over my comparing LLMs to humans when the tools you argue are comparable are humans.
> when the tools you argue are comparable are humans.
No, they are abstractions over humans. A group of people is not a person, they behave differently than people even though they are composed of them. Abstractions are hard to compare but still much easier than people.
It's not a meaningless difference, it's a crucial one. The contradiction only exists if you collapse all the differences between people and abstractions of people -- crucially that the former are people and the latter are abstractions -- and claim they're the same. Which they are not.
Anyway, we've gotten far from the point, which is that LLMs are not people and you can't treat them as such.
> Checking is usually faster than writing from scratch
Famous last words. Checking trivial code for trivial bugs, yes. In science, you can have very subtle bugs that bias your results in ways that aren't obvious for a while until suddenly you find yourself retracting papers.
I've used LLMs to write tedious code (that should probably have been easier if the right API had been thought through), but when it comes to the important stuff, I'll probably always write an obviously correct version first and then let the LLM try to make a faster/more capable version, that I can check against the correct version.
Humans are notoriously bad at checking for meticulous detail—hence why copyediting was a specialized role. We’ve seen what’s happened since the newspapers eliminated it for cost savings.
I only used an LLM for the first time recently, to rewrite a YouTube transcript into a recipe. It was excellent at the overall restructuring, but it made a crucial and subtle mistake. The recipe called for dividing 150g of sugar, adding 30g of cornstarch to one half, and blanching eggs in that mixture. ChatGPT rewrote it so that you blanched the eggs in the other half, without the cornstarch. This left me with a boiling custard that wasn’t setting up.
I did confirm that the YouTube transcript explicitly said to use the sugar and cornstarch mixture. But I didn’t do a side by side comparison because the whole reason for doing the rewrite is that transcripts are difficult to read!
> I'll probably always write an obviously correct version first
I’m not usually so confident in my own infallibility, so I prefer to think of it as “I might get this wrong, the LLM might get this wrong, our failure modes are probably not very correlated, so the best thing is for us both to do it and compare.”
Agree it is always better for the human engineer to try writing the critical code first, since they are susceptible to being biased by seeing the LLM’s attempt. Whereas you can easily hide your solution from the LLM.
> I’m not usually so confident in my own infallibility
I'll concede that point, but let me put it differently: when I write the "obviously correct" version, what I mean is that I can explain what every line of code is supposed to be doing. I can say what the expected result would be, at each step and test it.
When the LLM writes an inscrutable version for me, I might be able to test the final result, but I'll never know if there are corner cases (or not even that) in between where it will go spectacularly wrong. If I have to untangle what it's written to get that kind of confidence, then it's going to take me much longer. Reading code, especially somebody else's, nevermind an alien's code, is just so much harder for me.
It seems that on some level, you have to in order to not just constantly reflecting upon your thoughts and researching facts. Whether you trust a given source should surely depend upon its reputation regarding the validity of its claims.
I agree and by reputation you mean accuracy. We implicitly know not to judge anything as 100% true and implicitly apply skepticism towards sources - the skepticism is decided by our past experience with the sources.
Think of LLMs as the less accurate version of scientific journals.
Accuracy certainly does play a role, but this in itself is not sufficient for preventing an infinite regress – how does one determine the accuracy of a source if not by evaluating claims about the source, that themselves have sources that need to be checked for accuracy? Empirical inquiries are optimal but often very unpractical. Reputation is accuracy as imperfectly valued by society or specific social groups collectively.
"Based on my research, zstd compression is not natively supported by iOS or Apple's frameworks, which means you cannot use zstd compression without adding some form of external code to your project"
Most likely the key here is web search. Later I will try the post’s example with gpt-5 with search. I would be surprised if it didn’t say the same thing.
From a product point of view, it seems clear that just as they have work to get the model to dynamically decide to use reasoning when it would help, they have to do the same with web search.
gpt-5 with medium reasoning, no search: "Short answer: use the system zstd C library that ships with iOS; Apple’s Compression framework still doesn’t expose zstd. Link libzstd and call its streaming API from Swift." (and then like 50 lines of code plus some explanation)
gpt-5 with medium reasoning + search: "Short answer: you can’t. Apple’s built‑in Compression framework on iOS exposes LZFSE, LZ4, ZLIB/DEFLATE, and LZMA—not Zstandard—so there is no system API to do zstd without bundling an external library. (https://www.hackingwithswift.com/example-code/system/how-to-...)
If you can switch to a built‑in algorithm, here’s a minimal streaming compressor using Compression (LZFSE shown):" (etc etc)
* LLMs, when presented with a question to which they don't know the answer, tend to make up a wrong answer rather than say they don't know.
* Therefore, LLMs are useless.
Yes, LLMs do that, and it is a problem.
No, that does not make them useless. It's usually much easier to check their answer than it would have been to search for an answer yourself. If they are right, you saved time. If they are wrong, at worst you wasted a couple minutes. But they are right often enough that it can be worthwhile, especially if you have a feel for what questions they are likely to know the answer to and what questions they aren't.
An implication from the 1985 paper of Peter Naur on programming as theory building is that the current LLM coding tool would be very effective in generating technical debt even when it works ... use at your own risk.
I'm currently working as a hands-off VP, so I don't use LLMs for coding at work, only for emails and document editing. I do use it for my hobby weekend coding stuff, which usually involves writing short 100-1000 LOC Python toy programs for my own education/entertainment. My way-of-working is quite primitive, I have zero integrations, nothing agentic, I just copy/paste with ChatGPT.
For this use-case it's been very useful, it can usually generate close-to-complete solutions, as long as it's one of the major programming languages, and it's a reasonably standard problem. So in general I'm always surprised when people say that LLMs are completely useless for coding --- this is just not true, and I feel sorry for people who shut themselves off from a useful tool.
But even at this small scale, even the best (o3) models sometimes totally fail. Recently I started a series of posts on distributed algorithms [1], and when I was working on the post/code for the Byzantine Generals / Consensus algorithm, o3 --- to my honest surprise --- just totally failed. I tried about 10 different times (both from scratch and describing the incorrect behaviour of its code), also showing it the original Lamport paper, and it just couldn't get it right... even though the toy implementation is just ~100 LOC, the actual algorithm portion is maybe 25 LOC. My hypothesis is that there are very few implementations online, and additionally I find the descriptions of the algorithm a bit vague (interleaves message cascade and decision logic).
ChatGPT 5> Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?
(It auto-routed to its thinking model, took 25s)
> Short answer: you can’t. iOS doesn’t ship a public zstd encoder/decoder. The system Compression and AppleArchive frameworks only expose lzfse, lz4, zlib/deflate, and lzma. So with only Apple SDKs, there’s no way to produce zstd-compressed bytes.
>
> Two practical paths:
> 1. If zstd isn’t mandatory, use a built-in codec (lzfse or lz4 are fastest) via Compression’s streaming API. Example (pure Swift, no deps) that compresses an arbitrary byte stream:
>
> <code example>
> 2. If zstd is mandatory, you must embed zstd yourself. That can mean:
> - Vendoring the official C sources (libzstd) directly into your project and calling them via a tiny Swift wrapper/bridging header
> - Statically linking a prebuilt libzstd.a you compile for iOS
I think the useful takeaway here is that Top 1 operation is generally not a good idea, especially not for making judgements. This doesn't address the main points of the blog though.
what's funny is that newer models will now be trained on the exact question, "Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?" and similar questions to it, because of this post
maybe the key to training future llm's is to write angry blog posts about the things they aren't good at and get them to the front page of hn?
I wonder if one reasons new versions of GPT appear to get better - say at coding tasks is just because they have new knowledge.
When ChatGPT4 comes out, new versions of API’s will have less blog post / examples / documentation in their training data. So ChatGPT 5 comes out and seems to solve all the problems that ChatGPT4 had, but then of course fail on newer libraries. Rinse and repeat
I’ve heard of this idea of training on synthetic data, I wonder what is that data and does this increase or decrease hallucinations? Is the goal of training on synthetic data to better wear certain paths, or to increase the amount of knowledge / types of data.
Because the second seems vaguely impossible to do.
I see that GitHub Copilot actually runs code, writes simple exploratory programs, iteratively tests its hypothesis. it is astoundingly effective and fast.
same here. nothing stops this AI to actually trying to implement whatever this AI suggested, compile it, and see if this is actually works.
grounding in reality at inference time, so to speak.
LLMs can be a useful tool, maybe. But don’t anthropomorphize them.
(but, earlier)
If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it’s a useless tool. I don’t like collaborating with chronic liars.
Hallucinations are not a bug or an exception, but a feature. Everything outputted by LLMs is 100% made-up, with a heavy bias towards what has been fed to it at first (human written content).
The fundamental reason why it cannot be fixed is because the model does not know anything about the reality, there is simply no such concept here.
To make a "probability cutoff" you first need a probability about what the reality/facts/truth is, and we have no such reliable and absolute data (and probably never will).
>To make a "probability cutoff" you first need a probability about what the reality/facts/truth is, and we have no such reliable and absolute data (and probably never will).
Can a human give a probability estimate to their predictions?
Humans can explain how they arrived at the conclusion, an LLM fundamentally cannot do that since they don't remember why they picked the tokens they did, they just make up an explanation based on explanations it has seen before.
You use a lot of anthropomorphisms: doesn't "know" anything (does your hard drive know things? Is it relevant?), "making things up" is even more linked to continuous intent. Unless you believe the LLMs are sentient this is a strange choice of words.
have you seen Iris flowers dataset? it is fairly simple to find cutoffs to classify flowers.
or are you claiming in general that there is no objective truth in reality in philosophical sense? well, you can go on that more philosophical side of the road, or you can get more pragmatic. things just work, regardless how we talk about them.
I don't mean it in a philosophical sense, more in a rigorous scientific one.
Yes, we do have reliable datasets as in your example, but those are for specific topics and are not based on natural language. What I would call "classical" machine learning is already a useful technology where it's applied.
Jumping from separate datasets focused on specific topics to a single dataset describing "everything" at once is not something we are even close to doing, if it's even possible. Hence the claim of having a single AI able to answer anything is unreasonable.
The second issue is that even if we had such a hypothetical dataset, ultimately if you want a formal response from it, you need a formal question and a formal language (probably something between maths and programming?) in all the steps of the workflow.
LLMs are only statistical models about natural languages, so it's the antithesis of this very idea. To achieve that would have to be a completely different technology that has yet to even be theoretized.
I call this genre of article "I'm too lazy to think about how to deal with the fact that LLMs are sometimes wrong, especially when they aren't using thinking and are given short, leading prompts."
Ok great! For those of us who aren't too lazy for it, LLMs are providing a lot of value right now.
I love posts like these because it just reinforces that I made the right decision in spending as much time as I do in getting really really really good at using llms.
if you treat it like stochastic software it is, how is 95% chance of one-shotting a problem not useful? I feel like the biggest problem most people have with LLMs is the "lack of accountability", and then the same people say "don’t anthropomorphize them"
I find it hard to empathize with these sorts of articles. I think in, in the spate of GPT-5 related content recently, I've been seeing a lot of articles that boil down to "I tried GPT-5 on a single hard question, and it gave an answer. This proves that all LLMs are useless." And I don't think I'm distorting the author's viewpoint. He goes on to say "This concludes all the testing for GPT5 I have to do" after conducting a single test.
This seems like particularly harsh criteria; what would happen if I applied this to other tools?
- I used Typescript, but it missed a bug that crashed prod, so it is "absolute horseshit"
- I used Rust, but one of my developers added an unsafe block, so it's trash.
My issue with LLMs isnt really the tech itself. They're occassionally useful, although HOW useful seems to depend on a persons skillset.
My main beef with the AI hype is that it's allowing a lot of idiots to significantly devalue our profession in a really noxious and irritating way to people that generally don't understand what we do but would like to pay us less or pay less of us. I'm annoyed at other software developers that don't seem to see how harmful this will be for us when the insane investment bubble bursts and AI becomes a lot more expensive to use. We will probably have lost a generation of junior developers who have become dependent on a suddenly expensive tool. And execs will just think the seniors need to pick up the slack. And expectations on AI will be a lot higher when the subscription is more like 200 or 2000 a month.
And that's just for coding! I'd be furious if I was an artist and generative AI was being trained on my portfolio to plagiarize my work. (Badly)
What I never see justified is why any of this is good for society. At best it lets billionionaires save some money by getting rid of jobs, or vibe coders pretend they can build a product until they hit a wall where real understanding is neccessary. If you follow the trail of who is supposed to benefit from these things its not many of us. If AI were to disappear today I don't think my life would be any worse.
How is this relevant? Modern LLMs don't rearrange words at all in a meaningful sense of the word, and are certainly better than just using random chance to sample tokens.
Indeed it is exactly that process. They cutup words. Then categorize them based on some metric of nearness (not random). Then link them up. Obviously this process is much more sophisticated than what I have described here.
The top comments are focusing on what "know" means in the context of an LLM but I think that while it's an interesting discussion it misses the point. Even if we agree that an LLM can "know" things, the tech has been heavily advertised as nothing less that transformative. If it can't help itself from inventing non existent answers or count the number of "r"s in the word blueberry (a test I tried myself the other day on my phone's chat-gpt app, it said "three") it's not transformative.
I'm almost positive that he didn't use thinking mode, because I've asked GPT-5 thinking his question verbatim in 10 parallel threads and it said "you can't" and thoroughly explained why every time. Below is the first response I got back.
---
Short answer: you can’t. iOS doesn’t ship a Zstandard encoder/decoder. Apple’s built-in `Compression` framework only exposes LZFSE, LZ4(/RAW), zlib/DEFLATE, and LZMA—no ZSTD knob to turn. ([Microsoft Learn][1], [Apple Developer][2])
If you’re okay with a built-in alternative that’s “zstd-ish” in speed/ratio, use LZFSE via `Compression`’s streaming API. Here’s a minimal streaming compressor you can drop in; it takes chunks of `Data` and yields compressed chunks (no third-party code):
```swift
import Compression
final class StreamCompressor {
private var stream = compression_stream()
private let algo: compression_algorithm
private var isInitialized = false
init(algorithm: compression_algorithm = COMPRESSION_LZFSE) {
self.algo = algorithm
}
func append(_ input: Data, isFinal: Bool, outChunkSize: Int = 64 * 1024) -> [Data] {
if !isInitialized {
let status = compression_stream_init(&stream, COMPRESSION_STREAM_ENCODE, algo)
precondition(status != COMPRESSION_STATUS_ERROR, "compression_stream_init failed")
isInitialized = true
}
var outputs: [Data] = []
input.withUnsafeBytes { (srcBuf: UnsafeRawBufferPointer) in
var flags: Int32 = isFinal ? Int32(COMPRESSION_STREAM_FINALIZE.rawValue) : 0
stream.src_ptr = srcBuf.bindMemory(to: UInt8.self).baseAddress
stream.src_size = srcBuf.count
var out = Data(count: outChunkSize)
out.withUnsafeMutableBytes { dstBuf in
while true {
stream.dst_ptr = dstBuf.bindMemory(to: UInt8.self).baseAddress
stream.dst_size = dstBuf.count
let status = compression_stream_process(&stream, flags)
let produced = dstBuf.count - stream.dst_size
if produced > 0 {
outputs.append(Data(bytes: dstBuf.baseAddress!, count: produced))
}
if status == COMPRESSION_STATUS_OK && stream.src_size == 0 && stream.dst_size > 0 {
break // need more input to continue
} else if status == COMPRESSION_STATUS_OK && stream.dst_size == 0 {
// output buffer filled; loop to drain more
continue
} else if status == COMPRESSION_STATUS_END {
break
} else if status == COMPRESSION_STATUS_ERROR {
preconditionFailure("compression_stream_process failed")
} else {
// Shouldn’t happen
break
}
}
}
}
return outputs
}
deinit {
if isInitialized {
compression_stream_destroy(&stream)
}
}
}
```
Usage:
```swift
let compressor = StreamCompressor(algorithm: COMPRESSION_LZFSE)
var compressed: [Data] = []
for (i, chunk) in chunks.enumerated() {
let final = i == chunks.count - 1
compressed.append(contentsOf: compressor.append(chunk, isFinal: final))
}
// `compressed` now contains the zstd-adjacent (LZFSE) stream
```
If you truly need the ZSTD format (interop requirement), you’ll have to ship the reference C library (`libzstd`) in your app (static link it and call through a tiny Swift wrapper). That’s still “third-party,” even if you vendor the sources instead of using SPM/CocoaPods. There is no system-provided ZSTD on iOS today. ([GitHub][3])
tl;dr: iOS doesn’t expose ZSTD. Use `Compression` with LZFSE for built-in streaming, or vendor `libzstd` and bridge to it if you must emit/ingest ZSTD. ([Microsoft Learn][1], [Apple Developer][2], [GitHub][3])
I am sympathetic to the reasoning as to why LLMs should not be used to help some programmers right now. But I get a little frustrated seeing many of these kinds of posts that talk about fundamental limitations of LLMs vs humans on the grounds that it cannot "logically reason" like a human does. These are limitations in the current approach to training and objectives; internally, we have no clue what is going on.
> it’s “just a statistical model” that generates “language” based on a chain of “what is most likely to follow the previous phrase”
Humans are statistical models too in an appropriate sense. The question is whether we try to execute phrase by phrase or not, or whether it even matters what humans do in the long term.
> The only way ChatGPT will stop spreading that nonsense is if there is a significant mass of humans talking online about the lack of ZSTD support.
Or you can change the implicit bias in the model by being more clever with your training procedure. This is basic stats here, not everything is about data.
> They don’t know anything, they don’t think, they don’t learn, they don’t deduct. They generate real-looking text based on what is most likely based on the information it has been trained on.
This may be comforting to think, but it's just wrong. It would make my job so much easier if it were true. If you take the time to define "know", "think", and "deduct", you will find it difficult to argue current LLMs do not do these things. "Learn" is the exception here, and is a bit more complex, not only because of memory and bandwidth issues, but also because "understand" is difficult to define.
>Humans are statistical models too in an appropriate sense.
No, we aren't and I'm getting tired of this question begging and completely wrong statement. Human beings are capable of what Kant in fancy words called "transcendental apperception", we're already bringing our faculties to bear on experience without which the world would make no sense to us.
What that means in practical terms for programming problems of this kind is that, we can say "I don't know", which the LLM can't, because there's no "I", in the LLM, no unified subject that can distinguish what it knows and what it doesn't, what's within its domain of knowledge or outside.
>If you take the time to define "know", "think", and "deduct", you will find it difficult to argue current LLMs do not do these things
No, only if you don't spend the time to think about what knowledge is you'd make such a statement. What enables knowledge, which is not raw data but synthesized, structured cognition, is the faculties of the mind a priori categories we bring to bear on data.
That's why these systems are about as useless as a monkey with a typewriter when you try to have them work on manual memory management in C, because that's less of a task in auto completion and requires you to have in your mind a working model of the machine.
This is interesting philosophy, and others have better critiques here in that regard. I'm a mathematician, so I can only work in what I can define symbolically. Humans most certainly ARE statistical models by that definition: without invoking the precise terminology, we take input, yield output, and plausibly involve uncertain elements. One can argue as to whether this is the correct language or not, but I prefer to think this way, as the arrogance of human thinking has otherwise failed us in making good predictions about AI.
If you can come up with a symbolic description of a deficiency in how LLMs approach problems, that's fantastic, because we can use that to alter how these models are trained, and how we approach problems too!
> What that means in practical terms for programming problems of this kind is that, we can say "I don't know", which the LLM can't, because there's no "I", in the LLM, no unified subject that can distinguish what it knows and what it doesn't, what's within its domain of knowledge or outside.
We seriously don't know whether there is an "I" that is comprehended or not. I've seen arguments either way. But otherwise, this seems to refer to poor internal calibration of uncertainty, correct? This is an important problem! (It's also a problem with humans too, but I digress). LLMs aren't nearly as bad as this as you might think, and there are a lot of things you can do (that the big tech companies do not do) that can better tune it's own self-confidence (as reflected in logits). I'm not aware of anything that uses this information as part of the context, so that might be a great idea. But on the other hand, maybe this actually isn't as important as we think it is.
> I'm a mathematician, so I can only work in what I can define symbolically.
This is a limitation of you, not an argument. Of course everything will look the same to you if that's the only way you can represent them.
We have more disciplines than mathematics because mathematically is not the only valid way to explore things.
Yes, of course. I'm not trying to establish further argument or to criticise. But it isn't fruitful to have a conversation where both parties have different definitions in mind. Since my original comment referred to the mathematical definition, it is important for me to clarify that.
The rest of my comment is to try to find some middle ground, of which there is plenty in philosophy. If the raw theory of mind arguments for humans were already applicable for recognizing the limitations of LLM designs, we would be much further along than we currently are.
The position of Kant does not align with the direction modern neuroscience is heading towards. Current evidence seems to prefer decentralized theories of consciousness like Dennett's multiple drafts model[1], suggesting there is no central point where everything comes together to form conscious experience, but instead that it itself is constituted by collaborative processes that have multiple realizations.
[1]: https://en.wikipedia.org/wiki/Multiple_drafts_model
>Current evidence seems to prefer decentralized theories of consciousness like Dennett
There is no such thing as consciousness in Dennett's theory, his position is that it doesn't exist, he is a Eliminativist. This is of course an absurd position with no evidence for it as people like Chalmers have pointed out (including in that Wikipedia article), and it might be the most comical and ideological position in the last 200 years.
From the link:
> However, Dennett is not denying the existence of the mind or of consciousness, only what he considers a naive view of them
It doesn't seem like he's Eliminativist. It also seems like the criticisms rely on harping on about qualia, which is one of the sillier schools of sophistry. I'd need to see actual criticisms before believing that Dennett is pushing for something comical.
It is kind of interesting you are arguing with a human about the ability to distinguish what is known and not known.
Where your claim is humans know and LLMs don’t, but the op is wrong and does not know what he does not know.
Kant was a dualist, of course he didnt think humans were statistical models. It just turns out he was (probably) wrong.
Does a chess engine know, think, or deduct?
> because there's no "I", in the LLM, no unified subject
is there in humans?
> Human beings are capable of
are they?
> without which the world would make no sense to us
does it?
While the normal distribution meme is notoriously overused, I think it fits the scenario here.
LLMs know so much (when you just use ChatGPT for the first time like it's an Oracle machine) -> LLMs don't know anything (when you understand how machine learning works) -> LLMs know so much (when you actually think about what 'know' means)
> it cannot "logically reason" like a human does
Reason? Maybe. But there's one limitation that we currently have no idea how to overcome; LLMs don't know how much they know. If they tell you they don't something it may be a lie. If they tell you they do, this may be a lie too. I, a human, certainly know what I know and what I don't and can recall from where I know the information
I have never met a human who has a good grasp of what they know and don't know. They may have a better graps of it than an LLM, but humans are awfully bad at understanding the limits of our own knowledge, and will argue very strongly in favour of knowing more than we demonstrably do in all kinds of contexts.
> I have never met a human who has a good grasp of what they know and don't know
yep. There are 2 processes underlying our behaviors.
1) a meme-copier which takes in information from various sources and regurgitates it. A rote-memory machine of sorts. Here, memory is populated with info without deconstructing it into a theory. This has been termed "know-that". Here, explanations are shallow, and repeated questioning of why will fail with: I was told so.
2) a builder which attempts to construct a theory of something. Here, a mechanism is built and understood, and information is deconstructed in the context of the mechanism. This has been termed "know-how". Here, explanations are deeper, and repeated questioning of why will end up with: these are the "givens" in our theory.
Problem is that we operate in the "know-that" territory most of the time, and have not taken the effort to build theories for ourselves in the "know-how" territory.
LLMs are not humans, they are ostensibly tools. Tools are supposed to come with a list of things they can do. LLMs don’t and are therefore bad at being tools, so we anthropomorphize them. But they are also not good at being people, so LLMs are left in this weird in-between state where half the people say they’re useful and half the people say they cause more problems than they solve.
They are fantastic tools to me. The provide abilities that no other tools we have can provide.
That they also have flaws does not remove those benefits.
There's no question they have capabilities that no other tool has. But a good tool goes beyond just doing something, there are some generally agreed upon principles of tool design that make something good versus just useful.
For example, I think a hammer is a good tool because every time I swing it at a nail, it provides the force necessary to drive it into the wood. It's reliable. Sure, sometimes a hammer breaks, but my baseline expectation in using one is that every time I swing the hammer it will behave the same way.
Something more complicated, like a Rust compiler is also a good tool in the same way. It's vastly more intricate than the hammer, yet it still has the good tool property of being reliable; every time I press compile, if the program is wrong, then the compiler tells me that. If it's right, then the compiler passes every time. It doesn't lie, it doesn't guess, it doesn't rate limit, it doesn't charge a subscription, it doesn't silently update causing code to fail, it informs when changes are breaking and what they are, it allows me to pick my version and doesn't silently deprecate me without recourse, etc.
There are of course ecosystems out there where building a project is more like a delicate dance, or a courtship ritual, and those ecosystems are pain in the ass to deal with. I'm talking XKCD #1987, or NodeJS circa 2014, or just the entire rationale behind Docker. People exit their careers to not have to deal with such technology, because it's like working at the DMV or living in Kafka's nightmares. LLMs are more in that direction, and no one is going to like where we end up if we make them our entire stack, as seems to be the intent by the powers that be.
There's a difference between what LLMs are and what they're being sold as. For what they are, they can be useful and may one day they will be turned into good tools if some of the major flaws are fixed.
On the other hand, we are in the process of totally upending the way our industry works on the basis of what these things will be, which they are selling as essentially an oracle. "The smartest person who know in your pocket", "a simultaneous expert PhD, MD, JD", "Smarter than all humans combined". But there's a giant gulf between what they're selling and what it is, and that gulf is what makes LLMs a poor tool.
They are good tools to me.
We won't agree on this.
The provide abilities no other tool provides me with. They could be better, but they've still provided me with possibilities that I have never had before without hiring humans.
I'm sure they're good for you, I'm not suggesting otherwise. What I'm saying is, if you ask 100 engineers to describe the properties of the best tools they use, the set of adjectives and characteristics come up with will largely not be applicable to LLMs, and ChatGPT 5 doesn't change that.
This is pure, unsubstantiated conjecture.
It's also wildly unrealistic conjecture, in my opinion.
The first, and most important, measure to me of a good tool is whether it makes me more productive. LLMs does. Measurably so.
You will certainly find people who don't believe LLMs do for them, but that won't change the fact that for a lot of us it is an immensely good tool. And a tool doesn't need to fit everyones processes to be a good tool.
> It's also wildly unrealistic conjecture, in my opinion.
The only property of good tools you've mentioned that LLMs have is they do something useful.
But are they reliable? No, they inexplicably work sometimes and don't work other times. Do they have a clear purpose? No, their purpose is muddled and it's sold as being capable of literally everything. Are the ergonomics good? No, the interface is completely opaque, accessed only behind a natural language text box that comes with no instructions on how to use. But then it must be intuitive, right? No, common wisdom on how to use them sound more like astrological forecasts, or the kind of advice you give people trying to get something from a toddler -- "If you want good results, first you have to get into character!"... etc. etc.
> it makes me more productive.
Awesome! I'm sure they are doing exactly what you say and your experiences with them are amazing. But yours and my personal productivity isn't the question facing the industry at the moment or the topic of this discussion.
The question isn't whether you personally as an induvial find these things useful in your process. If that were the question I wouldn't be here complaining about them. I'm here because the powers that be are telling us that we must adopt these things into every aspect of our work lives as soon as possible, and if we don't, we deserve to be left behind. THAT is what I'm here to talk about.
> And a tool doesn't need to fit everyones processes to be a good tool.
I haven't argued this. Of course a tool doesn't have to fit everyone's process to be good, but we have some generally accepted principles of good tool design, and LLMs don't follow them. That doesn't preclude you from using them well in your process.
> I have never met a human who has a good grasp of what they know and don't know
But humanity has a good grasp over it, we even created science to solve this. So out of all existences we are aware of humanity is by far the best at this, nothing else even comes remotely close.
No, we really don't. We have major disagreements about what we know and don't know all the time.
Like right now.
A LLM needs to be told what it knows. You don't It can never with reasonable accuracy say "I don't know" as a human would
And humans are often wrong both when we say we don't know, and when we claim to know. There likely is a difference in degree of accuracy, but the point I was making was that despite your claim to "certainly know what [you] know", we don't in fact know what we know with anything remotely near precision.
We know some of what we know, but we can both be pressed into doing things we are certain we don't know how to do but where the knowledge is still there, and we will confidently proclaim to know things (such as the extent of our knowledge) that we don't.
I will agree that LLMs need to acquire a better idea of what they know, but there is no reason to assume that knowing the limits of your own knowledge with any serious precision matters, given how bad humans are at knowing this.
So much of human culture, politics, and civil life is centered around resolving conflicts that arise out of our lack of knowledge of our own limits, that this uncertainty is fairly central to what it means to be human.
I’m afraid that sense of knowing what you know is very much illusory for humans as well. Everyone is just slowly having to come to terms with that.
No. I can tell you what skills I possess. For example: programming, writing music. A LLM can not do this unless it's told what it knows
You are judging this based on what the LLM outputs, not on its internals. When we peer into its internals, it seems that LLMs actually have a pretty good representation of what they do and don't know; this just isn't reflected in the output because the relevant information is lost in future context.
Do you really know what you don't know? This would rule out unknown unknowns entirely.
Yes, it's not that people know specifically what they don't know, it's that they develop the wisdom to know those boundaries and anticipate them and reduce their likelihood and impact.
For example, if I use the language of my expertise for a familiar project then the boundaries where the challenges might lie are known. If I start learning a new language for the project I won't know which areas might produce unknowns.
The LLM will happily give you code in a language it's not trained well on. With the same confidence as using any other language.
Sorry for copypasting from another comment but this is relevant
I can tell you what skills I possess. For example: programming, writing music. A LLM can not do this unless it's told what it knows. I could also tell you whether I studied thing X or attempted to do it and what success I had. So I'm pretty good at assessing my abilities. A LLM has no idea
This is interesting, because I wouldn't be too sure about that. Whether I am able to play chess well exclusively depends on the strengths of my opponents, because there is no absolute baseline. If society somehow decided that what you were writing wasn't "music" anymore, you would be the only person left stating you had that skill.
I believe that most claims about one's own skills do come from outward judgement and interpretation of one's actions, not introspection. The only thing humans have is, to radically appropriate the jargon, a way longer context window (spanning over an entire lifetime!), together with many ways to compress it.
> OpenAI researcher Noam Brown on hallucination with the new IMO reasoning model:
> Mathematicians used to comb through model solutions because earlier systems would quietly flip an inequality or tuck in a wrong step, creating hallucinated answers.
> Brown says the updated IMO reasoning model now tends to say “I’m not sure” whenever it lacks a valid proof, which sharply cuts down on those hidden errors.
> TLDR, the model shows a clear shift away from hallucinations and toward reliable, self‑aware reasoning.
Source: https://x.com/chatgpt21/status/1950606890758476264
Socrates would beg to differ
Yeah. The empty “it’s just a statistical model” critique (or the dressed-up “stochastic parrots” version of it) is almost a sign at this point that the person using it formed their opinions about AI back when ChatGPT first came out, and hasn’t really bothered to engage with it much since then.
If in 2022 I’d tried to convince AI skeptics that in three years we might have tools on the level of Claude Code, I’m sure I’d have heard everyone say it would be impossible because “it’s just a statistical model.” But it turned out that there was a lot more potential in the architecture for encoding structured knowledge, complex reasoning, etc., despite that architecture being probabilistic. (Don’t bet against the Bitter Lesson.)
LLMs have a lot of problems, hallucination still being one of them. I’d be the first to advocate for a skeptical hype-free approach to deploying them in software engineering. But at this point we need careful informed engagement with where the models are at now rather than cherry-picked examples and rants.
Unless what you work on is very simple and mostly mindless, using tools like Claude Code is the exact opposite of how to make the current SotA LLMs useful for coding. The models can help and boost your productivity, but it doesn't happen by letting them do more stuff autonomously. Quite the contrary.
And when what you usually work on actually is very simple and mostly mindless, you'd probably benefit more from doing it yourself, so you can progress beyond the junior stuff one day.
Where it really has value is if what you work on is like 33% extremely difficult and 66% boilerplate/tedious. Being able to offload the tedious bits can make more senior engineers 2-3x more productive without the coordination effort of "find a junior engineer to do this, schedule their time, assign the work, follow up on it".
(The problem of course is you still need these junior engineers to exist in order to have the next generatino of senior engineers, so we must now also think about what our junior folks should be doing to be valuable AND learn).
"real programmers use ed", 2025 edition: you're forever stuck with "junior stuff" if you let a language model handle language
god forbid I don't have to read 10k lines of logs to fix a typo
> If in 2022 I’d tried to convince AI skeptics that in three years we might have tools on the level of Claude Code, I’m sure I’d have heard everyone say it would be impossible because “it’s just a statistical model.”
We already had these coding models in 2022, they were already pattern matching engines with variables back then, all your imagination needed to do to go from that to Claude code today is to give it more code examples and make it bigger.
They still can't replace even a junior engineers ability to navigate tasks over even short periods of time, just like back then they need constant handholding to get anything done. So I don't see what changed except the model being larger with more examples and therefore you can get larger chunks of coherent code out of them.
People repeating the "stochastic parrot" meme in all kinds of variations if anything appear to be more like stochastic parrots than the typical LLM is.
Actually internally we do know what’s going on these days. Anthropic put out a white paper detailing how Claude can’t math but many math examples are out there so Claude can fake it.
I wish you’d stop magic LLMs some kind of magic thing they aren’t.
I work in the theory of deep learning, so I can say with some authority that while we know a really good number of things, and are able to probe the internals much better than most of the public realises, when it comes to philosophical questions that compare their nature with humans, we have a long, long way to go. The biggest problem is that we're still trying to work out what it is we even want to know that will tell us whether we have achieved AGI or not.
Linear probes and autoencoders have been useful, but we're quickly reaching the limits of those techniques. And don't even get me started on approaches to theory that operate by cherry-picked examples. Anthropic's contributions have been beneficial to the field, but are far from conclusive.
Humans build theories of how things work. llms dont. Theories are deterministic symbolic representation of the chaotic worlds of meaning . Take the turing machine for example as a theory of computation in general, euclidean geometry as a theory for space, and newtonian mechanics as a theory for motion.
A theory gives 100% correct predictions. Although the theory itself may not model the world accurately. Such feedback between the theory, and its application in the world causes iterations to the theory. From newtonian mechanics to relativity etc.
Long story short, the LLM is a long way away from any of this. And to be fair to LLMs, the average human is not creating theories, it takes some genius to create them (newton, turing, etc).
Understanding something == knowing the theory of it.
> Humans build theories of how things work. llms dont. Theories are deterministic symbolic representation of the chaotic worlds of meaning
What made you believe this is true? Like it or not, yes, they do (at least to the best extent of our definitions of what you've said). There is a big body of literature exploring this question, and the general consensus is that all performant deep learning models adopt an internal representation that can be extracted as a symbolic representation.
> What made you believe this is true?
I am yet to see a theory coming of the LLM that is sufficiently interesting. My comment was answering your question of what does it mean to "understanding something". My answer to that is: understanding something is knowing the theory of it.
Now, that begs the question of what is a theory. And to answer that, a theory comprises of building block symbols and a set of rules to combine them. for example, building blocks for space (and geometry) could be points, lines, etc. The key point in all of this is symbolism as abstractions to represent things in some world.
The "sufficiently interesting" part is the most important qualifier here. My response was talking about theories and representations that we already know, either instinctively from near-birth, or from learned experience. We have not seen anything unique from LLMs because they do not appear to have an internal understanding (in the same sense that I was talking about) that is as broad as an adult human. But that doesn't mean it lacks any understanding.
> The key point in all of this is symbolism as abstractions to represent things in some world.
The difficulty is understanding how to extract this information from the model, since the output of the LLM is actually a very poor representation of its internal state.
> This concludes all the testing for GPT5 I have to do. If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it’s a useless tool.
Yeah, except it isn't. You can get enormous value out of LLMs if you get over this weird science fiction requirement that they never make mistakes.
And yeah, their confidence is frustrating. Treat them like an over-confident twenty-something intern who doesn't like to admit when they get stuff wrong.
You have to put the effort in to learn how to use them with a skeptical eye. I've been getting value as a developer from LLMs since the GPT-3 era, and those models sucked.
> The only way ChatGPT will stop spreading that nonsense is if there is a significant mass of humans talking online about the lack of ZSTD support.
We actually have a robust solution for this exact problem now: run the prompt through a coding agent of some sort (Claude Code, Codex CLI, Cursor etc) that has access to the Swift compiler.
That way it can write code with the hallucinated COMPRESSION_ZSTD thing in it, observe that it doesn't compile and iterate further to figure out what does work.
Or the simpler version of the above: LLM writes code. You try and compile it. You get an error message and you paste that back into the LLM and let it have another go. That's been the main way I've worked with LLMs for almost three years now.
If that same intern, when asked something, responded that they checked, gave you a link to a document they claim have the proof / answer but does in fact no and continued to do that they wouldn’t be an intern very long. But somehow this is acceptable behaviour for an AI?
I use AI for sure, but only on things that I can easily verify is correct (run a test or some code ), because I have had the AI give me functions in an API with links to online documentation for those functions, the document exists, the function is not in it, when called out instead of doing a basic tool call the AI will double down that it is correct and you the human are wrong. That would get an intern fired but here you are standing on the interns side.
Because LLMs aren't human beings.
I wrote a note about that here: https://simonwillison.net/2025/Mar/11/using-llms-for-code/#s...
> Don’t fall into the trap of anthropomorphizing LLMs and assuming that failures which would discredit a human should discredit the machine in the same way.
> And yeah, their confidence is frustrating. Treat them like an over-confident twenty-something intern who doesn't like to admit when they get stuff wrong.
I was explicitly calling out this comment, that intern would get fired if when explicitly called out they not only don’t want to admit they are wrong but vehemently disagree.
The interaction was “Implement X”, it gave an implementation, I responded “function y does not exist use a different method”, it instead of following that instruction gave me a link to the documentation for the library that it claim’s contains that function and told me I am wrong.
I said the documentation it linked does not contain that function and to do something different and yet it still refused to follow instructions and pushed back.
At that point I “fired” it and wrote the code myself.
> Or the simpler version of the above: LLM writes code. You try and compile it. You get an error message and you paste that back into the LLM and let it have another go. That's been the main way I've worked with LLMs for almost three years now.
I’m going to comment here about this but it’s a follow on to the other comment, this is exactly the workflow I was following. I had given it the compiler error and it blamed an environment issue, I confirmed the environment is as it claims it should be, it linked to documentation that doesn’t state what it claims is stated.
In a coding agent this would have been an endless feedback loop that eats millions of tokens.
This is the reason why I do not use coding agents, I can catch hallucinations and stop the feedback loop from ever happening in the first place without needing to watch an AI agent try to convince itself that it is correct and the compiler must be wrong.
You're responding to simonw, the guy who could be considered the single leading voice in practical applications of llms, with an anecdote about how one time the bot gave you a compiler error, and llm coding is therefore useless.
I would agree with you if it was just one time, this is multiple times.
Also look up appeal to authority, everyone can be better, I respect Simon don’t get me wrong, but I don’t read names before I comment and you would probably do well to do the same, let what people say stand on their own.
I read what you wrote and let it stand on its own. Your comment mentioned a single time the bot had failed - there was no mention of it occurring multiple times. Maybe you should flesh out your anecdotes properly.
If you are treating anonymous posts on forums with the same authority as experts on those topics then it makes sense why you are struggling to use new tooling.
> I use AI for sure, but only on things that I can easily verify is correct
Where did I say I was struggling, I stated a preference, I even stated lived experience with using a coding agent and a chat based LLM interface and have a preference for the chat based version.
I don’t struggle to use modern tooling, I pointed out that the analogy is terrible and how my lived experience proves that, I do use LLMs in anger daily for coding tasks, I just don’t treat them like interns.
Many time the points I used indicate that what I am doing is not in the training set, when you work on hard problems you run into edge cases a little more reliably than when you don’t.
Stop defending a man you don’t know on the internet, his words defend him well enough and I can hope he just stops using that analogy because it is fundamentally flawed, which is the point of my comments. The rest of his comment makes perfect sense and is the way I use LLMs and have to work around their idiosyncrasies anyway.
Wow... people unironically anthropomorphize AI to the point that they expect AI to work exactly like an human intern, otherwise it's unacceptable...
>...it’s a useless tool. I don’t like collaborating with chronic liars who aren’t able to openly point out knowledge gaps...
I think a more correct take here might be "it's a tool that I don't trust enough to use without checking," or at the very least, "it's a useless tool for my purposes." I understand your point, but I got a little caught up on the above line because it's very far out of alignment with my own experience using it to save enormous amounts of time.
and as others have pointed out, this issue of “how much should I check” is really just a subset of an old general problem in trust and knowledge (“epistemology” or what have you) that people have recognized since at least the scientific revolution. The Royal Society’s motto on its founding in the 1660s was “take no man’s word for it.”
Coding agents have now got pretty good at checking themselves against reality, at least for things where they can run unit tests or a compiler to surface errors. That would catch the error in TFA. Of course there is still more checking to do down the line, in code reviews etc, but that goes for humans too. (This is not to say that humans and LLMs should be treated the same here, but nor do I treat an intern’s code and a staff engineer’s code the same.) It’s a complex issue that we can’t really collapse into “LLMs are useless because they get things wrong sometimes.”
> Coding agents have now got pretty good at checking themselves against reality, at least for things where they can run unit tests or a compiler to surface errors.
YMMV. I've seen Claude go completely batshit insane saying that tests all passed. Then I run them and I see 50+ failures. I copy the output tell him to fix it and he goes on his sycophantic apologia before spinning his wheels doing nothing and saying all tests are back to green.
It’s a tool that fundamentally can’t be used reliably without double checking everything it. That is rather different than you’re presenting it.
We double check human work too in all kinds of contexts.
A whole lot of my schooling involved listening to teachers repeating over and over to us how we should check our work, because we can't even trust ourselves.
(heck, I had to double-check and fix typos in this comment)
It’s very weird to me when someone says “this tool does not have this common property good tools have” and someone replies “humans also do not have those properties!” As if that is responsive to a complaint about a tool lacking a common property of tools.
We use tools because they work in ways that humans do not. Through centuries of building and using tools, we as a society have learned what makes a tool good versus bad.
Good tools are reliable. They have a clear purpose and ergonomic user interface. They are straightforward to use and transparent in how they operate.
LLMs are none of these things. It doesn’t matter that humans also are none of these things, if we are trying to use LLMs as tools.
The closest human invention resembling an LLM is the idea of a bureaucracy — LLMs are not good tools, they are not good humans, they are mindless automatons that stand in the way and lead you astray.
At best, LLMs are poor tools and also poor human replacements, which is why it’s so frustrating to me we are so intent on replacing good tools and humans with LLMs.
The reason for making that observation is that we don't have any other comparable tools and it is more reasonable to benchmark LLMs against what humans are capable of, because whether or not they are good approximations, we are trying to model human abilities.
One day maybe we exceed human abilities, but it's unreasonable to expect early attempts - and they are still early attempts - to do things we don't know how to beat other than by putting all kinds of complex process on top of very flawed human thinking.
We do have comparable tools to LLMs. There are plenty of human-composed tools that can do what LLMs do, like Mechanical Turk for instance. The human-composed tool that most closely resembles a LLM is the "bureaucracy".
An LLM is like a little committee you send a request to, and based on capricious, opaque rules that can change at any time, the committee returns a response that may or may not service your request. You can't know ahead of time if it will, and depending on the time of day, the political environment, or the amount of work before the committee, the delay of servicing and the quality of service may or may not degrade. Quality may range from a quick accurate response, to flat refusal to service without explanation, or outright lies to your face. There's no way to guarantee a good result, and there's no recourse or explanation for why things go wrong or changed.
LLMs feel like a customer service agent turned into a computer program, which is probably why it was people's first thought to use LMMs to automate customer service agents. They are a perfect fit there, but I don't want them to be my primary interface to do work. I have enough bureaucracies to deal with as it is.
> We do have comparable tools to LLMs. There are plenty of human-composed tools that can do what LLMs do, like Mechanical Turk for instance.
If you are going to treat humans as tools, then sure. In which case measuring LLMs against human ability is exactly the right thing, given that with Mechanical Turk the tasks are carried out by humans - sometimes with the help of LLMs...
It's utterly bizarre to argue over my comparing LLMs to humans when the tools you argue are comparable are humans.
> when the tools you argue are comparable are humans.
No, they are abstractions over humans. A group of people is not a person, they behave differently than people even though they are composed of them. Abstractions are hard to compare but still much easier than people.
This is a meaningless difference that does not alter any of what I wrote.
You're just trying to evade dealing with the contradiction in your argument.
It's not a meaningless difference, it's a crucial one. The contradiction only exists if you collapse all the differences between people and abstractions of people -- crucially that the former are people and the latter are abstractions -- and claim they're the same. Which they are not.
Anyway, we've gotten far from the point, which is that LLMs are not people and you can't treat them as such.
Checking is usually faster than writing from scratch so this is still +EV
What does +EV mean? I'm looking but can't find a definition
Positive expected value. In other words, it’s likely using a LLM saves you time relative to performing the same task without using an LLM.
"Positive expected value", i.e. it'll pay off on average.
Expected Value?
> Checking is usually faster than writing from scratch
Famous last words. Checking trivial code for trivial bugs, yes. In science, you can have very subtle bugs that bias your results in ways that aren't obvious for a while until suddenly you find yourself retracting papers.
I've used LLMs to write tedious code (that should probably have been easier if the right API had been thought through), but when it comes to the important stuff, I'll probably always write an obviously correct version first and then let the LLM try to make a faster/more capable version, that I can check against the correct version.
Humans are notoriously bad at checking for meticulous detail—hence why copyediting was a specialized role. We’ve seen what’s happened since the newspapers eliminated it for cost savings.
I only used an LLM for the first time recently, to rewrite a YouTube transcript into a recipe. It was excellent at the overall restructuring, but it made a crucial and subtle mistake. The recipe called for dividing 150g of sugar, adding 30g of cornstarch to one half, and blanching eggs in that mixture. ChatGPT rewrote it so that you blanched the eggs in the other half, without the cornstarch. This left me with a boiling custard that wasn’t setting up.
I did confirm that the YouTube transcript explicitly said to use the sugar and cornstarch mixture. But I didn’t do a side by side comparison because the whole reason for doing the rewrite is that transcripts are difficult to read!
> I'll probably always write an obviously correct version first
I’m not usually so confident in my own infallibility, so I prefer to think of it as “I might get this wrong, the LLM might get this wrong, our failure modes are probably not very correlated, so the best thing is for us both to do it and compare.”
Agree it is always better for the human engineer to try writing the critical code first, since they are susceptible to being biased by seeing the LLM’s attempt. Whereas you can easily hide your solution from the LLM.
> I’m not usually so confident in my own infallibility
I'll concede that point, but let me put it differently: when I write the "obviously correct" version, what I mean is that I can explain what every line of code is supposed to be doing. I can say what the expected result would be, at each step and test it.
When the LLM writes an inscrutable version for me, I might be able to test the final result, but I'll never know if there are corner cases (or not even that) in between where it will go spectacularly wrong. If I have to untangle what it's written to get that kind of confidence, then it's going to take me much longer. Reading code, especially somebody else's, nevermind an alien's code, is just so much harder for me.
So similar to wikipedia
Similar to anything really. Can I really trust anything without verifying? Scientific journals?
It seems that on some level, you have to in order to not just constantly reflecting upon your thoughts and researching facts. Whether you trust a given source should surely depend upon its reputation regarding the validity of its claims.
I agree and by reputation you mean accuracy. We implicitly know not to judge anything as 100% true and implicitly apply skepticism towards sources - the skepticism is decided by our past experience with the sources.
Think of LLMs as the less accurate version of scientific journals.
Accuracy certainly does play a role, but this in itself is not sufficient for preventing an infinite regress – how does one determine the accuracy of a source if not by evaluating claims about the source, that themselves have sources that need to be checked for accuracy? Empirical inquiries are optimal but often very unpractical. Reputation is accuracy as imperfectly valued by society or specific social groups collectively.
Do you verify everything? When your wife puts food in front of you, do you make her take a bite off your plate first to check for poison?
A crucial difference between LLMs and people is that you can build a mutual trust relationship with a person.
Exactly - also, if I were 14 I'd be hyperventilating right now. I love your blog posts!
[dead]
"Based on my research, zstd compression is not natively supported by iOS or Apple's frameworks, which means you cannot use zstd compression without adding some form of external code to your project"
Thanks Sonnet.
Full response:
https://www.perplexity.ai/search/without-adding-third-party-...
Most likely the key here is web search. Later I will try the post’s example with gpt-5 with search. I would be surprised if it didn’t say the same thing.
From a product point of view, it seems clear that just as they have work to get the model to dynamically decide to use reasoning when it would help, they have to do the same with web search.
gpt-5 with medium reasoning, no search: "Short answer: use the system zstd C library that ships with iOS; Apple’s Compression framework still doesn’t expose zstd. Link libzstd and call its streaming API from Swift." (and then like 50 lines of code plus some explanation)
gpt-5 with medium reasoning + search: "Short answer: you can’t. Apple’s built‑in Compression framework on iOS exposes LZFSE, LZ4, ZLIB/DEFLATE, and LZMA—not Zstandard—so there is no system API to do zstd without bundling an external library. (https://www.hackingwithswift.com/example-code/system/how-to-...)
If you can switch to a built‑in algorithm, here’s a minimal streaming compressor using Compression (LZFSE shown):" (etc etc)
Author is arguing:
* LLMs, when presented with a question to which they don't know the answer, tend to make up a wrong answer rather than say they don't know.
* Therefore, LLMs are useless.
Yes, LLMs do that, and it is a problem.
No, that does not make them useless. It's usually much easier to check their answer than it would have been to search for an answer yourself. If they are right, you saved time. If they are wrong, at worst you wasted a couple minutes. But they are right often enough that it can be worthwhile, especially if you have a feel for what questions they are likely to know the answer to and what questions they aren't.
An implication from the 1985 paper of Peter Naur on programming as theory building is that the current LLM coding tool would be very effective in generating technical debt even when it works ... use at your own risk.
I'm currently working as a hands-off VP, so I don't use LLMs for coding at work, only for emails and document editing. I do use it for my hobby weekend coding stuff, which usually involves writing short 100-1000 LOC Python toy programs for my own education/entertainment. My way-of-working is quite primitive, I have zero integrations, nothing agentic, I just copy/paste with ChatGPT.
For this use-case it's been very useful, it can usually generate close-to-complete solutions, as long as it's one of the major programming languages, and it's a reasonably standard problem. So in general I'm always surprised when people say that LLMs are completely useless for coding --- this is just not true, and I feel sorry for people who shut themselves off from a useful tool.
But even at this small scale, even the best (o3) models sometimes totally fail. Recently I started a series of posts on distributed algorithms [1], and when I was working on the post/code for the Byzantine Generals / Consensus algorithm, o3 --- to my honest surprise --- just totally failed. I tried about 10 different times (both from scratch and describing the incorrect behaviour of its code), also showing it the original Lamport paper, and it just couldn't get it right... even though the toy implementation is just ~100 LOC, the actual algorithm portion is maybe 25 LOC. My hypothesis is that there are very few implementations online, and additionally I find the descriptions of the algorithm a bit vague (interleaves message cascade and decision logic).
[1] https://bytepawn.com/tag/distributed.html
ChatGPT 5> Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?
(It auto-routed to its thinking model, took 25s)
> Short answer: you can’t. iOS doesn’t ship a public zstd encoder/decoder. The system Compression and AppleArchive frameworks only expose lzfse, lz4, zlib/deflate, and lzma. So with only Apple SDKs, there’s no way to produce zstd-compressed bytes.
> > Two practical paths:
> 1. If zstd isn’t mandatory, use a built-in codec (lzfse or lz4 are fastest) via Compression’s streaming API. Example (pure Swift, no deps) that compresses an arbitrary byte stream: >
> <code example>
> 2. If zstd is mandatory, you must embed zstd yourself. That can mean:
> - Vendoring the official C sources (libzstd) directly into your project and calling them via a tiny Swift wrapper/bridging header
> - Statically linking a prebuilt libzstd.a you compile for iOS
Here's what I got with ChatGPT GPT-5, it must not have thought about it because it was near instantaneous:
>On iOS, you can use Apple’s built-in Zstandard (zstd) compression API from the Compression framework — no third-party dependencies required.
>Here’s how you can compress a Data stream with zstd: >...
https://chatgpt.com/share/68976c8f-7ae0-8012-b7a8-58e016246d...
Looks like that time it didn't route to the thinking model. The router mechanism is deeply confusing right now: https://twitter.com/emollick/status/1954210778321465634
I think the useful takeaway here is that Top 1 operation is generally not a good idea, especially not for making judgements. This doesn't address the main points of the blog though.
what's funny is that newer models will now be trained on the exact question, "Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?" and similar questions to it, because of this post
maybe the key to training future llm's is to write angry blog posts about the things they aren't good at and get them to the front page of hn?
good point. nobody knows you are a dog on internet anyways
The prompt works for me and correctly identifies that zstd doesn't work https://chatgpt.com/share/689769c5-bd68-800b-ae63-c6a337dcfa...
"Short answer: you can’t. iOS doesn’t ship a Zstandard (zstd) encoder/decoder in any first-party framework. Apple’s built-in Compression framework supports LZFSE, LZ4, zlib/deflate, and LZMA—not zstd."
I wonder if one reasons new versions of GPT appear to get better - say at coding tasks is just because they have new knowledge.
When ChatGPT4 comes out, new versions of API’s will have less blog post / examples / documentation in their training data. So ChatGPT 5 comes out and seems to solve all the problems that ChatGPT4 had, but then of course fail on newer libraries. Rinse and repeat
> ... just because they have new knowledge.
This means there is a future where AI is training on data it self generated, and I worry that might not be sustainable.
A software based Habsburg Jaw if you will.
I’ve heard of this idea of training on synthetic data, I wonder what is that data and does this increase or decrease hallucinations? Is the goal of training on synthetic data to better wear certain paths, or to increase the amount of knowledge / types of data.
Because the second seems vaguely impossible to do.
This is already occurring, is not sustainable, and produces an effect known as Model Collapse.
possible solution: "reality checks"
I see that GitHub Copilot actually runs code, writes simple exploratory programs, iteratively tests its hypothesis. it is astoundingly effective and fast.
same here. nothing stops this AI to actually trying to implement whatever this AI suggested, compile it, and see if this is actually works.
grounding in reality at inference time, so to speak.
LLMs can be a useful tool, maybe. But don’t anthropomorphize them.
(but, earlier)
If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it’s a useless tool. I don’t like collaborating with chronic liars.
With chronic liar being a severe anthropomorphization by itself due to assuming some level of intent by the LLM, correct?
Yes. They had it right at the end of the piece, when they said "don't anthropomorphize". LLMs don't "lie".
> “Not having an answer” is not a possibility in this system - there’s always “a most likely response”, even if that makes no sense.
simple fix - probability cutoff. but in all seriousness this is something that will be fixed. don't see fundamental reason why not.
and I myself seen such hallucinations (about compression too actually) as well.
Hallucinations are not a bug or an exception, but a feature. Everything outputted by LLMs is 100% made-up, with a heavy bias towards what has been fed to it at first (human written content).
The fundamental reason why it cannot be fixed is because the model does not know anything about the reality, there is simply no such concept here.
To make a "probability cutoff" you first need a probability about what the reality/facts/truth is, and we have no such reliable and absolute data (and probably never will).
>To make a "probability cutoff" you first need a probability about what the reality/facts/truth is, and we have no such reliable and absolute data (and probably never will).
Can a human give a probability estimate to their predictions?
Humans can explain how they arrived at the conclusion, an LLM fundamentally cannot do that since they don't remember why they picked the tokens they did, they just make up an explanation based on explanations it has seen before.
You use a lot of anthropomorphisms: doesn't "know" anything (does your hard drive know things? Is it relevant?), "making things up" is even more linked to continuous intent. Unless you believe the LLMs are sentient this is a strange choice of words.
I originally put quotes around "know" and somehow lost it in an edit.
I'm precisely trying to criticize the claims of AGI and intelligence. English is not my native language, so nuances might be wrong.
I used the word "makes-up" in the sense of "builds" or "constructs" and did not mean any intelligence there.
have you seen Iris flowers dataset? it is fairly simple to find cutoffs to classify flowers.
or are you claiming in general that there is no objective truth in reality in philosophical sense? well, you can go on that more philosophical side of the road, or you can get more pragmatic. things just work, regardless how we talk about them.
I don't mean it in a philosophical sense, more in a rigorous scientific one.
Yes, we do have reliable datasets as in your example, but those are for specific topics and are not based on natural language. What I would call "classical" machine learning is already a useful technology where it's applied.
Jumping from separate datasets focused on specific topics to a single dataset describing "everything" at once is not something we are even close to doing, if it's even possible. Hence the claim of having a single AI able to answer anything is unreasonable.
The second issue is that even if we had such a hypothetical dataset, ultimately if you want a formal response from it, you need a formal question and a formal language (probably something between maths and programming?) in all the steps of the workflow.
LLMs are only statistical models about natural languages, so it's the antithesis of this very idea. To achieve that would have to be a completely different technology that has yet to even be theoretized.
I call this genre of article "I'm too lazy to think about how to deal with the fact that LLMs are sometimes wrong, especially when they aren't using thinking and are given short, leading prompts."
Ok great! For those of us who aren't too lazy for it, LLMs are providing a lot of value right now.
I'm not sure that the people who still prefer to do their own work rather than circuitously delegate it are the ones that seem lazy.
Finding a useless way to use a tool is neither interesting or novel. I hit my thumb with a hammer, useless tool.
>Use your brains.
I love posts like these because it just reinforces that I made the right decision in spending as much time as I do in getting really really really good at using llms.
I think people often misread AGI as Artificial God Intelligence.
if you treat it like stochastic software it is, how is 95% chance of one-shotting a problem not useful? I feel like the biggest problem most people have with LLMs is the "lack of accountability", and then the same people say "don’t anthropomorphize them"
If you have an Apple docs mcp, it does let you know and offers alternatives. So this is another “your holding it wrong” problem.
I find it hard to empathize with these sorts of articles. I think in, in the spate of GPT-5 related content recently, I've been seeing a lot of articles that boil down to "I tried GPT-5 on a single hard question, and it gave an answer. This proves that all LLMs are useless." And I don't think I'm distorting the author's viewpoint. He goes on to say "This concludes all the testing for GPT5 I have to do" after conducting a single test.
This seems like particularly harsh criteria; what would happen if I applied this to other tools?
- I used Typescript, but it missed a bug that crashed prod, so it is "absolute horseshit"
- I used Rust, but one of my developers added an unsafe block, so it's trash.
> I used Typescript...it is "absolute horseshit"
That's right
My issue with LLMs isnt really the tech itself. They're occassionally useful, although HOW useful seems to depend on a persons skillset.
My main beef with the AI hype is that it's allowing a lot of idiots to significantly devalue our profession in a really noxious and irritating way to people that generally don't understand what we do but would like to pay us less or pay less of us. I'm annoyed at other software developers that don't seem to see how harmful this will be for us when the insane investment bubble bursts and AI becomes a lot more expensive to use. We will probably have lost a generation of junior developers who have become dependent on a suddenly expensive tool. And execs will just think the seniors need to pick up the slack. And expectations on AI will be a lot higher when the subscription is more like 200 or 2000 a month.
And that's just for coding! I'd be furious if I was an artist and generative AI was being trained on my portfolio to plagiarize my work. (Badly)
What I never see justified is why any of this is good for society. At best it lets billionionaires save some money by getting rid of jobs, or vibe coders pretend they can build a product until they hit a wall where real understanding is neccessary. If you follow the trail of who is supposed to benefit from these things its not many of us. If AI were to disappear today I don't think my life would be any worse.
There was a writer who in order to get ideas to write about used to cut up words from newspaper headlines and then rearrange them.
In one rearrangement, he got "Son sues father for xyz". That headline came true 2 years later.
How is this relevant? Modern LLMs don't rearrange words at all in a meaningful sense of the word, and are certainly better than just using random chance to sample tokens.
Indeed it is exactly that process. They cutup words. Then categorize them based on some metric of nearness (not random). Then link them up. Obviously this process is much more sophisticated than what I have described here.
The top comments are focusing on what "know" means in the context of an LLM but I think that while it's an interesting discussion it misses the point. Even if we agree that an LLM can "know" things, the tech has been heavily advertised as nothing less that transformative. If it can't help itself from inventing non existent answers or count the number of "r"s in the word blueberry (a test I tried myself the other day on my phone's chat-gpt app, it said "three") it's not transformative.
I'm almost positive that he didn't use thinking mode, because I've asked GPT-5 thinking his question verbatim in 10 parallel threads and it said "you can't" and thoroughly explained why every time. Below is the first response I got back.
---
Short answer: you can’t. iOS doesn’t ship a Zstandard encoder/decoder. Apple’s built-in `Compression` framework only exposes LZFSE, LZ4(/RAW), zlib/DEFLATE, and LZMA—no ZSTD knob to turn. ([Microsoft Learn][1], [Apple Developer][2])
If you’re okay with a built-in alternative that’s “zstd-ish” in speed/ratio, use LZFSE via `Compression`’s streaming API. Here’s a minimal streaming compressor you can drop in; it takes chunks of `Data` and yields compressed chunks (no third-party code):
```swift import Compression
final class StreamCompressor { private var stream = compression_stream() private let algo: compression_algorithm private var isInitialized = false
} ```Usage:
```swift let compressor = StreamCompressor(algorithm: COMPRESSION_LZFSE) var compressed: [Data] = [] for (i, chunk) in chunks.enumerated() { let final = i == chunks.count - 1 compressed.append(contentsOf: compressor.append(chunk, isFinal: final)) } // `compressed` now contains the zstd-adjacent (LZFSE) stream ```
If you truly need the ZSTD format (interop requirement), you’ll have to ship the reference C library (`libzstd`) in your app (static link it and call through a tiny Swift wrapper). That’s still “third-party,” even if you vendor the sources instead of using SPM/CocoaPods. There is no system-provided ZSTD on iOS today. ([GitHub][3])
tl;dr: iOS doesn’t expose ZSTD. Use `Compression` with LZFSE for built-in streaming, or vendor `libzstd` and bridge to it if you must emit/ingest ZSTD. ([Microsoft Learn][1], [Apple Developer][2], [GitHub][3])
[1]: https://learn.microsoft.com/en-us/dotnet/api/compression.com... "CompressionAlgorithm Enum (Compression)" [2]: https://developer.apple.com/documentation/compression/compre... "COMPRESSION_LZFSE | Apple Developer Documentation" [3]: https://github.com/facebook/zstd?utm_source=chatgpt.com "facebook/zstd: Zstandard - Fast real-time compression ..."