> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful.
What a great way of framing it. I've been trying to explain this to people, but this is a succinct version of what I was stumbling to convey.
I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better.
In my company I feel that we getting totally overrun with code that's 90% good, 10% broken and almost exactly what was needed.
We are producing more code, but quality is definitely taking a hit now that no-one is able to keep up.
So instead of slowly inching towards the result we are getting 90% there in no time, and then spending lots and lots of time on getting to know the code and fixing and fine-tuning everything.
Maybe we ARE faster than before, but it wouldn't surprise me if the two approaches are closer than what one might think.
What bothers me the most is that I much prefer to build stuff rather than fixing code I'm not intimately familiar with.
Technology has always helped people. Are you one of the people that say optimizing compilers are bad? Do you not use the intellisense? Or IDEs? Do you not use higher level languages? Why not write in assembly all the time? No free lunch right.
Yes there are trade offs, but at this point if you haven’t found a way to significantly amplify and scale yourself using llms, and your plan is to instead pretend that they are somehow not useful, that uphill battle can only last so long. The genie is out of the bag. Adapt to the times or you will be left behind. That’s just what I think.
Technology does not always help people, in fact often it creates new problems that didn't exist before.
Also telling someone to "adapt to the times" is a bit silly. If it helped as much as its claimed, there wouldn't be any need to try and convince people they should be using it.
A LOT of parallels with crypto, which is still trying to find its killer app 16 years later.
I don’t think anyone needs to be convinced at this point. Every developer is using LLM and I really can’t believe someone who has made a career out of automating things wouldn’t be immediately drawn to trying them at least. Every single company seems convinced and using it too. The comparison to crypto makes no sense.
> Technology moves forward and productivity improves for those that move with it.
It does not, technology regresses just as often and linear deterministic progress is just a myth to begin with. There is no guarantee for technology to move forward and always make things better.
There are plenty of examples to be made where technology has made certain things worse.
Paul Krugman (Nobel laureate in economy) said in 1998 that the internet is no biggie. Many companies needed convincing to adopt the internet (heck, some still need convincing).
Would you say the same thing ("If it helped as much as its claimed, there wouldn't be any need to try and convince people they should be using it.") about the internet?
> Certainly if we ever ask a hallucination engine for a numeric answer, we should ask it at least three times, so we get some sense of the variation.
This works on people as well!
Cops do this when interrogating. You tell the same story three times, sometimes backwards. It's hard to keep track of everything if you're lying or you don't recall clearly so you can get a sense of confidence. Also works on interviews, ask them to explain a subject in three different ways to see if they truly understand.
Only within certain conditions or thresholds that we're still figuring out. There are many cases where the more someone recalls and communicates their memory, the more details get corrupted.
> Cops do this when interrogating.
Sometimes that's not to "get sense of the variation" but to deliberately encourage a contradiction to pounce upon it. Ask me my own birthday enough times in enough ways and formats, and eventually I'll say something incorrect.
Care must also be taken to ensure that the questioner doesn't change the details, such as by encouraging (or sometimes forcing) the witness/suspect to imagine things which didn't happen.
Triple modular redundancy. I remember reading that's how Nasa space shuttles calculate things because a processor / memory might have been affected by space radiation https://llis.nasa.gov/lesson/18803
For a hot second I thought LLMs were coming for our jobs. Then I realized they were just as likely to end up creating mountains of things for us to fix later. And as things settle down, I find good use cases for Claude Code that augment me but are in no danger of replacing me. It certainly has its moments.
I get a lot of productivity out of LLMs so far, which for me is a simple good sign. I can get a lot done in a shorter time and it's not just using them as autocomplete. There is this nagging doubt that there's some debt to pay one day when it has too loose a leash, but LLMs aren't alone in that problem.
One thing I've done with some success is use a Test Driven Development methodology with Claude Sonnet (or recently GPT-5). Moving forward the feature in discrete steps with initial tests and within the red/green loop. I don't see a lot written or discussed about that approach so far, but then reading Martin's article made me realize that the people most proficient with TDD are not really in the Venn Diagram intersection of those wanting to throw themselves wholeheartedly into using LLMs to agent code. The 'super clippy' autocomplete is not the interesting way to use them, it's with multiple agents and prompt techniques at different abstraction levels - that's where you can really cook with gas. Many TDD experts have great pride in the art of code, communicating like a human and holding the abstractions in their head, so we might not get good guidance from the same set of people who helped us before. I think there's a nice green field of 'how to write software' lessons with these tools coming up, with many caution stories and lessons being learnt right now.
It feels like Tdd/llm connection is implied — “and also generate tests”. Thought it’s not cannonical tdd of course. I wonder if it’ll turn the tide towards tech that’s easier to test automatically, like maybe ssr instead of react.
Yep, it's great for generating tests and so much of that is boilerplate that it feels great value. As a super lazy developer it's great as the burden of all that mechanical 'stuff' being spat out is nice. Test code being like baggage feels lighter when it's just churned out as part of the process, as in no guilt just to delete it all when what you want to do changes. That in itself is nice. Plus of course MCP things (Playwright etc) for integration things is great.
But like you said, it was meant more TDD as 'test first' - so a sort of 'prompt-as-spec' that then produces the test/spec code first, and then go iterate on that. The code design itself is different as influenced by how it is prompted to be testable. So rather than go 'prompt -> code' it's more an in-between stage of prompting the test initially and then evolve, making sure the agent is part of the game of only writing testable code and automating the 'gate' of passes before expanding something. 'prompt -> spec -> code' repeat loop until shipped.
The only thing I dislike is what it chooses to test when asked to just "generate tests for X": it often chooses to build those "straitjacket for your code" style tests which aren't actually useful in terms of catching bugs, they just act as "any change now makes this red"
As a simple example, a "buildUrl" style function that put one particular host for prod and a different host for staging (for an "environment" argument) had that argument "tested" by exactly comparing the entire functions return string, encoding all the extra functionality into it (that was tested earlier anyway).
A better output would be to check startsWith(prodHost) or similar, which is what I changed it into, but I'm still trying to work out how to get coding agents to do that in the first or second attempt.
But that's also not surprising: people write those kinds of too-narrow not-useful tests all the time, the codebase I work on is littered with them!
LLMs (Sonnet, Gemini from what I tested) tend to “fix” failing tests by either removing them outright or tweaking the assertions just enough to make them pass. The opposite happens too - sometimes they change the actual logic when what really needs updating is the test.
In short, LLMs often get confused about where the problem lies: the code under test or the test itself. And no amount of context engineering seems to solve that.
I think in part the issue is that the LLM does not have enough context. The difference between a bug in the test or a bug in the implementation is purely based on the requirements which are often not in the source code and stored somewhere else(ticket system, documentation platform).
Without providing the actual feature requirements to the LLM(or the developer) it is impossible to determine which is wrong.
Which is why I think it is also sort of stupid by having the LLM generate tests by just giving it access to the implementation. That is at best testing the implementation as it is, but tests should be based on the requirements.
Oh, absolutely, context matters a lot. But the thing is, they still fail even with solid context.
Before I let an agent touch code, I spell out the issue/feature and have it write two markdown files - strategy.md and progress.md (with the execution order of changes) inside a feat_{id} directory. Once I’m happy with those, I wipe the context and start fresh: feed it the original feature definition + the docs, then tell it to implement by pulling in the right source code context. So by the time any code gets touched, there’s already ~80k tokens in play. And yet, the same confusion frequently happens.
Even if I flat out say “the issue is in the test/logic,”, even if I point out _exactly_ what the issue is, it just apologizes and loops.
At that point I stop it, make it record the failure in the markdown doc, reset context, and let it reload the feature plus the previous agent’s failure. Occasionally that works, but usually once it’s in that state, I have to step in and do it myself.
> hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful.
I'd rather say that LLMs live in a world that consists entirely of stories, nothing but words and their combinations. Thy have no other reality. So they are good at generating more stories that would sit well with the stories they already know. But the stories are often imprecise, and sometimes contradictory, so they have to guess. Also, LLMs don't know how to count, but they know that two usually follows one, and three is usually said to be larger than two, so they can speak in a way that mostly does not contradict this knowledge. They can use tools to count, like a human who knows digits would use a calculator.
But much more than an arithmetic engine, the current crop of AI needs an epistemic engine, something that would help follow logic and avoid contradictions, to determine what is a well-established fact, and what is a shaky conjecture. Then we might start trusting the AI.
this was true, but then it wasn't... the research world several years ago, had a moment when the machinery could reliably solve multi-step problems.. there had to be intermediary results; and machinery could solve problems in a domain where they were not trained specifically.. this caused a lot of excitement, and several hundred billion dollars in various investments.. Since no one actually knows how all of it works, not even the builders, here we are.
This is a misconception, we absolutely do know how LLMs work, that's how we can write them and publish research papers.
The idea we don't is tabloid journalism, it's simply because the output is (usually) randomised - taken to mean, by those who lack the technical chops, that programmers "don't know how it works" because the output is indeterministic.
This is not withstanding we absolutely can repeat the output by using not randomisation (temperature 0).
This vaguely relates to a theory about human thought: that our subconscious constantly comes up with random ideas, then filters the unreasonable ones, but in people with delusions (e.g. schizophrenia) the filter is broken.
Salience (https://en.wikipedia.org/wiki/Salience_(neuroscience)), "the property by which some thing stands out", is something LLMs have trouble with. Probably because they're trained on human text, which ranges from accurate descriptions of reality to nonsense.
More of a error-correcting feedback loop rather than a filter, really. Which is very much what we do as humans, apparently. One recent theory of neuroscience that is becoming influential is Predictive Processing --https://en.wikipedia.org/wiki/Predictive_coding -- this postulates that we also constantly generate a "mental model" of our environment (a literal "prediction") and use sensory inputs to correct and update it.
So the only real difference between "perception" and a "hallucination" is whether it is supported by physical reality.
You ever see something out of the corner of your eye and you were mistaken? It feels like LLM hallucinations. "An orange cat on my counter?!" And an instant later, your brain has reclassified "a basketball on my counter" as that fits the environment model better as several instant-observations gather contexts: not moving, more round, not furry, I don't own a cat, yesterday my kid mentioned something about tryouts, boop insta-reclassification from cat to basketball.
I can recognize my own meta cognition there. My model of reality course corrects the information feed interpretation on the fly. Optical illusions feel very similar whereby the inner reality model clashes with the observed.
For general ai, it needs a world model that can be tested against and surprise is noted and models are updated. Looping llm output with test cases is a crude approximation of that world model.
I leaned heavily on my own meta cognition recently when withdrawing from a bereavment benzo habit recently. The combo of flu symptoms, anxiety and hallucinations are fierce. I knew O was seeing things that were not real; light fittings turning into a rotating stained glass slideshow. So I'm totally on board with the visual model hypothesis. My own speculation is that audio perception is less predictive as audio structure persists deeper into my own DMT sessions than does vision, where perspective quickly collapses and vision becomes kaleidoscopic. Which may be a return to the vision we had as newborns. Maybe normal vision is only attained via socialisation?
No, that's exactly what an agent isn't. What makes an agent an agent is all the not-LLM code. When an agent generates Golang code, it runs the Go compiler, which is in the agent's architecture an extension of the agent. The Go compiler does not hallucinate.
I have a very similar (probably unoriginal) thought about some human mental illnesses.
So, we VALUE creativity, we claim that it helps us solve problems, improves our understanding of the universe, etc.
BUT people with some mental illnesses, their brain is so creative that they lose the understanding of where reality is and where their imagination/creativity takes over.
eg. Hearing voices? That's the brain conjuring up a voice - auditory and visual hallucinations are the easy example.
But it goes further, depression is where people's brains create scenarios where there is no hope, and there's no escape. Anxiety too, the brain is conjuring up fears of what's to come
Nah I don't agree with this characterization. The problem is, the majority of those hallucinations are true. What was said would make more sense if the majority of the responses were, in fact, false, but this is not the case.
I think you're both correct but have different definitions of hallucinations. You're judging it as a hallucination based on the veracity of the output. Whereas Fowler is judging it based on the method by which the output is achieved. By that judgement, everything is a hallucination because the user cannot differentiate between when the LLM is telling the truth and isn't.
This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.
> In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called confabulation,[1] or delusion)[2] is a response generated by AI that contains false or misleading information presented as fact.[3][4]
You say
> This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.
For consistency you might as well say everything the human mind does is hallucination. It's the same sort of claim. This claim at least has the virtue of being taken seriously by people like Descartes.
They're using the term 'bullshit' as it is understood as a term of art, which doesn't imply lying. It's closer to creating a response without any regard for telling the truth. Bullshitting is often most effective when you happen to be telling the truth, although the bullshitter has no commitment to that.
> We should ask the LLM the question more than once
For any macOS users, I highly recommend an Alfred workflow so you just press command + space then type 'llm <prompt>' and it opens tabs with the prompt in perplexity, (locally running) deepseek, chatgpt, claude and grok, or whatever other LLMs you want to add.
This approach satisfies Fowler's recommendation of cross referencing LLM responses, but is also very efficient and over time gives you a sense of which LLMs perform better for certain tasks.
What would such a workflow look like? I have Alfred but mainly just use the clipboard feature. I've tried to get into automation but struggled for inspiration. This one seems good!
Modify to your taste. Example: https://github.com/stevecondylios/alfred-workflows/tree/main (you should be able to download the .alfredworkflow file and double click on it to import it straight into alfred, but creating your own shouldn't take long, maybe 5-10 mins if it's your first workflow)
There are many I've worked with that idolize Martin Fowler and have treated his words as gospel. That is not me and I've found it to be a nuisance, sometimes leading me to be overly critical of the actual content. As for now, I'm not working with such people and can appreciate the article shared without clouded bias.
I like this article, I generally agree with it. I think the take is good. However, after spending ridiculous amounts of time with LLMs (prompt engineering, writing tokenizers/samplers, context engineering, and... Yes... Vibe coding) for some periods 10 hour days into weekends, I have come to believe that many are a bit off the mark. This article is refreshing, but I disagree that people talking about the future are talking "from another orifice".
I won't dare say I know what the future looks like, but the present very much appears to be an overall upskilling and rework of collaboration. Just like every attempt before, some things are right and some are simply misguided. e.g. Agile for the sake of agile isn't any more efficient than any other process.
We are headed in a direction where written code is no longer a time sink. Juniors can onboard faster and more independently with LLMs, while seniors can shift their focus to a higher level in application stacks. LLMs have the ability to lighten cognitive loads and increase productivity, but just like any other productivity enhancing tool doing more isn't necessarily always better. LLMs make it very easy to create and if all you do is create [code], you'll create your own personal mess.
When I was using LLMs effectively, I found myself focusing more on higher level goals with code being less of a time sink. In the process I found myself spending more time laying out documentation and context than I did on the actual code itself. I spent some days purely on documentation and health systems to keep all content in check.
I know my comment is a bit sparse on specifics, I'm happy to engage and share details for those with questions.
Written code is an auditable entity in regulated businesses, like banks. Fowler suggests we get more comfortable with uncertainty because other eng domains have developed good risk mgmt. Cart before horse. Other engineering domains strive to mitigate the irreducible uncertainty of the physical world. AI adds uncertainty to any system. And there are many applications where increased uncertainty will lead to an increase in human suffering.
It still is, and should be. It’s highly unlikely that you provided all the required info to the agent at first try. The only way to fix that is to read and understand the code thoroughly and suspiciously, and reshaping it until we’re sure it reflects the requirements as we understand them.
Vibe coding is not telling an agent what to do and checking back. It's an active engagement and best results are achieved when everything is planned and laid out in advance — which can also be done via vibe coding.
No, written code is no longer a time sink. Vibe coding is >90% building without writing any code.
The written code and actions are literally presented in diffs as they are applied, if one so chooses.
> It's an active engagement and best results are achieved when everything is planned and laid out in advance
The most efficient way to communicate these plans is in code. English is horrible in comparison.
When you’re using an agent and not reviewing every line of code, you’re offloading thinking to the AI. Which is fine in some scenarios, but often not what people would call high quality software.
Writing code was never the slow part for a competent dev. Agent swarming etc is mostly snake oil by those who profit off LLMs.
ime this is the problem. When I have to deeply understand what an llm created, I don't see much of a speed improvement vs writing it myself.
With an engineer you can hand off work and trust that it works, whereas I find code reviewing llm output something that I have to treat as hostile. It will comment out auth or delete failing tests.
> It's an active engagement and best results are achieved when everything is planned and laid out in advance — which can also be done via vibe coding.
No.
The general assumed definition of vibe coding, hence the vibe word, is that coding becomes an iterative process guided by intuition rather than spec and processes.
What you describe is literally the opposite of vibe coding, it feels the term is being warped into "coding with an LLM".
Have a documentation system in place and have the LLM plan the high level before having the LLM write any code.
You can always just wing it, but if you do so and there isn't adequate existing context you're going to struggle with slop and hallucinations more frequently.
> We are headed in a direction where written code is no longer a time sink.
Written code has never been a time sink. The actual time that software developers have spent actually writing code has always been a very low percentage of total time.
Figuring out what code to write is a bigger deal. LLMs can help with part of this. Figuring out what's wrong with written code, and figuring out how to change and fix the code, is also a big deal. LLMs can help with a smaller part of this.
> Juniors can onboard faster and more independently with LLMs,
Color me very, very skeptical of this. Juniors previously spent a lot more of their time writing code, and they don't have to do that anymore. On the other hand, that's how they became not-juniors; the feedback loop from writing code and seeing what happened as a result is the point. Skipping part of that breaks the loop. "What the computer wrote didn't work" or "what the computer wrote is too slow" or even to some extent "what the computer wrote was the wrong thing" is so much harder to learn from.
Juniors are screwed.
> LLMs have the ability to lighten cognitive loads and increase productivity,
I'm fascinated to find out where this is true and where it's false. I think it'll be very unevenly distributed. I've seen a lot of silver bullets fired and disintegrate mid-flight, and I'm very doubtful of the latest one in the form of LLMs. I'm guessing LLMs will ratchet forward part of the software world, will remove support for other parts that will fall back, and it'll take us way too long to recognize which part is which and how to build a new system atop the shifted foundation.
> Figuring out what code to write is a bigger deal. LLMs can help with part of this. Figuring out what's wrong with written code, and figuring out how to change and fix the code, is also a big deal. LLMs can help with a smaller part of this.
I found exactly this is what LLMs are great at assisting with.
But, it also requires context to have guiding points for documentation. The starting context has to contain just enough overview with points to expand context as needed. Many projects lack such documentation refinement, which causes major gaps in LLM tooling (thus reducing efficacy and increasing unwanted hallucinations).
> Juniors are screwed.
Mixed, it's like saying "if you start with Python, you're going to miss lower level fundamentals" which is true in some regards. Juniors don't inherently have to know the inner workings, they get to skip a lot of the steps. It won't inherently make them worse off, but it does change the learning process a lot. I'd refute this by saying I somewhat naively wrote a tokenizer, because the >3MB ONNX tokenizer for Gemma written in JS seemed absurd. I went in not knowing what I didn't know and was able to learn what I didn't know through the process of building with an LLM. In other words, I learned hands on, at a faster pace, with less struggle. This is pretty valuable and will create more paths for juniors to learn.
Sure, we may see many lacking fundamentals, but I suppose that isn't so different from the criticism I heard when I wrote most of my first web software in PHP. I do believe we'll see a lot more Python and linguistic influenced development in the future.
> I'm guessing LLMs will ratchet forward part of the software world, will remove support for other parts that will fall back, and it'll take us way too long to recognize which part is which and how to build a new system atop the shifted foundation.
I entirely agree, in fact I think we're seeing it already. There is so much that's hyped and built around rough ideas that's glaringly inefficient. But FWIW inefficiency has less of an impact than adoption and interest. I could complain all day about the horrible design issues of languages and software that I actually like and use. I'd wager this will be no different. Thankfully, such progress in practice creates more opportunities for improvement and involvement.
> Sure, we may see many lacking fundamentals, but I suppose that isn't so different from the criticism I heard when I wrote most of my first web software in PHP.
It's not just the fundamentals, though you're right that is an easy casualty. I also agree that LLMs can greatly help with some forms of learning -- previously, you kind of had to follow the incremental path, where you couldn't really do anything complex without have the skills that it built on, because 90% of your time and brain would be spent on getting the syntax right or whatever and so you'd lose track of the higher-level thing you were exploring. With an LLM, it's nice to be able to (temporarily) skip that learning and be able to explore different areas at will. Especially when that motivates the desire to now go back and learn the basics.
But my real fear is about the skill acquisition, or simply the thinking. We are human, we don't want to have to go through the learning stage before we start doing, and we won't if we don't have to. It's difficult, it takes effort, it requires making mistakes and being unhappy about them, unhappy enough to be motivated to learn how to not make them in the future. If we don't have to do it, we won't, even if we logically know that we'd be better off.
Especially if the expectations are raised to the point where the pressure to be "productive" makes it feel like you're wasting other people's time and your paycheck to learn anything that the LLM can do for you. We're reaching the point where it feels irresponsible to learn.
(Sometimes this is ok. I'm fairly bad at long division now, but I don't think it's holding me back. But juniors can't know what they need to know before they know it!)
> But my real fear is about the skill acquisition, or simply the thinking. We are human, we don't want to have to go through the learning stage before we start doing, and we won't if we don't have to. It's difficult, it takes effort, it requires making mistakes and being unhappy about them, unhappy enough to be motivated to learn how to not make them in the future. If we don't have to do it, we won't, even if we logically know that we'd be better off.
I've noticed the effects of this first hand from intense LLM engagement.
I relate it more to the effects of portable calculators, navigation systems, and tools like Wikipedia. I'm under the impression this is valid criticism, but we may be overly concerned because it's new and a powerful tool. There's even surveys/studies showing differences in how LLM are perceived WRT productivity between different generations.
I'm more concerned with potential loss of critical thinking skills, more than anything else. And on a related note, there have been concerns of critical thinking skills before this mass adoption of LLMs. I'm also concerned with the impact of LLMs on the quality of information. We're seeing a huge jump in quantity while some quality lacks. It bothers me when I see an LLM confidently presenting incorrect information that's seemingly trivial to validate. I've had web searches give me more incorrect information from LLM tooling at a much greater frequency than I've ever experienced before. It's even more unsettling when the LLM gives the wrong answer and the correct answer is in the description of the top result.
"I'm also concerned with the impact of LLMs on the quality of information."
You have finally made an astute observation...
I have already made the assumption that use of LLMs is going to add new mounds of BS atop the mass of crap that already exists on the internet, as part of my startup thesis.
These things are not obvious in the here and now, but I try to take the view of - how would the present day look, 50 years out in the future looking backwards?
The issue is a lot of times it takes a senior level programmer to reason with a LLM to get the results needed. What happens when there are no Juniors to replace the Seniors. I guess by then AI will be able to program efficiently enough.
So far AI seems to be a great augmentation, but not a replacement.
> we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers,
Yup, this matches my recommended workflow exactly. Why waste time trying to turn an initially bad answer into a passable one, when you could simply re-generate (possibly with different context)
> One of the consequences of this is that we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers, indeed perhaps ask the LLM to compare answers for us. The difference in the answers can be as useful as the answers themselves.
There was once a coding agent which achieved SOTA performance on SWE Bench Verified by "just" running the agent 5 times on each instance, scoring each attempt and picking the attempt with the highest score: https://aide.dev/blog/sota-bitter-lesson
> One of the consequences of this is that we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers, indeed perhaps ask the LLM to compare answers for us. The difference in the answers can be as useful as the answers themselves.
This is what LLM "reasoning" does. More than "reasoning" in the human sense, it just reduces variance from variations in the prompt and random next token prediction.
The conclusion I reached was different, though. We learnt how to do outsourcing "properly" pretty quickly after some initial high-profile failures, which is why it has only continued to grow into such a huge industry. This also involved local talent refocusing on higher-value tasks, which is why job losses were limited. Those same lessons and outcomes of outsourcing are very relevant to "bot-sourcing'.
However, I do feel concerned that AI is gaining skill-levels much faster than the rate at which people can upskill themselves.
> Prompting (overseas) humans [...] After over a decade of trying that, we learned that had... flaws.
I think this is a US-centric point of view, and seems (though I hope it's not!) slightly condescending to those of us not in the US.
Software engineering is more than what happens to US-based businesses and their leadership commanding hundreds or thousands of overseas humans. Offshoring in software is certainly a US concern (and to a lesser extent, other nations suffer it), but is NOT a universal problem of software engineering. Software engineering happens in multiple countries, and while the big money is in the US, that's not all there is to it.
Software engineering exists "natively" in countries other than the US, so any problems with it should probably (also) be framed without exclusive reference to the US.
Talking to LLMs reminds me of my time as a system administrator for my university's physics department. A bunch of incredibly smart, PhD level people...just not in the subject they were talking to me about, most of the time.
To me, it is more specific to say that many futurists don't take into account the social and economic network effects of changes taking place. Many just act as if the future will continue on completely unchallenged in this current state. But if you look at someone like Kurzweil, you can see the very narrow and specific focus of a prediction, which has proved to be more informative to me as a high bar of futurism.
Part of what got me into software was this: no matter how complex or impressive the operation, with enough time and determination, you could trace each step and learn how a tap on a joystick lead to the specific pixels on a screen changing.
There’s a beautiful invitation to learn and contribute baked into a world where each command is fully deterministic and spec-ed out.
Yes, there have always been poorly documented black boxes, but I thought the goal was to minimize those.
People don’t understand how much is going to be lost if that goal is abandoned.
Agreed. The beauty of programming is that you're creating a "mathematical artifact." You can always drill down and figure out exactly what is going on and what is going to happen with a given set of inputs. Now with things like concurrency that's not exactly true, but, I think the sentiment still holds.
The more practical question is though, does that matter? Maybe not.
Yep, LLM’s still run on hardware with the same fundamental architecture we’ve had for years, sitting under a standard operating system. The software is written in the same languages we’ve been using for a long time. Unless there’s some seismic shift where all of these layers go away we’ll be maintaining these systems for a while.
Yes, but : not in all cases really. There are plenty of throw away, one off, experimental, trial and error, low stakes, spammy, interactions with dumb-assery, manager replacing instances that are acceptable as being quickly created and forgotten black boxes of temporary barely working trash. (not that people will limit to the proper uses)
In a way this is also a mathematical artifact — after all tokens are selected through beam searching or some random sampling of likely successor tokens.
As pertaining to software development, I agree. I've been hearing accounting (online and from coworkers) of using LLMs to do deterministic stuff. And yet, instead of at least prompting once to "write a script to do X," they just keep prompting "do X" over and over again. Seems incredibly wasteful. It feels like there is this thought of "We are not making progress if we aren't getting the LLM to do everything. Having it write a script we can review and tweak is anti-progress." No one has said that outright, but it's a gut feeling (and it wouldn't surprise me if people have said this out loud).
These are hardly equivalent. One is someone preferring one deterministic way over another. The other is more akin to arguing that it's better to ask someone to manually complete a task for you instead of caching the instructions on your computer. Now if the LLM does caching then you have more of a point, I don't have enough experience there.
I am one of these people. For my scripting needs, it would probably take me longer to find the script I saved rather than just asking it again and getting an answer in 15 seconds. I haven't saved a smallish script in a year.
I used to be one of those people, then I started saving these scripts in a folder and realized just how much time it saved me. Especially for nontrivial scripts which require a lot of typing or input specification.
This was me before LLMs. Had tons of scripts saved. I just don't see the point anymore "Take this list of users, remove from x group and reset their passwords" takes less time than trying to remember what I named that file and where it's saved. Anything that can be done in under 100 lines isn't worth saving anymore.
That was an interesting take, but "probabilistic" is to me different from "random". In particular other field get error tolerances, LLMs give us nothing like that.
We're introducing chaos monkeys, not just variability.
> Process engineers for example have to account for human error rates. [...] Designing systems to detect these errors (which are highly probabilistic!)
> Likewise even for regular mechanical engineers, there are probabilistic variances in manufacturing tolerances.
I read them as relatively confined, thus probabilistic. When a human pushes the wrong button, an elephant isn't raining from the sky. Same way tolerances are bounded.
When requesting a JSON array from an LLM, it could as well decide this time that JSON is a mythological Greek fighter.
adding to this, software deals with non-determinism all the time.
For example, web requests are non-deterministic. They depend, among other things, on the state of the network. They also depend on the load of the machine serving the request.
One way to think about this is: how easy is it for you to produce byte-for-byte deterministic builds of the software you're working on? If it's not trivial there's more non-determinism than is obvious.
Mostly the engineering part of software is dealing with non-determinism, by avoiding it or enforcing determinism. Take something like TCP, it's all about guaranteeing the determinism that either the message is sent and received or it is not. And we have a lot of algorithms that tries to guarantee consistency of information between the elements of a system.
But there is an underlying deterministic property in the TCP example. A message is either received within a timeout or not.
How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not? Has the halting problem been solved?
> How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not?
That's the catch 22 with LLM. You're supposed to be both the asker and the verifier. Which in practice, it's not that great. LLMs will just find the snippets of code that matches somehow and just act on it (It's the "I'm feeling Lucky" button with extra steps)
In traditional programming, coding is a notation too more than anything. You supposed to have a solution before coding, but because of how the human brain works, it's more like a blackboard, aka an helper for thinking. You write what you think is correct, verify your assumptions, then store and forget about all of it when that's true. Once in a while, you revisit the design and make it more elegant (at least you hope you're allowed to).
LLM programming, when first started, was more about a direct english to finished code translation. Now, hope has scaled down and it's more about precise specs to diff proposal. Which frankly does not improve productivity as you can either have a generator that's faster and more precise (less costly too) or you will need to read the same amount of docs to verify everything as you would need to do to code the stuff in the first place (80% of the time spent coding).
So no determinism with LLMs. The input does not have any formal aspects, and the output is randomly determined. And the domain is very large. It is like trying to find a specific grain of sand on a beach while not fully sure it's there. I suspect most people are doing the equivalent of taking a handful of sand and saying that's what they wanted all along.
No? These kinds of analyses all seem to rely on the notion that the LLM-caller needs to accept whatever output the LLM provides. In practice, they discard all the outputs that don't compile, and then a further subset of the ones that don't --- those outputs that aren't instantly clear to the caller.
My intuition for the problem here is that people are fixated on the nondeterminism of the LLM itself, which is of limited importance to the actual problem domain of code generation. The LLM might spit out ancient Egyptian hieroglyphics! It's true! The LLM is completely nondeterministic. But nothing like that is ever going to get merged into `main`.
It's fine if you want to go on about how bad "vibe coding" is, with LLM-callers that don't bother to read LLM output, because they're not competent. But here we're assuming an otherwise competent developer. You can say the vibe coder is the more important phenomenon, but the viber doesn't implicate the halting problem.
Valid programs are almost infinite. Context free grammars (which describe valid programs) are generative. When you're programming, you are mostly restricting the set of valid program to include only the few that satisfy the specs. Adding an extra 0 to a number is valid, but put that in the context of money transactions, it's a "hell breaks loose" situation.
SO that's why "it compiles" is worthless in a business settings. Of course it should compile. That's the bare minimum of expectations. And even "it passes the tests" is not that great. That just means you have not mess things up. So review and quality (accountability for both) is paramount, so that the proper stuff get shipped (and fixed swiftly if there was a mistake).
> Having the code compile means everything type checks and is not worthless. That's a huge constraint on valid programs.
In a business settings, what usually matter is getting something into prod and not have bug reports thrown back. And a maintainable code that is not riddled down with debts.
Compiled code is as basic as steering left and right for a F1 driver. Having the tests pass is like driving the car at slow speed and completing a lap with no other cars around. If you're struggling to do that, then you're still in the beginner phase and not a professional. The real deal is getting a change request from Product and and getting it to Production.
It feels like you stopped reading before "and then a further subset of those".
Again: my claim is simply that whatever else is going on, the halting problem doesn't enter into it, because the user in this scenario isn't obligated to prove arbitrary programs. Here, I can solve the halting problem right now: "only accept branchless programs with finite numbers of instructions". Where's my Field Medal? :)
It always feels like the "LLMs are nondeterministic" people are relying on the claim that it's impossible to tell whether an arbitrary program is branchless and finite. Obviously, no, that's not true.
No, I'd have indicated that in my comment if I had. Sorry, I think you just missed it.
I did add the last paragraph of the comment you just responded to (the one immediately above this) about 5 seconds after I submitted it, though. Doesn't change the thread.
> LLMs will just find the snippets of code that matches somehow
This suggests a huge gap in your understanding of LLMs if we are to take this literally.
> LLM programming, when first started, was more about a direct english to finished code translation
There is no direct english to finished code translation. A prompt like "write me a todo app" has infinitely many codes it maps to with different tradeoffs and which appeal to different people. Even if LLMs never made any coding mistakes, there is no function that maps a statement like that to specific pieces of code unless you're making completely arbitrary choices like the axiom of choice.
So we're left with the fact that we have to specify what we want. And at that LLMs do exceptionally well.
It’s not trivial largely because we didn’t bother to design deterministic builds because it didn’t seem to matter. There is not much about the actual problem that makes it difficult.
Yes, I spent years working on exactly that. I personally was working with a compiler team to validate changes to eliminate non-determinism. That’s why I felt qualified to make the statement I did.
It’s non trivial because you have to back through decades of tools and fix all of them to remove non determinism because they weren’t designed with that in mind.
The hardest part was ensuring build environment uniformity but that’s a lot easier with docker and similar tooling.
I would rather say it like this: Very good, very hardworking engineers spent years of their lives building the machine that raised us from the non-determinism of messy physical reality. The technology that brought us perfect, replicable, and reliable math from sending electrons through rocks has been deeply underappreciated in the "software revolution".
The engineers at TSMC, Intel, Global Foundries, Samsung, and others have done us an amazing service, and we are throwing all that hard work away.
It's uncertainty, not non-determinism. I don't design a rocket with no idea about the chain of events. I have inscrutable uncertainty everywhere, from manufacture to flight.
More seriously, this is not a fair comparison. Adding LLM output to your source code is not analogical to quantum physics; it is analogical to letting your 5 years old child transcribe the experimentally measured values without checking and accepting that many of them will be transcribed wrong.
> I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?
A junior engineer can't write code anywhere nearly as fast. It's apples vs oranges. I can have the LLm rewrite the code 10 times until its correct and its much cheaper than hiring an obsequious jr engineer
Junior engineers get better and learn from their mistakes. An LLM will happily make then same mistake 10 times if you try asking it to do something similar in 3 months. In the long term, I don't think LLMs actually save you time considering the amount of extra time spent reviewing/verifying its code and fixing tech debt.
>I’m often asked, “what is the future of programming?” Should people consider entering software development now? Will LLMs eliminate the need for junior engineers? Should senior engineers get out of the profession before it’s too late? My answer to all these questions is “I haven’t the foggiest”
I just want to point out that this answer implicitly means that, at the very least, the profession is at least questionably uncertain which isn't a good sign for people with a long future orientation such as students.
Students shouldn't ever have become so laser-focused on a single career path anyway, and even worse than that is how colleges have become glorified trade schools in our minds. Students should focus on studying for their classes, getting their electives and extracurriculars in, getting into clubs... Then depending on which circles they end up in, they shape their career that way. The thought that getting a Computer Science major would guarantee students a spot in the tech industry was always ridiculous, because the industry was just never structured that way, it's always been hacker groups, study groups, open source, etc. bringing out the best minds
It has never been anywhere close to certain, we just had 20 years of wild, unsustainable growth that encouraged people to cover their eyes and pretend the ride would go on forever. 20 years of telling everyone under the age of 30 that of course they should learn to code and that CS was the new medical or legal degree. 20 years of smugly acting like we are the inevitable future when we are, in fact, subject to the same ups and downs as every other career.
The answers are even kind of contradictory. If the answer to "will we ever need software engineers again in the future?" is "we don't know" then the answer to "should I spend time and money to enter software engineering?" should be "no".
I like the idea of AI usage comes down to a measurement of "tolerances". With enough specificity, LLMs will 100% return what you want. The goal is to find the happy tolerance between "acceptable" and "I did it myself" via prompts.
> I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?
Reminds me of a recent experience when I asked CC to implement a feature. It wrote some code that struck me as potentially problematic. When I said, "why did you do X? couldn't that be problematic?" it responded with "correct; that approach is not recommended because of Y; I'll fix it". So then why did it do it in the first place? A human dev might have made the same mistake, but it wouldn't have made the mistake knowing that it was making a mistake.
I think this gets lost on far too many discussions. There is no “why” - it’s statistics, and the closest we have to a “why” is that a lot of code out there sucks
> One of the big problems with these surveys is that they aren’t taking into account how people are using the LLMs. From what I can tell the vast majority of LLM usage is fancy auto-complete, often using co-pilot.
This is a completely wrong assumption and negates a bunch of the points of the article...
I would expect it to be almost tautologically true. It's the easiest to implement, and the most deployed. It has to be the most used, even if a lot of people don't know they're using it.
It was the original success of LLMs -- code autocomplete. All of the agentic coding have autocomplete under the hood. Autocomplete is almost literally what LLMs do + post-processing.
> All major technological advances have come with economic bubbles, from canals and railroads to the internet.
Is this actually correct? I don't see any evidence for a "airflight bubble" or a "car bubble" or a "loom bubble" at the technologies' invention. Also the "canal bubble" wasn't about the technology, it was about the speculation on a series of big canals but we had been making canals for a long time. More importantly, even if it was correct, there are plenty of bubbles (if not significantly more) around things that didn't have value or tech that didn't matter.
Apologies for arguing from first principles, but for anything that spurs a lot of activity, the only alternative to a bubble is this: people ramp up investment and activity and enthusiasm only as much as the underlying thing can handle, then gradually taper off the increase and gently level off at the equilibrium "carrying capacity" of the new technology.
Does that sound like any human, ever, to you?
(The only time there isn't a bubble is when the thing just isn't that interesting to people and so there's never a big wave of uptake in the first place.)
I don't know enough about the early history of the airline industry but there was very definitely a long series of huge bubbles in the railroad industry.
I would be very interested in reading about huge bubbles in railroad, airline, or any other industry. Do you happen to have references (genuinely asking; the original article didn't include any references)?
There was just a long Derek Thompson podcast with a Transcontinental Railroad scholar about this! (That's why I knew about it.)
The whole subtext of that podcast was how eerily similar the Transcontinental Railroad was to AI (as an investment/malinvestment/prediction of future trends).
The Intelligent Investor by Graham talks about investors putting so much money into airlines and air freight that it became impossible to make a return. I don't know if you would call that a bubble, maybe just over exuberance.
A bubble is asset prices systematically diverging from reasonable expectations of future cash flows. Bubbles are driven by financial speculation.
The claim in the blog post that all technology leads to speculative asset bubbles I find hard to believe. Where was the electricity bubble? The steel bubble? The pre-war aviation bubble? (The aviation bubble appeared decades later due to changes in government regulation.)
Is this an AI bubble? I genuinely don't know! There is a lot of real uncertainty about future cash flows. Uncertainty is not the foundation of a bubble.
I knew dot-com was a bubble because you could find evidence, even before it popped. (A famous case: a company held equity in a bubble asset, and that company had a market cap below the equity it held, because the bubble did not extend to second-order investments.)
Just taking your first example, yes I believe you could characterize a lot of the investments in electrification as speculative bubbles even though there was underlying value that was borne out.
To me a bubble reflects a market disconnect from fundamentals - wherein prices go up steeply, with no help from the fundamentals (expected growth in base year cash flows and risk).
Indeed there is a subtle difference between a bubble and speculation of what could come of a technology. But the two are connected because the effects of technology are reflected by investors in asset prices.
Besides electricity (answered above), there were bubbles related to steel production in its early days intertwined with the railroad bubble that drove huge investments in steel production for railroad development; when the railroad companies crashed it brought down steel as well; ultimately both survived in a more subdued form (as with the dot coms).
"Hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature".
I used to avidly read all his stuff, and I remember 20ish years ago he decided to rename Inversion of Control to Dependency Injection. In doing so, and his accompany blog, he showed he didn't actually understand it at a deep level (and hence his poor renaming).
This feels similar. I know what he's trying to say, but he's just wrong. He's trying to say the LLM is hallucinating everything, but Fowler is missing is that Hallucination in LLM terms refers to a very specific negative behavior.
That's the point. It is meaningless. When it first coined, there were already detractors to the term, that it is an incorrect description of the phenomenon. But it stuck.
No, it’s just an attempt to pretend wrong outputs are some special case when really they aren’t. They aren’t imagine something that doesn’t exist they are just running the same process they do for everything else and it just didn’t work.
If you disagree then I would ask what exactly is the “specific behaviour” you’re talking about?
> I’ve often heard, with decent reason, an LLM compared to a junior colleague.
No, they're like an extremely experienced and knowledgeable senior colleague – who drinks heavily on the job. Overconfident, forgetful, sloppy, easily distracted. But you can hire so many of them, so cheaply, and they don't get mad when you fire them!
These metaphors all suck. Well, ok, yours is funny. But anyway, LLMs are just very different from any human.
They are extremely shallow, even compared to a junior developer. But extremely broad, even compared to the most experienced developer. They type real fuckin fast compared to anyone on earth, but they need to be told what to do much more carefully than anyone on earth.
I've gotten Claude Code to make CUDA kernels and all kinds of advanced stuff that there's zero percent chance a junior would pull off.
AI is like a super advanced senior wearing a blindfold. It knows almost everything, it's super fast, and it gets confused pretty quickly about things after you tell it.
it's not a senior though, because of the amount of oversight and guidance required. I trust senior-plus human developers to do the right thing and understand why it's the right thing. For mission critical things I get another human senior to verify. There's no way I'd autonomously trust 2, 10 or any number of LLMs to do the same thing.
You'd be surprised at what juniors can pull off. I have seen fresh-out-of-college new grads write performant GPU kernels that are used in real world library implementations for particular architectures.
Most of the juniors I've worked with would make numerical errors and give up/declare premature victory before getting the implementation to a robust state. I'm sure there are exceptional young folks out there though.
And today I asked Claude code to carefully look at and follow the structure of my tests and write some more, and after 5-10 mins it completely ignored it, wrote one class with all the helper methods and a bunch of compilation errors from fields that arent even in my model.
I figured there's probably a ton more logical issues and deleted it immediately
I asked Claude to design me a UI, and it made a lovely one... but I wanted a web ui. It very happily through away all its work and made a brand new web UI.
I can't imagine any employee being that quick to just move on after something like that.
I think we had the analogy right with "fancy autocomplete". Sometimes, the completions are excellent and do exactly what we want, other times the completions fail to match our expectations and need human attention. It's a tool.
I just spent a good 2 hours trying to debug a SM6 Vulkan issue with unreal engine using an LLM, it had got me to good state but UE kept falling to load a project, it transpired that the specific error message would provide a fix as the top Google result, which I found when I eventually decided to look for myself.
LLM did help a lot to get some busy work out of the way, but it's difficult to know when you need to jump out of the LLM loop and go old skool.
Fwiw I think the ratio of times I needed to go to google for a solution instead of an LLM is like 20:1 for me so your mileage may vary. Depends a lot on the specific niche you're working in.
Unrelated to software but recently I wanted to revive an old dumbphone I haven't used since 2014 and apparently I had it password protected and forgot the password and wanted to factory reset it. I found the exact model of the phone and google had only content farm articles that didn't help me at all but Gemini gave me the perfect solution first try. I went to google first because I had no faith in Gemini since to me it seemed like a pretty obscure question but guess I was wrong.
In the interest of full disclosure my setup is quite esoteric for unreal dev. Linux and nixos no less. To be honest I'd probably have given up on nixos long ago without LLM support. It's actually really quite handy to be able to share a declarative specification of my environment.
No, they're like a completely nontechnical marketing person who has a big library of papers on related subjects and who's been asked to generate a whitepaper by pulling phrases. What comes out will probably have proper grammar and seem perfectly reasonable to another person with no knowledge of the field, but if actually read by someone knowledgeable may be complete gibberish taken as a whole.
Individual sentences and paragraphs may mostly work, but it's an edifice built on sand out of poorly constructed bricks plus mortar with the wrong proportions (or entirely wrong ingredients).
LLM output is "truthy" - it looks like it might be true, and sometimes it will even be accurate (see also, "stopped clock") but depending on it is foolish because what's generating the output doesn't actually understand what it's putting out - it's just generating output that looks like the kind of thing you've requested.
Overconfident, but also easy to sway their opinion.
They follow directions for maybe an hour and then go off and fix random shit because they forgot what their main task was.
They'll tell you to your face how great your ideas were, and that you're absolutely right about something, then go implement it completely incorrectly.
They add comments to literally everything even when you ask them to stop. But they also ignore said comments sometimes.
My analogy is a cursed monkey paw with unlimited wishes. It's actually really really powerful but you have to be careful of leaving any possible ambiguity in your wishes.
One good yardstick, if one has to anthropomorphize, is that LLMs know and believe what's popular. If you ask it to do something that popular developer opinion gets right, it will do fine.
Ask it for things that many people get wrong or just do badly, or can be mistakenly likened to a popular thing in a way that produces a wrong result, and it'll often err.
The trick is having an awareness of what correct solutions are prevalent in training data and what the bulk of accessible code used for training probably doesn't have many examples of. And this experience is hard to substitute for.
Juniors therefore use LLMs in a bumbling fashion and are productive either by sheer luck, or because they're more likely to ask for common things and so stay in a lane with the model.
A senior developer who develops a good intuition for when the tool is worth using and when not can be really efficient. Some senior developers however either overestimate or underestimate the tool based on wrong expectations and become really inefficient with them.
“Here’s the code:” looks totally plausible but hallucinated libFakeCall.
“libFakeCall doesn’t exist. Use libRealCall instead of libFakeCall.”
“You’re absolutely correct. I apologize for blah blah blah blah. Here’s the updated code with libRealCall instead. :[…]”
“You just replaced the libFakeCall reference with libRealCall but didn’t update the calls themselves. Re-write it and cite the docs. “
“Sorry about the confusion! I’ve found these calls in the libRealCall docs. Here’s the new code and links to the docs.”
“That’s the same code but with links to the libRealCall docs landing page.”
“You’re absolutely correct.” It appears that these calls belong to another library with that functionality:” looks totally plausible but hallucinated libFakeCall.
You forgot to include it's best excuse for why the documentation it cites is hallucinated: "The links may be broken/website is down".
For all the hot air I hear about the user having to give the system the proper context to give you good answers - does anyone claim to have a solution for dealing with such a belligerent approach to bullshit?
They are by no means useless, but once they fall into that hole, there's no further value in interrogating them.
It’s funny how people acknowledge the railroads and the similarity to AI, but then jump back to comparing it to the internet when it comes to drawing conclusions.
The internet build out left massive amounts of useful infrastructure.
The railroads left us with lots of railroads that fell into disuse and eventually left us with a complete joke of a railway system. Made a few people so rich we started calling them robber barons and talking the gilded age.
Are we going to continue to use the 60 billion dollar data centers in Louisiana when the bubble bursts? Is it valuable infrastructure or just a waste of money that gets written off?
I don’t disagree, but was Louisiana chosen because it offers cheap energy/cooling/infra or because the government doesn’t bother to ask questions about the impact to thinks like energy and water usage (which is why we are seeing more and more reports of residents upset about pollution, water usage, and energy costs).
There are only 3 things that I have strong empirical evidence for with respect to LLMs
1. Routinely some task or domain of work that some expert claims that LLM’s will able to do, LLM’s start being able to reliably perform that task within 6 months to a year, if they haven’t already
2. Whenever AI gets better, people move the goalposts regarding what “intelligence” counts as
3. Still, LLM’s reveal that there is an element to intelligence that is not orthogonal to the ability to do well on tests or benchmarks
If all you’ve done with AI is just use it for autocomplete, you’re missing out big time. I built a slick react app using Lovable yesterday and then created a Node BE using Claude Code today. I told Claude Code to look through the FE code to understand the requirements and purpose of the site, and to build a detailed plan (including proposed DB schemas) for a system that could support that functionality.
It generated a thousand line file with a robust breakdown of everything that needed to be done and at my command it did it. We went module by module and I made sure that each module
Had comprehensive unit test coverage and that the repo built well as we went. After a few hours of back and forth we made 9 modules, 60+ APIs across 10 different tables, and hundreds of unit tests that all are passing.
Does that mean that I’m all done and ready to deploy to prod? Unlikely. But it does mean that I got a ton of boilerplate stuff put into place really quickly and that I’m eight hours into a project that would have taken at least a month before.
Once the BE was done I had it generate extensive documentation for the agent that would handle the FE integration as a sort of instruction guide - in case we need it. As issues and bugs arise during integration (they will!) the model has everything it needs to keep on track and finish the job it set out to do.
Even this post by Martin Fowler shows he's an aging dinosaur stuck in denial.
> I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?
I don't know what "LLM's" he's using but I just simply don't get hallucinations like that with cursor or claude code.
He ends with this:
> LLMs create a huge increase in the attack surface of software systems. Simon Willison described the The Lethal Trifecta for AI agents: an agent that combines access to your private data, exposure to untrusted content, and a way to externally communicate (“exfiltration”). That “untrusted content” can come in all sorts of ways, ask it to read a web page, and an attacker can easily put instructions on the website in 1pt white-on-white font to trick the gullible LLM to obtain that private data.
Not sure why he is re-iterating well known prompt injection vulnerability, passing it off as a general weakness of LLM's that applies to all LLM use when that's not the reality.
> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful.
What a great way of framing it. I've been trying to explain this to people, but this is a succinct version of what I was stumbling to convey.
I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better.
https://jstrieb.github.io/posts/llm-thespians/
Intelligence in a way is the ability to filter out useless information. Be it, thoughts or sensory information
In my company I feel that we getting totally overrun with code that's 90% good, 10% broken and almost exactly what was needed.
We are producing more code, but quality is definitely taking a hit now that no-one is able to keep up.
So instead of slowly inching towards the result we are getting 90% there in no time, and then spending lots and lots of time on getting to know the code and fixing and fine-tuning everything.
Maybe we ARE faster than before, but it wouldn't surprise me if the two approaches are closer than what one might think.
What bothers me the most is that I much prefer to build stuff rather than fixing code I'm not intimately familiar with.
"but quality is definitely taking a hit now that no-one is able to keep up."
And its going to get worse! So please explain to me how in the net, you are going to be better off? You're not.
I think most people haven't taken a decent economics class and don't deeply understand the notion of trade offs and the fact there is no free lunch.
Technology has always helped people. Are you one of the people that say optimizing compilers are bad? Do you not use the intellisense? Or IDEs? Do you not use higher level languages? Why not write in assembly all the time? No free lunch right.
Yes there are trade offs, but at this point if you haven’t found a way to significantly amplify and scale yourself using llms, and your plan is to instead pretend that they are somehow not useful, that uphill battle can only last so long. The genie is out of the bag. Adapt to the times or you will be left behind. That’s just what I think.
Technology does not always help people, in fact often it creates new problems that didn't exist before.
Also telling someone to "adapt to the times" is a bit silly. If it helped as much as its claimed, there wouldn't be any need to try and convince people they should be using it.
A LOT of parallels with crypto, which is still trying to find its killer app 16 years later.
I don’t think anyone needs to be convinced at this point. Every developer is using LLM and I really can’t believe someone who has made a career out of automating things wouldn’t be immediately drawn to trying them at least. Every single company seems convinced and using it too. The comparison to crypto makes no sense.
My parents could have said your first paragraph when I tried to teach them they could Google their questions and find answers.
Technology moves forward and productivity improves for those that move with it.
Why is productivity so important? When do regular people get to benefit from all this "progress?"
> Technology moves forward and productivity improves for those that move with it.
It does not, technology regresses just as often and linear deterministic progress is just a myth to begin with. There is no guarantee for technology to move forward and always make things better.
There are plenty of examples to be made where technology has made certain things worse.
Paul Krugman (Nobel laureate in economy) said in 1998 that the internet is no biggie. Many companies needed convincing to adopt the internet (heck, some still need convincing).
Would you say the same thing ("If it helped as much as its claimed, there wouldn't be any need to try and convince people they should be using it.") about the internet?
Someone already pointed out that we're at the point when it's longer possible to know if comments like the above are satire or not.
As Fowler himself states, there's a need to learn to use these tools properly.
In any case poor work quality is a failure of tech leadership and culture, it's not AI's fault.
It’s funny how nothing seems to be AI’s fault.
No one seems to be able to grasp the possibility that AI is a failure
> No one seems to be able to grasp the possibility that AI is a failure.
Do you think by the time GPT-9 comes, we'll say "That's it, AI is a failure, we'll just stop using it!"
Or do you speak in metaphorical/bigger picture/"butlerian jihad" terms?
You've failed to figure out when and how to use it. It's not a binary failed/succeeded thing.
None of the copyright issues or suicide cases are handled in the court yet. There are many aspects.
Metaverse was...
If poor work gets merged, the responsibility lies in who wrote it, who merged it, and who allows such a culture.
The tools used do not hold responsibilities, they are tools.
How could a tool be at fault? If an airplane crashes is the plane at fault or the designers, engineers, and/or pilot?
> Certainly if we ever ask a hallucination engine for a numeric answer, we should ask it at least three times, so we get some sense of the variation.
This works on people as well!
Cops do this when interrogating. You tell the same story three times, sometimes backwards. It's hard to keep track of everything if you're lying or you don't recall clearly so you can get a sense of confidence. Also works on interviews, ask them to explain a subject in three different ways to see if they truly understand.
> This works on people as well!
Only within certain conditions or thresholds that we're still figuring out. There are many cases where the more someone recalls and communicates their memory, the more details get corrupted.
> Cops do this when interrogating.
Sometimes that's not to "get sense of the variation" but to deliberately encourage a contradiction to pounce upon it. Ask me my own birthday enough times in enough ways and formats, and eventually I'll say something incorrect.
Care must also be taken to ensure that the questioner doesn't change the details, such as by encouraging (or sometimes forcing) the witness/suspect to imagine things which didn't happen.
Triple modular redundancy. I remember reading that's how Nasa space shuttles calculate things because a processor / memory might have been affected by space radiation https://llis.nasa.gov/lesson/18803
Who remembers that scene on Better Call Saul between Lalo, Saul, and Kim?
For a hot second I thought LLMs were coming for our jobs. Then I realized they were just as likely to end up creating mountains of things for us to fix later. And as things settle down, I find good use cases for Claude Code that augment me but are in no danger of replacing me. It certainly has its moments.
I get a lot of productivity out of LLMs so far, which for me is a simple good sign. I can get a lot done in a shorter time and it's not just using them as autocomplete. There is this nagging doubt that there's some debt to pay one day when it has too loose a leash, but LLMs aren't alone in that problem.
One thing I've done with some success is use a Test Driven Development methodology with Claude Sonnet (or recently GPT-5). Moving forward the feature in discrete steps with initial tests and within the red/green loop. I don't see a lot written or discussed about that approach so far, but then reading Martin's article made me realize that the people most proficient with TDD are not really in the Venn Diagram intersection of those wanting to throw themselves wholeheartedly into using LLMs to agent code. The 'super clippy' autocomplete is not the interesting way to use them, it's with multiple agents and prompt techniques at different abstraction levels - that's where you can really cook with gas. Many TDD experts have great pride in the art of code, communicating like a human and holding the abstractions in their head, so we might not get good guidance from the same set of people who helped us before. I think there's a nice green field of 'how to write software' lessons with these tools coming up, with many caution stories and lessons being learnt right now.
edit: heh, just saw this now, there you go - https://news.ycombinator.com/item?id=45055439
It feels like Tdd/llm connection is implied — “and also generate tests”. Thought it’s not cannonical tdd of course. I wonder if it’ll turn the tide towards tech that’s easier to test automatically, like maybe ssr instead of react.
Yep, it's great for generating tests and so much of that is boilerplate that it feels great value. As a super lazy developer it's great as the burden of all that mechanical 'stuff' being spat out is nice. Test code being like baggage feels lighter when it's just churned out as part of the process, as in no guilt just to delete it all when what you want to do changes. That in itself is nice. Plus of course MCP things (Playwright etc) for integration things is great.
But like you said, it was meant more TDD as 'test first' - so a sort of 'prompt-as-spec' that then produces the test/spec code first, and then go iterate on that. The code design itself is different as influenced by how it is prompted to be testable. So rather than go 'prompt -> code' it's more an in-between stage of prompting the test initially and then evolve, making sure the agent is part of the game of only writing testable code and automating the 'gate' of passes before expanding something. 'prompt -> spec -> code' repeat loop until shipped.
The only thing I dislike is what it chooses to test when asked to just "generate tests for X": it often chooses to build those "straitjacket for your code" style tests which aren't actually useful in terms of catching bugs, they just act as "any change now makes this red"
As a simple example, a "buildUrl" style function that put one particular host for prod and a different host for staging (for an "environment" argument) had that argument "tested" by exactly comparing the entire functions return string, encoding all the extra functionality into it (that was tested earlier anyway).
A better output would be to check startsWith(prodHost) or similar, which is what I changed it into, but I'm still trying to work out how to get coding agents to do that in the first or second attempt.
But that's also not surprising: people write those kinds of too-narrow not-useful tests all the time, the codebase I work on is littered with them!
> It feels like Tdd/llm connection is implied — “and also generate tests”.
That sounds like an anti-pattern and not true TDD to get LLMs to generate tests for you if you don't know what to test for.
It also reduces your confidence in knowing if the generated test does what it says. Thus, you might as well write it yourself.
Otherwise you will get these sort of nasty incidents. [0] Even when 'all tests passed'.
[0] https://sketch.dev/blog/our-first-outage-from-llm-written-co...
LLMs (Sonnet, Gemini from what I tested) tend to “fix” failing tests by either removing them outright or tweaking the assertions just enough to make them pass. The opposite happens too - sometimes they change the actual logic when what really needs updating is the test.
In short, LLMs often get confused about where the problem lies: the code under test or the test itself. And no amount of context engineering seems to solve that.
I think in part the issue is that the LLM does not have enough context. The difference between a bug in the test or a bug in the implementation is purely based on the requirements which are often not in the source code and stored somewhere else(ticket system, documentation platform).
Without providing the actual feature requirements to the LLM(or the developer) it is impossible to determine which is wrong.
Which is why I think it is also sort of stupid by having the LLM generate tests by just giving it access to the implementation. That is at best testing the implementation as it is, but tests should be based on the requirements.
Oh, absolutely, context matters a lot. But the thing is, they still fail even with solid context.
Before I let an agent touch code, I spell out the issue/feature and have it write two markdown files - strategy.md and progress.md (with the execution order of changes) inside a feat_{id} directory. Once I’m happy with those, I wipe the context and start fresh: feed it the original feature definition + the docs, then tell it to implement by pulling in the right source code context. So by the time any code gets touched, there’s already ~80k tokens in play. And yet, the same confusion frequently happens.
Even if I flat out say “the issue is in the test/logic,”, even if I point out _exactly_ what the issue is, it just apologizes and loops.
At that point I stop it, make it record the failure in the markdown doc, reset context, and let it reload the feature plus the previous agent’s failure. Occasionally that works, but usually once it’s in that state, I have to step in and do it myself.
> hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful.
Nice.
I'd rather say that LLMs live in a world that consists entirely of stories, nothing but words and their combinations. Thy have no other reality. So they are good at generating more stories that would sit well with the stories they already know. But the stories are often imprecise, and sometimes contradictory, so they have to guess. Also, LLMs don't know how to count, but they know that two usually follows one, and three is usually said to be larger than two, so they can speak in a way that mostly does not contradict this knowledge. They can use tools to count, like a human who knows digits would use a calculator.
But much more than an arithmetic engine, the current crop of AI needs an epistemic engine, something that would help follow logic and avoid contradictions, to determine what is a well-established fact, and what is a shaky conjecture. Then we might start trusting the AI.
this was true, but then it wasn't... the research world several years ago, had a moment when the machinery could reliably solve multi-step problems.. there had to be intermediary results; and machinery could solve problems in a domain where they were not trained specifically.. this caused a lot of excitement, and several hundred billion dollars in various investments.. Since no one actually knows how all of it works, not even the builders, here we are.
"Since no one actually knows how all of it works, not even the builders, here we are."
To me this is the most bizarre part. Have we ever had a technology deployed at this scale without a true understanding of its inner workings?
My fear is that the general public perception of AI will be damaged since for most LLMs = AI.
Humanity used fire for like a bazillion years before figuring out thermodynamics
This is a misconception, we absolutely do know how LLMs work, that's how we can write them and publish research papers.
The idea we don't is tabloid journalism, it's simply because the output is (usually) randomised - taken to mean, by those who lack the technical chops, that programmers "don't know how it works" because the output is indeterministic.
This is not withstanding we absolutely can repeat the output by using not randomisation (temperature 0).
Are you sure you're talking about LLMs? These sound more like traditional ML systems like AlphaFold or AlphaProof.
In that framing, you can look at an agent as simply a filter on those hallucinations.
This vaguely relates to a theory about human thought: that our subconscious constantly comes up with random ideas, then filters the unreasonable ones, but in people with delusions (e.g. schizophrenia) the filter is broken.
Salience (https://en.wikipedia.org/wiki/Salience_(neuroscience)), "the property by which some thing stands out", is something LLMs have trouble with. Probably because they're trained on human text, which ranges from accurate descriptions of reality to nonsense.
More of a error-correcting feedback loop rather than a filter, really. Which is very much what we do as humans, apparently. One recent theory of neuroscience that is becoming influential is Predictive Processing --https://en.wikipedia.org/wiki/Predictive_coding -- this postulates that we also constantly generate a "mental model" of our environment (a literal "prediction") and use sensory inputs to correct and update it.
So the only real difference between "perception" and a "hallucination" is whether it is supported by physical reality.
You ever see something out of the corner of your eye and you were mistaken? It feels like LLM hallucinations. "An orange cat on my counter?!" And an instant later, your brain has reclassified "a basketball on my counter" as that fits the environment model better as several instant-observations gather contexts: not moving, more round, not furry, I don't own a cat, yesterday my kid mentioned something about tryouts, boop insta-reclassification from cat to basketball.
I can recognize my own meta cognition there. My model of reality course corrects the information feed interpretation on the fly. Optical illusions feel very similar whereby the inner reality model clashes with the observed.
For general ai, it needs a world model that can be tested against and surprise is noted and models are updated. Looping llm output with test cases is a crude approximation of that world model.
I leaned heavily on my own meta cognition recently when withdrawing from a bereavment benzo habit recently. The combo of flu symptoms, anxiety and hallucinations are fierce. I knew O was seeing things that were not real; light fittings turning into a rotating stained glass slideshow. So I'm totally on board with the visual model hypothesis. My own speculation is that audio perception is less predictive as audio structure persists deeper into my own DMT sessions than does vision, where perspective quickly collapses and vision becomes kaleidoscopic. Which may be a return to the vision we had as newborns. Maybe normal vision is only attained via socialisation?
Sounds like a kalman filter, which suggests to me that it’s too simplistic a perspective.
thats a fascinating way to put it
Isn't an "agent" not just hallucinations layered on top of other random hallucinations to create new hallucinations?
No, that's exactly what an agent isn't. What makes an agent an agent is all the not-LLM code. When an agent generates Golang code, it runs the Go compiler, which is in the agent's architecture an extension of the agent. The Go compiler does not hallucinate.
The most common "agent" is an letting an LLM run a while loop (“multi-step agent”) [1]
[1] https://huggingface.co/docs/smolagents/conceptual_guides/int...
That's not how Claude Code works (or Gemini, Cursor, or Codex).
Yes yes, with yet to be discovered holes
I've prefered to riff off of the other quote:
"All (large language) model outputs are hallucinations, but some are useful."
Some astonishingly large proportion of them, actually. Hence the AI boom.
Isn’t that why people argue against calling them hallucinations?
It implies that some parts of the output aren’t hallucinations, when the reality is that none of it has any thought behind it.
I have a very similar (probably unoriginal) thought about some human mental illnesses.
So, we VALUE creativity, we claim that it helps us solve problems, improves our understanding of the universe, etc.
BUT people with some mental illnesses, their brain is so creative that they lose the understanding of where reality is and where their imagination/creativity takes over.
eg. Hearing voices? That's the brain conjuring up a voice - auditory and visual hallucinations are the easy example.
But it goes further, depression is where people's brains create scenarios where there is no hope, and there's no escape. Anxiety too, the brain is conjuring up fears of what's to come
Nah I don't agree with this characterization. The problem is, the majority of those hallucinations are true. What was said would make more sense if the majority of the responses were, in fact, false, but this is not the case.
I think you're both correct but have different definitions of hallucinations. You're judging it as a hallucination based on the veracity of the output. Whereas Fowler is judging it based on the method by which the output is achieved. By that judgement, everything is a hallucination because the user cannot differentiate between when the LLM is telling the truth and isn't.
This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.
an LLM hallucination is defined by its truth
> In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called confabulation,[1] or delusion)[2] is a response generated by AI that contains false or misleading information presented as fact.[3][4]
You say
> This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.
For consistency you might as well say everything the human mind does is hallucination. It's the same sort of claim. This claim at least has the virtue of being taken seriously by people like Descartes.
https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...
Even the colloquial term outside of AI is characterized by the veracity of the output.
LLMs don't hallucinate : they bullshit (which is not caring about truth).
This isn't a good characterization of it either. I don't think LLMs know the difference. Bullshit implies they are lying.
It's possible LLMs are lying but my guess is that they really just can't tell the difference.
They're using the term 'bullshit' as it is understood as a term of art, which doesn't imply lying. It's closer to creating a response without any regard for telling the truth. Bullshitting is often most effective when you happen to be telling the truth, although the bullshitter has no commitment to that.
> We should ask the LLM the question more than once
For any macOS users, I highly recommend an Alfred workflow so you just press command + space then type 'llm <prompt>' and it opens tabs with the prompt in perplexity, (locally running) deepseek, chatgpt, claude and grok, or whatever other LLMs you want to add.
This approach satisfies Fowler's recommendation of cross referencing LLM responses, but is also very efficient and over time gives you a sense of which LLMs perform better for certain tasks.
What would such a workflow look like? I have Alfred but mainly just use the clipboard feature. I've tried to get into automation but struggled for inspiration. This one seems good!
Are you just opening a browser tab?
Go to the 'Workflows' tab, make a new one with keyword of your choice (e.g. llm), and map it to open these urls in your default browser:
http://localhost:3005/?q={query}
https://www.perplexity.ai/?q={query}
https://x.com/i/grok?text={query}
https://chatgpt.com/?q={query}&model=gpt-5
https://claude.ai/new?q={query}
Modify to your taste. Example: https://github.com/stevecondylios/alfred-workflows/tree/main (you should be able to download the .alfredworkflow file and double click on it to import it straight into alfred, but creating your own shouldn't take long, maybe 5-10 mins if it's your first workflow)
There are many I've worked with that idolize Martin Fowler and have treated his words as gospel. That is not me and I've found it to be a nuisance, sometimes leading me to be overly critical of the actual content. As for now, I'm not working with such people and can appreciate the article shared without clouded bias.
I like this article, I generally agree with it. I think the take is good. However, after spending ridiculous amounts of time with LLMs (prompt engineering, writing tokenizers/samplers, context engineering, and... Yes... Vibe coding) for some periods 10 hour days into weekends, I have come to believe that many are a bit off the mark. This article is refreshing, but I disagree that people talking about the future are talking "from another orifice".
I won't dare say I know what the future looks like, but the present very much appears to be an overall upskilling and rework of collaboration. Just like every attempt before, some things are right and some are simply misguided. e.g. Agile for the sake of agile isn't any more efficient than any other process.
We are headed in a direction where written code is no longer a time sink. Juniors can onboard faster and more independently with LLMs, while seniors can shift their focus to a higher level in application stacks. LLMs have the ability to lighten cognitive loads and increase productivity, but just like any other productivity enhancing tool doing more isn't necessarily always better. LLMs make it very easy to create and if all you do is create [code], you'll create your own personal mess.
When I was using LLMs effectively, I found myself focusing more on higher level goals with code being less of a time sink. In the process I found myself spending more time laying out documentation and context than I did on the actual code itself. I spent some days purely on documentation and health systems to keep all content in check.
I know my comment is a bit sparse on specifics, I'm happy to engage and share details for those with questions.
Written code is an auditable entity in regulated businesses, like banks. Fowler suggests we get more comfortable with uncertainty because other eng domains have developed good risk mgmt. Cart before horse. Other engineering domains strive to mitigate the irreducible uncertainty of the physical world. AI adds uncertainty to any system. And there are many applications where increased uncertainty will lead to an increase in human suffering.
> written code is no longer a time sink
It still is, and should be. It’s highly unlikely that you provided all the required info to the agent at first try. The only way to fix that is to read and understand the code thoroughly and suspiciously, and reshaping it until we’re sure it reflects the requirements as we understand them.
Vibe coding is not telling an agent what to do and checking back. It's an active engagement and best results are achieved when everything is planned and laid out in advance — which can also be done via vibe coding.
No, written code is no longer a time sink. Vibe coding is >90% building without writing any code.
The written code and actions are literally presented in diffs as they are applied, if one so chooses.
> It's an active engagement and best results are achieved when everything is planned and laid out in advance
The most efficient way to communicate these plans is in code. English is horrible in comparison.
When you’re using an agent and not reviewing every line of code, you’re offloading thinking to the AI. Which is fine in some scenarios, but often not what people would call high quality software.
Writing code was never the slow part for a competent dev. Agent swarming etc is mostly snake oil by those who profit off LLMs.
ime this is the problem. When I have to deeply understand what an llm created, I don't see much of a speed improvement vs writing it myself.
With an engineer you can hand off work and trust that it works, whereas I find code reviewing llm output something that I have to treat as hostile. It will comment out auth or delete failing tests.
> When I have to deeply understand what an llm created
Which should be always in my opinion
Are people really pushing code to production that they don't understand?
> It's an active engagement and best results are achieved when everything is planned and laid out in advance — which can also be done via vibe coding.
No.
The general assumed definition of vibe coding, hence the vibe word, is that coding becomes an iterative process guided by intuition rather than spec and processes.
What you describe is literally the opposite of vibe coding, it feels the term is being warped into "coding with an LLM".
I've described an iterative process where one never needs to touch code or documents directly.
Leaving out specs and documentation leads to more slop and hallucinations, especially with smaller models.
How could you possibly plan out "everything" in advance? Code itself would be the only way to explicitly specify the "everything".
Have a documentation system in place and have the LLM plan the high level before having the LLM write any code.
You can always just wing it, but if you do so and there isn't adequate existing context you're going to struggle with slop and hallucinations more frequently.
> We are headed in a direction where written code is no longer a time sink.
Written code has never been a time sink. The actual time that software developers have spent actually writing code has always been a very low percentage of total time.
Figuring out what code to write is a bigger deal. LLMs can help with part of this. Figuring out what's wrong with written code, and figuring out how to change and fix the code, is also a big deal. LLMs can help with a smaller part of this.
> Juniors can onboard faster and more independently with LLMs,
Color me very, very skeptical of this. Juniors previously spent a lot more of their time writing code, and they don't have to do that anymore. On the other hand, that's how they became not-juniors; the feedback loop from writing code and seeing what happened as a result is the point. Skipping part of that breaks the loop. "What the computer wrote didn't work" or "what the computer wrote is too slow" or even to some extent "what the computer wrote was the wrong thing" is so much harder to learn from.
Juniors are screwed.
> LLMs have the ability to lighten cognitive loads and increase productivity,
I'm fascinated to find out where this is true and where it's false. I think it'll be very unevenly distributed. I've seen a lot of silver bullets fired and disintegrate mid-flight, and I'm very doubtful of the latest one in the form of LLMs. I'm guessing LLMs will ratchet forward part of the software world, will remove support for other parts that will fall back, and it'll take us way too long to recognize which part is which and how to build a new system atop the shifted foundation.
> Figuring out what code to write is a bigger deal. LLMs can help with part of this. Figuring out what's wrong with written code, and figuring out how to change and fix the code, is also a big deal. LLMs can help with a smaller part of this.
I found exactly this is what LLMs are great at assisting with.
But, it also requires context to have guiding points for documentation. The starting context has to contain just enough overview with points to expand context as needed. Many projects lack such documentation refinement, which causes major gaps in LLM tooling (thus reducing efficacy and increasing unwanted hallucinations).
> Juniors are screwed.
Mixed, it's like saying "if you start with Python, you're going to miss lower level fundamentals" which is true in some regards. Juniors don't inherently have to know the inner workings, they get to skip a lot of the steps. It won't inherently make them worse off, but it does change the learning process a lot. I'd refute this by saying I somewhat naively wrote a tokenizer, because the >3MB ONNX tokenizer for Gemma written in JS seemed absurd. I went in not knowing what I didn't know and was able to learn what I didn't know through the process of building with an LLM. In other words, I learned hands on, at a faster pace, with less struggle. This is pretty valuable and will create more paths for juniors to learn.
Sure, we may see many lacking fundamentals, but I suppose that isn't so different from the criticism I heard when I wrote most of my first web software in PHP. I do believe we'll see a lot more Python and linguistic influenced development in the future.
> I'm guessing LLMs will ratchet forward part of the software world, will remove support for other parts that will fall back, and it'll take us way too long to recognize which part is which and how to build a new system atop the shifted foundation.
I entirely agree, in fact I think we're seeing it already. There is so much that's hyped and built around rough ideas that's glaringly inefficient. But FWIW inefficiency has less of an impact than adoption and interest. I could complain all day about the horrible design issues of languages and software that I actually like and use. I'd wager this will be no different. Thankfully, such progress in practice creates more opportunities for improvement and involvement.
> Sure, we may see many lacking fundamentals, but I suppose that isn't so different from the criticism I heard when I wrote most of my first web software in PHP.
It's not just the fundamentals, though you're right that is an easy casualty. I also agree that LLMs can greatly help with some forms of learning -- previously, you kind of had to follow the incremental path, where you couldn't really do anything complex without have the skills that it built on, because 90% of your time and brain would be spent on getting the syntax right or whatever and so you'd lose track of the higher-level thing you were exploring. With an LLM, it's nice to be able to (temporarily) skip that learning and be able to explore different areas at will. Especially when that motivates the desire to now go back and learn the basics.
But my real fear is about the skill acquisition, or simply the thinking. We are human, we don't want to have to go through the learning stage before we start doing, and we won't if we don't have to. It's difficult, it takes effort, it requires making mistakes and being unhappy about them, unhappy enough to be motivated to learn how to not make them in the future. If we don't have to do it, we won't, even if we logically know that we'd be better off.
Especially if the expectations are raised to the point where the pressure to be "productive" makes it feel like you're wasting other people's time and your paycheck to learn anything that the LLM can do for you. We're reaching the point where it feels irresponsible to learn.
(Sometimes this is ok. I'm fairly bad at long division now, but I don't think it's holding me back. But juniors can't know what they need to know before they know it!)
> But my real fear is about the skill acquisition, or simply the thinking. We are human, we don't want to have to go through the learning stage before we start doing, and we won't if we don't have to. It's difficult, it takes effort, it requires making mistakes and being unhappy about them, unhappy enough to be motivated to learn how to not make them in the future. If we don't have to do it, we won't, even if we logically know that we'd be better off.
I've noticed the effects of this first hand from intense LLM engagement.
I relate it more to the effects of portable calculators, navigation systems, and tools like Wikipedia. I'm under the impression this is valid criticism, but we may be overly concerned because it's new and a powerful tool. There's even surveys/studies showing differences in how LLM are perceived WRT productivity between different generations.
I'm more concerned with potential loss of critical thinking skills, more than anything else. And on a related note, there have been concerns of critical thinking skills before this mass adoption of LLMs. I'm also concerned with the impact of LLMs on the quality of information. We're seeing a huge jump in quantity while some quality lacks. It bothers me when I see an LLM confidently presenting incorrect information that's seemingly trivial to validate. I've had web searches give me more incorrect information from LLM tooling at a much greater frequency than I've ever experienced before. It's even more unsettling when the LLM gives the wrong answer and the correct answer is in the description of the top result.
"I'm also concerned with the impact of LLMs on the quality of information."
You have finally made an astute observation...
I have already made the assumption that use of LLMs is going to add new mounds of BS atop the mass of crap that already exists on the internet, as part of my startup thesis.
These things are not obvious in the here and now, but I try to take the view of - how would the present day look, 50 years out in the future looking backwards?
The issue is a lot of times it takes a senior level programmer to reason with a LLM to get the results needed. What happens when there are no Juniors to replace the Seniors. I guess by then AI will be able to program efficiently enough.
So far AI seems to be a great augmentation, but not a replacement.
I will believe his rant when Fowler creates any notable software.
> we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers,
Yup, this matches my recommended workflow exactly. Why waste time trying to turn an initially bad answer into a passable one, when you could simply re-generate (possibly with different context)
I wrote up an example of this workflow here: https://github.com/sutt/agro/blob/master/docs/case-studies/a...
> One of the consequences of this is that we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers, indeed perhaps ask the LLM to compare answers for us. The difference in the answers can be as useful as the answers themselves.
There was once a coding agent which achieved SOTA performance on SWE Bench Verified by "just" running the agent 5 times on each instance, scoring each attempt and picking the attempt with the highest score: https://aide.dev/blog/sota-bitter-lesson
> One of the consequences of this is that we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers, indeed perhaps ask the LLM to compare answers for us. The difference in the answers can be as useful as the answers themselves.
This is what LLM "reasoning" does. More than "reasoning" in the human sense, it just reduces variance from variations in the prompt and random next token prediction.
For my money, I used this analogy at work:
Before AI, we were trying to save money, but through a different technique: Prompting (overseas) humans.
After over a decade of trying that, we learned that had... flaws. So round 2: Prompting (smart) robots.
The job losses? This is just Offshoring 2.0; complete with everyone getting to re-learn the lessons of Offshoring 1.0.
Funnily enough, I left a similar comment just the other day: https://news.ycombinator.com/item?id=44944717
The conclusion I reached was different, though. We learnt how to do outsourcing "properly" pretty quickly after some initial high-profile failures, which is why it has only continued to grow into such a huge industry. This also involved local talent refocusing on higher-value tasks, which is why job losses were limited. Those same lessons and outcomes of outsourcing are very relevant to "bot-sourcing'.
However, I do feel concerned that AI is gaining skill-levels much faster than the rate at which people can upskill themselves.
I'm sure the blacksmiths and weavers will find solace in that take. Their time will return!
> Prompting (overseas) humans [...] After over a decade of trying that, we learned that had... flaws.
I think this is a US-centric point of view, and seems (though I hope it's not!) slightly condescending to those of us not in the US.
Software engineering is more than what happens to US-based businesses and their leadership commanding hundreds or thousands of overseas humans. Offshoring in software is certainly a US concern (and to a lesser extent, other nations suffer it), but is NOT a universal problem of software engineering. Software engineering happens in multiple countries, and while the big money is in the US, that's not all there is to it.
Software engineering exists "natively" in countries other than the US, so any problems with it should probably (also) be framed without exclusive reference to the US.
The problem isn't that there aren't high quality offshore developers - far from it. Or even high quality AI models.
The problems are inherent with outsourcing to a 3rd party and having little oversight. Oversight is, in both cases, way harder than it appears.
Talking to LLMs reminds me of my time as a system administrator for my university's physics department. A bunch of incredibly smart, PhD level people...just not in the subject they were talking to me about, most of the time.
My favorite quote to borrow: “Furthermore I think anyone who says they know what this future will be is talking from an inappropriate orifice.”
Predicting the future isn't about being correct tomorrow, rather it’s about selling something to someone today.
An insight I picked up along the way…
Reminds me of the classic Yogi Berra, “It's tough to make predictions, especially about the future”.
To me, it is more specific to say that many futurists don't take into account the social and economic network effects of changes taking place. Many just act as if the future will continue on completely unchallenged in this current state. But if you look at someone like Kurzweil, you can see the very narrow and specific focus of a prediction, which has proved to be more informative to me as a high bar of futurism.
> Other forms of engineering have to take into account the variability of the world.
> Maybe LLMs mark the point where we join our engineering peers in a world on non-determinism.
Those other forms of engineering have no choice due to the nature of what they are engineering.
Software engineers already have a way to introduce determinism into the systems they build! We’re going backwards!
Part of what got me into software was this: no matter how complex or impressive the operation, with enough time and determination, you could trace each step and learn how a tap on a joystick lead to the specific pixels on a screen changing.
There’s a beautiful invitation to learn and contribute baked into a world where each command is fully deterministic and spec-ed out.
Yes, there have always been poorly documented black boxes, but I thought the goal was to minimize those.
People don’t understand how much is going to be lost if that goal is abandoned.
Agreed. The beauty of programming is that you're creating a "mathematical artifact." You can always drill down and figure out exactly what is going on and what is going to happen with a given set of inputs. Now with things like concurrency that's not exactly true, but, I think the sentiment still holds.
The more practical question is though, does that matter? Maybe not.
> The more practical question is though, does that matter?
I think it matters quite a lot.
Specifically for knowledge preservation and education.
Yep, LLM’s still run on hardware with the same fundamental architecture we’ve had for years, sitting under a standard operating system. The software is written in the same languages we’ve been using for a long time. Unless there’s some seismic shift where all of these layers go away we’ll be maintaining these systems for a while.
Yes, but : not in all cases really. There are plenty of throw away, one off, experimental, trial and error, low stakes, spammy, interactions with dumb-assery, manager replacing instances that are acceptable as being quickly created and forgotten black boxes of temporary barely working trash. (not that people will limit to the proper uses)
In a way this is also a mathematical artifact — after all tokens are selected through beam searching or some random sampling of likely successor tokens.
"Computers are deterministic!"
If I wanted to plumb together badly documented black boxes, I'd have become an EE.
Underrated comment - computers can be decidedly non-deterministic.
https://en.wikipedia.org/wiki/Metastability_(electronics)
> People don’t understand how much is going to be lost if that goal is abandoned.
Ah but you see, imagine the shareholder value we can generate for a quarter or two in the meanwhile!
Editors note: please read that as absolutely dripping with disdain and sarcasm.
As pertaining to software development, I agree. I've been hearing accounting (online and from coworkers) of using LLMs to do deterministic stuff. And yet, instead of at least prompting once to "write a script to do X," they just keep prompting "do X" over and over again. Seems incredibly wasteful. It feels like there is this thought of "We are not making progress if we aren't getting the LLM to do everything. Having it write a script we can review and tweak is anti-progress." No one has said that outright, but it's a gut feeling (and it wouldn't surprise me if people have said this out loud).
This is the 2025 equivalent of the people who once wrote 2000 word blog posts about how bad it was to use "cat" instead of just shell redirection.
These are hardly equivalent. One is someone preferring one deterministic way over another. The other is more akin to arguing that it's better to ask someone to manually complete a task for you instead of caching the instructions on your computer. Now if the LLM does caching then you have more of a point, I don't have enough experience there.
I am one of these people. For my scripting needs, it would probably take me longer to find the script I saved rather than just asking it again and getting an answer in 15 seconds. I haven't saved a smallish script in a year.
I used to be one of those people, then I started saving these scripts in a folder and realized just how much time it saved me. Especially for nontrivial scripts which require a lot of typing or input specification.
This was me before LLMs. Had tons of scripts saved. I just don't see the point anymore "Take this list of users, remove from x group and reset their passwords" takes less time than trying to remember what I named that file and where it's saved. Anything that can be done in under 100 lines isn't worth saving anymore.
'potatolicious says we're going forwards: https://news.ycombinator.com/item?id=44978319
That was an interesting take, but "probabilistic" is to me different from "random". In particular other field get error tolerances, LLMs give us nothing like that.
We're introducing chaos monkeys, not just variability.
Note that he's talking about the same nondeterminism in that post that we're talking about here.
From the linked comment
> Process engineers for example have to account for human error rates. [...] Designing systems to detect these errors (which are highly probabilistic!)
> Likewise even for regular mechanical engineers, there are probabilistic variances in manufacturing tolerances.
I read them as relatively confined, thus probabilistic. When a human pushes the wrong button, an elephant isn't raining from the sky. Same way tolerances are bounded.
When requesting a JSON array from an LLM, it could as well decide this time that JSON is a mythological Greek fighter.
I'm just saying, it's a cite to a thread about the implications of AI-based nondeterminism, just like this one. That's all.
adding to this, software deals with non-determinism all the time.
For example, web requests are non-deterministic. They depend, among other things, on the state of the network. They also depend on the load of the machine serving the request.
One way to think about this is: how easy is it for you to produce byte-for-byte deterministic builds of the software you're working on? If it's not trivial there's more non-determinism than is obvious.
Mostly the engineering part of software is dealing with non-determinism, by avoiding it or enforcing determinism. Take something like TCP, it's all about guaranteeing the determinism that either the message is sent and received or it is not. And we have a lot of algorithms that tries to guarantee consistency of information between the elements of a system.
But there is an underlying deterministic property in the TCP example. A message is either received within a timeout or not.
How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not? Has the halting problem been solved?
You don't need to solve the halting problem in this situation, because you only need to accept a subset of valid, correct programs.
> How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not?
That's the catch 22 with LLM. You're supposed to be both the asker and the verifier. Which in practice, it's not that great. LLMs will just find the snippets of code that matches somehow and just act on it (It's the "I'm feeling Lucky" button with extra steps)
In traditional programming, coding is a notation too more than anything. You supposed to have a solution before coding, but because of how the human brain works, it's more like a blackboard, aka an helper for thinking. You write what you think is correct, verify your assumptions, then store and forget about all of it when that's true. Once in a while, you revisit the design and make it more elegant (at least you hope you're allowed to).
LLM programming, when first started, was more about a direct english to finished code translation. Now, hope has scaled down and it's more about precise specs to diff proposal. Which frankly does not improve productivity as you can either have a generator that's faster and more precise (less costly too) or you will need to read the same amount of docs to verify everything as you would need to do to code the stuff in the first place (80% of the time spent coding).
So no determinism with LLMs. The input does not have any formal aspects, and the output is randomly determined. And the domain is very large. It is like trying to find a specific grain of sand on a beach while not fully sure it's there. I suspect most people are doing the equivalent of taking a handful of sand and saying that's what they wanted all along.
No? These kinds of analyses all seem to rely on the notion that the LLM-caller needs to accept whatever output the LLM provides. In practice, they discard all the outputs that don't compile, and then a further subset of the ones that don't --- those outputs that aren't instantly clear to the caller.
My intuition for the problem here is that people are fixated on the nondeterminism of the LLM itself, which is of limited importance to the actual problem domain of code generation. The LLM might spit out ancient Egyptian hieroglyphics! It's true! The LLM is completely nondeterministic. But nothing like that is ever going to get merged into `main`.
It's fine if you want to go on about how bad "vibe coding" is, with LLM-callers that don't bother to read LLM output, because they're not competent. But here we're assuming an otherwise competent developer. You can say the vibe coder is the more important phenomenon, but the viber doesn't implicate the halting problem.
Valid programs are almost infinite. Context free grammars (which describe valid programs) are generative. When you're programming, you are mostly restricting the set of valid program to include only the few that satisfy the specs. Adding an extra 0 to a number is valid, but put that in the context of money transactions, it's a "hell breaks loose" situation.
SO that's why "it compiles" is worthless in a business settings. Of course it should compile. That's the bare minimum of expectations. And even "it passes the tests" is not that great. That just means you have not mess things up. So review and quality (accountability for both) is paramount, so that the proper stuff get shipped (and fixed swiftly if there was a mistake).
I have LLMs generate Haskell. Having the code compile means everything type checks and is not worthless. That's a huge constraint on valid programs.
> Having the code compile means everything type checks and is not worthless. That's a huge constraint on valid programs.
In a business settings, what usually matter is getting something into prod and not have bug reports thrown back. And a maintainable code that is not riddled down with debts.
Compiled code is as basic as steering left and right for a F1 driver. Having the tests pass is like driving the car at slow speed and completing a lap with no other cars around. If you're struggling to do that, then you're still in the beginner phase and not a professional. The real deal is getting a change request from Product and and getting it to Production.
It feels like you stopped reading before "and then a further subset of those".
Again: my claim is simply that whatever else is going on, the halting problem doesn't enter into it, because the user in this scenario isn't obligated to prove arbitrary programs. Here, I can solve the halting problem right now: "only accept branchless programs with finite numbers of instructions". Where's my Field Medal? :)
It always feels like the "LLMs are nondeterministic" people are relying on the claim that it's impossible to tell whether an arbitrary program is branchless and finite. Obviously, no, that's not true.
> It feels like you stopped reading before "and then a further subset of those".
Pretty sure you've just edited to add that part.
No, I'd have indicated that in my comment if I had. Sorry, I think you just missed it.
I did add the last paragraph of the comment you just responded to (the one immediately above this) about 5 seconds after I submitted it, though. Doesn't change the thread.
> LLMs will just find the snippets of code that matches somehow
This suggests a huge gap in your understanding of LLMs if we are to take this literally.
> LLM programming, when first started, was more about a direct english to finished code translation
There is no direct english to finished code translation. A prompt like "write me a todo app" has infinitely many codes it maps to with different tradeoffs and which appeal to different people. Even if LLMs never made any coding mistakes, there is no function that maps a statement like that to specific pieces of code unless you're making completely arbitrary choices like the axiom of choice.
So we're left with the fact that we have to specify what we want. And at that LLMs do exceptionally well.
Right, you handle determinism by applying engineering. E.g. having fail safes, redundancies, creating robust processes, etc.
It’s not trivial largely because we didn’t bother to design deterministic builds because it didn’t seem to matter. There is not much about the actual problem that makes it difficult.
if you don't have deterministic builds then you can't tell whether the executable you're running comes from the source code you can see.
It is definitely non-trivial and large organizations spend money to try to make it happen.
Yes, I spent years working on exactly that. I personally was working with a compiler team to validate changes to eliminate non-determinism. That’s why I felt qualified to make the statement I did.
It’s non trivial because you have to back through decades of tools and fix all of them to remove non determinism because they weren’t designed with that in mind.
The hardest part was ensuring build environment uniformity but that’s a lot easier with docker and similar tooling.
I would rather say it like this: Very good, very hardworking engineers spent years of their lives building the machine that raised us from the non-determinism of messy physical reality. The technology that brought us perfect, replicable, and reliable math from sending electrons through rocks has been deeply underappreciated in the "software revolution".
The engineers at TSMC, Intel, Global Foundries, Samsung, and others have done us an amazing service, and we are throwing all that hard work away.
It's uncertainty, not non-determinism. I don't design a rocket with no idea about the chain of events. I have inscrutable uncertainty everywhere, from manufacture to flight.
It was forward when Newton discovered the beautiful simple determinism of physics.
Was it going backwards when the probabilistic nature of quantum mechanics emerged?
Two words: many-world interpretation.
More seriously, this is not a fair comparison. Adding LLM output to your source code is not analogical to quantum physics; it is analogical to letting your 5 years old child transcribe the experimentally measured values without checking and accepting that many of them will be transcribed wrong.
Hopefully even Newton already had some awareness of "deterministic chaos" (or whatever terms they would have used back then) ?
And on the side hand, no transistors without quantum mechanics.
> I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?
A junior engineer can't write code anywhere nearly as fast. It's apples vs oranges. I can have the LLm rewrite the code 10 times until its correct and its much cheaper than hiring an obsequious jr engineer
Junior engineers get better and learn from their mistakes. An LLM will happily make then same mistake 10 times if you try asking it to do something similar in 3 months. In the long term, I don't think LLMs actually save you time considering the amount of extra time spent reviewing/verifying its code and fixing tech debt.
If an AI can’t write the code after two attempts, I’ve never had success trying ten times
>I’m often asked, “what is the future of programming?” Should people consider entering software development now? Will LLMs eliminate the need for junior engineers? Should senior engineers get out of the profession before it’s too late? My answer to all these questions is “I haven’t the foggiest”
I just want to point out that this answer implicitly means that, at the very least, the profession is at least questionably uncertain which isn't a good sign for people with a long future orientation such as students.
(slightly off-topic)
Students shouldn't ever have become so laser-focused on a single career path anyway, and even worse than that is how colleges have become glorified trade schools in our minds. Students should focus on studying for their classes, getting their electives and extracurriculars in, getting into clubs... Then depending on which circles they end up in, they shape their career that way. The thought that getting a Computer Science major would guarantee students a spot in the tech industry was always ridiculous, because the industry was just never structured that way, it's always been hacker groups, study groups, open source, etc. bringing out the best minds
It has never been anywhere close to certain, we just had 20 years of wild, unsustainable growth that encouraged people to cover their eyes and pretend the ride would go on forever. 20 years of telling everyone under the age of 30 that of course they should learn to code and that CS was the new medical or legal degree. 20 years of smugly acting like we are the inevitable future when we are, in fact, subject to the same ups and downs as every other career.
The answers are even kind of contradictory. If the answer to "will we ever need software engineers again in the future?" is "we don't know" then the answer to "should I spend time and money to enter software engineering?" should be "no".
Let me guess, they’re not as good as the hype and Real Developers don’t have anything to worry about. Also something about the environment.
I like the idea of AI usage comes down to a measurement of "tolerances". With enough specificity, LLMs will 100% return what you want. The goal is to find the happy tolerance between "acceptable" and "I did it myself" via prompts.
> With enough specificity, LLMs will 100% return what you want.
By now I’m sure it won’t. Even if you provide the expected code verbatim, LLMs might go on a side quest to “improve” something.
Stop comparing LLMs to Humans for fucks sake. Anthropomorphization is fine as long as it serves a purpose, it doesn't in this case.
> I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?
Reminds me of a recent experience when I asked CC to implement a feature. It wrote some code that struck me as potentially problematic. When I said, "why did you do X? couldn't that be problematic?" it responded with "correct; that approach is not recommended because of Y; I'll fix it". So then why did it do it in the first place? A human dev might have made the same mistake, but it wouldn't have made the mistake knowing that it was making a mistake.
It did so because his training, weights and context and a billion of matrix calculations led him there.
I think this gets lost on far too many discussions. There is no “why” - it’s statistics, and the closest we have to a “why” is that a lot of code out there sucks
> One of the big problems with these surveys is that they aren’t taking into account how people are using the LLMs. From what I can tell the vast majority of LLM usage is fancy auto-complete, often using co-pilot.
This is a completely wrong assumption and negates a bunch of the points of the article...
How is it wrong?
Do you have any hard number and data to state that tab/auto complete are less popular than agentic coding?
I would expect it to be almost tautologically true. It's the easiest to implement, and the most deployed. It has to be the most used, even if a lot of people don't know they're using it.
It was the original success of LLMs -- code autocomplete. All of the agentic coding have autocomplete under the hood. Autocomplete is almost literally what LLMs do + post-processing.
> All major technological advances have come with economic bubbles, from canals and railroads to the internet.
Is this actually correct? I don't see any evidence for a "airflight bubble" or a "car bubble" or a "loom bubble" at the technologies' invention. Also the "canal bubble" wasn't about the technology, it was about the speculation on a series of big canals but we had been making canals for a long time. More importantly, even if it was correct, there are plenty of bubbles (if not significantly more) around things that didn't have value or tech that didn't matter.
Apologies for arguing from first principles, but for anything that spurs a lot of activity, the only alternative to a bubble is this: people ramp up investment and activity and enthusiasm only as much as the underlying thing can handle, then gradually taper off the increase and gently level off at the equilibrium "carrying capacity" of the new technology.
Does that sound like any human, ever, to you?
(The only time there isn't a bubble is when the thing just isn't that interesting to people and so there's never a big wave of uptake in the first place.)
If you were right then we'd have a lot more than a dozen listed bubbles on wikipedia.
That's an absurd framing for a cute quip.
Why do you assume that Wikipedia can be trusted to provide an exhaustive answer to this question ?
I don't know enough about the early history of the airline industry but there was very definitely a long series of huge bubbles in the railroad industry.
I would be very interested in reading about huge bubbles in railroad, airline, or any other industry. Do you happen to have references (genuinely asking; the original article didn't include any references)?
Edits--
Found one: https://en.wikipedia.org/wiki/Panic_of_1893
Another good one:https://en.wikipedia.org/wiki/Public_Utility_Holding_Company... (from cake_robot here: https://news.ycombinator.com/item?id=45056621)
For reference, apple and spotify links to the Derek Thompson podcast in reply below (thank you!):
https://podcasts.apple.com/us/podcast/plain-english-with-der...
https://open.spotify.com/show/3fQkNGzE1mBF1VrxVTY0oo
There was just a long Derek Thompson podcast with a Transcontinental Railroad scholar about this! (That's why I knew about it.)
The whole subtext of that podcast was how eerily similar the Transcontinental Railroad was to AI (as an investment/malinvestment/prediction of future trends).
The Intelligent Investor by Graham talks about investors putting so much money into airlines and air freight that it became impossible to make a return. I don't know if you would call that a bubble, maybe just over exuberance.
The AI bubble also isn't about the technology.
Dont forget the gas lighting when they make a mistake and try to act like it was you.
A bubble is asset prices systematically diverging from reasonable expectations of future cash flows. Bubbles are driven by financial speculation.
The claim in the blog post that all technology leads to speculative asset bubbles I find hard to believe. Where was the electricity bubble? The steel bubble? The pre-war aviation bubble? (The aviation bubble appeared decades later due to changes in government regulation.)
Is this an AI bubble? I genuinely don't know! There is a lot of real uncertainty about future cash flows. Uncertainty is not the foundation of a bubble.
I knew dot-com was a bubble because you could find evidence, even before it popped. (A famous case: a company held equity in a bubble asset, and that company had a market cap below the equity it held, because the bubble did not extend to second-order investments.)
Just taking your first example, yes I believe you could characterize a lot of the investments in electrification as speculative bubbles even though there was underlying value that was borne out.
https://en.wikipedia.org/wiki/Public_Utility_Holding_Company...
I want to re-phrase your definition.
To me a bubble reflects a market disconnect from fundamentals - wherein prices go up steeply, with no help from the fundamentals (expected growth in base year cash flows and risk).
Indeed there is a subtle difference between a bubble and speculation of what could come of a technology. But the two are connected because the effects of technology are reflected by investors in asset prices.
Besides electricity (answered above), there were bubbles related to steel production in its early days intertwined with the railroad bubble that drove huge investments in steel production for railroad development; when the railroad companies crashed it brought down steel as well; ultimately both survived in a more subdued form (as with the dot coms).
avation: likewise there was the "Lindberg Boom" https://en.wikipedia.org/wiki/Lindbergh_Boom which led to overspeculation and the crash of many early aviation companies
"Hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature".
I used to avidly read all his stuff, and I remember 20ish years ago he decided to rename Inversion of Control to Dependency Injection. In doing so, and his accompany blog, he showed he didn't actually understand it at a deep level (and hence his poor renaming).
This feels similar. I know what he's trying to say, but he's just wrong. He's trying to say the LLM is hallucinating everything, but Fowler is missing is that Hallucination in LLM terms refers to a very specific negative behavior.
As far as an LLM is concerned, there is no difference between "negative" hallucination and a positive one. It's all just tokens and embeddings to it.
Positive hallucinations are more likely to happen nowadays, thanks to all the effort going into these systems.
This basically ruins the term “hallucination” and makes it meaningless, when the term actually describes a real phenomenon.
That's the point. It is meaningless. When it first coined, there were already detractors to the term, that it is an incorrect description of the phenomenon. But it stuck.
No, it’s just an attempt to pretend wrong outputs are some special case when really they aren’t. They aren’t imagine something that doesn’t exist they are just running the same process they do for everything else and it just didn’t work.
If you disagree then I would ask what exactly is the “specific behaviour” you’re talking about?
I'll take the bait.
What didn't he understand properly about inversion of control then.
That it is ultimately about object lifetime management. Dependency injection is a big part of it, but not the only thing.
> I’ve often heard, with decent reason, an LLM compared to a junior colleague.
No, they're like an extremely experienced and knowledgeable senior colleague – who drinks heavily on the job. Overconfident, forgetful, sloppy, easily distracted. But you can hire so many of them, so cheaply, and they don't get mad when you fire them!
These metaphors all suck. Well, ok, yours is funny. But anyway, LLMs are just very different from any human.
They are extremely shallow, even compared to a junior developer. But extremely broad, even compared to the most experienced developer. They type real fuckin fast compared to anyone on earth, but they need to be told what to do much more carefully than anyone on earth.
> but they need to be told what to do much more carefully than anyone on earth.
have you ever managed an offshore team. holy cow
"Offshore team." "Holy cow." I see what you did there.
I've gotten Claude Code to make CUDA kernels and all kinds of advanced stuff that there's zero percent chance a junior would pull off.
AI is like a super advanced senior wearing a blindfold. It knows almost everything, it's super fast, and it gets confused pretty quickly about things after you tell it.
it's not a senior though, because of the amount of oversight and guidance required. I trust senior-plus human developers to do the right thing and understand why it's the right thing. For mission critical things I get another human senior to verify. There's no way I'd autonomously trust 2, 10 or any number of LLMs to do the same thing.
You'd be surprised at what juniors can pull off. I have seen fresh-out-of-college new grads write performant GPU kernels that are used in real world library implementations for particular architectures.
Most of the juniors I've worked with would make numerical errors and give up/declare premature victory before getting the implementation to a robust state. I'm sure there are exceptional young folks out there though.
What's your sample size?
And today I asked Claude code to carefully look at and follow the structure of my tests and write some more, and after 5-10 mins it completely ignored it, wrote one class with all the helper methods and a bunch of compilation errors from fields that arent even in my model.
I figured there's probably a ton more logical issues and deleted it immediately
have you ever asked a junior developer to write a cuda kernel?
I've asked juniors to write easier things without success, and I applied the transitive property.
And they are incessantly cheerful.
I asked Claude to design me a UI, and it made a lovely one... but I wanted a web ui. It very happily through away all its work and made a brand new web UI.
I can't imagine any employee being that quick to just move on after something like that.
I think we had the analogy right with "fancy autocomplete". Sometimes, the completions are excellent and do exactly what we want, other times the completions fail to match our expectations and need human attention. It's a tool.
I just spent a good 2 hours trying to debug a SM6 Vulkan issue with unreal engine using an LLM, it had got me to good state but UE kept falling to load a project, it transpired that the specific error message would provide a fix as the top Google result, which I found when I eventually decided to look for myself.
LLM did help a lot to get some busy work out of the way, but it's difficult to know when you need to jump out of the LLM loop and go old skool.
Fwiw I think the ratio of times I needed to go to google for a solution instead of an LLM is like 20:1 for me so your mileage may vary. Depends a lot on the specific niche you're working in.
Unrelated to software but recently I wanted to revive an old dumbphone I haven't used since 2014 and apparently I had it password protected and forgot the password and wanted to factory reset it. I found the exact model of the phone and google had only content farm articles that didn't help me at all but Gemini gave me the perfect solution first try. I went to google first because I had no faith in Gemini since to me it seemed like a pretty obscure question but guess I was wrong.
In the interest of full disclosure my setup is quite esoteric for unreal dev. Linux and nixos no less. To be honest I'd probably have given up on nixos long ago without LLM support. It's actually really quite handy to be able to share a declarative specification of my environment.
Google search has been enshittified. Kagi is where you get real search results now.
Unsure why this is downvoted. "Google search has been enshittified" should be a common sentiment here.
No, they're like a completely nontechnical marketing person who has a big library of papers on related subjects and who's been asked to generate a whitepaper by pulling phrases. What comes out will probably have proper grammar and seem perfectly reasonable to another person with no knowledge of the field, but if actually read by someone knowledgeable may be complete gibberish taken as a whole.
Individual sentences and paragraphs may mostly work, but it's an edifice built on sand out of poorly constructed bricks plus mortar with the wrong proportions (or entirely wrong ingredients).
LLM output is "truthy" - it looks like it might be true, and sometimes it will even be accurate (see also, "stopped clock") but depending on it is foolish because what's generating the output doesn't actually understand what it's putting out - it's just generating output that looks like the kind of thing you've requested.
Overconfident, but also easy to sway their opinion.
They follow directions for maybe an hour and then go off and fix random shit because they forgot what their main task was.
They'll tell you to your face how great your ideas were, and that you're absolutely right about something, then go implement it completely incorrectly.
They add comments to literally everything even when you ask them to stop. But they also ignore said comments sometimes.
Maybe they are kinda like us lol.
Like a person that has memorized every single answer in the world to every single question ever asked - but is completely dead behind the eyes.
My analogy is a cursed monkey paw with unlimited wishes. It's actually really really powerful but you have to be careful of leaving any possible ambiguity in your wishes.
> who drinks heavily on the job
Yes, that's why you should add "... and answer in the style of a drunkard" to every prompt. Makes it easier to not forget what you are dealing with.
Experienced and knowledgeable and they also believe in the technical equivalent of flat-Earthism in many, many non-trivial corners.
And if you push back on that insanity, they'll smile and nod and agree with you and in the next sentence, go right back to pushing that nonsense.
One good yardstick, if one has to anthropomorphize, is that LLMs know and believe what's popular. If you ask it to do something that popular developer opinion gets right, it will do fine.
Ask it for things that many people get wrong or just do badly, or can be mistakenly likened to a popular thing in a way that produces a wrong result, and it'll often err.
The trick is having an awareness of what correct solutions are prevalent in training data and what the bulk of accessible code used for training probably doesn't have many examples of. And this experience is hard to substitute for.
Juniors therefore use LLMs in a bumbling fashion and are productive either by sheer luck, or because they're more likely to ask for common things and so stay in a lane with the model.
A senior developer who develops a good intuition for when the tool is worth using and when not can be really efficient. Some senior developers however either overestimate or underestimate the tool based on wrong expectations and become really inefficient with them.
“Here’s the code:” looks totally plausible but hallucinated libFakeCall.
“libFakeCall doesn’t exist. Use libRealCall instead of libFakeCall.”
“You’re absolutely correct. I apologize for blah blah blah blah. Here’s the updated code with libRealCall instead. :[…]”
“You just replaced the libFakeCall reference with libRealCall but didn’t update the calls themselves. Re-write it and cite the docs. “
“Sorry about the confusion! I’ve found these calls in the libRealCall docs. Here’s the new code and links to the docs.”
“That’s the same code but with links to the libRealCall docs landing page.”
“You’re absolutely correct.” It appears that these calls belong to another library with that functionality:” looks totally plausible but hallucinated libFakeCall.
You forgot to include it's best excuse for why the documentation it cites is hallucinated: "The links may be broken/website is down".
For all the hot air I hear about the user having to give the system the proper context to give you good answers - does anyone claim to have a solution for dealing with such a belligerent approach to bullshit?
They are by no means useless, but once they fall into that hole, there's no further value in interrogating them.
> Overconfident, forgetful, sloppy, easily distracted
And constantly microdosing, sometimes a bit too much.
And sometimes it seems like a macro-dose huffing xylene-based glue.
> and they don't get mad when you fire them!
No, typically __you__ are mad when you fire them ...
If you’re making hire/fire decisions while emotional, you’re doing it wrong.
Whoosh
“You’re absolutely right!”
Yes. Funny
But it is nothing like a wetware colleague. It is a machine.
It’s funny how people acknowledge the railroads and the similarity to AI, but then jump back to comparing it to the internet when it comes to drawing conclusions.
The internet build out left massive amounts of useful infrastructure.
The railroads left us with lots of railroads that fell into disuse and eventually left us with a complete joke of a railway system. Made a few people so rich we started calling them robber barons and talking the gilded age.
Are we going to continue to use the 60 billion dollar data centers in Louisiana when the bubble bursts? Is it valuable infrastructure or just a waste of money that gets written off?
> Are we going to continue to use the 60 billion dollar data centers in Louisiana when the bubble burst
Why not?
Tracks in the middle of nowhere are useless.
Data centers in the middle of nowhere are useful, as long as energy, cooling and uplink are cost-effective.
(Not saying Louisiana is in the middle of nowhere!)
I don’t disagree, but was Louisiana chosen because it offers cheap energy/cooling/infra or because the government doesn’t bother to ask questions about the impact to thinks like energy and water usage (which is why we are seeing more and more reports of residents upset about pollution, water usage, and energy costs).
.
There are only 3 things that I have strong empirical evidence for with respect to LLMs
1. Routinely some task or domain of work that some expert claims that LLM’s will able to do, LLM’s start being able to reliably perform that task within 6 months to a year, if they haven’t already
2. Whenever AI gets better, people move the goalposts regarding what “intelligence” counts as
3. Still, LLM’s reveal that there is an element to intelligence that is not orthogonal to the ability to do well on tests or benchmarks
If all you’ve done with AI is just use it for autocomplete, you’re missing out big time. I built a slick react app using Lovable yesterday and then created a Node BE using Claude Code today. I told Claude Code to look through the FE code to understand the requirements and purpose of the site, and to build a detailed plan (including proposed DB schemas) for a system that could support that functionality.
It generated a thousand line file with a robust breakdown of everything that needed to be done and at my command it did it. We went module by module and I made sure that each module Had comprehensive unit test coverage and that the repo built well as we went. After a few hours of back and forth we made 9 modules, 60+ APIs across 10 different tables, and hundreds of unit tests that all are passing.
Does that mean that I’m all done and ready to deploy to prod? Unlikely. But it does mean that I got a ton of boilerplate stuff put into place really quickly and that I’m eight hours into a project that would have taken at least a month before.
Once the BE was done I had it generate extensive documentation for the agent that would handle the FE integration as a sort of instruction guide - in case we need it. As issues and bugs arise during integration (they will!) the model has everything it needs to keep on track and finish the job it set out to do.
What a time to be alive!
I feel exactly the same way.
Even this post by Martin Fowler shows he's an aging dinosaur stuck in denial.
> I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?
I don't know what "LLM's" he's using but I just simply don't get hallucinations like that with cursor or claude code.
He ends with this: > LLMs create a huge increase in the attack surface of software systems. Simon Willison described the The Lethal Trifecta for AI agents: an agent that combines access to your private data, exposure to untrusted content, and a way to externally communicate (“exfiltration”). That “untrusted content” can come in all sorts of ways, ask it to read a web page, and an attacker can easily put instructions on the website in 1pt white-on-white font to trick the gullible LLM to obtain that private data.
Not sure why he is re-iterating well known prompt injection vulnerability, passing it off as a general weakness of LLM's that applies to all LLM use when that's not the reality.
Okay, but LLMs will produce a thousand lines of code for Hello World.
Plus, there is no telling how many bugs you have in that code.