> Why are we talking about “graduate and PhD-level intelligence” in these systems if they can’t find and verify relevant links — even directly after a search?
This is my pet peeves, and recently OpenAI's models seem to have become very militant in how they stand by and push their obviously hallucinated sources. I'm talking about hallucinating answers, when pressed to cite sources they also hallucinate URLs that never existed, when repeatedly prompted to verify how the are hallucinating the stick to their clearly wrong output, and ultimately fall back to claiming they were right but the URL somehow changed even though it never existed ever.
In order to start talking about PhD-level intelligence, in the very least these LLMs must support PhD-level context-seeking and information verification. It is not enough to output a wall of text that reads quite fluently. You must stick to verifiable facts.
The approach of generating something and then looking for hallucinations is just stupid. To validate the output I have to be an expert. How do I become an expert if rely on LLMs? It's a dead end.
"and suport the claims" is doing some *extremely* heavy lifting there.
I can't write a software program, give the source to the greengrocer and expect him to be able to say anything about its quality. Just like I can't really say much about vegetables.
If the output is interpreting sources rather than just regurgitating quotes from them, you need to exert judgment to verify they support its claims. When the LLM output is about some highly technical subject, it can require expert knowledge just to judge whether the source supports the claims.
Or why the LLM doesn’t do a lookup into a subset of the training data as a database and reject the output if it seems to be wrong. A billion of the most urls and the entirety of Wikipedia, arkiv and stackoverflow would go a long way.
> Seems like the LLM is giving correct output if it’s generating a plausible string of tokens in response to your string of tokens.
No. If you prompt it to get a response and then you ask it to cite sources, if it outputs broken links that never existed then it clearly failed to deliver correct output.
"correct" for an llm means "fits the statistical distributions in the training data"
"correct" for you is "truth that corresponds to the real world"
They are two very different things. The llm's output is, very much, correct. Because it was never meant to mean anything other than similarity of probability distributions.
It's not what you wanted, but that doesn't make it incorrect. You're just under a wrong assumption about what you were asking for. You were asking for something that looks like it could be true. Even if you ask it to not hallucinate, you're just asking it to make it look like it is not hallucinating. Meanwhile you thought you were asking for the actual, real, answer to your question.
Right, the dialogue between the user and the LLM closely resembles documents used in training the LLM. People argue with, lie to, and misunderstand others on the internet. Here's a totally plausible hypothetical forum discussion:
Person A: I believe X.
Person B: Do you have a source for that?
A: Yes, it was shown by blah blah in the paper yada yada.
B: I don't think that study exists. Share a link?
A: [posts a URL]
B: That's not a real paper. The URL doesn't even work!
A: Works on my machine.
---
I've seen those kind of chats so many times online. Know what I haven't seen very often? When person A says "You're right, I made up that article. Let me look again for a real one, and I might change my opinion depending on what it says."
Why isn't the LLM under the wrong assumption? So I don't get from my tool what I need and it's still me at fault? I am not yet ready to bow to the AI overlords, sorry.
But are the links plausible text given the training data?
If the purpose is to accurately cite sources, how is it even possible to hallucinate them? Seems like folks are expecting way too much from these tools. They are not intelligent. Useful, perhaps.
Seems that's just expecting things that LLMs were not designed for.
It's a token producer based on trained weights, it doesn't use any sources.
Even if it were "fixed" so that it only generates URLs that exist, it's still incorrect because it did not use any sources so those URLs are not sources.
I have search enabled 100% of the time with ChatGPT and would never go back to raw-dogging LLM citations. O3 especially has passed the threshold of “not always annoying”. Had an argument with Gemini yesterday where it was insisting on some hallucinated implementation of a function even while giving me a GitHub link to the correct source.
if this refers to how often i post in bursts, i have wicked insomnia. I really should remove my credentials from the HN app on my phone. if this is what you meant, you're the first person to call me out!
I wouldn't use it in a workplace either, perhaps not at all, but here is a pseudonymous forum. The expectations—or repercussions—of decorum aren't the same.
algospeak originated on Tiktok, with all that implies. You couldn't post videos about sexual assault, suicides, murders, any of that directly, so people started self-censoring and using words - unalive maybe came from Deadpool in 2013? I'm not real interested in the actual etymology right now.
and you jest but there's been pushes to change things like "master/slave", "kill", "whitelist/blacklist", and the like to something different. I don't know how much traction, and i may have been tricked by april 1st posts or something.
I apologize if it came off as tone policing. "Raw dog" sounds innocuous, if a bit strange. There was a radio station that had an internet stream out of Topeka Kansas called "Raw Dog Radio" that was a comedy station/stream; as an example - but it was a direct reference to the sexual connotation. For people who don't know, and notice that "raw dogging" is entering zeitgeist, maybe they'll say it because it sounds silly. It was merely a warning about what it means. The "definition shift" really isn't, people are comparing taking a flight without water, cellphones, etc to sex without a condom (Doesn't matter, flew through the air at hundreds of miles per hour, don't care!) The person i replied to was comparing using an LLM without RAG to risky sex.
I'd also avoid saying "So. How are we going to fuck this pig?" in a meeting about infra problems.
The key thing I got from this article is that the o3 and Claude 4 projects (I'm differentiating from the models here because the harness of tools around them is critical too) are massively ahead of GPT 4.1 and Gemini 2.5 when it comes to fact checking in a way that benefits from search and web usage.
Both o3 and Claude 4 have a crucial new ability: they can run tools such as their search tool as part of their "reasoning" phase. I genuinely think this is one of the most exciting new advances in LLMs in the last six months.
If anyone is interested in a larger sample size comparing how often LLMs confabulate answers based on provided texts, I have a benchmark at https://github.com/lechmazur/confabulations/. It's always interesting to test new models with it because the results can be unintuitive compared to those from my other benchmarks.
Useful benchmark. I noticed o3-high hallucinating too often for such a good model, but it is usually great with search. In my experience, Claude Opus & Sonnet 4 consistently lie, cheat, and try to hide their tracks. Maybe they are good in writing code but I don't trust them with other things.
It's not just that they get links wrong, it's how they get them wrong – like, totally fabricating them and then doubling down! A human messing up a citation is one thing, but this feels... different, almost like a creative act of deception, lol.
I do wonder about the role of test time compute in the blog post in terms of document understanding. A non reasoning output (or low test time compute setting) might easily misinterpret the text, but reasoning models can second guess, consider multiple objectives in turn, and can right the ship.
I note that Gemini 2.5 has one of the lowest confabulation/hallucination rates according to this benchmark [1], so am surprised by the results in the blog.
Also, I have found link hallucination and output quality improve when you restrict searches to, for example, only pubmed sources, and to provide the source link directly into the text (as opposed to Gemini deep research usual method for citation).
One reason, I think, is that unrestricted search will get the paper, the related blog posts and press releases, weight them as equal (and independent!) sources of a fact, when we know that nuance is lost in the latter, and maybe because it will then spend more test time compute in the quality sources, not the press-releases.
> Why are we talking about “graduate and PhD-level intelligence” in these systems if they can’t find and verify relevant links
For exactly the same reason the author markets his tool as a research assistant
> It also models an approach that is less chatbot, and more research assistant in a way that is appropriate for student researchers, who can use it to aid research while coming to their own conclusions.
I have a strange feeling: it seems that original insights and hallucinations are related. One seems to come very frequently with the other.
I've noticed that o3 is the one that lies with the most conviction (compared to Gemini Pro and Claude Sonnet). It will be the hardest to convince that it is wrong, will invent excuses and complex explanations for its lies, almost to a Trump level of lying and deception.
But it is also the one that provides the most interesting insights, that will look at what others don't see.
There might some kind deep truth in this correlation. Or it might be myself having an hallucination...
Taken from the blog:
> Why are we talking about “graduate and PhD-level intelligence” in these systems if they can’t find and verify relevant links — even directly after a search?
This is my pet peeves, and recently OpenAI's models seem to have become very militant in how they stand by and push their obviously hallucinated sources. I'm talking about hallucinating answers, when pressed to cite sources they also hallucinate URLs that never existed, when repeatedly prompted to verify how the are hallucinating the stick to their clearly wrong output, and ultimately fall back to claiming they were right but the URL somehow changed even though it never existed ever.
In order to start talking about PhD-level intelligence, in the very least these LLMs must support PhD-level context-seeking and information verification. It is not enough to output a wall of text that reads quite fluently. You must stick to verifiable facts.
The approach of generating something and then looking for hallucinations is just stupid. To validate the output I have to be an expert. How do I become an expert if rely on LLMs? It's a dead end.
> The approach of generating something and then looking for hallucinations is just stupid. To validate the output I have to be an expert.
No. You only need to check for sources, and then verify these sources exist and they support the claims.
It's the very definition of "fact".
In some cases, all you need to do is check if a URL that was cited does exist.
"and suport the claims" is doing some *extremely* heavy lifting there.
I can't write a software program, give the source to the greengrocer and expect him to be able to say anything about its quality. Just like I can't really say much about vegetables.
If the output is interpreting sources rather than just regurgitating quotes from them, you need to exert judgment to verify they support its claims. When the LLM output is about some highly technical subject, it can require expert knowledge just to judge whether the source supports the claims.
Including literal 404s... As an outsider it has always struck me as absurd that they don't just do the equivalent of wget over all provided sources.
Or why the LLM doesn’t do a lookup into a subset of the training data as a database and reject the output if it seems to be wrong. A billion of the most urls and the entirety of Wikipedia, arkiv and stackoverflow would go a long way.
If that could be done, then we would be using that and skipping the llms entirely
Can’t see why that couldn’t be done? You save a http request for a ton of the urls.
Seems like the LLM is giving correct output if it’s generating a plausible string of tokens in response to your string of tokens.
> Seems like the LLM is giving correct output if it’s generating a plausible string of tokens in response to your string of tokens.
No. If you prompt it to get a response and then you ask it to cite sources, if it outputs broken links that never existed then it clearly failed to deliver correct output.
"correct" for an llm means "fits the statistical distributions in the training data"
"correct" for you is "truth that corresponds to the real world"
They are two very different things. The llm's output is, very much, correct. Because it was never meant to mean anything other than similarity of probability distributions.
It's not what you wanted, but that doesn't make it incorrect. You're just under a wrong assumption about what you were asking for. You were asking for something that looks like it could be true. Even if you ask it to not hallucinate, you're just asking it to make it look like it is not hallucinating. Meanwhile you thought you were asking for the actual, real, answer to your question.
Right, the dialogue between the user and the LLM closely resembles documents used in training the LLM. People argue with, lie to, and misunderstand others on the internet. Here's a totally plausible hypothetical forum discussion:
Person A: I believe X.
Person B: Do you have a source for that?
A: Yes, it was shown by blah blah in the paper yada yada.
B: I don't think that study exists. Share a link?
A: [posts a URL]
B: That's not a real paper. The URL doesn't even work!
A: Works on my machine.
---
I've seen those kind of chats so many times online. Know what I haven't seen very often? When person A says "You're right, I made up that article. Let me look again for a real one, and I might change my opinion depending on what it says."
Oh okay, guess all LLMs are just fine then and we don't need to do any further development on them.
Why isn't the LLM under the wrong assumption? So I don't get from my tool what I need and it's still me at fault? I am not yet ready to bow to the AI overlords, sorry.
But are the links plausible text given the training data?
If the purpose is to accurately cite sources, how is it even possible to hallucinate them? Seems like folks are expecting way too much from these tools. They are not intelligent. Useful, perhaps.
Seems that's just expecting things that LLMs were not designed for.
It's a token producer based on trained weights, it doesn't use any sources.
Even if it were "fixed" so that it only generates URLs that exist, it's still incorrect because it did not use any sources so those URLs are not sources.
Then let's face it: LLMs were not designed to give proper answers. Now that we settled this and the emperor is obviously naked, what?
[dead]
I have search enabled 100% of the time with ChatGPT and would never go back to raw-dogging LLM citations. O3 especially has passed the threshold of “not always annoying”. Had an argument with Gemini yesterday where it was insisting on some hallucinated implementation of a function even while giving me a GitHub link to the correct source.
[flagged]
Are you being paid to post here? They're giving me nothing. Cheapskates.
if this refers to how often i post in bursts, i have wicked insomnia. I really should remove my credentials from the HN app on my phone. if this is what you meant, you're the first person to call me out!
I wouldn't use it in a workplace either, perhaps not at all, but here is a pseudonymous forum. The expectations—or repercussions—of decorum aren't the same.
I think the usage and all its connotations are perfectly cromulent here.
Languages evolve and words get new meanings all the time.
Yeah the meaning was what I said until about 2 weeks ago when someone went viral talking about air travel without a cellphone.
Do vulgarities often become accepted?
Are we allowed to say "sex" or "kill"? Or do we have to start s*lf-c*nsoring everything?
Anyway, my last search to how to un-alive children processes gave me nothing. I wonder if those m*n pages are actually wr*tten by pr*f*ss*n*ls.
algospeak originated on Tiktok, with all that implies. You couldn't post videos about sexual assault, suicides, murders, any of that directly, so people started self-censoring and using words - unalive maybe came from Deadpool in 2013? I'm not real interested in the actual etymology right now.
and you jest but there's been pushes to change things like "master/slave", "kill", "whitelist/blacklist", and the like to something different. I don't know how much traction, and i may have been tricked by april 1st posts or something.
Some of them do, yes. Tone policing has always been around as well, just surprises me to happen in an online forum about "vulgarities".
I apologize if it came off as tone policing. "Raw dog" sounds innocuous, if a bit strange. There was a radio station that had an internet stream out of Topeka Kansas called "Raw Dog Radio" that was a comedy station/stream; as an example - but it was a direct reference to the sexual connotation. For people who don't know, and notice that "raw dogging" is entering zeitgeist, maybe they'll say it because it sounds silly. It was merely a warning about what it means. The "definition shift" really isn't, people are comparing taking a flight without water, cellphones, etc to sex without a condom (Doesn't matter, flew through the air at hundreds of miles per hour, don't care!) The person i replied to was comparing using an LLM without RAG to risky sex.
I'd also avoid saying "So. How are we going to fuck this pig?" in a meeting about infra problems.
This is trivial to overcome by using a REST client to verify the link through MCP, and by caching results it wouldn't even add much latency.
The key thing I got from this article is that the o3 and Claude 4 projects (I'm differentiating from the models here because the harness of tools around them is critical too) are massively ahead of GPT 4.1 and Gemini 2.5 when it comes to fact checking in a way that benefits from search and web usage.
The o3 finding matches my own experience: https://simonwillison.net/2025/Apr/21/ai-assisted-search/#o3...
Both o3 and Claude 4 have a crucial new ability: they can run tools such as their search tool as part of their "reasoning" phase. I genuinely think this is one of the most exciting new advances in LLMs in the last six months.
Products, not projects.
If anyone is interested in a larger sample size comparing how often LLMs confabulate answers based on provided texts, I have a benchmark at https://github.com/lechmazur/confabulations/. It's always interesting to test new models with it because the results can be unintuitive compared to those from my other benchmarks.
Useful benchmark. I noticed o3-high hallucinating too often for such a good model, but it is usually great with search. In my experience, Claude Opus & Sonnet 4 consistently lie, cheat, and try to hide their tracks. Maybe they are good in writing code but I don't trust them with other things.
It's not just that they get links wrong, it's how they get them wrong – like, totally fabricating them and then doubling down! A human messing up a citation is one thing, but this feels... different, almost like a creative act of deception, lol.
I do wonder about the role of test time compute in the blog post in terms of document understanding. A non reasoning output (or low test time compute setting) might easily misinterpret the text, but reasoning models can second guess, consider multiple objectives in turn, and can right the ship.
I note that Gemini 2.5 has one of the lowest confabulation/hallucination rates according to this benchmark [1], so am surprised by the results in the blog.
Also, I have found link hallucination and output quality improve when you restrict searches to, for example, only pubmed sources, and to provide the source link directly into the text (as opposed to Gemini deep research usual method for citation).
One reason, I think, is that unrestricted search will get the paper, the related blog posts and press releases, weight them as equal (and independent!) sources of a fact, when we know that nuance is lost in the latter, and maybe because it will then spend more test time compute in the quality sources, not the press-releases.
[1]https://github.com/lechmazur/confabulations/
Took some time to realize the SIFT toolbox mentioned in the article is not a Scale-Invariant Feature Transform toolbox.
> Why are we talking about “graduate and PhD-level intelligence” in these systems if they can’t find and verify relevant links
For exactly the same reason the author markets his tool as a research assistant
> It also models an approach that is less chatbot, and more research assistant in a way that is appropriate for student researchers, who can use it to aid research while coming to their own conclusions.
In such cases, I get better answers to questions starting with "What" and not "Did".
Prompt engineering!
I have a strange feeling: it seems that original insights and hallucinations are related. One seems to come very frequently with the other.
I've noticed that o3 is the one that lies with the most conviction (compared to Gemini Pro and Claude Sonnet). It will be the hardest to convince that it is wrong, will invent excuses and complex explanations for its lies, almost to a Trump level of lying and deception.
But it is also the one that provides the most interesting insights, that will look at what others don't see.
There might some kind deep truth in this correlation. Or it might be myself having an hallucination...