What's more alarming isn't that AI is limited to existing domain data, it's that when people push it to deviate outside those known data points it confidently hallucinates nonsense.
Models are not trained to self-evaluate. Or only as an afterthought during tuning. So they are poor at it.
It isn’t mysterious.
Humans are trained incrementally, educated informally and formally, and along the way tested by context and classroom. Evaluated by people in their circle and strangers. The training to evaluate ourselves is near constant.
Even then, many people habitually believe they understand things they clearly don’t.
And can even be hostile to feedback.
Many forms of hallucination are canonized, or socially encouraged at varying demographic scales. While others are idiosyncratic.
Once self-examination is a first-class part of models training process, I expect they will respond like they have to being trained on vast troves of information, which is far excel human beings.
But, until then, their poor self-assessment isn’t mysterious, it is a mundane result of being trained not to do that. Or only as a tuning after thought.
Not as different from humans as we might like to think.
And models are noticeably improving on this measure. Humans in my experiencing may be regressing.
And many of today's publicly-accessible platforms are designed to steer results away from nonsense back to... too much sense. "Hypernormal" is a good word for it. I've spent a lot of time prompt-yelling, "Please make something as weird as I'm imagining, stop veering back to what normal people want to see/read."
If I had a nickel for every AI-poisoned "researcher" I'd seen with a preprint full of nonsense buzzwords like "quantum fractal holographic resonance matrix"... well, I wouldn't be rich, but I'd probably at least have enough to buy a coffee.
The article presumes that the models we have today describing everything could still be subject to a major paradigm shift.
Maybe they could be, but it seems pretty unlikely. The edges of a lot of scientific understanding are now past practical applicability. The edges are essentially models of things impossible to test. In fact, relativity was only recently fully backed up with experimental data.
I don't think paradigm shifts have to be 'better' in some march-toward-progress sense, they can be lateral or even regressive in that way and still lead to longer-horizon improvements.
I think also what's practically applicable changes constantly. Perhaps we're truly at the End of Science, but empirically we've been wrong every other time we've said that. My money is that there's more race to run.
On that note, Terence Tao gave a good interview to Dwarkesh Patel talking about Kepler. He pointed out that the previous geocentric models were actually more accurate than Kepler's at the time, in part because they'd had so much complexity piled on to solve minor errors. Kepler's theory was more elegant, but at the time it wasn't necessarily a better model.
I think important paradigm shifts can often look like this - there's not necessarily a reason to expect them to be instantly optimal. Deep Learning vs 'good old-fashioned AI' is another example of this dichotomy; it took a long time for deep learning to establish itself.
> I don't think paradigm shifts have to be 'better'
But they do. Paradigm shifts happen because the new paradigm explains the unexplained and importantly also covers the old model. If prior data is unexplained with a paradigm shift, the shift will never be adopted.
> Perhaps we're truly at the End of Science
Who said that? Just because the core of our current models seem pretty rock steady doesn't mean there's not more science. It simply means that we can mostly just expect refining rather than radical discovery.
There will be sub-paradigm shifts, but there's likely not going to be major "relativity" moments from here on out.
> Paradigm shifts happen because the new paradigm explains the unexplained and importantly also covers the old model
Empirically it seems that paradigm shifts are more driven by deaths and retirement rather than improved fit to the data. Moreover the way that you reconcile old data with the new model can be contestable; it's not like everyone all at once says "oh this new model is clearly a strict superset of the previous one, time to adopt it". With all that said I think one could argue that this stuff is basically noise and that the process still 'trends toward progress' (and I'd agree). But I would say that the scale of noise can also be quite large relative to things a human might experience in their life. I was sort of imagining social-disruption (like a dark-age type regression) as the 'backwards paradigm shift'.
> but there's likely not going to be major "relativity" moments from here on out
I cannot understand how anyone treat this as something that can be objectively concluded; by definition these kinds of radical paradigm shifts are basically unforeseeable up until they happen. I called it the "End of Science" to draw a parallel to "End of History"-type thinking because both (IMO) take this view of "there will be no more revolutions, only incremental adjustments on an unshakeable core into infinity", which I feel is personally a 'vibes based' assessment of things. It's not even that I disagree with it so much as I feel like the statement is basically (and will always be) a pure guess, one which many people have made and been wrong about in the past.
> I cannot understand how anyone treat this as something that can be objectively concluded
Mostly because the room for the unexplained in physics is really small. It's possible that we end up finding some sort of big revelation about quantum physics that completely changes how we view relativity. But even in that case we are more likely to find that relativity is just a simplification of a more complex model with better explanatory power. Very much like how Newtonian physics still works really well from quite small things to anything most humans will deal with on earth. It's only when you start talking about uncommon experiences in extreme environments where relativity starts being a requirements to make the math work.
> there will be no more revolutions, only incremental adjustments on an unshakeable core into infinity
I guess I'm just more comfortable with that position. A lot of the revolutions in science circled around detecting and measure things previously immeasurable and unsee-able. The study of EMF exploded when it did because that's also when our ability to generate and measure electricity in more than just a party trick happened.
We are at a point where things are more of an unknown unknowns with no theoretical way to observe. The physics models at the fringes are mostly centered around things we can't measure.
There just aren't interactions we can't currently predict. The only one I know about is radioactive decay.
And a lot of this shows in modern society. In physics, the last major paradigm shift was relativity. That's a nearly 100 year old model at this point. Everything we have currently is just incremental improvements on the physics model.
I don't think this is because we just aren't as smart today as we once were. Quiet the opposite. There are far more people on the planet. There are almost certainly a lot more "Einsteins" trying to find a new paradigm and they've simply failed over the decades because it's seemingly increasingly unlikely that there is something to find.
> It simply means that we can mostly just expect refining rather
The practical issue is if there will enough funds for just "refining", instead of "paradigm shifts", which I understand as new and "exciting" discoveries. I'm not a scientist, of course, this is just my layman's understanding.
Physics is a bit of a special case. This certainly doesn't apply to, say, biology, medicine, cognition, not to mention any of the social sciences—i.e. most research.
I'm also a little skeptical about the practical value of the bleeding edge of both experimental and theoretical physics. Interesting? Sure.
cognition is just a special case of medicine which is a special case of biology which is a special case of chemistry which is a special case of physics.
And the closer you get to physics, the less likely any sort of major paradigm shift will be discovered (though the article focuses pretty heavily on physics which is why I do as well).
But even in those fields, there are core parts that aren't likely to ever see any sort of paradigm shift. For example, in biology, I doubt we'll see a shift from evolution as it'll be impossible for a new model to also explain what evolution does.
I agree that at the edges you'll possibly see more paradigm shifts and discovery, but those are all going to be working from things that will not see paradigm shifts. For example, biology can't escape things like single celled organisms made up from atoms and chemical compounds.
But ultimately, what I disagree with in the article is the notion that discovery won't ultimately be a process of hypernormalization. In medicine, we are unlikely to see a new paradigm that isn't germ theory. When it comes to the research, it'll mostly be focused on finding new compounds and delivery mechanisms for treatment rather than finding a new paradigm for how to treat a disease.
The softer sciences are the only place where you might find new paradigms, but that's simply because the data itself is so squishy and poor anyways that it's easy to shift around. There it's less a question of the science and more of the utility of the model (regardless of whether or not it aligns with reality).
> article presumes ... everything could still be subject to a major paradigm shift. ...seems pretty unlikely
Alternatively: there's plenty of mainstream, accepted science that's plain, flat out, provably wrong. Yet, it is against good taste (job security, people's feelings, status quo bias, etc.) to point this out.
Hence, it can actually be tricky to catch wind of, or get a grasp on, such issues to begin with, much less pursue such issues toward meaningful, published, recognized change in understanding (that is to say: paradigm shift).
I'd name some examples, but you wouldn't believe me.
With respect to the article, it seems the current LLMs can (though, obviously, do not necessarily have to) return text that appears to reason (pretty reasonably!) about paradigm shifts, when given the context required and nudged quite forcefully toward particular directions. But, as the article seems to indicate, the LLMs seem to not tend toward finding, investigating, and reporting on paradigm shifts all on their own very much. (But maybe part of that is intrinsic to how they are programmed and/or their context?)
> there's plenty of mainstream, accepted science that's plain, flat out, provably wrong. Yet, it is against good taste (read: job security, people's feelings, etc.) to point this out.
I highly doubt that.
There are a lot of people that think they've proving the mainstream wrong. But more often than not, it's cranks using bad non-repeated tests. These bad tests are propped up, ironically, because of people's feelings and job security more than a built up body of evidence.
They also almost always have to ignore the mainstream body of evidence and just say it's wrong and bad because of a conspiracy.
For example, plenty of creationists believe they have irrefutable evidence that evolution is provably wrong. It's usually a few cherry picked or poorly interpreted results or sometimes just flat out lying. And often they simply flat out lie about the existing body of evidence that support evolution.
Another example is the antivaxx movement. Wakefield and RFK both built careers that made them a lot of money talking about how the mainstream was wrong. Even when the industry adopted some of the recommendations (abandoning Thimerosal), they simply ignored the fact that further data didn't support their claims.
> In fact, relativity was only recently fully backed up with experimental data.
Can you elaborate on the assertion you made here? In addition to the important points @elbasti made about tests performed approximately a century ago, what does it even mean for a scientific theory to be "fully backed up"? Such theories can be tested and the tests either passed or the theory disproven but it's not possible to _prove_ such a theory. And to some extent we already know that relativity cannot be the final answer because it doesn't mesh well with quantum mechanics (which has been experimentally tested substantially, arguably even more than relativity has).
Nope. The edge for a lot of interesting science is difficult to use scientific software and scientists not using basic statistical techniques correctly.
which contains Heathrow Terminals 1, 2, 3, 4 & 5 on the Picadilly line. For about 15 seconds I imagined a world where Heathrow has had 5 terminals since 1933, then I read the map itself: "Recreated by Arthurs D". Phew.
Awesome example of improving information conveyance through abstractions though!
> AI could repeat this pattern at a larger scale — generating faster results within the existing paradigm, while the structural conditions for disruptive science remain unchanged or worsen.
Worsen. LLMs discard/loses and mixes data on their statistical "compression" to create their vectorial database model. Across the time, successive feed back will be homologous to create a jpg image sourcing a jpg image that was created from another jpg image, through this "gaussian" loop.
Those faster (but worst) results will degrade real valuable data and science at a speed/rate that will statistically discard good done science on a regular basis, systematically.
My hot take is that mathematical and scientific 'soundness' is ultimately more of an aesthetic preference than an objective quality of reality. Good science makes sense to humans, and 'what makes sense' is ultimately what fits satisfyingly in your brain. There's nothing inherently wrong with an enormous epicycle model of reality from the perspective of the God of Math; so long as your formal system is consistent and expressive enough to represent everything then meh, it's a model. But the model that humans want to elevate to canonical status has far stricter requirements, and ultimately it's the one which the majority of sufficiently credentialed tastemakers decide is 'best'. Parsimony works well in physics where you have closed form expressions for all your stuff, but the biology cases are so much messier because it turns out that sometimes reality isn't parsimonious. All this to say that good science is a matter of taste, and while AI can gist the broad strokes of taste I've yet to see it take on the role of genuine tastemaker.
If biology, or some other subject area, is inherently, irredeemably hard to explain, and always will be, then I don't care about it much, because it doesn't mean very much. I care about explanations, not "reality" in the sense of every arbitrary muddle of knotted nerve fibers and confused flour beetles. If all the world's messy, inexplicable things were to gang up and cause us trouble such that we have to pay them attention, we can still ultimately deal with them in the ways that matter by using clarity and the things we can explain well.
As another commenter mentioned, the point of the story from Borges is that a perfectly detailed map is rather useless, because you need abstraction (it's a repeating theme in some other stories from him like the Library of Babel, and Funes the Memorious). LLMs are likely already able to exhaust the conceptual space for any given field, but some judgement is still going to be required about what to pursue. In biology and other fields this problem is even bigger because experimentation is so difficult and expensive.
The process of judgement and resource allocation will still be human for quite a while, but it's quite likely some humans will outsource their responsibility to AI to cut corners.
meh. I would be happier with this article if it demonstrated familiarity with the source material. "Del Rigor en la Ciencia" (On Exactitude in Science) was Borges (hilarious) investigation into Korzybski's General Semantics, a fact that was surprisingly absent from the text. Borges implied "the map is not the territory," but Korzybski actually came out in said it a decade or so before Borges wrote the story in question. Understanding the themes of Borges story is greatly informed by a passing familiarity with Korzybski's work. WorldCat tells me the 4th edition of "Science and Sanity" is widely held by libraries, and if you're interested in the assertions made in this article, you might enjoy reading (at least parts of) it.
The author completely missed the point Borges (and Korzybski) made about the utility of maps. Maps (according to both) are abstractions which allow the user to ignore irrelevant aspects of reality so other, more interesting facets come into sharper resolve. This might be why Beck's London Tube map is so well regarded. It allows the user to easily ignore aspects that are not germane to the task of deciding where and when to get on and off the tube.
But is a scientific paradigm like a map? Certainly it is an abstraction, if we take Kuhn's definition. If you're interested, I can recommend both "The Structure of Scientific Revolutions" and "The Essential Tension : Selected Studies in Scientific Tradition and Change" by Kuhn.
If you're going to talk about new paradigms, Maxwell is a great example, but his story is not complete without mentioning Heaviside, Michelson and Morley.
Also... I bristle at the phrase "Hypernormal Science." It's also introduced without definition or reference. Collins, et al describe it as distinct from (though seemingly related to) the word "Hypernormal" as coined by Yurchak in "Everything Was Forever Until It Was No More."
Or if you're short on time, you can get an entertaining (though not as enlightening) description from Adam Curtis' 2016 documentary HyperNormalization. You won't come away from it with a better understanding of AI, General Semantics or Popperian falsifiability, but it has a striking visual style and a very good soundtrack. And may lead to a better understanding of "hypernormal science."
And getting back to the Michelson-Morley experiment. The author talks about how their results did not cause the scientific establishment to abandon the concept of luminiferous aether. Certainly there is conservatism in science. Gigging science-monkeys tend to want to see interesting results replicated.
And this was one of the issues with the MM experiment. It took a while to replicate. We're MUCH better at replicating it these days and I would guess that thousands (maybe hundreds) of physics undergrads did this very task last year. But we've had over a century of pedagogical experience w/ this experiment. We know how to structure it to get the results we want. This was not the case in the late 1800s and in fact, several early attempts to replicate the experiment suggested the existence of an aether which was drifting slowly towards Cleveland.
And what does it say that heat flow, fluid flow, diffusion and electrostatics share equations? Does it say there's something fundamental in reality? Or does it say there's something fundamental in the way we model reality?
That being said... I think the author has hit upon something here... people are often wary of evidence which contradicts experience, even when that evidence (and not experience) is more correct.
But each of the examples he provides glosses over the process by which new paradigms overrode the old.
I deeply appreciate the author avoiding slavish fealty to fashionable AI trends. He probably could have gone further to describe more representational weakness of ESM3 and GNoME.
I fear, however, he has missed the point. It's less interesting to describe the messy ways in which AI fails than to describe the messy ways in which humans succeed. The process by which paradigms shift is messy, social and fundamentally human. It often has more to do with qualitative explanations than quantitative science. Science, as a human endeavor, is very much a story-telling exercise.
"Designing AI for Disruptive Science" is a bit market-ey, but "AI Risks 'Hypernormal' Science" is just a trimmed section heading "Current AI Training Risks Hypernormal Science".
It's been shown in other fields that training models on the output of other models produces subtly broken models, not a flattening to the statistical mean. Why would science be different?
I don't think this argument is wrong. But also debatable. At the end of the day, we are talking about the manifold of the reality (as compressed by LLM through language abstraction). It is remain to be seen if supervised fine-tuning on the best human can produce would nag the model enough to generate surprising findings.
We know the pre-trained models do tend to revert to mean, but I don't think that's enough to say SFT / RL models will do the same, although some might argue RL only sharpens the distribution, even for that, I am skeptical about that paper.
I find it funny how people are so concerned that AI cannot innovate, that AI coding agents only give the most bland solutions to any problem etc. when the next step in OpenAI's 5 stages to AGI is literally called "Innovators".
Do you mean to say my current AI workflow doesn't involve secret agents running around Bond-style sabotaging those that'd impede my efforts to build a super secret RSS forwarder that pig-lantinifies the text before sending it to my client?
My two step plan is to go to sleep and then wake up the next day and be a billionaire. Surely because that's my stated next step that means when I wake up tomorrow I'll be rich.
What's more alarming isn't that AI is limited to existing domain data, it's that when people push it to deviate outside those known data points it confidently hallucinates nonsense.
Models are not trained to self-evaluate. Or only as an afterthought during tuning. So they are poor at it.
It isn’t mysterious.
Humans are trained incrementally, educated informally and formally, and along the way tested by context and classroom. Evaluated by people in their circle and strangers. The training to evaluate ourselves is near constant.
Even then, many people habitually believe they understand things they clearly don’t.
And can even be hostile to feedback.
Many forms of hallucination are canonized, or socially encouraged at varying demographic scales. While others are idiosyncratic.
Once self-examination is a first-class part of models training process, I expect they will respond like they have to being trained on vast troves of information, which is far excel human beings.
But, until then, their poor self-assessment isn’t mysterious, it is a mundane result of being trained not to do that. Or only as a tuning after thought.
Not as different from humans as we might like to think.
And models are noticeably improving on this measure. Humans in my experiencing may be regressing.
And many of today's publicly-accessible platforms are designed to steer results away from nonsense back to... too much sense. "Hypernormal" is a good word for it. I've spent a lot of time prompt-yelling, "Please make something as weird as I'm imagining, stop veering back to what normal people want to see/read."
So much nonsense.
If I had a nickel for every AI-poisoned "researcher" I'd seen with a preprint full of nonsense buzzwords like "quantum fractal holographic resonance matrix"... well, I wouldn't be rich, but I'd probably at least have enough to buy a coffee.
The article presumes that the models we have today describing everything could still be subject to a major paradigm shift.
Maybe they could be, but it seems pretty unlikely. The edges of a lot of scientific understanding are now past practical applicability. The edges are essentially models of things impossible to test. In fact, relativity was only recently fully backed up with experimental data.
I don't think paradigm shifts have to be 'better' in some march-toward-progress sense, they can be lateral or even regressive in that way and still lead to longer-horizon improvements.
I think also what's practically applicable changes constantly. Perhaps we're truly at the End of Science, but empirically we've been wrong every other time we've said that. My money is that there's more race to run.
On that note, Terence Tao gave a good interview to Dwarkesh Patel talking about Kepler. He pointed out that the previous geocentric models were actually more accurate than Kepler's at the time, in part because they'd had so much complexity piled on to solve minor errors. Kepler's theory was more elegant, but at the time it wasn't necessarily a better model.
I think important paradigm shifts can often look like this - there's not necessarily a reason to expect them to be instantly optimal. Deep Learning vs 'good old-fashioned AI' is another example of this dichotomy; it took a long time for deep learning to establish itself.
I like this a lot. The Innovators Dilemma for science.
The new simpler tool always competes with highly adapted complex tools to get to a region of value generation.
Starting where it’s greater simplicity, despite less complementary adaptations, is of great advantage.
Then slowly accumulates its own version of practical complements that let it excel overall.
> I don't think paradigm shifts have to be 'better'
But they do. Paradigm shifts happen because the new paradigm explains the unexplained and importantly also covers the old model. If prior data is unexplained with a paradigm shift, the shift will never be adopted.
> Perhaps we're truly at the End of Science
Who said that? Just because the core of our current models seem pretty rock steady doesn't mean there's not more science. It simply means that we can mostly just expect refining rather than radical discovery.
There will be sub-paradigm shifts, but there's likely not going to be major "relativity" moments from here on out.
> Paradigm shifts happen because the new paradigm explains the unexplained and importantly also covers the old model
Empirically it seems that paradigm shifts are more driven by deaths and retirement rather than improved fit to the data. Moreover the way that you reconcile old data with the new model can be contestable; it's not like everyone all at once says "oh this new model is clearly a strict superset of the previous one, time to adopt it". With all that said I think one could argue that this stuff is basically noise and that the process still 'trends toward progress' (and I'd agree). But I would say that the scale of noise can also be quite large relative to things a human might experience in their life. I was sort of imagining social-disruption (like a dark-age type regression) as the 'backwards paradigm shift'.
> but there's likely not going to be major "relativity" moments from here on out
I cannot understand how anyone treat this as something that can be objectively concluded; by definition these kinds of radical paradigm shifts are basically unforeseeable up until they happen. I called it the "End of Science" to draw a parallel to "End of History"-type thinking because both (IMO) take this view of "there will be no more revolutions, only incremental adjustments on an unshakeable core into infinity", which I feel is personally a 'vibes based' assessment of things. It's not even that I disagree with it so much as I feel like the statement is basically (and will always be) a pure guess, one which many people have made and been wrong about in the past.
> I cannot understand how anyone treat this as something that can be objectively concluded
Mostly because the room for the unexplained in physics is really small. It's possible that we end up finding some sort of big revelation about quantum physics that completely changes how we view relativity. But even in that case we are more likely to find that relativity is just a simplification of a more complex model with better explanatory power. Very much like how Newtonian physics still works really well from quite small things to anything most humans will deal with on earth. It's only when you start talking about uncommon experiences in extreme environments where relativity starts being a requirements to make the math work.
> there will be no more revolutions, only incremental adjustments on an unshakeable core into infinity
I guess I'm just more comfortable with that position. A lot of the revolutions in science circled around detecting and measure things previously immeasurable and unsee-able. The study of EMF exploded when it did because that's also when our ability to generate and measure electricity in more than just a party trick happened.
We are at a point where things are more of an unknown unknowns with no theoretical way to observe. The physics models at the fringes are mostly centered around things we can't measure.
There just aren't interactions we can't currently predict. The only one I know about is radioactive decay.
And a lot of this shows in modern society. In physics, the last major paradigm shift was relativity. That's a nearly 100 year old model at this point. Everything we have currently is just incremental improvements on the physics model.
I don't think this is because we just aren't as smart today as we once were. Quiet the opposite. There are far more people on the planet. There are almost certainly a lot more "Einsteins" trying to find a new paradigm and they've simply failed over the decades because it's seemingly increasingly unlikely that there is something to find.
> Empirically it seems that paradigm shifts are more driven by deaths and retirement rather than improved fit to the data.
Indeed, Kuhn's own work acknowledged this.
> It simply means that we can mostly just expect refining rather
The practical issue is if there will enough funds for just "refining", instead of "paradigm shifts", which I understand as new and "exciting" discoveries. I'm not a scientist, of course, this is just my layman's understanding.
> relativity was only recently fully backed up with experimental data.
Gravitational deflection (General relativity) received pretty important confirmation in 1919, only 8 years after Einstein first proposed it.
Time dilation (Special realativity) was experimentally confirmed in 1932.
Physics is a bit of a special case. This certainly doesn't apply to, say, biology, medicine, cognition, not to mention any of the social sciences—i.e. most research.
I'm also a little skeptical about the practical value of the bleeding edge of both experimental and theoretical physics. Interesting? Sure.
cognition is just a special case of medicine which is a special case of biology which is a special case of chemistry which is a special case of physics.
And the closer you get to physics, the less likely any sort of major paradigm shift will be discovered (though the article focuses pretty heavily on physics which is why I do as well).
But even in those fields, there are core parts that aren't likely to ever see any sort of paradigm shift. For example, in biology, I doubt we'll see a shift from evolution as it'll be impossible for a new model to also explain what evolution does.
I agree that at the edges you'll possibly see more paradigm shifts and discovery, but those are all going to be working from things that will not see paradigm shifts. For example, biology can't escape things like single celled organisms made up from atoms and chemical compounds.
But ultimately, what I disagree with in the article is the notion that discovery won't ultimately be a process of hypernormalization. In medicine, we are unlikely to see a new paradigm that isn't germ theory. When it comes to the research, it'll mostly be focused on finding new compounds and delivery mechanisms for treatment rather than finding a new paradigm for how to treat a disease.
The softer sciences are the only place where you might find new paradigms, but that's simply because the data itself is so squishy and poor anyways that it's easy to shift around. There it's less a question of the science and more of the utility of the model (regardless of whether or not it aligns with reality).
> article presumes ... everything could still be subject to a major paradigm shift. ...seems pretty unlikely
Alternatively: there's plenty of mainstream, accepted science that's plain, flat out, provably wrong. Yet, it is against good taste (job security, people's feelings, status quo bias, etc.) to point this out.
Hence, it can actually be tricky to catch wind of, or get a grasp on, such issues to begin with, much less pursue such issues toward meaningful, published, recognized change in understanding (that is to say: paradigm shift).
I'd name some examples, but you wouldn't believe me.
With respect to the article, it seems the current LLMs can (though, obviously, do not necessarily have to) return text that appears to reason (pretty reasonably!) about paradigm shifts, when given the context required and nudged quite forcefully toward particular directions. But, as the article seems to indicate, the LLMs seem to not tend toward finding, investigating, and reporting on paradigm shifts all on their own very much. (But maybe part of that is intrinsic to how they are programmed and/or their context?)
> ... there's plenty of mainstream, accepted science that's plain, flat out, provably wrong : : > I'd name some examples, but you wouldn't believe me.
I probably would not. You would probably be wrong
> there's plenty of mainstream, accepted science that's plain, flat out, provably wrong. Yet, it is against good taste (read: job security, people's feelings, etc.) to point this out.
I highly doubt that.
There are a lot of people that think they've proving the mainstream wrong. But more often than not, it's cranks using bad non-repeated tests. These bad tests are propped up, ironically, because of people's feelings and job security more than a built up body of evidence.
They also almost always have to ignore the mainstream body of evidence and just say it's wrong and bad because of a conspiracy.
For example, plenty of creationists believe they have irrefutable evidence that evolution is provably wrong. It's usually a few cherry picked or poorly interpreted results or sometimes just flat out lying. And often they simply flat out lie about the existing body of evidence that support evolution.
Another example is the antivaxx movement. Wakefield and RFK both built careers that made them a lot of money talking about how the mainstream was wrong. Even when the industry adopted some of the recommendations (abandoning Thimerosal), they simply ignored the fact that further data didn't support their claims.
> In fact, relativity was only recently fully backed up with experimental data.
Can you elaborate on the assertion you made here? In addition to the important points @elbasti made about tests performed approximately a century ago, what does it even mean for a scientific theory to be "fully backed up"? Such theories can be tested and the tests either passed or the theory disproven but it's not possible to _prove_ such a theory. And to some extent we already know that relativity cannot be the final answer because it doesn't mesh well with quantum mechanics (which has been experimentally tested substantially, arguably even more than relativity has).
Nope. The edge for a lot of interesting science is difficult to use scientific software and scientists not using basic statistical techniques correctly.
I wasn’t aware of the map empire, thank you!
Taking away some complexity comes at a price, and for some people, it’s hard to see that it outweighs the practicality.
Reminds me of the coastline paradox - https://en.wikipedia.org/wiki/Coastline_paradox
I got stuck for a minute on the caption "Harry Beck’s 1933 map of the London Underground" to: https://substackcdn.com/image/fetch/$s_!VsWm!,f_auto,q_auto:...
which contains Heathrow Terminals 1, 2, 3, 4 & 5 on the Picadilly line. For about 15 seconds I imagined a world where Heathrow has had 5 terminals since 1933, then I read the map itself: "Recreated by Arthurs D". Phew.
Awesome example of improving information conveyance through abstractions though!
> AI could repeat this pattern at a larger scale — generating faster results within the existing paradigm, while the structural conditions for disruptive science remain unchanged or worsen.
Worsen. LLMs discard/loses and mixes data on their statistical "compression" to create their vectorial database model. Across the time, successive feed back will be homologous to create a jpg image sourcing a jpg image that was created from another jpg image, through this "gaussian" loop.
Those faster (but worst) results will degrade real valuable data and science at a speed/rate that will statistically discard good done science on a regular basis, systematically.
IMHO.
Can you explain further how we can prevent this
My hot take is that mathematical and scientific 'soundness' is ultimately more of an aesthetic preference than an objective quality of reality. Good science makes sense to humans, and 'what makes sense' is ultimately what fits satisfyingly in your brain. There's nothing inherently wrong with an enormous epicycle model of reality from the perspective of the God of Math; so long as your formal system is consistent and expressive enough to represent everything then meh, it's a model. But the model that humans want to elevate to canonical status has far stricter requirements, and ultimately it's the one which the majority of sufficiently credentialed tastemakers decide is 'best'. Parsimony works well in physics where you have closed form expressions for all your stuff, but the biology cases are so much messier because it turns out that sometimes reality isn't parsimonious. All this to say that good science is a matter of taste, and while AI can gist the broad strokes of taste I've yet to see it take on the role of genuine tastemaker.
https://en.wikipedia.org/wiki/99_Variations_on_a_Proof
If biology, or some other subject area, is inherently, irredeemably hard to explain, and always will be, then I don't care about it much, because it doesn't mean very much. I care about explanations, not "reality" in the sense of every arbitrary muddle of knotted nerve fibers and confused flour beetles. If all the world's messy, inexplicable things were to gang up and cause us trouble such that we have to pay them attention, we can still ultimately deal with them in the ways that matter by using clarity and the things we can explain well.
[dead]
>...nothing inherently wrong with an enormous epicycle model of reality...
That would be pretty hopeless for launching satellites and the like.
What use does the God of Math have for satellites and the like?
Well maybe not much for the God of math but Newtonian mechanics is more practical for life, beyond just matters of taste.
The piano accompaniment and human narration are a nice touch.
As another commenter mentioned, the point of the story from Borges is that a perfectly detailed map is rather useless, because you need abstraction (it's a repeating theme in some other stories from him like the Library of Babel, and Funes the Memorious). LLMs are likely already able to exhaust the conceptual space for any given field, but some judgement is still going to be required about what to pursue. In biology and other fields this problem is even bigger because experimentation is so difficult and expensive.
The process of judgement and resource allocation will still be human for quite a while, but it's quite likely some humans will outsource their responsibility to AI to cut corners.
meh. I would be happier with this article if it demonstrated familiarity with the source material. "Del Rigor en la Ciencia" (On Exactitude in Science) was Borges (hilarious) investigation into Korzybski's General Semantics, a fact that was surprisingly absent from the text. Borges implied "the map is not the territory," but Korzybski actually came out in said it a decade or so before Borges wrote the story in question. Understanding the themes of Borges story is greatly informed by a passing familiarity with Korzybski's work. WorldCat tells me the 4th edition of "Science and Sanity" is widely held by libraries, and if you're interested in the assertions made in this article, you might enjoy reading (at least parts of) it.
https://search.worldcat.org/title/369632
The author completely missed the point Borges (and Korzybski) made about the utility of maps. Maps (according to both) are abstractions which allow the user to ignore irrelevant aspects of reality so other, more interesting facets come into sharper resolve. This might be why Beck's London Tube map is so well regarded. It allows the user to easily ignore aspects that are not germane to the task of deciding where and when to get on and off the tube.
But is a scientific paradigm like a map? Certainly it is an abstraction, if we take Kuhn's definition. If you're interested, I can recommend both "The Structure of Scientific Revolutions" and "The Essential Tension : Selected Studies in Scientific Tradition and Change" by Kuhn.
https://search.worldcat.org/title/4660423077
https://search.worldcat.org/title/3034084
Calling scientific paradigms maps isn't wrong, per se, but it does create more of a meta-metaphor, and a weak one at that.
Also. No. Maxwell did not replace a patchwork of equations with four short ones. That was Heaviside.
https://en.wikipedia.org/wiki/Oliver_Heaviside
Something we don't mention in polite society these days is that Maxwell proposed electromagnetic waves as propagating through an aether:
https://en.wikisource.org/wiki/A_Treatise_on_Electricity_and...
If you're going to talk about new paradigms, Maxwell is a great example, but his story is not complete without mentioning Heaviside, Michelson and Morley.
Also... I bristle at the phrase "Hypernormal Science." It's also introduced without definition or reference. Collins, et al describe it as distinct from (though seemingly related to) the word "Hypernormal" as coined by Yurchak in "Everything Was Forever Until It Was No More."
https://direct.mit.edu/posc/article-abstract/31/2/262/112751...
https://search.worldcat.org/title/1572419463
Or if you're short on time, you can get an entertaining (though not as enlightening) description from Adam Curtis' 2016 documentary HyperNormalization. You won't come away from it with a better understanding of AI, General Semantics or Popperian falsifiability, but it has a striking visual style and a very good soundtrack. And may lead to a better understanding of "hypernormal science."
https://youtu.be/u1Tp-ryQPFI
And getting back to the Michelson-Morley experiment. The author talks about how their results did not cause the scientific establishment to abandon the concept of luminiferous aether. Certainly there is conservatism in science. Gigging science-monkeys tend to want to see interesting results replicated.
And this was one of the issues with the MM experiment. It took a while to replicate. We're MUCH better at replicating it these days and I would guess that thousands (maybe hundreds) of physics undergrads did this very task last year. But we've had over a century of pedagogical experience w/ this experiment. We know how to structure it to get the results we want. This was not the case in the late 1800s and in fact, several early attempts to replicate the experiment suggested the existence of an aether which was drifting slowly towards Cleveland.
And what does it say that heat flow, fluid flow, diffusion and electrostatics share equations? Does it say there's something fundamental in reality? Or does it say there's something fundamental in the way we model reality?
That being said... I think the author has hit upon something here... people are often wary of evidence which contradicts experience, even when that evidence (and not experience) is more correct.
But each of the examples he provides glosses over the process by which new paradigms overrode the old.
I deeply appreciate the author avoiding slavish fealty to fashionable AI trends. He probably could have gone further to describe more representational weakness of ESM3 and GNoME.
I fear, however, he has missed the point. It's less interesting to describe the messy ways in which AI fails than to describe the messy ways in which humans succeed. The process by which paradigms shift is messy, social and fundamentally human. It often has more to do with qualitative explanations than quantitative science. Science, as a human endeavor, is very much a story-telling exercise.
Please don't editorialize titles unless they're clearly clickbait.
"Designing AI for Disruptive Science" is a bit market-ey, but "AI Risks 'Hypernormal' Science" is just a trimmed section heading "Current AI Training Risks Hypernormal Science".
Thanks, but please email us (hn@ycombinator.com) when you see things like this, so we can take action more quickly.
Will do. I wanted mostly to (re)inform the submitter.
[flagged]
[dead]
[flagged]
It's been shown in other fields that training models on the output of other models produces subtly broken models, not a flattening to the statistical mean. Why would science be different?
I don't think this argument is wrong. But also debatable. At the end of the day, we are talking about the manifold of the reality (as compressed by LLM through language abstraction). It is remain to be seen if supervised fine-tuning on the best human can produce would nag the model enough to generate surprising findings.
We know the pre-trained models do tend to revert to mean, but I don't think that's enough to say SFT / RL models will do the same, although some might argue RL only sharpens the distribution, even for that, I am skeptical about that paper.
I find it funny how people are so concerned that AI cannot innovate, that AI coding agents only give the most bland solutions to any problem etc. when the next step in OpenAI's 5 stages to AGI is literally called "Innovators".
It's marketing.
Do you mean to say my current AI workflow doesn't involve secret agents running around Bond-style sabotaging those that'd impede my efforts to build a super secret RSS forwarder that pig-lantinifies the text before sending it to my client?
My two step plan is to go to sleep and then wake up the next day and be a billionaire. Surely because that's my stated next step that means when I wake up tomorrow I'll be rich.
At no risk to myself I will try your plan and if it doesn't work you owe me your billion.