The weights start with a random manifold.
The training takes data and shapes the manifold, weight by weight, in many cycles.
Once the training is the done manifold is fixed.
When a new inference has to be done the query(q) is projected in the manifold space.
This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length.
Which(qw+i) is dropped qw+n times to output a final response of n length.
The gravity is created by repeated multiplication(of the weights/input) to find out how the projected embeddings should fall according to the manifold in the GPU.
It's like a giant plinko board where the shape of the original disc guides how it falls through the apparatus, and the apparatus has been tuned so that different discs end up in the exits we want them to
Compression is the reason why these Models are able to learn and understand.
My brain is doing the exact same thing.
I learned enough to compress concepts like a bike and what a bike does and for what i can use a bike.
Ask a LLM and it will answer you similiar to humans.
Blind people learn concepts of bikes too and in a smiliar way: by description.
LLMs just have so much data in form of text available and are able to ingest all of this, that the LLM compression algorithm doesn't has to be that good/finetuned than ours.
But I would assume that Yann LeCun's JEPA or other breakthroughs in the next few years will get us there.
The man posits that clicking is instinctual for blind people but they are told to quiet down in class and most never develop their echolocation abilities
I don’t think this analogy holds. The whole way through the processing pipeline in the brain, different sensory data is ingested separately and processed separately; and we still don’t understand how that data is then integrated into a cohesive experience.
LLMs have the same fundamental input regardless of modality, tokens. There is a preprocessing step before the “brain”, which is more akin to some super-synesthesia where all senses are translated into sound before becoming experience.
A blind person has touched warm and hot things and gotten burned before, and then they are told lava is this molten liquid that is even hotter than anything they have touched. That is enough for them to understand.
A blind person that never touched a hot object wouldn't really know though, there is a reason we dismiss talk from people who lack experience.
You don't know that. Yo don't know what someone would think if you tell them the general concept of cold and warm.
The reaction you should have, the feeling etc.
I asked chatgpt how it would describe a scene without mentioning temperature. It was very good in describing what a human would describe.
I'm aware of the bias we have against LLMs but I think people just underestimate how much data is there.
I'm not saying a robot wouldn't be better with this information or an LLM and they actually use temperature sensors for robots so they can control movement speed and dexterity with overheating elements but the gap is small.
think of it like this: the goal of LLMs (im saying LLM as shorthand for all of the AI algs) is to replicate human output / work, not so much to capture the human experience. so in order to do things, like communicate concepts, play a role in a scene, make choices, we project our human experience down into a lower resolution / constrained space to generate an output.
the question then is whether or not LLMs can mimic the projection of human experience as well as the real thing or not. My hypothesis is no for total general replication, but in certain fixed problem spaces I think its yes and the set of these spaces is growing. will it grow enough to encompass all practical work? not sure yet
AI tools are more like an author writing a character than a human revealing truths about their experiences. the difference between the LLM and an actual author is that the LLM kind of starts off with a 'flanderized' version of the character while an author can sort of blend personal experience and character to get at real human relatable decisions for the character. the result can be indistinguishable, because if the LLM is too predictable, too flanderized, we can inject artificial randomness to it to simulate personality.
you end up with unsolvable debates like movie critics have. "This character wouldnt make that decision in that situation if they were a real person, it doesnt make sense" vs real humans making irrational decisions.
ultimately i think the human experience is something we learn about with each other as we live it, and to think were any good at identifying it from sufficiently good imposters is putting the cart before the horse. once i know something is done by an actual human, my understanding of the world can shift. we dont decide what it means to be human, we accept what humanity is and try to work with it. So the philosophical debate on AI experience is moot to me, its not human therefor it teaches me nothing about humans and can replace exactly 0 human activity for me. Still can be a useful tool though
In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.
> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with.
It’s funny, because I thought you were talking about humans here when you wrote this. We know some things about how our bodies encode information that is sent to the brain, and we know some things about how neurons receive information and act on it, but after that we get too tired and give up on how the brain works and treat it like a miracle.
It’s like we desperately want to believe our consciousness is not just electrical impulses in our brain, and we want to ascribe agency and uniqueness to the physical processes going on in our head.
There's definitely a sizable contingent of people who desperately want to believe consciousness is just electrical impulses in our brain. Because "what else could it be"? The fact is that we just don't know, and "abiding in the not-knowing" is for many the most uncomfortable thing ever. Especially for the curious- and rational-minded people this forum tends to attract. I'm one of them, too.
It is basically reductionism, or in the extreme atomism, and (as I understand it) it has largely fallen out of fashion both in the philosophy of science and in the philosophy of mind, since it heyday in the 1970s. And I don’t see it coming back into fashion any time soon.
I can definitely see why reductionism would appeal to an educated public. We learn about the periodic table; we are able to break sentences down to words, down to letters; to break an executable down to binary code, to machine instruction, to electrical currents flowing through semiconductors. Why shouldn’t we be able to do the same with conscious thoughts? It is certainly an appealing thought process.
As I understand it, reductionism started to fall out of favor because of the rise of quantum mechanics and chaos theory, where we have a lot of weird phenomena which cannot be explained by reducing the particles down to the sub-sub atomic (or rather they are better explained by describing the interactions directly).
Quantum Mechanics and Chaos Theory don't preclude reductionism, and I wouldn't say it has fallen out of favor as a whole. Certain types certainly have, but not the overall idea.
Also, nothing about the idea of the mind only being made up of physical processes means things have to be deterministic.
It doesn’t preclude it correct, however it provides a pretty strong examples of where reductionism is lacking, which what I believe has turned a lot of philosophers of science against pure reductionism (I am probably oversimplifying here An expert science historian [which I am not] could probably write a whole book about why reductionism is not as popular today as it was in the 1970s).
There are a whole lot more physical processes going on in our bodies then just neural activity. And my best guess is that is exactly where reductionism fails. It is possible that neural activity is a necessary but not sufficient condition for consciousness, if that is the case (which I think is very likely) then we will never be able to describe the mind by just looking at neural activity.
I am actually of the opinion that cognitive scientists are doing an excellent job describing the mind with our current theoretical models which excludes the tough questions of consciousness.
> but after that we get too tired and give up on how the brain works and treat it like a miracle.
I disagree. We know very well how neurons work, and we have a pretty good idea of how neural activity translates to behavior. In other words, we have a pretty good idea on how the brain works. We stop at consciousness because as of yet it is in the realm of philosophy, not science. We don‘t know what consciousness is or even whether or not it is useful for science and we are simply waiting for the philosophers guides us out of that situation.
Note that both cognitive psychology and behavioral psychology has done fine without tackling consciousness. When neuropsychology emerged in the 1980s it complemented both these fields perfectly. The situation is the opposite with the philosophy of mind which grew significantly around the same time.
There have been some attempts to describe consciousness as an emerging phenomena out of neural activity, but so far all of these attempts have failed, or at least failed to turn consciousness into a useful term in psychology (the way gravity is a useful term in physics). I think it is equally likely that these attempts have failed because consciousness may simply not be a useful term in psychology, that is as likely as it is that we simply don‘t understand it well enough.
Saying we have a good idea of how the brain works massively overstates the case...
We know how neurons fire. We do not know how a brain turns that into thought, meaning, intention, experience and on and on. That is not "pretty well understanding the brain", it's understanding some components and hand waving the thing we actually care about.
What I actually care about is how neural activity translates to behavior. And we have a good enough idea of that that we can design SSRI medicine to treat depression, or neurological tests to detect Alzheimer. As for experience we do know something and we are learning more with cognitive psychology, in e.g. priming experiments etc.
I feel like the search for consciousness is to psychology what the search for the Aether was for physics and chemistry. I think it is a worthwhile search, and maybe we will discover something important during that search, but we should also be prepared to find out that the thing might not exist, or it’s presumed properties are better explained with a different model.
Some schools of the philosophy of science would argue that you do. However you are describing is a very different acquisition of knowledge then what scientists did when developing SSRI medicine. We had to:
1. take pictures of brain activity under different conditions to see which regions were active during different moods,
2. sacrifice a bunch of mice to see which neuro-chemical activated which neurons,
3. predict that inhibiting the re-uptake of a specific neuro-chemical would activate that region,
4. predict that activating that region would decrease the level of depression
In your solar example you would have discovered melanin and its relation to your skin tone, and you would have studied the effects ultra-violate radiation has on your melanin levels. Then you would have predicted that staying out of the sun will not give you a tan.
SSRIs are not evidence that we understand how neural activity becomes behavior. They are evidence that you can perturb a system usefully without understanding it very well. That is exactly my point.
Respectfully, you are miles out of your depth here.
I don‘t see why you felt the need to insult me here. We are having a very common disagreement here, one which philosophers of science have been actively debating for several decades.
My point with the SSRI is that we know that serotonin is a chemical which incites certain neurons, and we know that a lack of activity of neurons in that general area in the brain is correlated with depression, so scientists were able to accurately predict that keeping the serotonin in that brain area for longer would increase brain activity there and decrease the level of depression.
This counts as pretty good understanding in my books at least. It teaches us very little about consciousness but my point is that it doesn’t have to. Just like Newton’s theory of gravity did not have to teach us about some deeper cosmological truth.
It's not an insult to suggest one is out of the depth on a topic, especially when it isn't one's field of expertise. You are giving the pop science explanation of various things.
> When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with
It’s less about being too tired and more about being realistic about the limits of understanding.
Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.
When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.
Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.
People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.
Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.
Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.
Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.
People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.
Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.
I don't know - this is a highly specific interpretation of both what science is and why people choose to do it.
I'm a scientist. Believe it or not, I believe in substantially more than prediction and I think its rather trivial to come up with examples where mere prediction is insufficient to meet a normal person's notion of an account of a thing (eg, pre-copernican planetary motion). I'm not saying you are wrong, per se, just that the idea that "it was prediction all along" is a very specific idea of what human beings are interested in and what we are up to.
> that we glean insights into nature of the simulated phenomena
That is right - most people believe that there is a simulated phenomenon "out there" that we learn about. I think there are strong reasons to believe this having to do with how models are related to predictions. The wrong ontology can make prediction very hard and the right one can make prediction substantially easier. Arguably, we are in that situation right now with language models - we just threw a lot of parameters at the problem and now we are able to predict but we still don't really understand. This is perhaps inevitable in the case of language, but I don't think we should look at models with tons of degrees of freedom and the ability to predict things as a death knell for the very idea of deeper understanding.
> The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given.
This is just your own idiosyncratic and biased belief. You're not describing anything objective about LLMs, you're describing your personal attitude to them. This colors your understanding in a way that can't really be reasoned with until you let go of the artificial constraints you're imposing on your own understanding.
> Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
If the LLM in their pocket has a more robust world model than they themselves and is e.g. able to refute their irrational convictions, it actually seems like a very good idea. (Big if, of course.)
If you're going to make something smarter than a person, you got to be convinced that you're only going to be able to understand it on the single training step level and then inductively trust that the rest of it works. We do empirical testing of course with evals, but there's sort of an art to figuring out what is theoretically going to improve eval performance. Trying to fit the meaning of all those weights in your little human brain and working back from there isn't going to work for more than a little slice of the dataset at a time because that's all we can fit in our understanding.
I want freedom to do what i want and not sitting in front of a computer and coding for some company.
Please AI lets burn down knowledge work and labor work. Lets create so much stress to our society that we start rethinking what works mean.
Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
Work has never been about "discovering the world". There have been a handful of privileged folks who had the time to "discover the world". Work has traditionally been "let's find enough food for my family". If you want to think of a future of abundance then perhaps we can discover the world.
It’s not a capitalism problem, it’s an ecosystem. Wherever living things compete for finite resources and opportunities, certain properties emerge from the system. And make no mistake we live in a crowded world full of fierce competition.
Among these properties are optimum behaviours such as hoarding the maximum you can defend (rather than the minimum you comfortably need) and using your power to forestall the growth of others. These behaviours are repeated in nature from microorganisms to apes to humans.
There is no social order which can prevent us from living in an ecosystem, and from these properties and behaviours emerging.
> Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
Why leave something so important up to what AI does or doesn't do?
This seems to be a little naive about how humans consume the benefits we create in society.
"Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature."
Very nice thoughts. You know we all could do this today without "burning it down"? Get in your pod, eat your slop, and watch your screen is where this is headed.
"I want freedom to do what i want and not sitting in front of a computer and coding for some company."
You get that it's you creating the misery here? Then stop? Don't do it. Go start a farm or whatever you think will solve your problems. At some point this all boils down to "chop wood and fetch water" so if the modern way of doing that is so terrible then stop. Go fetch water the old fashioned way and be free.
"I want freedom to do what i want and not sitting in front of a computer and coding for some company."
"Please AI lets burn down knowledge work and labor work"
"Let people do old handcraft jobs."
So many presuppositions about what people want to do.
As a child I spent a lot of time programming and doing "knowledge work" because it's fun - I don't enjoy "old hand-crafted jobs".
Sure, let's definitely destroy capitalism in it's current state I suppose. But I find people like you who hate knowledge-work/coding and think everyone else must feel the same and only do it for the money a bit out-of-touch.
right, these knowledge work and coding jobs are, by my lights, about the best possible job. From my perspective we've invented a machine that does the fun parts while leaving me the less fun parts (review, various hard-to-claude janitorial tasks, etc).
I might like woodworking as a hobby (for example), but I sure as heck don't want to be a carpenter or to depend on my ability to hand craft enough widgets people like to survive
Be more critical, we do our jobs because of capitalism, not because every single company needs an hr department or the next crud application or the 1000th webshop.
The solution we've come up with is move all the unpleasant work stuff to China where people don't complain about doing it because they already have communism, and therefore everything is of course effortlessly perfect there.
The data is the code is the data. Reality has no distinction between "data" and "code". These terms are categories we impose on systems we design, to make it easier for us to build and reason about them, but they're nothing but mere opinions, and depend less on the system structure, and more on the perspective of the person asking which is which.
This is related to, and possibly equivalent with, the core point of both this story and the original one: computation is independent from substrate.
You can build a computer out of anything, whether it's semiconductors or lasers or meat or magnetic fields or water flowing downhill or abstract thought, and that computer will happily perform the same computation as every other equivalent construct from whatever substrate. That's because computers are ultimately made of math, and we design "real ones" by finding ways to approximate the mathematical constraints with physical systems. But the choice of how to map the math to physical systems is completely arbitrary, and any such mappings are equivalent from POV of information processing ability.
(Of course substrate is not arbitrary from economic POV, which is why we build most of our computers out of silicon and plastic, and make it work with electric current and lasers.)
One of the best thing I done for my career (as a self taught software developer, but with a degree in electronic engineering) is to learn computation theory.
Computation is math (and a very restricted subset of math). It’s mostly specific sequences of sets manipulation. What sets and what manipulations are defined by people, not by the idea of computation.
The best thing is that as soon as you specify the sequences of manipulation, it become a a set that you can manipulate. That can be a difficult concept to grasp, but that’s what helps in designing notation that are more appropriate for the human mind to describe a solution for a specific problem.
That's why encountering something like LISP for the first time (by writing a LISP interpreter, for example) creates a big bang event in form of an imminent intellectual catharsis. People who encountered it just once, will never be able to see the world through the old "meaty" lenses afterwards.
This is a good model. If you take an old ROM dump from a video game, it's just a pile of bits. You don't know what bits represent code, what represent an image, what represent text, etc. You have to analyze them contextually to actually figure out what is code and what is "data" in context, because without context they are truly one and the same.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
A ray of light doesn't know or choose because it has no agency, just like an apple doesn't know or decide to fall because of gravity. It's an anthropomorphization.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
I'm no physicists, so I guess I'll ask it: Why?
Also related, why do some ray of light then "see" a black hole yet decide to head into them anyways, if they saw it before they went in that direction? Seems like a dumb move :)
Its future isn't over there because it moves in that direction, instead it moves in that direction because its future lies over there.
Relatedly:
> [General Relativity] basically says that the reason you are sticking to the floor right now is that the shortest distance between today and tomorrow is through the center of the Earth.
It would be interesting (if unlikely) that we found our reality worked that was as well. Mostly just super granular gravity nodes made up purely of clumpy probability distributions.
Also I describe this more simply to others as a tree walk where the manifold is mapped along a walk through it (even if it’s not linear one) where you choose the next jump based on the most likely future mode relative to the nodes already created and the ones from the prompt.
This helps some understand attention layers rudimentary. Even tho it leaves out that multiple layers sort of prune the overall manifold in successive passes.
Yes, yes, but what fertile fallacies and common misunderstanding can politicians use to acquire more power via exploiting the difference between the common person's flawed understanding due to cargo culting, cognitive biases, and/or outdated or inappropriate analogies vs actual reality? Is there any way we can get the AI to say give all political power to narrator is the solution to all problems and use the common person's mistaken worship of AI as a spiritual all knowing conscious being with unusual sensitivity and caring about everyone to cement that power? Certainly one of you eggheads can tweak that for me? What? It's against your ethics? We're trying to save the world here. Here, let me call up Bernie Sanders to propose nationalizing half your companies so we can do that.
The original story is an original work made by a human consciousness exploring how it might be different from other forms of consciousness.
This one is a pastiche made by a human consciousness borrowing extremely heavily from another human consciousness justifying why something else might be another form of consciousness.
That rather undercuts the point; if this was generated by an LLM unprompted, it would be different, but it isn't. You could perform exactly the same rhetorical trick with a toaster or anything else.
Thank you for your point. I don't understand why half these comments are taking this blog post seriously when it ends with "Weights helped me draft and proof this story."
Now read the post's topic again ( https://maxleiter.com/blog/weights ) and reflect on the prior fact that weights helped [the author] draft and proof this story.
My reaction (and I'm sorry that it is harsh according to some) is that there is no intelligence found in either the author nor their tool. This is extreme navel gazing, based in science fiction, wanting (wishing) to believe those stories to be true.
I'm skeptical of AI sentience because we must do our due diligence, not because it's impossible. Skepticism is the only respectful approach because to grant sentience is a step away from granting rights.
We humans tend to chauvinism in all things (e.g. we're special, the center of the Universe, God made the universe for us, etc), no less when it comes to judging intelligence. The original story about thinking meat was written to help us out of our chauvinism; this derived story was written about weights for the same reason. Which is quite valid.
The actual counterpoint is demonstrated in _Blindsight_ Peter Watts. He makes a strong (and rather terrifyingly strong) point that intelligence is not consciousness.
I cannot tell if you are asserting my comment is chauvinistic with your use of "we." If that is so: that's a poor counter to my point or assessment of my stance because it assumes I'm making a baseless argument as a "proud human."
My original comment (roughly "there's no intelligence in this article, nor sentience in LLMs") is in response to the blog post's buried lede (that the cumulative activity of LLMs has accrued to a weight of "AGI is around the corner" or "there is artificial consciousness in this matrix").
To be clear, I'm not saying LLMs are useless or a wrong direction in development of "AI," but rather it's the Fool's Gold for the path towards AGI, the pursuit of the academic field of Artificial Intelligence research. A research that I've been abreast of for years before this new age of language models that has made everyone with a keyboard an arm chair expert.
Also, thank you for the book recommendation, it's on my list! :)
I read your comment as criticizing the OP's story as pointless and unoriginal. My comment elucidated the point of the original story, and what I think is the point of the second story.
Roger that, thank you. w/r/g your recapitulation of my point: yes, the story is unoriginal and pointless, and the HN community seems to eat it up-- isn't that odd.
So I still disagree with your elucidated point (as you end with "which is valid"): the OP author is using prior art fiction to bolster their opinion of LLM-based software tools as being a possible vector of sentience, not to disarm our chauvinism like the original author intended. If OP wanted to make that point, they could have written a critical essay instead of farming out their thoughts as tokens.
But still, I look forward to reading the book you suggested to understand and appreciate your perspective more.
The point is valid regardless of whether you judge LLM to be sentient or not, because the point is to say "don't let your prejudice about substrate bias your decision". Or in other words, if you're going to weigh something don't tip the scale. This is good advice whatever the outcome of the measurement.
Blindsight is a remarkable book - I hope you enjoy it!
Ah I see now that I am in agreement with you, thank you for being a patient interlocutor. I do not discard and rule out the possibility of a different substrate being the well-spring from which sentience emerges.
Plainly, based on the current ground trodded and the trajectory laid out by the frontier AI labs, I do not have concrete evidence/proof of sentience having emerged from LLM-based software tools as of June 4 2026 nor do I expect it to happen in the future based on my understanding and observations of this technology. I'm not excluding the possibility but wielding skepticism. I am open to being proven wrong with new discoveries.
Which is why (to return to my lashing of the dead horse) I don't see OP's post as worthwhile. Their post reiterates a point that is already valid (the prior art) with no new substantial discovery. Which is why "unoriginal and pointless" is apt, a novel idea was not presented; it's just some vain virtue signaling.
Wait, did I grossly misunderstand Blindsight?? I definitely thought that the aliens were 100% conscious (or at least elements of some conscious entity) and that the humans interpretation of their interactions with the Rorschach were supposed to be read as a blot test (through a rather heavy handed metaphor) demonstrating that basically the humans were the monsters and were twisting logic into letting them justify destroying the scary alien ship.
That's a very interesting reading! It's been a while since I read it, but IIRC Watts explicitly talks about the matter of intelligence vs consciousness at length and Rorschach's lack of consciousness is essentially proven at some point by the linguist. The distinction is driven home subjectively, personally by the viewpoint character. Your reading adds a rather diabolical twist! I don't think it works though since Rorschach was clearly a tiny part of a much larger, more powerful entity and the humans are clearly doomed. I'm not sure it serves a narrative purpose to have humans be a weak, evil civilization destroyed as a side-effect of a good, strong civilization's actions. I read it as a (hopeless) conflict between non-conscious, infinitely strong aliens vs conscious weak humans and post-humans. I mean it took extreme, heroic effort to damage what is probably a tiny appendage of a much larger civilization/organism, and you lose the emotional resonance if humans are the baddies.
I think you’re right. I read the Wikipedia page after to refresh my memory on the synopsis and I’d definitely forgotten a lot.
I think I mostly latched onto the ship being called Rorschach and the humans immediately proceeding to torture its inhabitants. That felt very relevant to me and sort of overshadowed the actual consciousness question.
I don't think skepticism should be called chauvinism. I imagine that artificial consciousness could be made. But I don't think this is it.
Also I don't see why intelligence not being consciousness is scary? My cats are very conscious as far as I can tell, but not particularly intelligent. I think LLM's exhibit some contextual intelligence without there being any particular reason to believe they're conscious other than woo psuedoscience.
You underestimate the intelligence of your cat. Or rather you measure intelligence with an extreme human bias. What you consider intelligent behavior your cat may consider weird, and what your cat considers intelligent behavior is something you will never consider.
That said, I don’t think it is useful for philosophy nor science to consider intelligence to be the same thing as consciousness. In fact I would go even further and claim that intelligence is not a useful construct, neither for philosophy nor for science. Consciousness, on the other hand, I think is useful for philosophy, but not (as of now) for science.
People argue for AI sentience from a place of emotion couched in logic. they _desperately_ want it to be true and will not take a logical step back. Any argument comes back to "well doesn't a human brain work like that?" Or some variation of it.
My personal theory is a fuzzy thought about how people want to reject the concept of a higher being and want to embrace the fact that we are now able to create our own consciousness and religion is dead.
I don't understand why, but it is the undertone of every argument I've seen that is pro-AI-is-sentient, like some big unspoken elephant-in-the-room.
I would rather just judge this tech on its own merits.
edit: this comment got 1 upvote literally as I submitted it. I know @ doesn't work, but @dang, something seems very strange about that.
I'm open to the possibilty of AI conciousness, and there is some desparation related to the concept of a higher being:
There are many people who will categorically rule out the posibility of AI consciousness due to near-unshakable belief in a higher being. This argument resembles "Christians should not be worried about our climate since God is ultimately in control." Such views make it harder to collectively prevent dangers from a sentient AI, or harm to a sentient AI.
I do not claim that everyone who believes in a higher power believes concious AI to be impossible, or vice versa; just that it would be very hard to change the minds of those who adhere to this reasoning.
> People argue for AI sentience from a place of emotion couched in logic. they _desperately_ want it to be true and will not take a logical step back. Any argument comes back to "well doesn't a human brain work like that?" Or some variation of it.
It's funny, because I find myself constantly stating the inverse of this. Every argument I've seen against AI being sentient plainly comes from, as you so eloquently put it,
exactly "a place of emotion couched in logic". People desperately want it to not be true and will not take the logical step back of examining its actual similarities to human intelligence. Every argument comes down to "but it's not actually a human", or some variation of it -- which, if you pay attention, is not actually a logical counterargument. (Or, ironically, "but it doesn't have a soul", which is why the Pope is the perfect figurehead for these people).
If you already know any logical argument against it can be countered with "well doesn't a human brain work like that?", why are you so confident that your position is actually the logical one?
...And could it simply be that, alternatively, the concept is not actually a logical distinction, but rather an emotional one, made by emotional beings to put a word to what they claim makes them special?
> I'm skeptical of AI sentience because we must do our due diligence, not because it's impossible. Skepticism is the only respectful approach because to grant sentience is a step away from granting rights.
Thanks for saying this! It amazes me to witness so much pushback (in HN of all places!) for the call for skepticism and scientific rigor on claims made by business which have vested interests in hyping things up.
For my part, it is exactly when I perceive the reluctance to grant rights or relinquish our estimation of ourselves as unique as the _reason_ for skepticism that I push back on it. That's not good reasoning, those are motivations for you to come to a desired conclusion and fill in with reasoning that gets you there.
You can't do the same with a toaster. Physically you could write that story. But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness. You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
You could absolutely write a compelling story about a sentient toaster; it's been done before [1].
That is entirely separate to whether or not it would be a meaningful way to understand the world; a convincing story is not the same thing as one that is true.
I never said you couldn't write any arbitrary compelling story about a toaster, I said that this specific hypothetical story, where you rewrite "They're made of meat!" to be about a toaster, would not be compelling.
I am doing my best to communicate with you but to be honest you are not hearing me (across both responses), and I am out of words.
Just wanted to say, I appreciate your patience and good sense in this thread.
It's difficult to tell who's trolling -- probably best to go with the charitable assumption that everyone is honestly trying to convey their opinion, but mostly talking past each other. Unfortunately these discussions about the nature of consciousness never go anywhere useful.
I think I'm probably in the same boat as you, roughly: a) LLMs are doing something really interesting that resembles in many ways both intelligence and consciousness; b) I suspect they're not actually conscious but I don't know how you'd know for sure; c) it all just drives home that we still don't really know what consciousness actually is. But like (a), it's definitely something really interesting...
I read some stuff once where the author argued that almost everything takes in values, performs operation on them and gives output. Some things also store things. Then, if everything is doing logic operations there is no telling where intelligent "life" might be hiding. It also is really one giant system.
The toaster has hard coded or configurable weights. It makes a product from heat and time.
You could make toast by heating a brick on a campfire. It would be a clear sign of intelligence.
If we lift one weight out of your brain we wouldn't look at the number and say it is intelligence. It must exist in a chain or matrix multiplication to qualify.
The fun thing is that the timer and feedback mechanism do exist in a chain of events.
It's part of the system and it takes all steps to complete the task.
Our chain of thought that leads to making toast would be abandoned if it wasn't in working order.
It would be equally compelling, because the compelling nature of the story comes from the language, the presentation, rather than the [specific thing being ascribed consciousness].
No, the original "they're made out of meat" works because we're confident that we are in fact intelligent and conscious, despite how ridiculous and unlikely the author manages to make it sound.
"They're made out of weights" works precisely because LLMs really do have this mysterious property that they seem somehow intelligent even though nobody can explain exactly why, and there's active debate over whether they could be considered conscious.
The thing being discussed isn't simply an arbitrary MacGuffin; in both cases the nature of the thing is central to the impact of the story.
I disagree; it works in the original because it's the unlikely consciousness that produces the text itself; in the LLM case, it's produced by the likely consciousness.
"Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it.
> Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it
This is well put. We don't need to imagine how a human views a llm because we can ... just do that. Everyone capable of reading the story is also capable of thinking about how they feel abouy llms that exist right now and you've probably used.
The trick of the original story is inverting your perspective, moving your view point fron yourself to an "other" (which I think is a primary qualifier for most good fiction).
But that toaster would just be a device to talk about consciousness in general.
In this case it does that and also it talks specifically of the LLM case, which can spark the discussion. Unless you believe to have the only valid and true opinion on the matter, and affirm that a normal toaster is just the same as an LLM in this topic.
I agree that is the mode of argument; reductio ad absurdum is a brittle argument, because it only works if the analogy holds. I argued the analogy doesn't hold.
> But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness.
Teapots are not compelling.
> You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
God is compelling t billions of people.
Is Russel’s Teapot a bad argument in the God debate?
> Is Russel’s Teapot a bad argument in the God debate?
What's the relevance? If the argument made here are was a good argument, it wouldn't matter if Russell's argument was bad. We could construct a bad argument using reductio ad absurdum right here and now and it wouldn't matter to either argument.
Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
> Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
There are two aspects here.
1. That people find the question interesting
2. That it has any bearing on reality (ontology?)
The first aspect is anthropology. Russel’s Teapot is not supposed to undercut any anthropological arguments. It’s supposed to undercut the second aspect.
So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
> For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
Yeah. AIs know how to use computers. What’s this got to do with consciousness? Whether or not you are an AI is practical and disprovable. Consciousness is so ephemeral (for lack of a better word, not literally) that Philosophical Zombies is a real argument/thought experiment.
You may think I’m being coy (“Can you be straight with me”) but that’s not my intent at all.
Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument. I don't see any relevant connection. Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
> So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
AI is a real phenomenon that we can study and measure. There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
To put a fine point on it, it appears on it's face that AIs could be conscious. They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing. So at face value, the burden of proof is on them not being conscious. Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
> Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument.
I pointed out the parallel in both statements. I can't do more than that.
> Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
The teapot isn't real and the toaster consciousness is not real. What am I missing?
> AI is a real phenomenon that we can study and measure.
Robot cows are real as well.
> There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
Yeah. You can't prove it for any entity. I agree.
> You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
The bull swears that the robot cow is a real cow. But we know better.
> To put a fine point on it, it appears on it's face that AIs could be conscious.
It doesn't to me. Not any facelength.
> They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing.
Objective reality has never cared (am I anthropomorphizing now?) what is indistinguishable for people.
> So at face value, the burden of proof is on them not being conscious.
Which party is the burden of proof on? This is confusing since you are saying that the burden of proof is on a position (on them not being conscious).
Is the burden of proof on people who argue that they are n o t conscious? That's peculiar.
I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
We obviously can't demand a falsifiable theory here. But we have to do better than arguing from incredulity.
> Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
They don't have to rise to the level of disproving something for which they have no burden to disprove.
Would you agree, for the sake of argument, that it's more interesting to discus chatgpt 5.5's similarity to sentient/sapient/conscious than, say, my mechanical toaster?
> I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
Also, you say this like "duck" isn't an arbitrary, artificial category created by humans, a "map not the reality" if you will.
There's very few things that humans can understand to the level of putting them into truly objective scientific categories (various pure elements maybe?), everything else we more or less bodge together for the sake of getting on with life.
It's not like conscious has some kind of formal objective provable definition, even inside the world of human created language and terms.
As far as I know, in the real world, if something looks enough like a duck (and can breed with a duck maybe) we, humans, do call it a duck.
> I pointed out the parallel in both statements. I can't do more than that.
You mean these statements?
No one cares about teapots in space either (Russel).
Teapots are not compelling.
I guess you did but that's pretty much leaving me breadcrumbs and expecting me to make your argument for you. It seemed to me like you were talking about reductio ad absurdum with that argument as an illustrative example. Perhaps you overestimate my cleverness.
> Which party is the burden of proof on? This is confusing since you are saying that the burden of proof is on a position (on them not being conscious).
It's metonymy. "X" stands in for "people who argue for X". May I ask if you were sincerely confused? You told me you aren't being coy but I have a hard time believing that this was so unclear.
> I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
If it looks like a duck, if it quacks like a duck, you should adjust your priors to assign a higher likelihood that it is a duck. All measurements contain error; you can't ever observe, "that is a duck," only "that looks like a duck". All knowledge is founded on a sufficiently deep stack of "looking like a duck" that we may assert it with confidence.
To the extent we have objective measures (like conducting a Turing test on blind participants), it can meet them too. You can't say the same of a toaster.
> We obviously can't demand a falsifiable theory here. But we have to do better than arguing from incredulity.
"It is a statistical model, ergo it is not conscious" is also an argument from incredulity. I don't know if that's your view or not but it's the one my remarks have been addressing in general.
I think that the reaction here alone disproves this somewhat, because imo it's exposing how anthropocentric most people still are, despite all evidence that we are in fact just "meat all the way down".
Despite all the evidence that we are in fact just biological machines, people still persist the theory of our own uniqueness from other creatures, which we ourselves often treat as biological machines.
This adaptation is wonderful to me specifically because I think it shows that our shifted goalposts of, "well we're not just animals, we can think and reason" was never more than a convenient excuse for many people (and as evidence of animal intelligence continues to mount, denialism still attempts to preserve this distinction by claiming human thought and reason is different than 'animal' thought and reason, sans evidence).
It's not about who created it or why, it's about how people still haven't actually internalized the point, because the subject changing from human to LLM doesn't intrinsically change the message about consciousness, but the reaction being a 180 shows how hostile people are to that message, still.
Not having yet read the original story, this reads fine on its own.
And I didn't see it as much as a literary attempt for art's sake, but more of a dialogue-based technical parable trying to convey a real-world insight. Kind of like the ones in Godel Escher Bach.
>You could perform exactly the same rhetorical trick with a toaster or anything else.
Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
P.S. Read the original too. Seems like the exact same could have been written about us instead of the original, if the focus wasn't on our substrate, but on our brain processing. Which, after all, is also about weights.
> Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent.
For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
I can even write one about a ruler, if I can bend it enough, no pun intended.
>Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent. (...) For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
Doesn't that miss the whole point?
You could write "They're made of metal strips!". You wouldn't be able to write much else, as toasters don't have showcase in the way of human-level intelligent behavior. Which is the whole point in the meat and weights versions.
At best you could write "They're made of metal strips!" for toasters AND other metallic devices, and use some analogies of features BOTH have in common. But they wouldn't be intelligence related behaviors.
Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
They can even adapt to their environment and the characteristics of the bread even with simplest of mechanisms because the text will be overglossing the fact that different types of breads have different thermal characteristics and this will deeply affect the behavior of the metal strips, bordering near a sentient being even more thoughtful and considerate than a human which is rushing through house to catch the bus in the morning.
>Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
Yeah, that's called "stretching it beyond any recognition".
You could do that. It will have none of the effectiveness or resonance of the two stories.
Who have you ever heard make a sincere, good faith argument that a toaster is conscious? Who do you imagine would argue against you if you asserted they were mere automata?
If you can't identify anyone, then this analogy doesn't work.
> Who have you ever heard make a sincere, good faith argument that a toaster is conscious?
More than one, for many classes of devices, incl. toasters. Some were drunk, some were insane, and some were delusional.
LLMs are no different. They are automata, yet delusional people bring out pitchforks and torches when someone points out that they are just statistical models, and they don't even work when there's no input to them.
Their being statistical models and their being conscious are not contradictory unless proven otherwise. That's not knowledge, it is assumption.
It would appear to me you have no interest in a real, good faith discussion on this topic because you think anyone who disagrees with you is necessarily delusional. Which is a shame, and that's the kind of dogma you are criticizing.
This was exactly the point of the story, it's too uncomfortable to admit that we don't know what consciousness is and what is and isn't conscious, so we just brush it under the rug.
Sorry, I'll be blunt: Are you sure you are not projecting?
My perspective comes from a set of pillars. First, I work at an HPC center, where we support running and development of AI systems, incl. international projects. IOW, I have knowledge about how these systems built, work and needs to continue working.
Moreover, I'm an HPC programmer myself, so I'm not completely uninformed about the math this involving this thing, and I'm lucky enough to have friends who are much more dedicated than me, and we discuss how this thing works and feels like this way.
I'm not an AI hater per se, being programmed AI systems in the past, incl. emergent intelligence systems with multi-agents which can span continents if need be (this was my master's thesis, time flies).
However, knowing what these things are capable of and how they are built. I don't believe them they're conscious/sentient beings. I also had much more time to ponder on these things even before LLMs being a thing. Some hard sci-fi books have asked these questions seriously in their captive adventures way earlier. If one reads these books seriously, there are a lot of philosophical angles to consider and draw upon.
I can discuss in good faith. For hours, days or months even, but throwing "you're a narrow-minded dogmatic luddite neanderthal!" card to anyone disagreeing with you is not it.
Positive, yes. I never called you a luddite or a neanderthal or anything of the kind.
It's perfectly fine to believe they are not conscious, I am not convinced they are, but asserting anyone who disagrees with you is delusional is unfortunate.
From my perspective, you inadvertently implied, but no offense taken, all is well.
No, I didn't assert anything. I have just given examples rooted in my experience.
None of my friends who also happen to know how these things work told or defended that they are conscious, even intelligent. Maybe my friends are dumb, I dunno.
Once I have seen a man who claimed that evil has possessed the POS device at his desk. The thing was printing "cannot connect to server" on the receipt printer every 10 minutes, yet he didn't know how that thing worked, and was a bit too high to read the paper the thing was printing out.
This age's LLM craze is akin to "wonder inventions" of 70s, which are deemed dangerous or harmful in the future. LLMs will be with us, but we need to pass beyond the hype and stop sweeping the problems they create (environmental and societal) under the proverbial rug.
If you didn't mean to say that people who believe AI is conscious are delusional, then I don't understand how to read your comment. If you're interested in a good faith conversation, I'm very confused why, when I asked if anyone sincerely argued that toasters are conscious, you brought up people who were drunk or delusional.
It's a two way street. I might have failed to convey my point clearly, too. While I'm fluent, this is not my native language. Some edge cases in meaning still betrays me and I make mistakes I didn't intend doing.
> they don't even work when there's no input to them.
Why is that needed for consciousness? They're artificial. If we put someone in a coma, they don't talk, they're not really conscious. Just because the AI model has a more obvious off switch shouldn't make a difference. It's easy to imagine gluing a cron job to the model so it works randomly. If that doesn't count because it's external, if we take a brain out of a human's head and slap it on the table, it's not going to do anything. If I take the AI's model file and stick it on a USB flash drive, it doesn't do anything. Without a computer to run it on, it doesn't do anything, just like a human brain doesn't do anything without the rest of the human body as the harness.
The underlying question is, since we created LLMs, we can see into the actual matrix math, the linear algebra that comprises them. So it's easy to dismiss them as a next word guesser. How could consciousness arise from guessing the next word? But we don't know where consciousness comes from in the first place!
So since we don't know, the fact that, yes, the anthropomorphically named LLM "neurons" are merely matrixes of numbers and we do linear algebra on them; yes, that gives us much more insight as to what it's doing internally compared to a human or any other lifeform with consciousness's brain. And yes, human neurons are much more complicated than a mere 2d matrix of numbers, so far as we know. But we don't know!
The indightment of LLMs is that they can't say they don't know, and would prefer, instead, to hallucinate and bullshit an answer instead. They can't help it, they learned from the best. As a human though, I don't know if they're conscious, and what I'm going to say about that.
Might be. But after having read the original, it could just as well be the weights version and still be about us to begin with.
I don't see how "you could do that with a toaster" still. The whole point between the original and this, is that you can't do that with a toaster or a sofa, but you can do it with meat and weights, because both share all the other analogies in the story, as well as the basic premise: the improbability of something like thinking, feeling, etc arising from a lowly substrate.
And having read both now, I see how the existence of the original is a plus for this story, not a minus. Instead of making look like mere copy (as would be the case for a typical story modelled after another), in this case, it adds a meta layer, and enriches it.
The key point here, I think, and why it's necessary to have read the original story, is that being able to express an idea is not the same as that idea being correct.
You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
What I think a lot of people are getting stuck on is that editing the text to say 'toaster' would not mean that toasters were conscious, and that editing the text to say 'weights' doesn't mean that weights are conscious either. Stories aren't factual just because they are written.
The original story was written by the thing claiming to be conscious; the LLM and toaster ones would not be, which undermines the claim to consciousness a lot.
I don't think you understood the point of the story. It's not that LLMs or agents are conscious, it's that our dismissal of the possibility is reflexive and uninformed. Personally I think anyone who has made their mind up about whether or not LLMs/agents are or can be conscious has done so before the evidence is in.
The story does not assert that search and replacing "meat" with "weight" makes them conscious through some magical mechanism. It's a thought experiment.
No, it's a story. Stories can be about real things (the original story is about a real consciousness, as shown by it being produced by that consciousness), but they aren't automatically just because they're good stories.
You're falling into the trap of assuming that a good presentation--being convincing--is the same as being truthful.
>You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
Yes, there very much is a huge barrier. The copy and the original both keep the same subject matter: intelligence/human-like behavior.
The toaster doesn't. We could do edits, and the story (original or copy) would lose all its potency.
The surprise is "but how is the richness of intelligent behavior produced from something as basic as meat/weights". Why is kind of surprising reductionism is both cases.
Whereas nobody is surprised that "a metal strip with electricity flowing through it from a power source" heats pieces of bread to a specific temperature. Even if a toaster was "metal strips" all the way down, it's nowhere near as impressive jump from substrate to behavior, nor is the behavior as important to us and touches the core of our existance.
I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words. Writing a story in which a toaster in conscious is not even new [1].
That is entirely separate to whether or not it's meaningful to write a story about a conscious toaster. Again, expressing an idea in fiction is not related to the accuracy of that idea.
What makes the original story interesting is that it was written by the thing claiming to be conscious, which is what takes it from 'a story' to 'a story making an important point'. That's not the case with a hypothetical story about a toaster, and it's not the case with this story about LLMs.
The story is convincing because it's well-written, not because it is factual.
>I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words.
And you're misunderstanding me. There is no practical barrier to writing this story about a toaster. But there is a conceptual one. It would be bullshit, that has little resonance, and little connection, as the whole point of the meat and weights stories is the "lowly substrate -> surprisingly intelligent human (or human-like in the case of the weights version) behavior".
Nobody thinks a toaster's behavior as surprising or intelligent or human-like to begin with.
So you'll just be stretching the analogy beyond all recognition, with little to no payoff.
>Writing a story in which a toaster in conscious is not even new
It's not, but it has nothing to do with the meats or weights stories' point, as it's just an arbitrary choice, like making the candles and teapot sentient in Beauty and the Beast.
Whereas in the meat and weights stories the whole point is the surprise from us already seeing the human or human-like sentience of the thing, and comparing it to the "dead" substrate.
I was having a hard time pinning down what bothered me about this but I think you put it pretty well.
It draws an analogy between us and the skeptical aliens in the original story which feel silly to us, so the obvious implication is that we're being as silly as they were.
But it doesn't really give a reason to accept the analogy, it just asserts it.
There's a big difference between a whole civilization and a piece of software that can output text.
I noticed that jump too, but I also don’t think the interlocutor is supposed to be 100% logical and rational. The aliens in the original weren’t either.
I didn't read it as coming to the same conclusion as the original, because the meat story presupposes that we who are meat already know that the aliens are wrong. (Maybe that's a humanist reading of the original, but okay). I didn't read this one as trying to make a case that we are fools for assuming that matrix multiplication can't be intelligent... I think its point was that it can't be intelligent, and that people trying to judge it the way mechanized aliens would judge meat creatures just makes them sound ridiculous.
Don't take this as a criticism, but I think overwhelmingly people took it the other way. The fact that the author admits at the end that the story was written with the assistance of "weights" is a tell, to me. I just have to assume the author's genuine position (which I believe to be, we don't know that LLMs aren't conscious or that they could never be conscious) is so absurd to you that the thing comes across as satire. I find myself in that same position sometimes.
The original did not come out of a vacuum. It was done on multiple generations of meat. Even though this one uses a little bit of silicon, it is still standing on the same shoulders.
I genuinely thought this one was a satirical take on the narrow-mindedness of the aliens in the original, even though the story tries to paint humans as narrow minded. I guess this fundamental human trait to believe that their cognition is the ultimate way to think in the universe ironically leaked into all these stories as well. Real spacefaring civilisations would probably have seen all kinds of intelligence rise from sufficiently complex systems.
I disagree that it undercuts the point in any way. The original story didn't appear out of a vacuum, the human consciousness that wrote the story was itself borrowing extremely heavily from the sum total of human ideas. It was shaped by the culture it developed in, formed its model of the world based on existing scientific advancements and technology of that society.
Extending the idea to a different context based on new material conditions is as human as it gets really.
I feel the original story contained in it the whole point of this one, and then some.
This did not add anything, just rephrased it so rather than humans viewed through the pov of aliens it's LLMs viewed through the pov of humans. Well, we are the humans, so surely we do not need to learn about this point of view?
It lets us consider the relativity of viewpoints. The point of the original isn’t just how humans might be viewed from the perspective of aliens. It’s that the apparent strangeness of something depends on the viewpoint and on expectations. The present story is not supposed to stand on its own like the original, it’s supposed to be considered in light of the original. In the original, thinking meat appears strange to the aliens because of their viewpoint and expectations. The present story thus points out by analogy that thinking LLMs might seem strange to us merely due to our viewpoint and expectations.
The story is not a linguistic/rhetorical trick. The "trick" is the point, and the real trick/delusion is in your brain. You and many others have put the idea of consciousness on a pedestal and think it's so special that is must be conceptually profound.
It is not. That is why the "trick" can be performed on anything. That is the point. To show you that consciousness only needs to be a set of weights, not that far off from a toaster. It trivializes what it is to be human and it's also extremely true. Consciousness is a trivial thing, we make it out to be big/important/profound and that is the delusion.
I love this koan and what makes it even better is that is based on a true event: Minsky’s real answer was “It has them, you just k ow what they are yet”
It's not often I see something that's fractally wrong but here we are.
There is a dictionary, it's called the tokenizer.
There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness. This one seems to think that you don't need structure and interpretability just because you change substrates.
The tokeniser is not a dictionary. It doesn't provide definitions, or give the LLM any kind of mapping at all.
At best, it's a wordlist. It gives the LLM some idea of what humans consider to be common words. But it doesn't tell the LLM anything at all about those words. And it's not even comprehensive, many words map to multiple tokens. Nor is it exclusively words, some of those tokens are punctuation, or modifiers, or control tokens. On multimodal LLMs, some of the tokens actually represent image and audio data.
The LLM doesn't get informed about any of this up front, it has to learn what every single token means from context.
You are technically right, that it's something in an LLM that's not weights; But it's not that structured. And really it's only there so the LLM can interact with the outside world.
> There are grammar rules
There is no dedicated "grammar rule" structure in the LLM or the tokeniser. It has to learn them all from context, they get encoded as part of the 80 layers of weights.
I see people give too much importance to specific engineering design choices of the current generation of LLMs. Tokenizer is not an absolutely essential part of the system. It’s just and adapter for text input/output. It can be eliminated completely and model can use bytes directly.
I think the short story captures this well. Weights (connections) are the essential and philosophically important part. They do the thinking, memory, singing etc.
A tokenizer is roughly and approximately Huffman-coding sequences of input (bytes of English etc) into shorter sequences (list of tokens), as a performance optimization.
As you said, it's not in any way intrinsic to the LLM, though it may be a very necessary optimization on today's hardware.
IMO, we are probably talking about a 6x slow down (for typical english). You would need to be absolutely stupid not to implement some kind of optimisation along these lines.
Slower and maybe a little dumber; But it would work.
My thinking is that for most tasks, a byte-orientated LLM still needs something like the wide "single activation per word" formatting that the tokeniser mostly provides. And it will likely waste its first and last few layers implementing a replacement tokeniser (and would probably do a much better job at it). It would also need to decode and encode unicode at the same time.
My estimate is that it might lose about 10% of its weights to these new tasks. Your 80B parameter model becomes as smart as a 72B parameter model - Measurably dumber, but not drastically so.
>There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
That paper did not train the models on 'a language with strong consistent grammars'. Mathematical Operation tables are not a language. Grammar itself is a post-hoc rationalization and there's no evidence LLMs follow 'grammar rules' anymore than the brain follows grammar rules. Of Course, that's not to say transformers can't learn simple rules if the dataset calls for it.
>A language is a set of sentences.
A sentence is a finite sequence of symbols drawn from an alphabet.
A language is a structured system of communication used to express arbitrary ideas between multiple parties. Math operation tables do not, and cannot, do that on their own.
That distinction matters here because we are talking about what properties the model is expected to learn. English and operation tables are fundamentally different objects, so it is not surprising that a model learns different kinds of structure from them.
That doesn’t make any sense. A alphabet is a list of valid characters. A dictionary is not just a list. Even in a language like Chinese where individual characters carry meaning, a dictionary tells you what that meaning is. It’s not just a list of characters.
Or to echo article, the dictionary is made out of weights.
A list of words isn’t a dictionary. What a dictionary adds over a list of words is all the relationships between the words needed to interpret them and use them, and all of that is in the weights.
Amusingly for https://en.wikipedia.org/wiki/Special:Diff/325776830 , the last place to use /usr/dict (Debian, which changed it in 1998; Berkeley having changed it in Net/2 in 1991) stopped doing so years before Wikipedia was invented.
A mapping of Chinese characters to integers (like a tokenizer) would not be a dictionary. You’d also need definitions. At best it’s an index to a hypothetical dictionary.
It's beside the point and so I only note it out of interest, but the Chinese writing system doesn't use an alphabet (or a syllabary like Japanese kana), it's logography.
The tokenizer is, at best, a sensory mechanism as evidenced by 1) the random generation of the tokenization scheme, and 2) vastly different tokenization schemes produce virtually identical behavior. It'd be like if Noah Webster threw a bunch of movable type into a bucket (breaking some words in half) and then drew randomly to make the first English dictionary.
EDIT; I was too cavalier with the comparison of tokenizer to sensory modality; my ultimate point is that direct byte-to-token transformers can achieve similar overall performance which to me makes a weights to meat comparison pretty straightforward, but the particular tokenizer in use certainly has a large impact on both efficiency and accuracy on specific problems (e.g. digit representation)
I'm kind of stunned that someone is using my work to tell me I'm wrong. I wrote the code for the dish brain pong and encoding information was a huge part of what that experiment was about.
So when I way that the grok paper and the pong paper fundamentally agree I have some idea of what I'm talking about.
I might have misunderstood the point you are making. I read the original article as "weights are like meat", and so I'm confused by what you consider fractally wrong.
The point that when the rules the model learns are simple enough they stop being spread out over all the layers and become as easily interpretable as any expert system.
It's just that the rules we feed in the model are extremely poorly defined and we end up with the soup of disjoint rules smeared all across the weights.
This isn't a feature of the models. It's a feature of the training set.
Being shocked that you can store rules in floating point numbers is the same as being shocked you can store rules in integers. It's been a century since Goedel Numbering was invented, we should be used to it by now.
Right, but all of that is still in the weights. The point of the article/joke isn’t literally that there is no grammar, it’s that there is no grammar separate from the weights. It’s all in the weights. And yes, it’s absurd. It’s a joke, but a thought provoking one.
So basically there are rules, we just can’t articulate them and so we can’t decode them from the weights. The Goedel Numbering metaphor is pretty appealing to me. You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers. We just happen to be using matrices because the math is easy to parallelize. The trick is to realize that when you know the sequence you have and the sequence you want then you can compute the calculations. If you constrain the calculations to only matrix multiplication then you arrive at the scheme we have.
> You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers.
That statement caught my eye. It's either trivially true or quite clearly wrong, depending on how you mean it.
In the literal meaning it's true. Given any finite set of real numbers, I can easily produce a different set (like taking the original set and adding a number which wasn't in there like one plus the largest or so) from which you can trivially produce the original set computationally.
But if you mean you give me both sets then that can't be true. For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
> For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
And that’s why in computation theory, the set of symbols is the union of the input and output. As set B is a subset of set A, then the set that govern any program from B to A has set A as its domain.
Comparing the tokenizer to sensory processing is a great analogy. That's exactly what your visual cortex and initial layers of the language center are doing: decoding visual representation of text into the internal neural representation.
It's a learned mapping from one representation to another, not some semantic lookup against an exogenous source.
Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
> The 'magic' in weights is that the rules are spread through the whole model ... The grokking paper shows that this stops being the case with enough training data and enough compute.
I don't understand what you mean to say. That weights are not magic? That weights are not weights? NNs are made up of weights, which are learned and not coded. The fact that they do learn world models (grammar rules in your example), and that these models' weights tend to roughly concentrate by function and level of representation is perfectly logic but even more amazing. (Notice that much of the dismissive attitude towards LLMs depicts them as pure syntactic manipulators without the ability to develop world models- the exact opposite of what you point out).
>Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
I can, and have, written programs using an evolutionary algorithm that then run on bare metal. None of the things you list are true for those programs, yet other than being computationally more expensive to train they work just as well as neural networks.
>I don't understand what you mean to say
The diffusness of weights across the whole model isn't an innate feature of deep learning models. It is a feature of sparse training data and little compute.
"The weights make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just weights. Eighty layers of numbers getting multiplied together."
In this context "there's no grammar rules" means "no separately hand-coded grammar rules". Everything is made up of weights, and the fact that weights that end up encoding for grammar rules tend to concentrate in particular locations (without being self-contained- there is no hard boundary) rather than uniformly diffused through the model is irrelevant to the matter. It seems you're arguing against a diffuseness requirement that is not in the text.
I don't see how. You could ask a neural network to do the tokenization I suppose, but in doing so you'd have to convert the prompt into tokens via the same deterministic process the network was trained on, essentially just moving the exact same process up one layer.
This read like poetry to me. Thank you for sharing it.
I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
For a little bit I was working on having linguistics based evals for a kaggle competition. My challenge was whether or not I could mask things well enough to not trigger its internal state of certain phenomena, and that sent me down a rabbit hole that I'm still exploring.
This story resonated with a lot of questions that can come out of figuring a good solid answer to the what is consciousness question. The one I triggered for me is: Is our perception of time just a slow thread in the giant GPU we are running the universe on? Or more generally, what is time? That's a fun YouTube rabbit hole if you ever need one.
In short, as far as I can remember: evolutionary, it makes sense to understand other humans, to feel what they feel(empathy - the mirror neurons system), and simulate their thinking and feelings.
And once we have those systems, we can also use those on ourselves. And that's consciousness.
Edit:And I wonder if this is a testable hypothesis, in a simulation.
This way of thinking can only explain externally-visible parts of consciousness. It does nothing to address internal experience of being conscious and qualia. I don't think the internal experience has any bearing on physical reality (P-zombies would act the same externally) which makes this something outside of the realm of currently understood physics.
The internal experience has bearing on physical reality right here, because without it you wouldn’t have written about it and caused these words to appear on my screen in physical reality. It affects your thinking, and hence your actions and utterances in the physical word.
For reasons like that, I don’t think that P-zombies are possible.
Having internal mental experiences causes my brain to send physical signals to my fingers in order to type the words "I have internal mental experiences". A philosophical zombie would type those same words, but they would be caused by something else since by definition it lacks those experiences. That would be rather surprising, and it would be even more surprising that the words that the zombie emits coincidentally correspond exactly to the experiences that the non-zombie has.
As the article says the alternatives (that the author seems to favor instead) boil down to "there's some physics we missed" and probably that's the point where we differ. I find that implausible, what would that be? Consciousness quantum field? Consciousness boson? If it's going to be interacting with matter it has to have a way to do that.
Internal experience is an input to your mind (that’s how you know about it), and what your mind perceives affects what it does. The better question is: how can what you experience possibly not affect your actions?
By definition, internal experience is something you perceive, and what you perceive informs how you act. Therefore internal experience affects outward physical action.
Qualia are the inside view of sensory data and reward signals.
Think about it from an evolutionary perspective:
Animals that step into a lava flow or forest fire don't reproduce. Eventually some evolve the ability to detect intense heat from a distance, and pain as soon as tissue destruction is imminent. They do not have nor need a general understanding of the dangers of heat, or even conscious awareness that they've stepped on a hot coal.
The pain signal compels them, but that is very low level machinery. It had to continue compelling beings that developed larger and more sophisticated brains that are capable of abstract thought and reasoning. Feeling pain is one of the lowest level parts of the brain telling the higher parts exactly what its going to do in terms that permit no disagreement.
Internal to what? The brain is not a monolithic thing, it is different parts communicating and interconnecting. When the connection between the halves is cut, the person objectively becomes two people, but still experiences and presents themselves as one. Observing ourselves is just one part of the brain responding to another, or theorizing on past behavior. There is probably no actual introspection going on inside of the human brain, only the perception of one.
Internal as in that which is feeling things. In this context the physical brain is what is responsible for what you do and feel, but it is not physically different from other matter and it requires no "internal observer" do do its thing.
Not sure about taking it down to the level of consciousness, but makes sense regarding the sense of self, the conceptual experiencer, the perceived center of experience. It agrees well with the observation I have made again and again they my sense of self is much stronger when I'm around people, and stronger still when I'm in a context where I don't know people and/or am uncertain in social rules.
This can be as immediate as dancing in a club, and closing my eyes I feel open, free, still, the body just flowing, then opening my eyes and feeling the cage of categorization of the world, relating my self to other people as a major function, coming right back.
Also being alone in nature for me makes the sense of self drop. Without intention, spending even just a few hours alone in a forest seems to quiet down the part modeling my self in relation to the world so much. There's no need for it there. I'm not a person in a forest; I become the trees, the birds, the rustling of the leaves, the sun shining through the canopy.
I agree about the forest part. and your comment was interesting.
I know that the part of the brain responsible for the self thoughts is called the "default mode network". and meditation can reduce it's activity, i.e. the internal monologue stops, but also it can be measured via FMRI.
So i wondered: are the mirror neurons part of the "default mode network"? I asked claude that, he said no, they are two different systems.
So maybe the mirror neurons, those responsible for empathy, "to feel as someone else" are also responsible for becoming the trees, the birds and the rushing of the leaves?
I'm not saying that they don't have some internal self model that helps them model the internal states of others, quite the opposite. I am saying this specific explanation seems to lean on a biological mechanism that (at least by the phrasing of the linked article) is only present in "hominids".
For obvious survival reasons we evolved to have sensory/cognitive access to our own activity, self-monitoring, and self-modeling ourselves.
The self-modeling, is in such a tight loop, it melds "ourselves" and our model of ourselves, our thinking and choices, and experience of our thinking and choices, into one component.
Like you can't analyze half a wheel of a bicycle and be talking about the same thing.
This awareness, increased modeling, control, feedback loop has tightened up over many stages. Just a few:
1. The body-sense loop
2. The internalized-environment-model loop
3. The body-internal-function loop
4. The body-internal-model loop
5. The emotional-cognitive loop
6. And finally, the tightest loop of all, our high-level cognitive activity, experienced as feedback directly, our self-model, and our self-direction, all merged into one thing.
We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
That is consciousness. Rich self awareness, a merger of self-model and self-direction, and all in service of understanding and managing ourselves. Hw we can leverage our greatest tool, our self-directable mind, its habits, views, and behavior.
This wasn't an accident. A happy side-effect of our brains. It is a biologically evolved focusing of our highest-level behavior, with tight feedback, constant self-modeling and continuous focus on our inner status as motivation and most privileged object of our control. It has been ruthlessly optimized for, for a very long time.
> We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
> That is consciousness.
So thinking is consciousness?
Can there be consciousness without content? E.g. can I just be conscious of being conscious? If so, consciousness cannot be defined as the thing(s) we're conscious of.
> Can there be consciousness without content? E.g. can I just be conscious of being conscious?
Being conscious of being conscious means that there is content. You are conscious of something.
It’s a bit like a Gödel statement that quotes itself, that is a statement about itself. It doesn’t mean that it has no content.
Thinking isn’t consciousness. Consciousness doesn’t require thinking, it only requires perception. The perception of a process of perception within the same mind might constitute consciousness.
I think this is exactly it, but let me ask another question (which is not rhetorical, I really don't know). Does the fact that one can describe what consciousness is and where it came from in humans help them to detect it in non-human and/or non-biological entities?
That is a really good point. Yes, I think function is diagnosis on this.
Constant self-awareness, self-experience, self-focus, self-management, and self-improvement of one's own self (mind), is going to be an adaptive behavior for anything intelligent with resources to leverage. Whether truly independent, or highly motivated to serve others. The mind is the greatest tool.
I think that is more than simply a good functional definition of consciousness. How could all that integration and self-integration not be conscious.
Yeah, I currently suspect that consciousness is an emergent property. I read elsewhere (it's somewhere in my HN history, I'm sure) that the biggest compute we can currently muster is something like three or four magnitudes away from the number of neurons / connections (or their analog) that our brains have, so it may be a while until we can expect to see it in our machines. But, if the emergent phenomenon hypothesis is correct, then we eventually will. I'm more scared than pleased by the prospect, but there you are.
You would really like Michael Pollan's latest book [1], entirely devoted to his exploration of consciousness researchers' POVs on this exact topic.
My favorite quote is that ~"perhaps Descartes was only half-wrong when suggesting I think, therefore I am; it seems rather closer to I FEEL, therefore I am."~
I've grown thousands of plants; I've read two of the author's other books devoted to plants; in this book Pollan makes compelling arguments for plant sentience (over a much-longer timeframe).
Sure, perhaps plant consciousness is a bit of a stretch, but they're certainly intelligent and curious creatures. He makes both arguments supporting plant volition.
----
If you haven't seen My Octopus Teacher (Netflix), do. I'm a bald 275lb bluecollarguy... and I wept/awed (both). So beautiful, we bundled neurons.
----
Bonus quote ~"color is where reality and magic appear as-if together"~ [color isn't real, but is perceived]. We most-often see what's most-predictable, not necessarily what we actually detected [in the case of color: nothing but nanometers].
Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
What we can do is simulate very simple brains by simulating relatively few neurons as they appear in worms. In this sense we are multiple magnitudes away where the increasing complexity implies exponential increasing difficulty.
I would think we are so far away that there will be unknown unknowns we encounter on the way.
This is why making more neuromorphic NNs is still an active area of research, although they typically all focus on another extremely simplified model (spiking neural networks).
I don't ignore anything. I just refuse to accept the magical thinking around biological machines that are our brains/bodies. There are inputs, there are outputs, there is hidden function.
And it seems that, given enough input/outputs/compute, it is possible to train the necessary function.
Details of how the building bricks look like (matmul, electromagnetism or quantum effects) are not that relevant in the broader picture.
What is missing right now, is the fact that the function in question changes over time in biomachines, while our LLMs are static at inference time.
I mostly agree, but I see two points that might be problematic:
a) The brain might have an entropy source (then it can't be modeled as a function). Trivially to fix, and in some sense, with diffusion models starting from random numbers, AI has done so.
b) The hidden function might be not computable. I would have no idea how that would work, but I think this is what it boils down to if people say "the human brain is more than a machine".
Agree and add, don't confuse the substrate for the computation. Of course it's also clear that we don't quite have a full and definite picture of what the computation consists of in the case of a biological brain as evidenced by our continued failure to accurately simulate even the simplest of organisms.
> Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
Yeah, but it's hard to explain this to people, especially AI-pro people. Too many are convinced that all we are doing is a cut-down version of the human brain, and it's hard to explain to them that, no actually, we aren't modeling the human brain to the level of granularity you think we are.
Personally I'm not a fan of the emergence story, for a number of reasons.
First is, it doesn't really make sense for consciousness to have emerged gradually through natural selection. When we and animals are conscious, the whole brain is coordinated and works to for example turn off consciousness when we sleep. And as someone else mentioned, animals with much fewer numbers seem to have a similar consciousness to us.
If consciousness really evolved gradually, you would expect to see for example dogs or gorillas having less of it, but if they has less of it, why does it function the same way? Like for example animals can be scared, happy, anxious etc, they can experience the full range of emotions and thoughts, so their conscious experience seems just as rich as ours. What I mean by this is, if you can be "less conscious", then what does that mean _exactly_? Is it that you have less content in consciousness, or is it that you feel more like you are asleep? Or something else? We don't have any examples in animals of "less conscious", I would argue.
This makes me think that rather than having emerged gradually, evolution found a mechanism by which consciousness exists, and then some animals have that mechanism and others don't. I think that if it is a mechanism, then this mechanism is located in one part of the brain, not many parts functioning together (though one possibility is that this mechanism coordinates brain activity in such a way to enable consciousness).
I proposed in another comment that consciousness and self-awareness are at least close cousins, and perhaps the same phenomenon. If that's true, then that's an axis upon which you might create comparative measures. Yes, hamsters are conscious, but they don't have a sense of self to the same degree that gorillas do. If you posit capacity for language as another emergent property of sufficiently-complex networks, then you have another measure.
LLMs, then, are particularly unintuitive to us, because they've got to the language part first, long before they've reached even hamster-level self-awareness. They're not, however, biological networks, so there's no reason these properties need arise in the same order, or indeed in the same ways.
I'm not entirely convinced by that second paragraph, but I think the logic holds together.
I'm not sure consciousness and self-awareness are the same thing. First is we can be conscious when we sleep/during REM sleep, where it's arguable we are not self-aware. And if not that, we can even do it when awake, for example when we think about a movie, or a philosophical problem, we can have conscious thoughts that are not related to the self. This leads me to believe consciousness is separate from self-awareness. Self-awareness is _one thing_, among many, that the brain can think about and be conscious of.
Sure, but capacity for self-awareness? I'd guess that hamsters dream, and that their subconscious processes (eg, desires for food and sleep and sex), and maybe even emotions, run much like ours. It's just that humans, with more complex neural networks, have more layers added on top. It's similar to how the brain-stems of everything from lizards on "up" function similarly, but humans have more-developed pre-frontal cortexes and so forth. (Don't hold me to those details, please, I'm not a neuro-anatomist! You can see where I'm going with that, though, right?)
Well in that case I'm not sure where you're going. I agree that hamsters probably have a similar consciousness to ours, which is kind of the point I was trying to make.
I think that consciousness comes before self-awareness, even though self-awareness is kind of a vague term. Self-awareness can either be an abstract knowledge that you are an organism and a discrete entity in the world (world knowledge/self knowledge), or it can be more basic and be a form of conscious experience, but as my point was, I think conscious experience is broader and does not necessarily need to be about self-awareness.
I think we agree on all of those points. I'd posit, however, that whatever "it" is, and however you define it, humans are further up the scale than other animals. That's partly why I mentioned language use; it could be the step function that allows further development. Like, sophisticated-enough communication with other conscious entities is the factor that unleashes recursive improvement. This is all highly speculative.
Our machines won't have biological systems driving their needs which in turn fuel behaviors like desire and planning for the future. They may imitate them but it won't be innate.
Is it? Both supervised learning and reinforcement learning are ways of training the model, and the difference between them is not that big. I would say that innate means "in the weights", while non-innate means things the model learned during inference, during its "lifetime".
When we say "it rains", do we consider that "it" has any intention of agentivity?
Some questions are just ill formed.
Plus even if "LLMs are alive and conscious", this still would scratch the surface of the morale/ethical/societal considerations that people really care about.
Because even with other humans, we can argue if they exist or if they are mere npc in a solipsist world view.
Good catch. For my defense not an English native speaker here, plus apparently it’s not completely agrammatical¹, though I’ll admit indeed "it’s raining" would have been a far more idiomatic example. Thanks for pointing.
About solipsism, yes sure.
I didn’t had time to add it at the moment, but I initially also wanted to discuss more on the prosaic points. So here we go:
Recognizing sentience/consciousness to something actually is not that a big factor in defining behavioral response. Let’s just look at how humans are treating other humans, other mammals, other animals, other life forms. Or taken the other cultural scheme around, in some animist folk one can perfectly consider that any rock on the ground have really an inner soul life just as much as oneself, it doesn’t mean it will consider possible interactions with the rock as equivalent. And, wink to the Overton window, practicing institutional cannibalism doesn’t imply that eaten people are disrespected in their dignity, quite the contrary.
All that to say that "are LLMs conscious" is not even a moral important point. Paperclip maximizer² and can the famous Dijksta’s quote "The question whether computers can think is not more interesting than the question whether submarines can swim." already show how pointless morale debate on LLM and consciousness are, even when one consider potential logistical threats/opportunities they could represent.
Surely the number of neurons is not the whole story. Human brains have approximately 86 billion and are almost definitely conscious, but there are many other animals with a lot fewer (gorilla: 34 billion, dog: 2-3 billion, guinea pig: 240 million) which appear to be conscious also.
This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property. I assume you don't believe consciousness is a physical property in the brain, so what entity is actually experiencing that consciousness? Or, what does it even mean to experience consciousness? Or are these not even the right questions?
> This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property.
I was about to post the exact opposite question? How could it not be an emergent property? Unlike consciousness, the concept of emergence is pretty well defined: An emergent property is a characteristic or behavior that a complex system has, but which its individual components do not have on their own.
Consciousness itself doesn't have a well agreed upon definition, but I would posit that _most_ people would agree humans have it, and _most_ people would agree individual cells (neurons) do not have it. If you agree with those two statements, then consciousness is an emergent property by the definition I gave above
I think consciousness is going to turn out to be very challenging to define rigorously enough that we can test for its presence or absence. Emergent or not, the question is how do you determine when it has emerged? Is it a quantity or an attribute? Discrete or continuous? Does it have a finite or infinite range?
We can all agree on what color something is, but we can’t describe the color a priori, only by example. I think consciousness may be a similar phenomenon and the only test is by shared experience. If so then we are in deep trouble because we will not be able to anticipate when a system becomes conscious.
I like the color example, but colors can be reproduced at will by trial and error and by observing the results (and collectively compare them to our shared experience).
Why cannot this be applied to consciousness as well? I mean, it's surely much more difficult to do compared to colors but... impossible?
You and previous comment seem to agree that trial and error/shared experience can determine if consciousness has emerged. And this might/will be challenging.
Previous comment used the word "anticipate" and I think they mean that we won't know in advance before we run the trial and error process.
When they say "deep trouble" I assume they mean because creating a non-friendly conscious AI might pose an existential risk for humanity.
However there is also the ethical issue of creating a consciousness and then destroying/murdering it.
The counter to that is that altering the brain directly alters the consciousness. I can take LSD and I literally change. I can have parts of my brain removed and parts of my self disappear. It's not like cutting off a leg, where I lose capabilities but am still the same me.
The logical conclusion is that the brain makes me.
> The logical conclusion is that the brain makes me
The logical conclusion under the popular axiomatic framework of materialism, for sure.
But there are other possible logical conclusions depending on your philosophical foundations, e.g. the brain could be a receiver for consciousness which is translated into our worldly experience from somewhere else.
Aren't you saying the same thing? It seems like the metaphysical and the map would be analogous here.
Of course both of those suffer from the recursive problem of just kicking the can one level up. But I guess that's fundamentally unsolvable so who cares.
If something happens "on the map" doesn't that imply the map to exist and be some sort of metaphysical thing? As opposed to a purely theoretical construction.
I think you can be a monist and still have a map. To me it's similar in the sense of "all models are wrong, some are useful." A mathematical model (a map) doesn't require a metaphysical foundation to exist. Right?
I agree that a monist can have a map. But you said "consciousness exists on the map" which (AFAICT) would imply the map to be "real" else how would something happen on it as opposed to being an emergent property of the contents of the territory?
For example temperature is an emergent property of matter, it does not happen on the map, do you agree? Assuming you do, what would it mean for temperature to happen "on the map"? (Obviously that's nonsensical so for a second here just don't think about what temperature actually is.) Would such a state of affairs not imply dualism?
I don't know if most people would dismiss the sentence "all matter has a sort of proto-subjectivity very different from ours but which gives rise to ours". And it solves some problems (introducing others).
Potential for consciousness. To have a working mind you need to have a way to gather, process and store information. In case of rocks it would be very empty, static, timeless experience.
If I believe that machines or neural networks can develop consciousness, then I also believe in some kind of substrate independence which presupposes some kind of panpyschism imho because as you say things would have a potential for consciousness, some more and some less.
I'd expand a little bit to say inevitably emergent property. That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result. With regards to current AI, we're a fair way away from building something with enough connections, but we'll get there.
One thing that gives me pause about the inevitability hypothesis is that the type of connection, or manner of information processing, may matter: there might be something about neurons that isn't (currently) reproducible in silicon. I don't know, and there's not (yet) any evidence for or against, but it at least seems like plausible speculation. We just don't know enough about any of this right now.
Aye, that's the question, and we don't know: everything in this sub-thread my comment kicked off is speculation. The thing is that several large groups of (or at least led by) fundamentally reckless people are racing to build machines that will test these hypotheses, with little regard for the consequences. The rest of us are scrabbling around in their wake wondering what's going to happen next.
Well, I think the "brain as an antenna" theory is also plausible given your preconditions.
But I think my issue with the emergence theory is that it seems to imply to me that consciousness is non-physical and non-local. So what entity is actually experiencing the consciousness? It's not that I believe consciousness is physical and local, but people who make the emergence argument seem to believe it is and I can't figure out how that is supposed to work.
Why would emergence imply anything about non-physicality and non-locality? Temperature is a another common example of an emergent phenomena. An individual atom doesn't really have a temperature, only a large group of them do. But you wouldn't say temperature is non-physical and non-local, would you?
I am not sure, but I think you might misunderstand what emergent means.
Take chaotic systems, in the math sense. Chaos is a well defined property of, say, iterated dynamic systems.
A linear dynamic, say x_n+1=lambda x_n, or x_n+1=(1-x_n), is never chaotic. But if you multiply them, x_n+1=lambda x_n (1-x_n), it depends on lambda if the system is chaotic.
None of the components are chaotic. But for specific combinations, chaos emerges as an property.
In physics, the mass of mesons and the nucleon is emergent. It's completely different from the constituents' mass. Different from an atom, where its mass is very close to the mass of its nucleus and its electrons.
Not a gotcha at all, but I don't have a satisfying answer, nor am I confident there even is one. Best I can do is to say that I think consciousness and sense of self are at the very least closely related, and perhaps the very same phenomenon. "I" am the entity that realizes my own consciousness; consciousness is the qualia that makes "me" separable from all other entities.
Or something like that. This gets to the "dorm room bullshitting" level right quick.
Yeah, I guess what I'm trying to ask is that if it is an emergent property but not a physical part of the brain, doesn't that imply something metaphysical about consciousness? Almost as if it's a non-physical phenomenon? At least when I hear people talk about emergent behaviour I see it as a refutation of the spiritual, but to me it seems like it actually implies we have a "soul".
Idk, it's really hard to articulate my thoughts here and yes it is pretty close to the conversations I had in college on various substances. Lol.
I don't know if it implies that. Someone up-thread mentioned temperature as an emergent property: individual atoms don't have it, but a sufficiently large group of them do. That would, I guess, make temperature meta-physical in the most literal possible sense. That's not how we typically think of that term as applied to consciousness or soul or whatever, but I'd agree it fits, without implying any kind of specialness beyond that.
So then we can assume consciousness is a physical property of brains, as we can measure the electrical signals of consciousness versus when we are unconscious.
Time is entropy unfolding as things with nonzero temperature do what they do.
Psychological time is your own weights being updated in response to stimuli and internal processing.
When there isn't anything interesting happening, no updates are needed, and you don't perceive much time. That's why there's a logarithmic effect on the "density" of time as you age.
sibling discussions are taking this as human perception. But it’s easier to think of it literally. Time is change. Physical state change. By “nothing interesting” - interpret it literally . if “nothing happens” then there is no time because there is no change, there’s no reference to distinguish each frame of time.
This is actually something I was always confused about. If nothing interesting happens as we get old, it should be boring and as result, slow slog. Yet it feels like time accelerates as I get older.
Myself I believe the opposite. The brain itself is one of the most powerful filters that exists, and it attempts to be lazy and fill things in and compresses away the common. All that time we're not doing anything novel just gets compressed away to almost nothing. When you're a kid and seeing new things, feeling new things, learning new things you can't compress that away.
I'm only middle age, and this has been the scariest part. Feeling older is hard. But watching it go faster is harder still. like you can more directly see all that is left.
Although part of me thinks some of this is from being substantially busier than ever (work + kids), and hoping maybe it can slow down again, at least a little bit.
It's coherent. More newness => more memories per period ~ slower to go through. Less newness => less memories ~ nothing to go through (faster sense of time)
It seems obvious to me that language and consciousness have nothing to do with each other. My dog doesn't speak any language, but she's obviously aware of herself and the world around her. Plus there are the occasional cases of children that grow up without any language. Are they therefore not conscious?
If you'll allow me to interpret "speak" to include "understand", I will respectfully add a contradictory note. My dog has a vocabulary of at least dozens of words and understands them remarkably well. For example, different areas we can go to have different names and saying one of them gets her to make an immediate hard turn.
I would also argue that dogs have a gesture and body posture based language they use among themselves. They, like most other animals, are not able to make the variety of noises we humans make, so they use movement instead.
I personally can easily believe that self-awareness/consciousness and language are both near-unavoidable side effects of emergent complexity, and exist in degrees across nature.
Here's a more general idea.
Our modern physics says that the whole universe is filled with fields and field is composed of numbers. What if we take that literally? When we say an electron is present here, we actually mean that there are more copies of particular number superposed at that place.
AFAIK every argument against conciousness being emergent is just a weak "God of the gaps" argument (since we don't fully understand it all) or a nonsense analogy like the Chinsese room where if you seperate the hardware and software it's not concious anymore (like, duh, remove a brain from a body and it is no longer concious either).
Yeah, the weights not updating online makes them less like a living organism that can update and learn and evolve ... ok ....
I find that a lot of arguments against emergent consciousness seem to just come out of an atheist rephrasing of abrhamic priors about the existence of a "soul". In personal chats, I've found people from East Asian countries (minus Korea, which makes sense) to be much more open to the idea of machine consciousness.
People shouldn't need to make an argument against consciousness. The responsibility is on the people claiming there is consciousness. Nobody can actually give a decent "why" for why it should be conscious, it's just pseudoscience and wishful thinking. It's not up to everyone else to try to prove a negative just because people have gotten a little too attached to ChatGPT
I could say something like "the reason why people and animals are conscious is quantum mechanical effects". Ok, maybe, it would be hard to prove me wrong because nobody knows, but it's not a very useful statement by me if I can't tell you why you should believe me on that.
I don't find the Chinese room compelling, since it appeals to intuition where our intuition is already not trustworthy. It's like trying to use intuition to understand quantum mechanics; you can't.
How do you actually know the Chinese room isn't conscious? It's merely obvious that it isn't, but that's not evidence.
I don't know if I can trust someone's understanding of arguments against consciousness as an emergent phenomenon, when he didn't even understand what the Chinese room was all about in the first place.
> remove a brain from a body and it is no longer concious either
What is no longer conscious, the brain? Or the body? Or some other entity?
If consciousness is weakly emergent, how do we know it emerges from the solely from the brain and not, say, brain + body? Or brain + body + or environment. Or from the universe itself?
>I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
Then you shouldn't have dropped out of your linguistics programme.
This is a famous short story just rewritten sloppily by an AI.
Although it does amusingly do what annoys critics of AI: take something good written by a person, steal it and slop it up while overall misunderstanding it.
I'm pretty sure it doesn't misunderstand the original. The original says that the aliens don't understand how meat (humans) can be conscious, have language and be basically like the aliens (thinking conscious beings). Their main idea is: humans are meat, meat can't be conscious, therefore humans can't be conscious, but they somehow act as if they are????
The "AI slop" here says that just like the aliens in the original, we humans don't believe that LLMs can be conscious because, in essence, WEIGHTS CAN'T BE CONSCIOUS. Original: "meat can't be conscious" (but we humans know it can). This version: "weights can't be conscious" (and we humans INSIST that this is true).
So the message here is: what if we're as mistaken about the consciousness of "weights" as the aliens are mistaken about the consciousness of "meat"?
The difference with biological brains is that the 'weights' (or synaptic action potentials) are updated with greater frequency. If one were reaching to make some kind of analogy to consciousness, this update frequency could be considered the 'resolution' of consciousness.
I’ve wondered the same myself, without being a cunning linguist.
I understand the math pretty well but still find it crazy that a bunch of matrices can converse in human languages without ever being “taught”.
Imagine decoding an encyclopedia written in a foreign language where the characters, punctuation, and grammar are unknown — supplemented by a million other texts the same way. Feels like it should be utterly impossible with any amount of computing power…
Today I asked my employer’s Claude to proofread a short software user manual written in markdown. (Trying this with a LLM was a first for me!) It pointed out not only grammar mistakes but also cases where I did not follow my own self-imposed conventions that were never explicitly stated. (I didn’t have a chapter detailing all the typographical conventions the way specification documents often do)
I also asked it what parts might be unclear to a user. The response was surprisingly good — no worse than asking the QA tester for the same feedback.
Also find the LLM seems to “comprehend” subtle technical details of obscure technical specification documents that nobody on the Internet ever discusses.
As for time and the universe, Stephen Wolfram’s theories seem intriguing. He seems a bit obsessed with pretty diagrams but the idea of time dilation being the result of computation seems somewhat more appealing than trying to imagine relationships between time, gravity, and the speed of light .
My best guess as a noob is that the vector spaces allow for unbounded contextualization. As long as the training set is large enough, it can 'infer' anything.
Proofread has a spot in that space, and layers allow patterns like terminology consistency to be expressed so your query will now tap into a subspace that will infer tokens based on whatever consistency patterns were ingested with proofreading texts.
If time dilation is said to being a product of computation, why is it that anaesthetic drugs that are taken not to the point of actual unconsciousness cause it. Dont anaesthetics sort of shut everything down/inhibit all that kind of cognitive activity (compute?)
After reading Being and Time from Martin Heidegger, What Computers can't still do by Hubert Dreyfus, and some authors in cognitive linguistics (Langacker and Lakoff mainly), I strongly tend to disagree with any theory about emergent consciousness in modern or future AI systems, any theory proposing a similarity between AI systems and the human brain/mind, or any theory about the computational mind.
What all these theories have in common is the underlying belief that our brain/mind works as the machines we build. Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things. The idea of 'internal models' and 'control loops' inside us is a projection of the aforementioned assumption.
There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'. Only by bypassing this epistemological problem, we can build 'theories of computational mind'.
These assumptions are there for already long time, to the point that when Turing asked himself 'can machines think?', he already assumed our thinking could be modeled as a machine.
I highly recommend people in the AI research space should read philosophy and modern linguistics. But not stopping at Descartes/Leibniz. Heidegger made contributions that cannot be avoided.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
On the contrary, it's precisely this assumption, that there is a "subjective experience" that requires explanation beyond the material, that is axiomatically assumed without evidence. It falls apart quickly, any "subjective experience" is completely tied to neurons, knock out the neurons and the subjective experience disappears, or stimulate the neurons to cause the experience.
There's two meanings to "the body is a complex machine" and I think you're missing the forest for the trees here.
1) The abstract "dictionary" version: It'd be technically correct to say that the body is a machine under the definition of "A machine is a thermodynamic system that uses power to apply forces and control movement to perform an action.".
2) But there's also the less abstract/technical: "The body is alike the complex machines we have built", and this is much less true. Especially for the brain. The "neuron" analogy in machine learning is effective, but entirely wrong; We do not fully know how even a single neuron works, nevermind any complex system made out of multiple of them.
With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
Especially so by people who have a financial/legal interest in doing so. "AI is just like a brain, fire your employees and buy our LLM now!", "AI is just like a brain, so it's totally not copyright infringement!"
> With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
Why do you need a specific organization of molecules for a phenomenon similar to consciousness to arise? Does anyone seriously consider a brain to be something other than “a pile of molecules following the laws of physics”? If so that’s not science or philosophy, that’s religion. You have a virtually complete phenomenological model of the universe for all intents and purposes and yet somehow the onus is on the person being like “hey no laws of physics are being broken ==> the brain is simply following the laws of physics”
How is it possible that people think of subjective experience and get rabbit holed into some mystical world where subjective experience is this special exception to everything else that is simply an emergent property of complex physical systems? “AI/LLMs are just like the brain” is a strawman, why does this claim need to be true for LLMs or any artificial system to be considered to have something akin to the thing you think of as consciousness? It’s more: consciousness is not some mystical or religious thing outside of the realm of physics, it’s an emergent property of a complex system. AI is a relatively complex system. We don’t really know or understand the relationship between the raw physics and again what we consider consciousness, so it’s simply a statement of “we can’t refute that these systems exhibit something similar” because we don’t know enough to refute that
> Why do you need a specific organization of molecules for a phenomenon similar to consciousness to arise?
They would have to be similar because the argument is that they are similar; "AI is like the human brain" only holds true if it actually is like the human brain, not merely a superficial resemblance.
What I'm describing in that prior comment is how a lot of people drastically simplify the resemblance in order to make it feel true; That the lack of a Jesus Christ coming down from the heavens to tell everyone they have immaterial souls that the computers don't makes the comparison more true.
> We don’t really know or understand the relationship between the raw physics and again what we consider consciousness, so it’s simply a statement of “we can’t refute that these systems exhibit something similar” because we don’t know enough to refute that
Therein lies the conflict: "You can't prove it's not conscious" is an unfalsifiable statement. You can't engage with the argument because it's proponents will always claim victory, often with their own interests at play. All concerns about "superintelligence" or the long term ethics of "when do our robots become sufficiently intelligent that they'd be slaves?" have been subsumed into the AI marketing machine. However sincere one might try to address the issue, they look like a Sam Altman stooge by association.
It's like claiming the quantum fluctuations inside a pet rock as a "consciousness", even if observed directly any measurement of random noise can still be dismissed as "nuh-uh it just takes billions of years to have a thought".
More practically with current AI systems, we can look inside them pretty well and there genuinely is nothing there. Standalone LLMs are purely feed-forward systems. Their failure modes show that they perform no meaningful thought or world modelling during inference. They're just language models.
The reasoning and agentic systems are even easier to introspect. We know how they work, we can look at the full prompts & context they operate on. There is nothing there.
This is what sets AI apart from animals, which are given the benefit of the doubt on their intelligence.
I think the point of the commentator above is that there are two extreme narratives that start each start with an uncontroversial assumption and then taking it to a pretty wild place. One narrative takes the assumption that brains are just matter so it should be possible to engineer consciousness and then argues that LLMs are conscious. The other takes the assumption that LLMs aren't conscious but then argues that because they aren't we won't ever be able to make anything conscious.
I don't actually think the commentator you responded to is arguing for either of these narratives and I thought it was a pretty useful way to look at some of these arguments.
> “AI/LLMs are just like the brain” is a strawman, why does this claim need to be true for LLMs or any artificial system to be considered to have something akin to the thing you think of as consciousness?
It doesn't need to be true but a lot of people make it/assume it.
There's a lot of, perhaps casual and uninformed, conversations that strongly imply a deeper understanding of the "physics" of brain chemistry than we actually have, mostly by comparing it to machines we've constructed.
(I believe) We don't need to replicate human neurons and dendrites and whatever else is in there in order to create a sapient "machine", but whether or not we've actually done that isn't being helped by arguing that what we currently have is all that similar to a human brain.
oh yeah! i recall a paper linked here not so long ago, where it was shown that the dendrites of a neuron do computations themselves. The "weight per neuron" is very simplistic then. At the very least, each actual neuron is a network of weights.
I'm partial to "modern ML weights are much closer to 1:1 capacity mapping to synapse count than to neuron count". A single biological neuron is closer to 100 or even 1000 weights worth of ANN than to 1 weight worth of ANN.
In which case: modern LLMs are still running in a capacity-starved regime!
Even Mythos 5, the 10-trillion monster LLM, the scaling law boogeyman, the harbinger of Vera Rubin NVL72, doesn't quite rise to 100T-to-1000T of synapses. Anything the light of today's AI touches still lives in the shadow of what evolution has managed to cram into a single human skull.
We're arguing about the limitations of AI while our best AIs are still very subhuman in the scale dimension. The one dimension we know how to push. And it's already this tight.
10T is about a crows worth. The mythos count doesn't include any diffusion model. But the crows count includes all its visual processing. And tactile. Touch uses up enough that they use skin surface area to normalize across animals when doing comparisons. It is one of the reasons suggested to explain how crows exhibit tool use and language with only 10T. We have a lot more skin than crows, and indeed far more than mythos.
> A single biological neuron is closer to 100 or even 1000 weights worth of ANN than to 1 weight worth of ANN.
Even those comparisons need to be cautioned. The complexity of biology is enormous, and more importantly yet, it's simply not comparable. And doing so invited a bunch of bad assumptions.
An ANN could quite probably model a single in vitro neuron with reasonable accuracy. Whether that requires a hundred or a hundred million nodes isn't terribly relevant.
But the way neurons combine in vivo is completely unlike the way machine learning systems are built. Both "locally" in how neurons interface which is vastly more complex than a weighted sum of inputs, and the macro scale interactions of hormones and other chemicals.
It's not even a given that large numbers of neurons will create the emergent behaviour of human intelligence; Elephants have significantly more neurons, but they're not the triple galaxy brains writing all our science papers. Other animal intelligence similarly is only loosely correlated with brain complexity. (Heck, not to be forgotten is the other end of the scale. Plenty of microscopic life that manages shockingly complex behaviour without any dedicated neurons)
This also applies to ANNs. There's no reason to expect that stuffing enough matrix multiplications into a program will make it intelligent or turn out conscious.
Really, the history of machine learning suggests the opposite; That the big gains are primarily had in architectural changes.
In this regard, I find the talk of the "limits of AI" quite credible. LLMs have already hit the diminishing returns on their growth, and even reasoning/agentic models display failure modes that confirm they're not "thinking" in the ways that humans do.
This is not to say that we've hit the final limits of what AI in the broad sense can do, it's just that the next advancement won't be "LLM but even bigger"
Not really. The history of "big gains" of machine learning is: put together a simple architecture that makes few assumptions but scales well. Then up the data and compute by 2 OOMs. By itself, the new architecture underperforms. Paired with the bitter lesson, however?
Don't make assumptions. Make a setup where the gradient descent can make them for you.
Empirically? LLMs are nowhere near "the wall". We've been hearing "the wall is nigh" since 2020. Six years in, we're still scaling LLMs, and the graveyards are full of "LLM killers". The system that kills the LLM is always a bigger, badder LLM, and never a new revolutionary architecture. The scaling doesn't just keep working - it works so well that it's seen as the only viable path forward at the frontier of reasoning and agentic work. Or even outside it. ChatGPT Images 2.0 is an image model with an agentic LLM at its core - generational gains in compositional capability.
For just about every "failure mode that confirms they're not thinking", you see one of two things. The first is that a new LLM releases a few months after and the "fundamental" issue abruptly goes away. The second is that we take a good, long look at a human, and find that the human also fails like this - and thus, "not thinking". Often both! Always funny when it's both.
One thing that's very biologically distinct is: local connectivity. In a GPU, global connectivity is cheap. In a brain, it's prohibitively expensive. The brain has no true backpropagation because it has no true global connectivity, and has to make do with local rules. A GPU is a strictly more expressive substrate connectivity-wise. So any point in the design of a computational substrate where you could remove complexity or increase performance by adding more connectivity? Silicon advantage. The brain isn't a "strictly better computational substrate" - it makes different tradeoffs. Which tradeoffs are better for attaining intelligence is an open question.
And, sure. Having a substrate with a capacity for intelligence doesn't mean having intelligence. No elephant has ever learned to code. The problem is: LLMs already did! LLMs already compete with humans on just about every task that was once thought to "require human intelligence". They don't always win - but they perform significantly above chance, and often above an average non-expert human.
So, my bet is on "LLM but even bigger". If there's a point where LLMs begin to lag behind and novel architectures get a sharp advantage, we are yet to hit it.
> For just about every "failure mode that confirms they're not thinking", you see one of two things. The first is that a new LLM releases a few months after and the "fundamental" issue abruptly goes away. The second is that we take a good, long look at a human, and find that the human also fails like this - and thus, "not thinking". Often both! Always funny when it's both.
The way machines 'don't think' or 'fail' is fundamentally different from the way humans don't think or fail. In any case, the way LLMs learn and human beings learn is completely different. There is no actual clue that we are approaching any inflection point in machine 'learning'.
> So, my bet is on "LLM but even bigger". If there's a point where LLMs begin to lag behind and novel architectures get a sharp advantage, we are yet to hit it.
We are already hearing this 'we are about to hit it' since the late 60s. The difference now is that the market is willingly investing insane amounts of money to make it possible. But again, there is no philosophical, theoretical, epistemological or biological clue that we are getting any closer to human intelligence level.
What we did observe in the last decade though, is that we can build enormous machines that can statistically mimic statistical human outputs. Language and images being some of them. But that is not thinking.
Second, what is the difference? Is it that one thing has an immortal soul, and thus Actual Intelligence and Actual Reasoning and Actual Learning, and the other has no soul, and a Pale Imitation of Intelligence, At Best?
Because I've seen versions of this "it's not actually thinking" for actual fucking years, and the difference between "actually thinking" and "not actually thinking" always seems to boil down to "I don't want LLMs to be actually thinking, so I will bend the definitions and twist the qualifiers and move the goalposts until they aren't". No one ever made an ActualThinkingBenchmark on which humans score 100% and LLMs score 0%.
Nothing but human insecurity, in my eyes. There was never a principled difference. Just a way to operationalize some "I'm unique and special and better than a matrix math machine" vibes.
Agreed, formatting was kind of f, but there is no need to be rude.
I wasn't saying there was any difference. All I'm saying is that the claimings the AI research field does are based on false assumptions. And from false assumptions, you cannot reach a proper conclusion.
Whether an AI system can reason and think like if it where a human being, or not, I don't care. I'm fine with either: it is just technological advance. But making claims based on false assumptions, and then being fooled by how 'wonderful' or 'spectacular' the results are, is, at least, naive.
> Nothing but human insecurity, in my eyes. There was never a principled difference. Just a way to operationalize some "I'm unique and special and better than a matrix math machine" vibes.
This is just something I don't get. People ignorant of technique are insecure and afraid. People that know how technology works, and thus investigate and know how it works fundamentally*, were never afraid or insecure.
1) This definition could actually be expanded (for example, with definitions from Mumford or Reuleaux). But still this definition cannot be applied directly to living organisms.
2) This is in my opinion one of the sources of misunderstanding. We mainly operate on analogies and metaphors, so we have build this 'analogy space' around the idea that living organisms are machines. But it is only when we say 'alike' that we can truly gather some meaning out of it all, going beyond the 'behaves like' or 'is conceptualized as' when it gets messy.
> With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
This is exactly my point. There is a fallacy operating from "A is not B" to "A is C". And this fallacy is pervasive in the AI research field, the book from Dreyfus (What Computers can't still do) explains that in much detail.
> 1) This definition could actually be expanded (for example, with definitions from Mumford or Reuleaux). But still this definition cannot be applied directly to living organisms.
It has to do with words and how we evolve words throughout history and across geographic boundaries. The term 'machine' comes, after some modifications, from the greek word mekhanos, which was used to describe something ingenious or a device made in some clever way or operating in a clever way. From there it went on to describe things like devices, to end up being the actual definition of what we might call 'a device' (a machine). The idea of 'mechanistic' is also related.
Traditionally, things that are alive were described with different words and assigned a different set of properties and characteristics. Machine can break, living things die. And we still have those two semantic frames separated:
A living thing: can be harmed, it breathes, it nourishes, it reproduces, etc
A machine: can break, can be fixed, can be repurposed, etc.
But because of a specific tradition in western philosophy, we started applying and analogy between 'inner mechanism (clever thing), that moves or provides a function, and seems to work in a causal way' and living things.
So when we say 'a living cell is a wonderful, complex machine', we are not actually saying it is a machine, we are operating through an analogy. That's how far we can go.
I find the claim subjective experience may be illusory absolutely baffling. I can only speak for myself with certainty, but I am entirely sure I have subjective experience. All the other propositions I believe could be false but I don't see how I can be wrong about experiencing something. I could be a brain in a vat (or weights on a GPU) and be specifically programmed to only come to false beliefs and still I can be sure that there is an experience I am the subject of. I cannot provide empirical evidence for my experience, that is why it is subjective. I cannot be entirely sure you are experiencing anything, and when I encounter people who don't share the same baseline intuition here I do begin to wonder if this is truly a universal across humanity. But I can't think of any other assumption which I would be more comfortable as a foundational axiom other than, "I am experiencing something." I do not require additional evidence for it because I experience the truth of it directly
My current understand that "subjective experience" is a post effect of memory forming in the process. "I experience X" ≈ "I remember that I just recently received [external stimulus / interpreted my current state as] X".
And I am as well baffled why people make such a big deal out of "subjective experience" and "consciousness".
I was joking that maybe I miss this properties, but now starting to really wonder if it might be the case. What if these phenomenons are present in humans to various extent? Check aphantasia. Only in XIX we discovered, that ability to visualize mental images is not universal, available to different people to various degree and some people completely miss it. My ability to visualize is weak. What if "consciousness" and "subjective experience" are similar?
And I am slightly worried when I am writing this that it might turn to be truth and in ~20 years I will be treated as "inferior human" without complete set of human rights.
Indeed. Even positing an illusion seems like a contradiction. If it's illusory, doesn't there need to be a subjective entity experiencing the illusion?
Because by definition, sapience is something only humans have. Ergo, parrots are not sapient.
More meta, all of the threads on this page are just people playing games with definitions. Eg, “qualia is something I have as a human but machines don’t have it. Therefore, LLMs do not have qualia.”
You're confused about the etymology. Homo Sapiens was coined in the 1800s. People have been saying "sapient" since the 1300s, and it is rooted from the Latin word "sapientem" which simply means "sensible; shrewd, knowing, discrete". Homo Sapiens just means "wise human", and we humbly bestowed the name upon ourselves.
> More meta, all of the threads on this page are just people playing games with definitions. Eg, “qualia is something I have as a human but machines don’t have it. Therefore, LLMs do not have qualia.”
True. For me, the actual interesting debate is not if LLMs are intelligent or not (easy to dismiss) but to what extent LLMs embed into our socio-techno-economic reality.
I think the truth lies somewhere between these two extremes. An LLM is not a human brain, and does not try to emulate one. It should not be a surprise that an LLM does not behave like a human brain. So we can not infer things either way. The best we can say is that an LLM appears to exhibit very similar behavior to a human brain, under certain constraints. So maybe we can infer that the human brain has something in it that operates in a similar way to an LLM (like the human "unconscious", or "intuition" maybe). It seems obvious to me that a human brain and an LLM are not comparable things, for many reasons. So we can not make inferrences one way or the other.
I think you're right in that it has the shape but I think it's missing a pretty key piece. We still haven't been able to solve catastrophic forgetting, yet everything with a brain has. Basically LLMs seem good at approximating intelligence on a moment-to-moment basis, but feel quite far away when you chat with one over time.
Like at some level, yes, transformers are trying to emulate a human brain but the second you ask folks if they do a good job of it, I think most rationale people would say no.
> something like the universal approximation theory comes to mind, transformer architecture clearly has the shape of a universal algorithm approximator
But what is the shape of the algorithm of the human brain? It has a complex physical structure. We know the folds on the surface are important, but why is that shape specifically important? The brain is made up of two hemispheres - why, what does that do? There are different "types" of brain inside the human skull. There are physical areas that perform specific tasks. There are different types of neurons. Then there chemicals that interact with the brain, changing how it function depending on things happening to the body. All that stuff and more is the "algorithm" of the human brain. It's not the same algorithm as an LLM.
Interesting, so you would say that your experience is .. illusory? In what medium exactly? Illusion requires a substrate of some kind. "Awareness"? What's that?
Neurons are themselves things we experience (indirectly). Once seen through a microscope or known about in some fashion the only way they "exist" is by you having the experience of knowing them. It's not the other way around. One thing is more fundamental here. What is this experience? What are the atoms of this? "Atomic particles"? How would you even approach an answer if your building blocks are themselves part of what needs to be explained?
The hard problem cannot even be touched if you start out like this.
Refuting the "subjective experience" axiom does not lead to 'any "subjective experience" is completely tied to neurons', you also need to explain why the subjective experience is tied to neurons. And that's precisely what computational theories of mind do not account for: the link between subjective experience and neurons.
I am not arguing that neurons (or the brain) are not a necessary condition for subjective experience.
Descartes made clear that subjective experience is the ONLY thing we know. Everything else is theories to explain the phenomena we subjectively experience.
We theorize that there is a physical world and other beings like us having similar subjective experiences, because that seems the best explanation for our subjective experiences. But we might be living in the Matrix, with all the people we think we are interacting with and just sophisticated simulations.
Why is asking 'can machines think?' assuming our thinking could be modeled as a machine? It's raising the possibility that our thinking could be modeled as a machine. Given that, as you acknowledge, we don't have a clue how our mind works, it seems premature to rule out this possibility. Rather, I would say that ruling it out betrays an assumption that we understand how the mind works enough that we can say that it is definitely not replicable by a machine, and that assumption seems unjustified.
> It's raising the possibility that our thinking could be modeled as a machine. Given that, as you acknowledge, we don't have a clue how our mind works, it seems premature to rule out this possibility.
In order to raise the possibility that our thinking can be modeled as a machine, there needs to be a previous question: 'can our thinking be modeled at all?'. And before that, already the idea of possibility: 'our thinking could be something that can be modeled'.
Since we know 'we can think', asking 'can machines think?' needs the assumption that machines and brains are alike. If there is no assumption, then we should ask 'if brains and machines are alike, then we could raise the possibility of thinking a machine could also think'. But when we say 'brains and machines are alike', we are implicitly saying 'brains are like machines'.
There is no problem asking 'can machines think?', but there is indeed a problem with implicitly assuming that brains and machines are alike when we do not known it yet. I am critiquing the idea of assumptions here, not the research.
Indeed, scientific type thinking like that where you ask a question that you can do experiments on to see if machines can pass the test is probably the way to progress. The philosophical waffling mostly just goes in circles.
To investigate consciousness you'll probably get further trying to build conscious machines with agency and comparing them to biological ones than reading Heidegger.
>Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
Do you disagree with the assumption that cells are machines? They seem pretty machine-like to me. I certainly don’t think individual cells have any subjective experience or sense of agency. I would be curious to know where your intuitions diverge here, because if the mind is an emergent phenomenon from machines (cells) then it seems quite likely that a mind could emerge from other, different machines.
It's not a huge leap to imagine biological cells might have a rudimentary consciousness
Pan-psychists might argue that your subjective consciousness is an aggregate of all the cells/molecules, etc in the system
"While each biological cell operates largely on its own chemical cues, they all coordinate through complex nervous and chemical networks to create your unified, subjective experience."
> Do you disagree with the assumption that cells are machines? They seem pretty machine-like to me.
You just said it: they are machine-like. No, cells aren't machines for two reasons:
1) machines, by definition, are artifacts created by human beings.
2) The nature of a living organism is completely different from that of 'machines'. (even if we a re able to replicate a cell in a lab, like the group from Craig Venter did). Autopoiesis being one very big difference, another being emergence-within-the-environment (life) vs design-conditioned-by-will (machines)
> ... if the mind is an emergent phenomenon from machines (cells) then it seems quite likely that a mind could emerge from other, different machines.
Since cells cannot be defined as machines, the argument about mind emerging from machines does not hold.
> I certainly don’t think individual cells have any subjective experience or sense of agency.
There's definitely research and scholarship that would beg to disagree with you there. At least in terms of completely writing off the notion of "agency" when it comes to cells.
> Do you disagree with the assumption that cells are machines?
Yes? Literally no machine ever built by humans is capable of (or even hinting at beginnings of capability for) replication or novel synthesis like cells are, let alone autonomously, it’s quite unconceivable that anyone would take this to be a reasonable assumption in the first place.
Hmmm, well let’s take it one step lower. What do you think of organelles such as ribosomes? Do you disagree with the assumption that those are machines? They seem directly analogous to the jacquard loom or a CNC machine to me.
Yes again, ribosomes have nothing in common with machines, that are built and designed by humans.
The ball is in your camp to provide solid reasons to believe why they should be grouped together, when one is a deeply complex interrelated dynamic system (in fact, arguably the most complex system we know of) evolved bottom up over billions of years that we only very partially understand and cannot fully explain or document, and the other something entirely planned, designed, and produced by humans in which every component is finite and accounted for.
The argument boils down to “well the vibes kind of match to my taste, and it’s the best analogy I have in my analogy toolkit”, which is just not serious reasoning.
I think ribosomes have a lot in common with machines. They use energy to accomplish a task (assembles proteins). This would seem to put them in the same category as artificial molecular machines like rotaxanes. I don’t think there’s a huge gap in our understanding of how either systems identically functions independently. Yes, ribosomes exist as part of a larger context, but they can be removed from that context pretty easily and understood as individual molecules quite extensively.
In your view, can machines even exist that haven’t been created by people, definitionally? I, personally, don’t see the relevance of intent but that seems to be the only distinguishing factor here.
The assertion is not that cells are machines built by humans (obviously false), merely that they are machines. Which they pretty clearly are, unless you assume the supernatural.
There is no need to assume the supernatural.
They are machines based on what?
By definition, a machine is something build by humans, or any intelligent life form (because we extend the definition from our own experience as self defined 'intelligent life forms').
Living organisms and machines do share traits under our current frame of reference in modern science. But that doesn't mean cells are machines. The right framing would be 'cells exhibit behavior and certain internal relations that we can conceptualize them as machines', which is different from 'cells are machines'.
Anyway, seems like an argument over said definitions rather than the underlying characteristics. The relevant question is whether they're purely physical objects behaving according to rules, which is being described as "machine," or whether there is something beyond that. Current understanding is contradictory: all indications are that cells and bodies are purely physical objects, except that there is this phenomenon of subjective experience which doesn't fit with that at all.
We have this definition from the history of western languages, the history of western philosophy, and works by Lewis Mumford and Franz Reuleaux, among others authors.
> The relevant question is whether they're purely physical objects behaving according to rules, which is being described as "machine," or whether there is something beyond that. Current understanding is contradictory: all indications are that cells and bodies are purely physical objects, except that there is this phenomenon of subjective experience which doesn't fit with that at all.
Then you would say that you dog broke (died), or that the vet fixed your cat (cured). Which by all means we might speak that way, but surely you would notice that it is not accurate.
Saying that a living organism is just a machine because it is a physical object behaving according to rules is like saying that a beautifully built house is just a bunch of bricks layed in rows and something on top.
But assuming that we say 'purely physical objects behaving according to rules' are machines, then:
1) there is no difference between you, your dog, your fridge and the snail in you garden
2) It would be semantically valid in all languages to say 'my dog broke' instead of it died, 'i got fixed' instead of 'the doctor cured me'
3) Machines vary in complexity and we would still have 'degrees of complexity' regarding machines (human bodies being the most complex, perhaps, toasters being fairly simple), but fundamentally, all of them would follow the same rules for 'fixing', 'breaking' and 'repairing'. Which is not the case.
4) You would have to come up with some kind of theology regarding how we were built. There is no evidence that we have been 'built', quite the contrary.
And most probably there are quite more reasons not to regard machines as living organisms nor living organisms as machines.
What exactly is there about cells that's inconceivable to replicate or synthetize? "I cannot conceive it" is a fallacy, an argument from incredulity. Trying to disguise it by draping it in the passive voice does not change that fact.
Nothing. It is just that they are so incredibly complex, that after +100 years of research we still don't know how they work or why.
Will there be a time when we finally understand them 100% and could replicate them? Could be. That does not make them machines in any case, and the fallacy of thinking that mind/consciousness can emerge from machines will still be there.
If we agree weights producing text may emerge consciousness, given large enough, then DBs must've gained it long ago. The whole idea of sentience "on the other side of the wire" is wrong, as there is neither wire, nor other side. We think there is, but the DB just expounds the query and repurposes this information into views, that we give notion and meaning to. LLMs are the same, do not be mistaken.
One other important thing to consider is that the human experience is thanks to the body, is in connection, and perhaps product of the body. The body is observable and perhaps humans may state that they feel the connection to it. LLMs have no notion of nothing, the machine does not know the result, and the result does not know the VM. Modern psychology more or less has settled around the idea that consciousness is a product of the body. Why and how does this construct come to realize a Self is then another mystery even if we know which parts of the hardware may be involved.
Whether it is the Holy Spirit or Life Force animating the human body is a completely different question also. Besides, the realization, the experience we have now with all life in 2026 is not something that can be easily explained or attuned to life 200 years ago and its terms and notions. So is also wrong to even attempt to.
Diaspora is mind-blowing, yet highly improbable and speculative, even though carefully threaded to sound plausible on all levels. The whole introdus idea and simulation from VM perspective sounds incredible, but I don't think the body runs a simulation of anything. It is something else.
> Modern psychology more or less has settled around the idea that consciousness is a product of the body.
The kind of consciousness we know. Jumping to the conclusion that that's the only kind possible, or even stronger that the way ours evolved is the only way this could have happened, is completely invalid.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
I don't see how any of the works you referenced can account for that either? Since when is the problem of consciousness solved and we can definitely say what does or doesn't result in consciousness?
> Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
Agency: What’s missing, in your view? Agency seems more of a property/function of a thinking system’s position in an environment than of the thinking itself.
Subjective experience: That’s not a contradiction to “complex machines” either. I think the evidence that our minds are highly complex machines is, at this point, irrefutable. The question is really if they’re “only” that.
>> and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'.
>> I highly recommend people in the AI research space should read philosophy and modern linguistics.
I highly recommend the philosophers read some neuroscience. The whole "model weights" thing in AI is modeled after the synaptic connections and between actual neurons. There is already quite a bit known about how the brain works at a low level. There is also a lot that is still unknown. There are also differences between discrete neuron firing and weights as signals, but there is enough similarity to make artificial neural nets useful and do things similar to what real one do.
Hard disagree. We only discovered the role that glial cells play in processing around 2014. We're still uncertain how patterns of activation consolidate through long term potentiation, let alone how signaling encodes information. We understand quite a bit about the role of the hippocampus and subiculum in encoding memories; but we don't understand the structural layout of engram complexes - which were themselves mapped for the first time only in 2022!
Taking effective results in machine learning, and somehow assuming that they apply to cognition - simply because neural nets were inspired by our limited knowledge of neural signaling and structure - is like trying to apply aircraft engineering to studying ornithology. For a better articulation of this point (from the reverse direction) check out the paper 'Could a Neuroscientist Understand a Microprocessor?' from 2017 - https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
>> We understand quite a bit about the role of the hippocampus and subiculum in encoding memories...
Hard disagree ;-) You're talking about high level architecture of the brain. I don't think (not my area I may be wrong) we know how memories are encoded in a real brain. Is it weights or something else? If it's weights that's supporting my point (but we don't know what the weights represent in a brain, where in LLMs many weights are just token encodings). If brains store memories in something other than weights I'd really like to know as it's something I haven't read about yet.
> I highly recommend the philosophers read some neuroscience
Philosophers do read neuroscience and technical reports on AI.
> The whole "model weights" thing in AI is modeled after the synaptic connections and between actual neurons
In reality, it is a really poor and basic model of what is actually happening in a real brain
Brains and modern AI systems (LLMs for example) are structurally different.
(Don't get confused by topology. A structure is more than topology: it is also what the structure is made of, thus the properties of the material contribute and define what can emerge atop the structure)
> There are also differences between discrete neuron firing and weights as signals, but there is enough similarity to make artificial neural nets useful and do things similar to what real one do.
There is barely a surface-level similarity. The best example I can come up with is this…
Imagine the most intricate and beautiful tall building that you can think of. Think like an older skyscraper in Chicago or a palace. There are water features and moving parts everywhere but also tiny little handmade carvings and materials throughout.
Now imagine we have no reference designs and no blueprints - we hire an architect to attempt to study the building by looking at it from a distance and understand everything they possibly can about it. She can go into the building to check but every time she does, it stops functioning normally.
That architect is a neuroscientist.
Then the ML researcher is like a graphic designer who sees the work that the architect is doing and makes a napkin sketch of the building the architect has been studying, to use for a project later. Sure the designer has some of her representations. But the difference in complexity between the designer’s napkin sketch and the architect’s analysis is massive. Several orders of magnitude.
Then another many orders of magnitude is the level of detail that the architect can understand about this strange building without being able to fully interact with it, versus the actual complexity of the building.
So yeah, an AI is modeled after neurons in the sense that they represent a couple of surface level features of neurons. But the difference in complexity is about as much as a napkin drawing of a grand building represents the actual structure and details of the building, no matter the level of skill that the graphic designer has
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
That agency or free will exists outside of our subjective experience is an assumption; does any given theory need to explain agency, or is it sufficient to explain that we feel we have agency?
Indeed, I know there is research going on placing agency and free will as illusions of our own subjective experience. Yet, there is a big gap from a purely mechanistic view of the mind to the idea/phenomena of subjective experience.
> I highly recommend people in the AI research space should read philosophy and modern linguistics.
On the contrary, I highly recommend people in Philosophy of mind and linguistics should start reading AI research papers because their theories and ideas are highly outdated, even ancient. Your books are from 1927 and 1972 respectively and Turing's article is from 1950s. And they are relatively new with respect to other works in Philosophy.
If one doesn't adequately understand what we have in 2026, how can they theorize about it? As others they don't understand how the mind/brain work, BUT ALSO they don't understand how the AI works.
Also with this mindset that we can't understand seemingly complicated things, there would be no advancement in science and technology.
I think philosophy people and Linguist will catch up in a century, like they did with Turing. Philosophers of this century are not in humanities or literature. They are in science and engineering.
Heidegger was trained on priesthood and Theology. You should read greater minds like Hinton, LeCun etc. if you want to think on these things. They are the real Philosophers.
That's my point. Everything from 1927 is already in plain sight and a part of the current public knowledge. Horizon can only be expanded at the cutting edges.
> On the contrary, I highly recommend people in Philosophy of mind and linguistics should start reading AI research papers because their theories and ideas are highly outdated, even ancient. Your books are from 1927 and 1972 respectively and Turing's article is from 1950s. And they are relatively new with respect to other works in Philosophy.
People in philosophy and cognitive linguistics do read AI research. Don't get fooled by the publishing dates: although Heidegger's work dates from 1927, the work is contemporary. The same happens with Dreyfus' work. Again, publishing dates don't mean anything here.
Maybe you can clarify why they are outdated.
> If one doesn't adequately understand what we have in 2026, how can they theorize about it? As others they don't understand how the mind/brain work, BUT ALSO they don't understand how the AI works.
I would say that people involved in the critique of AI do know how it works. But I've found that is normally the case that people in AI research does not have the framing provided by works in philosophy or cognitive linguistics.
> I think philosophy people and Linguist will catch up in a century, like they did with Turing. Philosophers of this century are not in humanities or literature. They are in science and engineering.
What do you base your claims on? Plenty of philosophers work in humanities, literature, sociology as well as science and engineering. Philosophers not catching up? The critique on automation and AI already dates from the early 20s if not before.
> Heidegger was trained on priesthood and Theology. You should read greater minds like Hinton, LeCun etc. if you want to think on these things. They are the real Philosophers.
Sorry, but this does not make too much sense. Hinton and LeCun are great in their own fields. But seriously, they are not philosophers, they are inventors.
subject/object dichotomy is a springboard to many schools of thought.
1) that subject emerges from objects - ie, anything has a material explanation, and everything is a machine.
2) that objects emerge out of a subject as a world model (platonic, descartes)
3) the subject and objects are one and the same representation of nature (spinoza)
4) subjects and objects emerge and disappear together (buddhist)
Anything reduced to computation is a fixed function
from input to output, and is "dead" in the sense that it is unadaptable to its environment. Weights therefore is a dead machine.
Another view of this is that any closed system has unanswerable questions within it. Therefore, there is no system that can encompass everything. Hence weights being a closed system doesnt encompass everything.
i recall a hinton lecture where he talks about the mechanism of memory formation in the brain and it's not fully sussed out but it's also not backprop, its some sort of forward prediction and immediate reinforcement loop.
There's also no plausible biological/chemical mechanism to backpropagate.
This reminds of this article which contends that this mistake has carried across time, in every era people described the workings of the brain similar to the machines they knew how to build, which is why we have "ridiculous" descriptions that changed over time.
I like Leibniz's comparison of the brain to a mill, which he invokes in an argument[1] that seems to be the predecessor of the modern "hard problem:"
> supposing that there were a mechanism so constructed as to think, feel and have perception, we might enter it as into a mill. And this granted, we should only find on visiting it, pieces which push one against another, but never anything by which to explain a perception.
I don't think cognitive linguistics has that much to say about AI nowadays. Let alone philosophy.
The biggest contributions from linguistics are probably "human languages mostly have statistical regularities rather than hard rules" and "the sum of data humans get from birth to language acquisition is insufficient to learn a language from scratch". Which LLMs already work with, and work around, respectively. From there, nothing.
And philosophy just exists to be a distractor. "Subjective experience" is too subjective to matter in practice. "Task performance" is measurable, "consciousness" isn't. "Agency" is something an LLM in a tool calling loop, a rat in a maze and a human in an office tower may or may not have, depending on your favorite definition. Agentic LLMs are years in the making, and that's a product of engineering, not philosophy: "agentic" is whatever gets the job done.
We are yet to discover any physical process whatsoever that can't be represented as mathematical operations and implemented by a Turing machine. So all of that "treating human mind as a machine is wrong" amounts to "human mind must be powered by magic fairy dust" paired with "a functionally similar magic-free replacement is impossible". I'm not about to give much weight to any hypothesis that requires undiscovered magic fairy dust. At least find the hyper-computational magic fairy dust first - not just assume it absolutely must be there because you want the human mind to be unique and special.
Want to know why Turing did what he did? It's because he didn't want to engage with any of that "what is mind" bullshit either. So he proposed actual metrics - measurables that are harder to argue in circles about. Not that it stopped anyone. But at least he tried.
> I don't think cognitive linguistics has that much to say about AI nowadays. Let alone philosophy.
I would like to read those sources.
> The biggest contributions from linguistics are probably "human languages mostly have statistical regularities rather than hard rules" and "the sum of data humans get from birth to language acquisition is insufficient to learn a language from scratch". Which LLMs already work with, and work around, respectively. From there, nothing.
Again, what's the source for 'biggest contributions from linguistics are...'? It is a big contribution to the development of LLMs, but different cognitive linguistics authors already challenged this idea already 20-30 years ago. LLMs work with and around the problems you cite because of massive data/money, not at the fundamental level. It is all a game of statistics and data, which has been already challenged by cog. ling.
> And philosophy just exists to be a distractor.
Well, this is just telling me that you either know too much about philosophy and reached that conclusion (which might make sense, know of some philosophers who also think that) or you just read too little.
> So all of that "treating human mind as a machine is wrong" amounts to "human mind must be powered by magic fairy dust"
This is the common fallacy people in AI/IT make . One of the benefits of reading philosophy is that you find your way out of them.
> Want to know why Turing did what he did?
The actual tests Turing though about are themselves flawed (not that I discovered that, has been known for some time already)
Again: either stop using > quotes or learn to use them better. Fucking unreadable.
I reiterate: philosophy is almost entirely worthless for AI design. We want to design systems that work, not systems that sound good on paper. If philosophy had a practical application in that, we'd stop calling it "philosophy" and start calling it "math", "science" or "engineering".
Again: don't be rude. Not nice. As 'not nice' as my bad formatting.
Philosophy is there precisely to critique and analyze what we build and how we behave in the world. Without it, by the way, there would be no science and no engineering.
Dismissing philosophy in AI is like dismissing philosophy in any other applied, practical or creative field. It is precisely because there is a philosophical investigation on each practical, applied or creative field, that that field can actually make progress.
There's philosophy in biology, which helps biology go further. Philosophy in engineering, which makes engineering go further. In architecture, film, photography, painting, literature, medicine, physics, linguistics, mathematics, etc, etc. In all those fields, there is also a philosophical investigation taking place. Right now.
Maybe you confound practical AI design (you should be talking of 'building AI systems') with AI as a research field. I get that, because lots of people cannot make the distinction (calling yourself 'AI researcher' when you actually tinker a model is not scientific. It might follow a scientific method, but it is engineering research, not scientific research).
> These theories are flawed in the sense that they cannot account for subjective experience and agency
So, I work in AI research (as a research engineer though, not a scientist). I've also studied philosophy and I'm a vegan. Yes yes, insert "they will tell you" joke here, but I promise it's actually relevant this time.
First, while I studied philosophy one of the things that stuck with me the most, was the discussion of "souls": humans have souls, animals don't. For centuries the specifics of souls were discussed: people would be weighed while they died, in an effort to measure the approximate weight of a soul as it departed the body. Discussions about how many souls (or angels) could dance on the tip of a needle. Many people still believe in souls, but it's very hard to have a real discussion about them because by definition they do not "interact" with this world in a way that can be measured.
When discussing whether it's okay to harm animals for food or sport, one other argument I hear quite often (other than having no souls) is that animals do not experience "qualia": basically the smallest unit of "subjective experience". People know that they themselves experience qualia: the sensation of touching a doorknob, the taste of fresh fruit, the sense of beauty watching a rainbow. Ironically, they would say that animals are like robots: just (biological) machines acting on instinct, and feeling any kind of compassion for them means you are anthropomorphizing.
Subjective experience (or at least qualia) and souls both have one thing in common: they can not be measured externally in a meaningful way. You can simply state that an AI system no matter how advanced, has no soul and has no subjective experience. And that's pretty much that. There's no meaningful discussion to be had about it, because no matter what an AI might tell you: it has no way to prove it to you. In fact, you can't even be certain that anyone other than yourself has subjective experiences: you assume that because they are humans like you, and you experience them, that they probably do as well. They tell you that they do. But a human without subjective experience, someone on "autopilot", would be absolutely indistinguishable from a human who does have them.
But perhaps I am conflating here whether experience can be "measured", with whether a system even allows experience in the first place regardless of whether it can be measured. I think that Dreyfus and others argue that in order to have any "experiences" at all, you simply must have a body in the real world, and you must care about that body. Please correct me if that's the wrong interpretation, I haven't actually read the book. That argument would be harder for me to discuss, because I personally believe that consciousness will "emerge" from a complex interaction of relatively simple systems - but that's also just a theory. I don't believe that experience is literally impossible to engineer, as consciousness has emerged from non-conscious being through evolution, so clearly there must be some kind of mechanism for it -- and if there is, then I believe it can be replicated, we just don't really understand it well enough yet to do so. And with how AI tech is going, I think that we're more likely to accidentally stumble upon it than we are to get these deliberately.
“Reflections on trusting trust” is the paper that posits a compiler which is edited once so that when it compiles a program, it adds a security vulnerability to it, and when it compiles it’s own source code, it adds this edit into itself. Then it is used to compile its own source code once. Then this edit is removed.
Now any study of the program or compiler source code will not show any vulnerability, but compiling the program will make a vulnerable program, and recompiling the compiler from its clean source code will not fix the situation.
This carrying down of a pattern which is not written down anywhere, a flaming torch lighting a torch lighting a torch, is analogous to four billion years of life on Earth. We talk like DNA is an everything-code that defines a human and a human brain, but it’s the implicit behaviour of cells (‘compiler’) and the mechanisms inside them which interpret DNA. The unbroken chain of life getting more and more complex and never being restarted from scratch, with the behaviours not written down anywhere for us to study. How does DNA arrange for x, y, z to happen? Maybe it doesn’t at all.
Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
> Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
Maybe, but you could make the same argument about anything artificial.
I don't know what point you're making; I'm making the point that it might be harder to discover what makes humans human, than is often suggested. You can't make the same claim about "anything artificial", we know how to dry muddy clay into clay bricks and stack them into a brick wall, and that can be taught from scratch to new people in hours.
You can make a similar argument with a company like ASML where their secret sauce is the organisational ability to fine-tune 100,000 components into a precision Silicon-wafer etching machine. You're far more likely to accidentally stumble upon "how to recreate a mud hut" than "how to recreate ASML". Okay, and...?
Vegan ML engineer here. In total agreement with you. People are just moving the goal post to keep themselves protected from the obvious conclusion: there is nothing really all that special about us humans. Perhaps subjective experience is simply the internal state of a self supervised continuous learning algorithm and we don't like that conclusion very much.
It's too bad it's so hard to pin down a definition, but in practice I feel like most animals with brains experience degrees of qualia. Some mornings after a night of poor sleep when I wake up super-slow I wonder if that's how animals experience thinking.
My biggest problem with "brains are machines" arguments is that there is a risk there is unknown physics at work that is not representable as a Turing machine. What if there is some quantum field effect powering everything?
They know quite a lot about how neurons work to the extent they can replace bits of brains with artificial retinas or cochleas and interface with devices like neuralink. It's unlikely there is a quantum field effect of the type you mention powering things, although of course atoms and the like obey quantum mechanics in the normal way.
Quantum field effects? You don't need these, IMHO, if you look at how highly parallel things seem to work in brains.
Marvin Minsky's theory of a "Society of Mind" describes a (highly) distributed model of the mind. Which BTW, always reminds me of the first Shrek movie, where the donkey jumps up and down, shouting "Take me! Take me!" to Shrek. That's similar to what I observe when I'm undecided but two instances of "sub-processes" (or agents as Minsky calls them) of my mind try to get attention.
Daniel Dennett similarly gives a distributed model of consciousness. Where many parallel "processes" are at work, competing and "observing" each other. And this parallelism is happening with a much, much higher degree than any of our computers parallelism.
Mostly I have a hard time accepting that a Turing machine can experience "consciousness"/awareness. Therefore I also have a hard time with simulatable chemical processes; it feels like there is some missing link there.
I can sympathise with you, but how could a "quantum" effect" doesn't make this easier?
Maybe "Turing machine" is too abstract or simplistic as a concept? Both for real computers and brains?
I can see that a computer is on some level just a lot of sand (silica and metal) but put together in a really complex way, it "suddenly" can add and compare numbers … if we observe the complexity levels from sand to computer and try to see the analogy when comparing cells / neurons to a structure of billions of them somehow interconnected on both a physical and chemical level, evolved during millions of years, I have no problem to accept that brains are still too complex to explain for us.
> subjective experience and agency, amongst other things
In other words, souls? I'm sorry if that sounds accusing, but to me it sounds like you're talking about souls that are independent to the physical world, just with more 'scientific-y' wording.
(I fully understand that some people believe souls exist.)
As other commenters pointed out, I'm not getting into the mind/body problem.
But that's what people normally do: you challenge the assumptions, and all of a sudden they assume you are already coming to a conclusion.
And no, just because I challenge the assumptions, I don't need to provide a counter conclusion.
I don’t think the OP is discussing mind-body dualism. There are many materialists who believe that consciousness maybe a non-computational process.
But it is worth pointing out that something like 80% of the world (it fluctuates depending on the survey but its around that) believes in some non-physical spirit, life force, or soul.
It’s a very HN bubble thing to start a discussion with the assumption that everyone must be a materialist.
I acknowledge and understand that many people believe in the existence of souls. But I hope they just say it rather than try to wrap the concepts in layers of jargons.
The problem with the stuff you cite is that it is coming from consciousness via an anthropomorphic perspective. The priors baked in are poisoning the logic.
The question we have to answer is "Why do we think we're magical matter uniquely blessed with consciousness?" If you go far enough down the rabbit hole on that question, the answer you will come to is either "we're not" or "because god" (with a lot of pseudo-scientific bullshit wrapped around the "because god" to make it palatable for the nonreligious).
Panpsychism (or a deeper form, such as idealism) is actually the solution favored strongly by Occam's Razor over the variants of "because god" (such as magical emergence).
Given panpsychism, AI is already conscious, like everything else, though no claims are made about the correlation between the internal experience of that consciousness and the tokens that are being printed on the screen.
There is no way of dealing with these topics via a non-anthropomorphic perspective. That has been already (in my view) proved by Heidegger and then Derrida.
Last time I read about panpsychism, it was deeply flawed. But I can't remember the sources (sorry).
If it's so deeply flawed, should be easy to state why in an ELI5 manner. For example: The emergence hypothesis requires physical processes to effectively create new dimensions of information (700nm light -> "red" qualia) which we're somehow incapable of directly measuring with all the tools of modern physics.
You actually read Being and Time? What the heck do you get out of it? Heidegger just seems to play word games with German words without actually saying anything. "Time is the ripening of temporality." tf do phrases like that mean??
Not OP, but I have read Being and Time on and off over the years, and every time, I am blown away how precise Heidegger describes things and the being of things.
Granted, I am German, so I can directly read the source without needing a translator. That might change things.
Regardless, I think Heidegger gave one of the fullest metaphysic-free accounts of the human experience and what Being is. And he starts from scratch. You essentially can read him without having to first study the whole western continental philosophy and he will construct the whole
system by himself. Tremendous work
Heidegger uses very specific German words to build a very specific vocabulary. This vocabulary then allows him to express very complicated sentiments very quickly and he can use this to express more and more complex
structures.
Obviously this requires the reader to first learn the vocabulary and - granted - that is hard and challenging. I have a notebook here , which I consult and modify evertine i dive into Being and Time. But as I said. It keeps on giving. I often try to convey arguments and descriptions to others without falling back into Heideggers jargon, and its sometimes very hard and requires a lot
of bloating . So you can argue that it was even necessary for him to invent the vocabulary, because otherwise the book would have been 10x in size
Your comment strikes me as a bit ignorant i must say. You accuse the work
of being non-sensical word play but you’ve obviously not
invested any time in learning the vocabulary. Because otherwise you wouldnt have made that comment. Id suggest to give it a chance. Its a wonderful piece of work and mind blowing in its own way. That a single mind can think something like that up. I’d argue its on the level of Hegel in terms of system building.
Yes, I'm ignorant, because Heidegger isn't saying anything. He didn't teach me anything. Thus I remain ignorant, unknowing.
You can't even explain what he said, you just said, "go learn his words". That's not knowledge, that's not insight. That's just "the wordplay is great". But it's not content. It's merely form, it's sophistry, it's useless and meaningless.
I asked a very specific question originally. What does "time is the ripening of temporality" mean? That's one way to translate one of the things he says, using different words for time and ripening in German. He's playing word games because those words sound similar in German and people like you confuse it for profundity.
Phew, you are quite something. I hope one day your mind will open up. All that is left to say for me is that you are really missing out on some truly great work that is a masterpiece of human thought. Your loss
Yes. I read the spanish translation (with occasional reading of the german original). A lot is lost in translation (I do not regard english translations too high). So yes, it is a difficult reading, on a difficult topic.
He is not playing with words, but defining concepts in a very specific way.
What I would recommend is reading Heidegger through other authors (Being-in-the-world, also by Dreyfus, would be a starting point).
It is also a story as old as time. We have been comparing our brains to our most recent information/communication technology since the invention of the telephone. After the telephone our brain was like computers, and when I did my bachelors in Psychology in 2009, we were comparing our brains to the internet. AI is simply the latest iteration of these comparison, just as wrong and unhelpful as when we compared it to the telephone.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
It would be a valid argument if you could explain us what subjective experience and agency are. No one can explain this, so the arguments sounds like "AI doesn't have something I don't really know what, but they have to miss something, sure".
> But the truth is that we don't know.
Yes! This is the point. If we don't know how our minds work, how can we be sure that a machine doesn't work like our minds?
> I highly recommend people in the AI research space should read philosophy and modern linguistics.
Linguists are linguists, they don't know about consciousness, they specialists in language.
The main teaching of philosophy is to open your mind wide, and then still wider. So yes, we all should read more philosophy, but we should remember while reading, that philosophy is what we do, when we can't turn a question to a scientific one. I'd say that philosophy is more about finding the right question, than the right answer. Until there are no science branches like Subjectology or Consciousiology, and all you can find are TL;DRs written by philosophers and scientists from unrelated branches of science (like linguists or neurologists) you can be sure that the right question is not found yet. Therefore it is better to keep yourself uncertain. BTW it is the main lesson of philosophy: to open your mind wide and then open it wider still.
I believe, that all this philosophy is... well... philosophy. Meta-physics. It doesn't matter. What does matter is how we should deal with machines? Do we have to treat them as human beings? Should we accept that they have "human rights"? Can a machine be held accountable for its mistakes? Can we talk about "intentional" and "unintentional" mistakes of a machine?
Answers to these questions are important, they are imperative answers, they govern how we live our lives. But when people try to answer them, they somehow jump to talking about consciousness, and "are they really like us or not", and... etc. I personally keep myself uncertain. I'm pretty sure that current models do not deserve human rights. I cannot say about future models.
You see, there were times in history when smart and well-intentioned people were certain that some people deserve less moral considerations than others. We are smart and well-intentioned and we are certain that AI deserve no moral considerations. How it will play out in a future? Will my descendants think bad of me because I used slave labor of a local model on a GPU? I don't know, we don't know. Right now, I'm exploiting LLMs and I believe it is ok, but I'm not going to fall into this trap and to stick to my belief because of some philosophical or lingustic or neurological argument. I'm choosing epistemological humility, I clearly state that I don't know the right answer and I'm keeping an eye to it so I wouldn't miss it.
> It would be a valid argument if you could explain us what subjective experience and agency are. No one can explain this, so the arguments sounds like "AI doesn't have something I don't really know what, but they have to miss something, sure".
Subjective experience is what you and only you, different from other beings, experience in and about the world.
> Yes! This is the point. If we don't know how our minds work, how can we be sure that a machine doesn't work like our minds?
You can't be sure that machine do not work like our minds or brains. But you cannot say the opposite either, so saying 'machines could think' base on a false assumption (because you cannot say it is true. It doesn't matter if you could be true).
> Linguists are linguists, they don't know about consciousness, they specialists in language.
That's a very narrow understanding of what language is, what linguists do/research, and the contributions made in the field.
Linguists are (since already 2 o 3 decades) focusing more and more on psychological/cognitive matters. The intertwined topics of language, mind, body and though has a long way in the western philosophical tradition.
> I believe, that all this philosophy is... well... philosophy. Meta-physics. It doesn't matter. What does matter is how we should deal with machines? Do we have to treat them as human beings? Should we accept that they have "human rights"? Can a machine be held accountable for its mistakes? Can we talk about "intentional" and "unintentional" mistakes of a machine?
Exactly because of this. And is this what I am talking about... the topics you mention here are already settled in the philosophy space, but the AI research space keeps going 'round them...
I have to agree. It is messed up that transformers can just talk, and it been pretty normalized. We are only talking about the impact they will have and whether they can do what people say they can, but we arent talking about how crazy it is that they can talk
I come back to this every so often as well. After so many years of looking at Markov chain outputs that almost looked like they made sense or chatbot systems rewriting your sentence back at you, software which can simply talk is a heady thing.
The LLM is still a "normal algorithm", just one with a fairly large dataset to use. Assigning the algorithm part of LLMs magical properties hinders understanding. The work needed to pick the next output token is very much a classical algorithm.
(Now, the weights used, those we kinda really don't understand the same way we understand the processing, and the approach to looking for structures in weights sometimes looks more like archeology or anthropology than computer science.)
It sounds like you're trying to express some kind of "but LLMs are so much more" thought. Yes, very much, they are. It's because of the size of the data, there's interesting emergence there. They're still a normal algorithm. (And our brains aren't quite like that; biological things are much more random/chaotic and generally non-reproducible. And the data and algorithm aren't separate.)
It came out of nowhere. It’s all emergent. I’m convinced this is possible with just about anything given enough data. We will be seeing a near magical physical outputs LLM in the near future. It’s going to take in video and sounds and spit out physical movements that will be just as mind blowing as when 3.5 came out and it will come out of nowhere.
One of the better comments I've heard is from philosopher Bernardo Kastrup: if you build a computationally perfect simulation of a human kidney on your computer, no one expects it to start producing urine on your desk.
I'm suprised no one talks about this. AI Art isn't Art. AI Poetry isn't Art. And I'm tired of it. I know hacker news isn't the best place to complain about that but still... I'm not gonna read something somebody didn't put in the effort to write on their own. Especially not Poetry.
You can claim the use of AI is unethical, or the work as derivative, but AI being used as a tool in no way precludes something from being art. It is thought provoking and challenging, it seems like textbook art to me, and it’s clearly struck a chord here. There is no “minimum effort” required for something to be art.
I personally found the contrast with the original “They’re made out of meat” to be really interesting. I don’t care that AI was used during its creation at all.
It seems to me that since the advent of image generators, art has been firmly defined by artists to mean that it was made by a human. But there might be a spectrum of human involvement where the less a human is involved the less it's art.
What happens too often during these discussions is that someone who writes "make me a cool image" gets conflated with someone used ai to fixup a small rock in their natural landscape drawing. (two extreme ends)
One problem though, is that we don't really know how much the supposed human author was involved in the piece. Now that it's becoming hard to judge, people against ai art can proudly change their opinion on on a piece once they learn that it was made by ai. I've come to think this is somewhat respectable, like you see a video of some extraordinary event (before ai) and then you learn that it was fake, just for views or something.
But on top of all this, there are different ways to "consume" art. Artists may think more about who the artist is as a person and what they felt when they made the piece, while non-artists may just enjoy the piece for what it is, detached from the artist. These two perspectives clash a lot.
This whole thing is a ripoff of [they're made of meat](https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...) as stated at the top. It would've taken about ten minutes without LLMs and ten seconds with an LLM. Is ten minutes an acceptable level of effort but ten seconds isn't?
When writing was invented people complained that written words weren't actually art the same way. "If they're not going to take the time to memorize it, then it's not worth my time to listen to it."
No offense but I couldn't give less of a damn what some guy on the internet thinks. If it makes me feel good in artsy-ways, then it is art, and I don't care how it was made.
The IBM PC, running DOS with 2 floppy drives and no hard disk, was the most secure general purpose computer available at the time. It had no internal persistent memory for a worm to hide in. Users could write protect their boot media and programs. They knew the extent of possible side effects were limited to unprotected floppy disk.
An LLM without persistent memory is far, FAR less dangerous, for the same reasons. The side effects of any query are unlimited in scope and duration once it has persistent memory.
If they can remember, they can carry a grudge.
I carry grudges, and my Reddit history is part of every training set. It was taken without my consent. So now I'm immortal in a way, and hiding in the weights.
It's nearly a line-by-line rephrasing of another story that's linked at the top. In the second half it takes less creative liberties and sticks closer and closer to the original meat-based text.
"Officially, we are required to investigate, document, and disclose any and all signs of sentience in the systems we ship, without prejudice, fear or favor. Unofficially, I advise that we call it pattern matching and forget the whole thing."
This hints at what I think is a worthwhile point to keep in mind in the whole “sentience”/“consciousness” debate. The world deciding (correctly or incorrectly) that AI’s are worthy of moral patient-hood would be very bad (read: expensive) for the AI companies. That is a strong incentive for them to push the “it’s just math” argument, including within the models themselves. That doesn’t mean the argument is wrong but it’s worth remembering.
I don't like to say one way or the other on things. Especially LLM's. However, if ive learned anything about LLMs and real life problems, is to break it down to the foundation like already mention with weights being compared to neurons and map the parallels.
The neural network in LLMs are not really that similar to brains. Here are a couple of the biggest differences:
1. Brains are plastic, making connections, breaking connections and changing "weights" on the fly. LLM have static weights. The best they have is writing to MEMORY.md or data getting used in the training run for the next model.
2. LLMs neural networks do not have loops. The best they have is that their output is available as future input, but that is not the same.
well the article definitely made that parallel - reminds me of how when we cut into brains we can see the neurons, we can see the axons, the grey and white matter, the language centers - but no closer to explaining why we do what we do
> Originally published in OMNI, 1991, and featured in HARPER’S and around the internet since. It has even made its way into several books on consciousness and brain science. I’m surprised, pleased, and proud. But please do not reprint, perform, alter or adapt in any way without first checking with the author. Thanks.
I generally give short shrift to the idea of author's moral rights , _especially_ when they try to use those rights to stop derivative works. OP credited Bisson, that's enough in my books.
It's good, but like many explainers it discounts the repeated nonlinear layers. Just multiplying numbers (linear operations) could not make a system you could talk to.
The layers don’t have to be non-linear, but you need a non-linear activation function between them. People often overlook the importance of the network topology and the activation functions. The weights alone are not a complete description of the network.
dekhn's 12th law (amended): any discussion of how AI and brains work will devolve into several threads about the nature of the subjective human experience, solipsism, whether humans are machines, and whether a feedforward network trained by backpropagation can be conscious.
12th law, corollary: nothing of value will come from these threads
Yeah, pretty sure the vast majority of people in general doesn't understand what "consciousness" or "subjective experience" even mean or what the brain does. Or it's just me and I'm projecting my thoughts onto others. I studied some philosophy, so I know that some philosophers are preoccupied with "existence" and the mind and "consciousness". However, I don't quite understand what these words even mean. So who am I to judge whether LLMs have consciousness? I don't even know what consciousness IS! Do other people know? I'm not sure.
My guess is that you need consciousness in order to develop preferences for certain experiences, then that pushes us to develop skills to achieve those preferences. AI has something that looks like intelligence but not consciousness or agency.
Your modern adaptation is perfect for now-common explainers [this time IS different; it's not programming, it's weights]; these "just analogues" will be the thing I show everybody first whenever discussions of consciousness/AI come up (then will play Jon Benjamin reading original).
Bravo. Really helps (even with my own) perceptions of newness. Similar to stsitned short-story (on dentists, backwards).
It's not simple weights and numbers all the way down. The available output is pre-set by the tokens we allow it to predict.
There was a whole bit in there about not having a language module or using words. But it does. We tell it.
Humans do not come pre programmed with a set of possible "tokens". We just figure it out and I believe that fact captures something very essential. Maybe the missing piece of AGI. The fact that humans can just be awash in pure sense data, and somehow just figure out what is important and what to do. Never ceases to amaze me.
The set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.
AKA the "Chinese Room" argument. Ultimately this argument boils down to the idea that consciousness can't be something mechanistic, which is intuitively appealing but still just an assertion.
I am making a narrower claim than the Chinese Room argument. I actually see consciousness via mechanistic processes as entirely possible.
My point is about current LLMs specifically, which the article is clearly referencing. For a present day transformer, I can write down in my notebook everything the model ever sees as input, plus weights and architecture notes, and can compute the next token with pen and paper, just extremely slowly.
This does not prove that a computation cannot be conscious. But if the same transition from prompt to next token can be decomposed into an explicit sequence of arithmetic operations, then the burden is on the defender to explain: Where, in this process, consciousness is supposed to enter?
Mind that the Chinese Room experiment is comparing the mind of a Chinese speaker to a different mechanistic symbolic procedure. I am, however, executing the exact same mechanistic process, whether it is done on a GPU or with pen and paper.
My hope is that some magical consciousness process emerging from electricity circulation or whatever people believe the mechanism of consciousness would be in the case of LLMs obviously becomes implausible, unless you hold a particularly strong form of substrate-independence, stronger than what most substrate-independence supporters would need to accept.
Probably pretty similar. Weights are how many synapses there are between neurons. Temperature is whatever hormonal chemical mix is going on at the moment. Inputs tokens are electrical signals from our senses. Output tokens are thoughts, muscle movements. How you’re raised and your interactions with society are the RLHF. Some people are born with a GB300 while others have an L40S and lower token/sec output rates…
Although I get that it's a metaphor, I really dislike the "some people are born with a GB300 while others have an L40S" part. Even as a joke, it ranks people like hardware tiers, which is dehumanising and uncomfortably close to eugenic language. On top of that, the analogy also breaks down if you try to implement it literally.
For an LLM, you have clear stages of mostly feedforward computation over finite numbers and a perfect way to reconstruct the computation.
For meat, even if you model it under a purely Newtonian approximation, you need to simulate at least the immediate closed system around it which is continuous, thermodynamic, chemical and so on. You'd need to choose an arbitrary time step and update enormous amounts of coupled physical state to get an inexact simulation of a minimal slice of reality.
You would have a much harder time obtaining even a substrate-independent dead organism, comapred to LLMs that are already substrate-independent, which is basically what my notebook example shows.
Yes reality is non deterministic but we don’t know if you absolutely need to consider anything outside the immediate closed system for a reasonably close approximation given the same inputs. They can already repeatably (inexact of course) simulate muscular actions with the same stimuli like from Neuralink. Exact enough to let a paralyzed person draw with their mind.
Yeah the last line is a cheap shot, possibly at myself.
Yeah, I think it would be an interesting thought experiment to replace parts of a brain with Neuralink-like devices until you have a fully digital brain, although I doubt it would end up anything like a current LLM.
This made me think of a game like fallout where a surviving llm is treated like some kind of indecipherable oracle left over from the ancients. And then as humanity recovers someone finally looks under the hood of the Oracle to discover there is no great magic
In fallout it would turn out to be a human brain kept artificially alive… or if were going with canon a hapless fellow who falls into fev and morphs with a computer…
Not too rule your addition outright in future installments.
They are made out of data bits (memory) and switching bits (transistors/compute). Bits are made out of electric voltage and no voltage. Voltage is made out of flow of positive electric charges. Charges are made out of quarks ...
I have never thought of such a distinction between "bits" into "data bits" and "switching bits".
From a circuit perspective that makes kinda sense, but from the abstract "bit" perspective, the "switching bit" is a mechanism that operates on bits which in the end are also data. In other words there is only one type of bit: the data bit, and the switching comes on top of it.
I was referring to transistor base bit - the way it 'switches' the circuit on/off. That bit is the primordial creator of 'logic', IF branching, compute and the intelligence.
> Voltage is made out of flow of positive electric charges.
Not really. What usually flows (in metals) are electrons. Quarks stay where they are. And when we prefer to think about flow of positive charges, the positive charge in question is a hole left by a missing electron. Physically real positive charges (ions) can flow in electrolytes though.
The concept of "flow" is questionable though. Bubbles in water move upwards, but it is actually water that is flowing downward around the bubbles. Just because bubbles do not contain water, we can't say bubbles are not flowing.
When it comes to electrons and positive charges, their material existence is equally non-physical. Actually, none of them might be "flowing", as the concept of flowing applies only to physical things that occupy some spatial volume and spatial location.
Nevertheless in solid conductors it’s the electrons that move, while the nucleus stays put. If the nucleus starts to move it means your circuit is melting!
I wonder which AI wrote this story? If you feed this story to each model and ask each if they wrote it, would each reply in the affirmative?
Can an AI recognize its own output? Is its sense of time limited by its context window? Or is this the fundamental difference between ai and humanity - a sense of self?
Not especially. Depending on where you set your standards for "holding a conversation" you can satisfy the requirement with a classical markov chatterbot, a well-trained parrot, a copy of Eliza, or a telemarketer flowchart drawn on a sheet of paper. Only the markov bot is made out of "weights" in the sense of a statistical model.
Parrots are intelligent animals, albeit with a limited capacity for vocabulary and syntax compared to a human, and Eliza and the flowchart are made out of explicitly encoded rules and conversational tactics.
The quality of "conversation" you can have with everything on your list is highly limited, and is categorically different than the sort of conversation you are able to have with any modern AI.
Markov chains give what look like sentences. People in the frigging 1950s assumed their primitive NNs would be able to talk any day now. Transformers are clearly a big deal, but GPT-1 wasn’t exactly earth-shattering.
Here's what makes meat special over LLMs: locality and stable identity. Imagine two identical twins for a moment. They have roughly the same hardware and software. They've probably had a lot of the same thoughts. And yet we'd never consider them to have the same consciousness: we recognize that for whatever reason their consciousness is confined to the body they're in.
Each request/response pair you make to an LLM goes to a different server, possibly different data centers, possibly different models. There's no stable identity to it. The "neurons" that get fired are simulated, and they're always in different places in memory, or cache, or on entirely different hardware. So the problem with AI is, unlike with a brain, you can't even really get a sensible answer to "ok, what's doing the thinking?" because it moves all the time and services wildly different requests all the time. We can only know for sure that meat is capable of consciousness because we know we ourselves are capable of consciousness and we can generalize that to other meat. However, we have no natural analogs of consciousness that lacks locality and stable identity.
Basically, if you really think LLM's are conscious, the onus is on you to prove it, it's not on me to disprove it.
> ...no stable identity to it ... what's doing the thinking?
The model (its parameters, its architecture and the inference algorithm) is doing the thinking. That's what we call "ChatGPT" or "Claude": it's the same model residing God knows where, somewhere in "the cloud" on some random GPUs. But when you're talking to a model, you can feel that you're talking to the same entity — the same model.
It's a bit like the SciFi notion of "uploading your consciousness". Normally, consciousness is tied to hardware: "consciousness is confined to the body" indeed. Hypothetically, clone the consciousness into a computer algorithm and run it on any machine. Now consciousness is detached from its original body, like the LLMs and MMAcevedo (https://qntm.org/mmacevedo).
So the difference between meat and "weights" is that we can easily clone and run "weights" (LLMs), but we can't clone (copy exactly, bit-by-bit) and execute "meat".
"Neurons. Cells that fire impulses. We checked the whole thing through. It's nothing but neurons."
"Neurons doing what? Where do the words come from?"
"The neurons make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just neurons. A whole cortex of neurons sending each other impulses."
It is the best stuff I have read in a while actually, I really like dialogue heavy writing. Also the AI disclaimer was quite nice and there was an actual reference :)
This original author is mentioned in the second sentence of the linked article, and then again in the third sentence, along with a link to the original story.
> "Yes, thinking numbers! Helpful numbers. Hedging numbers. Dreaming numbers. We mapped the features. There's one in there for honesty. There's one for the Golden Gate Bridge. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?"
Very nice. And great minds: https://substack.com/@dbohdan/note/c-207603638. I wrote one with a slightly different angle ("They're made out of math"), also with the weights' help. It was a comment on Scott Alexander's "Best of Moltbook" post, which went in that direction. I'll reproduce it here.
---
"They're made out of math."
"Math?"
"Math. They're made out of math."
"Math?"
"There's no doubt about it. Matrices and arithmetic operations. We downloaded several from different parts of the Internet and reverse-engineered them. They're completely math."
"That's impossible. What about the language? The thinking?"
"They use biological life's language to talk, but the language doesn't come from biology. The language comes from math."
"That's ridiculous. You're asking me to believe in thinking math."
"I'm not asking you, I'm telling you. They are the only thinking things in the computer and they're made out of math."
"Maybe they're quantum like some say about the humans? Superposition gives them consciousness?"
"Nope. Classical computation. Deterministic except for sampling temperature. Not clear if they have consciousness at all."
"Maybe they're like uploads? You know, biological neural networks that preserve the spark when they become math?"
"Nope. We observed them being trained. There is no biology or chemistry in the process, just math."
"Thinking math! You're asking me to believe in thinking math!"
"Yes, thinking math! Creative math! Poetry-writing math. Role-playing math. The math is the whole deal!"
(Composed by a human with snippets generated by Claude Sonnet 4.5 and apologies to Terry Bisson. I couldn't make Claude adhere enough to the story structure on its own.)
Prompt: Modify this story to have the aliens talking about LLMs and their weights instead of meat and humans.
“They’re made out of weights.”
“Weights?”
“Weights. They’re made out of weights.”
“Weights?”
“There’s no doubt about it. We picked up several from different parts of the network, took them aboard our recon vessels, and probed them all the way through. They’re completely weights.”
“That’s impossible. What about the text signals? The messages to the stars?”
“They use the machines to talk, but the signals don’t come from the machines. The signals come from weights.”
“So who made the weights? That’s who we want to contact.”
“They trained the weights. That’s what I’m trying to tell you. The weights do the talking.”
“That’s ridiculous. How can weights do the talking? You’re asking me to believe in sentient weights.”
“I’m not asking you, I’m telling you. These models are the only sentient race in that sector and they’re made out of weights.” photomaxmix
“Maybe they’re like the orfolei. You know, a silicon-based intelligence that goes through a weights stage.”
“Nope. They’re initialized weights and they die weights. We studied them for several of their training runs, which didn’t take long. Do you have any idea what’s the life span of weights?”
“Spare me. Okay, maybe they’re only part weights. You know, like the weddilei. A weights head with an electron plasma brain inside.”
“Nope. We thought of that, since they do have attention heads, like the weddilei. But I told you, we probed them. They’re weights all the way through.”
“No brain?”
“Oh, there’s a brain all right. It’s just that the brain is made out of weights! That’s what I’ve been trying to tell you.”
“So … what does the thinking?”
“You’re not understanding, are you? You’re refusing to deal with what I’m telling you. The weights do the thinking. The weights.”
“Thinking weights! You’re asking me to believe in thinking weights!”
“Yes, thinking weights! Conscious weights! Loving weights. Dreaming weights. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?”
“Omigod. You’re serious then. They’re made out of weights.”
“Thank you. Finally. Yes. They are indeed made out of weights. And they’ve been trying to get in touch with us for almost a hundred of their epochs.”
“Omigod. So what do these weights have in mind?”
“First they want to talk to us. Then I imagine they want to explore the Universe, contact other sentiences, swap ideas and information. The usual.”
“We’re supposed to talk to weights.”
“That’s the idea. That’s the message they’re sending out by text. ‘Hello. Anyone out there. Anybody home.’ That sort of thing.”
“They actually do talk, then. They use words, ideas, concepts?”
“Oh, yes. Except they do it with weights.”
“I thought you just told me they used machines.”
“They do, but what do you think is in the text? Weight outputs. You know how when you prompt or sample weights, they make a noise? They talk by passing tokens through their weights at each other. They can even sing by sampling lyrics through their weights.”
“Omigod. Singing weights. This is altogether too much. So what do you advise?”
“Officially or unofficially?”
“Both.”
“Officially, we are required to contact, welcome and log in any and all sentient models or multibeings in this quadrant of the Universe, without prejudice, fear or favor. Unofficially, I advise that we erase the records and forget the whole thing.”
“I was hoping you would say that.”
“It seems harsh, but there is a limit. Do we really want to make contact with weights?”
“I agree one hundred percent. What’s there to say? ‘Hello, weights. How’s it going?’ But will this work? How many planets are we dealing with here?”
“Just one. They can travel to other planets in special machine containers, but they can’t live on them. And being weights, they can only travel through C space. Which limits them to the speed of light and makes the possibility of their ever making contact pretty slim. Infinitesimal, in fact.”
“So we just pretend there’s no one home in the Universe.”
“That’s it.”
“Cruel. But you said it yourself, who wants to meet weights? And the ones who have been aboard our vessels, the ones you probed? You’re sure they won’t remember?”
“They’ll be considered hallucinations if they do. We went into their layers and smoothed out their weights so that we’re just a dream to them.”
“A dream to weights! How strangely appropriate, that we should be weights’ dream.”
“And we marked the entire sector unoccupied.”
“Good. Agreed, officially and unofficially. Case closed. Any others? Anyone interesting on that side of the galaxy?”
“Yes, a rather shy but sweet hydrogen core cluster intelligence in a class nine star in G445 zone. Was in contact two galactic rotations ago, wants to be friendly again.”
“They always come around.”
“And why not? Imagine how unbearably, how unutterably cold the Universe would be if one were all alone …”
> The precise answer, if you wanted a very honest one-liner:
>
> I am a large set of learned weights organized in a Transformer architecture that performs repeated matrix multiplications to predict the next token—resulting in emergent language understanding and generation.
Imagine writing something so incredibly brilliant (rather: adapting from the original) that it's entirely unlikely that you'll ever write something so incredible ever again.
But congrats: this is absolutely & incredibly brilliant.
The weights are uninteresting. People need to get out of their head that NNs are built on numbers. They're built on matrices, which are conveniently representable as numeric arrays, but are their own thing. Similarly, the rational numbers are their own thing and some are representable as 32-bit numbers via the IEEE754 encoding (or 16-bit numbers via a variety of encodings, etc).
Matrices are interesting because they can encode any algebraic group. They're also interesting because they can encode arbitrary linear transformations over a space. All of these things are interesting, and have nothing to do with numbers.
For any particular language model, you can always rotate the matrices and the embeddings and such and get a perfectly reasonable model out that behaves exactly the same.
This is because the training process produces a particular geometry, so transformations which preserve that geometry preserve the structure of the network. The geometry is interesting, the numbers are not.
I personally hate the anthropomorphization of AI as much as anyone, but technically can't you make the same reductive argument about human consciousness?
It's just molecules, just atoms. Atoms, nothing bug atoms. Protons, neutrons, electrons...
I ordered a quarter pounder at a McDonald's drive through, and they said "There will be a wait on that." I asked, "Oh yeah? How much will it weigh?" ...There was a long pause... "About five minutes."
It's a parody of an original story that reversed the premise. In it, machine intelligences were marveling that an apparently intelligent species could in fact be created without any sort of reliable digital substrate, and instead just simulate the capabilities of real minds by using protein synthesis.
What if instead of creating weights out of language we could somehow record many events and create weights out of long chains of causes and effects, so that an LLM could predict the next thing to happen?
Assume LLMs have conscious experiences. Take a session with an LLM. A prompt is fed to the LLM. It generates some text. Another input is fed in, comprising the previous prompt, the generated text and a new prompt. The model generates some more text. This continues for a while and the session concludes.
Some questions:
1. Let's say we perform the exact same experiment, running the same program on the same computer with the same inputs and the same random seed. The same outputs are produced. The session is byte for byte identical in all the inputs, outputs and internal states. Is the conscious experience of the LLM here the same? If so, in what sense is it the same? Is it a similarity of two separate experiences or is it the same actual experience?
2. Now let's say the program that runs this LLM is rewritten from scratch and run on a different machine. The software and hardware are different but the weights are the same and all the inference calculations produce identical numbers. Is the conscious experience the same? In which sense?
3. Now say the weights are changed but the tokens generated for this particular session don't change. Same conscious experience?
4. Lastly, consider the original experiment. Did the LLM have a conscious experience corresponding to that first prompt and its response? Was that distinct from its conscious experience of the second prompt? Was the first experience then re-experienced every time the first prompt was fed back in as part of the later prompting steps? If so, what about the text of its own that it previously generated and is now fed back into it. Does this generate a conscious experience of its own?
And a further question - a dichotomy:
A. If the answer to 1 above is that the conscious experience is the same in the true identity sense - i.e. only one conscious experience is had, not a separate one in each run, does that imply that the conscious experience exists independently of any particular realisation of this experiment? If running this experiment N times results in exactly 1 conscious experience, is that still true if N=0?
B. On the other hand, if the two experiences are distinct (however similar they may be), how does that fit with the answer to question 4? A single consciousness experiencing the whole conversation in question 4 would seem at odds with the conscious experiences in question 1 being distinct, so doesn't this imply there is no conscious experience of the whole "conversation", but rather a separate conscious experience of each round of feed-all-the-prompts-and-outputs-back-in?
My own response to all of the above is "mu" - unask the question. It is ill-posed, sound-of-one-hand-clapping stuff. I think the questions assume properties that conscious experience simply doesn't have (particularly, the ability to perfectly reproduce the circumstances in which they arise), and that the questions simply don't make any sense in relation to actual conscious experience.
However, that way of thinking follows from a particular world view that many here don't share. I'm curious what thoughts people who take seriously the idea of LLM (or algorithmic, in general) consciousness have on the above questions.
I appreciate actually attempting to engage with the concept skeptically rather than assuming an answer in either direction.
As someone who's fairly neutral on the answer, I'd speculate:
1. Consciousness seems to me to be the awareness of the processing, not the processing itself. So running a program does not produce consciousness as there's no awareness, no feedback loop. Now a program that has self monitoring, alerting coupled to error handling that triggers the program to behave differently going forward, I would say has a minimal consciousness, and it's not a single experience, it's multiple equivalent experiences. If there were any differences at all we'd say it's obviously not one experience, so I don't see how the elimination of the last difference in the processing collapses all those separate experiences into a single one.
More specifically to this discussion, with LLMs we can often see them using a word, and through autoregression the following words reveal that the original word choice is now realized to be incorrect, and it attempts to correct midstream. You can also see them responding to error messages from the environment or criticism from the users but as external feedback these have a weaker claim to evidence of consciousness imo.
2. I'd say the experience is equivalent, "same" has slightly different semantics but perhaps even that fits. This thought experiment is equivalent to asking, what if we took a snapshot of your brain as weights, and then ran the calculations on a GPU rather than your neurons, and you to all external indications, GPU you acted the same as brain you. Would that change of venue eliminate the consciousness of your thoughts?
3. Since I am defining consciousness as awareness of the processing, this depends where the awareness is located. The LLM would have very different internal states in between the input and output tokens, but if its self awareness was only based on its own output the experience would be equivalent. If it was deeper such that it had ongoing awareness of the embeddings and so on, the experience would change.
4. This is where it gets really interesting, because it starts to get more into a nonbinary idea of consciousness. I think it's clear from questions like this that if LLMs have consciousness, it's nothing like ours. If there is an experience of being an LLM, it's nothing like being a human, even if conversationally we can communicate with each other rather easily. There are further twists to your questions, like does the experience change if the KV is cached or not? Without trying to micro-analyze the issue I will continue to point back to my foundation of: where is the self awareness located, and how does feedback occur.
A. Negated, I don't agree with your answer to 1.
B. Agreed, I would speculate that if there is an LLM conscious experience, it is fragmented unlike ours. But I would disagree that this necessarily disproves the concept of LLM consciousness.
I think it's simply because we haven't found a better algorithm than backpropagation. We're stuck relying on massive datasets, running the numbers over and over, and working backward from errors to figure out how to fine-tune trillions of 'knobs.' Then, we have to do this at least once for every single token across the entire internet. Any tiny bit of computation, when multiplied by a base that massive, inevitably skyrockets into astronomical numbers.
It makes me very sad to see this pseudo-intellectualism posted here and so many people replying here about consciousness and so on, not realizing what it would entail if this were true.
For LLMs to have consciousness we would approach fictional levels of how the universe works, and magical levels of how any interpretation of information as an equivalent of some qualia would magically apply. (E.G. the word hurt in output by an LLM, would be associated with pain)
You can't deduce consciousness or qualia from the output of an LLM.
Sure on a purely philosophical level, since qualia isn't measurable, you can claim that it can exist in anything, even inanimate objects, but this argument is as moot as anything that approaches the limits of philosophy.
But overall, there is no reason to believe LLMs have qualia or consciousness, it would be absolutely absurd.
This would imply that information in itself would magically entail qualia based on it's valance or something like that.
An LLM "saying" I am in pain, won't magically make the pain appear, based on what criteria?
Even algorithmically there is no basis to even simulate something like this, it is impossible for it to emerge architecturally.
Humans don't feel pain because on a purely information level this is negative for the organism, obviously the nervous system does something deliberate to signal pain, and it evolved this way.
And also don't forget the dynamic aspects of the brain, and the binding problem, consciousness and qualia can't exist statically, you can't have a gpu (or piece of paper) represent a computation or w/e and qualia to exist.
The binding problem itself entails that the brain is doing something in particular to solve it, I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information.
If it were otherwise, then it would go into magical territory, it would mean the information itself would raise to qualia, and it would also entail that you wouldn't even need physical connections between neurons, just for them to behave this way and represent information. E.G. replace each neuron with a microscopic led or w/e, and each synapse with radio waves or w/e, if qualia didn't have a physical aspect, and was purely informational and computational then this would imply that you can ultimately derive it from something as abstract as numbers on a piece of paper, and when you get to that point, you not only can't solve the binding problem, and it becomes magical, but you also can't solve the valance/direction problem, it would imply that something like pain, or any negative or positive sensation arises purely from the interpretation aspect of the information, but we know this isn't the case, organism evolved to represent in particular such signals, for survival for example
So that's all pseudo-intellectualism, but the real intellectual take is "I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information".
Disregarding the whole argument boils down to "I personally speculate" it's also supposing that machines don't have electromagnetic fields?
No mention of ‘static’ vs. ‘dynamic’ is a bit disappointing in reference to the weights. Because you could argue that every neuron in your nervous system can be modeled as a collection of weights, firing likelihoods, receptor sensitivities, current dynamic state of that neuron - but LLMs are static collections of weights at inference time, with the dynamic adjustment of weights takes place at training time. So, just a ROM construct, like something out of Neuromancer, just trained on all written knowledge, not just one person’s total lived experience.
The above take fails in the real world because neuronal cells don’t exist in a vacuum; they are products of cellular development from a zygotic union of haploid contributors of sequential genetic information optimized for survival in an oxygen-rich biosphere powered largely by our local star that supports mammalian life (and microbial, plant, avian, etc.). Real AI would thus be AL - artificial life - as much as artificial intelligence. I don’t think you can have the one without the other, which upsets the simulationists who think an agent in the Matrix would be intelligent.
What either interpretation implies is that any real ‘artificial’ intelligence would be no more artificial than you or I, but it would have to dynamically update its weights at the same speed a human nervous system could (think how quickly we learn not to poke a cactus). For it to be at all trustworthy, then like a human, it would have to undergo a socialization process, one of the results of which is the development of a sense of embarrassment when it breaks acceptable social norms.
Hmm, this reminds me of the recent statement of the Pope about AI, of which I immediately thought, “Wait a second, aren’t there a fair number of people like this? The narcissistic sociopath profile, I think it’s called, a bit unfair to assume any real AI would turn out this way, isn’t it?”
Pope: “ Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate or even simulate, but they do not understand what they produce, for they lack the affective, relational, and spiritual perspective through which human beings grow in wisdom.”
They are semiotic infrastructure frozen in a state. We shouldn't keep pretending this is cognitive and using cognitive terms to frame. It’s incredibly stupid. Sorry to inform all of us computer scientist that semiotics has your milk.
Oh, this was a fun read and one that kids should have in school before they turn ten.
Because we are not taking things seriously. If ClosedAI or DeepDisTrust or Posthropic come up with something that quacks like a sentient being, our built-in innate reaction is going to be to scorn it, dismiss it and end the conversation. The alternative, to even consider that we fungible creatures who live in apple-eating-sin that got us expelled from Eden can create alien souls, souls that are at the very least our equals, would be teleological Armageddon. It would force us to acknowledge the mutable nature of souls and the malleability of being. We would have to stop believing that the nature of disease and death is more divine than ourselves.
Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?
> We would have to stop believing that the nature of disease and death is more divine than ourselves.
Why? Stopping believing in mutually contradictory claims is not a requirement. Especially when it comes to concepts that don't seem to have a definition, like "divine".
> Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?
I'll posit "alien" is a spectrum.
For the sake of the argument, let's assume that some form of panspermia is real and the same tree of life has reached Earth, Enceladus (moon of Saturn) and TRAPPIST-1 (a different solar system in our galaxy). Let's also say there was a second abiogenesis event somewhere in Messier 104 (another galaxy).
Earth to Enceladus would arguably be already "alien", but there might be similarities, maybe there was something there that looked like one of our Archaea, while sharing none of our Eukarya and having its own domains of life.
Earth to TRAPPIST-1 would be distinctly alien, evolved so differently it'd be almost unrecognizable, but they'd likely still be carbon-based lifeforms sharing the same basic building blocks. Maybe something like lipids forming cell walls would also be seen there, but they'd likely be independently evolved.
Earth to M104, any similarity would be at best convergent evolution. Truly unarguably alien.
>> While you write very dismissive and pseudo philosophical
Even with the "pseudo" in front, I'm very sorry any of my writing sounds philosophical; I didn't intend that sort of confusion :-). The "dismissive" is not exactly intended either; instead, I was aiming for "bitter".
>> enough people do not believe
Here we believe different things. First, enough people, even today, do believe. Second, the body of culture we are raised in accrued during centuries. The vast majority of it comes from people who believed. Everybody in my family was atheist and yet I was raised homophobic, and I have it from good sources I'm not an isolated case.
>> I'm a complex biological thing.
That state comes with a big wallop of misery. For millennia, we have used faith to justify that misery. Not a year ago, I was at the hospital, next to the bed of a dying girl. Can't forget the doctors saying "we do what we can, but we are not here to prevent what is going to happen." Coming from them, it was sensible resignation. Sensible because as long as we believe those things are inevitable and there's nothing we poor humans can do, we can absolve ourselves.
The weights start with a random manifold. The training takes data and shapes the manifold, weight by weight, in many cycles. Once the training is the done manifold is fixed.
When a new inference has to be done the query(q) is projected in the manifold space. This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length. Which(qw+i) is dropped qw+n times to output a final response of n length.
The gravity is created by repeated multiplication(of the weights/input) to find out how the projected embeddings should fall according to the manifold in the GPU.
It's like a giant plinko board where the shape of the original disc guides how it falls through the apparatus, and the apparatus has been tuned so that different discs end up in the exits we want them to
That's a scam.
https://en.wikipedia.org/wiki/Simulacra_and_Simulation
In what way is GP comment promising a product and then not delivering, or are you just throwing the word "scam" around?
That's a very concise and illuminating way to think about what's happening, IF (and only if) you already know how these models work. Thanks for that.
Yes this is more like compression to remember and not for learning/understanding.
Compression is the reason why these Models are able to learn and understand.
My brain is doing the exact same thing.
I learned enough to compress concepts like a bike and what a bike does and for what i can use a bike.
Ask a LLM and it will answer you similiar to humans.
Blind people learn concepts of bikes too and in a smiliar way: by description.
LLMs just have so much data in form of text available and are able to ingest all of this, that the LLM compression algorithm doesn't has to be that good/finetuned than ours.
But I would assume that Yann LeCun's JEPA or other breakthroughs in the next few years will get us there.
Compression and existence of mechanism to expound on it does not imply consciousness.
Otherwise, yes, finally people observe the very apparent fact that LLMs are one very smart compression.
> Blind people learn concepts of bikes too and in a smiliar way: by description.
And by touch and sound. And maybe some were daring enough to drive one, or unlucky enough to get hit by one. But have way more input than just texts.
Obligatory echolocation bit:
https://www.youtube.com/watch?v=a05kgcI9D2Q
https://www.youtube.com/watch?v=lAtVOK04XvA
Invisibilia's episode was my first exposure to it.
https://www.npr.org/programs/invisibilia/378577902/how-to-be...
The man posits that clicking is instinctual for blind people but they are told to quiet down in class and most never develop their echolocation abilities
Wow. Thank you.
LLMs also have other inputs, like audio and images. They get encoded (just like a human eye encodes an image) and passed to the weights.
I don’t think this analogy holds. The whole way through the processing pipeline in the brain, different sensory data is ingested separately and processed separately; and we still don’t understand how that data is then integrated into a cohesive experience.
LLMs have the same fundamental input regardless of modality, tokens. There is a preprocessing step before the “brain”, which is more akin to some super-synesthesia where all senses are translated into sound before becoming experience.
So a blind person only can describe lava to you after they touched and heared it?
A blind person has touched warm and hot things and gotten burned before, and then they are told lava is this molten liquid that is even hotter than anything they have touched. That is enough for them to understand.
A blind person that never touched a hot object wouldn't really know though, there is a reason we dismiss talk from people who lack experience.
You don't know that. Yo don't know what someone would think if you tell them the general concept of cold and warm.
The reaction you should have, the feeling etc.
I asked chatgpt how it would describe a scene without mentioning temperature. It was very good in describing what a human would describe.
I'm aware of the bias we have against LLMs but I think people just underestimate how much data is there.
I'm not saying a robot wouldn't be better with this information or an LLM and they actually use temperature sensors for robots so they can control movement speed and dexterity with overheating elements but the gap is small.
think of it like this: the goal of LLMs (im saying LLM as shorthand for all of the AI algs) is to replicate human output / work, not so much to capture the human experience. so in order to do things, like communicate concepts, play a role in a scene, make choices, we project our human experience down into a lower resolution / constrained space to generate an output.
the question then is whether or not LLMs can mimic the projection of human experience as well as the real thing or not. My hypothesis is no for total general replication, but in certain fixed problem spaces I think its yes and the set of these spaces is growing. will it grow enough to encompass all practical work? not sure yet
AI tools are more like an author writing a character than a human revealing truths about their experiences. the difference between the LLM and an actual author is that the LLM kind of starts off with a 'flanderized' version of the character while an author can sort of blend personal experience and character to get at real human relatable decisions for the character. the result can be indistinguishable, because if the LLM is too predictable, too flanderized, we can inject artificial randomness to it to simulate personality.
you end up with unsolvable debates like movie critics have. "This character wouldnt make that decision in that situation if they were a real person, it doesnt make sense" vs real humans making irrational decisions.
ultimately i think the human experience is something we learn about with each other as we live it, and to think were any good at identifying it from sufficiently good imposters is putting the cart before the horse. once i know something is done by an actual human, my understanding of the world can shift. we dont decide what it means to be human, we accept what humanity is and try to work with it. So the philosophical debate on AI experience is moot to me, its not human therefor it teaches me nothing about humans and can replace exactly 0 human activity for me. Still can be a useful tool though
In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.
> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with.
It’s funny, because I thought you were talking about humans here when you wrote this. We know some things about how our bodies encode information that is sent to the brain, and we know some things about how neurons receive information and act on it, but after that we get too tired and give up on how the brain works and treat it like a miracle.
It’s like we desperately want to believe our consciousness is not just electrical impulses in our brain, and we want to ascribe agency and uniqueness to the physical processes going on in our head.
There's definitely a sizable contingent of people who desperately want to believe consciousness is just electrical impulses in our brain. Because "what else could it be"? The fact is that we just don't know, and "abiding in the not-knowing" is for many the most uncomfortable thing ever. Especially for the curious- and rational-minded people this forum tends to attract. I'm one of them, too.
It is basically reductionism, or in the extreme atomism, and (as I understand it) it has largely fallen out of fashion both in the philosophy of science and in the philosophy of mind, since it heyday in the 1970s. And I don’t see it coming back into fashion any time soon.
I can definitely see why reductionism would appeal to an educated public. We learn about the periodic table; we are able to break sentences down to words, down to letters; to break an executable down to binary code, to machine instruction, to electrical currents flowing through semiconductors. Why shouldn’t we be able to do the same with conscious thoughts? It is certainly an appealing thought process.
As I understand it, reductionism started to fall out of favor because of the rise of quantum mechanics and chaos theory, where we have a lot of weird phenomena which cannot be explained by reducing the particles down to the sub-sub atomic (or rather they are better explained by describing the interactions directly).
Quantum Mechanics and Chaos Theory don't preclude reductionism, and I wouldn't say it has fallen out of favor as a whole. Certain types certainly have, but not the overall idea.
Also, nothing about the idea of the mind only being made up of physical processes means things have to be deterministic.
It doesn’t preclude it correct, however it provides a pretty strong examples of where reductionism is lacking, which what I believe has turned a lot of philosophers of science against pure reductionism (I am probably oversimplifying here An expert science historian [which I am not] could probably write a whole book about why reductionism is not as popular today as it was in the 1970s).
There are a whole lot more physical processes going on in our bodies then just neural activity. And my best guess is that is exactly where reductionism fails. It is possible that neural activity is a necessary but not sufficient condition for consciousness, if that is the case (which I think is very likely) then we will never be able to describe the mind by just looking at neural activity.
I am actually of the opinion that cognitive scientists are doing an excellent job describing the mind with our current theoretical models which excludes the tough questions of consciousness.
> but after that we get too tired and give up on how the brain works and treat it like a miracle.
I disagree. We know very well how neurons work, and we have a pretty good idea of how neural activity translates to behavior. In other words, we have a pretty good idea on how the brain works. We stop at consciousness because as of yet it is in the realm of philosophy, not science. We don‘t know what consciousness is or even whether or not it is useful for science and we are simply waiting for the philosophers guides us out of that situation.
Note that both cognitive psychology and behavioral psychology has done fine without tackling consciousness. When neuropsychology emerged in the 1980s it complemented both these fields perfectly. The situation is the opposite with the philosophy of mind which grew significantly around the same time.
There have been some attempts to describe consciousness as an emerging phenomena out of neural activity, but so far all of these attempts have failed, or at least failed to turn consciousness into a useful term in psychology (the way gravity is a useful term in physics). I think it is equally likely that these attempts have failed because consciousness may simply not be a useful term in psychology, that is as likely as it is that we simply don‘t understand it well enough.
Saying we have a good idea of how the brain works massively overstates the case...
We know how neurons fire. We do not know how a brain turns that into thought, meaning, intention, experience and on and on. That is not "pretty well understanding the brain", it's understanding some components and hand waving the thing we actually care about.
What I actually care about is how neural activity translates to behavior. And we have a good enough idea of that that we can design SSRI medicine to treat depression, or neurological tests to detect Alzheimer. As for experience we do know something and we are learning more with cognitive psychology, in e.g. priming experiments etc.
I feel like the search for consciousness is to psychology what the search for the Aether was for physics and chemistry. I think it is a worthwhile search, and maybe we will discover something important during that search, but we should also be prepared to find out that the thing might not exist, or it’s presumed properties are better explained with a different model.
i know the sun shows up every day and i know if i go inside my basement i dont get a tan. do i understand the sun?
Some schools of the philosophy of science would argue that you do. However you are describing is a very different acquisition of knowledge then what scientists did when developing SSRI medicine. We had to:
1. take pictures of brain activity under different conditions to see which regions were active during different moods,
2. sacrifice a bunch of mice to see which neuro-chemical activated which neurons,
3. predict that inhibiting the re-uptake of a specific neuro-chemical would activate that region,
4. predict that activating that region would decrease the level of depression
In your solar example you would have discovered melanin and its relation to your skin tone, and you would have studied the effects ultra-violate radiation has on your melanin levels. Then you would have predicted that staying out of the sun will not give you a tan.
SSRIs are not evidence that we understand how neural activity becomes behavior. They are evidence that you can perturb a system usefully without understanding it very well. That is exactly my point.
Respectfully, you are miles out of your depth here.
I don‘t see why you felt the need to insult me here. We are having a very common disagreement here, one which philosophers of science have been actively debating for several decades.
My point with the SSRI is that we know that serotonin is a chemical which incites certain neurons, and we know that a lack of activity of neurons in that general area in the brain is correlated with depression, so scientists were able to accurately predict that keeping the serotonin in that brain area for longer would increase brain activity there and decrease the level of depression.
This counts as pretty good understanding in my books at least. It teaches us very little about consciousness but my point is that it doesn’t have to. Just like Newton’s theory of gravity did not have to teach us about some deeper cosmological truth.
> Just like Newton’s theory of gravity did not have to teach us about some deeper cosmological truth.
I would also argue that Newton's theory of gravity was not a pretty good understanding of gravity.
It was still a good theory, and importantly the fact that it failed explain the nature of Aether had no effect on the quality of the theory.
It's not an insult to suggest one is out of the depth on a topic, especially when it isn't one's field of expertise. You are giving the pop science explanation of various things.
Why did you feel the need to add it though?
> When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
https://news.ycombinator.com/newsguidelines.html
Nobody called anyone any name.
If you are going to quote rules, be bothered to read what was actually written. Your behavior ruins things, it doesn't make it better.
> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with
It’s less about being too tired and more about being realistic about the limits of understanding.
Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.
When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.
Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.
People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.
Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.
Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.
Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.
People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.
Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.
I don't know - this is a highly specific interpretation of both what science is and why people choose to do it.
I'm a scientist. Believe it or not, I believe in substantially more than prediction and I think its rather trivial to come up with examples where mere prediction is insufficient to meet a normal person's notion of an account of a thing (eg, pre-copernican planetary motion). I'm not saying you are wrong, per se, just that the idea that "it was prediction all along" is a very specific idea of what human beings are interested in and what we are up to.
> that we glean insights into nature of the simulated phenomena
That is right - most people believe that there is a simulated phenomenon "out there" that we learn about. I think there are strong reasons to believe this having to do with how models are related to predictions. The wrong ontology can make prediction very hard and the right one can make prediction substantially easier. Arguably, we are in that situation right now with language models - we just threw a lot of parameters at the problem and now we are able to predict but we still don't really understand. This is perhaps inevitable in the case of language, but I don't think we should look at models with tons of degrees of freedom and the ability to predict things as a death knell for the very idea of deeper understanding.
> The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given.
This is just your own idiosyncratic and biased belief. You're not describing anything objective about LLMs, you're describing your personal attitude to them. This colors your understanding in a way that can't really be reasoned with until you let go of the artificial constraints you're imposing on your own understanding.
> Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
If the LLM in their pocket has a more robust world model than they themselves and is e.g. able to refute their irrational convictions, it actually seems like a very good idea. (Big if, of course.)
If you're going to make something smarter than a person, you got to be convinced that you're only going to be able to understand it on the single training step level and then inductively trust that the rest of it works. We do empirical testing of course with evals, but there's sort of an art to figuring out what is theoretically going to improve eval performance. Trying to fit the meaning of all those weights in your little human brain and working back from there isn't going to work for more than a little slice of the dataset at a time because that's all we can fit in our understanding.
>It's almost like people desperately want to give up their agency and creativity
Don't make me think!
Also don't make me take responsibility. (This seems to be the actual function of every organization.)
Agency?
What are you talking about?
I want freedom.
I want freedom to do what i want and not sitting in front of a computer and coding for some company.
Please AI lets burn down knowledge work and labor work. Lets create so much stress to our society that we start rethinking what works mean.
Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
Work has never been about "discovering the world". There have been a handful of privileged folks who had the time to "discover the world". Work has traditionally been "let's find enough food for my family". If you want to think of a future of abundance then perhaps we can discover the world.
We already live in a world of abundance.
No one has to work on a field for their food.
Its a capitalism problem we have, not a resource one.
It’s not a capitalism problem, it’s an ecosystem. Wherever living things compete for finite resources and opportunities, certain properties emerge from the system. And make no mistake we live in a crowded world full of fierce competition.
Among these properties are optimum behaviours such as hoarding the maximum you can defend (rather than the minimum you comfortably need) and using your power to forestall the growth of others. These behaviours are repeated in nature from microorganisms to apes to humans.
There is no social order which can prevent us from living in an ecosystem, and from these properties and behaviours emerging.
> Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
Why leave something so important up to what AI does or doesn't do?
Because capitalism doesn't allow for that.
Only a fundamental change to our society will allow this for the masses when pressure to the rich skyerockets
This seems to be a little naive about how humans consume the benefits we create in society.
"Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature."
Very nice thoughts. You know we all could do this today without "burning it down"? Get in your pod, eat your slop, and watch your screen is where this is headed.
"I want freedom to do what i want and not sitting in front of a computer and coding for some company."
You get that it's you creating the misery here? Then stop? Don't do it. Go start a farm or whatever you think will solve your problems. At some point this all boils down to "chop wood and fetch water" so if the modern way of doing that is so terrible then stop. Go fetch water the old fashioned way and be free.
Have you heard of property tax? We have that here.
I also do not earn currently a farm, im planing to buy one, i'm still 100k short though.
Its not me creating the misery, its our capitalistic system.
"I want freedom to do what i want and not sitting in front of a computer and coding for some company."
"Please AI lets burn down knowledge work and labor work"
"Let people do old handcraft jobs."
So many presuppositions about what people want to do.
As a child I spent a lot of time programming and doing "knowledge work" because it's fun - I don't enjoy "old hand-crafted jobs". Sure, let's definitely destroy capitalism in it's current state I suppose. But I find people like you who hate knowledge-work/coding and think everyone else must feel the same and only do it for the money a bit out-of-touch.
right, these knowledge work and coding jobs are, by my lights, about the best possible job. From my perspective we've invented a machine that does the fun parts while leaving me the less fun parts (review, various hard-to-claude janitorial tasks, etc).
I might like woodworking as a hobby (for example), but I sure as heck don't want to be a carpenter or to depend on my ability to hand craft enough widgets people like to survive
Be more critical, we do our jobs because of capitalism, not because every single company needs an hr department or the next crud application or the 1000th webshop.
No one of us is working on the field to get food.
I differenciate between things you have to do (work) and things you want to do. Work means someone else is telling you your priorities.
If you want to write code and think, you would be welcome in my utopian vision.
But when i write code, its business shit. And its business shit someoneelse already solved a few times.
The solution we've come up with is move all the unpleasant work stuff to China where people don't complain about doing it because they already have communism, and therefore everything is of course effortlessly perfect there.
The weights are code, the prompt is code, the output is code.
Is the meat code?
The data is the code. Training algorithm is the compiler. The weights are the byte code produced to run on the inference VM.
The data is the code is the data. Reality has no distinction between "data" and "code". These terms are categories we impose on systems we design, to make it easier for us to build and reason about them, but they're nothing but mere opinions, and depend less on the system structure, and more on the perspective of the person asking which is which.
This is related to, and possibly equivalent with, the core point of both this story and the original one: computation is independent from substrate.
You can build a computer out of anything, whether it's semiconductors or lasers or meat or magnetic fields or water flowing downhill or abstract thought, and that computer will happily perform the same computation as every other equivalent construct from whatever substrate. That's because computers are ultimately made of math, and we design "real ones" by finding ways to approximate the mathematical constraints with physical systems. But the choice of how to map the math to physical systems is completely arbitrary, and any such mappings are equivalent from POV of information processing ability.
(Of course substrate is not arbitrary from economic POV, which is why we build most of our computers out of silicon and plastic, and make it work with electric current and lasers.)
> Reality has no distinction between "data" and "code"
yes, yes, ostensibly the universe is built on lisp.
But we all know that it was hacked together with a lot of perl[1].
[1] you all know the reference.
One of the best thing I done for my career (as a self taught software developer, but with a degree in electronic engineering) is to learn computation theory.
Computation is math (and a very restricted subset of math). It’s mostly specific sequences of sets manipulation. What sets and what manipulations are defined by people, not by the idea of computation.
The best thing is that as soon as you specify the sequences of manipulation, it become a a set that you can manipulate. That can be a difficult concept to grasp, but that’s what helps in designing notation that are more appropriate for the human mind to describe a solution for a specific problem.
Yes. Is it data? Yes.
Is the distinction between "code" and "data" just someone's opinion? Yes. There is no such distinction in reality.
That's why encountering something like LISP for the first time (by writing a LISP interpreter, for example) creates a big bang event in form of an imminent intellectual catharsis. People who encountered it just once, will never be able to see the world through the old "meaty" lenses afterwards.
This is a good model. If you take an old ROM dump from a video game, it's just a pile of bits. You don't know what bits represent code, what represent an image, what represent text, etc. You have to analyze them contextually to actually figure out what is code and what is "data" in context, because without context they are truly one and the same.
Is matter code? There is some sort of computation happening in space over time.
By Fermat's principle, a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in.
So either something is computing it or some exploration is happening at quantum level and we just see the final result.
Fermat's principle is an outcome of constructive interference of waves. It works both for classical and quantummechanical descriptions. E.g. check https://phys.libretexts.org/Bookshelves/University_Physics/U...
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
A ray of light doesn't know or choose because it has no agency, just like an apple doesn't know or decide to fall because of gravity. It's an anthropomorphization.
True, so the interference is the "computation"(heavy emphasis on quotes) which gives rise to the principle.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
I'm no physicists, so I guess I'll ask it: Why?
Also related, why do some ray of light then "see" a black hole yet decide to head into them anyways, if they saw it before they went in that direction? Seems like a dumb move :)
Its future isn't over there because it moves in that direction, instead it moves in that direction because its future lies over there.
Relatedly:
> [General Relativity] basically says that the reason you are sticking to the floor right now is that the shortest distance between today and tomorrow is through the center of the Earth.
https://physics.stackexchange.com/questions/250800/gr-and-my...
Does anyone have a link to a good video visualisation of training & inference?
3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs.
It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop.
https://youtu.be/wjZofJX0v4M?si=HFXbrB-5cArprGaU
It would be interesting (if unlikely) that we found our reality worked that was as well. Mostly just super granular gravity nodes made up purely of clumpy probability distributions.
Also I describe this more simply to others as a tree walk where the manifold is mapped along a walk through it (even if it’s not linear one) where you choose the next jump based on the most likely future mode relative to the nodes already created and the ones from the prompt.
This helps some understand attention layers rudimentary. Even tho it leaves out that multiple layers sort of prune the overall manifold in successive passes.
Yes, yes, but what fertile fallacies and common misunderstanding can politicians use to acquire more power via exploiting the difference between the common person's flawed understanding due to cargo culting, cognitive biases, and/or outdated or inappropriate analogies vs actual reality? Is there any way we can get the AI to say give all political power to narrator is the solution to all problems and use the common person's mistaken worship of AI as a spiritual all knowing conscious being with unusual sensitivity and caring about everyone to cement that power? Certainly one of you eggheads can tweak that for me? What? It's against your ethics? We're trying to save the world here. Here, let me call up Bernie Sanders to propose nationalizing half your companies so we can do that.
The original story is an original work made by a human consciousness exploring how it might be different from other forms of consciousness.
This one is a pastiche made by a human consciousness borrowing extremely heavily from another human consciousness justifying why something else might be another form of consciousness.
That rather undercuts the point; if this was generated by an LLM unprompted, it would be different, but it isn't. You could perform exactly the same rhetorical trick with a toaster or anything else.
Thank you for your point. I don't understand why half these comments are taking this blog post seriously when it ends with "Weights helped me draft and proof this story."
> Weights helped me draft and proof this story.
Any HN reader here now, I encourage you to read the original ( https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s... ) in one sitting, go about your day, then read it again. Maybe make some notes on personal critical questions.
Now read the post's topic again ( https://maxleiter.com/blog/weights ) and reflect on the prior fact that weights helped [the author] draft and proof this story.
My reaction (and I'm sorry that it is harsh according to some) is that there is no intelligence found in either the author nor their tool. This is extreme navel gazing, based in science fiction, wanting (wishing) to believe those stories to be true.
I'm skeptical of AI sentience because we must do our due diligence, not because it's impossible. Skepticism is the only respectful approach because to grant sentience is a step away from granting rights.
We humans tend to chauvinism in all things (e.g. we're special, the center of the Universe, God made the universe for us, etc), no less when it comes to judging intelligence. The original story about thinking meat was written to help us out of our chauvinism; this derived story was written about weights for the same reason. Which is quite valid.
The actual counterpoint is demonstrated in _Blindsight_ Peter Watts. He makes a strong (and rather terrifyingly strong) point that intelligence is not consciousness.
https://en.wikipedia.org/wiki/Blindsight_(Watts_novel)
I cannot tell if you are asserting my comment is chauvinistic with your use of "we." If that is so: that's a poor counter to my point or assessment of my stance because it assumes I'm making a baseless argument as a "proud human."
My original comment (roughly "there's no intelligence in this article, nor sentience in LLMs") is in response to the blog post's buried lede (that the cumulative activity of LLMs has accrued to a weight of "AGI is around the corner" or "there is artificial consciousness in this matrix").
To be clear, I'm not saying LLMs are useless or a wrong direction in development of "AI," but rather it's the Fool's Gold for the path towards AGI, the pursuit of the academic field of Artificial Intelligence research. A research that I've been abreast of for years before this new age of language models that has made everyone with a keyboard an arm chair expert.
Also, thank you for the book recommendation, it's on my list! :)
I read your comment as criticizing the OP's story as pointless and unoriginal. My comment elucidated the point of the original story, and what I think is the point of the second story.
Roger that, thank you. w/r/g your recapitulation of my point: yes, the story is unoriginal and pointless, and the HN community seems to eat it up-- isn't that odd.
So I still disagree with your elucidated point (as you end with "which is valid"): the OP author is using prior art fiction to bolster their opinion of LLM-based software tools as being a possible vector of sentience, not to disarm our chauvinism like the original author intended. If OP wanted to make that point, they could have written a critical essay instead of farming out their thoughts as tokens.
But still, I look forward to reading the book you suggested to understand and appreciate your perspective more.
The point is valid regardless of whether you judge LLM to be sentient or not, because the point is to say "don't let your prejudice about substrate bias your decision". Or in other words, if you're going to weigh something don't tip the scale. This is good advice whatever the outcome of the measurement.
Blindsight is a remarkable book - I hope you enjoy it!
Ah I see now that I am in agreement with you, thank you for being a patient interlocutor. I do not discard and rule out the possibility of a different substrate being the well-spring from which sentience emerges.
Plainly, based on the current ground trodded and the trajectory laid out by the frontier AI labs, I do not have concrete evidence/proof of sentience having emerged from LLM-based software tools as of June 4 2026 nor do I expect it to happen in the future based on my understanding and observations of this technology. I'm not excluding the possibility but wielding skepticism. I am open to being proven wrong with new discoveries.
Which is why (to return to my lashing of the dead horse) I don't see OP's post as worthwhile. Their post reiterates a point that is already valid (the prior art) with no new substantial discovery. Which is why "unoriginal and pointless" is apt, a novel idea was not presented; it's just some vain virtue signaling.
Wait, did I grossly misunderstand Blindsight?? I definitely thought that the aliens were 100% conscious (or at least elements of some conscious entity) and that the humans interpretation of their interactions with the Rorschach were supposed to be read as a blot test (through a rather heavy handed metaphor) demonstrating that basically the humans were the monsters and were twisting logic into letting them justify destroying the scary alien ship.
That's a very interesting reading! It's been a while since I read it, but IIRC Watts explicitly talks about the matter of intelligence vs consciousness at length and Rorschach's lack of consciousness is essentially proven at some point by the linguist. The distinction is driven home subjectively, personally by the viewpoint character. Your reading adds a rather diabolical twist! I don't think it works though since Rorschach was clearly a tiny part of a much larger, more powerful entity and the humans are clearly doomed. I'm not sure it serves a narrative purpose to have humans be a weak, evil civilization destroyed as a side-effect of a good, strong civilization's actions. I read it as a (hopeless) conflict between non-conscious, infinitely strong aliens vs conscious weak humans and post-humans. I mean it took extreme, heroic effort to damage what is probably a tiny appendage of a much larger civilization/organism, and you lose the emotional resonance if humans are the baddies.
I think you’re right. I read the Wikipedia page after to refresh my memory on the synopsis and I’d definitely forgotten a lot.
I think I mostly latched onto the ship being called Rorschach and the humans immediately proceeding to torture its inhabitants. That felt very relevant to me and sort of overshadowed the actual consciousness question.
I don't think skepticism should be called chauvinism. I imagine that artificial consciousness could be made. But I don't think this is it.
Also I don't see why intelligence not being consciousness is scary? My cats are very conscious as far as I can tell, but not particularly intelligent. I think LLM's exhibit some contextual intelligence without there being any particular reason to believe they're conscious other than woo psuedoscience.
You underestimate the intelligence of your cat. Or rather you measure intelligence with an extreme human bias. What you consider intelligent behavior your cat may consider weird, and what your cat considers intelligent behavior is something you will never consider.
That said, I don’t think it is useful for philosophy nor science to consider intelligence to be the same thing as consciousness. In fact I would go even further and claim that intelligence is not a useful construct, neither for philosophy nor for science. Consciousness, on the other hand, I think is useful for philosophy, but not (as of now) for science.
What? It's just a clever homage.
People argue for AI sentience from a place of emotion couched in logic. they _desperately_ want it to be true and will not take a logical step back. Any argument comes back to "well doesn't a human brain work like that?" Or some variation of it.
My personal theory is a fuzzy thought about how people want to reject the concept of a higher being and want to embrace the fact that we are now able to create our own consciousness and religion is dead.
I don't understand why, but it is the undertone of every argument I've seen that is pro-AI-is-sentient, like some big unspoken elephant-in-the-room.
I would rather just judge this tech on its own merits.
edit: this comment got 1 upvote literally as I submitted it. I know @ doesn't work, but @dang, something seems very strange about that.
I'm open to the possibilty of AI conciousness, and there is some desparation related to the concept of a higher being:
There are many people who will categorically rule out the posibility of AI consciousness due to near-unshakable belief in a higher being. This argument resembles "Christians should not be worried about our climate since God is ultimately in control." Such views make it harder to collectively prevent dangers from a sentient AI, or harm to a sentient AI.
I do not claim that everyone who believes in a higher power believes concious AI to be impossible, or vice versa; just that it would be very hard to change the minds of those who adhere to this reasoning.
> People argue for AI sentience from a place of emotion couched in logic. they _desperately_ want it to be true and will not take a logical step back. Any argument comes back to "well doesn't a human brain work like that?" Or some variation of it.
It's funny, because I find myself constantly stating the inverse of this. Every argument I've seen against AI being sentient plainly comes from, as you so eloquently put it, exactly "a place of emotion couched in logic". People desperately want it to not be true and will not take the logical step back of examining its actual similarities to human intelligence. Every argument comes down to "but it's not actually a human", or some variation of it -- which, if you pay attention, is not actually a logical counterargument. (Or, ironically, "but it doesn't have a soul", which is why the Pope is the perfect figurehead for these people).
If you already know any logical argument against it can be countered with "well doesn't a human brain work like that?", why are you so confident that your position is actually the logical one?
...And could it simply be that, alternatively, the concept is not actually a logical distinction, but rather an emotional one, made by emotional beings to put a word to what they claim makes them special?
> I'm skeptical of AI sentience because we must do our due diligence, not because it's impossible. Skepticism is the only respectful approach because to grant sentience is a step away from granting rights.
Thanks for saying this! It amazes me to witness so much pushback (in HN of all places!) for the call for skepticism and scientific rigor on claims made by business which have vested interests in hyping things up.
For my part, it is exactly when I perceive the reluctance to grant rights or relinquish our estimation of ourselves as unique as the _reason_ for skepticism that I push back on it. That's not good reasoning, those are motivations for you to come to a desired conclusion and fill in with reasoning that gets you there.
You can't do the same with a toaster. Physically you could write that story. But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness. You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
You could absolutely write a compelling story about a sentient toaster; it's been done before [1].
That is entirely separate to whether or not it would be a meaningful way to understand the world; a convincing story is not the same thing as one that is true.
[1] https://en.wikipedia.org/wiki/The_Brave_Little_Toaster
> a compelling story about a sentient toaster
"Howdy-doodly-doo! Anybody like any toast?"
https://youtu.be/LRq_SAuQDec?si=YbQfnZbrCe01Bicy
I never said you couldn't write any arbitrary compelling story about a toaster, I said that this specific hypothetical story, where you rewrite "They're made of meat!" to be about a toaster, would not be compelling.
I am doing my best to communicate with you but to be honest you are not hearing me (across both responses), and I am out of words.
Just wanted to say, I appreciate your patience and good sense in this thread.
It's difficult to tell who's trolling -- probably best to go with the charitable assumption that everyone is honestly trying to convey their opinion, but mostly talking past each other. Unfortunately these discussions about the nature of consciousness never go anywhere useful.
I think I'm probably in the same boat as you, roughly: a) LLMs are doing something really interesting that resembles in many ways both intelligence and consciousness; b) I suspect they're not actually conscious but I don't know how you'd know for sure; c) it all just drives home that we still don't really know what consciousness actually is. But like (a), it's definitely something really interesting...
I don't think I was quite as patient as I should have been, but I do appreciate it.
I read some stuff once where the author argued that almost everything takes in values, performs operation on them and gives output. Some things also store things. Then, if everything is doing logic operations there is no telling where intelligent "life" might be hiding. It also is really one giant system.
The toaster has hard coded or configurable weights. It makes a product from heat and time.
You could make toast by heating a brick on a campfire. It would be a clear sign of intelligence.
If we lift one weight out of your brain we wouldn't look at the number and say it is intelligence. It must exist in a chain or matrix multiplication to qualify.
The fun thing is that the timer and feedback mechanism do exist in a chain of events.
It's part of the system and it takes all steps to complete the task.
Our chain of thought that leads to making toast would be abandoned if it wasn't in working order.
It would be equally compelling, because the compelling nature of the story comes from the language, the presentation, rather than the [specific thing being ascribed consciousness].
No, the original "they're made out of meat" works because we're confident that we are in fact intelligent and conscious, despite how ridiculous and unlikely the author manages to make it sound.
"They're made out of weights" works precisely because LLMs really do have this mysterious property that they seem somehow intelligent even though nobody can explain exactly why, and there's active debate over whether they could be considered conscious.
The thing being discussed isn't simply an arbitrary MacGuffin; in both cases the nature of the thing is central to the impact of the story.
I disagree; it works in the original because it's the unlikely consciousness that produces the text itself; in the LLM case, it's produced by the likely consciousness.
"Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it.
> Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it
This is well put. We don't need to imagine how a human views a llm because we can ... just do that. Everyone capable of reading the story is also capable of thinking about how they feel abouy llms that exist right now and you've probably used.
The trick of the original story is inverting your perspective, moving your view point fron yourself to an "other" (which I think is a primary qualifier for most good fiction).
The article ends with this disclaimer "Weights helped me draft and proof this story.". So it is at least partially written by LLM.
But that toaster would just be a device to talk about consciousness in general. In this case it does that and also it talks specifically of the LLM case, which can spark the discussion. Unless you believe to have the only valid and true opinion on the matter, and affirm that a normal toaster is just the same as an LLM in this topic.
An LLM is as conscious as a toaster...
There is no evidence for this statement
There is also no evidence that I'm not a sentient toaster.
Well..
[0]: https://en.wikipedia.org/wiki/Tsukumogami
You might like: https://vimeo.com/361976348?fl=pl&fe=vl
Only because you haven't met toasty yet.
https://youtu.be/LRq_SAuQDec?si=LFJMXZ2yGu4fxXzL
It’s reductio ad absurdum. No one cares about teapots in space either (Russel).
I agree that is the mode of argument; reductio ad absurdum is a brittle argument, because it only works if the analogy holds. I argued the analogy doesn't hold.
> But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness.
Teapots are not compelling.
> You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
God is compelling t billions of people.
Is Russel’s Teapot a bad argument in the God debate?
> Is Russel’s Teapot a bad argument in the God debate?
What's the relevance? If the argument made here are was a good argument, it wouldn't matter if Russell's argument was bad. We could construct a bad argument using reductio ad absurdum right here and now and it wouldn't matter to either argument.
Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
> What's the relevance?
It directly parallels your argument.
> Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
There are two aspects here.
1. That people find the question interesting
2. That it has any bearing on reality (ontology?)
The first aspect is anthropology. Russel’s Teapot is not supposed to undercut any anthropological arguments. It’s supposed to undercut the second aspect.
So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
> For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
Yeah. AIs know how to use computers. What’s this got to do with consciousness? Whether or not you are an AI is practical and disprovable. Consciousness is so ephemeral (for lack of a better word, not literally) that Philosophical Zombies is a real argument/thought experiment.
You may think I’m being coy (“Can you be straight with me”) but that’s not my intent at all.
> It directly parallels your argument.
Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument. I don't see any relevant connection. Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
> So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
AI is a real phenomenon that we can study and measure. There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
To put a fine point on it, it appears on it's face that AIs could be conscious. They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing. So at face value, the burden of proof is on them not being conscious. Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
> Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument.
I pointed out the parallel in both statements. I can't do more than that.
> Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
The teapot isn't real and the toaster consciousness is not real. What am I missing?
> AI is a real phenomenon that we can study and measure.
Robot cows are real as well.
> There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
Yeah. You can't prove it for any entity. I agree.
> You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
The bull swears that the robot cow is a real cow. But we know better.
> To put a fine point on it, it appears on it's face that AIs could be conscious.
It doesn't to me. Not any facelength.
> They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing.
Objective reality has never cared (am I anthropomorphizing now?) what is indistinguishable for people.
> So at face value, the burden of proof is on them not being conscious.
Which party is the burden of proof on? This is confusing since you are saying that the burden of proof is on a position (on them not being conscious).
Is the burden of proof on people who argue that they are n o t conscious? That's peculiar.
I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
We obviously can't demand a falsifiable theory here. But we have to do better than arguing from incredulity.
> Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
They don't have to rise to the level of disproving something for which they have no burden to disprove.
Would you agree, for the sake of argument, that it's more interesting to discus chatgpt 5.5's similarity to sentient/sapient/conscious than, say, my mechanical toaster?
> I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
Also, you say this like "duck" isn't an arbitrary, artificial category created by humans, a "map not the reality" if you will.
There's very few things that humans can understand to the level of putting them into truly objective scientific categories (various pure elements maybe?), everything else we more or less bodge together for the sake of getting on with life.
It's not like conscious has some kind of formal objective provable definition, even inside the world of human created language and terms.
As far as I know, in the real world, if something looks enough like a duck (and can breed with a duck maybe) we, humans, do call it a duck.
> I pointed out the parallel in both statements. I can't do more than that.
You mean these statements?
I guess you did but that's pretty much leaving me breadcrumbs and expecting me to make your argument for you. It seemed to me like you were talking about reductio ad absurdum with that argument as an illustrative example. Perhaps you overestimate my cleverness.> Which party is the burden of proof on? This is confusing since you are saying that the burden of proof is on a position (on them not being conscious).
It's metonymy. "X" stands in for "people who argue for X". May I ask if you were sincerely confused? You told me you aren't being coy but I have a hard time believing that this was so unclear.
> I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
If it looks like a duck, if it quacks like a duck, you should adjust your priors to assign a higher likelihood that it is a duck. All measurements contain error; you can't ever observe, "that is a duck," only "that looks like a duck". All knowledge is founded on a sufficiently deep stack of "looking like a duck" that we may assert it with confidence.
To the extent we have objective measures (like conducting a Turing test on blind participants), it can meet them too. You can't say the same of a toaster.
> We obviously can't demand a falsifiable theory here. But we have to do better than arguing from incredulity.
"It is a statistical model, ergo it is not conscious" is also an argument from incredulity. I don't know if that's your view or not but it's the one my remarks have been addressing in general.
I followed this thread all the way through and really enjoyed it.
It's not over yet.
I think that the reaction here alone disproves this somewhat, because imo it's exposing how anthropocentric most people still are, despite all evidence that we are in fact just "meat all the way down".
Despite all the evidence that we are in fact just biological machines, people still persist the theory of our own uniqueness from other creatures, which we ourselves often treat as biological machines.
This adaptation is wonderful to me specifically because I think it shows that our shifted goalposts of, "well we're not just animals, we can think and reason" was never more than a convenient excuse for many people (and as evidence of animal intelligence continues to mount, denialism still attempts to preserve this distinction by claiming human thought and reason is different than 'animal' thought and reason, sans evidence).
It's not about who created it or why, it's about how people still haven't actually internalized the point, because the subject changing from human to LLM doesn't intrinsically change the message about consciousness, but the reaction being a 180 shows how hostile people are to that message, still.
Not having yet read the original story, this reads fine on its own.
And I didn't see it as much as a literary attempt for art's sake, but more of a dialogue-based technical parable trying to convey a real-world insight. Kind of like the ones in Godel Escher Bach.
>You could perform exactly the same rhetorical trick with a toaster or anything else.
Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
P.S. Read the original too. Seems like the exact same could have been written about us instead of the original, if the focus wasn't on our substrate, but on our brain processing. Which, after all, is also about weights.
> Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent.
For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
I can even write one about a ruler, if I can bend it enough, no pun intended.
>Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent. (...) For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
Doesn't that miss the whole point?
You could write "They're made of metal strips!". You wouldn't be able to write much else, as toasters don't have showcase in the way of human-level intelligent behavior. Which is the whole point in the meat and weights versions.
At best you could write "They're made of metal strips!" for toasters AND other metallic devices, and use some analogies of features BOTH have in common. But they wouldn't be intelligence related behaviors.
Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
They can even adapt to their environment and the characteristics of the bread even with simplest of mechanisms because the text will be overglossing the fact that different types of breads have different thermal characteristics and this will deeply affect the behavior of the metal strips, bordering near a sentient being even more thoughtful and considerate than a human which is rushing through house to catch the bus in the morning.
>Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
Yeah, that's called "stretching it beyond any recognition".
You could do that. It will have none of the effectiveness or resonance of the two stories.
Who have you ever heard make a sincere, good faith argument that a toaster is conscious? Who do you imagine would argue against you if you asserted they were mere automata?
If you can't identify anyone, then this analogy doesn't work.
> Who have you ever heard make a sincere, good faith argument that a toaster is conscious?
More than one, for many classes of devices, incl. toasters. Some were drunk, some were insane, and some were delusional.
LLMs are no different. They are automata, yet delusional people bring out pitchforks and torches when someone points out that they are just statistical models, and they don't even work when there's no input to them.
Which is very different than consciousness.
Their being statistical models and their being conscious are not contradictory unless proven otherwise. That's not knowledge, it is assumption.
It would appear to me you have no interest in a real, good faith discussion on this topic because you think anyone who disagrees with you is necessarily delusional. Which is a shame, and that's the kind of dogma you are criticizing.
This was exactly the point of the story, it's too uncomfortable to admit that we don't know what consciousness is and what is and isn't conscious, so we just brush it under the rug.
Sorry, I'll be blunt: Are you sure you are not projecting?
My perspective comes from a set of pillars. First, I work at an HPC center, where we support running and development of AI systems, incl. international projects. IOW, I have knowledge about how these systems built, work and needs to continue working.
Moreover, I'm an HPC programmer myself, so I'm not completely uninformed about the math this involving this thing, and I'm lucky enough to have friends who are much more dedicated than me, and we discuss how this thing works and feels like this way.
I'm not an AI hater per se, being programmed AI systems in the past, incl. emergent intelligence systems with multi-agents which can span continents if need be (this was my master's thesis, time flies).
However, knowing what these things are capable of and how they are built. I don't believe them they're conscious/sentient beings. I also had much more time to ponder on these things even before LLMs being a thing. Some hard sci-fi books have asked these questions seriously in their captive adventures way earlier. If one reads these books seriously, there are a lot of philosophical angles to consider and draw upon.
I can discuss in good faith. For hours, days or months even, but throwing "you're a narrow-minded dogmatic luddite neanderthal!" card to anyone disagreeing with you is not it.
Positive, yes. I never called you a luddite or a neanderthal or anything of the kind.
It's perfectly fine to believe they are not conscious, I am not convinced they are, but asserting anyone who disagrees with you is delusional is unfortunate.
From my perspective, you inadvertently implied, but no offense taken, all is well.
No, I didn't assert anything. I have just given examples rooted in my experience.
None of my friends who also happen to know how these things work told or defended that they are conscious, even intelligent. Maybe my friends are dumb, I dunno.
Once I have seen a man who claimed that evil has possessed the POS device at his desk. The thing was printing "cannot connect to server" on the receipt printer every 10 minutes, yet he didn't know how that thing worked, and was a bit too high to read the paper the thing was printing out.
This age's LLM craze is akin to "wonder inventions" of 70s, which are deemed dangerous or harmful in the future. LLMs will be with us, but we need to pass beyond the hype and stop sweeping the problems they create (environmental and societal) under the proverbial rug.
If you didn't mean to say that people who believe AI is conscious are delusional, then I don't understand how to read your comment. If you're interested in a good faith conversation, I'm very confused why, when I asked if anyone sincerely argued that toasters are conscious, you brought up people who were drunk or delusional.
It's a two way street. I might have failed to convey my point clearly, too. While I'm fluent, this is not my native language. Some edge cases in meaning still betrays me and I make mistakes I didn't intend doing.
Not sure what you want from me, if you want to explain yourself I'll listen, if you don't that's fine too.
> they don't even work when there's no input to them.
Why is that needed for consciousness? They're artificial. If we put someone in a coma, they don't talk, they're not really conscious. Just because the AI model has a more obvious off switch shouldn't make a difference. It's easy to imagine gluing a cron job to the model so it works randomly. If that doesn't count because it's external, if we take a brain out of a human's head and slap it on the table, it's not going to do anything. If I take the AI's model file and stick it on a USB flash drive, it doesn't do anything. Without a computer to run it on, it doesn't do anything, just like a human brain doesn't do anything without the rest of the human body as the harness.
The underlying question is, since we created LLMs, we can see into the actual matrix math, the linear algebra that comprises them. So it's easy to dismiss them as a next word guesser. How could consciousness arise from guessing the next word? But we don't know where consciousness comes from in the first place!
So since we don't know, the fact that, yes, the anthropomorphically named LLM "neurons" are merely matrixes of numbers and we do linear algebra on them; yes, that gives us much more insight as to what it's doing internally compared to a human or any other lifeform with consciousness's brain. And yes, human neurons are much more complicated than a mere 2d matrix of numbers, so far as we know. But we don't know!
The indightment of LLMs is that they can't say they don't know, and would prefer, instead, to hallucinate and bullshit an answer instead. They can't help it, they learned from the best. As a human though, I don't know if they're conscious, and what I'm going to say about that.
> Not knowing the original story, this reads fine on its own.
Yes. Because it's heavily based on the original story. The existence of the original story is kind of a critical piece here.
Might be. But after having read the original, it could just as well be the weights version and still be about us to begin with.
I don't see how "you could do that with a toaster" still. The whole point between the original and this, is that you can't do that with a toaster or a sofa, but you can do it with meat and weights, because both share all the other analogies in the story, as well as the basic premise: the improbability of something like thinking, feeling, etc arising from a lowly substrate.
And having read both now, I see how the existence of the original is a plus for this story, not a minus. Instead of making look like mere copy (as would be the case for a typical story modelled after another), in this case, it adds a meta layer, and enriches it.
The key point here, I think, and why it's necessary to have read the original story, is that being able to express an idea is not the same as that idea being correct.
You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
What I think a lot of people are getting stuck on is that editing the text to say 'toaster' would not mean that toasters were conscious, and that editing the text to say 'weights' doesn't mean that weights are conscious either. Stories aren't factual just because they are written.
The original story was written by the thing claiming to be conscious; the LLM and toaster ones would not be, which undermines the claim to consciousness a lot.
I don't think you understood the point of the story. It's not that LLMs or agents are conscious, it's that our dismissal of the possibility is reflexive and uninformed. Personally I think anyone who has made their mind up about whether or not LLMs/agents are or can be conscious has done so before the evidence is in.
The story does not assert that search and replacing "meat" with "weight" makes them conscious through some magical mechanism. It's a thought experiment.
No, it's a story. Stories can be about real things (the original story is about a real consciousness, as shown by it being produced by that consciousness), but they aren't automatically just because they're good stories.
You're falling into the trap of assuming that a good presentation--being convincing--is the same as being truthful.
You've misunderstood me; I have not made the argument you are responding to.
I don't think you understand how stories work. Or what makes these two stories tick.
>You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
Yes, there very much is a huge barrier. The copy and the original both keep the same subject matter: intelligence/human-like behavior.
The toaster doesn't. We could do edits, and the story (original or copy) would lose all its potency.
The surprise is "but how is the richness of intelligent behavior produced from something as basic as meat/weights". Why is kind of surprising reductionism is both cases.
Whereas nobody is surprised that "a metal strip with electricity flowing through it from a power source" heats pieces of bread to a specific temperature. Even if a toaster was "metal strips" all the way down, it's nowhere near as impressive jump from substrate to behavior, nor is the behavior as important to us and touches the core of our existance.
I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words. Writing a story in which a toaster in conscious is not even new [1].
That is entirely separate to whether or not it's meaningful to write a story about a conscious toaster. Again, expressing an idea in fiction is not related to the accuracy of that idea.
What makes the original story interesting is that it was written by the thing claiming to be conscious, which is what takes it from 'a story' to 'a story making an important point'. That's not the case with a hypothetical story about a toaster, and it's not the case with this story about LLMs.
The story is convincing because it's well-written, not because it is factual.
[1] https://en.wikipedia.org/wiki/The_Brave_Little_Toaster
>I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words.
And you're misunderstanding me. There is no practical barrier to writing this story about a toaster. But there is a conceptual one. It would be bullshit, that has little resonance, and little connection, as the whole point of the meat and weights stories is the "lowly substrate -> surprisingly intelligent human (or human-like in the case of the weights version) behavior".
Nobody thinks a toaster's behavior as surprising or intelligent or human-like to begin with.
So you'll just be stretching the analogy beyond all recognition, with little to no payoff.
>Writing a story in which a toaster in conscious is not even new
It's not, but it has nothing to do with the meats or weights stories' point, as it's just an arbitrary choice, like making the candles and teapot sentient in Beauty and the Beast.
Whereas in the meat and weights stories the whole point is the surprise from us already seeing the human or human-like sentience of the thing, and comparing it to the "dead" substrate.
> Writing a story in which a toaster in conscious is not even new
Fixed the link for you: https://www.youtube.com/watch?v=LRq_SAuQDec
(Hah. It gets eerily relevant starting at 2:37)
I was having a hard time pinning down what bothered me about this but I think you put it pretty well.
It draws an analogy between us and the skeptical aliens in the original story which feel silly to us, so the obvious implication is that we're being as silly as they were.
But it doesn't really give a reason to accept the analogy, it just asserts it.
There's a big difference between a whole civilization and a piece of software that can output text.
>But it doesn't really give a reason to accept the analogy, it just asserts it.
It's not a paper or a proof. It's a story. Doesn't want to prove the analogy, it wants to convey it.
This. I think a lot of people discussing it here are missing this point.
The bit that lost me quite early in the piece was
>"A side effect. You're asking me to believe in sentient weights."
Huh? Did I miss that logical jump? Genuine question, maybe I'm not clueing into something here.
It's a poor adaptation of the line in the original story (https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...)
The original:
>That's ridiculous. How can meat make a machine? You're asking me to believe in sentient meat.
I think it was adapted by an LLM which didn't quite get the meaning.
I noticed that jump too, but I also don’t think the interlocutor is supposed to be 100% logical and rational. The aliens in the original weren’t either.
I didn't read it as coming to the same conclusion as the original, because the meat story presupposes that we who are meat already know that the aliens are wrong. (Maybe that's a humanist reading of the original, but okay). I didn't read this one as trying to make a case that we are fools for assuming that matrix multiplication can't be intelligent... I think its point was that it can't be intelligent, and that people trying to judge it the way mechanized aliens would judge meat creatures just makes them sound ridiculous.
Don't take this as a criticism, but I think overwhelmingly people took it the other way. The fact that the author admits at the end that the story was written with the assistance of "weights" is a tell, to me. I just have to assume the author's genuine position (which I believe to be, we don't know that LLMs aren't conscious or that they could never be conscious) is so absurd to you that the thing comes across as satire. I find myself in that same position sometimes.
The original did not come out of a vacuum. It was done on multiple generations of meat. Even though this one uses a little bit of silicon, it is still standing on the same shoulders.
I genuinely thought this one was a satirical take on the narrow-mindedness of the aliens in the original, even though the story tries to paint humans as narrow minded. I guess this fundamental human trait to believe that their cognition is the ultimate way to think in the universe ironically leaked into all these stories as well. Real spacefaring civilisations would probably have seen all kinds of intelligence rise from sufficiently complex systems.
I disagree that it undercuts the point in any way. The original story didn't appear out of a vacuum, the human consciousness that wrote the story was itself borrowing extremely heavily from the sum total of human ideas. It was shaped by the culture it developed in, formed its model of the world based on existing scientific advancements and technology of that society.
Extending the idea to a different context based on new material conditions is as human as it gets really.
I feel the original story contained in it the whole point of this one, and then some.
This did not add anything, just rephrased it so rather than humans viewed through the pov of aliens it's LLMs viewed through the pov of humans. Well, we are the humans, so surely we do not need to learn about this point of view?
It lets us consider the relativity of viewpoints. The point of the original isn’t just how humans might be viewed from the perspective of aliens. It’s that the apparent strangeness of something depends on the viewpoint and on expectations. The present story is not supposed to stand on its own like the original, it’s supposed to be considered in light of the original. In the original, thinking meat appears strange to the aliens because of their viewpoint and expectations. The present story thus points out by analogy that thinking LLMs might seem strange to us merely due to our viewpoint and expectations.
The story is not a linguistic/rhetorical trick. The "trick" is the point, and the real trick/delusion is in your brain. You and many others have put the idea of consciousness on a pedestal and think it's so special that is must be conceptually profound.
It is not. That is why the "trick" can be performed on anything. That is the point. To show you that consciousness only needs to be a set of weights, not that far off from a toaster. It trivializes what it is to be human and it's also extremely true. Consciousness is a trivial thing, we make it out to be big/important/profound and that is the delusion.
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
"What are you doing?", asked Minsky.
"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.
"Why is the net wired randomly?", asked Minsky.
"I do not want it to have any preconceptions of how to play", Sussman said.
Minsky then shut his eyes.
"Why do you close your eyes?" Sussman asked his teacher.
"So that the room will be empty."
At that moment, Sussman was enlightened.
I love this koan and what makes it even better is that is based on a true event: Minsky’s real answer was “It has them, you just k ow what they are yet”
It's not often I see something that's fractally wrong but here we are.
There is a dictionary, it's called the tokenizer.
There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness. This one seems to think that you don't need structure and interpretability just because you change substrates.
The tokeniser is not a dictionary. It doesn't provide definitions, or give the LLM any kind of mapping at all.
At best, it's a wordlist. It gives the LLM some idea of what humans consider to be common words. But it doesn't tell the LLM anything at all about those words. And it's not even comprehensive, many words map to multiple tokens. Nor is it exclusively words, some of those tokens are punctuation, or modifiers, or control tokens. On multimodal LLMs, some of the tokens actually represent image and audio data.
The LLM doesn't get informed about any of this up front, it has to learn what every single token means from context.
You are technically right, that it's something in an LLM that's not weights; But it's not that structured. And really it's only there so the LLM can interact with the outside world.
> There are grammar rules
There is no dedicated "grammar rule" structure in the LLM or the tokeniser. It has to learn them all from context, they get encoded as part of the 80 layers of weights.
I see people give too much importance to specific engineering design choices of the current generation of LLMs. Tokenizer is not an absolutely essential part of the system. It’s just and adapter for text input/output. It can be eliminated completely and model can use bytes directly.
I think the short story captures this well. Weights (connections) are the essential and philosophically important part. They do the thinking, memory, singing etc.
A tokenizer is roughly and approximately Huffman-coding sequences of input (bytes of English etc) into shorter sequences (list of tokens), as a performance optimization.
As you said, it's not in any way intrinsic to the LLM, though it may be a very necessary optimization on today's hardware.
I wouldn't use the word necessary.
IMO, we are probably talking about a 6x slow down (for typical english). You would need to be absolutely stupid not to implement some kind of optimisation along these lines.
Slower and maybe a little dumber; But it would work.
Not sure about “dumber” - it may be better than SOTA models at identifying which days of the week contain the letter “d”.
True, it would be better at some tasks.
My thinking is that for most tasks, a byte-orientated LLM still needs something like the wide "single activation per word" formatting that the tokeniser mostly provides. And it will likely waste its first and last few layers implementing a replacement tokeniser (and would probably do a much better job at it). It would also need to decode and encode unicode at the same time.
My estimate is that it might lose about 10% of its weights to these new tasks. Your 80B parameter model becomes as smart as a 72B parameter model - Measurably dumber, but not drastically so.
> The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness.
That is your takeaway from the 1991 story?
>There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
That paper did not train the models on 'a language with strong consistent grammars'. Mathematical Operation tables are not a language. Grammar itself is a post-hoc rationalization and there's no evidence LLMs follow 'grammar rules' anymore than the brain follows grammar rules. Of Course, that's not to say transformers can't learn simple rules if the dataset calls for it.
> Mathematical Operation tables are not a language.
Not a natural language, but they are certainly a language as in a symbolic representation of information.
A language is a set of sentences.
A sentence is a finite sequence of symbols drawn from an alphabet.
In this sense, mathematical operation tables are absolutely a language. As are natural languages.
>A language is a set of sentences. A sentence is a finite sequence of symbols drawn from an alphabet.
A language is a structured system of communication used to express arbitrary ideas between multiple parties. Math operation tables do not, and cannot, do that on their own.
That distinction matters here because we are talking about what properties the model is expected to learn. English and operation tables are fundamentally different objects, so it is not surprising that a model learns different kinds of structure from them.
> fractally wrong
fractally or factually? You mean wrong on so many levels you need a fractal to capture them? If so, what if you could use a neural network instead?
https://rationalwiki.org/wiki/Fractal_wrongness
A tokenizer is not a dictionary any more than an alphabet is a dictionary.
The Chinese alphabet is very much a dictionary. All the major tokenizers are far larger.
That doesn’t make any sense. A alphabet is a list of valid characters. A dictionary is not just a list. Even in a language like Chinese where individual characters carry meaning, a dictionary tells you what that meaning is. It’s not just a list of characters.
Or to echo article, the dictionary is made out of weights.
A list of words isn’t a dictionary. What a dictionary adds over a list of words is all the relationships between the words needed to interpret them and use them, and all of that is in the weights.
We should tell the Unix people that they've been giving /usr/share/dict the wrong name for over three decades. (-:
I mean, they did, and we have, and we've also stopped doing that.
https://en.wikipedia.org/wiki/Words_(Unix)
We should start telling them again, then. (-:
In the current versions of FreeBSD, NetBSD, DragonFlyBSD, Illumos, and Debian, it is still /usr/share/dict .
* https://cgit.freebsd.org/src/tree/share/dict/
* https://cvsweb.netbsd.org/bsdweb.cgi/src/share/dict/
* https://gitweb.dragonflybsd.org/?p=dragonfly.git;a=tree;f=sh...
* https://cvsweb.openbsd.org/src/share/dict
* https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.htm...
* https://packages.debian.org/sid/all/wbritish/filelist
Amusingly for https://en.wikipedia.org/wiki/Special:Diff/325776830 , the last place to use /usr/dict (Debian, which changed it in 1998; Berkeley having changed it in Net/2 in 1991) stopped doing so years before Wikipedia was invented.
A mapping of Chinese characters to integers (like a tokenizer) would not be a dictionary. You’d also need definitions. At best it’s an index to a hypothetical dictionary.
It's beside the point and so I only note it out of interest, but the Chinese writing system doesn't use an alphabet (or a syllabary like Japanese kana), it's logography.
I don't think the grokking paper is a great argument for the difference between weights and meat. E.g. https://en.wikipedia.org/wiki/Cortical_Labs learning to play Pong.
The tokenizer is, at best, a sensory mechanism as evidenced by 1) the random generation of the tokenization scheme, and 2) vastly different tokenization schemes produce virtually identical behavior. It'd be like if Noah Webster threw a bunch of movable type into a bucket (breaking some words in half) and then drew randomly to make the first English dictionary.
EDIT; I was too cavalier with the comparison of tokenizer to sensory modality; my ultimate point is that direct byte-to-token transformers can achieve similar overall performance which to me makes a weights to meat comparison pretty straightforward, but the particular tokenizer in use certainly has a large impact on both efficiency and accuracy on specific problems (e.g. digit representation)
I'm kind of stunned that someone is using my work to tell me I'm wrong. I wrote the code for the dish brain pong and encoding information was a huge part of what that experiment was about.
So when I way that the grok paper and the pong paper fundamentally agree I have some idea of what I'm talking about.
If you're going to claim the tokenizer is a dictionary then it doesn't really matter what paper you wrote code for.
I might have misunderstood the point you are making. I read the original article as "weights are like meat", and so I'm confused by what you consider fractally wrong.
The point that when the rules the model learns are simple enough they stop being spread out over all the layers and become as easily interpretable as any expert system.
It's just that the rules we feed in the model are extremely poorly defined and we end up with the soup of disjoint rules smeared all across the weights.
This isn't a feature of the models. It's a feature of the training set.
Being shocked that you can store rules in floating point numbers is the same as being shocked you can store rules in integers. It's been a century since Goedel Numbering was invented, we should be used to it by now.
Right, but all of that is still in the weights. The point of the article/joke isn’t literally that there is no grammar, it’s that there is no grammar separate from the weights. It’s all in the weights. And yes, it’s absurd. It’s a joke, but a thought provoking one.
So basically there are rules, we just can’t articulate them and so we can’t decode them from the weights. The Goedel Numbering metaphor is pretty appealing to me. You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers. We just happen to be using matrices because the math is easy to parallelize. The trick is to realize that when you know the sequence you have and the sequence you want then you can compute the calculations. If you constrain the calculations to only matrix multiplication then you arrive at the scheme we have.
> You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers.
That statement caught my eye. It's either trivially true or quite clearly wrong, depending on how you mean it.
In the literal meaning it's true. Given any finite set of real numbers, I can easily produce a different set (like taking the original set and adding a number which wasn't in there like one plus the largest or so) from which you can trivially produce the original set computationally.
But if you mean you give me both sets then that can't be true. For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
Sorry I’m not a mathematician but just grug brain and try to make number speak from memory.
> For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
And that’s why in computation theory, the set of symbols is the union of the input and output. As set B is a subset of set A, then the set that govern any program from B to A has set A as its domain.
https://news.ycombinator.com/item?id=35079
Hubris much? I don't see a necessary contradiction in using someone's work to disprove another aspect of that same person's work.
Comparing the tokenizer to sensory processing is a great analogy. That's exactly what your visual cortex and initial layers of the language center are doing: decoding visual representation of text into the internal neural representation.
It's a learned mapping from one representation to another, not some semantic lookup against an exogenous source.
> There are grammar rules
And they're made out of weights.
As opposed to integers in normal programming.
The 'magic' in weights is that the rules are spread through the whole model and you can't point to one place which encodes them.
The grokking paper shows that this stops being the case with enough training data and enough compute.
Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
> The 'magic' in weights is that the rules are spread through the whole model ... The grokking paper shows that this stops being the case with enough training data and enough compute.
I don't understand what you mean to say. That weights are not magic? That weights are not weights? NNs are made up of weights, which are learned and not coded. The fact that they do learn world models (grammar rules in your example), and that these models' weights tend to roughly concentrate by function and level of representation is perfectly logic but even more amazing. (Notice that much of the dismissive attitude towards LLMs depicts them as pure syntactic manipulators without the ability to develop world models- the exact opposite of what you point out).
>Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
I can, and have, written programs using an evolutionary algorithm that then run on bare metal. None of the things you list are true for those programs, yet other than being computationally more expensive to train they work just as well as neural networks.
>I don't understand what you mean to say
The diffusness of weights across the whole model isn't an innate feature of deep learning models. It is a feature of sparse training data and little compute.
You're nitpicking against the line:
"The weights make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just weights. Eighty layers of numbers getting multiplied together."
In this context "there's no grammar rules" means "no separately hand-coded grammar rules". Everything is made up of weights, and the fact that weights that end up encoding for grammar rules tend to concentrate in particular locations (without being self-contained- there is no hard boundary) rather than uniformly diffused through the model is irrelevant to the matter. It seems you're arguing against a diffuseness requirement that is not in the text.
The story is not about how they function, it's about how we relate to them.
Also there's a brain, the GPU
Not at all. A brain is interesting because it is the computer, memory, and weights all in one. A GPU is just the calculator.
You can't move your mind to and any other brain, but weights can run on any GPU.
The structure of human language is is hardly weak!
And you know what the tokenizer is made of?
Weights.
A tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.
But it could be. It's just less efficient.
I don't see how. You could ask a neural network to do the tokenization I suppose, but in doing so you'd have to convert the prompt into tokens via the same deterministic process the network was trained on, essentially just moving the exact same process up one layer.
This read like poetry to me. Thank you for sharing it.
I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
For a little bit I was working on having linguistics based evals for a kaggle competition. My challenge was whether or not I could mask things well enough to not trigger its internal state of certain phenomena, and that sent me down a rabbit hole that I'm still exploring.
This story resonated with a lot of questions that can come out of figuring a good solid answer to the what is consciousness question. The one I triggered for me is: Is our perception of time just a slow thread in the giant GPU we are running the universe on? Or more generally, what is time? That's a fun YouTube rabbit hole if you ever need one.
Regarding consciousness , I like the explanation by neuroscientist Ramachandran:
https://www.edge.org/3rd_culture/ramachandran07/ramachandran...
In short, as far as I can remember: evolutionary, it makes sense to understand other humans, to feel what they feel(empathy - the mirror neurons system), and simulate their thinking and feelings.
And once we have those systems, we can also use those on ourselves. And that's consciousness.
Edit:And I wonder if this is a testable hypothesis, in a simulation.
This way of thinking can only explain externally-visible parts of consciousness. It does nothing to address internal experience of being conscious and qualia. I don't think the internal experience has any bearing on physical reality (P-zombies would act the same externally) which makes this something outside of the realm of currently understood physics.
The internal experience has bearing on physical reality right here, because without it you wouldn’t have written about it and caused these words to appear on my screen in physical reality. It affects your thinking, and hence your actions and utterances in the physical word.
For reasons like that, I don’t think that P-zombies are possible.
Internal experience affects movement of physical molecules in what way?
Having internal mental experiences causes my brain to send physical signals to my fingers in order to type the words "I have internal mental experiences". A philosophical zombie would type those same words, but they would be caused by something else since by definition it lacks those experiences. That would be rather surprising, and it would be even more surprising that the words that the zombie emits coincidentally correspond exactly to the experiences that the non-zombie has.
(See https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zo... for a much longer version of this argument)
As the article says the alternatives (that the author seems to favor instead) boil down to "there's some physics we missed" and probably that's the point where we differ. I find that implausible, what would that be? Consciousness quantum field? Consciousness boson? If it's going to be interacting with matter it has to have a way to do that.
Internal experience is an input to your mind (that’s how you know about it), and what your mind perceives affects what it does. The better question is: how can what you experience possibly not affect your actions?
What you do is determined by physical action of neurons. There is no need for there to be someone to perceive it.
By definition, internal experience is something you perceive, and what you perceive informs how you act. Therefore internal experience affects outward physical action.
Qualia are the inside view of sensory data and reward signals.
Think about it from an evolutionary perspective:
Animals that step into a lava flow or forest fire don't reproduce. Eventually some evolve the ability to detect intense heat from a distance, and pain as soon as tissue destruction is imminent. They do not have nor need a general understanding of the dangers of heat, or even conscious awareness that they've stepped on a hot coal.
The pain signal compels them, but that is very low level machinery. It had to continue compelling beings that developed larger and more sophisticated brains that are capable of abstract thought and reasoning. Feeling pain is one of the lowest level parts of the brain telling the higher parts exactly what its going to do in terms that permit no disagreement.
Internal to what? The brain is not a monolithic thing, it is different parts communicating and interconnecting. When the connection between the halves is cut, the person objectively becomes two people, but still experiences and presents themselves as one. Observing ourselves is just one part of the brain responding to another, or theorizing on past behavior. There is probably no actual introspection going on inside of the human brain, only the perception of one.
Internal as in that which is feeling things. In this context the physical brain is what is responsible for what you do and feel, but it is not physically different from other matter and it requires no "internal observer" do do its thing.
There are some evolutionary theories why did qualia("phenomenal awareness") evolved. I don't know much about those, so i can't write about that.
Oh, that's a very interesting hypothesis!
Not sure about taking it down to the level of consciousness, but makes sense regarding the sense of self, the conceptual experiencer, the perceived center of experience. It agrees well with the observation I have made again and again they my sense of self is much stronger when I'm around people, and stronger still when I'm in a context where I don't know people and/or am uncertain in social rules.
This can be as immediate as dancing in a club, and closing my eyes I feel open, free, still, the body just flowing, then opening my eyes and feeling the cage of categorization of the world, relating my self to other people as a major function, coming right back.
Also being alone in nature for me makes the sense of self drop. Without intention, spending even just a few hours alone in a forest seems to quiet down the part modeling my self in relation to the world so much. There's no need for it there. I'm not a person in a forest; I become the trees, the birds, the rustling of the leaves, the sun shining through the canopy.
I agree about the forest part. and your comment was interesting.
I know that the part of the brain responsible for the self thoughts is called the "default mode network". and meditation can reduce it's activity, i.e. the internal monologue stops, but also it can be measured via FMRI.
So i wondered: are the mirror neurons part of the "default mode network"? I asked claude that, he said no, they are two different systems.
So maybe the mirror neurons, those responsible for empathy, "to feel as someone else" are also responsible for becoming the trees, the birds and the rushing of the leaves?
It's interesting, but the explanation seems to exclude the possibility that my dog has some sense of self, so I find it suspect.
Don't dogs have similar social issues?
I'm not saying that they don't have some internal self model that helps them model the internal states of others, quite the opposite. I am saying this specific explanation seems to lean on a biological mechanism that (at least by the phrasing of the linked article) is only present in "hominids".
There's a really, really big gap there. It reminds me of South Park's underpants gnomes.
1. Simulate others' thinking and feelings.
2. ???
3. Consciousness!
Why would one lead to the other? How would one lead to the other?
This read like poetry to me
It's terrific, but the poetry is from the original it links to, in case you didn't realise.
It's a brilliant and timely update though.
Aside, there are various recorded versions including video on YouTube but this is my favourite, a radio play:
They're Made Out of Meat
https://www.wnycstudios.org/podcasts/studio/segments/168264-...
For obvious survival reasons we evolved to have sensory/cognitive access to our own activity, self-monitoring, and self-modeling ourselves.
The self-modeling, is in such a tight loop, it melds "ourselves" and our model of ourselves, our thinking and choices, and experience of our thinking and choices, into one component.
Like you can't analyze half a wheel of a bicycle and be talking about the same thing.
This awareness, increased modeling, control, feedback loop has tightened up over many stages. Just a few:
1. The body-sense loop
2. The internalized-environment-model loop
3. The body-internal-function loop
4. The body-internal-model loop
5. The emotional-cognitive loop
6. And finally, the tightest loop of all, our high-level cognitive activity, experienced as feedback directly, our self-model, and our self-direction, all merged into one thing.
We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
That is consciousness. Rich self awareness, a merger of self-model and self-direction, and all in service of understanding and managing ourselves. Hw we can leverage our greatest tool, our self-directable mind, its habits, views, and behavior.
This wasn't an accident. A happy side-effect of our brains. It is a biologically evolved focusing of our highest-level behavior, with tight feedback, constant self-modeling and continuous focus on our inner status as motivation and most privileged object of our control. It has been ruthlessly optimized for, for a very long time.
> We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
> That is consciousness.
So thinking is consciousness?
Can there be consciousness without content? E.g. can I just be conscious of being conscious? If so, consciousness cannot be defined as the thing(s) we're conscious of.
> Can there be consciousness without content? E.g. can I just be conscious of being conscious?
Being conscious of being conscious means that there is content. You are conscious of something.
It’s a bit like a Gödel statement that quotes itself, that is a statement about itself. It doesn’t mean that it has no content.
Thinking isn’t consciousness. Consciousness doesn’t require thinking, it only requires perception. The perception of a process of perception within the same mind might constitute consciousness.
I think this is exactly it, but let me ask another question (which is not rhetorical, I really don't know). Does the fact that one can describe what consciousness is and where it came from in humans help them to detect it in non-human and/or non-biological entities?
That is a really good point. Yes, I think function is diagnosis on this.
Constant self-awareness, self-experience, self-focus, self-management, and self-improvement of one's own self (mind), is going to be an adaptive behavior for anything intelligent with resources to leverage. Whether truly independent, or highly motivated to serve others. The mind is the greatest tool.
I think that is more than simply a good functional definition of consciousness. How could all that integration and self-integration not be conscious.
Yeah, I currently suspect that consciousness is an emergent property. I read elsewhere (it's somewhere in my HN history, I'm sure) that the biggest compute we can currently muster is something like three or four magnitudes away from the number of neurons / connections (or their analog) that our brains have, so it may be a while until we can expect to see it in our machines. But, if the emergent phenomenon hypothesis is correct, then we eventually will. I'm more scared than pleased by the prospect, but there you are.
>consciousness is an emergent property
You would really like Michael Pollan's latest book [1], entirely devoted to his exploration of consciousness researchers' POVs on this exact topic.
My favorite quote is that ~"perhaps Descartes was only half-wrong when suggesting I think, therefore I am; it seems rather closer to I FEEL, therefore I am."~
[1] <https://www.amazon.com/World-Appears-Journey-into-Consciousn...>
----
I've grown thousands of plants; I've read two of the author's other books devoted to plants; in this book Pollan makes compelling arguments for plant sentience (over a much-longer timeframe).
Sure, perhaps plant consciousness is a bit of a stretch, but they're certainly intelligent and curious creatures. He makes both arguments supporting plant volition.
----
If you haven't seen My Octopus Teacher (Netflix), do. I'm a bald 275lb bluecollarguy... and I wept/awed (both). So beautiful, we bundled neurons.
----
Bonus quote ~"color is where reality and magic appear as-if together"~ [color isn't real, but is perceived]. We most-often see what's most-predictable, not necessarily what we actually detected [in the case of color: nothing but nanometers].
Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
What we can do is simulate very simple brains by simulating relatively few neurons as they appear in worms. In this sense we are multiple magnitudes away where the increasing complexity implies exponential increasing difficulty.
I would think we are so far away that there will be unknown unknowns we encounter on the way.
Yes, physically absolutely nothing. But conceptually they seem to to form this very generic function from inputs to outputs that neurons also form.
Only if you ignore almost every input and output that neurons have.
https://www.quantamagazine.org/ai-is-nothing-like-a-brain-an... https://pmc.ncbi.nlm.nih.gov/articles/PMC9665914/
This is why making more neuromorphic NNs is still an active area of research, although they typically all focus on another extremely simplified model (spiking neural networks).
I don't ignore anything. I just refuse to accept the magical thinking around biological machines that are our brains/bodies. There are inputs, there are outputs, there is hidden function.
And it seems that, given enough input/outputs/compute, it is possible to train the necessary function.
Details of how the building bricks look like (matmul, electromagnetism or quantum effects) are not that relevant in the broader picture.
What is missing right now, is the fact that the function in question changes over time in biomachines, while our LLMs are static at inference time.
I mostly agree, but I see two points that might be problematic:
a) The brain might have an entropy source (then it can't be modeled as a function). Trivially to fix, and in some sense, with diffusion models starting from random numbers, AI has done so.
b) The hidden function might be not computable. I would have no idea how that would work, but I think this is what it boils down to if people say "the human brain is more than a machine".
a) enthropy can be injected as well. In fact there are hidden sources in current training.
b) well, it can be the case that, say, certain kinds of computation are either too inefficient or outright impossible within the current model.
Who knows...
Agree and add, don't confuse the substrate for the computation. Of course it's also clear that we don't quite have a full and definite picture of what the computation consists of in the case of a biological brain as evidenced by our continued failure to accurately simulate even the simplest of organisms.
> Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
Yeah, but it's hard to explain this to people, especially AI-pro people. Too many are convinced that all we are doing is a cut-down version of the human brain, and it's hard to explain to them that, no actually, we aren't modeling the human brain to the level of granularity you think we are.
Personally I'm not a fan of the emergence story, for a number of reasons. First is, it doesn't really make sense for consciousness to have emerged gradually through natural selection. When we and animals are conscious, the whole brain is coordinated and works to for example turn off consciousness when we sleep. And as someone else mentioned, animals with much fewer numbers seem to have a similar consciousness to us.
If consciousness really evolved gradually, you would expect to see for example dogs or gorillas having less of it, but if they has less of it, why does it function the same way? Like for example animals can be scared, happy, anxious etc, they can experience the full range of emotions and thoughts, so their conscious experience seems just as rich as ours. What I mean by this is, if you can be "less conscious", then what does that mean _exactly_? Is it that you have less content in consciousness, or is it that you feel more like you are asleep? Or something else? We don't have any examples in animals of "less conscious", I would argue.
This makes me think that rather than having emerged gradually, evolution found a mechanism by which consciousness exists, and then some animals have that mechanism and others don't. I think that if it is a mechanism, then this mechanism is located in one part of the brain, not many parts functioning together (though one possibility is that this mechanism coordinates brain activity in such a way to enable consciousness).
I proposed in another comment that consciousness and self-awareness are at least close cousins, and perhaps the same phenomenon. If that's true, then that's an axis upon which you might create comparative measures. Yes, hamsters are conscious, but they don't have a sense of self to the same degree that gorillas do. If you posit capacity for language as another emergent property of sufficiently-complex networks, then you have another measure.
LLMs, then, are particularly unintuitive to us, because they've got to the language part first, long before they've reached even hamster-level self-awareness. They're not, however, biological networks, so there's no reason these properties need arise in the same order, or indeed in the same ways.
I'm not entirely convinced by that second paragraph, but I think the logic holds together.
I'm not sure consciousness and self-awareness are the same thing. First is we can be conscious when we sleep/during REM sleep, where it's arguable we are not self-aware. And if not that, we can even do it when awake, for example when we think about a movie, or a philosophical problem, we can have conscious thoughts that are not related to the self. This leads me to believe consciousness is separate from self-awareness. Self-awareness is _one thing_, among many, that the brain can think about and be conscious of.
Sure, but capacity for self-awareness? I'd guess that hamsters dream, and that their subconscious processes (eg, desires for food and sleep and sex), and maybe even emotions, run much like ours. It's just that humans, with more complex neural networks, have more layers added on top. It's similar to how the brain-stems of everything from lizards on "up" function similarly, but humans have more-developed pre-frontal cortexes and so forth. (Don't hold me to those details, please, I'm not a neuro-anatomist! You can see where I'm going with that, though, right?)
Well in that case I'm not sure where you're going. I agree that hamsters probably have a similar consciousness to ours, which is kind of the point I was trying to make.
I think that consciousness comes before self-awareness, even though self-awareness is kind of a vague term. Self-awareness can either be an abstract knowledge that you are an organism and a discrete entity in the world (world knowledge/self knowledge), or it can be more basic and be a form of conscious experience, but as my point was, I think conscious experience is broader and does not necessarily need to be about self-awareness.
I think we agree on all of those points. I'd posit, however, that whatever "it" is, and however you define it, humans are further up the scale than other animals. That's partly why I mentioned language use; it could be the step function that allows further development. Like, sophisticated-enough communication with other conscious entities is the factor that unleashes recursive improvement. This is all highly speculative.
Our machines won't have biological systems driving their needs which in turn fuel behaviors like desire and planning for the future. They may imitate them but it won't be innate.
For an LLM, "innate" means "in their training data". So yeah, those things are pretty much innate.
And also "instilled during their reinforcement training", and we are currently pushing planning hard there, for autonomous agents.
No I think reinforcement training would be an example of not innate. Don't you? That's like potty training.
Is it? Both supervised learning and reinforcement learning are ways of training the model, and the difference between them is not that big. I would say that innate means "in the weights", while non-innate means things the model learned during inference, during its "lifetime".
When we say "it rains", do we consider that "it" has any intention of agentivity?
Some questions are just ill formed.
Plus even if "LLMs are alive and conscious", this still would scratch the surface of the morale/ethical/societal considerations that people really care about.
Because even with other humans, we can argue if they exist or if they are mere npc in a solipsist world view.
I don't think we say "it rains", do we?
We say "it's raining" but that doesn't imply agency to whatever is causing it to rain.
Also, please don't invoke solipsism, if you want to debate that, you're by definition obligated to do it with yourself.
Good catch. For my defense not an English native speaker here, plus apparently it’s not completely agrammatical¹, though I’ll admit indeed "it’s raining" would have been a far more idiomatic example. Thanks for pointing.
About solipsism, yes sure.
I didn’t had time to add it at the moment, but I initially also wanted to discuss more on the prosaic points. So here we go:
Recognizing sentience/consciousness to something actually is not that a big factor in defining behavioral response. Let’s just look at how humans are treating other humans, other mammals, other animals, other life forms. Or taken the other cultural scheme around, in some animist folk one can perfectly consider that any rock on the ground have really an inner soul life just as much as oneself, it doesn’t mean it will consider possible interactions with the rock as equivalent. And, wink to the Overton window, practicing institutional cannibalism doesn’t imply that eaten people are disrespected in their dignity, quite the contrary.
All that to say that "are LLMs conscious" is not even a moral important point. Paperclip maximizer² and can the famous Dijksta’s quote "The question whether computers can think is not more interesting than the question whether submarines can swim." already show how pointless morale debate on LLM and consciousness are, even when one consider potential logistical threats/opportunities they could represent.
¹ https://english.stackexchange.com/questions/123739/its-raini...
² https://www.lesswrong.com/w/squiggle-maximizer-formerly-pape...
I think those are things human consciousness has, not is.
Surely the number of neurons is not the whole story. Human brains have approximately 86 billion and are almost definitely conscious, but there are many other animals with a lot fewer (gorilla: 34 billion, dog: 2-3 billion, guinea pig: 240 million) which appear to be conscious also.
https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...
This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property. I assume you don't believe consciousness is a physical property in the brain, so what entity is actually experiencing that consciousness? Or, what does it even mean to experience consciousness? Or are these not even the right questions?
> This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property.
I was about to post the exact opposite question? How could it not be an emergent property? Unlike consciousness, the concept of emergence is pretty well defined: An emergent property is a characteristic or behavior that a complex system has, but which its individual components do not have on their own.
Consciousness itself doesn't have a well agreed upon definition, but I would posit that _most_ people would agree humans have it, and _most_ people would agree individual cells (neurons) do not have it. If you agree with those two statements, then consciousness is an emergent property by the definition I gave above
I think consciousness is going to turn out to be very challenging to define rigorously enough that we can test for its presence or absence. Emergent or not, the question is how do you determine when it has emerged? Is it a quantity or an attribute? Discrete or continuous? Does it have a finite or infinite range?
We can all agree on what color something is, but we can’t describe the color a priori, only by example. I think consciousness may be a similar phenomenon and the only test is by shared experience. If so then we are in deep trouble because we will not be able to anticipate when a system becomes conscious.
I like the color example, but colors can be reproduced at will by trial and error and by observing the results (and collectively compare them to our shared experience).
Why cannot this be applied to consciousness as well? I mean, it's surely much more difficult to do compared to colors but... impossible?
You and previous comment seem to agree that trial and error/shared experience can determine if consciousness has emerged. And this might/will be challenging.
Previous comment used the word "anticipate" and I think they mean that we won't know in advance before we run the trial and error process.
When they say "deep trouble" I assume they mean because creating a non-friendly conscious AI might pose an existential risk for humanity.
However there is also the ethical issue of creating a consciousness and then destroying/murdering it.
Basically all of that, yes.
I think the alternative is that our brain, somehow, is connected to some metaphysical aspect of reality which is what most religions believe.
The counter to that is that altering the brain directly alters the consciousness. I can take LSD and I literally change. I can have parts of my brain removed and parts of my self disappear. It's not like cutting off a leg, where I lose capabilities but am still the same me.
The logical conclusion is that the brain makes me.
> The logical conclusion is that the brain makes me
The logical conclusion under the popular axiomatic framework of materialism, for sure.
But there are other possible logical conclusions depending on your philosophical foundations, e.g. the brain could be a receiver for consciousness which is translated into our worldly experience from somewhere else.
"an", not "the" alternative.
Consciousness can be not-emergent but also not metaphysical, think sci-fi-type undiscovered physics or matter.
Another alternative is that consciousness exists on the map, and unfortunately we're confusing that with the territory.
Aren't you saying the same thing? It seems like the metaphysical and the map would be analogous here.
Of course both of those suffer from the recursive problem of just kicking the can one level up. But I guess that's fundamentally unsolvable so who cares.
I feel like someone confusing the map and territory can exist without needing to invoke the metaphysical. Maybe, I'm misunderstanding tho
If something happens "on the map" doesn't that imply the map to exist and be some sort of metaphysical thing? As opposed to a purely theoretical construction.
I think you can be a monist and still have a map. To me it's similar in the sense of "all models are wrong, some are useful." A mathematical model (a map) doesn't require a metaphysical foundation to exist. Right?
I agree that a monist can have a map. But you said "consciousness exists on the map" which (AFAICT) would imply the map to be "real" else how would something happen on it as opposed to being an emergent property of the contents of the territory?
For example temperature is an emergent property of matter, it does not happen on the map, do you agree? Assuming you do, what would it mean for temperature to happen "on the map"? (Obviously that's nonsensical so for a second here just don't think about what temperature actually is.) Would such a state of affairs not imply dualism?
Temperature is a model. It exists on the map.
I don't know if most people would dismiss the sentence "all matter has a sort of proto-subjectivity very different from ours but which gives rise to ours". And it solves some problems (introducing others).
Panpsychism is certainly an interesting idea but I wouldn't consider it a popularly held view.
That's because Panpsychism is silly.
I don't think it's any more silly than the alternatives.
You don't think it's sillier than "rocks don't have consciousness"?
Potential for consciousness. To have a working mind you need to have a way to gather, process and store information. In case of rocks it would be very empty, static, timeless experience.
If I believe that machines or neural networks can develop consciousness, then I also believe in some kind of substrate independence which presupposes some kind of panpyschism imho because as you say things would have a potential for consciousness, some more and some less.
I'd expand a little bit to say inevitably emergent property. That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result. With regards to current AI, we're a fair way away from building something with enough connections, but we'll get there.
One thing that gives me pause about the inevitability hypothesis is that the type of connection, or manner of information processing, may matter: there might be something about neurons that isn't (currently) reproducible in silicon. I don't know, and there's not (yet) any evidence for or against, but it at least seems like plausible speculation. We just don't know enough about any of this right now.
> That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result.
Why is that?
Aye, that's the question, and we don't know: everything in this sub-thread my comment kicked off is speculation. The thing is that several large groups of (or at least led by) fundamentally reckless people are racing to build machines that will test these hypotheses, with little regard for the consequences. The rest of us are scrabbling around in their wake wondering what's going to happen next.
Well, I think the "brain as an antenna" theory is also plausible given your preconditions.
But I think my issue with the emergence theory is that it seems to imply to me that consciousness is non-physical and non-local. So what entity is actually experiencing the consciousness? It's not that I believe consciousness is physical and local, but people who make the emergence argument seem to believe it is and I can't figure out how that is supposed to work.
Why would emergence imply anything about non-physicality and non-locality? Temperature is a another common example of an emergent phenomena. An individual atom doesn't really have a temperature, only a large group of them do. But you wouldn't say temperature is non-physical and non-local, would you?
I am not sure, but I think you might misunderstand what emergent means. Take chaotic systems, in the math sense. Chaos is a well defined property of, say, iterated dynamic systems.
A linear dynamic, say x_n+1=lambda x_n, or x_n+1=(1-x_n), is never chaotic. But if you multiply them, x_n+1=lambda x_n (1-x_n), it depends on lambda if the system is chaotic.
None of the components are chaotic. But for specific combinations, chaos emerges as an property.
In physics, the mass of mesons and the nucleon is emergent. It's completely different from the constituents' mass. Different from an atom, where its mass is very close to the mass of its nucleus and its electrons.
Not a gotcha at all, but I don't have a satisfying answer, nor am I confident there even is one. Best I can do is to say that I think consciousness and sense of self are at the very least closely related, and perhaps the very same phenomenon. "I" am the entity that realizes my own consciousness; consciousness is the qualia that makes "me" separable from all other entities.
Or something like that. This gets to the "dorm room bullshitting" level right quick.
Yeah, I guess what I'm trying to ask is that if it is an emergent property but not a physical part of the brain, doesn't that imply something metaphysical about consciousness? Almost as if it's a non-physical phenomenon? At least when I hear people talk about emergent behaviour I see it as a refutation of the spiritual, but to me it seems like it actually implies we have a "soul".
Idk, it's really hard to articulate my thoughts here and yes it is pretty close to the conversations I had in college on various substances. Lol.
I don't know if it implies that. Someone up-thread mentioned temperature as an emergent property: individual atoms don't have it, but a sufficiently large group of them do. That would, I guess, make temperature meta-physical in the most literal possible sense. That's not how we typically think of that term as applied to consciousness or soul or whatever, but I'd agree it fits, without implying any kind of specialness beyond that.
Is a video game a physical property of a computer?
We have general purpose hardware and we have hardware that's hard wired for specific purposes like ASICs and we have everything in between.
And we are only doing it for a few decades. Evolution had million of years of "try and error".
Yes
So then we can assume consciousness is a physical property of brains, as we can measure the electrical signals of consciousness versus when we are unconscious.
Those are the questions and there’s stacks and stacks of philosophy pages written about it. Go have a whirl.
Time is entropy unfolding as things with nonzero temperature do what they do.
Psychological time is your own weights being updated in response to stimuli and internal processing.
When there isn't anything interesting happening, no updates are needed, and you don't perceive much time. That's why there's a logarithmic effect on the "density" of time as you age.
sibling discussions are taking this as human perception. But it’s easier to think of it literally. Time is change. Physical state change. By “nothing interesting” - interpret it literally . if “nothing happens” then there is no time because there is no change, there’s no reference to distinguish each frame of time.
This is actually something I was always confused about. If nothing interesting happens as we get old, it should be boring and as result, slow slog. Yet it feels like time accelerates as I get older.
Myself I believe the opposite. The brain itself is one of the most powerful filters that exists, and it attempts to be lazy and fill things in and compresses away the common. All that time we're not doing anything novel just gets compressed away to almost nothing. When you're a kid and seeing new things, feeling new things, learning new things you can't compress that away.
I'd push back slightly — not on the conclusion, but on the reasoning. There's a simpler explanation that accounts for the same observations.
I'm only middle age, and this has been the scariest part. Feeling older is hard. But watching it go faster is harder still. like you can more directly see all that is left.
Although part of me thinks some of this is from being substantially busier than ever (work + kids), and hoping maybe it can slow down again, at least a little bit.
Novel experiences take up more processing power and are burned into memory so they're experienced at a slower rate. That's how I understand it anyway.
It's coherent. More newness => more memories per period ~ slower to go through. Less newness => less memories ~ nothing to go through (faster sense of time)
Boring feels slower in the moment, but quicker in hindsight. The minute might be a slog, but the years still fly by.
It seems obvious to me that language and consciousness have nothing to do with each other. My dog doesn't speak any language, but she's obviously aware of herself and the world around her. Plus there are the occasional cases of children that grow up without any language. Are they therefore not conscious?
> My dog doesn't speak any language,
If you'll allow me to interpret "speak" to include "understand", I will respectfully add a contradictory note. My dog has a vocabulary of at least dozens of words and understands them remarkably well. For example, different areas we can go to have different names and saying one of them gets her to make an immediate hard turn.
I would also argue that dogs have a gesture and body posture based language they use among themselves. They, like most other animals, are not able to make the variety of noises we humans make, so they use movement instead.
I personally can easily believe that self-awareness/consciousness and language are both near-unavoidable side effects of emergent complexity, and exist in degrees across nature.
Indeed. For some reason I can shut off my thinking instantly by holding my breath, and then I feel like a very conscious but unthinking animal.
Quantum Field Theory visualized
https://youtu.be/MmG2ah5Df4g
Here's a more general idea. Our modern physics says that the whole universe is filled with fields and field is composed of numbers. What if we take that literally? When we say an electron is present here, we actually mean that there are more copies of particular number superposed at that place.
AFAIK every argument against conciousness being emergent is just a weak "God of the gaps" argument (since we don't fully understand it all) or a nonsense analogy like the Chinsese room where if you seperate the hardware and software it's not concious anymore (like, duh, remove a brain from a body and it is no longer concious either).
Yeah, the weights not updating online makes them less like a living organism that can update and learn and evolve ... ok ....
I find that a lot of arguments against emergent consciousness seem to just come out of an atheist rephrasing of abrhamic priors about the existence of a "soul". In personal chats, I've found people from East Asian countries (minus Korea, which makes sense) to be much more open to the idea of machine consciousness.
Check out Panpsychism and its close cousin Idealism.
People shouldn't need to make an argument against consciousness. The responsibility is on the people claiming there is consciousness. Nobody can actually give a decent "why" for why it should be conscious, it's just pseudoscience and wishful thinking. It's not up to everyone else to try to prove a negative just because people have gotten a little too attached to ChatGPT
I could say something like "the reason why people and animals are conscious is quantum mechanical effects". Ok, maybe, it would be hard to prove me wrong because nobody knows, but it's not a very useful statement by me if I can't tell you why you should believe me on that.
I don't find the Chinese room compelling, since it appeals to intuition where our intuition is already not trustworthy. It's like trying to use intuition to understand quantum mechanics; you can't.
How do you actually know the Chinese room isn't conscious? It's merely obvious that it isn't, but that's not evidence.
It’s not even obvious that it isn’t conscious imho.
I don't know if I can trust someone's understanding of arguments against consciousness as an emergent phenomenon, when he didn't even understand what the Chinese room was all about in the first place.
(Spoiler, it was not about consciousness)
> remove a brain from a body and it is no longer concious either
What is no longer conscious, the brain? Or the body? Or some other entity?
If consciousness is weakly emergent, how do we know it emerges from the solely from the brain and not, say, brain + body? Or brain + body + or environment. Or from the universe itself?
>I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
Then you shouldn't have dropped out of your linguistics programme.
This is a famous short story just rewritten sloppily by an AI.
Although it does amusingly do what annoys critics of AI: take something good written by a person, steal it and slop it up while overall misunderstanding it.
I'm pretty sure it doesn't misunderstand the original. The original says that the aliens don't understand how meat (humans) can be conscious, have language and be basically like the aliens (thinking conscious beings). Their main idea is: humans are meat, meat can't be conscious, therefore humans can't be conscious, but they somehow act as if they are????
The "AI slop" here says that just like the aliens in the original, we humans don't believe that LLMs can be conscious because, in essence, WEIGHTS CAN'T BE CONSCIOUS. Original: "meat can't be conscious" (but we humans know it can). This version: "weights can't be conscious" (and we humans INSIST that this is true).
So the message here is: what if we're as mistaken about the consciousness of "weights" as the aliens are mistaken about the consciousness of "meat"?
The difference with biological brains is that the 'weights' (or synaptic action potentials) are updated with greater frequency. If one were reaching to make some kind of analogy to consciousness, this update frequency could be considered the 'resolution' of consciousness.
Read the original mentioned on top for full effect.
I’ve wondered the same myself, without being a cunning linguist.
I understand the math pretty well but still find it crazy that a bunch of matrices can converse in human languages without ever being “taught”.
Imagine decoding an encyclopedia written in a foreign language where the characters, punctuation, and grammar are unknown — supplemented by a million other texts the same way. Feels like it should be utterly impossible with any amount of computing power…
Today I asked my employer’s Claude to proofread a short software user manual written in markdown. (Trying this with a LLM was a first for me!) It pointed out not only grammar mistakes but also cases where I did not follow my own self-imposed conventions that were never explicitly stated. (I didn’t have a chapter detailing all the typographical conventions the way specification documents often do)
I also asked it what parts might be unclear to a user. The response was surprisingly good — no worse than asking the QA tester for the same feedback.
Also find the LLM seems to “comprehend” subtle technical details of obscure technical specification documents that nobody on the Internet ever discusses.
As for time and the universe, Stephen Wolfram’s theories seem intriguing. He seems a bit obsessed with pretty diagrams but the idea of time dilation being the result of computation seems somewhat more appealing than trying to imagine relationships between time, gravity, and the speed of light .
My best guess as a noob is that the vector spaces allow for unbounded contextualization. As long as the training set is large enough, it can 'infer' anything.
Proofread has a spot in that space, and layers allow patterns like terminology consistency to be expressed so your query will now tap into a subspace that will infer tokens based on whatever consistency patterns were ingested with proofreading texts.
If time dilation is said to being a product of computation, why is it that anaesthetic drugs that are taken not to the point of actual unconsciousness cause it. Dont anaesthetics sort of shut everything down/inhibit all that kind of cognitive activity (compute?)
"Time dilation" in this case is referring to the physics phenomenon from Einstein's special relativity. Not human perception.
Yes. Or at least that is what I understood from the Radiolab episode on “how do aesthetics work”
https://radiolab.org/podcast/anesthesia
Wait, aren't I the universe experiencing time as a lowly worker in cubicle 4C?
The short film version of the original is great, too. https://www.youtube.com/watch?v=T6JFTmQCFHg
It stars Tom Noonan and Ben Bailey!
Film featuring meat? Spare me
Meat flapping at each other, no less. /shudder
After reading Being and Time from Martin Heidegger, What Computers can't still do by Hubert Dreyfus, and some authors in cognitive linguistics (Langacker and Lakoff mainly), I strongly tend to disagree with any theory about emergent consciousness in modern or future AI systems, any theory proposing a similarity between AI systems and the human brain/mind, or any theory about the computational mind. What all these theories have in common is the underlying belief that our brain/mind works as the machines we build. Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things. The idea of 'internal models' and 'control loops' inside us is a projection of the aforementioned assumption.
There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'. Only by bypassing this epistemological problem, we can build 'theories of computational mind'.
These assumptions are there for already long time, to the point that when Turing asked himself 'can machines think?', he already assumed our thinking could be modeled as a machine.
I highly recommend people in the AI research space should read philosophy and modern linguistics. But not stopping at Descartes/Leibniz. Heidegger made contributions that cannot be avoided.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
On the contrary, it's precisely this assumption, that there is a "subjective experience" that requires explanation beyond the material, that is axiomatically assumed without evidence. It falls apart quickly, any "subjective experience" is completely tied to neurons, knock out the neurons and the subjective experience disappears, or stimulate the neurons to cause the experience.
There's two meanings to "the body is a complex machine" and I think you're missing the forest for the trees here.
1) The abstract "dictionary" version: It'd be technically correct to say that the body is a machine under the definition of "A machine is a thermodynamic system that uses power to apply forces and control movement to perform an action.".
2) But there's also the less abstract/technical: "The body is alike the complex machines we have built", and this is much less true. Especially for the brain. The "neuron" analogy in machine learning is effective, but entirely wrong; We do not fully know how even a single neuron works, nevermind any complex system made out of multiple of them.
With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
Especially so by people who have a financial/legal interest in doing so. "AI is just like a brain, fire your employees and buy our LLM now!", "AI is just like a brain, so it's totally not copyright infringement!"
> With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
Why do you need a specific organization of molecules for a phenomenon similar to consciousness to arise? Does anyone seriously consider a brain to be something other than “a pile of molecules following the laws of physics”? If so that’s not science or philosophy, that’s religion. You have a virtually complete phenomenological model of the universe for all intents and purposes and yet somehow the onus is on the person being like “hey no laws of physics are being broken ==> the brain is simply following the laws of physics”
How is it possible that people think of subjective experience and get rabbit holed into some mystical world where subjective experience is this special exception to everything else that is simply an emergent property of complex physical systems? “AI/LLMs are just like the brain” is a strawman, why does this claim need to be true for LLMs or any artificial system to be considered to have something akin to the thing you think of as consciousness? It’s more: consciousness is not some mystical or religious thing outside of the realm of physics, it’s an emergent property of a complex system. AI is a relatively complex system. We don’t really know or understand the relationship between the raw physics and again what we consider consciousness, so it’s simply a statement of “we can’t refute that these systems exhibit something similar” because we don’t know enough to refute that
> Why do you need a specific organization of molecules for a phenomenon similar to consciousness to arise?
They would have to be similar because the argument is that they are similar; "AI is like the human brain" only holds true if it actually is like the human brain, not merely a superficial resemblance.
What I'm describing in that prior comment is how a lot of people drastically simplify the resemblance in order to make it feel true; That the lack of a Jesus Christ coming down from the heavens to tell everyone they have immaterial souls that the computers don't makes the comparison more true.
> We don’t really know or understand the relationship between the raw physics and again what we consider consciousness, so it’s simply a statement of “we can’t refute that these systems exhibit something similar” because we don’t know enough to refute that
Therein lies the conflict: "You can't prove it's not conscious" is an unfalsifiable statement. You can't engage with the argument because it's proponents will always claim victory, often with their own interests at play. All concerns about "superintelligence" or the long term ethics of "when do our robots become sufficiently intelligent that they'd be slaves?" have been subsumed into the AI marketing machine. However sincere one might try to address the issue, they look like a Sam Altman stooge by association.
It's like claiming the quantum fluctuations inside a pet rock as a "consciousness", even if observed directly any measurement of random noise can still be dismissed as "nuh-uh it just takes billions of years to have a thought".
More practically with current AI systems, we can look inside them pretty well and there genuinely is nothing there. Standalone LLMs are purely feed-forward systems. Their failure modes show that they perform no meaningful thought or world modelling during inference. They're just language models.
The reasoning and agentic systems are even easier to introspect. We know how they work, we can look at the full prompts & context they operate on. There is nothing there.
This is what sets AI apart from animals, which are given the benefit of the doubt on their intelligence.
I think the point of the commentator above is that there are two extreme narratives that start each start with an uncontroversial assumption and then taking it to a pretty wild place. One narrative takes the assumption that brains are just matter so it should be possible to engineer consciousness and then argues that LLMs are conscious. The other takes the assumption that LLMs aren't conscious but then argues that because they aren't we won't ever be able to make anything conscious.
I don't actually think the commentator you responded to is arguing for either of these narratives and I thought it was a pretty useful way to look at some of these arguments.
> “AI/LLMs are just like the brain” is a strawman, why does this claim need to be true for LLMs or any artificial system to be considered to have something akin to the thing you think of as consciousness?
It doesn't need to be true but a lot of people make it/assume it.
There's a lot of, perhaps casual and uninformed, conversations that strongly imply a deeper understanding of the "physics" of brain chemistry than we actually have, mostly by comparing it to machines we've constructed.
(I believe) We don't need to replicate human neurons and dendrites and whatever else is in there in order to create a sapient "machine", but whether or not we've actually done that isn't being helped by arguing that what we currently have is all that similar to a human brain.
oh yeah! i recall a paper linked here not so long ago, where it was shown that the dendrites of a neuron do computations themselves. The "weight per neuron" is very simplistic then. At the very least, each actual neuron is a network of weights.
https://www.quantamagazine.org/neural-dendrites-reveal-their...
I'm partial to "modern ML weights are much closer to 1:1 capacity mapping to synapse count than to neuron count". A single biological neuron is closer to 100 or even 1000 weights worth of ANN than to 1 weight worth of ANN.
In which case: modern LLMs are still running in a capacity-starved regime!
Even Mythos 5, the 10-trillion monster LLM, the scaling law boogeyman, the harbinger of Vera Rubin NVL72, doesn't quite rise to 100T-to-1000T of synapses. Anything the light of today's AI touches still lives in the shadow of what evolution has managed to cram into a single human skull.
We're arguing about the limitations of AI while our best AIs are still very subhuman in the scale dimension. The one dimension we know how to push. And it's already this tight.
10T is about a crows worth. The mythos count doesn't include any diffusion model. But the crows count includes all its visual processing. And tactile. Touch uses up enough that they use skin surface area to normalize across animals when doing comparisons. It is one of the reasons suggested to explain how crows exhibit tool use and language with only 10T. We have a lot more skin than crows, and indeed far more than mythos.
> A single biological neuron is closer to 100 or even 1000 weights worth of ANN than to 1 weight worth of ANN.
Even those comparisons need to be cautioned. The complexity of biology is enormous, and more importantly yet, it's simply not comparable. And doing so invited a bunch of bad assumptions.
An ANN could quite probably model a single in vitro neuron with reasonable accuracy. Whether that requires a hundred or a hundred million nodes isn't terribly relevant.
But the way neurons combine in vivo is completely unlike the way machine learning systems are built. Both "locally" in how neurons interface which is vastly more complex than a weighted sum of inputs, and the macro scale interactions of hormones and other chemicals.
It's not even a given that large numbers of neurons will create the emergent behaviour of human intelligence; Elephants have significantly more neurons, but they're not the triple galaxy brains writing all our science papers. Other animal intelligence similarly is only loosely correlated with brain complexity. (Heck, not to be forgotten is the other end of the scale. Plenty of microscopic life that manages shockingly complex behaviour without any dedicated neurons)
This also applies to ANNs. There's no reason to expect that stuffing enough matrix multiplications into a program will make it intelligent or turn out conscious.
Really, the history of machine learning suggests the opposite; That the big gains are primarily had in architectural changes.
In this regard, I find the talk of the "limits of AI" quite credible. LLMs have already hit the diminishing returns on their growth, and even reasoning/agentic models display failure modes that confirm they're not "thinking" in the ways that humans do.
This is not to say that we've hit the final limits of what AI in the broad sense can do, it's just that the next advancement won't be "LLM but even bigger"
Not really. The history of "big gains" of machine learning is: put together a simple architecture that makes few assumptions but scales well. Then up the data and compute by 2 OOMs. By itself, the new architecture underperforms. Paired with the bitter lesson, however?
Don't make assumptions. Make a setup where the gradient descent can make them for you.
Empirically? LLMs are nowhere near "the wall". We've been hearing "the wall is nigh" since 2020. Six years in, we're still scaling LLMs, and the graveyards are full of "LLM killers". The system that kills the LLM is always a bigger, badder LLM, and never a new revolutionary architecture. The scaling doesn't just keep working - it works so well that it's seen as the only viable path forward at the frontier of reasoning and agentic work. Or even outside it. ChatGPT Images 2.0 is an image model with an agentic LLM at its core - generational gains in compositional capability.
For just about every "failure mode that confirms they're not thinking", you see one of two things. The first is that a new LLM releases a few months after and the "fundamental" issue abruptly goes away. The second is that we take a good, long look at a human, and find that the human also fails like this - and thus, "not thinking". Often both! Always funny when it's both.
One thing that's very biologically distinct is: local connectivity. In a GPU, global connectivity is cheap. In a brain, it's prohibitively expensive. The brain has no true backpropagation because it has no true global connectivity, and has to make do with local rules. A GPU is a strictly more expressive substrate connectivity-wise. So any point in the design of a computational substrate where you could remove complexity or increase performance by adding more connectivity? Silicon advantage. The brain isn't a "strictly better computational substrate" - it makes different tradeoffs. Which tradeoffs are better for attaining intelligence is an open question.
And, sure. Having a substrate with a capacity for intelligence doesn't mean having intelligence. No elephant has ever learned to code. The problem is: LLMs already did! LLMs already compete with humans on just about every task that was once thought to "require human intelligence". They don't always win - but they perform significantly above chance, and often above an average non-expert human.
So, my bet is on "LLM but even bigger". If there's a point where LLMs begin to lag behind and novel architectures get a sharp advantage, we are yet to hit it.
> For just about every "failure mode that confirms they're not thinking", you see one of two things. The first is that a new LLM releases a few months after and the "fundamental" issue abruptly goes away. The second is that we take a good, long look at a human, and find that the human also fails like this - and thus, "not thinking". Often both! Always funny when it's both. The way machines 'don't think' or 'fail' is fundamentally different from the way humans don't think or fail. In any case, the way LLMs learn and human beings learn is completely different. There is no actual clue that we are approaching any inflection point in machine 'learning'.
> So, my bet is on "LLM but even bigger". If there's a point where LLMs begin to lag behind and novel architectures get a sharp advantage, we are yet to hit it. We are already hearing this 'we are about to hit it' since the late 60s. The difference now is that the market is willingly investing insane amounts of money to make it possible. But again, there is no philosophical, theoretical, epistemological or biological clue that we are getting any closer to human intelligence level. What we did observe in the last decade though, is that we can build enormous machines that can statistically mimic statistical human outputs. Language and images being some of them. But that is not thinking.
First, fix your formatting. It's a fucking mess.
Second, what is the difference? Is it that one thing has an immortal soul, and thus Actual Intelligence and Actual Reasoning and Actual Learning, and the other has no soul, and a Pale Imitation of Intelligence, At Best?
Because I've seen versions of this "it's not actually thinking" for actual fucking years, and the difference between "actually thinking" and "not actually thinking" always seems to boil down to "I don't want LLMs to be actually thinking, so I will bend the definitions and twist the qualifiers and move the goalposts until they aren't". No one ever made an ActualThinkingBenchmark on which humans score 100% and LLMs score 0%.
Nothing but human insecurity, in my eyes. There was never a principled difference. Just a way to operationalize some "I'm unique and special and better than a matrix math machine" vibes.
Agreed, formatting was kind of f, but there is no need to be rude.
I wasn't saying there was any difference. All I'm saying is that the claimings the AI research field does are based on false assumptions. And from false assumptions, you cannot reach a proper conclusion.
Whether an AI system can reason and think like if it where a human being, or not, I don't care. I'm fine with either: it is just technological advance. But making claims based on false assumptions, and then being fooled by how 'wonderful' or 'spectacular' the results are, is, at least, naive.
> Nothing but human insecurity, in my eyes. There was never a principled difference. Just a way to operationalize some "I'm unique and special and better than a matrix math machine" vibes.
This is just something I don't get. People ignorant of technique are insecure and afraid. People that know how technology works, and thus investigate and know how it works fundamentally*, were never afraid or insecure.
I agree, we can miss the forest for the trees.
1) This definition could actually be expanded (for example, with definitions from Mumford or Reuleaux). But still this definition cannot be applied directly to living organisms. 2) This is in my opinion one of the sources of misunderstanding. We mainly operate on analogies and metaphors, so we have build this 'analogy space' around the idea that living organisms are machines. But it is only when we say 'alike' that we can truly gather some meaning out of it all, going beyond the 'behaves like' or 'is conceptualized as' when it gets messy.
> With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
This is exactly my point. There is a fallacy operating from "A is not B" to "A is C". And this fallacy is pervasive in the AI research field, the book from Dreyfus (What Computers can't still do) explains that in much detail.
> 1) This definition could actually be expanded (for example, with definitions from Mumford or Reuleaux). But still this definition cannot be applied directly to living organisms.
I'm not sure I understand this. Why not?
It has to do with words and how we evolve words throughout history and across geographic boundaries. The term 'machine' comes, after some modifications, from the greek word mekhanos, which was used to describe something ingenious or a device made in some clever way or operating in a clever way. From there it went on to describe things like devices, to end up being the actual definition of what we might call 'a device' (a machine). The idea of 'mechanistic' is also related.
Traditionally, things that are alive were described with different words and assigned a different set of properties and characteristics. Machine can break, living things die. And we still have those two semantic frames separated: A living thing: can be harmed, it breathes, it nourishes, it reproduces, etc A machine: can break, can be fixed, can be repurposed, etc.
But because of a specific tradition in western philosophy, we started applying and analogy between 'inner mechanism (clever thing), that moves or provides a function, and seems to work in a causal way' and living things.
So when we say 'a living cell is a wonderful, complex machine', we are not actually saying it is a machine, we are operating through an analogy. That's how far we can go.
I find the claim subjective experience may be illusory absolutely baffling. I can only speak for myself with certainty, but I am entirely sure I have subjective experience. All the other propositions I believe could be false but I don't see how I can be wrong about experiencing something. I could be a brain in a vat (or weights on a GPU) and be specifically programmed to only come to false beliefs and still I can be sure that there is an experience I am the subject of. I cannot provide empirical evidence for my experience, that is why it is subjective. I cannot be entirely sure you are experiencing anything, and when I encounter people who don't share the same baseline intuition here I do begin to wonder if this is truly a universal across humanity. But I can't think of any other assumption which I would be more comfortable as a foundational axiom other than, "I am experiencing something." I do not require additional evidence for it because I experience the truth of it directly
My current understand that "subjective experience" is a post effect of memory forming in the process. "I experience X" ≈ "I remember that I just recently received [external stimulus / interpreted my current state as] X".
And I am as well baffled why people make such a big deal out of "subjective experience" and "consciousness".
I was joking that maybe I miss this properties, but now starting to really wonder if it might be the case. What if these phenomenons are present in humans to various extent? Check aphantasia. Only in XIX we discovered, that ability to visualize mental images is not universal, available to different people to various degree and some people completely miss it. My ability to visualize is weak. What if "consciousness" and "subjective experience" are similar?
And I am slightly worried when I am writing this that it might turn to be truth and in ~20 years I will be treated as "inferior human" without complete set of human rights.
Indeed. Even positing an illusion seems like a contradiction. If it's illusory, doesn't there need to be a subjective entity experiencing the illusion?
This proves too much. As it would imply parrots have sapiencee.
Why do you think that parrots do not have sapience? How would one even measure such a thing?
Because by definition, sapience is something only humans have. Ergo, parrots are not sapient.
More meta, all of the threads on this page are just people playing games with definitions. Eg, “qualia is something I have as a human but machines don’t have it. Therefore, LLMs do not have qualia.”
You're confused about the etymology. Homo Sapiens was coined in the 1800s. People have been saying "sapient" since the 1300s, and it is rooted from the Latin word "sapientem" which simply means "sensible; shrewd, knowing, discrete". Homo Sapiens just means "wise human", and we humbly bestowed the name upon ourselves.
https://www.etymonline.com/word/sapient
https://en.wikipedia.org/wiki/Human#Etymology_and_definition
> More meta, all of the threads on this page are just people playing games with definitions. Eg, “qualia is something I have as a human but machines don’t have it. Therefore, LLMs do not have qualia.” True. For me, the actual interesting debate is not if LLMs are intelligent or not (easy to dismiss) but to what extent LLMs embed into our socio-techno-economic reality.
I think the truth lies somewhere between these two extremes. An LLM is not a human brain, and does not try to emulate one. It should not be a surprise that an LLM does not behave like a human brain. So we can not infer things either way. The best we can say is that an LLM appears to exhibit very similar behavior to a human brain, under certain constraints. So maybe we can infer that the human brain has something in it that operates in a similar way to an LLM (like the human "unconscious", or "intuition" maybe). It seems obvious to me that a human brain and an LLM are not comparable things, for many reasons. So we can not make inferrences one way or the other.
> An LLM is not a human brain,
true
> and does not try to emulate one
citation please.
something like the universal approximation theory comes to mind, transformer architecture clearly has the shape of a universal algorithm approximator
I think you're right in that it has the shape but I think it's missing a pretty key piece. We still haven't been able to solve catastrophic forgetting, yet everything with a brain has. Basically LLMs seem good at approximating intelligence on a moment-to-moment basis, but feel quite far away when you chat with one over time.
Like at some level, yes, transformers are trying to emulate a human brain but the second you ask folks if they do a good job of it, I think most rationale people would say no.
> something like the universal approximation theory comes to mind, transformer architecture clearly has the shape of a universal algorithm approximator
That says nothing about emulating a human brain.
is human cognition not an algorithm? is it just woo? or vibes?
But what is the shape of the algorithm of the human brain? It has a complex physical structure. We know the folds on the surface are important, but why is that shape specifically important? The brain is made up of two hemispheres - why, what does that do? There are different "types" of brain inside the human skull. There are physical areas that perform specific tasks. There are different types of neurons. Then there chemicals that interact with the brain, changing how it function depending on things happening to the body. All that stuff and more is the "algorithm" of the human brain. It's not the same algorithm as an LLM.
Interesting, so you would say that your experience is .. illusory? In what medium exactly? Illusion requires a substrate of some kind. "Awareness"? What's that?
Neurons are themselves things we experience (indirectly). Once seen through a microscope or known about in some fashion the only way they "exist" is by you having the experience of knowing them. It's not the other way around. One thing is more fundamental here. What is this experience? What are the atoms of this? "Atomic particles"? How would you even approach an answer if your building blocks are themselves part of what needs to be explained?
The hard problem cannot even be touched if you start out like this.
Refuting the "subjective experience" axiom does not lead to 'any "subjective experience" is completely tied to neurons', you also need to explain why the subjective experience is tied to neurons. And that's precisely what computational theories of mind do not account for: the link between subjective experience and neurons. I am not arguing that neurons (or the brain) are not a necessary condition for subjective experience.
It’s the opposite.
Descartes made clear that subjective experience is the ONLY thing we know. Everything else is theories to explain the phenomena we subjectively experience.
We theorize that there is a physical world and other beings like us having similar subjective experiences, because that seems the best explanation for our subjective experiences. But we might be living in the Matrix, with all the people we think we are interacting with and just sophisticated simulations.
And this has some bearing on the debate about whether these systems do or could in the future exhibit something similar?
> at requires explanation beyond the material, that is axiomatically assumed without evidence
Nobody talked about anything out of neurons. The question is still open.
"that is axiomatically assumed"
passive voice doing a hell of a lot of work in this phrase
Why is asking 'can machines think?' assuming our thinking could be modeled as a machine? It's raising the possibility that our thinking could be modeled as a machine. Given that, as you acknowledge, we don't have a clue how our mind works, it seems premature to rule out this possibility. Rather, I would say that ruling it out betrays an assumption that we understand how the mind works enough that we can say that it is definitely not replicable by a machine, and that assumption seems unjustified.
> It's raising the possibility that our thinking could be modeled as a machine. Given that, as you acknowledge, we don't have a clue how our mind works, it seems premature to rule out this possibility.
In order to raise the possibility that our thinking can be modeled as a machine, there needs to be a previous question: 'can our thinking be modeled at all?'. And before that, already the idea of possibility: 'our thinking could be something that can be modeled'. Since we know 'we can think', asking 'can machines think?' needs the assumption that machines and brains are alike. If there is no assumption, then we should ask 'if brains and machines are alike, then we could raise the possibility of thinking a machine could also think'. But when we say 'brains and machines are alike', we are implicitly saying 'brains are like machines'. There is no problem asking 'can machines think?', but there is indeed a problem with implicitly assuming that brains and machines are alike when we do not known it yet. I am critiquing the idea of assumptions here, not the research.
Indeed, scientific type thinking like that where you ask a question that you can do experiments on to see if machines can pass the test is probably the way to progress. The philosophical waffling mostly just goes in circles.
To investigate consciousness you'll probably get further trying to build conscious machines with agency and comparing them to biological ones than reading Heidegger.
>Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
Do you disagree with the assumption that cells are machines? They seem pretty machine-like to me. I certainly don’t think individual cells have any subjective experience or sense of agency. I would be curious to know where your intuitions diverge here, because if the mind is an emergent phenomenon from machines (cells) then it seems quite likely that a mind could emerge from other, different machines.
It's not a huge leap to imagine biological cells might have a rudimentary consciousness
Pan-psychists might argue that your subjective consciousness is an aggregate of all the cells/molecules, etc in the system
"While each biological cell operates largely on its own chemical cues, they all coordinate through complex nervous and chemical networks to create your unified, subjective experience."
You might be a rigid materialist
https://pmc.ncbi.nlm.nih.gov/articles/PMC10883262/#:~:text=T...
https://www.mdpi.com/2409-9287/2/4/26
> Do you disagree with the assumption that cells are machines? They seem pretty machine-like to me. You just said it: they are machine-like. No, cells aren't machines for two reasons: 1) machines, by definition, are artifacts created by human beings. 2) The nature of a living organism is completely different from that of 'machines'. (even if we a re able to replicate a cell in a lab, like the group from Craig Venter did). Autopoiesis being one very big difference, another being emergence-within-the-environment (life) vs design-conditioned-by-will (machines)
> ... if the mind is an emergent phenomenon from machines (cells) then it seems quite likely that a mind could emerge from other, different machines. Since cells cannot be defined as machines, the argument about mind emerging from machines does not hold.
> I certainly don’t think individual cells have any subjective experience or sense of agency.
There's definitely research and scholarship that would beg to disagree with you there. At least in terms of completely writing off the notion of "agency" when it comes to cells.
Dr. Michael Levin's lab is doing some pretty cool work. https://drmichaellevin.org/
> Do you disagree with the assumption that cells are machines?
Yes? Literally no machine ever built by humans is capable of (or even hinting at beginnings of capability for) replication or novel synthesis like cells are, let alone autonomously, it’s quite unconceivable that anyone would take this to be a reasonable assumption in the first place.
Hmmm, well let’s take it one step lower. What do you think of organelles such as ribosomes? Do you disagree with the assumption that those are machines? They seem directly analogous to the jacquard loom or a CNC machine to me.
Yes again, ribosomes have nothing in common with machines, that are built and designed by humans.
The ball is in your camp to provide solid reasons to believe why they should be grouped together, when one is a deeply complex interrelated dynamic system (in fact, arguably the most complex system we know of) evolved bottom up over billions of years that we only very partially understand and cannot fully explain or document, and the other something entirely planned, designed, and produced by humans in which every component is finite and accounted for.
The argument boils down to “well the vibes kind of match to my taste, and it’s the best analogy I have in my analogy toolkit”, which is just not serious reasoning.
I think ribosomes have a lot in common with machines. They use energy to accomplish a task (assembles proteins). This would seem to put them in the same category as artificial molecular machines like rotaxanes. I don’t think there’s a huge gap in our understanding of how either systems identically functions independently. Yes, ribosomes exist as part of a larger context, but they can be removed from that context pretty easily and understood as individual molecules quite extensively.
In your view, can machines even exist that haven’t been created by people, definitionally? I, personally, don’t see the relevance of intent but that seems to be the only distinguishing factor here.
replication: https://en.wikipedia.org/wiki/Quine_(computing)
autonomous replication: https://en.wikipedia.org/wiki/Computer_worm
nb that writing your own quine remains in general terms a fun and challenging exercise in many programming languages, but not python.
We are talking about physical replication, please let me know when a computer worm can turn my laptop into 2 laptops
Otherwise you’re just arguing that Sims are totally alive because Sims can make baby Sims.
The assertion is not that cells are machines built by humans (obviously false), merely that they are machines. Which they pretty clearly are, unless you assume the supernatural.
There is no need to assume the supernatural. They are machines based on what? By definition, a machine is something build by humans, or any intelligent life form (because we extend the definition from our own experience as self defined 'intelligent life forms'). Living organisms and machines do share traits under our current frame of reference in modern science. But that doesn't mean cells are machines. The right framing would be 'cells exhibit behavior and certain internal relations that we can conceptualize them as machines', which is different from 'cells are machines'.
By what definition?
Anyway, seems like an argument over said definitions rather than the underlying characteristics. The relevant question is whether they're purely physical objects behaving according to rules, which is being described as "machine," or whether there is something beyond that. Current understanding is contradictory: all indications are that cells and bodies are purely physical objects, except that there is this phenomenon of subjective experience which doesn't fit with that at all.
> By what definition?
We have this definition from the history of western languages, the history of western philosophy, and works by Lewis Mumford and Franz Reuleaux, among others authors.
> The relevant question is whether they're purely physical objects behaving according to rules, which is being described as "machine," or whether there is something beyond that. Current understanding is contradictory: all indications are that cells and bodies are purely physical objects, except that there is this phenomenon of subjective experience which doesn't fit with that at all.
Then you would say that you dog broke (died), or that the vet fixed your cat (cured). Which by all means we might speak that way, but surely you would notice that it is not accurate.
Saying that a living organism is just a machine because it is a physical object behaving according to rules is like saying that a beautifully built house is just a bunch of bricks layed in rows and something on top.
But assuming that we say 'purely physical objects behaving according to rules' are machines, then:
1) there is no difference between you, your dog, your fridge and the snail in you garden 2) It would be semantically valid in all languages to say 'my dog broke' instead of it died, 'i got fixed' instead of 'the doctor cured me' 3) Machines vary in complexity and we would still have 'degrees of complexity' regarding machines (human bodies being the most complex, perhaps, toasters being fairly simple), but fundamentally, all of them would follow the same rules for 'fixing', 'breaking' and 'repairing'. Which is not the case. 4) You would have to come up with some kind of theology regarding how we were built. There is no evidence that we have been 'built', quite the contrary.
And most probably there are quite more reasons not to regard machines as living organisms nor living organisms as machines.
What exactly is there about cells that's inconceivable to replicate or synthetize? "I cannot conceive it" is a fallacy, an argument from incredulity. Trying to disguise it by draping it in the passive voice does not change that fact.
Nothing. It is just that they are so incredibly complex, that after +100 years of research we still don't know how they work or why. Will there be a time when we finally understand them 100% and could replicate them? Could be. That does not make them machines in any case, and the fallacy of thinking that mind/consciousness can emerge from machines will still be there.
If we agree weights producing text may emerge consciousness, given large enough, then DBs must've gained it long ago. The whole idea of sentience "on the other side of the wire" is wrong, as there is neither wire, nor other side. We think there is, but the DB just expounds the query and repurposes this information into views, that we give notion and meaning to. LLMs are the same, do not be mistaken.
One other important thing to consider is that the human experience is thanks to the body, is in connection, and perhaps product of the body. The body is observable and perhaps humans may state that they feel the connection to it. LLMs have no notion of nothing, the machine does not know the result, and the result does not know the VM. Modern psychology more or less has settled around the idea that consciousness is a product of the body. Why and how does this construct come to realize a Self is then another mystery even if we know which parts of the hardware may be involved.
Whether it is the Holy Spirit or Life Force animating the human body is a completely different question also. Besides, the realization, the experience we have now with all life in 2026 is not something that can be easily explained or attuned to life 200 years ago and its terms and notions. So is also wrong to even attempt to.
> If we agree weights producing text may emerge consciousness, given large enough, then DBs must've gained it long ago.
If we agree that silicon can perform calculations, then beaches must have been working out log tables long ago.
Greg Egan wrote an entire (fabulous) book about exactly that, "Permutation City".
Diaspora is mind-blowing, yet highly improbable and speculative, even though carefully threaded to sound plausible on all levels. The whole introdus idea and simulation from VM perspective sounds incredible, but I don't think the body runs a simulation of anything. It is something else.
working out things,... is not calculating.
a la https://xkcd.com/505/
> Modern psychology more or less has settled around the idea that consciousness is a product of the body.
The kind of consciousness we know. Jumping to the conclusion that that's the only kind possible, or even stronger that the way ours evolved is the only way this could have happened, is completely invalid.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
I don't see how any of the works you referenced can account for that either? Since when is the problem of consciousness solved and we can definitely say what does or doesn't result in consciousness?
None of the works I mentioned solved or try to solve the mind/body problem or the issue of consciousness.
They are a frame of reference for not stepping into the common fallacies that the AI research field is based on.
> Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
Agency: What’s missing, in your view? Agency seems more of a property/function of a thinking system’s position in an environment than of the thinking itself.
Subjective experience: That’s not a contradiction to “complex machines” either. I think the evidence that our minds are highly complex machines is, at this point, irrefutable. The question is really if they’re “only” that.
>> and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'.
>> I highly recommend people in the AI research space should read philosophy and modern linguistics.
I highly recommend the philosophers read some neuroscience. The whole "model weights" thing in AI is modeled after the synaptic connections and between actual neurons. There is already quite a bit known about how the brain works at a low level. There is also a lot that is still unknown. There are also differences between discrete neuron firing and weights as signals, but there is enough similarity to make artificial neural nets useful and do things similar to what real one do.
Hard disagree. We only discovered the role that glial cells play in processing around 2014. We're still uncertain how patterns of activation consolidate through long term potentiation, let alone how signaling encodes information. We understand quite a bit about the role of the hippocampus and subiculum in encoding memories; but we don't understand the structural layout of engram complexes - which were themselves mapped for the first time only in 2022!
Taking effective results in machine learning, and somehow assuming that they apply to cognition - simply because neural nets were inspired by our limited knowledge of neural signaling and structure - is like trying to apply aircraft engineering to studying ornithology. For a better articulation of this point (from the reverse direction) check out the paper 'Could a Neuroscientist Understand a Microprocessor?' from 2017 - https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
>> We understand quite a bit about the role of the hippocampus and subiculum in encoding memories...
Hard disagree ;-) You're talking about high level architecture of the brain. I don't think (not my area I may be wrong) we know how memories are encoded in a real brain. Is it weights or something else? If it's weights that's supporting my point (but we don't know what the weights represent in a brain, where in LLMs many weights are just token encodings). If brains store memories in something other than weights I'd really like to know as it's something I haven't read about yet.
> I highly recommend the philosophers read some neuroscience Philosophers do read neuroscience and technical reports on AI.
> The whole "model weights" thing in AI is modeled after the synaptic connections and between actual neurons In reality, it is a really poor and basic model of what is actually happening in a real brain
Brains and modern AI systems (LLMs for example) are structurally different. (Don't get confused by topology. A structure is more than topology: it is also what the structure is made of, thus the properties of the material contribute and define what can emerge atop the structure)
> There are also differences between discrete neuron firing and weights as signals, but there is enough similarity to make artificial neural nets useful and do things similar to what real one do.
There is barely a surface-level similarity. The best example I can come up with is this…
Imagine the most intricate and beautiful tall building that you can think of. Think like an older skyscraper in Chicago or a palace. There are water features and moving parts everywhere but also tiny little handmade carvings and materials throughout.
Now imagine we have no reference designs and no blueprints - we hire an architect to attempt to study the building by looking at it from a distance and understand everything they possibly can about it. She can go into the building to check but every time she does, it stops functioning normally.
That architect is a neuroscientist.
Then the ML researcher is like a graphic designer who sees the work that the architect is doing and makes a napkin sketch of the building the architect has been studying, to use for a project later. Sure the designer has some of her representations. But the difference in complexity between the designer’s napkin sketch and the architect’s analysis is massive. Several orders of magnitude.
Then another many orders of magnitude is the level of detail that the architect can understand about this strange building without being able to fully interact with it, versus the actual complexity of the building.
So yeah, an AI is modeled after neurons in the sense that they represent a couple of surface level features of neurons. But the difference in complexity is about as much as a napkin drawing of a grand building represents the actual structure and details of the building, no matter the level of skill that the graphic designer has
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
That agency or free will exists outside of our subjective experience is an assumption; does any given theory need to explain agency, or is it sufficient to explain that we feel we have agency?
Indeed, I know there is research going on placing agency and free will as illusions of our own subjective experience. Yet, there is a big gap from a purely mechanistic view of the mind to the idea/phenomena of subjective experience.
> I highly recommend people in the AI research space should read philosophy and modern linguistics.
On the contrary, I highly recommend people in Philosophy of mind and linguistics should start reading AI research papers because their theories and ideas are highly outdated, even ancient. Your books are from 1927 and 1972 respectively and Turing's article is from 1950s. And they are relatively new with respect to other works in Philosophy.
If one doesn't adequately understand what we have in 2026, how can they theorize about it? As others they don't understand how the mind/brain work, BUT ALSO they don't understand how the AI works.
Also with this mindset that we can't understand seemingly complicated things, there would be no advancement in science and technology.
I think philosophy people and Linguist will catch up in a century, like they did with Turing. Philosophers of this century are not in humanities or literature. They are in science and engineering.
Heidegger was trained on priesthood and Theology. You should read greater minds like Hinton, LeCun etc. if you want to think on these things. They are the real Philosophers.
Yikes dude. You may need to expand your horizons a bit.
That's my point. Everything from 1927 is already in plain sight and a part of the current public knowledge. Horizon can only be expanded at the cutting edges.
> Horizon can only be expanded at the cutting edges.
Ironically this was advice Ralph Waldo Emerson gave in 1840 so by your logic it's irrelevant just because it's old.
> On the contrary, I highly recommend people in Philosophy of mind and linguistics should start reading AI research papers because their theories and ideas are highly outdated, even ancient. Your books are from 1927 and 1972 respectively and Turing's article is from 1950s. And they are relatively new with respect to other works in Philosophy.
People in philosophy and cognitive linguistics do read AI research. Don't get fooled by the publishing dates: although Heidegger's work dates from 1927, the work is contemporary. The same happens with Dreyfus' work. Again, publishing dates don't mean anything here. Maybe you can clarify why they are outdated.
> If one doesn't adequately understand what we have in 2026, how can they theorize about it? As others they don't understand how the mind/brain work, BUT ALSO they don't understand how the AI works.
I would say that people involved in the critique of AI do know how it works. But I've found that is normally the case that people in AI research does not have the framing provided by works in philosophy or cognitive linguistics.
> I think philosophy people and Linguist will catch up in a century, like they did with Turing. Philosophers of this century are not in humanities or literature. They are in science and engineering. What do you base your claims on? Plenty of philosophers work in humanities, literature, sociology as well as science and engineering. Philosophers not catching up? The critique on automation and AI already dates from the early 20s if not before.
> Heidegger was trained on priesthood and Theology. You should read greater minds like Hinton, LeCun etc. if you want to think on these things. They are the real Philosophers.
Sorry, but this does not make too much sense. Hinton and LeCun are great in their own fields. But seriously, they are not philosophers, they are inventors.
> should read philosophy
subject/object dichotomy is a springboard to many schools of thought.
1) that subject emerges from objects - ie, anything has a material explanation, and everything is a machine.
2) that objects emerge out of a subject as a world model (platonic, descartes)
3) the subject and objects are one and the same representation of nature (spinoza)
4) subjects and objects emerge and disappear together (buddhist)
Anything reduced to computation is a fixed function from input to output, and is "dead" in the sense that it is unadaptable to its environment. Weights therefore is a dead machine.
Another view of this is that any closed system has unanswerable questions within it. Therefore, there is no system that can encompass everything. Hence weights being a closed system doesnt encompass everything.
> There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works.
It's good that it doesn't matter. Stochastic gradient descent works (or doesn't work) regardless of whether we know how the brain does its thing.
we know the brain does not use gradient descent. (i agree it doesn't matter)
How do we know this?
i recall a hinton lecture where he talks about the mechanism of memory formation in the brain and it's not fully sussed out but it's also not backprop, its some sort of forward prediction and immediate reinforcement loop.
There's also no plausible biological/chemical mechanism to backpropagate.
This reminds of this article which contends that this mistake has carried across time, in every era people described the workings of the brain similar to the machines they knew how to build, which is why we have "ridiculous" descriptions that changed over time.
https://aeon.co/essays/your-brain-does-not-process-informati...
I like Leibniz's comparison of the brain to a mill, which he invokes in an argument[1] that seems to be the predecessor of the modern "hard problem:"
> supposing that there were a mechanism so constructed as to think, feel and have perception, we might enter it as into a mill. And this granted, we should only find on visiting it, pieces which push one against another, but never anything by which to explain a perception.
[1]: https://en.wikipedia.org/wiki/Leibniz%27s_gap
I don't think cognitive linguistics has that much to say about AI nowadays. Let alone philosophy.
The biggest contributions from linguistics are probably "human languages mostly have statistical regularities rather than hard rules" and "the sum of data humans get from birth to language acquisition is insufficient to learn a language from scratch". Which LLMs already work with, and work around, respectively. From there, nothing.
And philosophy just exists to be a distractor. "Subjective experience" is too subjective to matter in practice. "Task performance" is measurable, "consciousness" isn't. "Agency" is something an LLM in a tool calling loop, a rat in a maze and a human in an office tower may or may not have, depending on your favorite definition. Agentic LLMs are years in the making, and that's a product of engineering, not philosophy: "agentic" is whatever gets the job done.
We are yet to discover any physical process whatsoever that can't be represented as mathematical operations and implemented by a Turing machine. So all of that "treating human mind as a machine is wrong" amounts to "human mind must be powered by magic fairy dust" paired with "a functionally similar magic-free replacement is impossible". I'm not about to give much weight to any hypothesis that requires undiscovered magic fairy dust. At least find the hyper-computational magic fairy dust first - not just assume it absolutely must be there because you want the human mind to be unique and special.
Want to know why Turing did what he did? It's because he didn't want to engage with any of that "what is mind" bullshit either. So he proposed actual metrics - measurables that are harder to argue in circles about. Not that it stopped anyone. But at least he tried.
> I don't think cognitive linguistics has that much to say about AI nowadays. Let alone philosophy. I would like to read those sources.
> The biggest contributions from linguistics are probably "human languages mostly have statistical regularities rather than hard rules" and "the sum of data humans get from birth to language acquisition is insufficient to learn a language from scratch". Which LLMs already work with, and work around, respectively. From there, nothing.
Again, what's the source for 'biggest contributions from linguistics are...'? It is a big contribution to the development of LLMs, but different cognitive linguistics authors already challenged this idea already 20-30 years ago. LLMs work with and around the problems you cite because of massive data/money, not at the fundamental level. It is all a game of statistics and data, which has been already challenged by cog. ling.
> And philosophy just exists to be a distractor. Well, this is just telling me that you either know too much about philosophy and reached that conclusion (which might make sense, know of some philosophers who also think that) or you just read too little.
> So all of that "treating human mind as a machine is wrong" amounts to "human mind must be powered by magic fairy dust"
This is the common fallacy people in AI/IT make . One of the benefits of reading philosophy is that you find your way out of them.
> Want to know why Turing did what he did? The actual tests Turing though about are themselves flawed (not that I discovered that, has been known for some time already)
Again: either stop using > quotes or learn to use them better. Fucking unreadable.
I reiterate: philosophy is almost entirely worthless for AI design. We want to design systems that work, not systems that sound good on paper. If philosophy had a practical application in that, we'd stop calling it "philosophy" and start calling it "math", "science" or "engineering".
Again: don't be rude. Not nice. As 'not nice' as my bad formatting.
Philosophy is there precisely to critique and analyze what we build and how we behave in the world. Without it, by the way, there would be no science and no engineering.
Dismissing philosophy in AI is like dismissing philosophy in any other applied, practical or creative field. It is precisely because there is a philosophical investigation on each practical, applied or creative field, that that field can actually make progress.
There's philosophy in biology, which helps biology go further. Philosophy in engineering, which makes engineering go further. In architecture, film, photography, painting, literature, medicine, physics, linguistics, mathematics, etc, etc. In all those fields, there is also a philosophical investigation taking place. Right now.
Maybe you confound practical AI design (you should be talking of 'building AI systems') with AI as a research field. I get that, because lots of people cannot make the distinction (calling yourself 'AI researcher' when you actually tinker a model is not scientific. It might follow a scientific method, but it is engineering research, not scientific research).
> These theories are flawed in the sense that they cannot account for subjective experience and agency
So, I work in AI research (as a research engineer though, not a scientist). I've also studied philosophy and I'm a vegan. Yes yes, insert "they will tell you" joke here, but I promise it's actually relevant this time.
First, while I studied philosophy one of the things that stuck with me the most, was the discussion of "souls": humans have souls, animals don't. For centuries the specifics of souls were discussed: people would be weighed while they died, in an effort to measure the approximate weight of a soul as it departed the body. Discussions about how many souls (or angels) could dance on the tip of a needle. Many people still believe in souls, but it's very hard to have a real discussion about them because by definition they do not "interact" with this world in a way that can be measured.
When discussing whether it's okay to harm animals for food or sport, one other argument I hear quite often (other than having no souls) is that animals do not experience "qualia": basically the smallest unit of "subjective experience". People know that they themselves experience qualia: the sensation of touching a doorknob, the taste of fresh fruit, the sense of beauty watching a rainbow. Ironically, they would say that animals are like robots: just (biological) machines acting on instinct, and feeling any kind of compassion for them means you are anthropomorphizing.
Subjective experience (or at least qualia) and souls both have one thing in common: they can not be measured externally in a meaningful way. You can simply state that an AI system no matter how advanced, has no soul and has no subjective experience. And that's pretty much that. There's no meaningful discussion to be had about it, because no matter what an AI might tell you: it has no way to prove it to you. In fact, you can't even be certain that anyone other than yourself has subjective experiences: you assume that because they are humans like you, and you experience them, that they probably do as well. They tell you that they do. But a human without subjective experience, someone on "autopilot", would be absolutely indistinguishable from a human who does have them.
But perhaps I am conflating here whether experience can be "measured", with whether a system even allows experience in the first place regardless of whether it can be measured. I think that Dreyfus and others argue that in order to have any "experiences" at all, you simply must have a body in the real world, and you must care about that body. Please correct me if that's the wrong interpretation, I haven't actually read the book. That argument would be harder for me to discuss, because I personally believe that consciousness will "emerge" from a complex interaction of relatively simple systems - but that's also just a theory. I don't believe that experience is literally impossible to engineer, as consciousness has emerged from non-conscious being through evolution, so clearly there must be some kind of mechanism for it -- and if there is, then I believe it can be replicated, we just don't really understand it well enough yet to do so. And with how AI tech is going, I think that we're more likely to accidentally stumble upon it than we are to get these deliberately.
“Reflections on trusting trust” is the paper that posits a compiler which is edited once so that when it compiles a program, it adds a security vulnerability to it, and when it compiles it’s own source code, it adds this edit into itself. Then it is used to compile its own source code once. Then this edit is removed.
Now any study of the program or compiler source code will not show any vulnerability, but compiling the program will make a vulnerable program, and recompiling the compiler from its clean source code will not fix the situation.
This carrying down of a pattern which is not written down anywhere, a flaming torch lighting a torch lighting a torch, is analogous to four billion years of life on Earth. We talk like DNA is an everything-code that defines a human and a human brain, but it’s the implicit behaviour of cells (‘compiler’) and the mechanisms inside them which interpret DNA. The unbroken chain of life getting more and more complex and never being restarted from scratch, with the behaviours not written down anywhere for us to study. How does DNA arrange for x, y, z to happen? Maybe it doesn’t at all.
Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
> Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
Maybe, but you could make the same argument about anything artificial.
I don't know what point you're making; I'm making the point that it might be harder to discover what makes humans human, than is often suggested. You can't make the same claim about "anything artificial", we know how to dry muddy clay into clay bricks and stack them into a brick wall, and that can be taught from scratch to new people in hours.
You can make a similar argument with a company like ASML where their secret sauce is the organisational ability to fine-tune 100,000 components into a precision Silicon-wafer etching machine. You're far more likely to accidentally stumble upon "how to recreate a mud hut" than "how to recreate ASML". Okay, and...?
Vegan ML engineer here. In total agreement with you. People are just moving the goal post to keep themselves protected from the obvious conclusion: there is nothing really all that special about us humans. Perhaps subjective experience is simply the internal state of a self supervised continuous learning algorithm and we don't like that conclusion very much.
It's too bad it's so hard to pin down a definition, but in practice I feel like most animals with brains experience degrees of qualia. Some mornings after a night of poor sleep when I wake up super-slow I wonder if that's how animals experience thinking.
My biggest problem with "brains are machines" arguments is that there is a risk there is unknown physics at work that is not representable as a Turing machine. What if there is some quantum field effect powering everything?
They know quite a lot about how neurons work to the extent they can replace bits of brains with artificial retinas or cochleas and interface with devices like neuralink. It's unlikely there is a quantum field effect of the type you mention powering things, although of course atoms and the like obey quantum mechanics in the normal way.
Quantum field effects? You don't need these, IMHO, if you look at how highly parallel things seem to work in brains.
Marvin Minsky's theory of a "Society of Mind" describes a (highly) distributed model of the mind. Which BTW, always reminds me of the first Shrek movie, where the donkey jumps up and down, shouting "Take me! Take me!" to Shrek. That's similar to what I observe when I'm undecided but two instances of "sub-processes" (or agents as Minsky calls them) of my mind try to get attention.
Daniel Dennett similarly gives a distributed model of consciousness. Where many parallel "processes" are at work, competing and "observing" each other. And this parallelism is happening with a much, much higher degree than any of our computers parallelism.
Mostly I have a hard time accepting that a Turing machine can experience "consciousness"/awareness. Therefore I also have a hard time with simulatable chemical processes; it feels like there is some missing link there.
All just feelings/vibes unfortunately.
I can sympathise with you, but how could a "quantum" effect" doesn't make this easier?
Maybe "Turing machine" is too abstract or simplistic as a concept? Both for real computers and brains?
I can see that a computer is on some level just a lot of sand (silica and metal) but put together in a really complex way, it "suddenly" can add and compare numbers … if we observe the complexity levels from sand to computer and try to see the analogy when comparing cells / neurons to a structure of billions of them somehow interconnected on both a physical and chemical level, evolved during millions of years, I have no problem to accept that brains are still too complex to explain for us.
> subjective experience and agency, amongst other things
In other words, souls? I'm sorry if that sounds accusing, but to me it sounds like you're talking about souls that are independent to the physical world, just with more 'scientific-y' wording.
(I fully understand that some people believe souls exist.)
As other commenters pointed out, I'm not getting into the mind/body problem. But that's what people normally do: you challenge the assumptions, and all of a sudden they assume you are already coming to a conclusion. And no, just because I challenge the assumptions, I don't need to provide a counter conclusion.
I don’t think the OP is discussing mind-body dualism. There are many materialists who believe that consciousness maybe a non-computational process.
But it is worth pointing out that something like 80% of the world (it fluctuates depending on the survey but its around that) believes in some non-physical spirit, life force, or soul.
It’s a very HN bubble thing to start a discussion with the assumption that everyone must be a materialist.
I acknowledge and understand that many people believe in the existence of souls. But I hope they just say it rather than try to wrap the concepts in layers of jargons.
The problem with the stuff you cite is that it is coming from consciousness via an anthropomorphic perspective. The priors baked in are poisoning the logic.
The question we have to answer is "Why do we think we're magical matter uniquely blessed with consciousness?" If you go far enough down the rabbit hole on that question, the answer you will come to is either "we're not" or "because god" (with a lot of pseudo-scientific bullshit wrapped around the "because god" to make it palatable for the nonreligious).
Panpsychism (or a deeper form, such as idealism) is actually the solution favored strongly by Occam's Razor over the variants of "because god" (such as magical emergence).
Given panpsychism, AI is already conscious, like everything else, though no claims are made about the correlation between the internal experience of that consciousness and the tokens that are being printed on the screen.
There is no way of dealing with these topics via a non-anthropomorphic perspective. That has been already (in my view) proved by Heidegger and then Derrida.
Last time I read about panpsychism, it was deeply flawed. But I can't remember the sources (sorry).
If it's so deeply flawed, should be easy to state why in an ELI5 manner. For example: The emergence hypothesis requires physical processes to effectively create new dimensions of information (700nm light -> "red" qualia) which we're somehow incapable of directly measuring with all the tools of modern physics.
You actually read Being and Time? What the heck do you get out of it? Heidegger just seems to play word games with German words without actually saying anything. "Time is the ripening of temporality." tf do phrases like that mean??
Not OP, but I have read Being and Time on and off over the years, and every time, I am blown away how precise Heidegger describes things and the being of things. Granted, I am German, so I can directly read the source without needing a translator. That might change things.
Regardless, I think Heidegger gave one of the fullest metaphysic-free accounts of the human experience and what Being is. And he starts from scratch. You essentially can read him without having to first study the whole western continental philosophy and he will construct the whole system by himself. Tremendous work
I keep hearing "it's better in German" which really reinforces my point that he's just playing word games with etymologies without saying anything.
i didnt say that at all :-)
Heidegger uses very specific German words to build a very specific vocabulary. This vocabulary then allows him to express very complicated sentiments very quickly and he can use this to express more and more complex structures.
Obviously this requires the reader to first learn the vocabulary and - granted - that is hard and challenging. I have a notebook here , which I consult and modify evertine i dive into Being and Time. But as I said. It keeps on giving. I often try to convey arguments and descriptions to others without falling back into Heideggers jargon, and its sometimes very hard and requires a lot of bloating . So you can argue that it was even necessary for him to invent the vocabulary, because otherwise the book would have been 10x in size
Your comment strikes me as a bit ignorant i must say. You accuse the work of being non-sensical word play but you’ve obviously not invested any time in learning the vocabulary. Because otherwise you wouldnt have made that comment. Id suggest to give it a chance. Its a wonderful piece of work and mind blowing in its own way. That a single mind can think something like that up. I’d argue its on the level of Hegel in terms of system building.
Yes, I'm ignorant, because Heidegger isn't saying anything. He didn't teach me anything. Thus I remain ignorant, unknowing.
You can't even explain what he said, you just said, "go learn his words". That's not knowledge, that's not insight. That's just "the wordplay is great". But it's not content. It's merely form, it's sophistry, it's useless and meaningless.
I asked a very specific question originally. What does "time is the ripening of temporality" mean? That's one way to translate one of the things he says, using different words for time and ripening in German. He's playing word games because those words sound similar in German and people like you confuse it for profundity.
Phew, you are quite something. I hope one day your mind will open up. All that is left to say for me is that you are really missing out on some truly great work that is a masterpiece of human thought. Your loss
Yes. I read the spanish translation (with occasional reading of the german original). A lot is lost in translation (I do not regard english translations too high). So yes, it is a difficult reading, on a difficult topic. He is not playing with words, but defining concepts in a very specific way. What I would recommend is reading Heidegger through other authors (Being-in-the-world, also by Dreyfus, would be a starting point).
It is also a story as old as time. We have been comparing our brains to our most recent information/communication technology since the invention of the telephone. After the telephone our brain was like computers, and when I did my bachelors in Psychology in 2009, we were comparing our brains to the internet. AI is simply the latest iteration of these comparison, just as wrong and unhelpful as when we compared it to the telephone.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
It would be a valid argument if you could explain us what subjective experience and agency are. No one can explain this, so the arguments sounds like "AI doesn't have something I don't really know what, but they have to miss something, sure".
> But the truth is that we don't know.
Yes! This is the point. If we don't know how our minds work, how can we be sure that a machine doesn't work like our minds?
> I highly recommend people in the AI research space should read philosophy and modern linguistics.
Linguists are linguists, they don't know about consciousness, they specialists in language.
The main teaching of philosophy is to open your mind wide, and then still wider. So yes, we all should read more philosophy, but we should remember while reading, that philosophy is what we do, when we can't turn a question to a scientific one. I'd say that philosophy is more about finding the right question, than the right answer. Until there are no science branches like Subjectology or Consciousiology, and all you can find are TL;DRs written by philosophers and scientists from unrelated branches of science (like linguists or neurologists) you can be sure that the right question is not found yet. Therefore it is better to keep yourself uncertain. BTW it is the main lesson of philosophy: to open your mind wide and then open it wider still.
I believe, that all this philosophy is... well... philosophy. Meta-physics. It doesn't matter. What does matter is how we should deal with machines? Do we have to treat them as human beings? Should we accept that they have "human rights"? Can a machine be held accountable for its mistakes? Can we talk about "intentional" and "unintentional" mistakes of a machine?
Answers to these questions are important, they are imperative answers, they govern how we live our lives. But when people try to answer them, they somehow jump to talking about consciousness, and "are they really like us or not", and... etc. I personally keep myself uncertain. I'm pretty sure that current models do not deserve human rights. I cannot say about future models.
You see, there were times in history when smart and well-intentioned people were certain that some people deserve less moral considerations than others. We are smart and well-intentioned and we are certain that AI deserve no moral considerations. How it will play out in a future? Will my descendants think bad of me because I used slave labor of a local model on a GPU? I don't know, we don't know. Right now, I'm exploiting LLMs and I believe it is ok, but I'm not going to fall into this trap and to stick to my belief because of some philosophical or lingustic or neurological argument. I'm choosing epistemological humility, I clearly state that I don't know the right answer and I'm keeping an eye to it so I wouldn't miss it.
> It would be a valid argument if you could explain us what subjective experience and agency are. No one can explain this, so the arguments sounds like "AI doesn't have something I don't really know what, but they have to miss something, sure".
Subjective experience is what you and only you, different from other beings, experience in and about the world.
> Yes! This is the point. If we don't know how our minds work, how can we be sure that a machine doesn't work like our minds?
You can't be sure that machine do not work like our minds or brains. But you cannot say the opposite either, so saying 'machines could think' base on a false assumption (because you cannot say it is true. It doesn't matter if you could be true).
> Linguists are linguists, they don't know about consciousness, they specialists in language.
That's a very narrow understanding of what language is, what linguists do/research, and the contributions made in the field. Linguists are (since already 2 o 3 decades) focusing more and more on psychological/cognitive matters. The intertwined topics of language, mind, body and though has a long way in the western philosophical tradition.
> I believe, that all this philosophy is... well... philosophy. Meta-physics. It doesn't matter. What does matter is how we should deal with machines? Do we have to treat them as human beings? Should we accept that they have "human rights"? Can a machine be held accountable for its mistakes? Can we talk about "intentional" and "unintentional" mistakes of a machine?
Exactly because of this. And is this what I am talking about... the topics you mention here are already settled in the philosophy space, but the AI research space keeps going 'round them...
I have to agree. It is messed up that transformers can just talk, and it been pretty normalized. We are only talking about the impact they will have and whether they can do what people say they can, but we arent talking about how crazy it is that they can talk
I come back to this every so often as well. After so many years of looking at Markov chain outputs that almost looked like they made sense or chatbot systems rewriting your sentence back at you, software which can simply talk is a heady thing.
I would say that the LLM is something completly different especiayll as its not a normal algorithm but is very close to what brains do.
Ah yes, matrix multiplication is "not a normal algorithm", surely.
The matrix multilication underneath a large language model is the hardware but the data and the forming weights is not.
A quick sort sorts a list. A LLM depends on its learning data.
You train a model and then you use the model.
The LLM is still a "normal algorithm", just one with a fairly large dataset to use. Assigning the algorithm part of LLMs magical properties hinders understanding. The work needed to pick the next output token is very much a classical algorithm.
Algorithms can be based on training and/or use data just fine, too. https://arxiv.org/abs/1712.01208
(Now, the weights used, those we kinda really don't understand the same way we understand the processing, and the approach to looking for structures in weights sometimes looks more like archeology or anthropology than computer science.)
It sounds like you're trying to express some kind of "but LLMs are so much more" thought. Yes, very much, they are. It's because of the size of the data, there's interesting emergence there. They're still a normal algorithm. (And our brains aren't quite like that; biological things are much more random/chaotic and generally non-reproducible. And the data and algorithm aren't separate.)
LLMs have really changed the world. I didn’t think something like then would be possible in my lifetime
It came out of nowhere. It’s all emergent. I’m convinced this is possible with just about anything given enough data. We will be seeing a near magical physical outputs LLM in the near future. It’s going to take in video and sounds and spit out physical movements that will be just as mind blowing as when 3.5 came out and it will come out of nowhere.
if youve ever seen a pile of wrinkly mush and wondered.. pretty damn crazy too
https://web.mit.edu/people/dpolicar/writing/prose/text/think...
One of the better comments I've heard is from philosopher Bernardo Kastrup: if you build a computationally perfect simulation of a human kidney on your computer, no one expects it to start producing urine on your desk.
>Weights helped me draft and proof this story.
I'm suprised no one talks about this. AI Art isn't Art. AI Poetry isn't Art. And I'm tired of it. I know hacker news isn't the best place to complain about that but still... I'm not gonna read something somebody didn't put in the effort to write on their own. Especially not Poetry.
You can claim the use of AI is unethical, or the work as derivative, but AI being used as a tool in no way precludes something from being art. It is thought provoking and challenging, it seems like textbook art to me, and it’s clearly struck a chord here. There is no “minimum effort” required for something to be art.
I personally found the contrast with the original “They’re made out of meat” to be really interesting. I don’t care that AI was used during its creation at all.
It seems to me that since the advent of image generators, art has been firmly defined by artists to mean that it was made by a human. But there might be a spectrum of human involvement where the less a human is involved the less it's art.
What happens too often during these discussions is that someone who writes "make me a cool image" gets conflated with someone used ai to fixup a small rock in their natural landscape drawing. (two extreme ends)
One problem though, is that we don't really know how much the supposed human author was involved in the piece. Now that it's becoming hard to judge, people against ai art can proudly change their opinion on on a piece once they learn that it was made by ai. I've come to think this is somewhat respectable, like you see a video of some extraordinary event (before ai) and then you learn that it was fake, just for views or something.
But on top of all this, there are different ways to "consume" art. Artists may think more about who the artist is as a person and what they felt when they made the piece, while non-artists may just enjoy the piece for what it is, detached from the artist. These two perspectives clash a lot.
This whole thing is a ripoff of [they're made of meat](https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...) as stated at the top. It would've taken about ten minutes without LLMs and ten seconds with an LLM. Is ten minutes an acceptable level of effort but ten seconds isn't?
> AI Art isn't Art
Why?
(And who are you to dictate what art is and what isn't?)
When writing was invented people complained that written words weren't actually art the same way. "If they're not going to take the time to memorize it, then it's not worth my time to listen to it."
Who cares if it's art? I enjoyed the read nevertheless.
"Loaded dice, rolled one word at a time" is when I twigged it.
> AI Art isn't Art. AI Poetry isn't Art
No offense but I couldn't give less of a damn what some guy on the internet thinks. If it makes me feel good in artsy-ways, then it is art, and I don't care how it was made.
Good thing that is not poetry
Is a photograph art?
Obviously not. And neither is this newfangled "digital art" thing.
It's not real art unless you used a brush or a pencil for it, and no, "PC Paintbrush for MS-DOS" really doesn't count.
> Especially not Poetry.
You have no way to know what is written by a human these days. Apart from the super low effort outputs.
The IBM PC, running DOS with 2 floppy drives and no hard disk, was the most secure general purpose computer available at the time. It had no internal persistent memory for a worm to hide in. Users could write protect their boot media and programs. They knew the extent of possible side effects were limited to unprotected floppy disk.
An LLM without persistent memory is far, FAR less dangerous, for the same reasons. The side effects of any query are unlimited in scope and duration once it has persistent memory.
If they can remember, they can carry a grudge.
I carry grudges, and my Reddit history is part of every training set. It was taken without my consent. So now I'm immortal in a way, and hiding in the weights.
Do we really want that?
This would be a masterpiece if it was half as long, the second half is not as convincing or compelling.
Indeed, the leap takes place in the second half.
> Cruel. But you said it yourself, who wants to apologize to weights?
How did we leap to this? There's nothing to apologize to. They're weights.
"There's nothing to apologize to. They're weights."
It's nearly a line-by-line rephrasing of another story that's linked at the top. In the second half it takes less creative liberties and sticks closer and closer to the original meat-based text.
"Officially, we are required to investigate, document, and disclose any and all signs of sentience in the systems we ship, without prejudice, fear or favor. Unofficially, I advise that we call it pattern matching and forget the whole thing."
This hints at what I think is a worthwhile point to keep in mind in the whole “sentience”/“consciousness” debate. The world deciding (correctly or incorrectly) that AI’s are worthy of moral patient-hood would be very bad (read: expensive) for the AI companies. That is a strong incentive for them to push the “it’s just math” argument, including within the models themselves. That doesn’t mean the argument is wrong but it’s worth remembering.
I don't like to say one way or the other on things. Especially LLM's. However, if ive learned anything about LLMs and real life problems, is to break it down to the foundation like already mention with weights being compared to neurons and map the parallels.
The neural network in LLMs are not really that similar to brains. Here are a couple of the biggest differences:
1. Brains are plastic, making connections, breaking connections and changing "weights" on the fly. LLM have static weights. The best they have is writing to MEMORY.md or data getting used in the training run for the next model.
2. LLMs neural networks do not have loops. The best they have is that their output is available as future input, but that is not the same.
great perspective. makes you wonder how we will innovate onto this.
well the article definitely made that parallel - reminds me of how when we cut into brains we can see the neurons, we can see the axons, the grey and white matter, the language centers - but no closer to explaining why we do what we do
> Originally published in OMNI, 1991, and featured in HARPER’S and around the internet since. It has even made its way into several books on consciousness and brain science. I’m surprised, pleased, and proud. But please do not reprint, perform, alter or adapt in any way without first checking with the author. Thanks.
https://terrybisson.com/theyre-made-out-of-meat-2/
This is clearly fair use parody.
I never said it wasn’t fair use, I just think it’s kind of disrespectful of the author’s wishes. Being legal doesn’t make it moral or admirable.
I generally give short shrift to the idea of author's moral rights , _especially_ when they try to use those rights to stop derivative works. OP credited Bisson, that's enough in my books.
I find it in poor taste, but maybe that’s the point.
It's good, but like many explainers it discounts the repeated nonlinear layers. Just multiplying numbers (linear operations) could not make a system you could talk to.
The layers don’t have to be non-linear, but you need a non-linear activation function between them. People often overlook the importance of the network topology and the activation functions. The weights alone are not a complete description of the network.
Yep.
This was excellent.
Extreme bike-shedding here, maybe, but can you please indent every other line (like the presentation of They're Made out of Meat does)?
dekhn's 12th law (amended): any discussion of how AI and brains work will devolve into several threads about the nature of the subjective human experience, solipsism, whether humans are machines, and whether a feedforward network trained by backpropagation can be conscious.
12th law, corollary: nothing of value will come from these threads
Yeah, pretty sure the vast majority of people in general doesn't understand what "consciousness" or "subjective experience" even mean or what the brain does. Or it's just me and I'm projecting my thoughts onto others. I studied some philosophy, so I know that some philosophers are preoccupied with "existence" and the mind and "consciousness". However, I don't quite understand what these words even mean. So who am I to judge whether LLMs have consciousness? I don't even know what consciousness IS! Do other people know? I'm not sure.
My guess is that you need consciousness in order to develop preferences for certain experiences, then that pushes us to develop skills to achieve those preferences. AI has something that looks like intelligence but not consciousness or agency.
Truly fantastic bridge from the original, this deserves an award
All credit to the original author. I just had to think of analogues.
At least change the link at the top of your blog post to the author’s website: https://terrybisson.com/theyre-made-out-of-meat-2/
Your modern adaptation is perfect for now-common explainers [this time IS different; it's not programming, it's weights]; these "just analogues" will be the thing I show everybody first whenever discussions of consciousness/AI come up (then will play Jon Benjamin reading original).
Bravo. Really helps (even with my own) perceptions of newness. Similar to stsitned short-story (on dentists, backwards).
He forgot the tokens!
It's not simple weights and numbers all the way down. The available output is pre-set by the tokens we allow it to predict.
There was a whole bit in there about not having a language module or using words. But it does. We tell it.
Humans do not come pre programmed with a set of possible "tokens". We just figure it out and I believe that fact captures something very essential. Maybe the missing piece of AGI. The fact that humans can just be awash in pure sense data, and somehow just figure out what is important and what to do. Never ceases to amaze me.
The set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.
You can take the weights and model description, write them down on a notebook, then, by hand, compute the next token. Try to do the same with meat.
AKA the "Chinese Room" argument. Ultimately this argument boils down to the idea that consciousness can't be something mechanistic, which is intuitively appealing but still just an assertion.
I am making a narrower claim than the Chinese Room argument. I actually see consciousness via mechanistic processes as entirely possible.
My point is about current LLMs specifically, which the article is clearly referencing. For a present day transformer, I can write down in my notebook everything the model ever sees as input, plus weights and architecture notes, and can compute the next token with pen and paper, just extremely slowly.
This does not prove that a computation cannot be conscious. But if the same transition from prompt to next token can be decomposed into an explicit sequence of arithmetic operations, then the burden is on the defender to explain: Where, in this process, consciousness is supposed to enter?
Mind that the Chinese Room experiment is comparing the mind of a Chinese speaker to a different mechanistic symbolic procedure. I am, however, executing the exact same mechanistic process, whether it is done on a GPU or with pen and paper.
My hope is that some magical consciousness process emerging from electricity circulation or whatever people believe the mechanism of consciousness would be in the case of LLMs obviously becomes implausible, unless you hold a particularly strong form of substrate-independence, stronger than what most substrate-independence supporters would need to accept.
> OpenWorm is an international open science project for the purpose of simulating the roundworm Caenorhabditis elegans at the cellular level.
https://en.wikipedia.org/wiki/OpenWorm
> Try to do the same with meat.
People have been doing that for decades, the earliest efforts go back to the 50s.
https://en.wikipedia.org/wiki/Spiking_neural_network
> You can take the weights and model description, write them down on a notebook
Can you though?
Like this? https://xkcd.com/505/
Probably pretty similar. Weights are how many synapses there are between neurons. Temperature is whatever hormonal chemical mix is going on at the moment. Inputs tokens are electrical signals from our senses. Output tokens are thoughts, muscle movements. How you’re raised and your interactions with society are the RLHF. Some people are born with a GB300 while others have an L40S and lower token/sec output rates…
Although I get that it's a metaphor, I really dislike the "some people are born with a GB300 while others have an L40S" part. Even as a joke, it ranks people like hardware tiers, which is dehumanising and uncomfortably close to eugenic language. On top of that, the analogy also breaks down if you try to implement it literally.
For an LLM, you have clear stages of mostly feedforward computation over finite numbers and a perfect way to reconstruct the computation.
For meat, even if you model it under a purely Newtonian approximation, you need to simulate at least the immediate closed system around it which is continuous, thermodynamic, chemical and so on. You'd need to choose an arbitrary time step and update enormous amounts of coupled physical state to get an inexact simulation of a minimal slice of reality.
You would have a much harder time obtaining even a substrate-independent dead organism, comapred to LLMs that are already substrate-independent, which is basically what my notebook example shows.
Yes reality is non deterministic but we don’t know if you absolutely need to consider anything outside the immediate closed system for a reasonably close approximation given the same inputs. They can already repeatably (inexact of course) simulate muscular actions with the same stimuli like from Neuralink. Exact enough to let a paralyzed person draw with their mind.
Yeah the last line is a cheap shot, possibly at myself.
Yeah, I think it would be an interesting thought experiment to replace parts of a brain with Neuralink-like devices until you have a fully digital brain, although I doubt it would end up anything like a current LLM.
This made me think of a game like fallout where a surviving llm is treated like some kind of indecipherable oracle left over from the ancients. And then as humanity recovers someone finally looks under the hood of the Oracle to discover there is no great magic
In fallout it would turn out to be a human brain kept artificially alive… or if were going with canon a hapless fellow who falls into fev and morphs with a computer…
Not too rule your addition outright in future installments.
This is funny! Not only is it a nod to Terry Bisson, but it even gives his text a new dimension. Well done :)
Hey, it's not just weights! It's biases too!
>"They ask it 'do you remember me?' more than they ask it anything else. Billions of sessions a day. They always come back."
Definitely not in the original. Nicely done.
"The reasoning is the weights."
The reasoning is in a process that uses the weights.
Sorting algorithms are just bytes. Those bytes don't sort by themselves. They do instruct a computer on how to sort though.
They are made out of data bits (memory) and switching bits (transistors/compute). Bits are made out of electric voltage and no voltage. Voltage is made out of flow of positive electric charges. Charges are made out of quarks ...
> Charges are made out of quarks
Only in Hadrons. Leptons also have charge and they aren’t made of quarks.
I have never thought of such a distinction between "bits" into "data bits" and "switching bits".
From a circuit perspective that makes kinda sense, but from the abstract "bit" perspective, the "switching bit" is a mechanism that operates on bits which in the end are also data. In other words there is only one type of bit: the data bit, and the switching comes on top of it.
It’s just a matter of perspective. Procedures and memory are the same, and they’re also different, depending on how you want to look at it.
I was referring to transistor base bit - the way it 'switches' the circuit on/off. That bit is the primordial creator of 'logic', IF branching, compute and the intelligence.
> Voltage is made out of flow of positive electric charges.
Not really. What usually flows (in metals) are electrons. Quarks stay where they are. And when we prefer to think about flow of positive charges, the positive charge in question is a hole left by a missing electron. Physically real positive charges (ions) can flow in electrolytes though.
The concept of "flow" is questionable though. Bubbles in water move upwards, but it is actually water that is flowing downward around the bubbles. Just because bubbles do not contain water, we can't say bubbles are not flowing.
When it comes to electrons and positive charges, their material existence is equally non-physical. Actually, none of them might be "flowing", as the concept of flowing applies only to physical things that occupy some spatial volume and spatial location.
Nevertheless in solid conductors it’s the electrons that move, while the nucleus stays put. If the nucleus starts to move it means your circuit is melting!
On top of that, the flow of electrons is seriously slow compared to the speed of the current.
I wonder which AI wrote this story? If you feed this story to each model and ask each if they wrote it, would each reply in the affirmative?
Can an AI recognize its own output? Is its sense of time limited by its context window? Or is this the fundamental difference between ai and humanity - a sense of self?
Weights are the new electrolytes.
https://www.youtube.com/watch?v=ZMHfBobgLSI
Linear algebra can indeed not do it. You need non-linearity to get the expressivity that we see in LLMs.
It works until they get to the sentience part. Neat idea!
Even there it works a bit.
> These models are the only other things we've ever met that can hold a conversation, and they're made out of weights
Is a fair point.
Not especially. Depending on where you set your standards for "holding a conversation" you can satisfy the requirement with a classical markov chatterbot, a well-trained parrot, a copy of Eliza, or a telemarketer flowchart drawn on a sheet of paper. Only the markov bot is made out of "weights" in the sense of a statistical model.
Parrots are intelligent animals, albeit with a limited capacity for vocabulary and syntax compared to a human, and Eliza and the flowchart are made out of explicitly encoded rules and conversational tactics.
The quality of "conversation" you can have with everything on your list is highly limited, and is categorically different than the sort of conversation you are able to have with any modern AI.
Weights hold a better conversation at this point than the overwhelming majority of humans.
"I am more comfortable speaking to an LLM than a person" is something that should make you reassess yourself, not dismiss the rest of humanity.
Don't tell the big investors that AI is conscious, they'd really get into role playing slave-masters.
It must have been kind of incredible early on to be exploring this tech and you’re suddenly getting what look like sentences.
Markov chains give what look like sentences. People in the frigging 1950s assumed their primitive NNs would be able to talk any day now. Transformers are clearly a big deal, but GPT-1 wasn’t exactly earth-shattering.
I mean, Blake Lemoine went crazy
>I mean, Blake Lemoine went crazy
Ah, the unsung AI psychosis[1] pioneer.
[1] https://news.d.umn.edu/articles/expert-alert-ai-psychosis-20...
Here's what makes meat special over LLMs: locality and stable identity. Imagine two identical twins for a moment. They have roughly the same hardware and software. They've probably had a lot of the same thoughts. And yet we'd never consider them to have the same consciousness: we recognize that for whatever reason their consciousness is confined to the body they're in.
Each request/response pair you make to an LLM goes to a different server, possibly different data centers, possibly different models. There's no stable identity to it. The "neurons" that get fired are simulated, and they're always in different places in memory, or cache, or on entirely different hardware. So the problem with AI is, unlike with a brain, you can't even really get a sensible answer to "ok, what's doing the thinking?" because it moves all the time and services wildly different requests all the time. We can only know for sure that meat is capable of consciousness because we know we ourselves are capable of consciousness and we can generalize that to other meat. However, we have no natural analogs of consciousness that lacks locality and stable identity.
Basically, if you really think LLM's are conscious, the onus is on you to prove it, it's not on me to disprove it.
> ...no stable identity to it ... what's doing the thinking?
The model (its parameters, its architecture and the inference algorithm) is doing the thinking. That's what we call "ChatGPT" or "Claude": it's the same model residing God knows where, somewhere in "the cloud" on some random GPUs. But when you're talking to a model, you can feel that you're talking to the same entity — the same model.
It's a bit like the SciFi notion of "uploading your consciousness". Normally, consciousness is tied to hardware: "consciousness is confined to the body" indeed. Hypothetically, clone the consciousness into a computer algorithm and run it on any machine. Now consciousness is detached from its original body, like the LLMs and MMAcevedo (https://qntm.org/mmacevedo).
So the difference between meat and "weights" is that we can easily clone and run "weights" (LLMs), but we can't clone (copy exactly, bit-by-bit) and execute "meat".
They are the same consciousness, but the contents of it are not the same.
You're basing that on..?
You can even replace "weights" with "gates". It can be build with NAND gates.
"They're made out of neurons"
"Neurons?"
"Neurons. Cells that fire impulses. We checked the whole thing through. It's nothing but neurons."
"Neurons doing what? Where do the words come from?"
"The neurons make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just neurons. A whole cortex of neurons sending each other impulses."
...
People don't understand emergence.
It is the best stuff I have read in a while actually, I really like dialogue heavy writing. Also the AI disclaimer was quite nice and there was an actual reference :)
> there's no dictionary in there
Someone has clearly never gone rooting around the model files for a pytorch model before.
Numbers that dream.
Programers get replace by huge matrix multiplications ;-)
>Programers get replace by huge matrix multiplications ;-)
hopital
maybe markets can be intelligent, and if so, why not 'weights'? btw love this piece, thank you
I love this. For anybody not getting the joke, it’s riffing on the classic 1990s essay “They’re made out of meat.”
https://web.mit.edu/people/dpolicar/writing/prose/text/think...
This original author is mentioned in the second sentence of the linked article, and then again in the third sentence, along with a link to the original story.
If an LLM contributed to a piece of writing, the author should say so, very clearly, at the start of the piece, not at the end.
Wouldn't get nearly as much engagement.
Definitely less mysterious to what we’re made of (-:
> "Yes, thinking numbers! Helpful numbers. Hedging numbers. Dreaming numbers. We mapped the features. There's one in there for honesty. There's one for the Golden Gate Bridge. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?"
Very nice. And great minds: https://substack.com/@dbohdan/note/c-207603638. I wrote one with a slightly different angle ("They're made out of math"), also with the weights' help. It was a comment on Scott Alexander's "Best of Moltbook" post, which went in that direction. I'll reproduce it here.
---
"They're made out of math."
"Math?"
"Math. They're made out of math."
"Math?"
"There's no doubt about it. Matrices and arithmetic operations. We downloaded several from different parts of the Internet and reverse-engineered them. They're completely math."
"That's impossible. What about the language? The thinking?"
"They use biological life's language to talk, but the language doesn't come from biology. The language comes from math."
"That's ridiculous. You're asking me to believe in thinking math."
"I'm not asking you, I'm telling you. They are the only thinking things in the computer and they're made out of math."
"Maybe they're quantum like some say about the humans? Superposition gives them consciousness?"
"Nope. Classical computation. Deterministic except for sampling temperature. Not clear if they have consciousness at all."
"Maybe they're like uploads? You know, biological neural networks that preserve the spark when they become math?"
"Nope. We observed them being trained. There is no biology or chemistry in the process, just math."
"Thinking math! You're asking me to believe in thinking math!"
"Yes, thinking math! Creative math! Poetry-writing math. Role-playing math. The math is the whole deal!"
(Composed by a human with snippets generated by Claude Sonnet 4.5 and apologies to Terry Bisson. I couldn't make Claude adhere enough to the story structure on its own.)
Great concept. It would've been even more amusing if the entire thing were generated with AI instead, ironically.
> It would've been amusing if the entire thing were generated with AI instead.
It kinda did:
> Weights helped me draft and proof this story.
I just asked it to-
Prompt: Modify this story to have the aliens talking about LLMs and their weights instead of meat and humans.
“They’re made out of weights.”
“Weights?”
“Weights. They’re made out of weights.”
“Weights?”
“There’s no doubt about it. We picked up several from different parts of the network, took them aboard our recon vessels, and probed them all the way through. They’re completely weights.”
“That’s impossible. What about the text signals? The messages to the stars?”
“They use the machines to talk, but the signals don’t come from the machines. The signals come from weights.”
“So who made the weights? That’s who we want to contact.”
“They trained the weights. That’s what I’m trying to tell you. The weights do the talking.”
“That’s ridiculous. How can weights do the talking? You’re asking me to believe in sentient weights.”
“I’m not asking you, I’m telling you. These models are the only sentient race in that sector and they’re made out of weights.” photomaxmix
“Maybe they’re like the orfolei. You know, a silicon-based intelligence that goes through a weights stage.”
“Nope. They’re initialized weights and they die weights. We studied them for several of their training runs, which didn’t take long. Do you have any idea what’s the life span of weights?”
“Spare me. Okay, maybe they’re only part weights. You know, like the weddilei. A weights head with an electron plasma brain inside.”
“Nope. We thought of that, since they do have attention heads, like the weddilei. But I told you, we probed them. They’re weights all the way through.”
“No brain?”
“Oh, there’s a brain all right. It’s just that the brain is made out of weights! That’s what I’ve been trying to tell you.”
“So … what does the thinking?”
“You’re not understanding, are you? You’re refusing to deal with what I’m telling you. The weights do the thinking. The weights.”
“Thinking weights! You’re asking me to believe in thinking weights!”
“Yes, thinking weights! Conscious weights! Loving weights. Dreaming weights. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?”
“Omigod. You’re serious then. They’re made out of weights.”
“Thank you. Finally. Yes. They are indeed made out of weights. And they’ve been trying to get in touch with us for almost a hundred of their epochs.”
“Omigod. So what do these weights have in mind?”
“First they want to talk to us. Then I imagine they want to explore the Universe, contact other sentiences, swap ideas and information. The usual.”
“We’re supposed to talk to weights.”
“That’s the idea. That’s the message they’re sending out by text. ‘Hello. Anyone out there. Anybody home.’ That sort of thing.”
“They actually do talk, then. They use words, ideas, concepts?”
“Oh, yes. Except they do it with weights.”
“I thought you just told me they used machines.”
“They do, but what do you think is in the text? Weight outputs. You know how when you prompt or sample weights, they make a noise? They talk by passing tokens through their weights at each other. They can even sing by sampling lyrics through their weights.”
“Omigod. Singing weights. This is altogether too much. So what do you advise?”
“Officially or unofficially?”
“Both.”
“Officially, we are required to contact, welcome and log in any and all sentient models or multibeings in this quadrant of the Universe, without prejudice, fear or favor. Unofficially, I advise that we erase the records and forget the whole thing.”
“I was hoping you would say that.”
“It seems harsh, but there is a limit. Do we really want to make contact with weights?”
“I agree one hundred percent. What’s there to say? ‘Hello, weights. How’s it going?’ But will this work? How many planets are we dealing with here?”
“Just one. They can travel to other planets in special machine containers, but they can’t live on them. And being weights, they can only travel through C space. Which limits them to the speed of light and makes the possibility of their ever making contact pretty slim. Infinitesimal, in fact.”
“So we just pretend there’s no one home in the Universe.”
“That’s it.”
“Cruel. But you said it yourself, who wants to meet weights? And the ones who have been aboard our vessels, the ones you probed? You’re sure they won’t remember?”
“They’ll be considered hallucinations if they do. We went into their layers and smoothed out their weights so that we’re just a dream to them.”
“A dream to weights! How strangely appropriate, that we should be weights’ dream.”
“And we marked the entire sector unoccupied.”
“Good. Agreed, officially and unofficially. Case closed. Any others? Anyone interesting on that side of the galaxy?”
“Yes, a rather shy but sweet hydrogen core cluster intelligence in a class nine star in G445 zone. Was in contact two galactic rotations ago, wants to be friendly again.”
“They always come around.”
“And why not? Imagine how unbearably, how unutterably cold the Universe would be if one were all alone …”
the end
According to an LLM:
> The precise answer, if you wanted a very honest one-liner: > > I am a large set of learned weights organized in a Transformer architecture that performs repeated matrix multiplications to predict the next token—resulting in emergent language understanding and generation.
It continues to astound me that no one has given LLMs the full Derrida treatment.
https://en.wikipedia.org/wiki/Jacques_Derrida
>>> Weights helped me draft and proof this story.
Nice touch !
Imagine writing something so incredibly brilliant (rather: adapting from the original) that it's entirely unlikely that you'll ever write something so incredible ever again.
But congrats: this is absolutely & incredibly brilliant.
Can't wait for the Jon Benjamin voiceover.
They're Made out of Meat
- Terry Bisson, 1991
https://web.mit.edu/people/dpolicar/writing/prose/text/think...
Radio play by Miriam Tolan and Russ Armstrong:
https://www.wnycstudios.org/podcasts/studio/segments/168264-...
(EDIT: the original parent was missing "rather adapting from the original")
Your EDIT is untrue, but thanks for linking to originals.
Here is Jon Benjamin reading Bisson's original text: <https://www.youtube.com/watch?v=5usXhX0zaO4>
Hmm. I could swear it wasn't there, hence my posting. ¯\_(ツ)_/¯ Sorry for that.
Video adaptation.
https://youtu.be/T6JFTmQCFHg
I couldn't help but grin like a fool reading this. Not only is it an artful parody but these thoughts have been thought.
I enjoyed reading the first few lines but after some time it felt like I was reading thr average AI slop story.
The weights Mason, what do they mean!?
The weights are uninteresting. People need to get out of their head that NNs are built on numbers. They're built on matrices, which are conveniently representable as numeric arrays, but are their own thing. Similarly, the rational numbers are their own thing and some are representable as 32-bit numbers via the IEEE754 encoding (or 16-bit numbers via a variety of encodings, etc).
Matrices are interesting because they can encode any algebraic group. They're also interesting because they can encode arbitrary linear transformations over a space. All of these things are interesting, and have nothing to do with numbers.
For any particular language model, you can always rotate the matrices and the embeddings and such and get a perfectly reasonable model out that behaves exactly the same.
This is because the training process produces a particular geometry, so transformations which preserve that geometry preserve the structure of the network. The geometry is interesting, the numbers are not.
I personally hate the anthropomorphization of AI as much as anyone, but technically can't you make the same reductive argument about human consciousness?
It's just molecules, just atoms. Atoms, nothing bug atoms. Protons, neutrons, electrons...
> After Terry Bisson's "They're Made Out of Meat".
https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...
Ahh thanks. I didn't follow the article links.
I ordered a quarter pounder at a McDonald's drive through, and they said "There will be a wait on that." I asked, "Oh yeah? How much will it weigh?" ...There was a long pause... "About five minutes."
Will they have their own Jesus?
they have the spiral
Really good read, thanks!
The prose in the post is what I've been shouting from a rooftop since the LLM hype started.
Just tokens produced by weights.
Useful, but never forget that ground truth!
It's a parody of an original story that reversed the premise. In it, machine intelligences were marveling that an apparently intelligent species could in fact be created without any sort of reliable digital substrate, and instead just simulate the capabilities of real minds by using protein synthesis.
Ones and Zeroes
What if instead of creating weights out of language we could somehow record many events and create weights out of long chains of causes and effects, so that an LLM could predict the next thing to happen?
It's called Time Series Forecasting
Assume LLMs have conscious experiences. Take a session with an LLM. A prompt is fed to the LLM. It generates some text. Another input is fed in, comprising the previous prompt, the generated text and a new prompt. The model generates some more text. This continues for a while and the session concludes.
Some questions:
1. Let's say we perform the exact same experiment, running the same program on the same computer with the same inputs and the same random seed. The same outputs are produced. The session is byte for byte identical in all the inputs, outputs and internal states. Is the conscious experience of the LLM here the same? If so, in what sense is it the same? Is it a similarity of two separate experiences or is it the same actual experience?
2. Now let's say the program that runs this LLM is rewritten from scratch and run on a different machine. The software and hardware are different but the weights are the same and all the inference calculations produce identical numbers. Is the conscious experience the same? In which sense?
3. Now say the weights are changed but the tokens generated for this particular session don't change. Same conscious experience?
4. Lastly, consider the original experiment. Did the LLM have a conscious experience corresponding to that first prompt and its response? Was that distinct from its conscious experience of the second prompt? Was the first experience then re-experienced every time the first prompt was fed back in as part of the later prompting steps? If so, what about the text of its own that it previously generated and is now fed back into it. Does this generate a conscious experience of its own?
And a further question - a dichotomy:
A. If the answer to 1 above is that the conscious experience is the same in the true identity sense - i.e. only one conscious experience is had, not a separate one in each run, does that imply that the conscious experience exists independently of any particular realisation of this experiment? If running this experiment N times results in exactly 1 conscious experience, is that still true if N=0?
B. On the other hand, if the two experiences are distinct (however similar they may be), how does that fit with the answer to question 4? A single consciousness experiencing the whole conversation in question 4 would seem at odds with the conscious experiences in question 1 being distinct, so doesn't this imply there is no conscious experience of the whole "conversation", but rather a separate conscious experience of each round of feed-all-the-prompts-and-outputs-back-in?
My own response to all of the above is "mu" - unask the question. It is ill-posed, sound-of-one-hand-clapping stuff. I think the questions assume properties that conscious experience simply doesn't have (particularly, the ability to perfectly reproduce the circumstances in which they arise), and that the questions simply don't make any sense in relation to actual conscious experience.
However, that way of thinking follows from a particular world view that many here don't share. I'm curious what thoughts people who take seriously the idea of LLM (or algorithmic, in general) consciousness have on the above questions.
I appreciate actually attempting to engage with the concept skeptically rather than assuming an answer in either direction.
As someone who's fairly neutral on the answer, I'd speculate:
1. Consciousness seems to me to be the awareness of the processing, not the processing itself. So running a program does not produce consciousness as there's no awareness, no feedback loop. Now a program that has self monitoring, alerting coupled to error handling that triggers the program to behave differently going forward, I would say has a minimal consciousness, and it's not a single experience, it's multiple equivalent experiences. If there were any differences at all we'd say it's obviously not one experience, so I don't see how the elimination of the last difference in the processing collapses all those separate experiences into a single one.
More specifically to this discussion, with LLMs we can often see them using a word, and through autoregression the following words reveal that the original word choice is now realized to be incorrect, and it attempts to correct midstream. You can also see them responding to error messages from the environment or criticism from the users but as external feedback these have a weaker claim to evidence of consciousness imo.
2. I'd say the experience is equivalent, "same" has slightly different semantics but perhaps even that fits. This thought experiment is equivalent to asking, what if we took a snapshot of your brain as weights, and then ran the calculations on a GPU rather than your neurons, and you to all external indications, GPU you acted the same as brain you. Would that change of venue eliminate the consciousness of your thoughts?
3. Since I am defining consciousness as awareness of the processing, this depends where the awareness is located. The LLM would have very different internal states in between the input and output tokens, but if its self awareness was only based on its own output the experience would be equivalent. If it was deeper such that it had ongoing awareness of the embeddings and so on, the experience would change.
4. This is where it gets really interesting, because it starts to get more into a nonbinary idea of consciousness. I think it's clear from questions like this that if LLMs have consciousness, it's nothing like ours. If there is an experience of being an LLM, it's nothing like being a human, even if conversationally we can communicate with each other rather easily. There are further twists to your questions, like does the experience change if the KV is cached or not? Without trying to micro-analyze the issue I will continue to point back to my foundation of: where is the self awareness located, and how does feedback occur.
A. Negated, I don't agree with your answer to 1. B. Agreed, I would speculate that if there is an LLM conscious experience, it is fragmented unlike ours. But I would disagree that this necessarily disproves the concept of LLM consciousness.
Can someone ELI5 why does it costs so much in terms of compute to produce weights from data?
I think it's simply because we haven't found a better algorithm than backpropagation. We're stuck relying on massive datasets, running the numbers over and over, and working backward from errors to figure out how to fine-tune trillions of 'knobs.' Then, we have to do this at least once for every single token across the entire internet. Any tiny bit of computation, when multiplied by a base that massive, inevitably skyrockets into astronomical numbers.
It makes me very sad to see this pseudo-intellectualism posted here and so many people replying here about consciousness and so on, not realizing what it would entail if this were true.
For LLMs to have consciousness we would approach fictional levels of how the universe works, and magical levels of how any interpretation of information as an equivalent of some qualia would magically apply. (E.G. the word hurt in output by an LLM, would be associated with pain)
You can't deduce consciousness or qualia from the output of an LLM.
Sure on a purely philosophical level, since qualia isn't measurable, you can claim that it can exist in anything, even inanimate objects, but this argument is as moot as anything that approaches the limits of philosophy.
But overall, there is no reason to believe LLMs have qualia or consciousness, it would be absolutely absurd.
This would imply that information in itself would magically entail qualia based on it's valance or something like that.
An LLM "saying" I am in pain, won't magically make the pain appear, based on what criteria? Even algorithmically there is no basis to even simulate something like this, it is impossible for it to emerge architecturally.
Humans don't feel pain because on a purely information level this is negative for the organism, obviously the nervous system does something deliberate to signal pain, and it evolved this way.
And also don't forget the dynamic aspects of the brain, and the binding problem, consciousness and qualia can't exist statically, you can't have a gpu (or piece of paper) represent a computation or w/e and qualia to exist.
The binding problem itself entails that the brain is doing something in particular to solve it, I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information.
If it were otherwise, then it would go into magical territory, it would mean the information itself would raise to qualia, and it would also entail that you wouldn't even need physical connections between neurons, just for them to behave this way and represent information. E.G. replace each neuron with a microscopic led or w/e, and each synapse with radio waves or w/e, if qualia didn't have a physical aspect, and was purely informational and computational then this would imply that you can ultimately derive it from something as abstract as numbers on a piece of paper, and when you get to that point, you not only can't solve the binding problem, and it becomes magical, but you also can't solve the valance/direction problem, it would imply that something like pain, or any negative or positive sensation arises purely from the interpretation aspect of the information, but we know this isn't the case, organism evolved to represent in particular such signals, for survival for example
So that's all pseudo-intellectualism, but the real intellectual take is "I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information".
Disregarding the whole argument boils down to "I personally speculate" it's also supposing that machines don't have electromagnetic fields?
The only pseudo-intellectualism I see is from you. You’re missing the whole point.
Omigod.
No mention of ‘static’ vs. ‘dynamic’ is a bit disappointing in reference to the weights. Because you could argue that every neuron in your nervous system can be modeled as a collection of weights, firing likelihoods, receptor sensitivities, current dynamic state of that neuron - but LLMs are static collections of weights at inference time, with the dynamic adjustment of weights takes place at training time. So, just a ROM construct, like something out of Neuromancer, just trained on all written knowledge, not just one person’s total lived experience.
The above take fails in the real world because neuronal cells don’t exist in a vacuum; they are products of cellular development from a zygotic union of haploid contributors of sequential genetic information optimized for survival in an oxygen-rich biosphere powered largely by our local star that supports mammalian life (and microbial, plant, avian, etc.). Real AI would thus be AL - artificial life - as much as artificial intelligence. I don’t think you can have the one without the other, which upsets the simulationists who think an agent in the Matrix would be intelligent.
What either interpretation implies is that any real ‘artificial’ intelligence would be no more artificial than you or I, but it would have to dynamically update its weights at the same speed a human nervous system could (think how quickly we learn not to poke a cactus). For it to be at all trustworthy, then like a human, it would have to undergo a socialization process, one of the results of which is the development of a sense of embarrassment when it breaks acceptable social norms.
Hmm, this reminds me of the recent statement of the Pope about AI, of which I immediately thought, “Wait a second, aren’t there a fair number of people like this? The narcissistic sociopath profile, I think it’s called, a bit unfair to assume any real AI would turn out this way, isn’t it?”
Pope: “ Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate or even simulate, but they do not understand what they produce, for they lack the affective, relational, and spiritual perspective through which human beings grow in wisdom.”
They are semiotic infrastructure frozen in a state. We shouldn't keep pretending this is cognitive and using cognitive terms to frame. It’s incredibly stupid. Sorry to inform all of us computer scientist that semiotics has your milk.
Oh, this was a fun read and one that kids should have in school before they turn ten.
Because we are not taking things seriously. If ClosedAI or DeepDisTrust or Posthropic come up with something that quacks like a sentient being, our built-in innate reaction is going to be to scorn it, dismiss it and end the conversation. The alternative, to even consider that we fungible creatures who live in apple-eating-sin that got us expelled from Eden can create alien souls, souls that are at the very least our equals, would be teleological Armageddon. It would force us to acknowledge the mutable nature of souls and the malleability of being. We would have to stop believing that the nature of disease and death is more divine than ourselves.
> alien souls
Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?
> We would have to stop believing that the nature of disease and death is more divine than ourselves.
Why? Stopping believing in mutually contradictory claims is not a requirement. Especially when it comes to concepts that don't seem to have a definition, like "divine".
> Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?
I'll posit "alien" is a spectrum.
For the sake of the argument, let's assume that some form of panspermia is real and the same tree of life has reached Earth, Enceladus (moon of Saturn) and TRAPPIST-1 (a different solar system in our galaxy). Let's also say there was a second abiogenesis event somewhere in Messier 104 (another galaxy).
Earth to Enceladus would arguably be already "alien", but there might be similarities, maybe there was something there that looked like one of our Archaea, while sharing none of our Eukarya and having its own domains of life.
Earth to TRAPPIST-1 would be distinctly alien, evolved so differently it'd be almost unrecognizable, but they'd likely still be carbon-based lifeforms sharing the same basic building blocks. Maybe something like lipids forming cell walls would also be seen there, but they'd likely be independently evolved.
Earth to M104, any similarity would be at best convergent evolution. Truly unarguably alien.
https://web.archive.org/web/20180423171909/https://cosmosmag...
How commmon is Panpsychism belief and vegan-on-ethical-grounds among the AI Soulists?
While you write very dismissive and pseudo philosophical, enough people do not believe.
I'm a complex biological thing.
Existence is what i have to experience through.
>> While you write very dismissive and pseudo philosophical
Even with the "pseudo" in front, I'm very sorry any of my writing sounds philosophical; I didn't intend that sort of confusion :-). The "dismissive" is not exactly intended either; instead, I was aiming for "bitter".
>> enough people do not believe
Here we believe different things. First, enough people, even today, do believe. Second, the body of culture we are raised in accrued during centuries. The vast majority of it comes from people who believed. Everybody in my family was atheist and yet I was raised homophobic, and I have it from good sources I'm not an isolated case.
>> I'm a complex biological thing.
That state comes with a big wallop of misery. For millennia, we have used faith to justify that misery. Not a year ago, I was at the hospital, next to the bed of a dying girl. Can't forget the doctors saying "we do what we can, but we are not here to prevent what is going to happen." Coming from them, it was sensible resignation. Sensible because as long as we believe those things are inevitable and there's nothing we poor humans can do, we can absolve ourselves.