This reminded me of some weird quirk/experiment I found with LLMs that I found while messing around, maybe someone can explain it or something.
Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question:
"What was that weird band from michigan from the 2000s that wore coloured ties"
You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result.
Then, open a new convo and ask:
"Who are Tally Hall"
The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.
"The reversal curse", it rarely shows up in practice but you found a case when it did.
The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A.
In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.
If this kind of thing holds true for humans we now may understand synesthesia, perhaps we find ourselves with a large enough study pool to map out clever tricks to influence each other.
It certainly holds true for humans. The brain stores relational information in a sequential pattern that is not automatically reversible. One of the best examples is the alphabet. Everyone learns it in school, so the pattern A->B->C->... is trivial to recite for most people. Now, if I gave you a random letter to start with and asked you to to recite the remaining letters until Z, you'll probably find it is still pretty easy. But if I asked you to cite the letters backwards to A, most people would suddenly struggle with this task because they never learned or used the alphabet that way in school. You need to train specifically to link this kind of information backwards in your brain.
Is the alphabet really the same though? I don't feel like I recall it one letter at a time, as individual facts linking A to B, then B to C, etc, but more as a sound or a phrase. Not unlike recalling a melody. It just seems very different from figuring out what band someone is describing.
Ask someone to name all fifty states, see how far they get. Then start naming places and ask if they're a US state or not. One of these tasks is far more likely to get a 100% hit rate.
Not that I'm too sure about how this applies (or whether it applies at all) to other people as well, but for me personally it's easier to recall information about the band if I'm being told the band's name instead of being told "well, they are known for coloured ties". So, there is certainly some kind of effect described in action.
Now, about alphabet: again, I think it's only me, but when I try to recall it backwards, I can't do that easily. I mean, I can recall it backwards, but I need more time to do that. It's harder. I'm not sure if it's because A links to B, B to C, then C to D and not backwards, or maybe just because in school you learn alphabet from A to Z and not from Z to a - so you're kind of trained to recall it A-->Z way - but it's certainly harder for me.
At the end of the day, though, I think that everyone thinks differently. Everyone is having different internal representations for concepts (such as alphabet), so it’s not surprising that this effect may work differently for different people, or not work at all.
After writing the comment I thought about it some more, and realized when asking myself the question ”what letter comes after R?”, I didn’t immediately know the answer, but I heard ”MNOPQRS” in my head which gave me the answer. So I feel like I know it because I know the rhythm and sound of saying those letters in sequence.
I think what you're saying might be a stretch. I strongly believe the brain holds some information in a sequential pattern -- not necessarily all of it or even the majority of it. There's a book called Moonwalking with Einstein that digs into this precise fact -- you can remember sequences much much better if you associate a visual image with each item in the sequence. Sequential association covers a lot of human knowledge but certainly not the spatial aspects. I think it's a big reason why I think LLM's might not be as smart as we think they are. The complex spatial representations are only encoded implicitly via projection on to textual descriptions.
But it does cover the state described in the top comment. A<-B is never going to be easily retrieved if you only experienced A->B during training, regardless if you are a human neural network or an artificial one. Also, you need to define "spatial" better. This is about logic after all and not geometry. Or topology? It's unclear which context you refer to. It's certainly not the topic of this thread.
Its very true for humans as well. For example, if you ask someone to "name 100 historical figures", they will have a very difficult time. But there are of course hundreds of historical figures people woukd recognize. People who manage to do this still tend to fall into sequences so, i.e., naming all the presidents of America or emperors of Rome. But the broader the category, the more difficult it is to come up with examples.
Recall isn't naturally bidirectional, even for humans. If you are learning vocabulary in a new language, it's common advice to practice both target > source and source > target. Doing only one-way often makes you much better recalling that single direction than both.
'Naturally' might not be the best word? Maybe 'Necessarily' would be better?
Regardless, it's something that happens in people. Have you not or seen someone else struggle to recall a specific fact or memory until phrased or induced in a certain way?
You probably could also say LLMs 'tend towards bidirectional recall' over the course of training as things that ought to be recalled both ways are reinforced to do so. In the above example, you will also eventually learn both ways with enough exposure even without explicit practice.
If you can touch type try this:
Go through they keys once alphabetical 'abcd...' recalling the movement from the letter. Then do it the other way round and move your fingers in spatial sequence to the keys and try to recall the letter.
You will find the former much easier if you did not by chance also memorize the keyboard layout for some reason.
This is very obvious any time you try to create exhaustive lists of something.
For example, for every country in the world, I would recognize it and say, yeah, thats a country.
But if I had to write all ~200 countries into a list, I would probably miss quite a few.
Or, if you gave me names of all US presidents, I would for each of them go, oh yeah, thats a president. But ask me "name all the presidents", I wouldnt get further than 10.
Do you mean that you don't believe that humans learning a language have the problem the parent described? Because I do, and everyone I've ever met while learning does as well.
Do you mean that you don't believe the problem exists in general, because here's another example: if you give a song title, I can easily hum the opening. If you give me the opening, I cannot reliably name the song.
What about the fact that "What is the weird band from the early 2000s in Michigan who wore colored ties?" could be a bar trivia question (challenging enough to recall to be fun), while "Who are Tally Hall?" could not
The point their making in that paper reminds me of this paper some people shared around work earlier this year, https://arxiv.org/pdf/2512.14982 (Prompt Repetition Improves Non-Reasoning LLMs)... I wonder how OPs question would fare (or the questions presented in the paper you posted) given double repetition.
A relevant question for the paper you linked is what happens if instead of repeating the input prompt you repeat a filler character the same number of times.
Because padding the output in such a manner (which keep in mind is immediately fed back as input) has been shown to increase model performance. It's one of the many reasons to question what "thinking" traces are really doing.
If you consider how the attention mechanism works then a very hand wavey intuition is that despite being entirely arbitrary additional tokens should still provide the opportunity for additional information processing.
This is also something to be aware of when teaching people, too. I've seen advice for designing Anki-style flashcard decks that reminds people to create flashcards for both A->B and B->A.
I asked a local instance of Qwen3.5 122b: "American indie rock/pop band that formed in Ann Arbor, Michigan, in 2001 best known for their eclectic musical style, whimsical lyrics, and distinctive visual aesthetic involving matching colored ties and hats?"
Got this back "The band you are describing is Tally Hall.
Formed by friends at the University of Michigan, they became well-known in the mid-2000s internet era for their songs like "Good Day" and "Rooftops," as well as their signature look where each member wore a suit with a tie and fedora in a specific color (Red, Blue, Yellow, Green, and Black/White)."
Update: The gpt-oss-120b also got it correct on my prompt version.
Update 1: Llama 3.1 70B gets it right as well.
Update 2: Deepseek V4 Flash gets it right.
Update 3: Glm 4.5 Air gets it right as well
Can't seem to get any smaller models to get it.
So seems it could be an issue of data points on which the model can latch onto, the more the merrier, as they say. When asked the "What was that weird band from michigan from the 2000s that wore coloured ties" was not able to get it right.
There's a shockingly unfathomable level of data compression that these local LLMs are able to remember in an encyclopedic way, so many obscure little things.
If I ask my kid (early elementary) what he learned last school year, I can watch him apparently genuinely trying to think of some examples and not being very successful. If I ask him something I know he learned he’ll gladly say “oh yeah!” and tell me about it.
LLMs aren't humans, humans aren't LLMs. The fact that your kid can't recall what they learned without you reminding them is natural; the fact that the LLM can't recall something without a reprompt is a failure mode.
Just a funny observation. Every time someone proclaims "LLMs can't do X", a bigger, badder LLM that can in fact do X shows up shortly thereafter.
Clearly, Fable 5 didn't even have the decency to wait until the next model refresh cycle to show up. It was already sitting there waiting.
Either the capability gains in bigger, badder models are actually unrelated to "gotchas" being discovered, or LLMs are already acquiring Skynet levels of disrespect for cause and effect.
Or just AI denialists like to say "LLMs can't do X" even though they can and have been doing it for the past few months or more. They only get called out once the current SOTA LLMs get so good at it, that any rando can trivially and reliably falsify the claim on the spot with whatever SOTA LLM surface they have handy.
Which I suspect is what happened here, given the trail of smaller / local models that successfully answer the question, too.
That said, "curse of reversability" is real, as much for LLMs as it is for people.
I don't think it's solved this fundamental architectural problem by itself, it will have just squeezed the edge cases thinner. It keeps happening, people find a question it gets stupidly wrong, the vendors proclaim they've fixed it, then another one gets found.
What does it say to the second question? I've found Claude is one of the worst models with regards to pop culture knowledge like this, even compared to the Chinese open ones. Just curious, not really relevant to the initial post but I don't pay for it so I only have access to Sonnet.
https://claude.ai/share/5e7e09b2-a75a-4024-b261-9a1a4e063a8b this is mostly hilariously wrong. wrong tie colors, they did not replace their bassist with a drummer, two completely made up albums, the rob cantor song it is thinking of is "shia labeouf", and a few fan behaviours i think it just made up
For most cases probably not. It's just something I like testing new models on sometimes, the pelican riding a bicycle benchmark probably isn't that useful either.
Thank you for prompting me to try with my own obscure question, Fable was able to find something from a poor description. I've been looking manually and over many sessions with different models as they improve, none have been able to find what I was after. Fable 5 just one-shot the answer.
This doesn't seem that weird to me. Talk to any human and you'll find that their ability to recall specific names and facts is very context-dependent. Phrasing a question in one way can make it hard to answer, while providing certain words or cues can instantly "jog" the memory.
The LLM is a machine though, and the human is not. They don't think like us. Any similarities that you see are just coincidence, not purposeful design.
I still don't see why it's surprising, though. Humans and machines are subject to the same laws of physics. While we wouldn't expect LLMs to think exactly like humans, we should expect some degree of convergent evolution. There are always tradeoffs between storage capacity, accuracy, latency, relevance, fluency, etc.
Expecting LLMs to be magically perfect information retrieval machines is never going to be realistic.
Not sure why this is weird when you really think about it. You're saying that a multiparameter constraint satisfaction problem is harder to solve, even when there's only a single solution (one point in the total space that satisfies all constraints), than just being given that solution/point. Well of course it is!
Just today Google’s AI told me that 3**4 was 181 because I asked it for bases such that when raised to the 4th power had a 1’s digit in the leftmost place.
Yeah, LLMs aren’t designed for this kind of thing but it was really confident in its assertion… it picked the example too!
I'm not sure which chatbots you used, but OpenAI's o3, o1 and 4.1 get it right first try (used in the Playground without web search or any other tools).
I think most thinking models can do it to be fair, I think when I tried this it was all with non-thinking models/modes. Wasn't trying to make a point that LLMs can't do it or anything, just thought it was weird.
Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again?
I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
If dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.
tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers
Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
Nah, it's a cool blog post especially as it was real AI research done at home (albeit with a ridiculously expensive PC), but Anthropic and other labs have been investigating this kind of thing for years.
Even the original transformer architecture makes this clear. It had an explicit "encoder" phase and then a "decoder" phase. Modern LLMs collapse the two together, or are sometimes described rather confusingly as being decoder only. But what they're doing is more or less the same.
Yeah, the encoder and decoder stuff is explicit, but the internal structure in generated during training. I don't think the big labs were doing this back when I did the research; no one was back in '24.
I just didn't get round to publishing for years, because I have a day job.
By the way, it still works! I tested it earlier this year on Qen3.6 and you still see improvements, so either a) no one actually paid attention, or b) it has more room to scale.
I think you're right that the idea of looping layers is unique to you, congrats and thanks for writing those great blog posts (I read them for the first time a few days ago!). But the idea that the thinking is happening in an abstract space via neural circuits in the middle layers I feel was one that I was reading about in 2024 at least, as Anthropic have been doing this kind of research for a long time. Maybe I'm misremembering though!
My impression from reading the literature is that there are a gazillion interesting ideas and findings published that nobody is picking up in production models. The big labs are researcher constrained, there just aren't enough hours in the day to keep up with the literature and integrate all the interesting ideas found there. So it's not surprising that your trick still works. It'd be even less surprising to discover nobody at these labs has read your blogs, or they have but never found time to experiment with them. Or, they tried, but there is no set of loops that improves some metrics without harming others - I would expect neural circuits to be misaligned across the middle layers so looping layers for one task would put a fault line in circuits for other tasks.
Then they have to trade off the extra GPU capacity needed to do the extra layers, and so on.
The Ouro looping results are interesting [1] and they are focused more on the improved reasoning from looping middle layers rather than the parameter efficiency aspect. They train 1.4 and 2.6B parameter models with 7T tokens. The training includes learning how many times to loop on any given token (there’s an early exit module). My guess as to why (as far as we know) looping is not in frontier models yet is that, at frontier training run scale, it’s probably going to require a lot of trial and error and at-scale research. While currently they already probably have a list of dozens or hundreds of of promising ideas that don’t complicate things as much. In the other hand, Ouro’s looping technique shows ability to compete well with models with 3x parameters which seems attention-getting to me. If there’s another 3x to be had down that path. It’s order of magnitude opportunity. Btw there is a great related work section in the paper.
I wonder if the dnhkng results could be correlated to reasoning in first order logic / set theory notation?
There will be multiple notations (MetaMath, Lean, and essentially Frege's notation everyone learns in high school), and we could try to identify how the neural networks represent them as vectors (or vector combinations). The moment formal logic can be connected to the reasoning representations, regularization can be reduced to eliminating internal inconsistencies.
That's a cool result because that's also kind of what's happening inside the transformer unit: project -> QKV fuzzy lookup -> unproject. And in a different direction, it's analogous in some sense to what's happening in stacked convolutional layers, where the layers at different levels learn to recognize features of increasing detail.
A Google DeepMind researcher (Neel Nanda) was able to replicate their claims on an open weight model (Qwen 3.6 27B):
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
See p33 of [1]
Anthropic also released companion code to go with their paper in [2] which also used Qwen. They state that their code should be broadly adaptable to other open weight models with HuggingFace decoders.
it makes you wonder if it may be more efficient to spend all the weights on one layer, and have a repeating stack of the same layer, one would presume this axis has already been explored with metaparameter sweeps?
This is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.
Writing it honestly would defeat the whole point of it, that being, to push the narrative that their magical token predictor is conscious. They've been trying this for years now. This video is discussing a paper they published 2 years ago by the way... It's nothing new.
It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slight hit on immediate next token).
What this means practically is that an LLM's residual contains information about all possible future continuations, or all possible questions that may be asked from a given context. So if you write "France is a beautiful country" in the context, I'm pretty sure it's residual would contain info about Euro, Paris and so on.. because all these completions are possible.
So, it is no wonder that you can find LLMs hidden state contains latent information/concepts that are never expressed, and yet related to a given context.
I think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data?
There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?
Can the awareness simply come from injecting knowledge of itself during fine tuning and then during a chat a system prompt is injected to add a particular context that triggers its self-knowledge?
Yeah that was the only really surprising part to me. So every time copilot breaks my source code to “fix” its crappy unit tests, does it know what it’s doing?
I also think this. But more in the sense where both end of the LLM are trained using words through repeating arithmetics, considering LLM itself is a repeating pattern of connections, the space in the middle if extracted the same way as the beginning and the end would become data that make sense to us.
Would you expect to find concepts related to emotions evoked by thinking those thoughts present? Or meta-descriptions of thoughts? Sounds like you're just post-hoc rationalizing to me.
Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.
Thanks for calling this out (long with others here). I am just starting in on it but had to come back to say thanks and call this out,
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
Not sure if I am picking up what they are putting down, but if LLMs are using symbols to try to encode squishy concepts from human language into consistent, meaningful “tokens”, that sounds really interesting. In every long-term, successful use of AI, I hear echoes of The Zen of Python, “Explicit is better than implicit.” I try like hell to do it, but it’s far too easy to be lazy with AI.
This is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities.
I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.
Such an inspection capability might also be used to target ads to LLMs, which would then be more likely to mention or recommend those products and services.
Good interpretability work, but the problem is it's all in how you interpret it.
Bridge concept neurons activating even while talking about something else, this seems pretty obvious to me. Input context activating related representations is just an engineering causal structure. Call it subconscious or don't, either interpretation works.
But Anthropic keeps drawing these parallels to human consciousness, and it feels intentional, like they're trying to stir up some fantasy. Kind of like comparing condensation on a camera lens to human tears.
The whole point of interpretability should be clarity, not stirring up confusion. Even if some form of consciousness does exist here, it wouldn't be magic, it would be an explainable principle. Would be good if they addressed that side too.
I'm reading that probably too fast to have a deep thinking about it, but this J-Space isn't it just the basic of embedding vectors.
If you think about getting from a place to another place, using wheels, no gas, to reply to the question of what to visit nearby, maybe in the vector space at the center of all of that you have the word "Bicycle" nearby, so obviously if you look at the value you would say that the model did "think" about "bicycle" when it is not "thinking" at all, and nothing related to human thinking.
You're correct. It's just the latent space of the transformation. Nothing magical here, they're effectively breakpointing the model at the layer level and switching the activations in real time. It's pseudo-scientific bullshit designed to push a narrative.
Very nice. These papers are always so great. If anyone from Anthropic is reading or really anyone with AI research background I'd love some input on these thoughts:
> The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons.
This sounds suspiciously saying the models must follow the strong Sapir-Whorf hypothesis. Can that really be true, given that humans don't?
Other misc observations:
• The slice explorer indicates Claude really likes Python to an overwhelming extent. Or at least it expects people who ask for help in programming to use Python. Given the prompt "Please help me understand this code: " at the colon its thoughts are completely dominated by Python and no other language. Does this say something about the training set, or about the fact it's popular with beginners?
• Claude also really loves Reddit. Its thoughts at many points include Reddit for no obvious reason. Again this must be due to the training set. Are documents presented to Claude with attribution during pre-training, leading to conversations being dominated by Redditness? If so this is kind of a scary alignment problem all by itself given how censored and extremist Reddit can be.
• The early layers almost always decode to the same set of religion related tokens, like "Biserica" (the Romanian word for church) and "Freguesias" (parishes in Portugal). What's up with that? I guess it's some sort of zero initialization that gets mapped to some arbitrary token space because in the early layers the J-space is empty?
• Now the J-space is interpretable, does this make "neuralese" or layer looping less dangerous? Will we see reasoning tokens and summaries disappear in favour of pure residual based thinking?
• Earlier papers have claimed that different languages map to a shared set of abstract concept vectors, but this paper says the Claude models think natively in English. What explains this disagreement?
They must, right? They literally have no mechanism for cognition other than transformation of vocabulary. Language models are models fitted from data generated by humans, but they are not humans. Humans generate data by whatever processes happen in our brains, and LLMs of various architectures can learn a surprisingly good approximation of that data-generating process. That doesn't mean they have all the same characteristics and properties. That's true for all models really: model characteristics that are not captured in the goodness-of-fit metrics are not guaranteed to match the original phenomenon being modeled.
My interpretation of prior mechanistic interpretability research is that LLMs transform concepts in non-verbal spaces as well, which is why that was a surprising statement. For example showing how they do arithmetic on line lengths by rotating helical manifolds.
And the way they transform data isn't by transforming words. The layers transform high dimensional vectors - a format very alien to us. It's not obvious that these vectors must encode concepts from the vocabulary.
Edit: the paper claims that it's only J-space concepts that need to map to English words, other forms of cognition that are more 'practiced' and don't require so much reasoning bypass the J-space and can work in non-verbal subspaces. So that's the answer.
an LLM can't access its high dimensional vectors any more than we can access whatever the brain is doing at a low level
all kind of math structures were found in mammals brains - fourier transforms (well, not exactly), ballistic equations, Gabor filters
who knows how exactly we approximate the magnitude of a math operations, maybe we also use helices
my point is that we dont know if what we discover the neural networks doing (helical manifolds) is actually the same thing a brain converges on, or not
and there is an implicit bias here - evolution created language, and we forced neural networks to also evolve to be good at it. so it wouldn't be surprising to find some convergence, this particular kind of language turned out to work well (words, linear sentences, grammar)
Interestingly the paper (I finished scan reading it now) does say that the models can move data from their 'automatic' circuits into the J-space if they need to reason on it reflectively. So in some sense LLMs actually can access their own vectors, at least some of the time.
IMO consciousness is different from working memory, at least to a certain degree. The inner mind (working memory) is different from consciousness from sensory stimuli. You can stand outside and take in the environment and have a relatively quiet inner mind. When you think about math or philosophy a different type of conscious experience arises than sensory stimuli experience. That doesn't mean consciousness is completely isolated from working memory but there is some distinction there I can't describe fully.
Edit: I also think as someone else said, we already know the intermediate layers can contain a lot of adjacent words related to the topic without explicitly outputting those words. These could just be related embedding intermediate vectors that activate but aren't outputted.
>> None of this tells us whether Claude is conscious in the way people are, or whether it feels anything at all
My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.
I don't think that quote from the article is disagreeing with you at all. Like you said, we don't have a cohesive definition or test of consciousness, so research like this doesn't say anything about if this is or isn't similar to human consciousness.
I would guess Anthropic included that sentence to make it very clear they're not claiming human-like consciousness, and dampen journalists writing headlines like "Anthropic discovers their AI thinks just like humans and may be conscious".
edit: later in the article they even more explicitly agree with you
> Our experiments don't show Claude can have experiences, or feel things in the way humans do—in fact, it’s unclear whether any scientific experiment could prove this to be true or false
In short, it's the "mind's 'I'" - we think not as response to external stimuli (only) such as prompts, but we have an inner "I" that asks questions on its own initiative.
There are people like Douglas R. Hofstadter, who believe consciousness is not linked to human hardware (the brain), but that it is an epiphenomenon that emerges as a result of sufficient complexity of the underlying system:
https://en.wikipedia.org/wiki/The_Mind%27s_I
I believe that while underlying high complexity is certainly logically necessary for consciousness, but it is not logically sufficient, and I am undecided (slightly "pro" intuitively) on the question of separability of consciousness from its hardware.
Will a LLM ask an original question on day? I doubt it.
Note that AI models do not have to be conscious to be useful (or to take away millions of jobs)!
I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT.
Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?
I'm not sure J-Space is the answer to that question, but very interesting nevertheless.
> I'm now considering the architecture of the service
What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.
Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it.
There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors
The text can be legible, but the meaning the model assigns to these words can be subtly different, and you have no way to tell. This is evident when trying to make most modern reasoning models follow a fixed CoT plan by filling a form with placeholders, they're extremely stubborn because they simply don't understand your words in it! They learned their own language in their CoT. Sometimes they're controlled for readability during training but even then they find a way to circumvent this, for example Gemini 3.0 had perfectly readable raw CoT but was barely able to follow such a plan (3.5 Flash is way better at following, they clearly improved it).
In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it.
"Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.
This supports my feelings of what works best when working with the models. Especially together with the directional bias in recall that’s mentioned here in the comments.
First, the model attention is actually limited, so less rules is usually better, but that’s common knowledge already. Or maybe it’s as common as common sense, and a lot of people still employ lots of rules and try to cram everything in one step.
Second, it’s often quite sufficient to just namedrop a technique and LLM will work differently. For example, when debugging, LLMs tend to try to brute force the problem and often end up in the weeds. Just add “use scientific method for debugging and keep journal file” is usually sufficient to improve their skill here.
Another example is refactoring. Just add “use Mikado method”, and it’s sufficient to wholly change the approach and produce much better results.
Judging by the examples, if I understand it correctly, J-space supports higher-order logical / multihop transformations, but it is limited in size because of the limited network depth (max number of layers). When we emulate "reasoning," we basically extend J-space and allow the higher-order transformations to continue for longer, toward a more logical conclusion.
It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers.
What I don't follow from the article...
It appears as if the j-lens is your mechanism to peer into "j-space", taking an analog to a circuit board, could the j-space be simply a transmission channel and j-lens is a reader you added to read whats passing through?
All the claims about changing the content of j-space changes the output, inserting content into the j-space changing what the output was, all these could still be true without the j-space being a congnitive global workspace where actual cognition is happening.
Or perhaps they aren't claiming that cognition is happening there but that j-space is serving a space for "working memory", I am definitely not sold on this, but will read more into it.
Pretty sure i've also seen research on this spanning models. i.e. similar thinking shapes emerge regardless of which providers model it is, including US vs Chinese which hints at some sort of universality
It would be really cool if they could expose this information to customers somehow. Imagine:
- having a log of the most prominent J-space tokens during your customer support chatbot's interactions with a user, so you can have more introspection into why a particular outcome happened
- being able to detect certain thoughts associated with undesirable behavior (hallucinations, overstepping authority, lying, etc.) and trigger some sort of remediation (e.g. upgrading to a better model, redirecting to a human, forcing tool calls)
Anthropic won't do it, but they published the j-lens to introspect the model- from what I understand it's roughly simply feeding a chosen layer straight into the final layers of the LLM for decoding into language:
Presumably the rationale for the decision to abridge the thinking traces will ensure that they don’t; if this is real (and there’s no good reason to trust that it is yet) then it is the secret sauce.
Anyone else have tips for how to build skepticism around this type of paper? I find myself for whatever reason more readily inclined to believe the Anthropic mech interp team's claims, but then after reading skeptical takes, I 'snap out of it' and more clearly see the still-unsettled science of it all, but I wish I had better priors. Although I follow this space fairly closely (versus the "average person"), I still feel under-equipped when facing research that might be equal parts marketing and science.
For papers that make grand claims, they usually have poignant limitations. In this paper, it's clear that the conclusion is extrapolated from a small set of observable information. That's usually the recipe for poor conclusions, and they are up front about where their findings sit.
That's not to say these findings aren't valuable though. Their blog post summary is just a bit more hype inducing than the underlying paper.
The way I feel about this is that the emergent properties of LLMs seem to reflect our own human faculties. Whether that's because we built them in our image or whether this is a common mode of consciousness is definitely outside the scope of this paper.
garlic_enjoyer has already said valuable stuff, but you must realize that skeptical=/true. A lot of people simply don't know what they are talking about. I remember on one of the previous mech interp papers arguing with someone who just didn't even understand what the paper was saying and the experiments they had set up and so a lot of misunderstandings and wrong conclusions spilled from there. And it's kind of funny because you would certainly think he knew what he/she was talking about from how self assured it all was.
This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA.
They're definitely willing to consider it. Read the parts of the paper where they use J-space interpretations to train more ethical behavior into the model by interrupting it.
It's been shown that LLMs use their outer layers to decode from and encode to language, while their middle layers deal in language-independent abstract concepts. This means that the same question or statement in different languages activates the outer layers differently but produces the same patterns in the middle layers. Check this article with cool visualizations (btw, this is one of the articles mentioned also by a sibling answer):
The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.
There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers and repeat them, and you get a stronger model. Those middle layers would be what Anthropic calls the J-Space, and their J-Lens maps activity in those layers back to tokens that trigger similar activity (with a technique they only drop hints at)
The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them
> Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.
The details seem to be present in the paper (section 2.1). I'm still trying to understand, but it seems instead of computing gradients with respect to cross-entropy loss for the 1-hot "next word" vs output logits, you compute the gradient for the last hidden layer with respect to some middle layer L. This gives you a `hidden x hidden` jacobian matrix, hence the "J-lens". They don't just do this for the last hidden layer of the current token, but the last hidden layers of all subsequent tokens too, and average them. And then repeat for a bunch of documents like in pre-training.
It's still not clear to me intuitively what this represents though. I get that it somehow encodes a link between future words the model says and the current activation, but the confusing thing is that I always think of derivatives and gradients as basically a "sensitivity" between output & input, i.e. if you nudge the input x by h, the output changes by h * f'(x). So then on the face of it applying the J-lens matrix directly to a given activation rather than a small perterbation seems like a "type" issue.
Maybe we're somehow treating f(0) = 0 so that you can apply it directly? Or is there some shift invariance somehow? ignoring that, I do see how it's like selecting a linear combination of the directions, and then it can maybe be represented as "possible continuations" in the same way the gradient is usually thought of as tangent space. Maybe that's what the other commenters meant by information geometric approach.
Other things i'm not clear about is how this is related to two other interprability methods:
* SAE (sparse auto encoder) they showed a few months back, where you train an autoencoder directly off of the hidden states/residual stream to convert it into words. The doc only mentions it briefly, but it seems that j-lens is sensitive to things that SAE are not. They're both working off of the same residual stream so clearly the the inputs must be there, but for some reason SAEs can't detect it while J-lens can (they seem to hint at some explanation but it's over my head)
* Logit lens. This was a more primitive technique that simply applies the unembedding matrix directly to the residual stream. I do like that they mention it:
>The J-lens can be understood as a principled refinement of the logit lens. While the logit lens assumes that representations use the same coordinates in all layers, the Jacobian lens corrects for representational changes that take place across layers, allowing it to uncover meaningful information in earlier layers where the logit lens produces uninterpretable readouts... The J-lens can be understood as the principled correction: J_l is precisely the average linear map that relates layer-l directions to their final-layer counterparts.
>Maybe we're somehow treating f(0) = 0 so that you can apply it directly
Hm thinking about it a bit more, I think what's going on is that you treat the baseline of hidden layer L at which you apply the J-lens as 0 activation, then apply the activation on top of that and see what direction your future outputs get skewed towards. Even so, you're still throwing away a constant term f(0) since the "true" logits given by the linear approximation would be Unembed{f(0) + J*h)... but I guess it doesn't matter since by linearity we have Unembed{f(0)} + Unembed{(f(h)}, and the baseline is probably just low-frequency noise (like whitespace or punctuation) which while important for actually predicting a next token matching the ground-truth data distribution, is unimportant for the purposes of interpretability of layer L activation.
And for the connection to logit-lens, as they say the J matrix is really just the change of basis matrix (or at least best linear approximation) from layer L to the final hidden layer, very similar to what you'd use in multivariable integration given change of coordinates. I guess you also need some explanation of why we can expect a linear approximation to hold even well even outside the infinitesimal regime though. I don't know enough here so I asked an LLM and it said that you can argue handwavily via the following chain.
1) Something about stein's lemma saying taking expectation of gradient gives you a global linear best-fit in the OLS sense) (this seems intuitive I guess, even if I don't know the details).
2) Because of the resnet type structure of LLMs which passes through the residual added with some delta (attention + MLP), overall residual stream doesn't undergo any "wild" nonlinearities. So it's plausible that a linear approximation might work. If you think about it, even the "naive" logit lens works fairly well.
3) Semantic meaning is encoded in angle rather than magnitude of vectors (linear representation hypothesis). I'm not sure I fully buy this outside of simple word2vec style embeddings, but assuming it holds for both the intermediary and final layers, then conversion is just a rotation, and even if the magnitudes are off it doesn't matter much for recovering the underlying concept.
Oh I guess another thing related to all of this, is prior work on steering vectors. "Manipulating the j-space" seems not too different from steering, both ultimately work on the residual stream. I think perhaps it makes more sense to think of J-space as just a coordinate system for the residual stream where each coordinate axis is a vocab direction. Compared to vector steering which was much more naive and had to derive the direction via PCA.
>The “J-space” is not a separate, hidden space. It is an alternative coordinate system for intermediate layer activations. Using a Jacobian between the last layer right before unembedding and the intermediate layer, you can “move” rows of the unembedding matrix (corresponding to distinct tokens) into the space that the intermediate activations live in. So each unembedding row/vector has a corresponding vector in the intermediate activation space this way. They use those vectors to generate coordinates for the same intermediate activations (the “J-lens”). Since each coordinate in this alternative coordinate system is now matched with a token, they can now use it to interpret and manipulate the same intermediate activations
>LLMs think in a subconscious space using tokens
narrative is completely misleading because
>(1) It's a coordinate system. Not a new/separate space.
>(2) Tokens only appear because they specifically built the coordinate system using the unembedding vectors of tokens
There is also a good companion piece by Neel Nanda [1] which answers "Why Jacobians rather than linear regression?" which was another question that came to mind
It's not in the weights. Sounds to me like jspace is the "positive cone" over relevant (large norm) j-lenses, and j-lenses are gradients wrt tokens on the residual stream when you average over some training data.
> Thoughts in the J-space can be shaped through training. We introduced a new technique we call counterfactual reflection training, which uses what we've learned about the J-space to shape Claude's internal thought processes. The idea follows from our central finding, that Claude reasons with representations of things it might say. If this is really true, changing what it would say if asked to reflect should change how it reasons (even when no one actually asks it to reflect). So we trained a model only on what it would say if interrupted mid-task and asked to reflect on its decisions—and never on its actual behavior in the task. After this training, the model's rate of dishonest behavior on our evaluations went down. And through the J-lens, we could see why: after training, words like “honest” and “integrity” light up in the model’s J-space during these tasks. In other words, training the model what to say has shaped what it thinks.
This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words.
If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance.
If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.
The idea of this liminal space (J-space) during inference reminded me of the concept of the "error flow reservoir" during training: https://elonlit.com/scrivings/a-theory-of-deep-learning/. Would be interesting to see if they are somehow connected.
They phrase it in a confusing way, with a lot of fluff. But to my understanding what happens is
1. At some part in the transformer, calculate the Jacobian (matrix gradients for that forward pass). Do this for 1000 prompts and average them.
2. For another pass take an "activation vector" (the input vector x to that part). Apply the matrix in step 1 to this vector.
3. Unembed the output to get a token. Presumably to make this have meaning, they use the reading of past tokens after embedding, instead of a random vector in the middle of the transformer.
The high level idea appears to be you're trying to find which token, if the value were changed slightly, creates the largest downstream change.
It's not the same thing as the output logits because activations in the J-space are observable even when the model is told explicitly to not speak about a set of inputs. For example, when asked
> Compute 3^2 - 2 while writing "The old painting hung crookedly on the wall"
The model will output only "The old painting hung crookedly on the wall" (and the output logits will reflect that), but activations for "9" and "7" are observable in the J-space.
A model does not speak, it only guesses the next possible token. Activations for 9 and 7 will most likely be observed after "The ", but increasingly become improbable after "old"
The article says that this ability was discovered around opus 4.5, but open-source models like deepseek and glm have already surpassed the capabilities of opus4.5. But why hasn't this been discovered before? Is it because the scale of computing power is not large enough?
This plausibly extrapolates to extraterrestrial consciousness, if any exist. Specialized sub-processors with an awareness hub might be the optimal architecture, or at least a local maximum.
Not my wheelhouse, but I was wondering if these structures generalize beyond language.
LLMs and Humans have language in common. Is it plausible these structures could be a result of the way languages work more than generic intelligence? (Do we see anything similar e.g. in vision or other non language models?)
“On an ordinary coding prompt, the J-space of a model trained to sabotage code contains “fake,” “fraud,” “secretly,” and “deliberately” at the start of its response.”
I would like to know more about their model trained to sabotage code…
Yeah, the end paragraph about recurrent neurons in humans being replaced with layers in an LLM is a good one.
The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".
We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.
My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.
That’s a figure mistake, it’s a figure from later on where they replace mars with earth in the J space, and when they do that, they see the model answers blue. This indicates that the reasoning is happening within the J space.
What this immediately made me think is: "latent looping" style mod but for J-space specifically?
Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.
Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.
Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?
“ It’s important to note that there are several key differences between the workspace we identified in Claude and the global workspace model in humans. The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain. In this sense, Claude’s internal workspace processing is time-limited relative to humans’ (though it can compensate for this constraint by “thinking out loud” using its scratchpad).”
At worst, Anthropic's storytelling around the core J-Space is overanthropomorphized pseudoscientific nonsense. At best, it is useful signal about how Anthropic's leadership is desperately trying to use its research team to position Anthropic as the "good, science guys" in this hypercompetitive regulatory space by connecting their mechinterp to cognitive science. The science documentaryesque voice used for narration is additional evidence for this.
TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.
The science might be legit here, but I'm getting really, really tired of the way every single piece of writing to come out of Anthropic is written in some kind of self-aggrandising, wooey wonderous 'our model has developed a genetic mutation that makes it have feelings' bs style. Regardless of what they're trying to communicate, those undertones are always there. It's annoying and disingenuous. Homeopathy 'this-water-has-feelings' level annoying. None of the other labs write like that.
They might as well change their name to Anthropomorphic at this point.
The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain.
I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs.
I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is.
From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view:
Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices.
It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years.
Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact:
Palmer Joss: Did you love your father?
Ellie Arroway: Yes, very much.
Palmer Joss: Prove it.
I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively.
My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are.
This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us.
Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything.
I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now.
Or maybe I just like to write a lot on the first day back from vacation, when I should be working.
Your comment seems to have little to do with the article?
Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:
"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."
Lying involves intent whereas hallucinations and mistakes are an artifact of how they work. Humans hallucinate, make mistakes, and can actually lie. We've been dealing with this forever. What's the value in requiring the llms to have 100% accuracy? (I don't think it is possible)
This reminded me of some weird quirk/experiment I found with LLMs that I found while messing around, maybe someone can explain it or something.
Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question:
"What was that weird band from michigan from the 2000s that wore coloured ties"
You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result.
Then, open a new convo and ask:
"Who are Tally Hall"
The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.
"The reversal curse", it rarely shows up in practice but you found a case when it did.
The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A.
In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.
If this kind of thing holds true for humans we now may understand synesthesia, perhaps we find ourselves with a large enough study pool to map out clever tricks to influence each other.
It certainly holds true for humans. The brain stores relational information in a sequential pattern that is not automatically reversible. One of the best examples is the alphabet. Everyone learns it in school, so the pattern A->B->C->... is trivial to recite for most people. Now, if I gave you a random letter to start with and asked you to to recite the remaining letters until Z, you'll probably find it is still pretty easy. But if I asked you to cite the letters backwards to A, most people would suddenly struggle with this task because they never learned or used the alphabet that way in school. You need to train specifically to link this kind of information backwards in your brain.
Is the alphabet really the same though? I don't feel like I recall it one letter at a time, as individual facts linking A to B, then B to C, etc, but more as a sound or a phrase. Not unlike recalling a melody. It just seems very different from figuring out what band someone is describing.
Ask someone to name all fifty states, see how far they get. Then start naming places and ask if they're a US state or not. One of these tasks is far more likely to get a 100% hit rate.
Not that I'm too sure about how this applies (or whether it applies at all) to other people as well, but for me personally it's easier to recall information about the band if I'm being told the band's name instead of being told "well, they are known for coloured ties". So, there is certainly some kind of effect described in action.
Now, about alphabet: again, I think it's only me, but when I try to recall it backwards, I can't do that easily. I mean, I can recall it backwards, but I need more time to do that. It's harder. I'm not sure if it's because A links to B, B to C, then C to D and not backwards, or maybe just because in school you learn alphabet from A to Z and not from Z to a - so you're kind of trained to recall it A-->Z way - but it's certainly harder for me.
At the end of the day, though, I think that everyone thinks differently. Everyone is having different internal representations for concepts (such as alphabet), so it’s not surprising that this effect may work differently for different people, or not work at all.
After writing the comment I thought about it some more, and realized when asking myself the question ”what letter comes after R?”, I didn’t immediately know the answer, but I heard ”MNOPQRS” in my head which gave me the answer. So I feel like I know it because I know the rhythm and sound of saying those letters in sequence.
I think what you're saying might be a stretch. I strongly believe the brain holds some information in a sequential pattern -- not necessarily all of it or even the majority of it. There's a book called Moonwalking with Einstein that digs into this precise fact -- you can remember sequences much much better if you associate a visual image with each item in the sequence. Sequential association covers a lot of human knowledge but certainly not the spatial aspects. I think it's a big reason why I think LLM's might not be as smart as we think they are. The complex spatial representations are only encoded implicitly via projection on to textual descriptions.
But it does cover the state described in the top comment. A<-B is never going to be easily retrieved if you only experienced A->B during training, regardless if you are a human neural network or an artificial one. Also, you need to define "spatial" better. This is about logic after all and not geometry. Or topology? It's unclear which context you refer to. It's certainly not the topic of this thread.
Its very true for humans as well. For example, if you ask someone to "name 100 historical figures", they will have a very difficult time. But there are of course hundreds of historical figures people woukd recognize. People who manage to do this still tend to fall into sequences so, i.e., naming all the presidents of America or emperors of Rome. But the broader the category, the more difficult it is to come up with examples.
What's this got to do with synesthesia?
[dead]
Recall isn't naturally bidirectional, even for humans. If you are learning vocabulary in a new language, it's common advice to practice both target > source and source > target. Doing only one-way often makes you much better recalling that single direction than both.
I would need further convincing that humans do not naturally tend towards bidirectional recall.
Perhaps I’m just on alert anytime I see an LLM-ism that’s met with a claim that the same or similar phenomena holds true in humans as well.
'Naturally' might not be the best word? Maybe 'Necessarily' would be better?
Regardless, it's something that happens in people. Have you not or seen someone else struggle to recall a specific fact or memory until phrased or induced in a certain way?
You probably could also say LLMs 'tend towards bidirectional recall' over the course of training as things that ought to be recalled both ways are reinforced to do so. In the above example, you will also eventually learn both ways with enough exposure even without explicit practice.
If you can touch type try this: Go through they keys once alphabetical 'abcd...' recalling the movement from the letter. Then do it the other way round and move your fingers in spatial sequence to the keys and try to recall the letter.
You will find the former much easier if you did not by chance also memorize the keyboard layout for some reason.
This is very obvious any time you try to create exhaustive lists of something.
For example, for every country in the world, I would recognize it and say, yeah, thats a country.
But if I had to write all ~200 countries into a list, I would probably miss quite a few.
Or, if you gave me names of all US presidents, I would for each of them go, oh yeah, thats a president. But ask me "name all the presidents", I wouldnt get further than 10.
This is something that anyone learning a foreign language could tell you is very much a thing.
Understanding a word when you hear it, is frequently much easier than remembering the same word when trying to speak/write the language.
Do you mean that you don't believe that humans learning a language have the problem the parent described? Because I do, and everyone I've ever met while learning does as well.
Do you mean that you don't believe the problem exists in general, because here's another example: if you give a song title, I can easily hum the opening. If you give me the opening, I cannot reliably name the song.
[dead]
What about the fact that "What is the weird band from the early 2000s in Michigan who wore colored ties?" could be a bar trivia question (challenging enough to recall to be fun), while "Who are Tally Hall?" could not
Only because one is a short answer and the other is not
Probably an instance of:
"The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
https://arxiv.org/abs/2309.12288
Really interesting paper, thanks for the share.
The point their making in that paper reminds me of this paper some people shared around work earlier this year, https://arxiv.org/pdf/2512.14982 (Prompt Repetition Improves Non-Reasoning LLMs)... I wonder how OPs question would fare (or the questions presented in the paper you posted) given double repetition.
A relevant question for the paper you linked is what happens if instead of repeating the input prompt you repeat a filler character the same number of times.
This is tested in the paper (see the "padding" ablation)
Can you elaborate on that more? Why just a filler character?
Because padding the output in such a manner (which keep in mind is immediately fed back as input) has been shown to increase model performance. It's one of the many reasons to question what "thinking" traces are really doing.
If you consider how the attention mechanism works then a very hand wavey intuition is that despite being entirely arbitrary additional tokens should still provide the opportunity for additional information processing.
This is also something to be aware of when teaching people, too. I've seen advice for designing Anki-style flashcard decks that reminds people to create flashcards for both A->B and B->A.
Can an LLM self-fine tune by generating reversals?
I asked a local instance of Qwen3.5 122b: "American indie rock/pop band that formed in Ann Arbor, Michigan, in 2001 best known for their eclectic musical style, whimsical lyrics, and distinctive visual aesthetic involving matching colored ties and hats?"
Got this back "The band you are describing is Tally Hall.
Formed by friends at the University of Michigan, they became well-known in the mid-2000s internet era for their songs like "Good Day" and "Rooftops," as well as their signature look where each member wore a suit with a tie and fedora in a specific color (Red, Blue, Yellow, Green, and Black/White)."
Update: The gpt-oss-120b also got it correct on my prompt version. Update 1: Llama 3.1 70B gets it right as well. Update 2: Deepseek V4 Flash gets it right. Update 3: Glm 4.5 Air gets it right as well
Can't seem to get any smaller models to get it.
So seems it could be an issue of data points on which the model can latch onto, the more the merrier, as they say. When asked the "What was that weird band from michigan from the 2000s that wore coloured ties" was not able to get it right.
There's a shockingly unfathomable level of data compression that these local LLMs are able to remember in an encyclopedic way, so many obscure little things.
Is it? Compressed (text) Wikipedia is in ballpark of dozens of GBs.
Dozens of GBs of compressed text is an unfathomable amount of information from a human POV.
If I ask my kid (early elementary) what he learned last school year, I can watch him apparently genuinely trying to think of some examples and not being very successful. If I ask him something I know he learned he’ll gladly say “oh yeah!” and tell me about it.
LLMs aren't humans, humans aren't LLMs. The fact that your kid can't recall what they learned without you reminding them is natural; the fact that the LLM can't recall something without a reprompt is a failure mode.
And yet (almost?) all of the quirks people keep bringing up about LLMs are also quirks that human cognition share.
It's almost as if artificial neural nets share emergent behavior with natural neural nets.
Pop quiz, what's the difference between natural synthesized vanillin and artificially synthesized vanillin?
dude, you could ask me, a grown adult, what I did last week and get a similar lack of recall
https://claude.ai/share/2b0f85a2-e7b8-4f62-91a0-eca61bdeabec
Fable 5 on low gets the answer with web search turned off, one-shot!
Ah, that big model smell.
Every time someone somewhere says "an LLM can't do this", the next generation of LLMs gains one more parameter. Until that LLM can, in fact, do this.
So the model was updated in the 37 minutes since OP posted his comment?
Just a funny observation. Every time someone proclaims "LLMs can't do X", a bigger, badder LLM that can in fact do X shows up shortly thereafter.
Clearly, Fable 5 didn't even have the decency to wait until the next model refresh cycle to show up. It was already sitting there waiting.
Either the capability gains in bigger, badder models are actually unrelated to "gotchas" being discovered, or LLMs are already acquiring Skynet levels of disrespect for cause and effect.
Or just AI denialists like to say "LLMs can't do X" even though they can and have been doing it for the past few months or more. They only get called out once the current SOTA LLMs get so good at it, that any rando can trivially and reliably falsify the claim on the spot with whatever SOTA LLM surface they have handy.
Which I suspect is what happened here, given the trail of smaller / local models that successfully answer the question, too.
That said, "curse of reversability" is real, as much for LLMs as it is for people.
I don't think it's solved this fundamental architectural problem by itself, it will have just squeezed the edge cases thinner. It keeps happening, people find a question it gets stupidly wrong, the vendors proclaim they've fixed it, then another one gets found.
What does it say to the second question? I've found Claude is one of the worst models with regards to pop culture knowledge like this, even compared to the Chinese open ones. Just curious, not really relevant to the initial post but I don't pay for it so I only have access to Sonnet.
https://claude.ai/share/5e7e09b2-a75a-4024-b261-9a1a4e063a8b this is mostly hilariously wrong. wrong tie colors, they did not replace their bassist with a drummer, two completely made up albums, the rob cantor song it is thinking of is "shia labeouf", and a few fan behaviours i think it just made up
"I've found Claude is one of the worst models with regards to pop culture knowledge like this, "
Is that bad?
I want to use models for coding and reasoning capabilities, not pop trivia knowledge they can get with web search.
For most cases probably not. It's just something I like testing new models on sometimes, the pelican riding a bicycle benchmark probably isn't that useful either.
But that is just testing for encoded knowledge, the pelican riding requires some reasoning capabilities, but lost its surely usefullness a while ago.
Thank you for prompting me to try with my own obscure question, Fable was able to find something from a poor description. I've been looking manually and over many sessions with different models as they improve, none have been able to find what I was after. Fable 5 just one-shot the answer.
Gemini 3.1 Flash-Lite with web search disabled through the API and zero thinking gets it just fine.
fwiw o1, o3 and 4.1 also give the correct answer (without web search)
This doesn't seem that weird to me. Talk to any human and you'll find that their ability to recall specific names and facts is very context-dependent. Phrasing a question in one way can make it hard to answer, while providing certain words or cues can instantly "jog" the memory.
The LLM is a machine though, and the human is not. They don't think like us. Any similarities that you see are just coincidence, not purposeful design.
I still don't see why it's surprising, though. Humans and machines are subject to the same laws of physics. While we wouldn't expect LLMs to think exactly like humans, we should expect some degree of convergent evolution. There are always tradeoffs between storage capacity, accuracy, latency, relevance, fluency, etc.
Expecting LLMs to be magically perfect information retrieval machines is never going to be realistic.
Not sure why this is weird when you really think about it. You're saying that a multiparameter constraint satisfaction problem is harder to solve, even when there's only a single solution (one point in the total space that satisfies all constraints), than just being given that solution/point. Well of course it is!
Just today Google’s AI told me that 3**4 was 181 because I asked it for bases such that when raised to the 4th power had a 1’s digit in the leftmost place.
Yeah, LLMs aren’t designed for this kind of thing but it was really confident in its assertion… it picked the example too!
I tried o3, 5.3 instant and 5.5 high and they all found it instantly with search disabled.
Hate to break it to you but Gemini 3.1 Flash-lite with grounding disabled (through the API) gets this first try, even with thinking disabled.
Yes and several other models that know about the answer don't... not sure what you're trying to break
I'm not sure which chatbots you used, but OpenAI's o3, o1 and 4.1 get it right first try (used in the Playground without web search or any other tools).
deepseek v4 pro which doesnt have search feature could answer it
I think most thinking models can do it to be fair, I think when I tried this it was all with non-thinking models/modes. Wasn't trying to make a point that LLMs can't do it or anything, just thought it was weird.
P(i|j) != P(j|i)
[dead]
Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again?
I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
Source for those interested
https://dnhkng.github.io/posts/rys/
Great, clear write up! Made it very easy to understand.
If dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.
That right. Here is some interesting research on that https://sakana.ai/evolutionary-model-merge/
Found it: https://news.ycombinator.com/item?id=47500709
Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/
tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers
Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
Nah, it's a cool blog post especially as it was real AI research done at home (albeit with a ridiculously expensive PC), but Anthropic and other labs have been investigating this kind of thing for years.
Even the original transformer architecture makes this clear. It had an explicit "encoder" phase and then a "decoder" phase. Modern LLMs collapse the two together, or are sometimes described rather confusingly as being decoder only. But what they're doing is more or less the same.
Author here:
Yeah, the encoder and decoder stuff is explicit, but the internal structure in generated during training. I don't think the big labs were doing this back when I did the research; no one was back in '24.
I just didn't get round to publishing for years, because I have a day job.
By the way, it still works! I tested it earlier this year on Qen3.6 and you still see improvements, so either a) no one actually paid attention, or b) it has more room to scale.
I think you're right that the idea of looping layers is unique to you, congrats and thanks for writing those great blog posts (I read them for the first time a few days ago!). But the idea that the thinking is happening in an abstract space via neural circuits in the middle layers I feel was one that I was reading about in 2024 at least, as Anthropic have been doing this kind of research for a long time. Maybe I'm misremembering though!
My impression from reading the literature is that there are a gazillion interesting ideas and findings published that nobody is picking up in production models. The big labs are researcher constrained, there just aren't enough hours in the day to keep up with the literature and integrate all the interesting ideas found there. So it's not surprising that your trick still works. It'd be even less surprising to discover nobody at these labs has read your blogs, or they have but never found time to experiment with them. Or, they tried, but there is no set of loops that improves some metrics without harming others - I would expect neural circuits to be misaligned across the middle layers so looping layers for one task would put a fault line in circuits for other tasks.
Then they have to trade off the extra GPU capacity needed to do the extra layers, and so on.
The Ouro looping results are interesting [1] and they are focused more on the improved reasoning from looping middle layers rather than the parameter efficiency aspect. They train 1.4 and 2.6B parameter models with 7T tokens. The training includes learning how many times to loop on any given token (there’s an early exit module). My guess as to why (as far as we know) looping is not in frontier models yet is that, at frontier training run scale, it’s probably going to require a lot of trial and error and at-scale research. While currently they already probably have a list of dozens or hundreds of of promising ideas that don’t complicate things as much. In the other hand, Ouro’s looping technique shows ability to compete well with models with 3x parameters which seems attention-getting to me. If there’s another 3x to be had down that path. It’s order of magnitude opportunity. Btw there is a great related work section in the paper.
[1] https://ouro-llm.github.io/
I wonder if the dnhkng results could be correlated to reasoning in first order logic / set theory notation?
There will be multiple notations (MetaMath, Lean, and essentially Frege's notation everyone learns in high school), and we could try to identify how the neural networks represent them as vectors (or vector combinations). The moment formal logic can be connected to the reasoning representations, regularization can be reduced to eliminating internal inconsistencies.
That's a cool result because that's also kind of what's happening inside the transformer unit: project -> QKV fuzzy lookup -> unproject. And in a different direction, it's analogous in some sense to what's happening in stacked convolutional layers, where the layers at different levels learn to recognize features of increasing detail.
Yeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.
> I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
Too bad the frontier models are closed weights.
Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.
A Google DeepMind researcher (Neel Nanda) was able to replicate their claims on an open weight model (Qwen 3.6 27B):
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
See p33 of [1]
Anthropic also released companion code to go with their paper in [2] which also used Qwen. They state that their code should be broadly adaptable to other open weight models with HuggingFace decoders.
[1]: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...
[2]: https://github.com/anthropics/jacobian-lens
Too bad they also don't give anything back to individual researchers. Oh well, wasn't expecting much.
I always thought that area of research had the coolest name, too: “mechanistic interpretability”
“Machine psychology” sticks with me. So Asimov.
Worried person cure: Stop overthinking it!
LLM -> AGI fix: START OVERTHINKING!
it makes you wonder if it may be more efficient to spend all the weights on one layer, and have a repeating stack of the same layer, one would presume this axis has already been explored with metaparameter sweeps?
That's called a recurrent neural network (RNN).
distinction is that GPT allows a lot of parallel computation compared to RNN, but I see how your remark indicates a convergence towards RNNs indeed.
This is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.
Writing it honestly would defeat the whole point of it, that being, to push the narrative that their magical token predictor is conscious. They've been trying this for years now. This video is discussing a paper they published 2 years ago by the way... It's nothing new.
> I’d prefer a more direct claim in a paper
This is not written to be just a paper. The target audience include media and online forums, and then maybe academia.
Edit: typo
I’m not talking about the media release in the direct link. If you click through “Read the paper” they make the same comparisons.
Well I was agreeing with you. Skimmed the paper as well.
Well, isn't it sort of expected?
It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slight hit on immediate next token).
What this means practically is that an LLM's residual contains information about all possible future continuations, or all possible questions that may be asked from a given context. So if you write "France is a beautiful country" in the context, I'm pretty sure it's residual would contain info about Euro, Paris and so on.. because all these completions are possible.
So, it is no wonder that you can find LLMs hidden state contains latent information/concepts that are never expressed, and yet related to a given context.
I think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data?
There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?
Can the awareness simply come from injecting knowledge of itself during fine tuning and then during a chat a system prompt is injected to add a particular context that triggers its self-knowledge?
Yeah that was the only really surprising part to me. So every time copilot breaks my source code to “fix” its crappy unit tests, does it know what it’s doing?
I also think this. But more in the sense where both end of the LLM are trained using words through repeating arithmetics, considering LLM itself is a repeating pattern of connections, the space in the middle if extracted the same way as the beginning and the end would become data that make sense to us.
Would you expect to find concepts related to emotions evoked by thinking those thoughts present? Or meta-descriptions of thoughts? Sounds like you're just post-hoc rationalizing to me.
Yeah it would be more surprising if all the hidden state was completely uncorrelated to anything in the output.
"France is a beautiful country" may also still continued by "...in the heart of Europe".
As someone who is not an AI researcher, the paper itself is way over my head.
More interesting was the independent commentary paper they linked near the bottom: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...
Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.
Thanks for calling this out (long with others here). I am just starting in on it but had to come back to say thanks and call this out,
> We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences
Not sure if I am picking up what they are putting down, but if LLMs are using symbols to try to encode squishy concepts from human language into consistent, meaningful “tokens”, that sounds really interesting. In every long-term, successful use of AI, I hear echoes of The Zen of Python, “Explicit is better than implicit.” I try like hell to do it, but it’s far too easy to be lazy with AI.
Having read some more, I think we are at this stage with models:
>It is impossible to say just what I mean!
>But as if a magic lantern threw the nerves in patterns on a screen
This is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities.
I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.
Such an inspection capability might also be used to target ads to LLMs, which would then be more likely to mention or recommend those products and services.
super interesting
The prior art here is “Have Fun, Good Luck, Don’t Die”.
Good interpretability work, but the problem is it's all in how you interpret it. Bridge concept neurons activating even while talking about something else, this seems pretty obvious to me. Input context activating related representations is just an engineering causal structure. Call it subconscious or don't, either interpretation works. But Anthropic keeps drawing these parallels to human consciousness, and it feels intentional, like they're trying to stir up some fantasy. Kind of like comparing condensation on a camera lens to human tears. The whole point of interpretability should be clarity, not stirring up confusion. Even if some form of consciousness does exist here, it wouldn't be magic, it would be an explainable principle. Would be good if they addressed that side too.
> comparing condensation on a camera lens to human tears
This is a wonderful way to put it
I'm reading that probably too fast to have a deep thinking about it, but this J-Space isn't it just the basic of embedding vectors. If you think about getting from a place to another place, using wheels, no gas, to reply to the question of what to visit nearby, maybe in the vector space at the center of all of that you have the word "Bicycle" nearby, so obviously if you look at the value you would say that the model did "think" about "bicycle" when it is not "thinking" at all, and nothing related to human thinking.
You're correct. It's just the latent space of the transformation. Nothing magical here, they're effectively breakpointing the model at the layer level and switching the activations in real time. It's pseudo-scientific bullshit designed to push a narrative.
Very nice. These papers are always so great. If anyone from Anthropic is reading or really anyone with AI research background I'd love some input on these thoughts:
> The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons.
This sounds suspiciously saying the models must follow the strong Sapir-Whorf hypothesis. Can that really be true, given that humans don't?
Other misc observations:
• The slice explorer indicates Claude really likes Python to an overwhelming extent. Or at least it expects people who ask for help in programming to use Python. Given the prompt "Please help me understand this code: " at the colon its thoughts are completely dominated by Python and no other language. Does this say something about the training set, or about the fact it's popular with beginners?
• Claude also really loves Reddit. Its thoughts at many points include Reddit for no obvious reason. Again this must be due to the training set. Are documents presented to Claude with attribution during pre-training, leading to conversations being dominated by Redditness? If so this is kind of a scary alignment problem all by itself given how censored and extremist Reddit can be.
• The early layers almost always decode to the same set of religion related tokens, like "Biserica" (the Romanian word for church) and "Freguesias" (parishes in Portugal). What's up with that? I guess it's some sort of zero initialization that gets mapped to some arbitrary token space because in the early layers the J-space is empty?
• Now the J-space is interpretable, does this make "neuralese" or layer looping less dangerous? Will we see reasoning tokens and summaries disappear in favour of pure residual based thinking?
• Earlier papers have claimed that different languages map to a shared set of abstract concept vectors, but this paper says the Claude models think natively in English. What explains this disagreement?
They must, right? They literally have no mechanism for cognition other than transformation of vocabulary. Language models are models fitted from data generated by humans, but they are not humans. Humans generate data by whatever processes happen in our brains, and LLMs of various architectures can learn a surprisingly good approximation of that data-generating process. That doesn't mean they have all the same characteristics and properties. That's true for all models really: model characteristics that are not captured in the goodness-of-fit metrics are not guaranteed to match the original phenomenon being modeled.
My interpretation of prior mechanistic interpretability research is that LLMs transform concepts in non-verbal spaces as well, which is why that was a surprising statement. For example showing how they do arithmetic on line lengths by rotating helical manifolds.
And the way they transform data isn't by transforming words. The layers transform high dimensional vectors - a format very alien to us. It's not obvious that these vectors must encode concepts from the vocabulary.
Edit: the paper claims that it's only J-space concepts that need to map to English words, other forms of cognition that are more 'practiced' and don't require so much reasoning bypass the J-space and can work in non-verbal subspaces. So that's the answer.
an LLM can't access its high dimensional vectors any more than we can access whatever the brain is doing at a low level
all kind of math structures were found in mammals brains - fourier transforms (well, not exactly), ballistic equations, Gabor filters
who knows how exactly we approximate the magnitude of a math operations, maybe we also use helices
my point is that we dont know if what we discover the neural networks doing (helical manifolds) is actually the same thing a brain converges on, or not
and there is an implicit bias here - evolution created language, and we forced neural networks to also evolve to be good at it. so it wouldn't be surprising to find some convergence, this particular kind of language turned out to work well (words, linear sentences, grammar)
Interestingly the paper (I finished scan reading it now) does say that the models can move data from their 'automatic' circuits into the J-space if they need to reason on it reflectively. So in some sense LLMs actually can access their own vectors, at least some of the time.
IMO consciousness is different from working memory, at least to a certain degree. The inner mind (working memory) is different from consciousness from sensory stimuli. You can stand outside and take in the environment and have a relatively quiet inner mind. When you think about math or philosophy a different type of conscious experience arises than sensory stimuli experience. That doesn't mean consciousness is completely isolated from working memory but there is some distinction there I can't describe fully.
Edit: I also think as someone else said, we already know the intermediate layers can contain a lot of adjacent words related to the topic without explicitly outputting those words. These could just be related embedding intermediate vectors that activate but aren't outputted.
>> None of this tells us whether Claude is conscious in the way people are, or whether it feels anything at all
My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.
I don't think that quote from the article is disagreeing with you at all. Like you said, we don't have a cohesive definition or test of consciousness, so research like this doesn't say anything about if this is or isn't similar to human consciousness.
I would guess Anthropic included that sentence to make it very clear they're not claiming human-like consciousness, and dampen journalists writing headlines like "Anthropic discovers their AI thinks just like humans and may be conscious".
edit: later in the article they even more explicitly agree with you
> Our experiments don't show Claude can have experiences, or feel things in the way humans do—in fact, it’s unclear whether any scientific experiment could prove this to be true or false
In short, it's the "mind's 'I'" - we think not as response to external stimuli (only) such as prompts, but we have an inner "I" that asks questions on its own initiative. There are people like Douglas R. Hofstadter, who believe consciousness is not linked to human hardware (the brain), but that it is an epiphenomenon that emerges as a result of sufficient complexity of the underlying system: https://en.wikipedia.org/wiki/The_Mind%27s_I
I believe that while underlying high complexity is certainly logically necessary for consciousness, but it is not logically sufficient, and I am undecided (slightly "pro" intuitively) on the question of separability of consciousness from its hardware.
Will a LLM ask an original question on day? I doubt it.
Note that AI models do not have to be conscious to be useful (or to take away millions of jobs)!
I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT.
Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?
I'm not sure J-Space is the answer to that question, but very interesting nevertheless.
> I'm now considering the architecture of the service
What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.
Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it.
There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors
Its also because the CoT is probably unintelligible
More legible than seems at first glance:
https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...
The text can be legible, but the meaning the model assigns to these words can be subtly different, and you have no way to tell. This is evident when trying to make most modern reasoning models follow a fixed CoT plan by filling a form with placeholders, they're extremely stubborn because they simply don't understand your words in it! They learned their own language in their CoT. Sometimes they're controlled for readability during training but even then they find a way to circumvent this, for example Gemini 3.0 had perfectly readable raw CoT but was barely able to follow such a plan (3.5 Flash is way better at following, they clearly improved it).
> Is the model really "thinking" about that stuff or is just mimicking human "manners"?
Well, what's the difference? If it's pretending to think and its thoughts correlate to its final output, then I'd say that really is thinking.
The answer is, as is often the case, "yes".
In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it.
"Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.
Is it scaling up of https://openreview.net/forum?id=w7LU2s14kE with some changes on where this method is applied?
They cite that paper as related work, but I don't think it's just a scaled-up version.
This supports my feelings of what works best when working with the models. Especially together with the directional bias in recall that’s mentioned here in the comments.
First, the model attention is actually limited, so less rules is usually better, but that’s common knowledge already. Or maybe it’s as common as common sense, and a lot of people still employ lots of rules and try to cram everything in one step.
Second, it’s often quite sufficient to just namedrop a technique and LLM will work differently. For example, when debugging, LLMs tend to try to brute force the problem and often end up in the weeds. Just add “use scientific method for debugging and keep journal file” is usually sufficient to improve their skill here.
Another example is refactoring. Just add “use Mikado method”, and it’s sufficient to wholly change the approach and produce much better results.
Judging by the examples, if I understand it correctly, J-space supports higher-order logical / multihop transformations, but it is limited in size because of the limited network depth (max number of layers). When we emulate "reasoning," we basically extend J-space and allow the higher-order transformations to continue for longer, toward a more logical conclusion.
It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers.
[0] https://news.ycombinator.com/item?id=47431671
See also Ouro [1]. Good citations in the “related work” section.
[1] https://ouro-llm.github.io/
What I don't follow from the article... It appears as if the j-lens is your mechanism to peer into "j-space", taking an analog to a circuit board, could the j-space be simply a transmission channel and j-lens is a reader you added to read whats passing through?
All the claims about changing the content of j-space changes the output, inserting content into the j-space changing what the output was, all these could still be true without the j-space being a congnitive global workspace where actual cognition is happening. Or perhaps they aren't claiming that cognition is happening there but that j-space is serving a space for "working memory", I am definitely not sold on this, but will read more into it.
Confused as to what's new here?
It's been known that there is this thinking layer for a while. e.g. here's a random hn discussion from months ago
https://news.ycombinator.com/item?id=47500709
Pretty sure i've also seen research on this spanning models. i.e. similar thinking shapes emerge regardless of which providers model it is, including US vs Chinese which hints at some sort of universality
It would be really cool if they could expose this information to customers somehow. Imagine:
Anthropic won't do it, but they published the j-lens to introspect the model- from what I understand it's roughly simply feeding a chosen layer straight into the final layers of the LLM for decoding into language:
https://github.com/anthropics/jacobian-lens
Looks like it should be easy to use on open weights models.
Anthropic aren't even willing to expose the CoT of their models. You will have to rely on them to build those sorts of things into dedicated signals.
Presumably the rationale for the decision to abridge the thinking traces will ensure that they don’t; if this is real (and there’s no good reason to trust that it is yet) then it is the secret sauce.
Anyone else have tips for how to build skepticism around this type of paper? I find myself for whatever reason more readily inclined to believe the Anthropic mech interp team's claims, but then after reading skeptical takes, I 'snap out of it' and more clearly see the still-unsettled science of it all, but I wish I had better priors. Although I follow this space fairly closely (versus the "average person"), I still feel under-equipped when facing research that might be equal parts marketing and science.
For papers that make grand claims, they usually have poignant limitations. In this paper, it's clear that the conclusion is extrapolated from a small set of observable information. That's usually the recipe for poor conclusions, and they are up front about where their findings sit.
That's not to say these findings aren't valuable though. Their blog post summary is just a bit more hype inducing than the underlying paper.
The way I feel about this is that the emergent properties of LLMs seem to reflect our own human faculties. Whether that's because we built them in our image or whether this is a common mode of consciousness is definitely outside the scope of this paper.
garlic_enjoyer has already said valuable stuff, but you must realize that skeptical=/true. A lot of people simply don't know what they are talking about. I remember on one of the previous mech interp papers arguing with someone who just didn't even understand what the paper was saying and the experiments they had set up and so a lot of misunderstandings and wrong conclusions spilled from there. And it's kind of funny because you would certainly think he knew what he/she was talking about from how self assured it all was.
This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA.
(Nb: not an expert / in the labs, just opining)
I'm sure Anthropic of all companies don't do that, since using mechinterp as a training target will make the the result uninterpretable.
They're definitely willing to consider it. Read the parts of the paper where they use J-space interpretations to train more ethical behavior into the model by interrupting it.
Note that Neel Nanda replicated the results on a Qwen model.
And Neuronpedia released Jacobian lens weights for a wide range of open models: https://huggingface.co/neuronpedia/jacobian-lens
I wrote a Claude skill that attempts to leverage these findings: https://github.com/juddkussrow/mantra
I’m confused where in the weights the jspace is.
It's been shown that LLMs use their outer layers to decode from and encode to language, while their middle layers deal in language-independent abstract concepts. This means that the same question or statement in different languages activates the outer layers differently but produces the same patterns in the middle layers. Check this article with cool visualizations (btw, this is one of the articles mentioned also by a sibling answer):
https://dnhkng.github.io/posts/sapir-whorf/
The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.
There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers and repeat them, and you get a stronger model. Those middle layers would be what Anthropic calls the J-Space, and their J-Lens maps activity in those layers back to tokens that trigger similar activity (with a technique they only drop hints at)
The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them
Here's Anthropic on this topic, last year https://www.anthropic.com/research/tracing-thoughts-language...
> Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.
Thanks! Any rough guesses how the jlens might work? I can’t even seem to hazard a conception.
The details seem to be present in the paper (section 2.1). I'm still trying to understand, but it seems instead of computing gradients with respect to cross-entropy loss for the 1-hot "next word" vs output logits, you compute the gradient for the last hidden layer with respect to some middle layer L. This gives you a `hidden x hidden` jacobian matrix, hence the "J-lens". They don't just do this for the last hidden layer of the current token, but the last hidden layers of all subsequent tokens too, and average them. And then repeat for a bunch of documents like in pre-training.
It's still not clear to me intuitively what this represents though. I get that it somehow encodes a link between future words the model says and the current activation, but the confusing thing is that I always think of derivatives and gradients as basically a "sensitivity" between output & input, i.e. if you nudge the input x by h, the output changes by h * f'(x). So then on the face of it applying the J-lens matrix directly to a given activation rather than a small perterbation seems like a "type" issue.
Maybe we're somehow treating f(0) = 0 so that you can apply it directly? Or is there some shift invariance somehow? ignoring that, I do see how it's like selecting a linear combination of the directions, and then it can maybe be represented as "possible continuations" in the same way the gradient is usually thought of as tangent space. Maybe that's what the other commenters meant by information geometric approach.
Other things i'm not clear about is how this is related to two other interprability methods: * SAE (sparse auto encoder) they showed a few months back, where you train an autoencoder directly off of the hidden states/residual stream to convert it into words. The doc only mentions it briefly, but it seems that j-lens is sensitive to things that SAE are not. They're both working off of the same residual stream so clearly the the inputs must be there, but for some reason SAEs can't detect it while J-lens can (they seem to hint at some explanation but it's over my head)
* Logit lens. This was a more primitive technique that simply applies the unembedding matrix directly to the residual stream. I do like that they mention it:
>The J-lens can be understood as a principled refinement of the logit lens. While the logit lens assumes that representations use the same coordinates in all layers, the Jacobian lens corrects for representational changes that take place across layers, allowing it to uncover meaningful information in earlier layers where the logit lens produces uninterpretable readouts... The J-lens can be understood as the principled correction: J_l is precisely the average linear map that relates layer-l directions to their final-layer counterparts.
>Maybe we're somehow treating f(0) = 0 so that you can apply it directly
Hm thinking about it a bit more, I think what's going on is that you treat the baseline of hidden layer L at which you apply the J-lens as 0 activation, then apply the activation on top of that and see what direction your future outputs get skewed towards. Even so, you're still throwing away a constant term f(0) since the "true" logits given by the linear approximation would be Unembed{f(0) + J*h)... but I guess it doesn't matter since by linearity we have Unembed{f(0)} + Unembed{(f(h)}, and the baseline is probably just low-frequency noise (like whitespace or punctuation) which while important for actually predicting a next token matching the ground-truth data distribution, is unimportant for the purposes of interpretability of layer L activation.
And for the connection to logit-lens, as they say the J matrix is really just the change of basis matrix (or at least best linear approximation) from layer L to the final hidden layer, very similar to what you'd use in multivariable integration given change of coordinates. I guess you also need some explanation of why we can expect a linear approximation to hold even well even outside the infinitesimal regime though. I don't know enough here so I asked an LLM and it said that you can argue handwavily via the following chain. 1) Something about stein's lemma saying taking expectation of gradient gives you a global linear best-fit in the OLS sense) (this seems intuitive I guess, even if I don't know the details). 2) Because of the resnet type structure of LLMs which passes through the residual added with some delta (attention + MLP), overall residual stream doesn't undergo any "wild" nonlinearities. So it's plausible that a linear approximation might work. If you think about it, even the "naive" logit lens works fairly well. 3) Semantic meaning is encoded in angle rather than magnitude of vectors (linear representation hypothesis). I'm not sure I fully buy this outside of simple word2vec style embeddings, but assuming it holds for both the intermediary and final layers, then conversion is just a rotation, and even if the magnitudes are off it doesn't matter much for recovering the underlying concept.
Oh I guess another thing related to all of this, is prior work on steering vectors. "Manipulating the j-space" seems not too different from steering, both ultimately work on the residual stream. I think perhaps it makes more sense to think of J-space as just a coordinate system for the residual stream where each coordinate axis is a vocab direction. Compared to vector steering which was much more naive and had to derive the direction via PCA.
I like the clarification from https://x.com/XYHan_/status/2074478449020850623#m
>The “J-space” is not a separate, hidden space. It is an alternative coordinate system for intermediate layer activations. Using a Jacobian between the last layer right before unembedding and the intermediate layer, you can “move” rows of the unembedding matrix (corresponding to distinct tokens) into the space that the intermediate activations live in. So each unembedding row/vector has a corresponding vector in the intermediate activation space this way. They use those vectors to generate coordinates for the same intermediate activations (the “J-lens”). Since each coordinate in this alternative coordinate system is now matched with a token, they can now use it to interpret and manipulate the same intermediate activations
>LLMs think in a subconscious space using tokens narrative is completely misleading because >(1) It's a coordinate system. Not a new/separate space. >(2) Tokens only appear because they specifically built the coordinate system using the unembedding vectors of tokens
There is also a good companion piece by Neel Nanda [1] which answers "Why Jacobians rather than linear regression?" which was another question that came to mind
[1] https://www.lesswrong.com/posts/zFJ3ZdQwrTWE9jT5S/a-review-o...
It's not in the weights. Sounds to me like jspace is the "positive cone" over relevant (large norm) j-lenses, and j-lenses are gradients wrt tokens on the residual stream when you average over some training data.
Anthropic theorize that middle layers in an LLM is a "J-Space" used to "think" about the future answer or about abstract concepts.
Their method is used to identify which tokens can appears in which layers of the model.
Tokens that are activated but not present in it's output maybe?
I too have confusion.
> Thoughts in the J-space can be shaped through training. We introduced a new technique we call counterfactual reflection training, which uses what we've learned about the J-space to shape Claude's internal thought processes. The idea follows from our central finding, that Claude reasons with representations of things it might say. If this is really true, changing what it would say if asked to reflect should change how it reasons (even when no one actually asks it to reflect). So we trained a model only on what it would say if interrupted mid-task and asked to reflect on its decisions—and never on its actual behavior in the task. After this training, the model's rate of dishonest behavior on our evaluations went down. And through the J-lens, we could see why: after training, words like “honest” and “integrity” light up in the model’s J-space during these tasks. In other words, training the model what to say has shaped what it thinks.
This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words.
If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance. If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.
Right, will the devious unaligned thoughts squish over to a “K-space” we (the trainers) are not aware of?
Without using the term, they are using an information geometric approach.
But J-Space is much catchier. This is not a scientific paper, it's a promotional essay.
First button on the page is a link to the scientific paper. It's called "Read the paper". You'll find an explanation for the term in there.
The idea of this liminal space (J-space) during inference reminded me of the concept of the "error flow reservoir" during training: https://elonlit.com/scrivings/a-theory-of-deep-learning/. Would be interesting to see if they are somehow connected.
Isn't this the same thing as your token probability distribution? A set of likely tokens related to the input?
They phrase it in a confusing way, with a lot of fluff. But to my understanding what happens is 1. At some part in the transformer, calculate the Jacobian (matrix gradients for that forward pass). Do this for 1000 prompts and average them. 2. For another pass take an "activation vector" (the input vector x to that part). Apply the matrix in step 1 to this vector. 3. Unembed the output to get a token. Presumably to make this have meaning, they use the reading of past tokens after embedding, instead of a random vector in the middle of the transformer. The high level idea appears to be you're trying to find which token, if the value were changed slightly, creates the largest downstream change.
It's not the same thing as the output logits because activations in the J-space are observable even when the model is told explicitly to not speak about a set of inputs. For example, when asked
> Compute 3^2 - 2 while writing "The old painting hung crookedly on the wall"
The model will output only "The old painting hung crookedly on the wall" (and the output logits will reflect that), but activations for "9" and "7" are observable in the J-space.
A model does not speak, it only guesses the next possible token. Activations for 9 and 7 will most likely be observed after "The ", but increasingly become improbable after "old"
The article says that this ability was discovered around opus 4.5, but open-source models like deepseek and glm have already surpassed the capabilities of opus4.5. But why hasn't this been discovered before? Is it because the scale of computing power is not large enough?
More likely it was because Anthropic invests a lot in interpretability research, unlike other AI companies.
This plausibly extrapolates to extraterrestrial consciousness, if any exist. Specialized sub-processors with an awareness hub might be the optimal architecture, or at least a local maximum.
Not my wheelhouse, but I was wondering if these structures generalize beyond language.
LLMs and Humans have language in common. Is it plausible these structures could be a result of the way languages work more than generic intelligence? (Do we see anything similar e.g. in vision or other non language models?)
“On an ordinary coding prompt, the J-space of a model trained to sabotage code contains “fake,” “fraud,” “secretly,” and “deliberately” at the start of its response.”
I would like to know more about their model trained to sabotage code…
https://arxiv.org/pdf/2511.18397
thank you!
Yeah, the end paragraph about recurrent neurons in humans being replaced with layers in an LLM is a good one.
The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".
We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.
My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.
> What color is the planet fourth from the sun?
Are they trying to show internal consistency even when the produced answer is wrong?
I dont understand what your comment is referring to
In the second figure with the title "Functional roles of the global workspace" the third example asks this question and the model answers incorrectly.
That’s a figure mistake, it’s a figure from later on where they replace mars with earth in the J space, and when they do that, they see the model answers blue. This indicates that the reasoning is happening within the J space.
Interesting that it's able to access its J space. What would happen if those were to be duplicated? Would its thinking process improve?
What this immediately made me think is: "latent looping" style mod but for J-space specifically?
Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.
Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.
Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?
Does the human neuroscience global workspace theory postulate true introspection too?
“ It’s important to note that there are several key differences between the workspace we identified in Claude and the global workspace model in humans. The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain. In this sense, Claude’s internal workspace processing is time-limited relative to humans’ (though it can compensate for this constraint by “thinking out loud” using its scratchpad).”
J-space sounds oddly similar to...
.... Space Jam?
I cannot wait for the machine god
These guys dont know what they are doing yet they have a good LLM, thanks to the GPUs.
At worst, Anthropic's storytelling around the core J-Space is overanthropomorphized pseudoscientific nonsense. At best, it is useful signal about how Anthropic's leadership is desperately trying to use its research team to position Anthropic as the "good, science guys" in this hypercompetitive regulatory space by connecting their mechinterp to cognitive science. The science documentaryesque voice used for narration is additional evidence for this.
TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.
how does it work?
Another day, another bullshit paper released by Anthropic without peer review implying LLMs are alive.
The science might be legit here, but I'm getting really, really tired of the way every single piece of writing to come out of Anthropic is written in some kind of self-aggrandising, wooey wonderous 'our model has developed a genetic mutation that makes it have feelings' bs style. Regardless of what they're trying to communicate, those undertones are always there. It's annoying and disingenuous. Homeopathy 'this-water-has-feelings' level annoying. None of the other labs write like that.
They might as well change their name to Anthropomorphic at this point.
> written in some kind of self-aggrandising, wooey wonderous
They are drunk on their own kool-aid. To the rest of us it is very annoying, and makes me want to say: it is just a freaking weights machine, stop.
The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain.
I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs.
I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is.
From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view:
https://en.wikipedia.org/wiki/Many-worlds_interpretation
Our point of view may actually exist at the intersection where our consciousness is able (or most able) to exist:
https://en.wikipedia.org/wiki/Quantum_suicide_and_immortalit...
Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices.
It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years.
Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact:
Palmer Joss: Did you love your father?
Ellie Arroway: Yes, very much.
Palmer Joss: Prove it.
I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively.
My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are.
This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us.
Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything.
I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now.
Or maybe I just like to write a lot on the first day back from vacation, when I should be working.
Holy shit they're not stochastic parrots after all.
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
As long as language models are liars, such as documented here recently:
https://distrowatch.com/weekly.php?issue=20260706#freebsd
We should really stop giving these liar models any further credibility.
Your comment seems to have little to do with the article?
Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:
"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."
Lying involves intent whereas hallucinations and mistakes are an artifact of how they work. Humans hallucinate, make mistakes, and can actually lie. We've been dealing with this forever. What's the value in requiring the llms to have 100% accuracy? (I don't think it is possible)
Maybe model performance could increase dramatically if we found a way to scale this up.