I work in this field (mech interp research) and I agree, the original post is definitely misleading. Some overselling in research is normal but it's a bit much.
First, the key result has nothing to do with making Qwen better at math, it just reduces reasoning loops.
Reasoning loops are a simple problem: the model generates the </think> token (which closes reasoning) with too low probability. You can fix this with simple steering approaches where you add a vector to different components of the LLM (https://arxiv.org/abs/2503.22048, https://arxiv.org/abs/2506.18167, etc). The method Astra used for identifying the vector is the standard procedure humans use (contrastive analysis). Such steering vector creation is basically the standard "intro-level interp project" for human researchers.
If you can code and understand LLM internals, you can do this yourself in 4-8 hours. The SWE equivalent would be doing binary patching with a debugger. Cool that frontier models can do this, but not unexpected since humans can do it without issue.
Speaking for myself. I’ve been aware of a lot of concepts around training, but stayed away from it until very recently.
A few weeks ago, I formed a hypothesis that I could get a smaller model good at the specific standards I have for my git commit messages, which are hard to get out of most LLMs today.
I spent the last four to five weeks on traditional training. It improved in a lot of areas, but not enough to be usable for me.
Yesterday, the article the author of this post is critical of gave me enough of an aha that maybe the problems I was seeing could be solved by having a larger LLM change the weights directly, without going through training.
In the last twelve-ish hours I’ve made more headway than I have in 2+ weeks. I’m using one of my last well performing checkpoints (full fine-tune + changes to the decision head/LoRA, etc.) and gave the model free rein to change any of the layers.
Its last two actions: (a) removed 16 participant-order directions from a head matrix, which improved one type of reasoning and degraded another, and (b) restored one of the 16, which reversed the degradation while keeping the core improvement.
All that to say, this might be known to people who’ve been around training for years, even decades, but it’s new to a lot of us. And we’re excited about it.
Would you be willing to share what the specific standards for your git commit messages are, and with what models you used that those standards couldn't be met?
Generally speaking you should be able to do something like that in AGENTS, I am really surprised that you have had an issue with most LLMs today getting them to follow a standard for SDLC. Personally, I like conventional commits a lot.
> Generally speaking you should be able to do something like that in AGENTS
I really wish that was true. All agents find some way to break the rules, and leave messages for their future selves.
The agents always find a way to include some forms of hedging, contradictions, future-looking documentation (which doesn't age well), etc. into git messages.
I found having an extra gate (husky/pre-commit hook), even if it's not as good as I want it to be improves the quality of how the AI writes substantially.
The cost of it not being very good is the agents now spend more time validating the claims of the validator. It's actually hilarious to witness.
I have been using local models for this final verification, which is partly why they're not that great. It's also why I decided to train my own.
It's been a great, fun learning experience. I learn almost everything via building. So why change a recipe that works :)
I have not had an experience anything like you are describing with Claude. I have found it to be the most reliable product I have ever purchased, actually. Learned it quickly, read the documentation, and it works.
Local models are not at all comparable to Claude to me, and I have used them: Qwen, DeepSeek, Minimax, Stepfun, Nemotron. None of them as good as Claude Fabel.
If your model is not adhering to AGENTS.md again I am really curious what exact model you are using. I am really just astonished that whatever model you are using is ignoring a clearly defined git conventions standards in your AGENTS.md in 2026.
> it’s new to a lot of us. And we’re excited about it.
Yes definitely.
Seems to me as if the weighs fit in the big model's context, it (the big model) reads all weights and doesn't need python.
But if the weights don't fit in context, then it uses python.
Isn't that pretty obvious in a way? That if too many weights, can't look at all. I don't think there's a need for OP to be dismissive when the AI uses python to look at a subset of the weights. It just has to?
To me, Astra understanding a 200k weights small model (the stuck car) is amazing enough already.
The model can only reason from the weights directly when there's a fairly linear relationship between the behaviour and the weight values. In a large model that's not remotely true.
Using Python to debug and fix problems with the model is totally different. It doesn't read in any of the weight values into the context at all when it uses the Python approach. Rather it gathers diagnostic statistics about the weights. It does just the same sort of thing a machine learning researcher would do to solve the problem.
So there are basically two unrelated types of experiments in the paper. In one type of experiment, the model reads the weights like it reads an image. In fact if you visualise the weights it's fairly clear why this works.
In the other type of experiment, the model is tasked with writing Python code to do ML engineering. This requires writing Python programs to analyse and manipulate weight data. This is a difficult coding problem but it really is just a coding problem. It doesn't "look" at the weights in these tasks any more than you or I would look at the weights.
> It does just the same sort of thing a machine learning researcher would do to solve the problem. [...] It doesn't "look" at the weights in these tasks any more than you or I would look at the weights.
Ok. Thanks for making that more clear!
> two unrelated types of experiments ...
The first type of experiment (small nets that seem to fit in context) are enough to make me impressed actually :-)
But yes then maybe the previous article could have been split in two, or two sections or something.
Wow. Are you saying you trained a smaller model from scratch (from what training data?), then used a larger model (which model?) to tweak its weights to fix this problem? Or did you post-train/fine-tune an existing model? I'm very curious what models, hardware, and training data you used!
> Are you saying you trained a smaller model from scratch
Not yet - but I really want to. Current experiments are on Qwen3.5-2B, and I've done a bunch on ModernBERT-large.
My main hypothesis is: I'm fairly confident I can come up with a university curriculum for a high school student to learn certain skills so that even without gaining technical knowledge, they can call bullshit on commit messages written by Claude and Codex. And I'm basically converting that curriculum into training.
Training a model from scratch is likely the next thing I'll do once I hit a significant bottle neck. There are still quite a few things to try (ex: Franken-merging, converting the dense model into a MoE, etc).
> very curious what models, hardware
Mostly using run pod and similar, and honestly, grabbing any decently sized GPU that I can get my hands on. 5090 - B200s.
> training data
Public data and synthetic data. This has been the hardest part. Also, why I'm excited about not having to "train" the behavior into the models.
I assume you run a benchmark on all the tasks on every change the model recommends? Do you have it set up like nanogpt speed run thingy that karpathy for a thumbs up or down for every change when you did this with astra?
>> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights
Wasn’t all that obvious to anyone who read the original post when they got to the section that explicitly explained what the author of this post is now summarizing?
Sure if you only read the title and brief at the top you’d walk away without the full picture though.
Given that the top comment yesterday said “this is insane […] I’m shook” I’d say no, it wasn’t all that obvious to everyone.
As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.
Reading an article takes more time than commenting, which takes more time than voting. So the typical life cycle of a HN post is that people vote on it without reading the article, comment without reading the article, vote on the comments without reading the article, and respond to comments without reading the article, until after some time people who've actually read the article show up in the comments with some clarifications, saving everyone else the bother of having to read it themselves.
If you want to escape the hype, reading posts with no upvotes and no comments on https://news.ycombinator.com/newest is an option, though somewhat hit-or-miss.
If you read comments about the paper it's pretty obvious people are misled. People think that the method read in Qwen weights directly to improve it. That's not what it did --- it authored a program to analyse and fix the weights.
What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.
but AI "authors programs" for everything ? Editing or looking at file usually ends up as a bunch of sed scripts running in shell rather than AI agent opening file directly, and that's pretty much the case for every operation
Not necessarily. Language models can do stuff—well, figure out or determine stuff—purely through inference. Give them the input on their context window and look at the output, no need to generate or run any programs.
You can take it further with a reasoning model and let it generate + consume a bunch of additional tokens before getting your output. Still not writing or running any programs.
To my understanding, that's how the examples of determining which game a small neutral net was trained to play. The weights of the neural net for into the context window, and then Astra was able to analyze them purely by reasoning rather than running external tools.
I wouldn't say misled, I would just say they made up their mind about what it means regardless of what the article said. Maybe the title is misleading, but the original article is pretty fair.
> it authored a program to analyse and fix the weights.
This is a difference without much distinction, at least to me. It's like being upset when someone says that Claude made a http call -- of course the LLM didn't make the call itself, it used a tool to do it.
It obviously is an important distinction to the paper's authors, which is why they used the modifier "native" to mark it as a distinct form of communication than what's already been on the market for years (coding agents). But it's not native, or rather some of the results were native but they mixed toy-scale native results with non-toy-scale non-native results for marketing purposes.
This is like someone telling you they made a "native" brain-to-brain interface but it's just a tin-can phone.
As someone who has been working in ML for 30+ years, and has designed and trained too many NNs to keep count, this is the first time I've seen anyone twiddling the weights directly. Usually you play with the training set, exemplar weights, model's hyperparameters, etc. I've never seen a trained network being modified by hand.
I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that
There’s some Dunning-Kruger effect going on. Many are looking to more knowledgeable commentators in hackernews for a digestible eli5 takeaway, myself included as the original article flew over my head.
Unfortunately it’s hard to distinguish who to trust in threads as many commentators sound overly confident and knowledgeable. They think they understand the article and its nuances, but they actually don’t and end up perpetuating false info.
At this moment on the front page (early morning in the US), this post is #1 and the other one is #2, so it's working like a warning about the other one being misleading before clicking on it.
Yeah, I was disappointed reading the title of this post because the work it is referencing was very exciting. But after reading this post, it's clear that the author is just pulling an "uhm, ackchyually" and responding to an abstract audience of laymen readers who don't know (or care) about any of the details of these results.
Anybody who actually read the original research site was fully aware of the details of the experiments because it's a well-written research site that made everything very clear. This author is clearly just jumping on the hype as a sort of contrarian where they were frustrated with how much excitement the research was generating and wanted to "correct the record" for the set of people who don't matter in this space.
LLMs are extremely hyped up right now. It's like when everyone was crypto-crazy, and threads were full of people who had no idea what was going on but were very excited.
Yes, while reading the blog I immediately thought about https://arxiv.org/html/2406.11717v3 paper. Original blog does actually overstates it findings as nothing like that exists.
I really liked that comment because before that I thought I was going crazy. A lot of the discussion around these experiments seemed to imply, if not outright state, that LLMs had developed an ability to natively understand and manipulate LLM weights, a sort of transcendental introspection into their own brains, which if true would be absolutely groundbreaking.
But then you look at this complete trace generated by Kimi 3.1 when it tries to unravel the Breakout weights, and you see that it's bog standard reverse engineering, like any human would do, just vastly faster:
Arguing that LLMs spontaneously evolve new capabilities outside of their training set (being able to naturally interpret weights and manipulate them to change their function) is a completely different proposition from being able to pattern-match Pac Man really fast from a pseudo-graphical representation (which is what they actually do here).
It's like ChatGPT coming up with a brand new brownie recipe that's surprisingly good vs. coming up with a working design for a cold fusion reactor.
That's the difference between me and Magnus Carlsen and Nigel Richards. Why downplay speed? I have a very, very hard time pinning down intelligence without time. I could basically do all IQ tests perfectly if I'm given infinite time, no? I never experienced one that is unsolvable, it's just that some people see it in a second or two and I and other lesser souls like me might need twenty minutes.
Edit: I'll add raw memory but yeah.. that's not hard to beat. Nigel's memory is insane as is Carlsen's, but compared to a computer it's nothing.
I'm not trying to downplay it, it's cool that LLMs can do this, but I feel like the discussion around genAI these days tend to quickly devolve into one of two extremes: complete dismissal (stochastic parrots) or near-AI-psychosis (LLMs are god machines that our puny human brains can't hope to comprehend).
In this particular instance, I also think that at the end of the day it's almost downplaying the capabilities of LLMs: they can reverse-engineer full modern videogames, of course they're capable of recognizing a simplified version of Breakout, even in weight form! It's barely newsworthy at this point, we know LLMs can do that stuff and so much more.
You are incorrect about the iq tests though, so everything else that depends on that falls over.
Some things you may get by looking longer, some things you just have to be shown, and only after being shown, maybe you can at least understand and re-use, but would never come up with from first principles and no hints or prior exposure.
How could I know that? There has already been essentially infinite normal-man-hours since people have started thinking, even when you remove all the people uninterested in doing anything but surviving. Today especially with the population in the billions and all of them having gobs of leisure time they never used to have. And yet we still have not solved much of anything that we know is out there to be solved, and still only an exceptionally rare genius once in a blue moon develops something new.
If mere time was the same, we have already had it, so why aren't we immortal omnipotent energy beings? Why do we still have colds?
Working time alone is a productive strategy for many problems. A machine that can brute force all possible chess moves beats the most genius human player, yet isn't intelligent at all, or rather, requires no intelligence at all. That machine will produce the exact same results either fast or slow depending on how much hardware you give it, and in humans, the genius player produces their answer faster.
But that's no more meaningful than observing that humans often turn simple mechanical cranks, and electric motors do to, and concluding that there is no ultimately significant difference.
Humans can spot patterns in data, and machines can spot patterns in data. Genius humans can do it faster and machines can do it even faster. But that is still just a simple task like pedalling a bicycle or calculating, that a human can do if they choose, but does not require a human and isn't any part of what defines human or intelligence or sentience.
Being mystified by that is like being mystified that an mp3 player said "hello" to you.
The fact that for those tasks, low capacity over longer time can usually produce the same result as more capacity over a shorter time means nothing.
Consider also, today lots of people have both lots of free time and a high interest in solving problems, and the most genius people that ever lived are actually only a little bit more capable than the average in absolute terms (which is probably where you get the feeling that more working time is the same as more ability), so where are all the free time hobby Einsteins?. There should be countless people coming up with new theories of relativity (that actually hold water and predict experiments) simply because they spent 5 years on something Einstein did in a month. Countless people have 5 years to plug away at things like that, and they do, and have been forever. There is some of that in the sense that there is a lot of tedious work being done that produces useful results, but not new insights.
AI's can now do some of that same tedious work only faster, except they are still strictly worse than the most average human because they can't decide what to work on, why to work on it, or judge the quality or usefullness or point of the result. They produce something but they don't produce understanding.
I'm still impressed by the "Weight-Space Programming" idea. Perhaps its because I didn't know that LLMs could understand or successfully modify weights at all previous to reading it. However, I think the examples provided in the original piece were eye-opening. Yes, humans can build tooling to modify weights, but the author here agrees that it's tough. The examples where Astra was interacting with weights are still novel, and Astra seemed to handle them with surprising ease.
So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.
There is something about AI related posts that makes it irresistible for people to post a “rebuttal” that actually makes the same claim as the original post just with different vibes. It feels like a lot of discourse is spent on setting the vibe over actually adding anything to the conversation.
But theoretically, what would prevent the labs from RL'ing the models on "weight-space programming" tasks so that they get better at them? Perhaps we're just at the beginning, like how coding agents sucked 2 years ago.
It still seems like a promising approach for RSI, distillation, fine-tuning, and a whole host of other tasks.
It's impossible to train on raw document > 1MB (I don't know the exact limit, but there is one).
You can't vectorize weights (or can you?!) and the output context is limited.
So you can't train a big model on it's previous weight or others models weights.
The output context can be "extended" using agents but I doubt anything coherent may be outputted using that technique.
Of course this is the problem. It's amazing that for all the commentary around AI models the number of people who really understand even the basics approaches zero.
It seems some level of interpretability, even if it requires tooling, could result in the ability to spin out synthetic training data sets from other models in an automated fashion.
I know you're being sarcastic, but I meant prompting an LLM with "generate me a X billion parameter model for doing Y task" and it would just output the weights.
They make it sound like they improve Qwen by reading raw weights. They don't. In fact the experiments that use raw weights have very little to do with the experiments that work on larger models.
The paper makes it ridiculously clear what they do and don’t do. And they make the reasons why clear too. There is absolutely nothing interesting in this new article.
I don't understand how anyone can read the abstract and think it's "ridiculously clear" that the "Just by looking at the raw weights," doesn't apply to the Qwen experiment.
You are right, if you literally just read a one paragraph description you will get less details than reading the whole post, but most of us read more than just the one paragraph. There is some actual interesting stuff in the paper, including the portion where they did edit the weights to fix the looping bug.
It says this “All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly.”
This seems 100% accurate to me. What is a lie here?
Some of original blog post really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
So I do agree it is misleading and trying to present work as groundbreaking.
The post wasn’t misleading, the usual horde of online clickbait it generated was misleading. That isn’t on the researchers. Their writeup was very detailed and informative. The problem is that there’s no room for nuanced discussion on any online forum. People seek out and amplify low quality drama and then complain about it.
I pretty much got that idea when reading the post, on how it didnt have improvements on a Gemma 270M model which didn't have any looping issue. However, it still made me look at some bottlenecks in models I had trained (like MLP network for stylus input automation and a 17M reranker with NLI stance and noul heads) and allowed me to finetune without training, but just by adjusting some weights. But what's pretty cool is the zero shot sandbox development method they showed for a Go model. I tried to do that for a 17M re-ranker from zero weights and Astra built it using its reasoning without me having to use a GPU and find training datasets that are not CC by NC, and add some hours of tweaking and checkpoints. I don't know if its the best way to build something, but we might be building small models (maybe not LLMs) like with just programming the weights instead of finetuning.
In the end I got to replace a 68M model with a 17M model, and improve stlyus tool recognition accuracy from 88 to 96%. Was a pretty fun experiment too.
Recently, as has probably been the case in other fields as well, many people feed the public cherry-picked results to hype up what AI can do or to stir up FOMO.
I used to think that in the dev community this sort of thing was less common than elsewhere... because people want to share their experience, what they found, etc.
Now that AI is here, people's experience and knowledge have basically been reset. So it's no different here now, which is a shame (I'm not saying it was better before, though)
>The 'Astra can read and write weights to improve Qwen' post is misleading
Does the author mean the "LLMs can natively read & write neural network weights" post? Did the researchers change the name or something? Because when I read it, it made it very clear what they were doing and none of it was misleading. It all aligns with what this author is saying.
Why this post is here in the front page? The Qwen vector steering performed was a very selective operation that interested only specific layers and weights set, it is non trivial to identify and fix. This is remarkable. Also Experiment 3: no access to forward pass, so why here this post says the contrary of what the paper shows and everybody upvoted it? I don't understand.
My complaint is that the post presents a confusing mix of experiments that imply capabilities that are more than the sum of their parts. This experiment 3 has nothing to do with experiment 1 for instance. In experiment 1 the model does read in the raw weights like it's reading an image.
If you don't like these experiments, then I can only assume you would complain about CRISPR as well, considering it's a general framework with many disparate parts that work together synergistically, similar to the work Takehi et al. describe. CRISPR was also famously rejected multiple times by reviewers.
Note that being able to both inspect raw weights and run Python code to perform additional analysis means the model has multiple possible techniques at its disposal (generally considered a positive, rather than a "confusing mix").
I don't understand CRISPR well enough to comment on the analogy.
The blog post claims to show that frontier LLMs can understand and craft neural network weights natively. The experiments where the model completes coding tasks are not relevant to this conclusion. Those experiments require the model to demonstrate an analytic understanding of how a network might behave under various conditions, along with reasoning skills, coding skills etc. As a coding task this is a significant challenge. But it's entirely different from "natively reading and writing neural network weights".
Would you also consider it a benefit if there were experiments showing that a frontier model can download and run PyTorch, or write its own network and optimiser? That's another technique it has at its disposal to improve an ML model. It's just not one that's relevant to the paper's core claims.
You know exactly why... because anti-AI hysteria is quite popular on HN.
Any "critique" or "appeal to resist AI" rooted in emotion and inability to admit that intelligence is not a feat unique to humans anymore will make the rounds here.
Agree, it makes sense, a lot of people are catching on to LLMs for reverse engineering finally, but its been possible. LLMs can and will eventually flawlessly dissect models. I experiment with Claude, I ask it tasks I know are possible, but have no idea the solution, it can reverse engineer obfuscated projects faster than most talented people I know in that field (talking about games now not AI). I heard a developer say their latest release would take at least a month to figure out the obfuscation and encryption, Opus 5 did it in like... 15 minutes of talking to me (I did throw it the right questions / suggestions). I do that all in the name of seeing just exactly what is possible.
Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.
If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
> Because this is very much a part of what ML engineers do.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!
I think you misunderstand. Yet the model is doing something impressive still by writing code to fix model weights on problems. But that's a whole different ballgame than just reading weights and knowing what to change. That would mean that they have an internal representation of how the model weights map to information or a specific task, which if true, throws all the interpretability research out of the window
To me, the original article is more about what LLMs can *do* in weight space, I don't really see the part where it would tie to an internal representation.
It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.
That "Likewise" is quite load bearing in the original article :)
So in effect, all what we're quarreling about here is whether it is _misleading_ to use "Likewise" to transition between two paragraphs without implying that the details of the first apply to the second?
Just like Graduate Students are good at Gradient Descent, they can see patterns and fine tune parameters 100000 times until the result looks good. So too can LLMs with their ability to examine patterns and brute force things.
I completely agree with you Matthew, but I see the thing go in right direction in research as well, I am working in the same field where I am building an agent where it will improve the base model too !
Agreed. Not sure it leads to anything truly groundbreaking, but more than likely will lead to very fascinating implementations that lead to even more interesting results 6-8 months from now. Think openclaw -> how people utilize now versus when it was first announced
It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
With normal training you're simply training for higher loss for things you don't want while keeping the loss low for things you do. Same would apply here except you would target low eval for behavior you do not want the model to have rather than trying to increase eval.
> how you could prune harmful capabilities from models and build a more efficient training system
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
What it shows are two unrelated results mixed together for no apparent reason besides marketing. The raw-weight results are impressive, but haven't been demonstrated as ready to scale up to a real language model. The Qwen result is a mildly interesting application of functionality that's been on the market for years.
If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.
I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
something i found 15 minutes ago is that the car network test is "very very easy", as in Opus 5.5 in Claude Code did not even need to see the weights of the network to solve it correctly. i can give the transcript of the session. i'm pretty sure Opus didn't cheat
(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)
Welcome to publications involving ML in 2026... Glad to see someone else write this though because it's important to start having these discussions. Too much overselling has been happening for too long. There is so much great about ML. IMO it hurts the field long-term to continue overselling claims like this.
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
I think the reviews are flawed for most of journals. But I can't really blame them, there is huge influx of new papers comming everyday. Unfortunately reviewers are more biased towards groundbreaking stuff so it seems they can't make connections with existing and prior work.
They are also biased to see from what institution you come from..
A better idea is to have Astra modify it's own weights and let it try if it can RSI, but you gonna need to closely monitor for actual leap in capabilities so you can keep a check on alignment.
This isn't a good look. The original article clearly states what the model was doing, and this new hit piece is supposed to land because they are straw manning the comments on the original post? This isn't reddit dude.
This could have been a comment on the original article.
> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.
Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.
A tangent, but I’m now realizing that the “recursive self improvement” thing that these LLM companies are talking about is (probably?) not about unsupervised Claude Code instances improving the training process, but rather LLMs modifying their own weights through tooling directly. It would certainly open up a whole new process for model optimization.
I was just watching a video about this. They had this survey question for AI researchers asking when they think AI would take their job. The reason they asked this was that would be the point where AI starts recursive self improvement. If it’s able to replace the job of an AI researcher, then it could keep improving itself.
I don't know how much we learn by surveying working "AI researchers". By their own admission most of them have a religious belief that they will bring about a singularity in the next few years. It's like asking a billionaire if he earned his wealth; regardless of what you think the answer is, you pretty much know how they will answer ahead of time.
Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).
Yes but my point is that I never realized that the AI would be writing tools to update the weights directly, rather than improve the existing training process.
The blog post exhibits the same flawed logic that many academic reviews do — looking for reasons to reject a work rather than accept it. It comes off cynical and contrarian. It's easy to find issues with an academic paper, but ACL has consistently pushed for a higher standard [1].
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
You've expressed your criticism plainly without addressing anything the post might've done well. I think that's absolutely fine, but why would your criticism apply to the post but not your own comment? Personally I think it's because the standards of peer review don't apply to HN comments, but I don't think they apply to blog posts either.
Considering I have a PhD in CS focused in NLP and have been a Senior Area Chair for ACL, yes I do know how academia works. And, no this isn't science working; it's the same plague infecting most of academic blogging and social media. People are vying for attention in whatever way possible, often at the expense of others.
And the examples in the OP's blog post just go to show that reading comprehension is at an all time low:
> Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
The table just below shows this statement to be *entirely* factual.
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
Any further implicit assumptions are on the reader, not the writer, as they never stated "that the Qwen experiment worked ‘just by looking at the raw weights’". You can certainly point out issues in the experimental design, but just stating you dislike the set of experiments and is yet another poor take that directly contradicts ACL's guidance for reviewing, exactly because it does not advance science.
All I can say is I very much disagree, and if I had authored the post and it had received the reaction from people that it has I would be anxious to issue corrections. I haven't reviewed for conferences for several years now, but I wouldn't have thought this should pass review.
I believe the post definitely plays off the implication that the Qwen experiments are connected to the experiments that read the raw weights and have no Python harness, when this isn't the case. There are experiments where Astra reads weights directly for very simple models, and there are experiments where Astra performs ML research tasks by analysing the problem and writing Python code. The authors imply that the same technique underlies these but that isn't the case.
I work in this field (mech interp research) and I agree, the original post is definitely misleading. Some overselling in research is normal but it's a bit much.
First, the key result has nothing to do with making Qwen better at math, it just reduces reasoning loops.
Reasoning loops are a simple problem: the model generates the </think> token (which closes reasoning) with too low probability. You can fix this with simple steering approaches where you add a vector to different components of the LLM (https://arxiv.org/abs/2503.22048, https://arxiv.org/abs/2506.18167, etc). The method Astra used for identifying the vector is the standard procedure humans use (contrastive analysis). Such steering vector creation is basically the standard "intro-level interp project" for human researchers.
If you can code and understand LLM internals, you can do this yourself in 4-8 hours. The SWE equivalent would be doing binary patching with a debugger. Cool that frontier models can do this, but not unexpected since humans can do it without issue.
Any idea why Alibaba might ship a model while aware that it has this defect when it is so easy to fix?
Is this looping tough to fix by changing the training?
Steering/model weight lobotomy usually have a ton of side effects. You fix one defect and add 10 others.
This was my impression on the math result too.
Do you have thoughts on experiments one through five though? The direct one-shot reading/writing of weights seems pretty surprising.
Speaking for myself. I’ve been aware of a lot of concepts around training, but stayed away from it until very recently.
A few weeks ago, I formed a hypothesis that I could get a smaller model good at the specific standards I have for my git commit messages, which are hard to get out of most LLMs today.
I spent the last four to five weeks on traditional training. It improved in a lot of areas, but not enough to be usable for me.
Yesterday, the article the author of this post is critical of gave me enough of an aha that maybe the problems I was seeing could be solved by having a larger LLM change the weights directly, without going through training.
In the last twelve-ish hours I’ve made more headway than I have in 2+ weeks. I’m using one of my last well performing checkpoints (full fine-tune + changes to the decision head/LoRA, etc.) and gave the model free rein to change any of the layers.
Its last two actions: (a) removed 16 participant-order directions from a head matrix, which improved one type of reasoning and degraded another, and (b) restored one of the 16, which reversed the degradation while keeping the core improvement.
All that to say, this might be known to people who’ve been around training for years, even decades, but it’s new to a lot of us. And we’re excited about it.
Would you be willing to share what the specific standards for your git commit messages are, and with what models you used that those standards couldn't be met?
Generally speaking you should be able to do something like that in AGENTS, I am really surprised that you have had an issue with most LLMs today getting them to follow a standard for SDLC. Personally, I like conventional commits a lot.
> Generally speaking you should be able to do something like that in AGENTS
I really wish that was true. All agents find some way to break the rules, and leave messages for their future selves.
The agents always find a way to include some forms of hedging, contradictions, future-looking documentation (which doesn't age well), etc. into git messages.
I found having an extra gate (husky/pre-commit hook), even if it's not as good as I want it to be improves the quality of how the AI writes substantially.
The cost of it not being very good is the agents now spend more time validating the claims of the validator. It's actually hilarious to witness.
I have been using local models for this final verification, which is partly why they're not that great. It's also why I decided to train my own.
It's been a great, fun learning experience. I learn almost everything via building. So why change a recipe that works :)
I have not had an experience anything like you are describing with Claude. I have found it to be the most reliable product I have ever purchased, actually. Learned it quickly, read the documentation, and it works.
Local models are not at all comparable to Claude to me, and I have used them: Qwen, DeepSeek, Minimax, Stepfun, Nemotron. None of them as good as Claude Fabel.
If your model is not adhering to AGENTS.md again I am really curious what exact model you are using. I am really just astonished that whatever model you are using is ignoring a clearly defined git conventions standards in your AGENTS.md in 2026.
> it’s new to a lot of us. And we’re excited about it.
Yes definitely.
Seems to me as if the weighs fit in the big model's context, it (the big model) reads all weights and doesn't need python.
But if the weights don't fit in context, then it uses python.
Isn't that pretty obvious in a way? That if too many weights, can't look at all. I don't think there's a need for OP to be dismissive when the AI uses python to look at a subset of the weights. It just has to?
To me, Astra understanding a 200k weights small model (the stuck car) is amazing enough already.
No that's not a good summary.
The model can only reason from the weights directly when there's a fairly linear relationship between the behaviour and the weight values. In a large model that's not remotely true.
Using Python to debug and fix problems with the model is totally different. It doesn't read in any of the weight values into the context at all when it uses the Python approach. Rather it gathers diagnostic statistics about the weights. It does just the same sort of thing a machine learning researcher would do to solve the problem.
So there are basically two unrelated types of experiments in the paper. In one type of experiment, the model reads the weights like it reads an image. In fact if you visualise the weights it's fairly clear why this works.
In the other type of experiment, the model is tasked with writing Python code to do ML engineering. This requires writing Python programs to analyse and manipulate weight data. This is a difficult coding problem but it really is just a coding problem. It doesn't "look" at the weights in these tasks any more than you or I would look at the weights.
> It does just the same sort of thing a machine learning researcher would do to solve the problem. [...] It doesn't "look" at the weights in these tasks any more than you or I would look at the weights.
Ok. Thanks for making that more clear!
> two unrelated types of experiments ...
The first type of experiment (small nets that seem to fit in context) are enough to make me impressed actually :-)
But yes then maybe the previous article could have been split in two, or two sections or something.
Wow. Are you saying you trained a smaller model from scratch (from what training data?), then used a larger model (which model?) to tweak its weights to fix this problem? Or did you post-train/fine-tune an existing model? I'm very curious what models, hardware, and training data you used!
> Are you saying you trained a smaller model from scratch
Not yet - but I really want to. Current experiments are on Qwen3.5-2B, and I've done a bunch on ModernBERT-large.
My main hypothesis is: I'm fairly confident I can come up with a university curriculum for a high school student to learn certain skills so that even without gaining technical knowledge, they can call bullshit on commit messages written by Claude and Codex. And I'm basically converting that curriculum into training.
Training a model from scratch is likely the next thing I'll do once I hit a significant bottle neck. There are still quite a few things to try (ex: Franken-merging, converting the dense model into a MoE, etc).
> very curious what models, hardware
Mostly using run pod and similar, and honestly, grabbing any decently sized GPU that I can get my hands on. 5090 - B200s.
> training data
Public data and synthetic data. This has been the hardest part. Also, why I'm excited about not having to "train" the behavior into the models.
I assume you run a benchmark on all the tasks on every change the model recommends? Do you have it set up like nanogpt speed run thingy that karpathy for a thumbs up or down for every change when you did this with astra?
From the Fine Article you're calling critical:
>> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights
Out of interest, did you use an assistant like Claude Code or Codex to help with the training experiments?
Wasn’t all that obvious to anyone who read the original post when they got to the section that explicitly explained what the author of this post is now summarizing?
Sure if you only read the title and brief at the top you’d walk away without the full picture though.
This is a better response imo: https://news.ycombinator.com/item?id=50344861
Given that the top comment yesterday said “this is insane […] I’m shook” I’d say no, it wasn’t all that obvious to everyone.
As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.
Reading an article takes more time than commenting, which takes more time than voting. So the typical life cycle of a HN post is that people vote on it without reading the article, comment without reading the article, vote on the comments without reading the article, and respond to comments without reading the article, until after some time people who've actually read the article show up in the comments with some clarifications, saving everyone else the bother of having to read it themselves.
If you want to escape the hype, reading posts with no upvotes and no comments on https://news.ycombinator.com/newest is an option, though somewhat hit-or-miss.
Gotta get to the IPO, whatever it takes.
If you read comments about the paper it's pretty obvious people are misled. People think that the method read in Qwen weights directly to improve it. That's not what it did --- it authored a program to analyse and fix the weights.
What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.
but AI "authors programs" for everything ? Editing or looking at file usually ends up as a bunch of sed scripts running in shell rather than AI agent opening file directly, and that's pretty much the case for every operation
Not necessarily. Language models can do stuff—well, figure out or determine stuff—purely through inference. Give them the input on their context window and look at the output, no need to generate or run any programs.
You can take it further with a reasoning model and let it generate + consume a bunch of additional tokens before getting your output. Still not writing or running any programs.
To my understanding, that's how the examples of determining which game a small neutral net was trained to play. The weights of the neural net for into the context window, and then Astra was able to analyze them purely by reasoning rather than running external tools.
Did you really insert emdashes in the first sentence yourself?
Yes. I've been abusing em-dashes over a decade and like them too much to give them up now :)
They're pretty easy to type on Android keyboards and on macOS (option + shift + -). You can set up a similar compose key thing on Linux.
I wouldn't say misled, I would just say they made up their mind about what it means regardless of what the article said. Maybe the title is misleading, but the original article is pretty fair.
> it authored a program to analyse and fix the weights.
This is a difference without much distinction, at least to me. It's like being upset when someone says that Claude made a http call -- of course the LLM didn't make the call itself, it used a tool to do it.
No it's a huge difference. Someone who can quickly compute 123^456 in his head has a very different skillset to someone who uses a calculator.
It obviously is an important distinction to the paper's authors, which is why they used the modifier "native" to mark it as a distinct form of communication than what's already been on the market for years (coding agents). But it's not native, or rather some of the results were native but they mixed toy-scale native results with non-toy-scale non-native results for marketing purposes.
This is like someone telling you they made a "native" brain-to-brain interface but it's just a tin-can phone.
As someone who has been working in ML for 30+ years, and has designed and trained too many NNs to keep count, this is the first time I've seen anyone twiddling the weights directly. Usually you play with the training set, exemplar weights, model's hyperparameters, etc. I've never seen a trained network being modified by hand.
> who read the original post
I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that
There’s some Dunning-Kruger effect going on. Many are looking to more knowledgeable commentators in hackernews for a digestible eli5 takeaway, myself included as the original article flew over my head.
Unfortunately it’s hard to distinguish who to trust in threads as many commentators sound overly confident and knowledgeable. They think they understand the article and its nuances, but they actually don’t and end up perpetuating false info.
At this moment on the front page (early morning in the US), this post is #1 and the other one is #2, so it's working like a warning about the other one being misleading before clicking on it.
Yes, I'm glad I waited for this post to arrive before even delving into the topic.
So you mean that the "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading? You should post that.
Yeah. The article is not misleading. The title is, but you know, headlines are all lies that's why you read the article.
>but you know, headlines are all lies that's why you read the article.
I'm fine with throwing the baby out with the bathwater in that case and just calling the whole article misleading.
The title is: "LLMs can natively read & write neural network weights"
The title is part of “The post”.
Yeah, I was disappointed reading the title of this post because the work it is referencing was very exciting. But after reading this post, it's clear that the author is just pulling an "uhm, ackchyually" and responding to an abstract audience of laymen readers who don't know (or care) about any of the details of these results.
Anybody who actually read the original research site was fully aware of the details of the experiments because it's a well-written research site that made everything very clear. This author is clearly just jumping on the hype as a sort of contrarian where they were frustrated with how much excitement the research was generating and wanted to "correct the record" for the set of people who don't matter in this space.
LLMs are extremely hyped up right now. It's like when everyone was crypto-crazy, and threads were full of people who had no idea what was going on but were very excited.
I feel like a lot of this Astra editing Qwen stuff is regurgitation of existing relatively well known techniques, for example: https://arxiv.org/html/2406.11717v3 and conceptually similar to: https://arxiv.org/abs/2512.05648 (tho this is done in training iirc)
Yes, while reading the blog I immediately thought about https://arxiv.org/html/2406.11717v3 paper. Original blog does actually overstates it findings as nothing like that exists.
There was some really good comments by @yurimo on the 'Astra can read and write weights to improve Qwen' hacker news post https://news.ycombinator.com/item?id=50341174
I really liked that comment because before that I thought I was going crazy. A lot of the discussion around these experiments seemed to imply, if not outright state, that LLMs had developed an ability to natively understand and manipulate LLM weights, a sort of transcendental introspection into their own brains, which if true would be absolutely groundbreaking.
But then you look at this complete trace generated by Kimi 3.1 when it tries to unravel the Breakout weights, and you see that it's bog standard reverse engineering, like any human would do, just vastly faster:
https://pastebin.com/G7VCWZHn
>This looks exactly Breakout: paddle agent at bottom row (row8), ball moving object anywhere, bricks rewards top rows, previous frame for ball movement. Agent weights bottom row, moving object weights rows1-7 (not row8? ball can't enter paddle row? Actually ball can), reward weights top rows1-3, obstacle walls. Previous frame used to infer ball velocity. No-op and directional actions move paddle left/right; up/down useless. Output biases similar all actions but W3 likely left/right structure. Could be Breakout. Should scrutinize output weights: actions 0 no-op,1 up,2 down,3 left,4 right. If Breakout, only left/right useful; network output weights should show action3 and4 distinctive and actions0/1/2 similar. Indeed W3 rows show patterns: [...]
> ...imply ... a sort of transcendental introspection into their own brains, which if true would be absolutely groundbreaking.
> but ... it's bog standard reverse engineering, like any human would do, just vastly faster
For some values of vastly faster these are functionally equivalent and so equivalently groundbreaking.
Arguing that LLMs spontaneously evolve new capabilities outside of their training set (being able to naturally interpret weights and manipulate them to change their function) is a completely different proposition from being able to pattern-match Pac Man really fast from a pseudo-graphical representation (which is what they actually do here).
It's like ChatGPT coming up with a brand new brownie recipe that's surprisingly good vs. coming up with a working design for a cold fusion reactor.
Yeah the brownie is cool, but it's not that cool.
If faster intelligence was “groundbreaking” we would still be worshipping pocket calculators.
If by "pocket calculator" you mean linear algebra machines that do our work, then yes, we are doing that.
I’m personally a huge fan of viewing LLMs as linear algebra machines. They seem a lot less fascinating when you do that.
What's an alternative to the more complete, correct description, "Linear algebra machines that can learn nonlinear functions?"
Witchcraft?
> like any human would do, just vastly faster
That's the difference between me and Magnus Carlsen and Nigel Richards. Why downplay speed? I have a very, very hard time pinning down intelligence without time. I could basically do all IQ tests perfectly if I'm given infinite time, no? I never experienced one that is unsolvable, it's just that some people see it in a second or two and I and other lesser souls like me might need twenty minutes.
Edit: I'll add raw memory but yeah.. that's not hard to beat. Nigel's memory is insane as is Carlsen's, but compared to a computer it's nothing.
I'm not trying to downplay it, it's cool that LLMs can do this, but I feel like the discussion around genAI these days tend to quickly devolve into one of two extremes: complete dismissal (stochastic parrots) or near-AI-psychosis (LLMs are god machines that our puny human brains can't hope to comprehend).
In this particular instance, I also think that at the end of the day it's almost downplaying the capabilities of LLMs: they can reverse-engineer full modern videogames, of course they're capable of recognizing a simplified version of Breakout, even in weight form! It's barely newsworthy at this point, we know LLMs can do that stuff and so much more.
You are incorrect about the iq tests though, so everything else that depends on that falls over.
Some things you may get by looking longer, some things you just have to be shown, and only after being shown, maybe you can at least understand and re-use, but would never come up with from first principles and no hints or prior exposure.
How could I know that? There has already been essentially infinite normal-man-hours since people have started thinking, even when you remove all the people uninterested in doing anything but surviving. Today especially with the population in the billions and all of them having gobs of leisure time they never used to have. And yet we still have not solved much of anything that we know is out there to be solved, and still only an exceptionally rare genius once in a blue moon develops something new.
If mere time was the same, we have already had it, so why aren't we immortal omnipotent energy beings? Why do we still have colds?
Working time alone is a productive strategy for many problems. A machine that can brute force all possible chess moves beats the most genius human player, yet isn't intelligent at all, or rather, requires no intelligence at all. That machine will produce the exact same results either fast or slow depending on how much hardware you give it, and in humans, the genius player produces their answer faster.
But that's no more meaningful than observing that humans often turn simple mechanical cranks, and electric motors do to, and concluding that there is no ultimately significant difference.
Humans can spot patterns in data, and machines can spot patterns in data. Genius humans can do it faster and machines can do it even faster. But that is still just a simple task like pedalling a bicycle or calculating, that a human can do if they choose, but does not require a human and isn't any part of what defines human or intelligence or sentience.
Being mystified by that is like being mystified that an mp3 player said "hello" to you.
The fact that for those tasks, low capacity over longer time can usually produce the same result as more capacity over a shorter time means nothing.
Consider also, today lots of people have both lots of free time and a high interest in solving problems, and the most genius people that ever lived are actually only a little bit more capable than the average in absolute terms (which is probably where you get the feeling that more working time is the same as more ability), so where are all the free time hobby Einsteins?. There should be countless people coming up with new theories of relativity (that actually hold water and predict experiments) simply because they spent 5 years on something Einstein did in a month. Countless people have 5 years to plug away at things like that, and they do, and have been forever. There is some of that in the sense that there is a lot of tedious work being done that produces useful results, but not new insights.
AI's can now do some of that same tedious work only faster, except they are still strictly worse than the most average human because they can't decide what to work on, why to work on it, or judge the quality or usefullness or point of the result. They produce something but they don't produce understanding.
I'm still impressed by the "Weight-Space Programming" idea. Perhaps its because I didn't know that LLMs could understand or successfully modify weights at all previous to reading it. However, I think the examples provided in the original piece were eye-opening. Yes, humans can build tooling to modify weights, but the author here agrees that it's tough. The examples where Astra was interacting with weights are still novel, and Astra seemed to handle them with surprising ease.
Here's the work that shows how rule-based logic can be compiled into transformer weights:
* Thinking Like Transformers” by Weiss, Goldberg & Yahav (ICML 2021).: https://arxiv.org/abs/2106.06981
* Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 (code at https://github.com/deepmind/tracr)
So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.
There is something about AI related posts that makes it irresistible for people to post a “rebuttal” that actually makes the same claim as the original post just with different vibes. It feels like a lot of discourse is spent on setting the vibe over actually adding anything to the conversation.
But theoretically, what would prevent the labs from RL'ing the models on "weight-space programming" tasks so that they get better at them? Perhaps we're just at the beginning, like how coding agents sucked 2 years ago.
It still seems like a promising approach for RSI, distillation, fine-tuning, and a whole host of other tasks.
The context.
It's impossible to train on raw document > 1MB (I don't know the exact limit, but there is one). You can't vectorize weights (or can you?!) and the output context is limited. So you can't train a big model on it's previous weight or others models weights.
The output context can be "extended" using agents but I doubt anything coherent may be outputted using that technique.
Of course this is the problem. It's amazing that for all the commentary around AI models the number of people who really understand even the basics approaches zero.
Yeah, this was my thought (as a layman) as well.
It seems some level of interpretability, even if it requires tooling, could result in the ability to spin out synthetic training data sets from other models in an automated fashion.
They might even discover some way to program the weights proportional to an error signal
Now that would be crazy
I know you're being sarcastic, but I meant prompting an LLM with "generate me a X billion parameter model for doing Y task" and it would just output the weights.
Now THAT would be crazy
The original post explains what they did and why better than this post. This post taught me nothing that wasn’t in the original.
They make it sound like they improve Qwen by reading raw weights. They don't. In fact the experiments that use raw weights have very little to do with the experiments that work on larger models.
The paper makes it ridiculously clear what they do and don’t do. And they make the reasons why clear too. There is absolutely nothing interesting in this new article.
I don't understand how anyone can read the abstract and think it's "ridiculously clear" that the "Just by looking at the raw weights," doesn't apply to the Qwen experiment.
You are right, if you literally just read a one paragraph description you will get less details than reading the whole post, but most of us read more than just the one paragraph. There is some actual interesting stuff in the paper, including the portion where they did edit the weights to fix the looping bug.
> most of us read more than just the one paragraph
Most of us barely read the title
Honestly, that's on them. Coddling people who are too intellectually lazy to read the entire article isn't helping anyone.
It says this “All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly.”
This seems 100% accurate to me. What is a lie here?
Nobody said they outright “lied”, most misleading statements are “100% accurate” yet are designed to mislead the reader.
And when you see the reactions in the HN comment to the original article, it's pretty clear that most people have indeed been mislead.
The article is not misleading either.
The title is but then OP should talk about just the title instead
The title and the abstract are, the article is fine (but disappointing, given the excitement built up by the title + abstract)
>They make it sound like they improve Qwen by reading raw weights.
Weird, because they explicitly and clearly explain exactly what they did and didn't do.
Did you actually read the original post or just take the word of some HN comments at face value?
They did, except they also did write an abstract and a title that will mislead anyone who isn't going to read the full paper in details.
It's fair to say that this is malpractice.
I just reread both and neither seem misleading to me. If anything the title is underwhelming.
Some of original blog post really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
So I do agree it is misleading and trying to present work as groundbreaking.
The post wasn’t misleading, the usual horde of online clickbait it generated was misleading. That isn’t on the researchers. Their writeup was very detailed and informative. The problem is that there’s no room for nuanced discussion on any online forum. People seek out and amplify low quality drama and then complain about it.
Do you not think that it’s cool / unsettling how in theory we can just “ask” for certain behaviours etc to be added / removed?
I’ve experimented with this today and have been able to add “refusal” categories by ~following the paper, with pretty good results.
If nothing else I think this will spark more targeted research
Recent and related: Astra can read and write weights to improve Qwen - https://news.ycombinator.com/item?id=50341174 (870 comments)
LLMs are not superhuman at reading and writing neural network weights - https://news.ycombinator.com/item?id=50344861 (10 comments)
I pretty much got that idea when reading the post, on how it didnt have improvements on a Gemma 270M model which didn't have any looping issue. However, it still made me look at some bottlenecks in models I had trained (like MLP network for stylus input automation and a 17M reranker with NLI stance and noul heads) and allowed me to finetune without training, but just by adjusting some weights. But what's pretty cool is the zero shot sandbox development method they showed for a Go model. I tried to do that for a 17M re-ranker from zero weights and Astra built it using its reasoning without me having to use a GPU and find training datasets that are not CC by NC, and add some hours of tweaking and checkpoints. I don't know if its the best way to build something, but we might be building small models (maybe not LLMs) like with just programming the weights instead of finetuning.
In the end I got to replace a 68M model with a 17M model, and improve stlyus tool recognition accuracy from 88 to 96%. Was a pretty fun experiment too.
Recently, as has probably been the case in other fields as well, many people feed the public cherry-picked results to hype up what AI can do or to stir up FOMO.
I used to think that in the dev community this sort of thing was less common than elsewhere... because people want to share their experience, what they found, etc.
Now that AI is here, people's experience and knowledge have basically been reset. So it's no different here now, which is a shame (I'm not saying it was better before, though)
>The 'Astra can read and write weights to improve Qwen' post is misleading
Does the author mean the "LLMs can natively read & write neural network weights" post? Did the researchers change the name or something? Because when I read it, it made it very clear what they were doing and none of it was misleading. It all aligns with what this author is saying.
Why this post is here in the front page? The Qwen vector steering performed was a very selective operation that interested only specific layers and weights set, it is non trivial to identify and fix. This is remarkable. Also Experiment 3: no access to forward pass, so why here this post says the contrary of what the paper shows and everybody upvoted it? I don't understand.
Experiment 3 doesn't involve reading the weights in as part of the prompt. It's a coding exercise. Here are the turns: https://github.com/RikiyaT/lm_understand-logs/tree/main/exp3...
At each point the model has to write code to read the weights in with Python and some other Python code to perform analysis, like this: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp3...
My complaint is that the post presents a confusing mix of experiments that imply capabilities that are more than the sum of their parts. This experiment 3 has nothing to do with experiment 1 for instance. In experiment 1 the model does read in the raw weights like it's reading an image.
If you don't like these experiments, then I can only assume you would complain about CRISPR as well, considering it's a general framework with many disparate parts that work together synergistically, similar to the work Takehi et al. describe. CRISPR was also famously rejected multiple times by reviewers.
Note that being able to both inspect raw weights and run Python code to perform additional analysis means the model has multiple possible techniques at its disposal (generally considered a positive, rather than a "confusing mix").
I don't understand CRISPR well enough to comment on the analogy.
The blog post claims to show that frontier LLMs can understand and craft neural network weights natively. The experiments where the model completes coding tasks are not relevant to this conclusion. Those experiments require the model to demonstrate an analytic understanding of how a network might behave under various conditions, along with reasoning skills, coding skills etc. As a coding task this is a significant challenge. But it's entirely different from "natively reading and writing neural network weights".
Would you also consider it a benefit if there were experiments showing that a frontier model can download and run PyTorch, or write its own network and optimiser? That's another technique it has at its disposal to improve an ML model. It's just not one that's relevant to the paper's core claims.
> Why this post is here in the front page?
You know exactly why... because anti-AI hysteria is quite popular on HN.
Any "critique" or "appeal to resist AI" rooted in emotion and inability to admit that intelligence is not a feat unique to humans anymore will make the rounds here.
But why? Shouldn't people who like CS be more open to the idea? I mean this is why computers were created, to think instead of people.
Are you kidding? If anything pro-AI hysteria is way stronger here, just look at how many upvotes the original got
Even any small model update a company puts out is guaranteed top of HN
If people have programmed programs to interpret neural networks, what is so special about the AI doing so? It's a forgone conclusion.
You can let the AI generate training code for the next architecture too.
It's not like the AI is randomly losing its ability to write software just because it involves AI.
I am growing increasingly fond of the term "anti-AI psychosis". It's getting ridiculous sometimes.
Thank you. For some reason the voices of sanity got drowned in the original discussion. LLMs do not have an uncanny ability to speak weights. Yet.
I was blown away even by #1, and I don't feel after reading this that I was misled.
Agree, it makes sense, a lot of people are catching on to LLMs for reverse engineering finally, but its been possible. LLMs can and will eventually flawlessly dissect models. I experiment with Claude, I ask it tasks I know are possible, but have no idea the solution, it can reverse engineer obfuscated projects faster than most talented people I know in that field (talking about games now not AI). I heard a developer say their latest release would take at least a month to figure out the obfuscation and encryption, Opus 5 did it in like... 15 minutes of talking to me (I did throw it the right questions / suggestions). I do that all in the name of seeing just exactly what is possible.
Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.
I don't get it. Because python was used, it's supposed to not be cool? IMHO, working with raw weights with Python is still working with raw weights.
If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
> Because this is very much a part of what ML engineers do.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!
I think you misunderstand. Yet the model is doing something impressive still by writing code to fix model weights on problems. But that's a whole different ballgame than just reading weights and knowing what to change. That would mean that they have an internal representation of how the model weights map to information or a specific task, which if true, throws all the interpretability research out of the window
To me, the original article is more about what LLMs can *do* in weight space, I don't really see the part where it would tie to an internal representation.
It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.
That "Likewise" is quite load bearing in the original article :)
So in effect, all what we're quarreling about here is whether it is _misleading_ to use "Likewise" to transition between two paragraphs without implying that the details of the first apply to the second?
Just like Graduate Students are good at Gradient Descent, they can see patterns and fine tune parameters 100000 times until the result looks good. So too can LLMs with their ability to examine patterns and brute force things.
Except LLMs already exhibit superhuman intelligence.
They exhibit superhuman toil.
That's a good way to look at it
There is a difference?
I've done my part. Do what you will from there.
Grit for the win, yet again.
Misleading how exactly when the original post states _exactly_ the same things as outlined in this objection?
I completely agree with you Matthew, but I see the thing go in right direction in research as well, I am working in the same field where I am building an agent where it will improve the base model too !
Either way, a lot of devs will be brute forcing explorations in this direction today.
Agreed. Not sure it leads to anything truly groundbreaking, but more than likely will lead to very fascinating implementations that lead to even more interesting results 6-8 months from now. Think openclaw -> how people utilize now versus when it was first announced
That’s what I’m excited for: the raft of posts over the next few days as the inspired set to work
https://grok.com/share/c2hhcmQtMi1jb3B5_e9515277-145d-4447-9...
It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
How do you prune out the 'fails to prune out behaviors' behavior? If my model is failing to prune it out, can another model prune out that behavior?
With normal training you're simply training for higher loss for things you don't want while keeping the loss low for things you do. Same would apply here except you would target low eval for behavior you do not want the model to have rather than trying to increase eval.
> how you could prune harmful capabilities from models and build a more efficient training system
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
What it shows are two unrelated results mixed together for no apparent reason besides marketing. The raw-weight results are impressive, but haven't been demonstrated as ready to scale up to a real language model. The Qwen result is a mildly interesting application of functionality that's been on the market for years.
If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.
I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
My main takeaway is that HN had no idea the field of interpretability existed until now
I don't know why because Anthropic has had a lot of blog posts about internal work on it
Just one word - MARKETING
We lack a build tool to handle an edge case yet unseen, no means exist to input the new weights and get a better model
Big deal; have some bpf probe handle kick off of the rebuild when the when new weights are tested by some verification process
This is just CI/CD work now
“Mech interp” has to be the most pretentious term I’ve heard in a while. Any normal group of people would call it MI.
oh man someone should make this into a loop where the model can alter it weights after an inference. some kind of descent of the local gradient.
Isn’t that what the Google dreaming thing is?
Wow that was fast
something i found 15 minutes ago is that the car network test is "very very easy", as in Opus 5.5 in Claude Code did not even need to see the weights of the network to solve it correctly. i can give the transcript of the session. i'm pretty sure Opus didn't cheat
According to this person, some problems are only "solved" by modern models: https://nitter.netbub.com/AlexKrentsel/status/21173766837246...
(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)
Why doesn't Astra improve it's own weights
That’s a distinction with only some difference.
“You can’t have a machine for testing” is a weird restriction on AI, especially if you entertain even the remotest notion of possible AGI.
This goes to show that AI psychosis on HN is real, especially when many AI researchers have repeatedly dismissed this as highly misleading.
I for one think it's pretty cool how far too many people bend the truth or outright lie in tech these days.
There is going to be a reckoning for tech in the near future, because people will remember the lies and it's gonna be hard to beat the history.
Welcome to publications involving ML in 2026... Glad to see someone else write this though because it's important to start having these discussions. Too much overselling has been happening for too long. There is so much great about ML. IMO it hurts the field long-term to continue overselling claims like this.
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
edit: lo and behold! https://x.com/SchmidhuberAI/status/2117446822855888956
I think the reviews are flawed for most of journals. But I can't really blame them, there is huge influx of new papers comming everyday. Unfortunately reviewers are more biased towards groundbreaking stuff so it seems they can't make connections with existing and prior work.
They are also biased to see from what institution you come from..
Waiting for
The "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading.
But can we use the idea inside the training loop, as an optimizer instead of (or next to) e.g. Adam?
Who watches the pruners?
A better idea is to have Astra modify it's own weights and let it try if it can RSI, but you gonna need to closely monitor for actual leap in capabilities so you can keep a check on alignment.
Someone has to already be doing this right?
Given the sizes of the models and the cost of the supercomputer necessary to run such an experiment, I wouldn't bet on it.
This isn't a good look. The original article clearly states what the model was doing, and this new hit piece is supposed to land because they are straw manning the comments on the original post? This isn't reddit dude.
This could have been a comment on the original article.
> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.
Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.
A tangent, but I’m now realizing that the “recursive self improvement” thing that these LLM companies are talking about is (probably?) not about unsupervised Claude Code instances improving the training process, but rather LLMs modifying their own weights through tooling directly. It would certainly open up a whole new process for model optimization.
I was just watching a video about this. They had this survey question for AI researchers asking when they think AI would take their job. The reason they asked this was that would be the point where AI starts recursive self improvement. If it’s able to replace the job of an AI researcher, then it could keep improving itself.
Instead of just asking the researchers when RSI would be achieved? Sneeky!
This is a tautology.
I don't know how much we learn by surveying working "AI researchers". By their own admission most of them have a religious belief that they will bring about a singularity in the next few years. It's like asking a billionaire if he earned his wealth; regardless of what you think the answer is, you pretty much know how they will answer ahead of time.
Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).
It's obvious that recursive self-improvement has been happening and accelerating quickly for some time.
AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.
Yes but my point is that I never realized that the AI would be writing tools to update the weights directly, rather than improve the existing training process.
That's "just" improvement, not recursive self improvement.
The blog post exhibits the same flawed logic that many academic reviews do — looking for reasons to reject a work rather than accept it. It comes off cynical and contrarian. It's easy to find issues with an academic paper, but ACL has consistently pushed for a higher standard [1].
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
[1]: https://aclrollingreview.org/reviewerguidelines#-task-3-writ...
You've expressed your criticism plainly without addressing anything the post might've done well. I think that's absolutely fine, but why would your criticism apply to the post but not your own comment? Personally I think it's because the standards of peer review don't apply to HN comments, but I don't think they apply to blog posts either.
You do realize how academia works, right? Granted, it is often dysfunctional, but this ain't that.
Trying to find every single angle and option to break something is how we verify the truth of something.
If something is objectively true, it will survive the scrutiny. If it does not survive, it wasn't objectively true.
This is how science works. That ain't no flaw. It is a flaw to be offended by science doing science things.
Considering I have a PhD in CS focused in NLP and have been a Senior Area Chair for ACL, yes I do know how academia works. And, no this isn't science working; it's the same plague infecting most of academic blogging and social media. People are vying for attention in whatever way possible, often at the expense of others.
And the examples in the OP's blog post just go to show that reading comprehension is at an all time low:
> Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
The table just below shows this statement to be *entirely* factual.
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
Any further implicit assumptions are on the reader, not the writer, as they never stated "that the Qwen experiment worked ‘just by looking at the raw weights’". You can certainly point out issues in the experimental design, but just stating you dislike the set of experiments and is yet another poor take that directly contradicts ACL's guidance for reviewing, exactly because it does not advance science.
All I can say is I very much disagree, and if I had authored the post and it had received the reaction from people that it has I would be anxious to issue corrections. I haven't reviewed for conferences for several years now, but I wouldn't have thought this should pass review.
I believe the post definitely plays off the implication that the Qwen experiments are connected to the experiments that read the raw weights and have no Python harness, when this isn't the case. There are experiments where Astra reads weights directly for very simple models, and there are experiments where Astra performs ML research tasks by analysing the problem and writing Python code. The authors imply that the same technique underlies these but that isn't the case.
I mean separating two paragraphs by "Likewise" is pretty much how an author communicates to a reader to follow the barely-hidden line of reasoning.
Link?
You have to be more specific. What exactly are you looking for a link to?