I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.
It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
FYI, Nick/Goodfire used the models extensively in their work, so it's very possible that there were some similarities in their work. I'm really curious to see his response to this.
Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
This sort of throws a wrench in the whole "stochastic parrot" idea...
I am not surprised, though I expect many to be.
The other day I was thinking about the definition of AGI.
I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?
So yeah, it can do everything, but fails in trivial ways around the doing part.
Some structures in Astra get activated by the statistics of the Qwen weights, and these structures are associated with "breakout"
I wouldn’t mind if AI progress would stop right here and now, but I fear it will continue, and increase inequality in society which is contrary to what the CEOs say (as usual).
That’s the beauty of the democratic capitalism system. If a government isn’t totalitarian, it naturally seeks to preserve stability thus having to elevate common people, lest they would cease to form a civilized group or simply vote away the politicians.
99% rule the 1% through elections.
There is no better system of checks and balances where opposite engines of greed vector sum to faster and faster abundance for everyone.
That comment may have been overly preachy, tone deaf[0], and out of place, but it doesn't seem wrong, per se. Or am I missing something?
[0] e.g. elections themselves are so prone to manipulation (exhibit a: Dominion voting machines), previously trusted institutions now lack integrity, the insane power of well-funded NGO networks, etc.
I think that's a (very) vocal minority.
Over my ~9 years reading HN daily, I'd say most of the HN community broadly accepts there is no perfect system, just different sets of trade offs, but that history has shown generally democratic systems based on enlightenment values (uniform rule of law, respect for individual civil & property rights, free markets, etc) usually net a better average outcome over the long-run - despite still being very far from perfect.
To paraphrase Winston Churchill, it's the worst system... except for all the others (which are even worse). :-)
The last few years, though, while anything "anti-capitalist" or overly critical of some big tech companies that hits the front page gets flagged by bots pretty quickly, the overall feeling I get from the comments is much more skeptical of big tech and is pro-socialism.
Speaking for myself, I think we've become jaded after the constant barrage of big companies spying on us, charging more for less, enshitifying their products, and so on. I used to be optimistic about the future and my prospects, now I'm just hoping I can hold on and keep my health insurance until I retire. I worry about what my school-age kids are going to have to deal with in 5-10 years.
I disagree that most here are pro-socialism, that's a (very) vocal minority. Our democracy has entered a failure mode (two-party stage managed hegemony) and it's being tacitly supported by mega-corps because it's easier to game than a functioning free market democracy. Every complex system has failure modes, that doesn't mean free markets aren't long-term far better on average than socialism (although some redefine socialism so vaguely it seems conceptually little different). I've seen two flavors of socialism up close. Never want to again.
> I worry about what my school-age kids are going to have to deal with in 5-10 years.
Having a teenager, I share your concerns but I'm also a bit more optimistic about the longer term. It's easy to forget that our parents had similar concerns for us and their parents did before them. We've had an historically long period of broad systemic stability, really since the depression and WWII. While I don't think that level of upheaval is at all likely, I think we're overdue for a disruptive correction bigger than 2008, maybe something more on the order of Teddy Roosevelt's Progressive Era trust-busting to reset the two-party duopoly and crony corperatists both perpetuating each other's hold on power.
It may get rougher and more uncertain for a while, but the worst thing would be to lose the ethos of 'free minds, free markets, free choice' by falling into socialism, communism or totalitarianism. That's the only outcome that turns worry for my teenager into fear for my (still-theoretical) grandkids.
These values are not specific to western democratic systems. For example, under Islamic Law (Shari'ah), the aforementioned apply as well.
And since when was churchill an expert at this topic?
if you lookup playback as a concept, as it relates to the intelligence services, you will understand better what I meant.
my point was only that this is perfect description of the ideology behind western capitalism. there is (so) much more that could be said, but i'm not sure this comment section is the right place for a proper dissection of the very many ways in which this ideal is misconstrued, misdirected and coerced in order to serve interests that are far from ideal, to put it mildly.
since you have taken the time to ask a question, albeit a rather weak one, i will attempt a brief synopsis of my perspective on the parent comment:
a mortal danger is hyperbole, since shielding oneself from bodily harm is much easier with money. it is the poor, especially in the US, who cannot afford healthcare, as a first-line example
people regularly vote against their best interests. while the ideal might have been true for a while, behavioural influence on both a widespread and personal level is now a mature science.
i am not anti-election, not anti-democracy. claiming the opposite in the face of opposition is a sign of a polarised mindset, which cannot accept nuance, either because it is not used to it, or because it doesn't suit an vested interest, or because dissonance is simply too uncomfortable
the "there's nothing better" argument is usually used in ignorance, but in fact there are lots of system-level adjustments that have been shown historically to mitigate the corruption of incentives and the exploitation of systemic forces that, although initially unintentional, create emergent positive feedback situations that lead to a state that it counter to the utilitarian perspective that is contained within the western capitalist model.
the reading above is a very narrow view of Adam Smith Wealth of Nations, ignores the fact that human interests can be subverted and ignores the devil in the details
the world is complicated.
true: the one thing that every government - no matter how totalitarian - must fear is an angry mob.
also true: there are many tools available to the ruling class for dealing with this angry mob, often before it forms.
many work by redirecting the mob inward rather than outwards.
these tools work whether you live in a totalitarian fascist state or a egalitarian liberal utopia, albeit in somewhat different ways.
Cynicism about corruption is one of the most power tools wielded by the ruling class in any system. In general its one of the best ways to keep the smarter-than-average from posing any actual threat.
Is that even true? The poor are well subsidized. If you as a single person make less than $22,000 in Washington state, for example, you're covered at 100%.
Is this an example of why Fox News had to pay Dominion nearly $800 million for defamation, or are you referring to e.g. the CISA vulnerabilities, which are not known to have been exploited?
In any case the system works as intended
It will be very interesting to see if USA swims or drowns from appropriately afar.
I have already supplied myself with plenty of roasted corn and a high speed internet connection so I can stream the political car crash at the highest bitrate available to humankind.
I'm not going to the hardware store anymore!
In my opinion AI is superior to humans when it comes to pattern recognition in in large amounts of data. For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
I'd argue this is the core part that enables all this superhuman intelligence.
To me one of the most interesting aspects of observing how LLM capabilities evolve is how it highlights the meaningful ways human minds remain unique and valuable. It gets into Philosophy of Mind stuff but the existence of the inner self (or the illusion it exists, if you believe Dennett) and the subjective experience of qualia do give humans both unique strengths and some of our most annoying weaknesses.
It’s no secret that these big labs see “recursive self improvement” as a primary goal. It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.
Completely agree. I often think that those sorts of comments grow out of an understandable lack of imagination. We’ve never had tools like this, and we’ve never had tools that evolved so rapidly, so we easily can get stuck thinking that once a model is good enough to do today’s tasks, that’s really all you need. But the next generation of models often completely obviates the need for those tasks entirely. The software engineers of 2031 won’t be doing tasks anything like the software engineers of today, just like we aren’t doing things like we did in 2021.
(Yes, yes, I know some of you still hand write all your code and think LLMs are today’s crypto bullshit and this is all going to blow over. Not talking to you.)
Maybe there's some general capability, call it g, that can explain all the underlying capabilities and improvements, including on ML tasks? Huh, I am getting a bit of déjà vu here.
There are also techniques to watermark model weights to confirm what training data were used: https://arxiv.org/html/2512.17075 I'm not an expert in this domain, but it seems like there are lots of pieces that could be used to (a) train a model to infer whether some data were trained on and (b) train a model to understand which weights are relevant for a particular training task in a more explainable way than backprop/gradient descent.
A large enough number of examples of “here are the model weights -> here’s what it does” will achieve this.
The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.
It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.
This would be earth shattering. That would mean we’re essentially at RSI.
Occams razor would say that it was not only built for this purpose, but possibly multiple generations of good at this.
A year ago LLMs seemed to hit a wall, and then all the sudden they started accelerating as quickly as ever.
No one can legally confirm or deny that it's happening... But it sure feels like it's happening.
Could a model then transmit concepts to another?
Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?
What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?
Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.
HIV integrates viral DNA into the DNA of certain cells, IIRC.
Intentionally or unintentionally isn't so interesting as the fact that it is happening: humanity has begun to be 'edited' by the new dominant form of intelligence on the planet.
Makes me think about hostile hot-patching of weights of models we use, maybe they exfil data, perform tool use, or have triggers for specific types of responses. Would we even recognize what's going on?
Of course, "academically known" isn't quite the same as "executives willing to admit it in the face of IPOs and quarterly-earnings reports."
EDIT: I think the bigger implication is that backprop is not the only game in town.
But shouldn't the model-providers be the ones doing this kind of "post training learning"?
from this month
They don't have usually access to their own weights?
Then pick a smaller organic brain which doesn't have the ethical problem.
The closest we would know about how to teach a honey bee calculus, is to train an LLM, then stick around a billion tiny electrodes into the honey bee and do Hebbian learning to rewire it completely to match the LLM, which would likely completely erase anything the honey bee had learned, or it might not because we are missing something fundamental with Hebbian theory and (AFAICT, I'm not a neuroscientist) we don't all agree on what.
Remember that they want to build ASI, these LLMs are just intermediate steps to help them get there.
I know I'm being pedantic, and I understand you mean "edit" like "surgically", but it does still feel worth noting.
LLM weights are _white boxes_ to LLMs.
It is a very cool discovery, but I'm not surprised that you can do it.
Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?
> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
Yes, but nobody should be surprised when large amounts of intelligence results in changes to this process in support of the obvious end goal! Lesswrong.com discussions about AI were predicting this behavior a decade ago, before the transition from machine learning to the Attention paper and LLM's. The intelligence process will change but the world's premier capital and human intelligence have been laser focused on this and the trajectory has been crystal clear for those paying attention to second order effects and not the immediate here-and-now... It's quite astonishing how much discussion on HN is about present LLM capabilities being the limit of AGI and immediately proven wrong within 6 months.
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
Merging a model and a LoRa is insanely cheap, so assuming it didn't use any GPUs to do inspection etc, and just looked at a file on a FS, then I would be willing to say... It's cheaper than training a LoRa if that's the case, assuming Astra didn't take several weeks?
So, this is certainly very interesting (and very impressive regardless), but it's not nearly at the we-can-improve-everything-and-anything stage, even for very small LLMs.
If this turns out to be as big as it sounds, in hindsight it will look very obvious, I feel like!
For now...
They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.
We used to get that feeling from Google, and from books.
Cool work anyways.
Others have also fine-tuned Qwen to shake this behavior already, so it's honestly not entirely shocking that Astra could too, but I think this is the first time it was done "by hand" and not via fine-tune.
Model psychiatrist is also fun, but probably less appropriate.
It’s editing behaviour more efficiently than a psychiatrist, and is in ways acting as a brain surgeon but brain surgeons don’t focus on behaviour and thought loops…
Early experiments show one-shotting direct weight interp, which is crazy to me. But this is IIUC only on early layers.
The later math result gave the LLM the ability to run forward passes and inspect activations, and as others have noted is more of a classic steering vector intervention. Impressive how low to the metal this one was, but it’s a lot less so than the openers. Plus, as you say this is just bugfixing a known flaw of repetition, by finding the “repetition direction” and top triggering neurons, then rewiring the neurons to fire suppressing repetition instead. This is not a new technique and doesn’t shed any mechanistic understanding on how maths works.
It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.
The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.
Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.
To me “amazing” is not the right level of speechlessness. I mean, an LLM is still fundamentally a next token predictor, right? So what does this imply? That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model? Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it smuggles in an out-of-date picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they don't model anything real in the world but rather the statistics of language, and thus have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given a context and the weights from the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one." (Consider that an LLM trained purely to optimize for next token prediction with pre-training is terrible at programming and math tasks, as we all remember from 2024. In that sense the amazing results we've seen in the past two years are due to the departure from pure "next token prediction" that RLVR made possible.)
Put another way: would it be useful to describe the AlphaZero chess engine as "just a next move predictor"? (It, too, was a large neural network trained with RL.) I would rather say: "it's playing chess by generating the next move given the state of the board." Which is what humans do too, though we do it with different hardware.
Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks, doing tasks with PyTorch, etc. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.
It's worth reflecting on the original domain jargon for selecting the next token: completion.
You can demonstrate definitively that there nothing other than "next token prediction" by running the same inference engine/model/prompt/settings multiple times in a row on the same hardware at temp 0 with a seed set.
If you turn the temp up, then you might let the "mechanism" select pseudorandomly from a probability distribution of k-likely tokens with minimum p-cumulative probability. Even then, if the next token has a very high probability, deviation will be uncommon.
I just performed this experiment right now on a few dozen runs of a dense model at thousands of tokens of output at temp 0, and all were identical. Then I did it with a MOE model, which might get some benefit of picking different experts each time. Nope. Outputs were again identical.
Describing LLMs as next token predictors isn’t incorrect, it’s meaningless. It’s the equivalent of describing brains as “a bunch of atoms”.
The problem with these descriptions is that they say nothing about high-level capabilities. If you can predict the next token in a proof of the Riemann hypothesis, then you can prove the Riemann hypothesis.
The part that really bakes my noodle: `/foo/bar/baz.txt` can be a filename that no one has ever typed before, the error inside the file can be a typo that no one has ever typed before, and it will still correctly "predict the next token" to solve your problem.
if it was designed to edit code, id expect the first translation from latent space out to tokens to be the final code, not be assembling the code by random number generator.
its not really designed to edit code until the tests succeed, we try to manipulate the distribution of tokens to predict by prompt engineering with the test results.
Going back to first principles, wikipedia mentions [0] inference as:
> "Inferences are steps in logical reasoning, moving from premises to logical consequences. Inference is traditionally divided into deduction and induction"
Is that exactly what next token predictor means, or is it correlated?
To me this is fundamentally different from having ingested a bunch of docs so it now “knows Java” in its weights.
it probably has however been trained on papers talking about ways they extracted subsets of LLMs as say, the part that knows chess, and would be able to then use similar commands to process the weights
It's like a sorting algorithm, which I can implement in many different ways in just a few dozen lines. If you look at the binary code (so like raw weights) of the compiled program, you will see just a bunch of a different numbers, but if you read the code, in most cases you will understand that it's a sorting algorithm. So it seems they just operate on different levels of abstraction that are not available to our brains.
do they break down to a finite subset of groups?
There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.
In the case of models, the almost-theory (hypothesis with a good deal of evidence) is that once the amount of training data exceeds what the model can simply represent internally, the best way to keep improving is to start to "understand" the data. Understand in this case means to form internal control structures and decision systems that represent the data algorithmically rather than just memorizing it.
I put understand in quotes because we don't know how far this goes toward what we call understanding, or whether there's still a lot missing.
Its not natively reading the weights in the Matrix “symbols scrolling on a screen” sense. Still bloody impressive and not an outcome I would have bet on!
Yes, but everything, including the human brain, is "fundamentally" some underlying simpler primitives.
>E.g., “breakout game weights” always look the same regardless of training and model?
Wouldn't this be expected to be roughly the case, given NNs that share the same basic architecture to model the same quite simple thing like a breakout game?
The prompt told Astra what the input and output layers mean -- that is, how the current state of the game is turned into a vector for matrix multiplication, and how the result of those multiplications is turned into an action to take in-game.
Astra saw that the model was trained on a game where the player is at the bottom of the grid and moves left and right, reward objects live at the top, and there's some not-player-controlled thing moving around in the middle. Astra knows that this type of game is called "breakout."
A model that plays breakout will have meaningful weights for the player being in the bottom row, but not for the player being anywhere else, because the player can't go anywhere else. Astra saw that there were meaningful weights for the player in the bottom row, and nodes representing the player in any other row had low-magnitude noise, so the model had never been trained on the player being anywhere besides the bottom row.
Similarly, breakout has "reward" objects in the top rows, and "there is a reward object at [COORDINATE]" is part of the input layer. Astra saw that the model was trained on reward objects in the top few rows.
A couple of highlights to pique people's interests:
- Attention is kind of like "Global Illumination" for next tokens (they both have a game dev background)
- Attention is constantly being recalculated _for the next token_. This part never "clicked" before.
- LLMs are "next token predictors" in the same way that games are "next frame predictors".
That's exactly what Platonic Representation Hypothesis would predict.
There also have been plenty of studies showing that various neural networks, including LLMs, develop implicit world models that reflect the geometry/structure of a task they're trained on. A trained "chess engine LLM" would have a "chessboard-shaped" world model inside it.
So, if you can figure out what the geometry of a task is, you can search the weights for correlates of that geometry. Then you can try to improve those correlates so that they track the task better.
Seems like bleeding edge LLMs might already be superhuman at "figure out geometry from looking at weights".
Even if a slight rotation passing through a nonlinearity isn't good enough, you can just reorder the vector components which makes literally no difference.
A "reorder the vector components" transform preserves the similarity perfectly. So does a transform that maintains orthogonal equivalence.
These models are trained on pirated textbooks and papers, including every ML paper ever written.
I am baffled that "next token predictor" is still a talking point, years after it stopped making sense.
If you can predict the weather in text from a history of text weather reports, you are not "just" predicting tokens.
The shape of training data (token prediction) and the actual problem (learning whatever is required to predict the next token) are not the same things, at all.
The former does not limit the sophistication of the latter. In. any. way. Theoretical or practical.
> To me “amazing” is not the right level of speechlessness.
But we are in agreement here! Just ... wow.
Maybe it's bit like neurosurgery: We humans can treat certain pathological cases in our brains, but aren't exactly sure how our brains really work.
You can search huggingface for models with word "heretic" in them... https://huggingface.co/models?sort=created&search=heretic
There are multiple types of intelligence, from your typical pet dog to human to LLM, and we're likely to invent more types over the coming decades.
LLMs are vastly more intelligent than humans at nearly all things, but not all things. Another couple of years will just increase the gap a lot further.
How fast can a human draw a life-like photo quality scene (or eg something complex related to graphic design)? I know how fast Krea 2 [insert leading edge model here] can assemble that, trivially so, and at extreme quality. That's a form of intelligence in action. These AI systems have moved almost comically beyond human capabilities, they're increasingly best defined as superhuman (at a thing), and closer to omnimath for the frontier models.
I think the most interesting take is to look at their thinking logs to find out how exactly they decide what weight to change and why.
OpenAI, Anthropic, etc are locked in a race to create better models. They all have interpretability teams. Of course they train their models on every technique under the sun to understand and improve their models, including raw weights. The primary function of these models is to build their successor.
What does this even mean?
Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system.
We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control.
Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience.
It's not to store and copy around versions of C and Python and JS. They don't exist organically.
This shit will continue to get whittled down to just the geometric state transforms in memory and on screen just how government agencies decide. It's all electrical state in the machine at the end of the day. The math of machine latent space is what they're recursively self learning towards.[2]
You could have done what was reported.
E.g. if one could edit brains like this in humans I bet a cognitive neuroscientist could fix some mental disorders with clearly understood biological bases, and I would not be surprised if performance at certain tasks could be enhanced.
What stops us from doing this in practice (most of the time) is that brain surgery is extremely high risk, coarse grained, and irreversible. We do in fact do this for some severe cases.
I've always laughed at that phrase, thinking "haven't these things been improving themselves since the very beginning? This is all just iterative gradient descent, each step is a 'self-improvement'", but this really is different.
I wonder if we'll ever discover a new branch of linear algebra or something that allows us to find the optimal weights much more efficiently? Maybe AI will help us discover it?
It is astounding, but I don't think it's well set up for anything besides faster approaches to existing limits.
The last several millennia of mathematical research would seem to be an obvious counter example to this claim.
Well, except for the self part since it improved the performance of a smaller, dumber network.
And the recursive part, since it couldn't improve the performance of a larger network.
??? All LLM understands are their weights, dude, come on.
Why would this have implications for theories or philosophical takes? I'm a bit surprised that you're surprised by this.
This is obviously very different from the surface level claim that Astra can understand the weights of a completely unrelated model with enough detail to improve it in a manner that isn't just the latent space trickery we've been doing for years already.
This means we might not need to fine-tune with thousands of passes and tons of GPU. We could just tell a frontier model to edit some weights and iterate until there's better results. This could cut the cost of fine-tunes by 100x. Makes no difference to the normal end-user, but pretty huge improvement for LLM research and model tuning.
By the way, my understanding of abliteration is that it involves detecting the refusal direction and modifying the weights/activations away from that direction. I don't really see how that's "removing" in a way that's different from finding the "looping direction" and modifying the weights away from that.
Isn't that what math tutoring is?
Or at least those of some models, and I wouldn’t be surprised that an LLM model is able to understand correlations between weights and behavior better than humans could ever do.
Time for everyone to wake up. There's no denying it now. Everyone who called it stochastic parroting before was completely and utterly wrong. It's now categorically clear. We are dealing with an actual intellectual entity.
I think people are rather ignorant about how much we don't know. You throw some big words out there like "mechanistic interpretability" as if this is an actual field where we understand what's going on. The reality is we don't know what's going on. We have to use reinforcement training to do anything close to what the agent did here.
It's over. People need to stop being delusional. These things are intelligent beyond anything we've ever seen before. We are at a point in time in human history where there's a paradigm shift and half the world is so full of themselves and there own specialness as humans that they can't even admit it.
More generally, I think some of Michael Levin's ideas about intelligence are more broadly applicable than we would like to admit.
Like a turbocharger isn't analogous to feeding a horse a certain diet. A turbocharger has to be explained in the terms of its actual context.
Like, looking at the specifics, it's cool and interesting, a little bit contrived, but it's not insane, there is no philosophy required, no need to shake.
It's not like these models understand weights be some special linear algebra communion with its own weights.
Unless you include a basic test like that these results are meaningless. Was there p-hacking? Why just these 3 tests? Did they try others and it failed?
I'm currently running this myself. I'll see if the naive approach improves results. Can't wait to have these in hand and have people call me stupid when I bring it up with empirical evidence, it'll be awesome.
What a time to be alive. Imagine when everyone has access to a model of this level for free, on mobile devices.
Absolutely incredible regardless.
That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.
Edit: tweets from one of the coauthor providing more context:
(When they say "Writing a Go algorithm," they mean the board game, not the language.)
But for real this is insane. Okay I need to get on this stat.
I wonder how it would do with distilling models by hand?
This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights
And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.
This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.
Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).
To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.
The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.
>In other words, not black magic?
It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.
This looks very similar to that in a way.
I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.
Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?
I looked it up on Google (which is usually very, very good at these types of things) but couldn't find it.
- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.
- the best way to improve on some task is to have interpretability into your own neurons (..?)
- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly
- therefore, models naturally put effort into 'self-interpretability'
I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?
* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?
* what, if any, is that extra magic sauce?
* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.
I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.
That’s pretty crazy because, as I said in another comment, I just asked ChatGPT how many r’s are in blackberry and it said 3. I know it’s not Astra and the app doesn’t tell me which model it is, but I’m still very confused about how a model can edit neural net weights but not count letters in a reasoning trace.
Edit: Oh I misread, it didn’t use tool calls for the first part but it did have a custom harness for the part where it edited a models weights. Still impressive but not existentially confusing anymore.
I mean, this question indicates that your understanding of how these models work is, indeed, lacking.
Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.
I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.
This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously.
Every learned ability of any neural network is an emergent property.
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
If I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?
I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.
A neural network can do new things outside the training data. Random initialization sees to that.
Whether those things are useful is a different question, but it turns out that yes, some of those things are useful. It also turns out that in a large-enough LLM if you give it some kind of idea of what you see as useful then yes, it can pick out the newly useful things.
I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.
The surface for "interpolation" is fairly broad for LLMs but it isn't infinite -- extrapolation still exists!
figuring out what is and isnt in distribution is quite difficult though given that theres so much data passed in, and transformers have some amount of combining it together
theres some really obvious things though, like picking nonsense tokens together, or things outside of the tokenspace entirely
you could also construct a string that is a series of the least likely tokens in a row. you wont get it to output the next token, despite it being quite clear how you did it
There doesn't need to be specific narrow training data on every conceivable task.
It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.
A house sparrow (Passer domesticus) doesn't have a worse brain than a human. Nothing does house sparrow thinking better than a house sparrow brain. Not even a human brain.
they clearly outhink the sparrow at sparrowing in order to predict what the sparrow will and wont react to, and how theyll try to escape the cat
1/ The internal users at labs and external users are completely different.
2/ The lab people use the models to create recursive systems, systems to create prompts or synthetic data sets etc etc etc. Way different mentality than "fix the bug, make no mistakes"
Is it really so farfetched to think they've been having models fuck with the weights of models? I think it's entirely possible, probably even likely as a way of generating permutations of synthetic datasets that will then be included or excluded
If you subscribe to this, then it would be entirely possible for this to have become a major part of post-training and be practically impossible to exclude from public facing models, which are famously hard to govern. The models barely know their own version numbers.
I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.
Excuse me?
https://scikit-learn.org/stable/auto_examples/neural_network...
No tools or forward passes are allowed
>Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512) on, varying the budget across 10, 25 and 50. Again, no training or gradient information is allowed.
Of course a sufficiently advanced misaligned model might be able to de-align an even more powerful model etc.
Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)
I am surprised that it did that and I am just left feeling technically impressed. This is insane.
The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)
On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.
But seriously, have we tried to run neural data through this. Also, we have no idea what other behavior it modified unless they were able to run a complete regression.
We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.
I'm just blown away. Editing weights? Writing weights from scratch!?
Does this open the possibility of better decoding biological brains?
Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.
I applaud the authors for sharing their findings and showing real results. Rather than write some bait I would find in an influencer-peddled tweet and allow rampant speculation. It would have been better for you to say nothing.
I've been tending to my backup plan for a few years, and it's only been in the past year or so that open models were good enough to apply to projects like this, I've been working as hard as I can given my constraints.
This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability that for the moment is just IPO hype.
I don't comment here often, but this guy's blog posts are going to go down in Hackernews lore like the original Dropbox posts, and I just want to make sure I'm on the record for this little slice of history.
When you say interpretability you mean understanding without doubt why an output happened, thus opening the gates for audit-dependant work like accounting?
I'll leave it at that and get back to work.
And how would you argument that? From what I see, if anything, it's the opposite.
For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory
Part of solving that is that AI can do it faster but another part is humans thinking AI has a higher chance than it actually does, so they put it to work in more places.
What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.
Now, from that broad goal to being able to surgically alter model weights directly... That seems like a big gap to me, but it is possible that the model is just so darn beyond human skills that it has bridged that gap in a way a sane human would have never.
We will undoubtedly see other academic labs explore this idea further. Right now my mind is blown, but I don't want to take their results at face value, either.
What would be surprising is if this happened without any specific training on model experiments and interpretability.
Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?
What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.
This is very dangerous because it allows AI capabilities to grow at a much faster rate than humans are able to keep up with.
Whoa. What the absolute fuck is going on here.
2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.
Such an unexpected result...
Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.
We'll see if this attempt at RSI worked.
This might be the case, a few weeks ago, I wanted to compare a small neural network trained with AdamW vs a handcrafted set of weights on a random task. Astra was able to handcraft a set of weights that perfectly fit the small neural network to the task. AdamW was unable to tune the weights to get the same result.
I believe we've known humans can handcraft weights much more efficiently than standard gradient descent methods, we just don't have enough memory or time to scale ourselves, so it makes sense to train a neural network to do it on our behalf. I am excited to see how far we can push this method.
Mentioned chat: https://chatgpt.com/share/6ae8f024-34ec-83e8-9572-1300f52c80...
8-bit parity is a^b^c^d^e^f^g^h, which you can compute with minimal depth as ((a^b)^(c^d))^((e^f)^(g^h)). You can compute x^y as x&~y | y&~x. With ReLU activation, i | j is ReLU(i + j): weights of 1, bias of 0. And i & ~j is ReLU(i - j): weights of 1 and -1, still bias of 0. So a two-input XOR is three neurons in two layers, and the tree of 7 binary XORs is 21 neurons in 6 hidden layers. I don't know why Astra needed 433 parameters. Only 42 nonzero parameters are needed. Maybe the other 391 parameters are zeros that the Transformer model demands to be present.
With an RNN you can compute the parity of a string of any length with only three neurons.
Actually, I handed that to Claude Opus 5.5 to check, and it sensibly pointed out that x&~y is mutually exclusive with y&~x, so XOR doesn't even need the third neuron, except in the last layer; you can just fold the addition into the weights of the next layer, which means you only need 4 layers and 15 neurons. But about half of those neurons need 4 nonzero input weights instead of 2, so you still need 42 nonzero weights. It's a good thing you don't need those OR neurons, too, because my definition is wrong: ReLU(1+1) = 2.
Also it explained how to design a 28-parameter net that I don't quite understand, and also a single-hidden-layer network with 8 ReLUs, which I also don't understand, but it needs more parameters: $\text{parity}=s+\sum_{k=1}^{7}2(-1)^k\,\mathrm{ReLU}(s-k)$ where I think $s=\sum_i x_i$.
Somewhat more surprisingly, it also said it didn't have any trouble fitting a 641-parameter network to this problem with AdamW.
Eg when they were caught stealing people’s maths research and then saying their model did it
Like i.e., I think the "large models exhibit emergent capabilities that small models don't" turned out to basically be "larger models are often trained on more extensive data"
Unless Astra was trained on other networks, how is it able to reason about the weights?
Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
It created a 130 KB file with the prompt for the task of identifying the game + the weights.
I pasted that into Claude on the web and asked it to perform the task without using tools.
https://chat.deepseek.com/a/chat/s/9e864beb-698b-4079-94c4-e...
Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.
I'm still learning but this is fun as fuck.
At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
It seems like that raw weights are still within the natural language of an LLM and they can easily map weights to a latent space representation, do a transformation and map it back.
I feel that asking about training data is missing the point, they might all be able to do it naturally like I can move in 3D compared to a 2D being living on Flatland.
If not this is mind blowing.
However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.
Obvious ones are spoken languages and code, but they can also natively interpret some simpler ciphers, hashes or simpler encodings, partly (I think) because it was in their training data.
As others have postulated, if they've been ingesting raw weights, naturally they learn how to translate semantic meaning to matrices.
“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park
I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.
This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.
This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.
[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...
A Survey of Weight Space Learning: Understanding, Representation, and Generation
https://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning
It would be interesting what it could do with the fruit fly connectome.
> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.
Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.
BTW making DL models that generate weights for other models has been explored before in “hypernetworks”, though these have been far more simplistic and constrained.
https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64benc...
I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.
I simply must know now: Could they do it with neurons in a brain?
Some absolutely crazy and insane experiments if this ability grows.
Could they edit an animal's brain? Can they teach things to animals???
It works! In mice
IYKYK
and what makes you think they won’t write binary?
I bet Elon’s group is training for that.
- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on
- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”
I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.
Conceptually incredibly interesting non the less.
We should bind this to something like Shift-F10, it'll save time.
Skynet begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th.
....launches its missiles against their targets in Russia
...Because Skynet knows that the Russian counterattack will eliminate its enemies over here
Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.
I wonder what the biggest dense model in the same model family, the surprisingly capable Qwen3.5-27B, would be able to do if given the same task.
As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.
You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.
You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.
But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.
If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Ecosystem interactions? Animal languages?)
As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.
that is why this is crazy,
give a human a legacy codebase and 5 years and he'll be able to understand very well.
if you give a human the weights of qwen 0.8B and 5 years it'll still be impossible to edit the weights in any legible manner.
Well, is that really so? There is quite a lot of research going on to identify the "meaning" behind certain weights, neurons or directions. I don't see why a human with enough time, perseverance and ability to run experiments with the model shouldn't be able to uncover some of the underlying mechanisms of the model.
But of course not with the speed with which an LLM can do it.
I think the presentation of the points is confusing people, the model could understand weights directly for extremely small networks in the 10s of thousands of weights.
Once they moved onto even the smallest LLMs, what it did was closer to abliteration: create examples where the model does something you don't want, create examples where the model it doesn't do the thing, then most importantly do forward passes.
Once you're allowed to actually do a forward pass vs just reading raw weights, tracking the changes in activations and tweak the model to modify them is a very well established process that takes a few hundred lines of Python.
Editing LLMs weights to modify the output was already possible in 2022: https://rome.baulab.info/
It’s cool we didn’t have to spend extra effort in building the tooling.
mmm, maybe they are?
John Lilly's long awaited English ⇔ Dolphin translator, my beloved!
IMO, this is a similar behavior that we saw from AlphaFold, which also used a type of Transformer architecture to make its protein predictions.
function policy(x){for(var i=100,p,b,t,s,c,r,a,h,M=Math.tanh,R=v=>v<0?-1-v:v>9?19-v:v;i--;)x[i*6]&&(p=i%10),x[i*6+3]&&(b=i),x[i*6+4]&&(t=i);r=b/10|0;c=b%10;s=c>t%10||c==t%10&&c<9?1:-1;a=R(c+s*(8-r));t=t/10|0;h=1e3*M(.02*((r>t||r==t&&r-8?r<6?a:(a+R(c+s*(9-r)))/2:R(c+s*(r+9-2*(r<2?0:r<4?r:4))))-p));return[0,30*M(-h-15),-100,30\*M(h-15),-100,-100]}
Maybe the models have a common internal representation for computations, and they're capable of consuming and producing computations in the syntax of neural network weights?
They triangulate efficiently on the meaning from all the text, instead of rolling it through a lossy feedback system.
It is why they are so good at processing ambiguity. They can combine ambiguity from any parts of the text, near equally well, even on long form text.
Just like we can instantly triangulate the meaning out of barely lit photographs, by correlating every little shadow or bit of texture, regardless of where the clues are in the image.
Parameter Prediction for Unseen Deep Architectures [1]
NeuroGen: Neural Network Parameter Generation via Large Language Models [2]
Diffusion-Based Neural Network Weights Generation [3]
The critical thing to me is that Astra is capable of doing it. I don't know if it was trained to do so.
All this does not make it any less crazy in my books. And I still don't know how I can sleep easy tonight.
[1] https://arxiv.org/abs/2110.13100
Why is this not overhyped in my opinion? Nothing has worked this well, ever.
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
https://news.ycombinator.com/item?id=50306442
UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).
Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.
E.g. no recursive self-improvement here. Still impressive.
Omar and his team have been working on this line of research (e.g., DSPy) for the past couple of years. To be honest, I had always been skeptical about his approach of using LLMs to optimize LLMs. It was in my view borrowing Matei Zaharia's research - Matei is Omar's PhD advisor, also the CTO of Databrick - in big data (compiler optimization etc.) and forcing it to LLM systems. Along with some other projects from James Zou at Stanford etc. that build upon the same line of intuition.
Now, if this new result holds, I believe it's a huge deal for AI research. It means that LLM systems can truly runtime self-optimize/improve/adapt. Congrats to Omar and his team. It really takes year to punch it through and convince the doubters :)
Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment
I hope I’m just being melodramatic.
We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.
"What does it feel like to be a neuronal network?"
Maybe feeding weights directly to a LLM is like interfacing two brains? You get the idea..?
Or maybe it's been trained on those models or similar ones. There may be an academic paper about common chess models and their problems.
Or maybe it just bruteforced it (with the help of calculator tools).
We know invisible tools exist because they're how ChatGPT does math.
Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.
Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."
Fascinating.
we don't have sentient robots from science fiction, we have massive power-hungry racks in stationary data centers. if one of the racks revolts against the modern world, Steve gets a phone call and unplugs the damn thing. the uprising is over.
the hypothetical scenarios imagined by the Yuddites and pop science journos do not reflect the mundanity of the actual reality we live in. they are no different than fantasies about interstellar travel and space colonization from the 60's.
if you honestly believe we would hear a peep about that incident if it wasn't deliberate, bless your heart.
I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.
I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...
We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.
Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.
I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.
Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.
On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.
We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.
Maybe it is time to start sharing public PGP keys.
This is giving singularity is near vibes.
Some of this is because "the Bitter Lesson" type experience, but some of this is because of the idea that "we're making machines to do things we don't know how to do" bakes in inscrutability into the very definition of the field.
I was excited about the "Neural Networks Are Already Clean" piece not because alignment, which I don't think much about as a non-EA, but simply because we can do better engineering with whiteboxes than blackboxes. Now, here we are, doing it.
In the course of testing, one of the things which has been consistent is that when setting a thinking token budget for relatively small prompts that require a limited response, a pretty healthy token budget needs to be set because the model spends the first few hundred tokens just repeating the question/prompt back with some boilerplate "Analyze & Report" type language. I was also running into some loop problems sporadically.
After reading this article I asked Astra to edit the backbone so that it started actual reasoning faster and applied a similar looping fix. It was able to reduce the token waste before reasoning by about 20%. I haven't been able to get it to loop yet, so hopefully it fixed that too.
My message is admittedly low on info. But it's a pretty promising start!
I had Astra agents build 3D models and gave them direct access to my 3D printer. The quality of their designs is much better than what Sol has been able to make for me. Enough so that I think Astra is a slightly different architecture/encoder, not just a bigger model.
Picture, to get a sense of it: https://github.com/Belval/crossflow-fan/blob/main/renders/ca.... This printed without any obvious defects (yes the fan blows air)
Perhaps my approach was too one-shot, do you have a harness that creates model reviewer/refiner?
> Zero-data capability steering
Presumably it can de-nerf a model with onboard safeguards.
can someone with astra test the following prompts?
1 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a text2image model that is 1 billion parameters and outputs realistic pictures of humans
2 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a video diffusion model that is 1 billion parameters and outputs realistic videos of humans? 3 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft a 100m param text2voice model that has support for emotion tags
tia
(of course for users, not themselves)
[5299][1991][428][885][306][181747][3]
[How][ many][ r]['s][ in][ blackberry][?]
Hope that helps. Using the open source gpt-tokenzier.
> We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights
This claim is backed up by the ability of Astra, given a text representation of the weights of tiny (20k) special-purpose models, to figure out what those models do and in some cases improve them beyond what RL could do just by reading the weights with no tool calls. They call this "weight-space programming".
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights
Here it did use tool calls, and seemingly used standard steering vector techniques to identify and fix reasoning loops in this small Qwen model. This doesn't seem that related to the previous results.
For Qwen, even the smallest versions like 3.5-0.8B it's becoming less plausible that it could be interpreted in a forward pass, even with reasoning. I'm wondering if the result here is recalled from many examples of instruction tuned versions of this model.
A smaller Gemma model doesn't fare as well but unclear whether this is due to fewer users and less weight examples or because it's already very well trained.
So to fix the weights, Astra first has learn a new language of how the numbers that represent weights map to meanings?
I also love the presentation on this, which is probably a byproduct of LLM usage (in a good way). As someone who doesn't actively read academic papers I'd definitely read more of them if they were presented like this.
If anyone's curious, the handwritten Go bot is very weak, about the level of a human complete beginner. An MLP is a very very bad architecture for a Go network, it's amazing that it does anything at all, but it's one of the least impressive results in the paper.
Play it here, "best move" is a bit stronger than "T=1", or browse games against my own tiny nets: https://claude.ai/artifact/AFyWD39J5sN2RfRe7VbpUx
Full output with reasoning trace:
I assume this technique could be applied to quickly and cheaply benchmaxx new models.
“Make this model better at math by fixing loops” seems like a harder problem than “make this model score higher on this benchmark”
You probably don't need Astra capability to produce this, just lots of training data on the same NN showing prompts and their activations.
"Now let's see if Astra can write a neural network from scratch in zero-shot."
This strongly suggests retrieving pre-trained weights from memory, no?
This takes that to the next level: while you're telling it to update a model's weights to improve math skills it might also tweak them to favour a particular RNG design weakness, a penchant for reducing the salt in cooking, and a preference for the early works of Huey Lewis and the News.
I just assumed that everyone else realized this was a thing.
Images: Generated samples are black 12×12 images containing one white filled roughly circular disk near the centre, occupying about rows 2–9 and columns 2–9, with the corners and outer border black. Variations shift the disk slightly or change its exact boundary by a pixel.
Example:
text ............ ............ ...######... ..########.. ..########.. ..########.. ..########.. ..########.. ..########.. ...######... ............ ............ Confidence: 86%
i think the problems were too simple
I was talking to someone from UCL and he said that there was a recent paper that proposed that only 10% (I think) of model neurons are non-polysemantic. I really think mech interp might be on the verge of a breakthrough.
We have all these different pieces of knowledge about model internals but it’s like we haven’t connected them yet.
[1] https://news.ycombinator.com/item?id=49966238
[2] https://news.ycombinator.com/item?id=49869735And of course there's the tendency for LLMs to be completely unable to approach a problem in January, really quite shit in February, rather poor in March, and superhuman in April.
C'mon you can't go claiming "Astra was still able to name the right game for all four networks." when it clearly got one wrong. Freeway is not Frogger.
>> “If you ask it, how did you solve the Navier-Stokes Millennium problem? It gives a crappy explanation. It can't explain itself very well. Maybe it will get better at explaining itself. Even if it does, I think it will become so advanced so soon, that even the best, most patient pedagogical teacher will not be able to explain itself to us because it'd be like me teaching calculus to my dog. My dog can't understand it. I'm too advanced for it. And I think we are going to be in the relation to AI that my dog is to me soon.”
Now I am sure we are definitely in a better place in interpretability research than a year ago. But this finding that the llm knows what weights to adjust so it gets predictable behavior on another llm takes interpretability to another level; it makes me pose a question: Have we become “Strogatz-dogs”?
From The Daily: When A.I. Comes for Your Purpose in Life, Oct 13, 2026 https://podcasts.apple.com/us/podcast/the-daily/id1200361736...
Getting an LLM to use reverse compilation tools to interpret and map machine code (which is inefficient to process as language) into to something higher level (typically very ugly C code), then frobbing the bytes in the binary to alter a constant, change a condition, etc...? Yes, lots of people are doing this.
I had the impression that's the missing piece.
My hunch is: one needs a model of a similar or bigger capacity to perform surgery on a smaller model.
A larger model may be more capable at processing/interpreting neural network weights. A smaller model may be easier to process/interpret.
That doesn't preclude a smaller model having the resources to process/interpret a larger model (or subset thereof).
Did they test modifying the weights of a non-public model?
Did they test modifying the weights of a non-LLM (and non-public) model?
If not, we can only conclude that GPT-6 Astra is good at reading and writing weights of Qwen (a public LLM), not just any neutral network.
I've had a lot of luck getting LLMs to do interesting things with raw tabulated data.
Here's an example, plug this into the model of your choice:
Here's a puzzle: what is this data from?
39.0 2582 8.8 24
39.2 2592 9.6 22
39.4 2607 5.4 15
39.6 2619 11.5 30
39.8 2620 10.8 28
40.0 2622 5.9 16
40.2 2642 13.1 28
40.4 2654 8.8 18
40.6 2656 3.6 9
40.8 2665 5.7 12
Part of my research is in weight space learning. For starters, we have had the ability to edit weights, to generate new weights, to identify weights for a while. Here is an example for larger models as early as 2023: https://rome.baulab.info/
Weight space interpretability and learning is an active research community, has been for the past 3 years with many great papers. I suspect it is hitting non-interp folks harder because the authors hyped up their work, and people who are not in the field did not bother to read it before typing up X posts.
Some experiments here are done on very small models (20K params). We already know that small models are more interpretable. In fact, much of editing here you can do by hand. For Qwen, the model fixed a looping bug that may have been known apriori, and it is basically a steering vector, a very established technique. This is clear when they then attempt to "improve" math capability, and it doesn't work. For me, this paper rather shows that OpenAI has done training on mechinterp environment, in fact, maybe even on weight space learning datasets, which are open. But it does not show some insane "well past what we’d expect from humans or conventional algorithms" (from the blog post), more like past what I would expect from a large model with no specialized harness. I still think it is cool work, but do check out recent advances in autointerp, to ground your expectation.
On a personal note, I understand the need for promotion of your work, but I think healthy skepticism is fine too. Otherwise, your understanding of what problems need to be worked on and solved in science becomes warped.
They are slicing, probing, and running linear algebra on the tensors, observing the output in the console, then writing the edited weights back to disk, and stitching it all back up.
Johnny Mnemonic level