The Information Machine
ConcludedRan 7 daysin the edition since 17 Oct 20263 sourcesupdated 23 Oct 2026

Nick Cammarata's interpretability reversal

The gist

Latest: Goodfire Paper Claims Most Large-Model Neurons Are Legible

If the finding that most neurons are legible holds up under scrutiny, it substantially lowers the estimated difficulty of mechanistic interpretability work. The result is contested, and researchers disagree sharply on whether neuron-level legibility translates to meaningful progress on aligning advanced AI systems.

When stories like this move, I write about them in an email each morning.

One free email each morning, around 8 AM US Eastern.

The full picture

Nick Cammarata of Goodfire published research arguing that the vast majority of neurons in large language models are legible, reversing his earlier pessimism about mechanistic interpretability. In preliminary audits of Qwen3-32B, Goodfire estimates roughly 90% of neuron poles have one main job, another approximately 7% have two related jobs, and only 2-4% remain not yet understood. The paper extends these findings to Kimi K3 with similar results. What previously appeared as polysemanticity, neurons encoding unrelated concepts, is characterized in the paper as recursive generalization, where neurons branch into related sub-concepts. An earlier large-scale automated project at OpenAI had found only about 2% of neurons legible with primitive tools; Goodfire attributes the difference to modern agentic tools and closer examination.

The work drew sharply divided reactions. Roon of OpenAI wrote that it 'may be remembered as a major moment in scientific history' and that it could make alignment into a 'normal science.' Dean Ball called it potentially one of the most important moments in the field. Sam Altman responded positively. Joshua Achiam framed it as a helpful incremental improvement rather than a paradigm shift, noting that optimization pressure could still produce alien, impenetrable systems.

Critics were more pointed. Eliezer Yudkowsky argued the work is at the stage of 'explaining where Coca-Cola comes from or how it gets made' by tracing how a quarter falls through a vending machine, and said: 'If you think there is any possible level of interpretability that makes align ASI be normal science rather than maaaybe possible then you do not understand what ASI is or don't believe in it.' Neel Nanda of DeepMind said representational superposition is still real and that weak neuron activations likely still carry polysemantic information not addressed by the paper. Jake Mendel argued the larger unsolved problem is bridging from low-level feature enumeration to high-level alignment questions. Leo Gao warned that explanations that cannot simulate a neuron's activations have not really explained it.

An independent test found that Qwen3's 'Colorado' neuron fires equally strongly on the Tornado Python web framework, a result commentators said casts doubt on how clean the paper's findings are in practice. Commentators also noted that interpretability advances are inherently dual-use, capable of improving capabilities as much as alignment.

Zvi Mowshowitz raised a concern about research methodology: training an AI on its interpretability analysis outputs would cause models to obfuscate their internal reasoning, defeating the monitoring tool. He wrote: 'If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T].' In a subsequent assessment, Zvi characterized initial reactions as overclaiming the significance while still calling it 'excellent news that makes our estimate of difficulty substantially lower going forward' and adding: 'The work can begin.'

How it developed
22 October 2026in the 23 Oct edition

In a subsequent assessment, Zvi Mowshowitz characterized the initial wave of reactions to Goodfire's finding that roughly 90% of neuron poles in large models have a single primary concept as overclaiming its significance.

Zvi still called the result 'excellent news that makes our estimate of difficulty substantially lower going forward,' and closed: 'The work can begin'.

19 October 2026in the 20 Oct edition

Goodfire's Qwen3-32B audit found roughly 90% of neuron poles have one primary job and only 2-4% remain unexplained, with the paper attributing earlier apparent polysemanticity to 'recursive generalization'.

Critics pushed back: Neel Nanda said representational superposition is still real, Eliezer Yudkowsky argued identifying what neurons represent falls far short of making alignment tractable, and an independent test found the paper's example 'Colorado' neuron in Qwen3 fires equally strongly on the Tornado Python web framework.

First in the edition · 17 October 2026
16 October 2026in the 17 Oct edition

Nick Cammarata, who co-authored early mechanistic interpretability research at OpenAI, reversed his earlier pessimism about the field on October 15-16, saying that where primitive automated tools once found only around 2% of neurons legible, modern agentic tools show the vast majority of neurons in models up to trillions of parameters are legible and their circuits comprehensible.

He attributed the earlier pessimism partly to misunderstood cases that were not genuinely uninterpretable. Sam Altman responded positively to the announcement.

Sources

Get the daily email

When stories like this move, I write about them in an email each morning.

One free email each morning, around 8 AM US Eastern.

See the latest edition · Privacy