Neural Network Interpretability: Why the “Black Box” May Be More Understandable Than We Thought

Neural Network Interpretability studies which internal processes lead a model to produce an answer and how to explain them to people. Researcher Nick Komarata believes modern models may be more structured and explainable than commonly assumed. For now, this is a working hypothesis, not a proven theory.

Developers know how to train neural networks, improve their accuracy, and expand the range of tasks they can perform. However, they do not necessarily understand every step that led a model to a particular answer. A modern system includes millions or billions of artificial neurons and tens or hundreds of billions of parameters tuned during training. It is difficult to examine such a construct in its entirety, even when information about its architecture, weights, and input data is available.

The problem is especially apparent in tasks where errors involve safety, deception, or a hidden objective. An external test shows behavior on selected examples. But a model may recognize that it is being tested and behave differently in real-world deployment.

What AI Interpretability Studies

Interpretability addresses two different questions. Which elements of the input are associated with the answer? And what internal mechanism transformed the input into the result? A map of the importance of words or images helps answer the first question, but does not always reveal the model’s actual algorithm.

Mechanistic interpretability goes deeper. A researcher breaks a model down into components, forms a hypothesis about their function, and then tests it by intervening in the network’s operation. If removing or amplifying a component predictably changes the result, the explanation gains causal support. Correlation alone is not enough to draw this conclusion.

Why One Neuron Does Not Always Mean One Concept

Polysemanticity means that one neuron responds to several outwardly unrelated features. One possible cause is the limited number of available components: a model encodes more concepts than it can assign to separate neurons, and distributes signals across a shared space.

The theory of superposition describes this idea. Several features can use the same activation directions if this encoding helps the model conserve computational resources. A neuron that a researcher tries to label with one name sometimes turns out to be part of a more complex combination of features.

The Example of Cats, Cars, and a V Shape

In Inception V1, one neuron was initially associated with cat faces and the fronts of cars. A repeat analysis described by Komarata proposed a different explanation: the shared feature may have been V-shaped geometry. A similar shape appears in flames, branching streams of water, fox faces, and other images.

This explanation changes the meaning of the observation. The neuron did not necessarily mix two random concepts, cats and cars. It may have responded to a geometric feature found in both image classes. According to the researcher, rounded O-shaped forms triggered the opposite response.

Sparse Autoencoders and the Risk of a Convincing Error

A sparse autoencoder takes a model’s internal activations and tries to represent them using a small number of active, more understandable components. This approach helps separate mixed signals and find features that are not visible when analyzing individual neurons.

But good reconstruction does not yet prove that the identified components correspond to real functions of the original model. An autoencoder may find a convenient geometric description that accurately reconstructs an activation but misrepresents its meaning. In materials related to Komarata’s work, this risk is identified as one reason interpretability may be overstated.

The result also depends on the evaluation method. If you check only how well a description predicts a neuron’s activation, you can miss its main function. A visualizer can sometimes produce convincing images of dog faces, even if dogs are not related to the main role of the component being studied.

Recursive Generalization: A Neuron’s Function Changes Gradually

Komarata suggests viewing neuron development not only as the random mixing of features. Researchers use the term “recursive generalization” to describe the gradual broadening or shifting of a function toward related features.

In one experiment described, a neuron initially responded to yellow fields and valleys, then to neighboring visual features, including apples, tennis balls, dogs, and yellow flowers, and by the end of training was primarily associated with tennis balls, terriers, and broccoli. This observation does not prove a universal law of learning. It shows that a component’s function can change considerably as parameters are updated.

What Experiments with Architectures and Large Models Showed

After replacing an older image recognition architecture with ConvNeXt, researchers estimated that neurons in the final layer were understandable in 99–100% of cases, while keeping the model size and training data unchanged. The result depends on how understandability is defined and does not mean that the entire model became fully explainable.

In models that differed in size by approximately 300 times, the internal structure, according to Komarata, remained largely understandable. Small models encoded broader concepts, while large ones divided functions into narrower categories. But understanding individual components is not the same as understanding all the connections between them.

An initial analysis of Alibaba’s open language model with 32 billion parameters estimated that about 90% of neurons had one main function, approximately 7% were associated with two functions, and 2–4% were not understood. The figure of 90% cannot be directly compared with OpenAI’s result, where about 1,7% of neurons were considered explained. The models, analysis methods, and evaluation criteria differed.

The final estimate depends on how a single function is defined. If weak signals are considered insignificant, the share of understandable neurons will increase. If rare but consistent responses are included, the same component may turn out to be multifunctional.

Examples of Functions in Language Models

The materials describe neurons whose responses changed during training. One component was successively associated with the letter I, the abbreviation DIY, and the expression do it yourself. Another responded to fragments of code and Bengali text that researchers linked to the letter combination PR.

A neuron associated with the Titanic activated in contexts where the next likely words could be “bot,” “boat,” or “vessel.” After further analysis, a component that responded to commas was linked to the state of Colorado. Amplifying its activity prompted the model to suggest hiking plans.

Analyzing the most prominent activations can be misleading. A neuron’s function sometimes appears not in the word that triggers it, but in which next tokens it supports and how it affects the subsequent course of computation.

Can Behavior Be Controlled Through Internal Components?

Experiments amplifying components show a causal connection between internal activity and a model’s response. In one study, amplifying a component associated with the Golden Gate Bridge made the system frequently mention the landmark and describe itself as a bridge. According to the experiment description, amplifying the “Colorado neuron” similarly affected the generation of hiking plans.

Removing strong signals from individual neurons also changed behavior. After this procedure, the model stopped continuing the beginning of Psalm 23, and generated a fictional declaration instead of the text of the U.S. Declaration of Independence. The result points to the importance of individual internal signals for reproducing well-known texts. Detailed technical results require a separate publication and independent verification.

A user under the pseudonym Brangus reported an observation that complicates the Colorado example: according to them, the same neuron responded to code from the Tornado library for Python. If confirmed, this observation would mean the component may not have one clear-cut function. For now, this is a report, not an established fact.

Why AI Agents Are Used in Research

AI agents were used as automated researchers. They proposed hypotheses about neuron functions, devised tests, and conducted experiments. This approach makes it possible to study more components than fully manual work does. However, an agent can also mistake a convenient explanation for a correct one.

A reliable process includes several stages: describing activations, testing on new data, intervening in a component, and comparing alternative explanations. A hypothesis becomes stronger when predictions, causal experiments, and independent verification all agree.

What Prevents Us from Declaring the Problem Solved

• A model may contain multifunctional neurons and complex connections between layers.
• The interpretability metric depends on the threshold used to distinguish strong signals from weak ones.
• Reconstructing an activation does not guarantee that the explanation is mechanistically faithful.
• Understanding the final layer does not reveal all the computations performed by previous layers.
• Large models with trillions of parameters remain too complex for complete analysis.
• Comparing percentages without using the same architecture, data, and methodology creates a false impression of progress.

The distorted claim of “99–1% of neurons explained” cannot be considered a confirmed figure. Exact numbers and methodology should be checked against the original technical publication. Komarata estimates his confidence in the main conclusions at approximately 85%, and a full paper on language model neurons is still in preparation.

Safety, Self-Improvement, and the Concentration of Power

Interpretability may help detect deception, hidden objectives, and dangerous internal processes before they manifest in the real world. Internal analysis can supplement behavioral tests, which check only scenarios selected in advance.

Roman Yampolskiy warned that the same tool could accelerate recursive self-improvement. If a system learns to understand its own weaknesses and change its internal structure, interpretability will become not only a means of control, but also a way to increase AI capabilities.

According to a number of researchers, deliberately designing model behavior could increase the concentration of power among companies and organizations that control computational resources, data, and intervention tools. This is no longer a matter of a single technical metric, but of access to controlling complex systems.

Conclusion: What Can Be Considered Well-Founded

Research shows that individual neurons and small chains of components can sometimes be described in understandable language and tested through causal interventions. Sparse autoencoders, circuit analysis, and AI agents are expanding the scale of this research. Experiments with ConvNeXt and open language models provide grounds for cautiously reconsidering the view of a completely chaotic “black box.”

But understanding individual components does not mean understanding the model in its entirety. Claims of 90–100% of understandable neurons depend on the criteria, architecture, and quality of testing. Researchers have not yet uncovered all the internal processes of advanced neural networks, and Komarata’s hypothesis requires independent replication. The black box may be more structured than we thought, but it is still far from transparent.