How Researchers Look Inside a Model

A common belief about AI chatbots is that they are black boxes. A question goes in, an answer comes out, and nobody can say why that answer appeared. It is a reasonable first guess, because these programs hold billions of internal numbers, and no person can read them the way one reads a recipe.

That belief matters to anyone who relies on these tools for real work, since it decides how far an answer can be trusted and whether a wrong answer can ever be diagnosed. This post covers what researchers at Anthropic reported in 2025: a method that follows one specific answer backward through the model's internal steps to the words in the question that shaped it, with published limits. I also wrote a very small imitation of the core trick. By the end, the reader should understand what this tracing can show, what it cannot show, and why "nobody can explain it" overstates the case.

Technical detail: in my toy I traced a network's output back to its inputs in about twenty lines of NumPy, a Python library for numerical work, and the arithmetic closed exactly. The logit, which is the raw score a network assigns to an answer before that score becomes a probability, was -10.793481. The sum of the contributions from the six input nodes was -10.793481. The two numbers matched to six decimal places because the decomposition is an identity. My reading is that the trick behind that identity is the one Anthropic's researchers use at far larger scale, and it is the reason I no longer accept the description of a language model as a black box.

Technical detail: every weight in a model, meaning every number fixed during training, is visible to its developers, and every activation, meaning the value a unit holds while processing a particular input, can be recorded. The obstacle has always been legibility, since billions of numbers do not explain themselves. Mechanistic interpretability is the research area that tries to turn those numbers into explanations a person can read. Anthropic's 2025 work reports that the current generation of tools can trace a path from an output token (a word or word fragment the model produces) back through intermediate concepts to the input tokens that caused it, with stated limits.

Why individual neurons are hard to read

Picture a plant where every worker handles dozens of unrelated jobs, and every job is spread across dozens of workers. An auditor who watches one worker learns almost nothing about any single job. This analogy is mine, and it matches the difficulty researchers describe with the internal units of a language model, which are called neurons.

Technical detail: neurons are the obvious place to look for meaning in a network. Researchers working on image models had some success finding neurons that responded to recognizable visual concepts. Language models resisted that approach. Anthropic's methods paper describes the core difficulty in one sentence: model neurons are often polysemantic, meaning each one represents a mixture of many unrelated concepts. The paper names superposition as the leading explanation, in which the model must represent more concepts than it has neurons and smears each concept across many of them.

Technical detail: Part 3 covered the storage side of that idea, and the reading side is the problem here. If a concept lives as a direction spread over thousands of neurons, and each neuron participates in thousands of concepts, then inspecting a single neuron tells a researcher almost nothing. Studying whole components fares no better. The paper notes that large components, such as an entire attention head (a unit that lets each word position look at other positions) or an entire MLP block (the layer stage that transforms each position's values on its own), play a multitude of unrelated roles across the data distribution.

Features and transcoders

The workaround is to re-sort the work into labeled job tickets, one ticket per concept, so that each ticket can be read on its own. Researchers call these tickets features, and the tool that builds them is a kind of substitute model made of readable parts.

Technical detail: sparse autoencoders train a wide layer to reconstruct a model's activations using only a few active units at a time, and the learned units, called features, often correspond to human-readable concepts. Part 3 described the large-scale version that Templeton et al. reported in 2024 on Claude 3 Sonnet, with dictionaries of up to 34 million features and about a hundred active on any given token.

Technical detail: features describe what is represented at a point in the network. Explaining how one concept leads to another requires something that models the computation between layers, and that is where transcoders enter. Anthropic's methods paper describes a transcoder as a replacement for a layer's MLP, built from a sparse set of features that read from the layer's input and write to its output. The variant Anthropic uses, the cross-layer transcoder, extends this so that each feature reads from the residual stream at one layer, the running shared record that each layer reads and adds to, and contributes to the outputs of that layer and every later one. The consequence is a shorter circuit. A concept that the original model builds up over many layers can appear as a single feature with a long reach.

Technical detail: the paper describes training as a two-term optimization. A reconstruction loss, which measures how far the features' output sits from the real MLP output, pushes the features to reproduce each MLP's output. A sparsity penalty pushes most features to stay silent on any given token. The team trained cross-layer transcoders at sizes from 300 thousand to 10 million features on a small 18-layer model, and from 300 thousand to 30 million features on Claude 3.5 Haiku.

The paper also reports how faithful the substitute is. The largest cross-layer transcoder on the 18-layer model, with the learned features substituted for the model's MLPs, reproduces the original model's next-token choice in about 50% of cases on a set of pretraining-style prompts, filtered to tokens the model predicts correctly with confidence below 80%. My reading is that half is a notable result for a substitute built from interpretable parts, and a clear sign of how much computation the replacement misses.

Freezing the nonlinearities

A spreadsheet full of IF statements cannot say how much of the final figure came from which input, because the answer switches depending on which branch fired. Lock every IF to the branch it took for today's numbers, and the sheet becomes straight multiplication. Every output then splits exactly into a contribution from each input. The Anthropic method applies the same kind of move to a model, one prompt at a time. The spreadsheet comparison is mine.

Technical detail: a nonlinearity is a step whose output changes in a bent or switch-like way as its input changes, and a system full of them has no clean way to say how much of an output came from which input. The method resolves that for one specific prompt. It freezes the attention patterns and the normalization denominators (the scaling factors that keep numbers in a workable range) at the values the original model produced, and the transcoder features bridge over the MLP nonlinearities by taking each feature's activation on that prompt as given. With those pieces fixed, the way one feature's input depends on earlier features becomes linear, and linear systems decompose exactly.

Technical detail: next come error terms. The replacement model's reconstruction of each MLP output differs from the real output, and the method adds the difference back as an extra input, so that on this one prompt the adjusted replacement matches the real model's activations and logits exactly. The authors call the result the local replacement model. It rewrites the real computation on that prompt in different units. They compare it to a Taylor approximation, a straight-line estimate of a curve that agrees with the curve at one point and drifts apart as one moves away from it.

What follows is my own toy, a small imitation of the freezing idea in Anthropic's method. I built the smallest version I could to see the principle at work. A three-layer network with random weights stands in for the model, and six input numbers stand in for token embeddings, which are the lists of numbers that represent each word of a prompt. Each unit is a ReLU, which outputs zero when its input is negative and passes the input through otherwise. For a fixed input, I freeze each ReLU as a gate: open if the unit was active, closed if not. The code then computes how strongly each input reaches the output, using a backward pass, a calculation of how much the output would change per unit change in each input. Because the weights are random, the toy says nothing about any real model. It shows only that frozen gates give an exact account.

import numpy as np
rng = np.random.default_rng(3)
W1 = rng.standard_normal((8, 6)); W2 = rng.standard_normal((8, 8)); w3 = rng.standard_normal(8)
x = rng.standard_normal(6)

def forward(x):
    a1 = W1 @ x;  h1 = np.maximum(a1, 0)
    a2 = W2 @ h1; h2 = np.maximum(a2, 0)
    return w3 @ h2, (a1 > 0).astype(float), (a2 > 0).astype(float)

y, m1, m2 = forward(x)
J = (w3 * m2) @ W2 @ (m1[:, None] * W1)   # gates frozen: trace output -> input
contrib = x * J                            # one edge per input node

In my toy, the output is -10.793481, and the six edge values sum to -10.793481. The contributions by input node come out as -5.861, 0.433, 0.018, -8.335, 3.276, and -0.324. Input 3 dominates with -8.335, and inputs 0 and 4 pull in opposite directions with -5.861 and +3.276. A reader can tell at a glance which inputs mattered and in which direction. The takeaway is that the account is exact for this input with this set of open gates, and I make no claim beyond that.

The same toy shows where the exactness ends. The frozen account predicts what happens if input 3 changes: scale it down by a fraction alpha, and the output should move by alpha times its edge value. I then ran that change through my toy network and compared the two.

alpha=0.05  predicted    0.417  actual    0.417
alpha=0.25  predicted    2.084  actual    2.084
alpha=0.5   predicted    4.167  actual    3.239
alpha=1.0   predicted    8.335  actual    2.176
alpha=2.0   predicted   16.669  actual   -1.043

For small changes the prediction and the toy network agree to three decimals. At a 50% reduction they have started to part ways, at full removal they differ by a factor of almost four, and when the input is reversed in sign, which is alpha = 2, the toy network moves in the opposite direction from the prediction. The cause is that large changes flip some gates, and the frozen account has no way to know. The takeaway is that a local explanation holds near the point where it was built. My reading is that this parallels the Taylor comparison the authors draw, and Part 10 returns to what it implies for how far one can generalize from a single traced prompt.

What an attribution graph contains

Think of a bill of materials for one answer. Each finished output traces back through sub-assemblies to raw inputs, and a separate line records the cost nobody could explain. That is the shape of what the Anthropic researchers call an attribution graph, and the comparison is mine.

Technical detail: the paper describes four kinds of nodes. Output nodes represent candidate next tokens, restricted to those needed to cover 95% of the probability mass, up to a maximum of ten, and the authors note that fewer than three were usually enough in the cases they studied. Intermediate nodes are the active transcoder features at each token position. Input nodes are the embeddings of the prompt tokens, and error nodes carry the part of each MLP output that the transcoder failed to explain.

Technical detail: edges represent linear attributions, the same kind of quantity my toy computes in miniature. The paper computes them with backward Jacobians, which are matrices of derivatives (rates of change) from one point in the network to another, taken with gradients stopped through every nonlinearity: the MLP outputs, the attention patterns, and the normalization denominators. Stopping a gradient tells the calculation to treat a value as a constant. Because of those stops, the pre-activation of every feature, its raw input sum before it fires, equals the sum of its incoming edges. By that construction the graph offers a linear decomposition of each feature's activity, exact within the local replacement model, in the way my twenty-line version was exact within its frozen toy network.

Tracing from output to input follows from that structure. Start at an output node for the token the model chose. Its incoming edges name the features that pushed it up or down. Each of those features has incoming edges from earlier features, and the chain continues until it reaches the embeddings of the prompt tokens, or reaches error nodes where the explanation runs out. The Austin example from Anthropic's 2025 summary, which Part 2 described, is one such trace: the path runs from "Austin" back through "capital of Texas" and "Dallas is in Texas" to the tokens of the prompt.

Taming the size of the graph

A raw bill of materials for a whole aircraft is too long to read, so auditors keep the lines that move the total. Researchers face the same problem with these graphs, and they prune.

Technical detail: according to the paper, the replacement model has on the order of a hundred active features per token position, and the edges number in the millions even for short prompts. A pruning algorithm keeps the nodes and edges that exert significant influence on the output, and with default settings it cuts the number of nodes by a factor of ten while reducing the behavior explained by about 20%.

What remains still needs a human. The paper describes analysts labeling features by inspecting where they activate and which output tokens they promote, then grouping related features into supernodes, such as the several features that all fire on the word "digital" for the same reason. The paper reports that automated grouping methods fell short of covering the range of groupings needed to illustrate specific claims. Anthropic's public summary adds a practical figure: it currently takes a few hours of human effort to understand the circuits in a graph, even on prompts of only tens of words.

Checking the explanation against the model

Aerospace engineers learn to distrust a simulation until a change made to the real hardware produces the result the simulation predicted. The interpretability researchers apply that logic, and the comparison to aerospace is mine. An explanation derived from a replacement model might describe the replacement and miss the original, and the authors of the methods paper treat that risk seriously.

Technical detail: they validate by intervention. A researcher picks a feature or a group of features, changes its activation in the real model, and checks whether downstream features and the output move as the graph predicts. The examples Part 2 covered came from Anthropic's March 2025 summary of this procedure: suppressing the "rabbit" feature produced "habit," injecting "green" produced a line ending in "green," and swapping Texas features for California features turned Austin into Sacramento. The methods paper reports that perturbation experiments are generally qualitatively consistent with the graphs, with some deviations. In those experiments the attention patterns stay fixed at their original values, a choice the authors flag.

Open-source tooling

Anthropic announced in May 2025 that it had released an open-source library that generates attribution graphs on open-weights models, meaning models whose trained numbers are publicly downloadable. The announcement describes a hosted interface from Neuronpedia for exploring the graphs. The launch demonstrations used Gemma-2-2b and Llama-3.2-1b, with the Gemma graphs built on transcoders from the Gemma Scope project. According to the announcement, a researcher can generate a graph for a prompt of their choosing, annotate and share it, and test hypotheses by changing feature values. My reading is that this capacity to test claims on a chosen prompt fits poorly with the black-box description.

Limits of the method

Every measuring instrument has a range where it can be trusted, and the authors of the methods paper supply the list of blind spots themselves.

Technical detail: the graphs hold attention patterns fixed, so they explain what information moves between positions and leave open why the model attended there. The paper calls out induction, the behavior of continuing a pattern that appeared earlier in the prompt, as an example where this limitation makes the graph essentially useless, and describes finding many such cases. Reconstruction error means a portion of the computation sits in error nodes that explain nothing, and when the critical computation lives there the graph reveals little. Inhibitory circuits, which are features that matter by being silent or by suppressing others, are harder to capture. Features sometimes sit at the wrong level of abstraction. The authors also ask how confident anyone should be that transcoders replicate the original mechanism at all. Global circuits that hold across many prompts are harder still than the single-prompt graphs.

In my reading, those limitations place the field somewhere between two slogans. The black-box description overstates the darkness, since a researcher can now trace a specific answer through named intermediate steps and check the trace by intervening in the model. The claim that models are fully understood overstates the light by a wide margin. What exists is measured coverage: a method with published fidelity numbers, known failure modes, and a cost in human hours per prompt. The next two articles apply it to two behaviors that people care about, the decision to answer or decline and the relationship between a model's written reasoning and its actual computation.

What this comes to

The belief was that an AI model is a sealed box and nobody can say why it gave a particular answer. The corrected picture is that researchers have built a way to follow one answer backward through the model's internal steps to the words that shaped it, and they have tested the result by changing the model and watching whether it responded as predicted. The method has real gaps. It explains some answers well and others barely at all, and a single answer can take a person hours to untangle. The most useful thing to remember is that "can anyone explain it" has no yes or no answer. The useful question is how much of a given answer can be explained, and how much effort that takes.


Sources: Anthropic, "Circuit Tracing: Revealing Computational Graphs in Language Models" (2025). Anthropic, "Tracing the thoughts of a large language model" (March 2025). Anthropic, "Open-sourcing circuit tracing tools" (May 2025). Templeton et al., "Scaling Monosemanticity" (2024). The frozen-gate tracing toy and its numbers are my own code, reproduced above.