What One Forward Pass Computes

A widely repeated belief says that an AI chatbot works like the autocomplete on a phone. According to this belief, the system keeps an enormous table of which word usually follows which, looks up the likeliest next word, and does no thinking in between. The belief matters because it sets expectations. A lookup table can only repeat what it has seen, while a system that computes can work out answers to questions nobody has asked before.

This post is part 2 of a 10-part series on how these systems work. It follows one trip of a piece of text through a model, from the moment the text goes in to the moment a guess for the next word comes out. I use published numbers, a small program I wrote that imitates the published design, and research from Anthropic that looked inside a working model. By the end, the reader should understand that the guess is the final step of a long chain of calculations, and that researchers have reported finding intermediate steps inside that chain which a lookup table would have no place to store.

How big the calculation is

Think of the model as an assembly line whose size can be read from its blueprint. The blueprint for GPT-3 is published, and counting the adjustable settings on the line takes a few lines of arithmetic. Those settings are called weights, and they are the numbers the model learned during training. The total count of them is the parameter count.

Technical detail: Brown et al. (OpenAI, 2020) published GPT-3's configuration. It has 96 layers (stages the data passes through in sequence), a model dimension of 12,288 (the length of the list of numbers that represents each token, where a token is a word or a piece of a word), and a vocabulary of 50,257 tokens. Kaplan et al. (OpenAI, 2020) give a rule of thumb that a transformer's parameters, apart from the embedding table, come to twelve times the layer count times the square of the dimension. Applying that rule to the Brown et al. configuration, my arithmetic gives 173,946,175,488. I then added the embedding table from Part 1 (the grid that assigns each vocabulary entry its list of numbers), which by my count holds 617,558,016 numbers. The total is 174,563,733,504, within a quarter of a percent of the 175 billion that the Brown et al. paper reports.

Every one of those numbers is a weight in a matrix, which is a grid of numbers, and every matrix exists to transform a vector, which is a list of numbers. My reading is that the model's behavior comes from composing 96 rounds of those transformations. That makes the object under discussion a very large piece of arithmetic.

Where the twelve comes from

Each stage of the line has two workstations. One lets the pieces of text consult each other, and the other lets each piece be worked on individually. The factor of twelve is the combined size of those two workstations.

Technical detail: the factor falls out of the architecture that Vaswani et al. (2017) introduced in "Attention Is All You Need." Each layer of a transformer has two sublayers. The attention sublayer contains four d × d matrices (d is the model dimension), which project the input into queries, keys, and values and then project the combined result back out: 4d² parameters. The feed-forward sublayer expands the vector from d to 4d dimensions and contracts it back, which costs 4d² for the way up and 4d² for the way down: 8d². Four plus eight is twelve per layer, and the layer count multiplies that. Layer norms, biases, and position embeddings add a smaller amount that I did not count, and 175 billion is a rounded figure.

I like starting here because it grounds the discussion in something checkable. The model is a specific stack of matrix multiplications with a known shape, and a person with a calculator can recover its size from its published configuration.

The block, in code

The code below is a miniature of one stage of the line. I wrote it in about thirty lines of Python using NumPy, a library for numeric work. It is my own small imitation of the design Vaswani et al. describe, and it uses random numbers where a real model has learned weights, so its predictions are meaningless. It exists to show every ingredient of a stage in a form short enough to read in a minute, and a reader who skips the code loses nothing in the sections that follow.

import numpy as np
rng = np.random.default_rng(7)
V, d, H, T = 50, 32, 4, 6
E  = rng.standard_normal((V, d)).astype(np.float32) * 0.5
Wq, Wk, Wv, Wo = [rng.standard_normal((d, d)).astype(np.float32) / np.sqrt(d)
                  for _ in range(4)]
W1 = rng.standard_normal((d, 4 * d)).astype(np.float32) / np.sqrt(d)
W2 = rng.standard_normal((4 * d, d)).astype(np.float32) / np.sqrt(4 * d)

def rmsnorm(x): return x / np.sqrt((x ** 2).mean(-1, keepdims=True) + 1e-6)

def forward(ids):
    x = E[ids]                                   # residual stream starts as embeddings
    h = rmsnorm(x)
    q, k, v = h @ Wq, h @ Wk, h @ Wv
    n, dh = len(ids), d // H
    out = np.zeros_like(x)
    for i in range(H):
        s = slice(i * dh, (i + 1) * dh)
        scores = q[:, s] @ k[:, s].T / np.sqrt(dh)
        scores = np.where(np.tril(np.ones((n, n))) == 1, scores, -1e9)  # causal mask
        w = np.exp(scores - scores.max(-1, keepdims=True)); w /= w.sum(-1, keepdims=True)
        out[:, s] = w @ v[:, s]
    x = x + out @ Wo                             # attention writes into the stream
    x = x + np.maximum(rmsnorm(x) @ W1, 0) @ W2  # MLP writes into the stream
    return rmsnorm(x) @ E.T                      # unembedding: logits [T, V]

Takeaway: one stage of the line is a short, readable recipe. The block contains multi-head causal attention, a feed-forward sublayer, normalization (rescaling numbers so they stay a manageable size), residual connections, and an unembedding step that turns the final vectors into one score per vocabulary entry. A full model repeats a recipe of this shape many times, and my toy runs it once.

Three ideas carry the whole design, and the first is the residual stream. Picture the work order that travels with a part down a manufacturing line. Each station writes its additions on the order, and no station erases what an earlier one wrote. By the end of the line, the order holds the original specification plus every contribution made along the way.

Technical detail: the residual stream is the running vector at each position. It begins as the token's embedding and receives additions from every layer. Attention and the feed-forward sublayer each compute something and add it to the stream, and nothing overwrites it. By the last layer, the stream holds the embedding plus the accumulated contribution of every component that ran. Researchers who study model internals describe it as the shared workspace of the model.

The second idea is a division of labor between attention and the feed-forward sublayer. Attention works like a status meeting where each piece of text hears from every earlier piece before deciding what to do. The feed-forward step works like each station doing its own job on its own part. Communication happens in the first and computation happens in the second.

Technical detail: in the design Vaswani et al. introduced, attention is the only place where information moves between positions. For each position, it scores every earlier position, converts the scores to weights with a softmax (a formula that turns a list of scores into percentages that add up to 100), and takes a weighted average of their value vectors. The feed-forward sublayer works on each position by itself, applying the same two matrices to every position in parallel. A model with 96 layers alternates between the two ninety-six times.

The third idea is the exit. At the end of the block, the final vector at each position gets multiplied by the transpose of the embedding table, which yields one raw score per vocabulary entry. Those raw scores are called logits. A softmax over them gives the probability of each possible next token. Many models reuse the input table as the output table in this way, and my toy does the same.

Two properties of one forward pass

A forward pass is one trip of the input through every stage to the final scores. I ran my toy block on a six-token input and checked two properties that matter for everything downstream. The first asks whether the same input always gives the same output. The second asks whether a token can influence the scores at earlier positions. The code prints True or False for each, and the comments show the results I got.

ids  = np.array([3, 17, 8, 42, 8, 5])
a, b = forward(ids), forward(ids)
print(np.array_equal(a, b))          # True: same input, same logits, bit for bit

ids2 = ids.copy(); ids2[-1] = 11
c = forward(ids2)
print(np.array_equal(a[:5], c[:5]))  # True: earlier positions ignore later tokens
print(np.array_equal(a[5], c[5]))    # False: the last position responds

The first check shows that my toy forward pass is a pure function, which means identical input always produces identical output. Identical tokens produce identical logits on this code path, with nothing random inside it. This result describes my toy only. I will return to the fact in Parts 5 and 6, because it makes the nondeterminism seen in production systems (the same input giving different outputs) a puzzle that needs an explanation.

The second check exposes the causal structure, meaning the rule that each position may read only the positions before it and itself. Because of the mask, the logits at position five depend on tokens zero through five, and the logits at position four never see position five. In my toy, changing the last token changed only the last position's scores.

Technical detail: this property gives training its efficiency. A single pass over a sequence of 2,048 tokens (an illustrative length of my choosing) yields 2,048 separate next-token predictions, one per position, and each prediction is scored against the token that actually followed. One forward pass during training therefore produces thousands of learning signals.

Generation as a loop

Generating text wraps that function in a loop, like a factory that feeds each finished item back in as raw material for the next. Run the forward pass on the prompt, read the logits at the final position, choose a token, append it to the sequence, and run again.

Technical detail: Kaplan et al. (2020) give a rule of thumb that one forward pass costs roughly two floating-point operations per parameter per token. A floating-point operation is one addition or multiplication on a decimal number. My arithmetic with that rule and the 175 billion parameters reported by Brown et al. gives hundreds of billions of operations for every token produced. The loop explains why long answers take visible time, and why serving systems devote so much engineering to caching, which means saving earlier work so it is not repeated. Part 4 covers caching.

The loop also explains why the phrase "predicts the next word" sounds smaller than it is. My argument is that predicting the next token well, across the whole range of text people write, requires some model of whatever produced the text. The continuation of a proof depends on the mathematics, and the continuation of a stack trace depends on how the program works. A sentence about the capital of the state containing Dallas needs geography to finish correctly. The training objective only measures the final guess. My reading is that the cheapest way for a deep network to guess well across all of that text is to build intermediate computations that track those underlying facts.

What the intermediate computations look like

Interpretability is the research field that opens up a trained model and examines what its internal parts do. In March 2025, Anthropic published a pair of papers that trace the internal computation of Claude 3.5 Haiku on short prompts. The researchers describe internal features, which are patterns of internal activity that they match to human concepts such as "Dallas is in Texas." Three of the results they report bear on the autocomplete claim, in my reading.

The first result concerns two-step reasoning. Asked for the capital of the state containing Dallas, the model answers Austin. A lookup model would store the question and its answer together. The researchers report tracing a two-stage chain: features for "Dallas is in Texas" activate first, and a separate set of features for "the capital of Texas is Austin" take that as input. They then intervened on the intermediate step, swapping the Texas features for California features, and report that the output changed from Austin to Sacramento. My reading is that an intermediate representation which can be swapped to change the answer is what a computation looks like from the inside.

Planning comes second. The team gave the model the first line of a rhyming couplet, "He saw a carrot and had to grab it," and report that they expected a model that wrote word by word and scrambled for a rhyme at the end. They found that features for candidate rhyme words, including "rabbit," became active before the model started writing the second line, and that the line was composed to arrive at the chosen word. They report that suppressing the "rabbit" feature led the model to end the line with "habit," and that injecting a "green" feature led to a line ending in "green." The authors write that they had set out to show that the model did not plan ahead, and that the evidence went the other way.

Arithmetic comes third. The model adds 36 and 59 correctly, and the researchers report two parallel pathways, one computing a rough magnitude for the sum and another determining the last digit precisely, with the two combining at the end. Asked how it did the addition, the model describes the schoolbook method with a carried one. My reading is that the description comes from the text the model learned to imitate, and that the computation underneath follows a different route. I will come back to that gap in Part 9.

All three results come with caveats that the authors state themselves. They report that their method captured a fraction of the total computation on short prompts, that the circuits come from one model, and that understanding a single circuit took researchers a few hours of effort. I read that as enough evidence to retire the lookup-table picture and well short of a complete account of any model's reasoning.

Depth as a budget

Picture an assembly line with a fixed number of stations. A job that needs more sequential steps than there are stations cannot finish in one trip down the line. The line can send partial work back to the start, and for a language model the way to do that is to write the partial work into the text.

Technical detail: my reading of the architecture is that a model with 96 layers can chain at most a bounded number of sequential operations in a single forward pass, since each layer adds one stage of computation. Problems that require a longer serial chain than the depth provides call for another mechanism, and I take writing intermediate steps into the text to be the standard one. Each generated token is an opportunity to feed new work back into the next pass. In my reading, that is the structural reason models answer multi-step problems better when they reason out loud, and Part 9 takes up the question of whether the reasoning they write down matches the reasoning they perform.

For now, the picture I carry from this exercise is simple. Tokens enter, a fixed stack of attention and feed-forward layers transforms a shared residual stream, and logits come out as a pure function of the whole prefix. Next-token prediction describes the interface, and a deep circuit sits behind it.

What this comes to

A common belief holds that an AI chatbot is a fancy autocomplete, a giant table of which word usually follows which. The picture that fits the evidence I read is a long assembly line. Text enters at one end and passes through stations that each read the work done so far and add to it. GPT-3 has 96 of those stations. A ranked list of candidate next words comes out the other end. Anthropic's researchers, studying a different model, reported seeing intermediate steps along the way, such as working out a state before naming its capital, and settling on a rhyme word before writing the line that ends with it. The single most useful thing to remember is that the final guess is the last step of a great deal of computation.


Sources: Brown et al., "Language Models are Few-Shot Learners" (2020), for the 96-layer, 12,288-dimension configuration and the 175 billion parameter figure. Vaswani et al., "Attention Is All You Need" (2017). Kaplan et al., "Scaling Laws for Neural Language Models" (2020), for the parameter-count and per-token compute rules of thumb. Anthropic, "Tracing the thoughts of a large language model" (2025). Anthropic, "On the Biology of a Large Language Model" (2025). My own work, reproduced above: the parameter arithmetic from the published specifications, and a NumPy block that imitates the Vaswani et al. design in miniature. Neither is evidence about production models.