What Training Puts in the Weights

A popular belief says an AI chatbot is a giant compressed copy of the internet, and that every answer is the program looking something up, much like a search engine fetching a stored page. The belief matters to anyone using these tools, because it sets expectations about what the tool can quote exactly, where its facts come from, and why it sometimes states wrong things with confidence.

This post explains what training leaves inside the model. My reading of the published research is that the model keeps a compact set of learned patterns about language and the world, spread across billions of adjustable numbers, and holds very few stored pages. A reader should come away understanding why that picture fits the evidence, where the old picture partly holds anyway, and why a chatbot's helpful manner comes from a separate round of training.

The size arithmetic

Picture a shipping container that has to hold cargo more than three times its own volume, while also housing the crane that sorts the cargo. Nothing can be a full copy of the cargo in that arrangement. The model's storage and its training material stand in the same relationship.

Technical detail: OpenAI's GPT-3 paper (Brown et al. 2020, Table 2.2) reports 175 billion parameters and 300 billion training tokens. A parameter is one adjustable number inside the model, and the full set of them is called the weights. A token is a chunk of text, usually a word or a piece of a word. At 16 bits, or two bytes, per parameter, my arithmetic puts the weights at about 350 gigabytes. A common rule of thumb for English puts a token at roughly four characters, so my estimate of the training stream is on the order of 1.2 terabytes of text. The container is smaller than the contents by a factor of more than three, and that is before accounting for the fact that the weights also have to hold the machinery that does the computing. In my reading, the picture of a model as a compressed copy of the internet that gets searched at question time fails on this arithmetic before it reaches any philosophy.

The retrieval picture says the weights contain the training data and an answer is the model looking something up. The mechanism I see in the research that replaces it has three parts. Training builds a lossy statistical compression of text, meaning a summary that gives up detail to save space. The compression stores knowledge as overlapping patterns spread across many parameters. A second round of training then reshapes behavior on top of what the first round built. Each part changes what a person should expect from a model.

Training in one paragraph

Training works like tuning a production line. The line makes a guess, someone measures how far the guess missed, every dial gets a tiny adjustment toward a smaller miss, and the cycle repeats an enormous number of times. In the model's case, the guess is the next piece of a real sentence.

Technical detail: training runs the forward pass from Part 2, the calculation that turns text into predicted probabilities for the next token, on batches of text, which are groups of samples processed together. At every position it compares the predicted probabilities with the token that actually came next. The comparison is cross-entropy loss, a score equal to the negative logarithm of the probability the model assigned to the correct token, so a model that was more surprised gets a larger score. A gradient computation then works backward through the network to find how each weight contributed to the error, and an optimizer nudges every weight a small step in the direction that would have reduced it. Repeat for hundreds of billions of tokens.

Technical detail: the starting point for that loss is easy to compute. A freshly initialized model, one whose weights hold random starting values, assigns roughly equal probability to every vocabulary entry, so for a 50,257-token vocabulary my arithmetic gives a loss of ln(50,257) = 10.82 nats per token, or about 15.6 bits. Nats and bits are two units for the same surprise score, one built on the natural logarithm and one on base 2. In my own toy block from Part 2, a miniature with a 50-entry vocabulary and random weights, the correct token received a loss of 3.58 nats on one position. That figure comes from my toy and says nothing about production models. It shows only the scale of the number on a small vocabulary. Training pushes the number down across the whole corpus.

Here is the idea behind the numbers. A loss of 2.0 nats would mean the model needs about 2.9 bits to specify the next token on average, by my arithmetic. A model that predicts well has found short descriptions of the text, and short descriptions require finding regularities. Regularities are what get stored. That chain is how I understand the link between prediction and compression.

Scale sets the budget

Manufacturing offers a useful comparison. Output quality improves in a steady, predictable way as a plant adds machines, raw material, and energy. A bigger plant that gets too little raw material wastes much of its capacity, so the three need to grow together. The scaling research describes that pattern for language models.

Technical detail: Kaplan and colleagues at OpenAI reported in 2020 that test loss, the score on text held back from training, falls as a smooth power law in the number of parameters, the amount of data, and the compute used, across several orders of magnitude. A power law is a steady curve in which equal multiplications of the input buy equal-ratio improvements in the output, and compute is the total calculation effort spent on training. Hoffmann and colleagues at DeepMind refined the recipe in 2022. They report that their Chinchilla model had 70 billion parameters and trained on 1.4 trillion tokens, using the same compute budget as Gopher, which had 280 billion parameters and about 300 billion training tokens. They report that Chinchilla outperformed Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG on a broad set of downstream tasks. The authors concluded that model size and training tokens should scale together. Dividing 1.4 trillion by 70 billion gives my figure of roughly twenty tokens per parameter for the Chinchilla ratio.

I read those results as damaging to the retrieval picture. If the weights were storage, I would expect the 280-billion-parameter model, with more room, to hold more and win. The reported outcome has the smaller model, trained on more data, winning at the same compute. A different account fits better, in which the quantity being optimized is compression quality, and compression quality depends on how many examples shaped each regularity.

The GPT-3 paper's own data table adds a detail I find revealing. An epoch is one full pass through a dataset. Brown and colleagues list the filtered Common Crawl portion, 410 billion tokens and 60% of the training mix, as sampled for 0.44 epochs, which means by my arithmetic that the model saw less than half of that corpus even once. They list Wikipedia, at 3 billion tokens, as seen 3.4 times. My reading of the table is that most of the web text that shaped GPT-3 passed through the optimizer a single time or never, and the model still absorbed the regularities in it.

Knowledge as directions in a shared space

Imagine many radio stations sharing one band. Each station sits on its own frequency, and because only a few broadcast loudly at any moment, the band carries far more stations than it has clean channels. The price is occasional bleed between neighbors. Interpretability research suggests a model packs concepts together in a similar way.

If the weights hold no copy of the data, where does a fact live? The best current answer, as I read it, comes from the interpretability work on superposition, which studies how a network packs more concepts than it has room for.

Technical detail: a dimension is one independent number in the list that describes a representation inside the model, and a feature is a recognizable concept the model tracks. Elhage and colleagues at Anthropic published a set of toy-model experiments in 2022 reporting that a network with a given number of dimensions will represent far more features than it has dimensions, by assigning each feature a direction that is nearly orthogonal to the others, meaning pointed in a nearly unrelated direction so the features barely interfere. They report that this works because most features are rarely active at the same time. The cost is interference: features share space, and their overlap shows up as noise.

Technical detail: Templeton and colleagues at Anthropic applied the idea to a production model in 2024. They trained sparse autoencoders, separate analysis tools that learn to split a model's internal activity into individual features, on the residual stream, the shared running workspace that each layer reads from and adds to, in the middle of Claude 3 Sonnet. They worked at three sizes, with roughly one million, four million, and thirty-four million learned features, and reported that only about a hundred features were active on any given token. They found the features interpretable and varied: specific places and famous people, programming concepts, and abstract notions like sarcasm. Their most memorable demonstration involved a feature for the Golden Gate Bridge. When the researchers clamped that feature to a high value, the model began describing itself as the bridge.

Together these two papers give me a different picture to put in place of lookup. A fact such as "the capital of Texas is Austin" corresponds, in my reading, to a pattern of activation along some directions in a high-dimensional space, built by the many layers that wrote to the residual stream. The Austin example is my own illustration, drawn from neither paper. The pattern is spread across many neurons, the individual computing units inside the network, and I take this to mean the weights contain no addressable entry for Austin.

Retrieving the fact then means computing the activation pattern from the context. In my reading, that accounts for why a differently worded question can succeed or fail, and for how a model can combine two facts it never saw together, as in the Dallas example from Part 2.

Verbatim memory exists at the edges

A compression archive can summarize most drawings loosely and still keep a few part numbers exactly. Models behave somewhat like that archive. They produce word-for-word text in some cases, and the published evidence is specific about it.

Technical detail: Carlini and colleagues reported in 2021 that they extracted hundreds of verbatim sequences from GPT-2 using only query access, meaning the ability to send prompts and read the replies. The sequences included names, phone numbers, email addresses, and code. They found that memorization grew with model size in their study, and that some of the extracted sequences appeared in only a single training document.

Both observations hold at once. In my reading, the compression is lossy on average and exact on certain items, and I would expect those items to include text that appears many times or has distinctive structure, though that expectation is my own inference, which I attribute to no paper. For anyone responsible for data governance, the extraction result carries weight: a model trained on sensitive text can leak fragments of it, and Carlini and colleagues needed only the ability to query. My conclusion is that the retrieval picture overstates the model's fidelity in general and understates its memorization in the specific cases where the privacy risk concentrates.

What the second round of training adds

Pretraining resembles building a general-purpose machine shop. The second round writes the work instructions that tell the shop which jobs to accept and how to hand over finished parts. The shop's capability comes from the first stage, and its conduct comes from the second.

Pretraining is the first round, where the model predicts text across a huge corpus. A model that has finished pretraining predicts text. Asked a question, it may continue the question with more questions, because that is a plausible continuation of a list of questions. The behaviors people associate with an assistant come from a second phase of fine-tuning on demonstrations and preference judgments, where human raters compare outputs and the model is trained toward the preferred ones.

Technical detail: the InstructGPT paper from OpenAI (Ouyang et al. 2022) put numbers on the effect. The authors report that human raters preferred the outputs of a 1.3-billion-parameter model fine-tuned with human feedback over the outputs of the 175-billion-parameter GPT-3, a model more than a hundred times larger. They describe fine-tuning data orders of magnitude smaller than the pretraining corpus. They also reported improvements in truthfulness and reductions in toxic output, along with minimal regressions on public benchmarks, which are standardized tests that models are scored on.

From that result I take a division of labor between the two kinds of training. My reading is that pretraining supplies knowledge and capability and costs most of the compute, while post-training shapes which of the model's many possible continuations gets produced and costs a small fraction. By my reading, persona, refusals, and format habits all come from the second phase operating on the first.

The incentive structure of the two phases also explains a behavior I take up in Part 8. Pretraining rewards a guess at every position, since the loss never offers a way to abstain. Anthropic's researchers state the consequence in their 2025 write-up: "At a basic level, language model training incentivizes hallucination: models are always supposed to give a guess for the next word." My own inference from that statement is that whatever restraint a deployed model shows has to be built afterward.

Reading the evidence together

In my reading, the arithmetic at the top settles the first point: 350 gigabytes of weights cannot be a copy of 1.2 terabytes of text. Smooth improvement with scale, as Kaplan and Hoffmann report it, points me to a system fitting regularities. The superposition research points me to facts being computed when a question arrives. Carlini's extraction results put the privacy risk at the margin, in exact memorization of some items. A thin layer of additional training then decides how all of that gets expressed.

The forward pass from Part 2 is the machine, and training is how its numbers got set. Parts 4 through 6 turn to how the machine gets used, where memory lives at serving time and where variation comes from when temperature is zero. Both answers sit in details of the serving stack that most people never look at.

What this comes to

The common belief says an AI model stores a copy of its training material and looks answers up. The published research I have read points to a different picture. Training adjusts a vast number of internal dials until the model predicts text well, and that skill comes from finding patterns across the material, with far too little room to hold a copy of everything it saw. Knowledge ends up spread across the whole system as overlapping patterns, so an answer gets computed fresh each time. A few exact passages can still be memorized, which matters for privacy. A second, smaller round of training then shapes the helpful manner. The single most useful thing to remember is that an answer is a computation from learned patterns, and exact quotation is the exception.


Sources: Brown et al., "Language Models are Few-Shot Learners" (2020), Table 2.2. Kaplan et al., "Scaling Laws for Neural Language Models" (2020). Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022). Elhage et al., "Toy Models of Superposition" (2022). Templeton et al., "Scaling Monosemanticity" (2024). Carlini et al., "Extracting Training Data from Large Language Models" (2021). Ouyang et al., "Training language models to follow instructions with human feedback" (2022). Anthropic, "Tracing the thoughts of a large language model" (2025). The size arithmetic, loss arithmetic, tokens-per-parameter division, and toy-block figure are my own.