Where a Language Model's Memory Lives

Most people who use an AI chat assistant believe it listens, remembers what was said earlier, and learns about them as the conversation goes on. That belief matters because it shapes what people trust the assistant with, whether they expect it to know them from one day to the next, and why long conversations get slower and more expensive than anyone expects.

The picture I have come to hold is that the AI model starts from a blank slate with every message. The appearance of a continuing conversation comes from the software wrapped around the model, which keeps a written record and hands the entire record back each time. By the end of this post I want a reader to understand where that record lives, why carrying it costs real hardware and money, and what the three different kinds of memory in this system are. The facts come from published research and documentation that I name as I go. The arithmetic and the small calculations are mine, and I label them that way.

Technical detail: a language model is a very large set of learned numbers, called weights, that were fixed during training. The weights are frozen when the model is served, and nothing a user says changes a single one of them. Memory exists in three places at three time scales, and the weights stay fixed at every one of them.

The transcript travels with every request

Picture a repair depot where every technician starts each shift with no recollection of earlier jobs. The part arrives with a folder listing every step already performed. The technician reads the whole folder, performs the next step, and a clerk adds a note. In a chat, the application plays the clerk and the transcript is the folder.

Technical detail: when I send my fifth message, the application assembles the system prompt (the standing instructions the provider writes), my first four messages, the model's first four replies, and my new message into one long sequence of tokens. A token is a small chunk of text, often a word or part of a word. Special marker tokens delimit each role. The model processes that whole sequence from the beginning, produces a reply, and forgets everything. The application appends the reply to the transcript, and the cycle repeats.

That design has a consequence that shows up on invoices. The next paragraph is my own arithmetic on an invented conversation, offered to show the growth pattern. It makes no claim about any real product's billing. Suppose a 500-token system prompt, 100 tokens per user message, and 300 tokens per reply. The model sees 600 tokens of input on turn one. On turn two it sees 1,000. By turn ten it sees 4,200. I summed the series: across ten turns the model processes 24,000 input tokens, to cover a conversation whose unique text totals 4,500 tokens. The ratio is about 5.3 to one, and it grows with every turn.

My takeaway is that every turn pays again for the entire history. A long conversation costs far more than the sum of its parts because the model holds no state of its own between turns, so the transcript supplies it.

Learning without changing a weight

Imagine the technician receives three worked examples at the top of a work order and then follows the same pattern on the next part. The tooling has not changed, and when the folder is closed, the examples go with it. Something similar happens inside a conversation with a model.

In the GPT-3 paper in 2020, Brown and colleagues at OpenAI described a behavior they called in-context learning. With a few examples of a task in the prompt, the model performs the task on a new input, using identical weights and no training step. My reading is that the examples shift the model's odds over what text comes next, the same way a different first paragraph would shift the continuation of an essay.

Technical detail: the adaptation lives entirely in the activations, which are the intermediate numbers the model computes while it reads the text in the window. The window is the maximum amount of text the model can read in one request. When the window clears, the adaptation vanishes, and the next request begins from the same frozen model.

In my reading, a product that advertises memory across sessions almost always works by storing text somewhere else, retrieving it in the application, and inserting it into the prompt. The model's behavior changes because its input changed, in the same way that a differently worded question changes the answer. The word memory fits the user's experience well and fits the architecture loosely.

Why the cache exists

Think of a machine shop that keeps finished sub-assemblies in a staging bin. When the next part number arrives, the shop machines only the new step and pulls the rest from the bin. A language model faces the same situation, because reprocessing the full transcript for every new token would be ruinous.

Technical detail: the saving works because of the causal structure from Part 2, where the result at each position depends only on the tokens before it. Attention is the step where the model compares the newest token with every earlier one to decide which matter. Each earlier token offers a key and a value, which are stored summaries used in that comparison, and the newest token contributes a query, which is its question to them. The keys and values for the first 1,000 positions are identical when the model generates token 1,001 to what they were at token 1,000. They can be computed once and stored, and a new decoding step needs only the new token's query, matched against the saved keys and values from every layer. A layer is one of the repeated processing stages stacked inside the model.

The saved keys and values together form the key-value cache. I take it to be the most important piece of serving infrastructure after the weights themselves, and it is large. The size follows from the architecture, and the figures in this paragraph are my own arithmetic on the GPT-3 architecture that Brown and colleagues published (96 layers, 12,288 dimensions, a 2,048-token context), assuming 2 bytes per number. Each token needs a key vector and a value vector at every layer, so the cost is 2 × 96 layers × 12,288 dimensions × 2 bytes, which is 4,718,592 bytes, or 4.5 mebibytes per token. A mebibyte is about a million bytes. A full 2,048-token context therefore occupies about 9.7 gigabytes of GPU memory for a single conversation, to be set against the 350 gigabytes of weights that I derived from the published parameter count and that every request shares.

Technical detail: modern models shrink the cache with grouped-query attention, which shares each key-value pair across several query heads. Meta's Llama 3 team (Grattafiori et al., 2024) describes the 70B model with 80 layers and 8 key-value heads. The published Llama 3.1 and 3.3 70B configurations give a head dimension of 128 and a context window of 131,072 tokens. My arithmetic on those figures puts the cache cost per token at 2 × 80 × 1,024 × 2 bytes, or 320 kibibytes. With the whole window filled, a single conversation holds about 43 gigabytes, which is 31% of the 140 gigabytes the weights occupy at 16 bits.

My reading is that one long context costs about a third of a model's weights in memory, and that a server handling dozens of such contexts at once has to budget for it explicitly.

Technical detail: a batch is a group of requests processed together in one pass. The weights get read from memory on every decoding step regardless of how many requests share the step, so adding requests to a batch spreads that cost across more users. The caches grow with each request and set a ceiling on how many requests fit. This is my own account of why serving systems talk about batching and memory pressure in the same breath.

What the window does to attention

Consider an inspector handed a very long work order. The first pages and the last pages get close attention, and the middle gets a skim. Liu and colleagues reported something shaped like that in 2023, in a study of how models use long contexts. They found a U-shaped pattern: accuracy on questions about a document was highest when the relevant information sat at the beginning or the end of the context and fell when it sat in the middle. The paper's title, "Lost in the Middle," stuck.

The study tested the models available in 2023, and how much of the effect persists in current models is an empirical question I have no number for. My reading is that a context window sets an upper bound on what the model can attend to, and that the model's practical use of that space can be uneven.

In my framing, each new window also resets the model's view of the world. Whatever the application chooses to put in the window constitutes the model's entire working knowledge of the present: the date, the user's files, the earlier decisions in a project. Everything else comes from the frozen weights, which stopped learning at the training cutoff.

Where the cache meets the numerics

Consider a ledger that rounds to the cent at every subtotal. Adding the same invoices in two different groupings can leave the grand totals a cent apart. Computers face a version of this all the time, because they store numbers in floating-point form, which keeps limited precision and rounds at each step. The order of the additions can change the last digits of the result.

One more role of the cache connects directly to Parts 5 and 6. As I understand serving systems, they reuse cached prefixes across requests, so a shared system prompt gets processed once and its keys and values serve many users. They also process long prompts in chunks. Both optimizations change how the attention computation gets sliced, and slicing determines the order in which floating-point numbers get added.

Horace He at Thinking Machines Lab gives a concrete example in "Defeating Nondeterminism in LLM Inference" (2025). Technical detail: an attention kernel, which is a small program that runs one piece of the math on the GPU, processes keys and values in blocks of 32. In their example a sequence has 80 tokens already in the cache with 48 new tokens arriving. A kernel that handles the cached portion separately from the new portion needs three blocks for the 80 cached tokens (two full, one partial) and two blocks for the 48 new tokens (one full, one partial), five blocks in total. The same 128 tokens processed in a single pass would need four blocks. Same tokens and the same mathematics, with a different number of blocks and therefore a different order of accumulation, which produces slightly different bits. A token's result then depends on how much of its sequence happened to be cached when it was processed.

Part 6 covers the fix. The Thinking Machines write-up describes it as updating the cache layout before the attention kernel runs, so that the kernel always sees the same arrangement.

My reading of their example is that it overturns a comfortable assumption. The cache looks like a pure optimization that returns identical results faster. In exact arithmetic, which has no rounding, it does. In floating-point arithmetic, the way the cache divides the work can leave fingerprints on the output.

Three memories

A factory has three kinds of memory too. The tooling is fixed at installation. The staging bin holds work in progress for the jobs on the floor. The work order folder carries the history of one specific job. I now account for a language model's memory in the same three layers.

The weights hold what training put there, they are frozen at serving time, and they stay the same for every user and every conversation. The key-value cache holds the computed state of the current context, it lives in GPU memory for the duration of a request or a short while longer, and it exists to avoid recomputation. The transcript, together with whatever the application retrieves and inserts, holds everything the model will treat as the history of this interaction, and the application owns it entirely.

Seen that way, forgetting is the default condition of a language model. Every form of continuity a user experiences, from the thread of a chat to a remembered preference, is a feature an application built around a model that starts fresh each time. In my view that design has real advantages for reliability and for privacy, because the model's behavior on a given input depends only on that input. It also raises the stakes on the one thing the next article examines: whether a given input produces a given output at all once temperature is set to zero. Temperature is the setting that controls how much randomness the model uses when it picks the next token, and zero is meant to remove it.

What this comes to

Many people believe an AI assistant listens to a conversation and learns from it. The picture that fits what I have read is different. The model starts fresh with every message and keeps nothing between them. The continuity comes from the software around it, which saves the whole conversation and sends all of it again each time. That is why long conversations cost more, and why a feature called memory usually amounts to saved notes pasted in front of the next question. Memory sits in three places: the model's fixed learned knowledge, the temporary working notes the system keeps while it answers, and the written record the software controls. The single most useful thing to remember is that whatever the assistant seems to remember was handed to it a moment ago.


Sources: Brown et al., "Language Models are Few-Shot Learners" (2020), for in-context learning and the GPT-3 architecture (96 layers, 12,288 dimensions, 2,048-token context). Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023). Grattafiori et al., "The Llama 3 Herd of Models" (2024), for the 80-layer, 8-KV-head architecture. Thinking Machines Lab (Horace He), "Defeating Nondeterminism in LLM Inference" (2025). The token-growth and cache-size arithmetic is my own. The published Llama 3.1 and 3.3 70B configurations supply the 131,072-token window and the 128-dimension head size.