What a Language Model Reads: Tokens and Embeddings
A language model has never seen a word. It has never seen a letter. The common picture says an AI chatbot reads a question the way a person does: the words arrive, the meaning gets understood, a reply gets composed. The machine works one step earlier than that picture suggests. Every model I have worked with first receives a list of whole numbers, each one standing for a piece of text from a fixed catalog, and everything that happens afterward happens to those numbers.
That detail matters to anyone who uses these tools at work. It explains why a chatbot can write a fluent page and still miscount the letters in a word, why long sums go wrong in odd ways, and why the bill depends on a unit of text that most users never see. This is the first of ten articles in which I take one common misunderstanding about large language models and work through the mechanism that replaces it. After this one, the reader will understand what the machine actually receives, why that changes its behavior and its price, and where its sense of meaning begins. The ten articles follow the life of a request: what goes in, what computation happens, what the learning process stored inside the model, where memory lives, where variation in answers comes from, and finally how researchers trace an answer back to its inputs. Part 1 covers the front door.
The input is a list of integers
Picture a parts warehouse where the robots read only catalog numbers and never part names. A clerk at the loading dock converts every order into catalog numbers before any robot touches it. In a language model, that clerk is the tokenizer, the program that cuts text into pieces called tokens and swaps each piece for its number in a fixed catalog called the vocabulary.
Brown and colleagues, in the 2020 paper that introduced GPT-3, reported a vocabulary of 50,257 entries. As an illustration of my own, a sentence of eight English words might become eleven numbers. Those eleven numbers are the entire input.
Technical detail: the catalog comes from a compression procedure that counts how often pairs of symbols sit side by side. Byte-pair encoding, introduced for neural translation by Sennrich and colleagues in 2015, starts with individual characters and repeatedly merges whichever adjacent pair occurs most often in a training corpus, a large body of text. It stops when the vocabulary reaches a target size. Frequent strings end up as single tokens, and rare strings stay in fragments.
The cleanest way I know to see this is to run the procedure on a corpus small enough to hold in mind. The program below is my own small re-implementation of the merging method Sennrich and colleagues describe, run on the four-word example corpus that their paper uses, with a frequency attached to each word. It shows how repeated merging of the most common neighbors builds bigger pieces out of letters, and it makes no claim about any real model's catalog.
from collections import Counter
corpus = {"l o w </w>": 5, "l o w e r </w>": 2,
"n e w e s t </w>": 6, "w i d e s t </w>": 3}
def pair_counts(c):
p = Counter()
for word, freq in c.items():
s = word.split()
for a, b in zip(s, s[1:]):
p[(a, b)] += freq
return p
def merge(c, pair):
a, b = pair
out = {}
for word, freq in c.items():
s, n, i = word.split(), [], 0
while i < len(s):
if i < len(s) - 1 and s[i] == a and s[i + 1] == b:
n.append(a + b); i += 2
else:
n.append(s[i]); i += 1
out[" ".join(n)] = freq
return out
c = corpus
for step in range(10):
p = pair_counts(c)
best = max(p, key=lambda k: (p[k], k))
print(step + 1, best, p[best])
c = merge(c, best)
print(c)
Each line of output below is one merge, showing the pair that was joined and how many times it occurred. The ten merges come out like this when I run it:
1 ('t', '</w>') 9
2 ('s', 't</w>') 9
3 ('e', 'st</w>') 9
4 ('o', 'w') 7
5 ('l', 'ow') 7
6 ('w', 'est</w>') 6
7 ('n', 'e') 6
8 ('ne', 'west</w>') 6
9 ('low', '</w>') 5
10 ('w', 'i') 3
{'low</w>': 5, 'low e r </w>': 2, 'newest</w>': 6, 'wi d est</w>': 3}
The final line holds the lesson of my toy run. The word "newest" has become a single token because it appeared six times. The word "lower" is still three pieces, low, e, r, because it appeared twice. The word "widest" is wi, d, est. The suffix est emerged on its own, because "newest" and "widest" share it, and frequency alone produced it.
The takeaway is that common strings get their own catalog number and rare strings get chopped into fragments. My reading is that the same procedure, run on a vastly larger corpus, produces the vocabularies that production models use, with common English words as single tokens and a long chemical name or a stretch of source code broken into whatever fragments happened to be frequent. My toy shows the mechanism at four words. It cannot show what any production catalog actually contains.
What the grain of the input changes
Picture a clerk who can see only sealed shipping crates and has to report how many bolts are inside one. The clerk can answer only from memory of what such crates usually hold. A model faces a similar situation with the letters inside a token.
Questions about the interior of a token ask about structure the input never exposed. How many times does a particular letter appear in a word? The word may arrive as one token, a single number with no letters inside it. In my reading, the model can still answer because its training text contains plenty of material that spells words out, so it recalls a pattern about that token. A child finishes the same task by looking at the page, and that gap is my explanation for why a model can write fluent paragraphs and still stumble on letter counts.
Numbers show the effect with a measurement attached. Singh and Strouse published a study in 2024 on how tokenization affects arithmetic in the large commercial models of the day. Think of a ledger whose digits are grouped from the wrong end, so that the columns never line up with ones, tens and hundreds.
Technical detail: Singh and Strouse describe tokenizers that group digits from the left in chunks of three, so the number 1234567 splits as 123, 456, 7, which misaligns the digits with their place values. The authors forced a right-to-left grouping by inserting commas, so the same number reads 1,234,567 and the chunks line up with thousands and millions. They reported that GPT-3.5's addition accuracy rose from 75.6% to 97.8%, and that GPT-4's rose from 84.4% to 98.9%. By my arithmetic from their figures, GPT-3.5 gained 22.2 points. The model's learned numbers and the prompt content stayed fixed, and only the cut of the digits changed.
My reading of that result is that it separates two things people routinely merge. Arithmetic skill lives in the weights, the numbers a model adjusts during training. The ease of expressing the problem lives in the tokenizer. A model with a good grasp of addition can still fail when the input format hides the place values from it.
From integer to vector: the embedding table
Each catalog number points to a spec sheet, a long row of numbers that describes the piece. Parts with similar uses end up with similar spec sheets. The first operation inside the model performs that lookup, and the row of numbers it returns is called a vector.
Technical detail: the model owns a matrix E, a grid of numbers with one row per vocabulary entry and one column per model dimension. Token 17 receives row 17. Brown and colleagues report a dimension of 12,288 for GPT-3 and a total of 175 billion parameters, the adjustable numbers learned in training. By my arithmetic from those published figures, the table holds 50,257 × 12,288 = 617,558,016 numbers. That single lookup table accounts for roughly 0.6 billion of the model's 175 billion parameters.
Mathematically the lookup equals multiplying a one-hot vector, a list that is all zeros except for a single one, by the matrix. The snippet below is my own small NumPy check of that standard identity, written in Python with a numerical library. It compares picking row 17 directly against the multiplication, and the table E and its row count V come from my test script, which I show only in part. The equivalence matters later, when the series reaches how researchers read features out of a model's internal activity.
oh = np.zeros(V, dtype=np.float32); oh[17] = 1
print(np.array_equal(oh @ E, E[17])) # True
The output True means lookup and multiplication gave identical results in my script. The identity is ordinary linear algebra, and my check says nothing about any production model. The takeaway is that "look up a row" and "multiply by a grid" are two descriptions of one operation.
Technical detail: the rows begin as random noise. Training adjusts them by gradient descent, a method that makes many small changes to reduce the model's errors, until tokens that behave similarly in text end up near each other in the space. The origin story for this idea is the word2vec work of Mikolov and colleagues, who reported in 2013 that arithmetic on such vectors captured relations: the vector for "king" minus the vector for "man" plus the vector for "woman" landed near "queen."
In my reading, modern models inherit the same principle at much larger scale. Word2vec assigned one fixed vector per word, and a transformer's embedding table behaves the same way at its first step, so "bank" starts with the same vector whether the sentence concerns rivers or loans. Context arrives afterward, through attention, the mechanism that lets each piece of text adjust its meaning using the pieces around it. Part 2 covers attention.
Technical detail: a bag of token vectors has no order, so the model needs position information. As I understand it, GPT-3 added a learned vector for each position, and many recent open models rotate the query and key vectors, the two lists of numbers that attention compares, by an angle that depends on position, a scheme called rotary embeddings. Either way, the model receives both what the token is and where it sits.
What sits between integers and integers
Between the order form and the shipped crate sits an assembly floor, and researchers have begun to map it. If the model reads numbers and writes numbers, a fair question is what happens in between.
Anthropic's interpretability team published research in March 2025 that traced the internal computation of Claude 3.5 Haiku, and it gives one of the more concrete answers I have seen. The team asked the model for "the opposite of small" in several languages and traced which internal features activated, where a feature is a pattern of internal activity that corresponds to a recognizable concept. They reported that the same core features for smallness and oppositeness fired regardless of the language of the question, and that a feature for largeness then fed into whichever language the answer needed. The team also reported that the shared fraction grows with model scale, and that Claude 3.5 Haiku shared more than twice the proportion of its features between languages compared with a smaller model.
In my reading, that pattern points to a division of labor. The surface layers handle the tokens of a particular language, the middle layers work in something closer to a shared space of concepts, and the final layers convert back to tokens in the language of the answer. Anthropic's authors present the result as suggestive evidence from one model on a few simple prompts, and I treat it the same way. I use it as a mental model: integers in, a concept-like workspace in the middle, integers out.
The grain of everything downstream
In my reading, tokenization looks like a preparation step and works as a design decision for everything after it. Think of freight billed per pallet: the pallet becomes the unit of handling, storage and cost, whatever the cargo happens to be.
The unit the tokenizer chooses becomes the unit of attention and the unit of cost. It also sets the unit of the context window, the maximum amount of text the model can consider at once, and the unit in which output probabilities are expressed. When a vendor quotes a context window of 200,000 tokens, which I use here as an illustrative figure, the number of words that fits depends on how well that tokenizer compresses the language and the content in question. When a model charges per token, a language that fragments heavily costs more for the same meaning.
Every later article in this series inherits that grain. The next one follows the numbers through the layers and shows what one forward pass, a single trip through the model that produces a prediction, computes. The probability distribution it produces covers tokens, and the sampling step in Part 5 draws numbers from that distribution. Even the interpretability graphs in Part 7 begin at input nodes that are tokens, because tokens are the only thing the model was ever given.
What this comes to
The common belief says an AI model reads a question the way a person does, word by word, taking in meaning as it goes. The corrected picture starts earlier. Text is cut into pieces, each piece is swapped for a catalog number, and the model works only with those numbers. Each number is then turned into a long list of values that describes the piece. Several odd behaviors trace back to where the cuts fall, including stumbles on letter counts, errors in long sums, and the way usage gets billed. The single most useful thing to remember, in my reading, is that the model sees the cuts and has no view of the letters inside them, so the way text gets divided shapes what comes easily and what comes hard.
Sources: Sennrich, Haddow, Birch, "Neural Machine Translation of Rare Words with Subword Units" (2015). Singh and Strouse, "Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs" (2024). Brown et al., "Language Models are Few-Shot Learners" (GPT-3, 2020). Mikolov et al., "Efficient Estimation of Word Representations in Vector Space" (2013). Anthropic, "Tracing the thoughts of a large language model" (2025). The BPE run is my own small re-implementation of the method in Sennrich et al., and the one-hot check is my own code. Both are reproduced above, and neither is evidence about a production model.