The token: the fundamental unit of large language models

The token: the basic unit of an LLM

A piece of text, not a word

An LLM reads neither letters nor words: it reads tokens. A token is a fragment of text, which may be a whole word, a part of a word, a punctuation mark or a number. Each token corresponds to a number in a dictionary fixed before training. Let us take as our prompt the sentence "Explique-moi la photosynthèse simplement."

The symbol _ represents a space, attached to the token that follows it. Dictionary sizes are rounded. A large dictionary tends to shorten texts, but this is not systematic: Gemma 3 (≈ 262,000 tokens) produces 10 tokens for this sentence, against 9 for Tekken (≈ 131,000). The split also depends on the languages and the corpus used to build the dictionary.

What the comparison reveals

The eight tokenizers in the table read the same sentence, but do not split it in the same way. These differences show what a token really is.

• "photosynthèse" has no entry of its own in any dictionary: this rare word is reconstructed from three or four fragments. Tekken, GPT-4o, DeepSeek and Gemma 3 split it into three (_photos, ynth, èse), whereas GPT-4, Llama 3 and Qwen3 also separate the ending into è and se.
• "simplement" forms a single token, including the space character, in all tokenizers except that of Mistral 7B. Its smaller dictionary splits it into _simple and ment.
• "Explique" is split into two (Expl, ique) by all the tokenizers, even though it is a common verb: the split follows the statistics of the corpus, not the grammar.
• "-moi" receives three different treatments: a single token in Tekken and GPT-4o, -m and oi in DeepSeek, GPT-4, Llama 3 and Qwen3, or - and moi in Gemma 3. Mistral 7B splits it into three tokens. The final period always forms a separate token.
• The total varies from 9 to 13 tokens. Dictionary size alone does not explain this: Gemma 3, with the largest dictionary in the table, produces 10 tokens against 9 for Tekken.

A token is therefore not a stable linguistic unit: it depends on the dictionary of each model. Only a few points are constant, such as the space attached to the word that follows it, isolated punctuation, or the splitting of "Explique" into Expl and ique.

Why split this way?

The dictionary is built before training by a method such as byte-pair encoding: we start from characters, then repeatedly merge the most frequent pairs in the corpus. Common sequences become whole tokens, rare sequences remain in pieces. It is a trade-off: a large dictionary shortens texts but weighs down the model, a small dictionary lengthens the sequences to be processed.

The same text is therefore not split in the same way from one model to another. Mistral 7B's SentencePiece tokenizer, whose dictionary contains about 32,000 tokens, produces 13 tokens for the same sentence instead of 9. In practice, the context window and the billing of AI services are measured in tokens, and languages less present in the corpus often require more of them to say the same thing.

From token to number

The model receives only the list of numbers: 60,795, 2,508, 33,001, 1,427, 17,159, 20,602, 24,380, 37,507, 1,046. These numbers are simply addresses in the dictionary and have no meaning by themselves. It is at the next step, the embedding, that each is replaced by a vector of thousands of numbers learned during training. The model therefore never "sees" the letters of "photosynthèse", which explains its difficulties in counting the letters of a word.

The embedding: from number to vector

A number has no meaning

After tokenization, the model has a list of numbers. But the number 17,159, which designates the token _photos in our example, is only an address in the dictionary: it is no closer to 17,160 than to 90,000. To compute, the model needs a representation where proximity reflects similarity of use. This is the role of the embedding.

A table of learned vectors

The embedding is an immense table that contains one row per token of the dictionary. Each row is a vector, that is, a list of thousands of numbers. To go from the number to the vector, the model simply reads the corresponding row:

_photos (number 17,159) → [0.021 ; −0.134 ; 0.087 ; … ; 0.052]

These values are fictitious and the real list contains, for example, 4,096 numbers. They are not written by humans: they are coefficients adjusted during training, just like those of attention. For a dictionary of 131,072 tokens and vectors of 4,096 numbers, this table alone represents about 537 million coefficients.

Vectors that draw closer according to use

Over the course of training, tokens used in similar contexts receive neighbouring vectors. The vector of _simplement thus ends up closer to that of a comparable adverb than to that of a proper noun. This space has thousands of dimensions: it cannot be drawn, but distances and angles can be measured in it.

The vector of a token is always the same, whatever its surroundings. A fragment such as ynth has no meaning on its own: it is the next step, attention, that puts it in relation with _photos and èse to reconstruct "photosynthèse".

The order of tokens

Since the same token gives the same vector everywhere, its position in the sentence must also be indicated, because "the cat eats the mouse" does not mean the same thing as "the mouse eats the cat". Depending on the model, this information is added to the vectors or, most often today, applied as a rotation inside the attention layers.

What the model receives

The 9 numbers of our example thus become 9 vectors, that is, a table of 9 rows and 4,096 columns, i.e. 36,864 numbers. It is this matrix, and no longer text, that enters the Transformer blocks.

Only 5 columns out of 4,096 are shown, i.e. 9 × 4,096 = 36,864 numbers in total. These values are illustrative: they are of the order of magnitude of those of a real embedding (small numbers, positive or negative, of the order of a hundredth), without being extracted from a specific model. In a real model, they are not limited to the interval from −1 to 1.