The journey from prompt to response: behind the scenes of an LLM

Phase 1: training, where the coefficients are born

N.B.:
Training phase: a corpus of text, cut into tokens, is used to adjust the model's coefficients. The orders of magnitude are indicative.

The sources: an immense corpus of text

Books, Wikipedia, articles, websites, computer code: the training corpus is first cut into tokens. For models whose figures are published, the order of magnitude runs into tens of thousands of billions of tokens: about 15,000 billion for Llama 3, from Meta (2024). The publishers of the most recent commercial models generally do not disclose these values.

Training: adjusting coefficients

The principle is simple. The model is shown the beginning of a text, it predicts the next token, its prediction is compared with the actual continuation, and then the coefficients are corrected very slightly so that the right answer becomes a little more probable. This correction, computed by gradient backpropagation, is repeated billions of times on thousands of GPUs, for weeks or months.

A second, shorter training then steers the model's behavior: answering questions, following instructions, refusing certain requests, adopting a tone. It relies in part on human evaluations.

What remains after training

Only the coefficients remain, also called the model's parameters: from 100 to 1000 billion numbers for today's large models. The texts are not stored in the model like books in a library: they have only shaped these values. A model can reproduce certain very widespread passages (partial memorization), but it cannot "re-read" a book.

Phase 2: inference, from prompt to response

N.B.:
Inference phase: four zones, from your device to the cloud, then back. The token numbers and probabilities are given as examples.

The diagram is divided into four colored zones that indicate where each component is. Zones 1 and 4 (blue) are on your device, zone 2 (orange) is the publisher's AI service, zone 3 (green) is the cloud, that is, a data center.

Zone 1: your device

The browser or mobile application merely collects your text and displays the answer. For large models, no AI computation takes place here, because the coefficients occupy hundreds of gigabytes. Only small compact models can run locally on a recent computer or phone.

Zone 2: the publisher's AI service

Your text travels over the Internet (HTTPS protocol) to the publisher's application servers. A gateway verifies your identity and your quotas, applies security filters, then assembles the complete prompt: the system instructions, the conversation history and your latest question.

This detail is essential: the model keeps no memory of the conversation. It is the service that sends it the whole history with each question, up to the context window, that is, the maximum number of tokens the model can read at once. This service, with its application, its filters and its history, is what is commonly called "the AI." The LLM is only one component of it.

Zone 3: the cloud and its GPUs

The complete prompt is transmitted over the data center's internal network to servers equipped with GPUs or TPUs, chips designed to perform huge matrix multiplications in parallel. The model, which occupies from 200 GB to several terabytes, is distributed across the memory of several chips. This is where the LLM in the strict sense is found: an inference program associated with the model.

In the cloud: seven computation steps

1. Tokenization

The text is cut into tokens using a dictionary fixed before training (the so-called byte-pair encoding method). Each token is given a number. Frequent words form a single token; rare words are split into pieces. The vocabulary has on the order of 100,000 to 200,000 tokens.

2. Embedding

Each token number is replaced by a vector, a list of thousands of numbers (16,384 for the largest Llama 3 model). Position information is added, because word order changes meaning. These vectors are themselves coefficients learned during training.

3. Transformer blocks: attention

The heart of the model is a stack of Transformer blocks, an architecture proposed in 2017. Large models have on the order of 80 to 130 of them. In each block, the attention mechanism compares each token with the preceding tokens and computes their relative importance: in "At the center of the Solar System is the," the words "center" and "Solar" weigh heavily for predicting the continuation. The matrices that decide what to pay attention to (denoted Q, K, V and O) are coefficients.

4. Transformer blocks: the feed-forward network

Each block also contains a feed-forward network, which transforms each vector by two large matrix multiplications separated by an activation function. Interpretability work suggests that these layers serve largely as memory for learned associations, but this interpretation remains a research hypothesis.

In these two steps, the logic is identical at each block, because it is the same code. Only the coefficients differ from one block to another.

5. Scores and probabilities

At the output of the last block, the model assigns a score (logit) to each token in the vocabulary. The softmax function converts these scores into probabilities whose sum equals 100%. In the diagram's example, "Sun" gets 92%, "Moon" 4%, "Jupiter" 3% and all the other tokens together 1%.

6. Sampling

The token is not always the most probable one. A setting called temperature flattens or sharpens the probabilities, and the top-p method limits the draw to the most probable candidates. At very low temperature, the answer is almost deterministic; higher, it varies from one run to the next. This is why the same question does not always produce the same answer.

7. The generation loop

The chosen token, here "Sun," is added to the text, and the whole cycle starts again to compute the next token, until the model produces a special end token. An answer of 300 words therefore requires several hundred successive passes through the model.

Zone 4: the return to your screen

The tokens are not all awaited together: each is sent back as soon as it is computed (streaming), passing again through the AI service and then the Internet. Your device displays them as they arrive, which gives the impression of an answer "written live."

Logic and coefficients: what does a model really contain?

Three things are often confused: the code that performs the computations, the coefficients that determine the result, and the data that were used to set those coefficients. The table below separates them.

Where is each component located?

The limits of the diagram

A deliberately simplified diagram

The diagrams leave out several elements: mixtures of experts, where only part of the coefficients is activated for each token; the cache memory of attention computations, which avoids recomputing everything at each token; the normalizations and residual connections inside the blocks; the external tools (web search, code execution) that the AI service can call; and the processing of images or sounds. Depending on the publisher, tokenization may also be performed in the AI service rather than on the GPU server.

Why an AI can be confidently wrong

Because the model chooses plausible tokens, not verified facts, it can produce a fluent but false sentence, which is called a hallucination. Nothing in the computation distinguishes a learned fact from a plausible sequence of words: only probability counts. Connections to external sources and human verification therefore remain indispensable.

What to remember

An LLM is first of all a program that multiplies matrices, associated with a model that is an immense table of coefficients. These coefficients were fixed by training on enormous text corpora, then frozen. With each question, the text travels from your device to a data center, goes through seven computation steps, produces a token, and then starts again.

This mechanism explains both the power of these systems and their limits: they excel at continuing a text plausibly, and they have no memory of their own of the conversation nor any guarantee of truth. Understanding where each component is located is also understanding where the issues of computation, energy and confidentiality lie.