12.7.1 — The Journey of a Prompt

A Cartwise customer opens the support chat and types: "Where is my order 4471?" About a second later, words start to appear: "Your order 4471 left our Pune warehouse yesterday and…", one or two at a time, until the answer is complete.

Between the customer pressing Enter and the last word appearing, a long chain of steps runs, and every one of them is something you will tune, pay for or debug as an AI engineer. The latency the customer feels, the cost on Cartwise's bill, why the assistant "forgets" things, why it sometimes invents an order status, and why the same question can get two different answers: all of these live somewhere on this path. This page walks the whole path once, end to end, so that every later page in this chapter has a place to fit.

The core fact to hold onto is simple: an LLM never reads or writes text. It reads a list of numbers and predicts the next number, over and over. Everything else is the machinery around that loop.

The full path from a customer's question to a streamed answer. Steps ② to ⑥ produce exactly one token; step ⑦ loops back to produce the next. The customer sees words appearing because the answer is literally being built one token at a time.

① The application builds the prompt

The customer typed five words, but the model receives far more. Before anything reaches the model, Cartwise's code assembles a complete prompt: a block of system instructions ("You are Cartwise's support assistant. Be concise. Never promise a refund…"), the conversation so far, any retrieved information (here, the status record of order 4471 looked up from the order database), and finally the new question. Deciding what goes into that prompt, and what to leave out, is called context engineering, and Chapter 12.8.2 is about doing it well.

This step explains a fact that surprises many people: the model has no memory between requests. Each call is processed from scratch. The feeling that the assistant "remembers" what you said three messages ago exists only because the application sends the whole conversation again, every time. That is why long conversations get slower and more expensive with each turn, and why there is a hard limit, the context window, on how much can be sent at once. Chapter 12.10.5 deals with what to do when the conversation outgrows it.

② Tokenize: text becomes numbers

The model's vocabulary is not words. It is a fixed list of tokens, typically 32,000 to 250,000 of them: whole common words, pieces of rarer words, punctuation, digits and spaces. A tokenizer splits the prompt into these pieces and replaces each with its number in the list. "Where is my order 4471?" might become seven or eight tokens, with "4471" perhaps split into "44" and "71". A useful rule of thumb for English is about four characters, or three-quarters of a word, per token.

Tokens are the unit of everything on the bill: providers charge per input token and per output token, and the context window is measured in tokens. Tokenisation also explains some famous failures, such as models miscounting the letters in a word, because the model never sees letters at all. Chapter 12.7.2 builds a tokenizer from scratch and shows exactly where these quirks come from.

③ Embed: numbers become vectors

Each token number is used to look up a row in the model's embedding table, the lookup table from 12.6.1. For a mid-sized open model such as Llama 2 7B, each row has 4,096 numbers, so our eight tokens become a grid of 8 rows by 4,096 columns. That grid, one vector per token, is what flows through the rest of the model.

Because attention, the core operation in the next step, does not by itself know the order of the tokens, information about each token's position is mixed in at this stage or inside the attention itself. Without it, "order 4471 shipped" and "4471 order shipped" would look identical to the model. Chapter 12.7.4 explains how.

④ The transformer stack: every token's vector is refined

Now the grid passes through a stack of identical blocks: 32 of them in Llama 2 7B, around 100 or more in the largest models. Each block does two things to every token's vector.

Attention lets tokens share information. Each token looks at the tokens before it and pulls in what it needs. The vector for "4471" absorbs the fact that it follows "my order", so it now represents "an order number this customer is asking about". The vector for "?" absorbs the whole question. This is the attention of 12.5.6, and Chapter 12.7.3 works it through with real matrices.

A feed-forward layer then processes each token on its own, the same small two-layer network from 12.5.1 applied to every position separately. A useful way to remember the split: attention decides what information to gather from other tokens, and the feed-forward layer works with what was gathered. Most of the model's parameters, and much of what it "knows" about the world, live in these feed-forward layers.

Both parts are wrapped in residual connections and layer normalisation from 12.5.4, so each block adds a refinement to every vector rather than replacing it. After 32 blocks, the vector at each position is a rich description of that token in its full context.

⑤ Unembed: the last vector becomes a score for every possible next token

To predict what comes next, the model takes only the vector at the last position, the one that has absorbed the whole prompt, and multiplies it by an output matrix with one row per vocabulary entry. The result is one raw score, a logit, for every token the model knows: 32,000 numbers for Llama 2, around 100,000 or more for many current models. A high logit for "Your" means the model considers "Your" a likely next token. This is exactly the softmax regression of 12.3.1, with the vocabulary as its classes.

⑥ Softmax and sampling: choose one token

Softmax from 12.2.6 turns the logits into probabilities that add up to 1. Then one token is chosen. Always taking the most probable token is one option; choosing randomly in proportion to the probabilities, reshaped by settings like temperature and top-p, is another. This choice is why the same question can produce different answers on different runs, and it is a setting you control on every API call. Chapter 12.7.7 covers every method and when to use each.

Notice what this step means for factual questions. The model chose "left our Pune warehouse yesterday" because those tokens were likely given the prompt, not because it looked anything up. If the real tracking record was in the prompt, likely and true coincide. If it was not, the model still produces fluent, likely-sounding tokens, and that is what an invented order status is. That is the whole reason for retrieval-augmented generation in Chapter 12.9: put the facts in the prompt so that the likely answer is the true one.

⑦ Append and repeat

The chosen token is added to the end of the sequence, and the model runs again to predict the token after it. Then again, and again. A 60-token answer means 60 trips around the loop.

That would be wasteful if every trip recomputed everything for the whole sequence. It does not. On the first trip, the model processes all the prompt's tokens at once, in parallel, which is called the prefill. On each later trip, it processes only the one new token, reusing intermediate results it saved for all the earlier ones in a structure called the KV cache. This later phase is called decode. The two phases have very different speeds and costs, which is why API pricing and latency have the shape they do, and Chapter 12.7.6 explains both.

⑧ and ⑨ Detokenize, stream, and stop

Each new token number is turned back into its piece of text and, in a chat interface, sent to the customer immediately. That is streaming, and it is why words appear progressively rather than all at once after a long pause. Chapter 12.8.5 shows how to build it.

Generation stops when the model produces a special end-of-sequence token, which it learned to emit at the natural end of an answer, or when it reaches a maximum length set by the application.

What the journey explains

With the whole path in view, many everyday facts about LLMs stop being mysterious:

• Why output tokens cost more and take longer than input tokens. Input tokens are processed together in one parallel prefill; output tokens are produced one at a time, each needing a full trip through the model.
• Why the first word takes a moment and later words flow quickly. The pause is the prefill over the whole prompt; after that, each token is one fast decode step. This first pause is called time to first token.
• Why long conversations and long documents are expensive. Every request resends and reprocesses everything, and attention's cost grows with the length of the sequence.
• Why the assistant can sound sure and be wrong. Step ⑥ picks likely tokens, not verified facts.
• Why the same prompt gives different answers. Sampling in step ⑥ involves randomness unless you turn it off.

Here is the path in a few lines of Python, using the small open model GPT-2 so it runs on any laptop. transformers is the open-source library from the company Hugging Face that provides most open models and their tokenizers behind one interface.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

ids = tok("Your order has been", return_tensors="pt").input_ids    # (1) ② tokenize
print(ids.shape)                                                    # → torch.Size([1, 4])

with torch.no_grad():
    logits = model(ids).logits                                      # (2) ③–⑤ the whole stack
print(logits.shape)                                                 # → torch.Size([1, 4, 50257])

probs = torch.softmax(logits[0, -1], dim=-1)                        # (3) ⑥ last position only
top = torch.topk(probs, 5)
print([tok.decode(int(i)) for i in top.indices])                    # (4) the five likeliest next tokens

Line (1) tokenizes four words into four token IDs. Line (2) runs the embedding table, all 12 transformer blocks of GPT-2, and the unembedding in one call; the output has a logit for each of GPT-2's 50,257 vocabulary entries at every position. Line (3) keeps only the last position, as in step ⑤, and applies softmax. Line (4) decodes the five most probable next tokens, which for this prompt will be continuations like " shipped" or " delivered" (the exact list depends on the model). Choosing one of them and running the model again with it appended is step ⑦.

What the interviewer will push on

"Explain how an LLM generates text." They are checking that you can walk the path rather than recite "it uses attention": tokenize, embed, a stack of attention and feed-forward blocks, unembed the last position into logits, softmax, choose a token, append, repeat. The tell of a strong answer is mentioning prefill versus decode and the KV cache.

"Does the model remember earlier messages?" They want: no, each request is stateless, and the application resends the conversation. Bonus for connecting this to cost growth and the context window.

"Why do LLMs hallucinate?" A good answer points at step ⑥: the model outputs likely tokens, and likely is not verified, so without the facts in the context it produces plausible text. The fix is grounding, not a better prompt alone.

"Where does the latency go?" They are checking you can split it into time to first token (dominated by prefill, so by prompt length) and time per output token (decode), and that output length usually dominates total time.

Recall

• An LLM reads a list of token numbers and predicts the next number, in a loop. The application first builds the full prompt (instructions, history, retrieved data, question); the model itself remembers nothing between requests.
• Tokenize into vocabulary pieces (about 4 English characters each); embed each into a vector; add position information.
• A stack of transformer blocks refines every token's vector: attention gathers information from other tokens, the feed-forward layer processes each token, with residuals and normalisation.
• The last position's vector is unembedded into one logit per vocabulary entry; softmax and sampling (temperature, top-p) choose one token. Likely is not the same as true, which is why grounding matters.
• The token is appended and the loop repeats: one parallel prefill, then one decode step per output token using the KV cache. Text streams out until an end token or a length limit.

Self-test: Why does doubling the length of the answer roughly double the generation time, while doubling the prompt length mostly increases the time before the first word? Where on the path would you look first if the assistant states a wrong delivery date? Why can the same prompt produce two different answers?

Next: 12.7.2 Tokenisation — the very first step decides what the model can and cannot see, what you pay, and several of its strangest mistakes; we build a tokenizer by hand to see why.