12.5.2 — Tokenizers and Embeddings

Ask a language model how many times the letter r appears in "strawberry" and it may well answer two.

This is not a reasoning failure. The model never saw the letters. It saw two or three tokens — something like str, aw, berry — and those are the atoms it works with. Asking it to count characters is like asking someone to count the brushstrokes in a word they read as a whole.

The same layer explains why the same sentence costs three times more in Hindi than in English, why arithmetic on long numbers is unreliable, and why a trailing space in your prompt can change the output. Tokenization is a preprocessing detail that leaks into everything.

1. Why subwords

Character-level — a vocabulary of about 100 symbols, so no word is ever unknown, and sequences become enormous. "internationalisation" is 20 tokens, and with O(n^2) attention (Chapter 12.5.1) length is expensive.

Word-level — short sequences, and a vocabulary that must be capped, so everything else becomes <UNK>. Every typo, every name, every new product is unknown, and the model cannot see that "run", "runs" and "running" are related.

Subword tokenization is the compromise everyone uses. Common words stay whole; rare words break into pieces. "tokenization" → token + ization. Nothing is ever unknown, and sequences stay short.

2. Byte pair encoding, walked

The dominant algorithm, and it is genuinely simple: start from characters, and repeatedly merge the most frequent adjacent pair.

Take a corpus of low low low lower lowest (counts shown):

Start:  l o w   (3)      l o w e r  (1)      l o w e s t  (1)
Most frequent pair: (l, o) → merge into "lo"
        lo w    (3)      lo w e r   (1)      lo w e s t   (1)
Most frequent pair: (lo, w) → merge into "low"
        low     (3)      low e r    (1)      low e s t    (1)
Next: (e, r) → "er",  then (e, s) → "es",  then (es, t) → "est"
Final vocabulary: l, o, w, e, r, s, t, lo, low, er, es, est

Frequent sequences became single tokens; the rest remain composable. Run this for 50,000 merges over a large corpus and you have a real tokenizer. The merge list is the tokenizer: encoding applies the same merges in the same order.

The variants you will meet:

• BPE — GPT models, and most open models.
• WordPiece — BERT. Merges by what most improves the likelihood of the corpus rather than by raw frequency.
• Unigram — starts with a large vocabulary and removes tokens, keeping those that hurt least. Used by SentencePiece.
• SentencePiece — treats the input as a raw byte stream including spaces (encoding a space as ▁), so it needs no language-specific pre-splitting. Essential for languages that do not separate words with spaces.

Byte-level BPE is the important refinement. Rather than starting from Unicode characters, start from the 256 possible bytes. The vocabulary can then represent absolutely any input — emoji, unusual scripts, corrupted text, binary — with no unknown token possible and no need for a character-level fallback.

3. What tokenization costs you

Languages are not priced equally. Tokenizers are trained mostly on English-heavy corpora, so English text is roughly 1 token per 4 characters, while many other languages — particularly those in non-Latin scripts — take two or three times as many tokens for the same meaning. That is a direct multiplier on cost and a direct divisor on effective context length, and it is a real fairness issue rather than a curiosity. Newer tokenizers with larger vocabularies have narrowed the gap without closing it.

Numbers tokenize badly. 1234567 may split as 123, 45, 67 — groupings with no arithmetic meaning, and inconsistent between numbers. This is a large part of why models are unreliable at multi-digit arithmetic, and why some models now force digits to tokenize individually. Use a calculator tool (Chapter 12.6.3) rather than trusting arithmetic.

Character-level tasks are structurally hard. Counting letters, reversing strings, checking rhymes, spotting anagrams — the model does not see characters. Some models compensate through training; the underlying limitation remains.

Whitespace matters more than it should. In most tokenizers a leading space is part of the token, so " hello" and "hello" are different tokens. A prompt ending in a trailing space puts the model in an unusual position — it must now produce a token that follows a space, which is off the distribution it usually sees — and can noticeably degrade output. Do not end prompts with a space.

Code tokenizes differently. Indentation becomes tokens, so deeply nested code costs more, and tokenizers with dedicated whitespace tokens handle it far better.

Token counts are model-specific. The same text is a different number of tokens per tokenizer. Count with the tokenizer of the model you are calling, not an estimate, when budgeting a context window — the standard rule of thumb of ~0.75 words per token is English-only and approximate.

4. Embeddings: from a symbol to a meaning

A token id is arbitrary — token 4,281 is not "more" than token 91. Feeding an id into a network is meaningless, and one-hot encoding (a vector of 50,000 zeros with a single 1) is enormous and says every word is equally unrelated to every other.

An embedding is a dense vector of a few hundred numbers, learned so that similar meanings sit in similar directions.

"Embedding" and "vector" are often used interchangeably, and the distinction is worth stating: a vector is the data structure (Chapter 12.2); an embedding is a vector that was learned to represent something. Every embedding is a vector; not every vector is an embedding.

How they were originally learned — word2vec (2013) — is the clearest illustration of why they work:

• Skip-gram: given a word, predict its neighbours.
• CBOW: given the neighbours, predict the word.

A single sentence explains the whole result: words appearing in similar contexts get similar vectors. "Cat" and "dog" appear near "pet", "vet", "fur", so they end up close together, without anyone defining what an animal is.

Negative sampling made it tractable: rather than a softmax over the whole vocabulary at every step, sample a handful of random words as negatives and push those apart. GloVe took a different route — factorising a global word co-occurrence matrix — with comparable results.

The famous property: king − man + woman ≈ queen. Directions in the space correspond to relationships, so a "gender" direction and a "capital city" direction genuinely exist. The honest caveat: the effect is real, weaker than the popular version suggests, and heavily dependent on excluding the input words from the answer. Do not build a product on vector arithmetic.

5. Static versus contextual

word2vec gives one vector per word, forever. So "bank" has a single vector averaging the riverbank and the financial sense, which is right in neither context.

Contextual embeddings — from BERT onwards — give a vector per occurrence. "River bank" and "savings bank" produce different vectors, because attention (Chapter 12.5.1) has already mixed in the surrounding words. This is the single largest quality jump in text representation, and it is why static embeddings are now historical.

Sentence embeddings are what a retrieval system actually needs: one vector per chunk of text. Naively averaging a sentence's token vectors works poorly, because the model was not trained to make the average meaningful.

The fix is contrastive training: take pairs known to mean the same thing, and train so that matching pairs are pulled together and random pairs pushed apart. That is what makes a dedicated embedding model much better than mean-pooling a general language model, and it is why you should use one.

6. Using embeddings

Choosing a model. Public benchmarks (MTEB is the common one) are a starting point, and evaluate on your own data — a model that is excellent on general text can be poor on your domain's vocabulary. Check: dimension (cost and quality), maximum input length, multilingual support if needed, and whether it distinguishes queries from documents.

That last point is easy to miss. Several models are trained asymmetrically — a short question and a long passage are different kinds of text — and require a prefix such as query: or passage:. Omitting it silently degrades retrieval, and there is no error.

Distance metric. Cosine similarity (Chapter 12.2) is the default. If the vectors are normalised to unit length, cosine similarity and dot product rank identically, and dot product is cheaper — which is why most vector databases normalise on insert. Euclidean distance also gives the same ranking on normalised vectors.

Dimensions. 384 to 1,536 is typical. Larger is usually slightly better and costs more storage and search time. Matryoshka embeddings are trained so the first k dimensions are a usable embedding on their own, so you can truncate a 1,536-dimension vector to 256 and keep most of the quality — a genuinely useful cost lever.

Chunking is a retrieval decision, not a preprocessing detail. Embedding a whole document averages away everything specific; embedding one sentence loses the context that made it meaningful. Chapter 12.6.2 covers strategies.

Storage. Vectors are large: a million documents at 1,536 dimensions in 32-bit floats is about 6 GB. Quantising to 8-bit integers cuts that four times with a small quality loss, and binary quantisation goes further and is used as a first-pass filter before rescoring. Approximate nearest-neighbour indexes (HNSW, IVF, product quantisation) make search sub-linear, trading a little recall for enormous speed.

Embeddings inherit the biases of their training data, and measurably so: occupation words carry gendered associations, and names associated with particular ethnicities cluster with different sentiment. A retrieval or ranking system built on them inherits that, which is a Chapter 12.10 concern with a concrete engineering consequence — measure outcomes by group rather than assuming neutrality.

7. The practical checklist

• Count tokens with the target model's tokenizer. Do not estimate.
• Do not end a prompt with a trailing space.
• Do not ask a model to count characters or do long arithmetic — give it a tool.
• Budget for non-English text costing two to three times more tokens.
• Use a dedicated embedding model, with its query/passage prefixes if it has them.
• Normalise vectors and use dot product if your store supports it.
• Re-embed everything when you change embedding model. Vectors from different models are not comparable — mixing them produces confidently wrong retrieval with no error message.

Recall

• Models see tokens, not characters, which is why letter counting and long arithmetic fail — those are representation limits, not reasoning failures.
• BPE starts from characters and repeatedly merges the most frequent adjacent pair; the merge list is the tokenizer. Byte-level BPE starts from the 256 bytes, so no input is ever unrepresentable.
• Tokenization is not language-neutral: many non-English texts cost two to three times more tokens, which multiplies price and divides effective context.
• Numbers split into meaningless groups, whitespace is part of a token (never end a prompt with a space), and code costs more through indentation. Token counts differ per model — count with the real tokenizer.
• An embedding is a vector learned to represent something. word2vec's principle: words in similar contexts get similar vectors. king − man + woman ≈ queen is real, weaker than advertised, and not a foundation to build on.
• Contextual embeddings give a vector per occurrence, so "river bank" and "savings bank" differ — the largest quality jump in text representation.
• Sentence embeddings need contrastive training, not mean-pooling. Many models are asymmetric and need query: / passage: prefixes — omitting them degrades retrieval silently.
• Normalised vectors make cosine and dot product rank identically. Quantise to 8-bit for 4× storage savings; Matryoshka embeddings truncate cleanly. Changing embedding model means re-embedding everything — vectors from different models are not comparable.

Self-test: Why can a model miscount letters in a word it spells correctly? · Walk one BPE merge step · Why does the same sentence cost more in some languages? · What is the practical difference between static and contextual embeddings? · When do cosine similarity and dot product give the same ranking? · What silently breaks if you swap embedding models?