12.7.2 — Tokenisation

Three things happened to Cartwise's support assistant in its first month, and the team filed them as three unrelated bugs.

1. A customer asked it to confirm that the coupon code SAVE10X had "exactly two X's". It confidently said yes.
2. The finance team noticed that conversations in Hindi cost about twice as much per message as conversations in English, for answers of similar length.
3. Asked to add up three item prices, it occasionally got a seven-digit total slightly wrong, while getting simpler sums right.

They are one bug, and it lives in the very first step of 12.7.1: the model never sees characters. It sees tokens, chunks produced by a tokenizer, and the way text is cut into chunks decides what the model can perceive, what you pay, and several of its most famous mistakes. This page builds a tokenizer by hand, then traces each of the three "bugs" back to it.

1. Why not letters, and why not words?

A model needs a fixed list of symbols it understands, its vocabulary, and every piece of text must be expressible in that list. There are two obvious choices, and both are bad.

Characters. A vocabulary of a few hundred symbols covers every word, typo and name in English, so nothing is ever unknown. But sequences become very long: "internationalisation" is 20 symbols. Attention compares every position with every other, so its cost grows with the square of the sequence length (Chapter 12.7.3 shows why). Four times as many symbols means sixteen times as much attention work. And the model wastes much of its capacity learning to spell before it can learn meaning.

Whole words. Sequences stay short. But the vocabulary must have a limit, and every word outside it becomes a single "unknown" symbol: every typo, every new product name, every brand, every word in a language the list did not cover. The model also cannot see that "charge", "charger" and "charging" share a root.

The answer every modern model uses sits in between: subword tokenisation. Common words are kept whole, and rarer words are built from smaller, reusable pieces. "charger" might be one token, "rechargeable" might be "re", "charge", "able", and a never-seen product code is spelled out in short fragments. Nothing is ever unknown, and common text stays short.

2. Byte pair encoding, built by hand

The most widely used subword method is byte pair encoding, or BPE. It began as a data compression algorithm, described by Philip Gage in 1994, and was adapted for language models by Rico Sennrich, Barry Haddow and Alexandra Birch in 2016. The training algorithm is genuinely simple: start from single characters, and repeatedly merge the most frequent adjacent pair into a new token.

Here is the example from the Sennrich paper, a tiny corpus of four words with how often each appears. An underscore _ marks the end of a word, so the tokenizer can tell "est" at the end of a word from "est" in the middle.

Word	Count
l o w _	5
l o w e r _	2
n e w e s t _	6
w i d e s t _	3

Merge 1. Count every adjacent pair, weighted by word count. "e" followed by "s" appears in "newest" (6) and "widest" (3): 9 times. Several other pairs, such as "s t", also reach 9; ties are broken by a fixed rule, and here "e s" is taken first. Merge it into a new token "es". The words become n e w es t _ and w i d es t _.

Merge 2. Now "es" followed by "t" appears 9 times. Merge it into "est".

Merge 3. "est" followed by "" appears 9 times. Merge into "est": the ending "-est" is now one token.

Merges 4 and 5. "l o" appears 7 times (5 in "low" plus 2 in "lower"), then "lo w" 7 times. Merging both gives the token "low".

Merges 6 and 7. "n e" then "ne w", 6 times each, give "new".

Merge 8. "new est_" appears 6 times, so "newest_" becomes a single token.

After eight merges, the vocabulary is the original letters plus es, est, est_, lo, low, ne, new, newest_. Frequent sequences have become single tokens, while rare ones, like "widest", are still built from smaller pieces: w, i, d, est_. A real tokenizer runs the same procedure over billions of words for 30,000 to 200,000 merges. The ordered list of merges is the tokenizer.

To tokenize new text, apply the same merges in the same order. Take a word the corpus never contained, "lowest". Start with l o w e s t _. Merge 1 joins "e s", merge 2 joins "es t", merge 3 joins "est _", merges 4 and 5 join "l o" and "lo w". The result is two tokens: low and est_. The tokenizer had never seen "lowest", yet it cut it at exactly the sensible place, into a root and an ending it learned from other words. That is the strength of subwords.

Bytes, so that nothing is ever unknown

One refinement matters a lot in practice. If you start from characters, a character that never appeared in training, an unusual emoji or a rare script, still has no token. Byte-level BPE, used by GPT-2 in 2019 and by most models since, starts instead from the 256 possible values of a byte, the unit of Chapter 1.4's UTF-8 encoding. Every possible text is a sequence of bytes, so every possible text can be tokenized, even emoji, corrupted data or a language the tokenizer never saw. In the worst case it is spelled out one byte at a time.

The other algorithms you will see named

• WordPiece, used by Google's BERT, works like BPE but chooses each merge by how much it improves the probability of the training text under a simple model, rather than by raw count.
• Unigram, by Taku Kudo, works in the opposite direction: start with a very large vocabulary and repeatedly remove the tokens whose loss hurts the least.
• SentencePiece, also by Kudo, is a library that implements BPE and Unigram on raw text, treating the space as an ordinary symbol (written ▁). That matters for languages such as Japanese, Chinese and Thai, which do not separate words with spaces.

3. Seeing it for real

tiktoken is the open-source tokenizer library that OpenAI publishes for its models. Its cl100k_base encoding has about 100,000 tokens.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def pieces(text):
    return [enc.decode([t]) for t in enc.encode(text)]          # (1)

print(pieces("strawberry"))      # (2) e.g. ['str', 'aw', 'berry']
print(pieces("SAVE10X"))         # e.g. ['SAVE', '10', 'X']
print(pieces("1234567"))         # (3) e.g. ['123', '456', '7']
print(pieces(" hello"), pieces("hello"))   # (4) different tokens: ' hello' vs 'hello'

Line (1) encodes text into token IDs and then decodes each ID on its own, so you can see the pieces. The pieces shown in the comments are what this family of tokenizers typically produces; other models' tokenizers cut differently, which is the point of section 5. Lines (2) to (4) are the examples the next section explains.

4. The three "bugs", explained

Counting letters: the model cannot see them

Ask how many letters "r" are in "strawberry", and a model may say two. Line (2) shows why: the model receives three tokens, something like str, aw, berry, as three numbers. It has never been shown that token number 496 (or whatever ID "str" has) contains an "r". It can only infer the spelling of each token from patterns in its training text, such as dictionaries and spelling exercises that happen to spell words out letter by letter. Counting the letters in a token is like asking someone to count the brushstrokes in a word they have only ever seen printed whole.

The coupon code is the same problem. SAVE10X might be SAVE, 10, X, and "how many X's" requires looking inside tokens. Tasks that depend on individual characters, such as counting letters, reversing strings, checking anagrams or validating the exact format of a code, are structurally hard for an LLM. Newer models handle the common cases better through training, but the limitation underneath remains. For anything that needs exact characters, use ordinary code, a one-line Python check, and give the model that code as a tool, as Chapter 12.8.4 shows.

Arithmetic: numbers are cut in arbitrary places

Line (3) shows a seven-digit number split into 123, 456, 7. Those chunks have no arithmetic meaning: the "456" is not "four hundred and fifty-six" in this number, it is "456 thousand". And a slightly different number, 2,345,678, is cut into different chunks, so the same digit in the same place value can arrive inside different tokens each time. Adding numbers requires lining up place values, and the tokenization actively scrambles them. That is a large part of why models make occasional mistakes on long arithmetic while managing short sums. Some model families now split every digit into its own token to help. The engineering rule is the same as for characters: never trust an LLM with arithmetic that matters; a cart total is computed by code, which was the "when not to use ML" rule of 12.1.3.

Hindi costs more: the tokenizer was trained mostly on English

A tokenizer's merges come from its training text, and the training text of most tokenizers is dominated by English. Common English words became single tokens long ago in the merge list. Words in Hindi, Bengali, Tamil and many other languages, especially in their own scripts, were rarely frequent enough to earn merges, so they are cut into many more, smaller pieces, sometimes byte by byte. A 2023 study by Aleksandar Petrov and colleagues measured this across many languages and found that the same sentence can take several times as many tokens in some languages as in English, and over ten times in the worst cases.

Since providers charge per token, and the context window is counted in tokens, this is a direct cost multiplier and a direct reduction in how much text fits. Newer tokenizers with larger vocabularies have narrowed the gap considerably, but for Cartwise the practical steps are to measure token counts per language on real messages with the tokenizer of the model actually in use, budget for the difference, and compare models partly on this.

5. Other things tokenisation decides

Spaces are part of tokens. Line (4) shows that " hello", with a leading space, and "hello" are different tokens. Most tokenizers attach the space to the start of the following word. So a prompt that ends with a stray trailing space leaves the model in an unusual position: it must produce a token that comes after a bare space, a pattern it rarely saw in training, and output quality can noticeably suffer. Do not end prompts with a space.

Code and formatting cost tokens. Indentation, long variable names and JSON punctuation all become tokens. Deeply indented code or verbose JSON can cost noticeably more than the same content written compactly, which matters when you put large structured data into prompts.

Every model counts differently. The same text is a different number of tokens for different models, sometimes very different. When you budget a context window or estimate costs, count with the tokenizer of the model you will actually call. The "about four characters per token" rule is an English-only rough guide.

The vocabulary size is a trade-off. A bigger vocabulary means shorter sequences (cheaper attention, more text per context window) but a bigger embedding table and output layer, and more rare tokens that are seen too seldom in training to be learned well. Vocabularies have grown over time, from about 32,000 tokens in many 2023 open models to 100,000 to 250,000 in many current ones, largely to serve more languages efficiently.

Tokens can be weirdly under-trained. Because the tokenizer is built before the model is trained, and from slightly different text, a few tokens end up almost never appearing in the model's actual training data. Prompts that contain them can produce strange behaviour. This is rare in practice, but it is one more reason that the tokenizer and the model are a matched pair that must never be mixed.

6. What this means for an AI engineer

• Budget in tokens, per language, measured on real data. Costs and context limits are in tokens, not words.
• Route exact work to code. Character checks, coupon validation, arithmetic and date calculations go to tools; the model decides when to call them and explains the results.
• Keep prompts tidy. No trailing spaces, compact structured data, consistent formatting.
• Never mix a tokenizer and a model. A model only understands the token numbers its own tokenizer produces; token 496 means something completely different to another model.

Recall

• Characters make sequences too long; whole words leave unknown words. Subword tokenisation keeps common words whole and builds rare ones from reusable pieces.
• Byte pair encoding repeatedly merges the most frequent adjacent pair; the ordered merge list is the tokenizer. In the worked example, eight merges produced "low", "est_" and "newest_", and the unseen word "lowest" was cut sensibly into "low" + "est_".
• Byte-level BPE starts from the 256 byte values, so any text can be tokenized. WordPiece, Unigram and SentencePiece are the other names you will meet.
• The model never sees characters, so letter counting and code-format checks are hard; numbers are split into arbitrary chunks, so long arithmetic is unreliable; English-heavy tokenizers make other languages cost several times more tokens.
• Leading spaces are part of tokens, so avoid trailing spaces; formatting costs tokens; count with the model's own tokenizer; route exact character and arithmetic work to code.

Self-test: Continue the BPE example by one more merge: which pair is most frequent after merge 8? Why might "2,345,678" and "1,234,567" be tokenized into chunks that line up differently? A Bengali support message costs 2.5 times its English translation; what single measurement would you take before choosing a model?

Next: 12.7.3 Attention Worked by Hand — once text is a row of token vectors, attention is how each token gathers information from the others; the next page computes it with real matrices, step by step.