12.6.1 — What an Embedding Is

A Cartwise customer types "sneakers for jogging" into the search bar. The best product in the catalogue is listed as "Men's Lightweight Running Shoes". The two phrases share no words at all. Cartwise's original search, which matched the words typed against the words in product titles, returned nothing useful, and the customer left.

To a person, the match is obvious: sneakers are shoes, and jogging is running. To a computer comparing strings of letters, "sneakers" and "shoes" are as different as "sneakers" and "invoice". The problem is that the computer is comparing spellings, and what we want it to compare is meanings. That needs a way to turn meaning into numbers, so that things that mean similar things get similar numbers. That is what an embedding is, and this chapter is about the product-search thread of Cartwise's four problems.

1. The problem with treating words as labels

The simplest way to give a computer a word is to number every word in the vocabulary and represent each one as a long list of zeros with a single 1 in its own position. With a vocabulary of five words, [shoes, sneakers, running, jogging, invoice], "shoes" is (1, 0, 0, 0, 0) and "sneakers" is (0, 1, 0, 0, 0). This is the one-hot encoding from 12.4.2.

Now measure how similar two words are with the dot product from 12.2.1. "Shoes" against "sneakers": 1 \times 0 + 0 \times 1 + 0 + 0 + 0 = 0. "Shoes" against "invoice": also 0. Every pair of different words has similarity exactly zero. In geometric terms, every one-hot vector sits at a right angle to every other one. The representation carries no information about meaning whatsoever, and it is enormous: a real vocabulary of 100,000 words means vectors of 100,000 numbers, almost all of them zero. A vector like that, mostly zeros, is called sparse.

What we want instead is a short list of numbers for each word, say a few hundred, where every number carries some information and where similar words end up with similar lists. A list like that, with every entry in use, is called dense.

2. An embedding: meaning as a position in space

An embedding is a dense vector, learned by a model, that represents an item so that items with similar meanings get vectors pointing in similar directions. The word comes from mathematics, where "embedding" one space in another means placing every point of the first somewhere in the second while keeping their relationships. Here, words (or sentences, products, photos) are placed as points in a space of a few hundred dimensions, so that their relationships survive as distances and angles.

Left: one-hot vectors, where every word is exactly as unrelated to every other word. Right: embeddings, drawn here in two dimensions for clarity (real ones use hundreds). Words used in similar ways land near each other, and a query lands in the neighbourhood of the products that match its meaning.

In the right panel, "shoes" and "sneakers" sit close together, near "running" and "jogging", while "invoice" and "refund" sit in a different region entirely. Search now becomes geometry: turn the query into a vector too, find which product vectors sit nearest to it, and return those. "Sneakers for jogging" lands next to "running shoes" because their meanings are close, whatever their spellings. That is semantic search, search by meaning, and it is the subject of the rest of this chapter.

Embedding or vector: what is the difference?

People use the two words loosely, so it is worth being exact. A vector is just the mathematical object: an ordered list of numbers, like (0.21, -1.4, 0.07). It can mean anything, or nothing; three random numbers are a vector. An embedding is a vector with a job: it was learned by a model to represent something, so that its position carries meaning. Every embedding is a vector, but not every vector is an embedding. The customer vector (4, 3) from 12.2.1 was a vector of hand-chosen counts; a 384-number vector produced by a trained model so that "sneakers" lands near "shoes" is an embedding. When engineers talk about a "vector database", they mean a database built to store and search embeddings.

An embedding layer is a lookup table

Inside a neural network, the simplest embedding is an embedding layer: a table with one row per word in the vocabulary, each row a learned vector. Looking up a word's row is exactly the same as multiplying its one-hot vector by the table, because the single 1 selects one row and the zeros erase the rest.

import numpy as np
import torch, torch.nn as nn

E = np.array([[ 0.9, 0.1],      # row 0: shoes
              [ 0.8, 0.2],      # row 1: sneakers
              [ 0.7, 0.6],      # row 2: running
              [-0.5, 0.9]])     # row 3: invoice
one_hot_running = np.array([0, 0, 1, 0])
print(one_hot_running @ E)      # (1) → [0.7 0.6]   the "running" row

table = nn.Embedding(num_embeddings=4, embedding_dim=2)   # (2) a learnable table
print(table(torch.tensor([2])).shape)                     # → torch.Size([1, 2])

Line (1) multiplies a one-hot vector by the table and gets back exactly the "running" row, which shows that an embedding lookup is a matrix multiplication with a one-hot input, done the cheap way. Line (2) creates PyTorch's version, with random starting values that training will adjust. Every LLM begins with such a table: one learned vector for each of the roughly 100,000 tokens it knows, as 12.7.2 explains.

3. Where the meanings come from: word2vec

Someone has to decide where each word goes. Nobody hand-places 100,000 words in a 300-dimensional space. The positions are learned, from one old idea in linguistics, usually quoted from the British linguist J. R. Firth in 1957: "You shall know a word by the company it keeps." Words that appear in similar surroundings tend to mean similar things. "Jogging" and "running" both appear near "shoes", "morning", "miles" and "track". "Invoice" appears near "payment", "due" and "GST". So if a model learns to predict a word's neighbours, it will be forced to give words with similar neighbours similar vectors.

In 2013, Tomas Mikolov and colleagues at Google turned this into a fast, simple training method called word2vec. Its most common form, called skip-gram, works like this.

How word2vec makes training data from plain text. A window slides along every sentence; each word in the window around the centre word forms a positive pair with it. Billions of such pairs come free from ordinary text.

Step 1: make pairs from text. Slide a window along every sentence. For each centre word, pair it with each word within, say, two positions of it. "Comfortable running shoes for morning jogging" gives pairs like (shoes, running) and (shoes, morning). These are positive pairs: words that really did appear together. This is the self-supervised learning of 12.1.2: the text labels itself, so billions of training examples come free.

Step 2: make some fake pairs. For each positive pair, also pair the centre word with a few words picked at random from the vocabulary, like (shoes, invoice). These are negative pairs, almost certainly words that did not appear together. Using a handful of random negatives instead of comparing against the whole vocabulary is called negative sampling, and it is what made word2vec fast.

Step 3: train a tiny classifier. Each word gets a vector. The model scores a pair by the dot product of the two words' vectors, turns that into a probability with a sigmoid from 12.2.6, and is trained with cross-entropy to give positive pairs high probability and negative pairs low probability. (Technically each word has two vectors, one for when it is the centre word and one for when it is a neighbour, which avoids a word trivially matching itself; the centre vectors are the ones kept as the embeddings.)

One training step, by hand

Here is one update with two-number vectors, to show the mechanism. The centre word "running" has vector \mathbf{v} = (0.2, 0.1). Its true neighbour "jogging" has vector \mathbf{u}_+ = (0.1, 0.3), and the random negative "invoice" has \mathbf{u}_- = (0.3, -0.1).

• Positive score: \mathbf{v} \cdot \mathbf{u}_+ = 0.02 + 0.03 = 0.05, so \sigma(0.05) \approx 0.512. The model is barely more than half sure these belong together.
• Negative score: \mathbf{v} \cdot \mathbf{u}_- = 0.06 - 0.01 = 0.05, also 0.512. It cannot tell the true neighbour from the random word.

The gradient of the loss with respect to \mathbf{v}, by the p - y rule from 12.2.6, is (0.512 - 1)\,\mathbf{u}_+ + (0.512 - 0)\,\mathbf{u}_- = -0.488 \times (0.1, 0.3) + 0.512 \times (0.3, -0.1) \approx (0.105, -0.198). Taking a step against it with a learning rate of 0.5 moves "running" to about (0.148, 0.199).

Check the effect. The positive score becomes 0.148 \times 0.1 + 0.199 \times 0.3 \approx 0.074, up from 0.05. The negative score becomes 0.148 \times 0.3 - 0.199 \times 0.1 \approx 0.024, down from 0.05. One step pulled "running" towards "jogging" and pushed it away from "invoice". (The neighbour vectors get updated too, in the same way.) Repeat this billions of times across real text, and every word ends up near the words it keeps company with, because those are the words it was pulled towards over and over.

GloVe: the same meaning, from counting

A year later, Jeffrey Pennington, Richard Socher and Christopher Manning at Stanford published GloVe, short for global vectors, which reaches similar embeddings from a different direction. Instead of sliding a window and learning one pair at a time, it first counts, over the whole corpus, how often every word appears near every other word, in one huge table. Then it learns vectors whose dot products reproduce the logarithms of those counts.

The GloVe paper illustrates why counting carries meaning with a famous example about the words "ice" and "steam". Look at how much more likely various words are to appear near "ice" than near "steam":

Nearby word	P(word near ice) ÷ P(word near steam)
solid	about 8.9
gas	about 0.085
water	about 1.4
fashion	about 1.0

"Solid" is far more associated with ice, "gas" far more with steam, and "water" and "fashion" are about equally associated (or unassociated) with both. The ratios pick out exactly the dimension of meaning that separates ice from steam, the solid-gas dimension, and cancel out everything irrelevant. GloVe is designed so that differences between word vectors capture such ratios. In practice, word2vec and GloVe produce embeddings of similar quality, and both were the standard way to represent words in the mid-2010s.

The famous arithmetic, and its honest limits

Embeddings from these methods showed a striking property: some relationships appear as consistent directions. The vector from "man" to "woman" is roughly parallel to the vector from "king" to "queen", so

\text{vector}(\text{king}) - \text{vector}(\text{man}) + \text{vector}(\text{woman}) \approx \text{vector}(\text{queen})

and similarly for capital cities and countries, or verb tenses. This is real and was one of the results that made the field take embeddings seriously. It is also weaker than it is usually presented: the tests exclude the input words from the possible answers (otherwise the nearest vector to the result is often "king" itself), and many relationships do not line up so neatly. Treat it as a beautiful illustration that embeddings encode structure, not as something to build a product on.

4. From one vector per word to one vector per meaning

word2vec and GloVe have a basic limitation: each word gets one vector forever. "Bank" has a single vector, somewhere between its "riverbank" meaning and its "savings bank" meaning, so it is correct for neither. "Apple" the fruit and Apple the company share one point. These are called static embeddings.

Modern models produce contextual embeddings instead: they read the whole sentence first and produce a vector for each occurrence of a word, coloured by its surroundings. "River bank" and "bank account" get different vectors for "bank", because the attention mechanism from 12.5.6 has already mixed information from "river" or "account" into it. This began with models like ELMo in 2018 and became standard with BERT, a transformer released by Google later that year, and it was one of the largest single improvements in how computers represent text. Every LLM produces contextual embeddings internally at every layer. Static word vectors are now mostly of historical and teaching interest, and the mechanism that produces contextual ones is the subject of Chapter 12.7.

5. Anything can be embedded

The idea is not limited to words. An embedding is any learned vector that places similar things near each other, and almost every trained network produces one along the way.

• Photos. The 512 numbers the ResNet in 12.5.5 produces just before its classifier are an embedding of the photo. Two photos of the same kettle, taken by different sellers, land close together. Cartwise uses exactly this to catch duplicate listings: embed every product photo, and flag pairs whose embeddings are nearly identical.
• Sentences and documents. One vector for a whole product description, review or support email, which is what search actually needs, built in 12.6.2.
• Products and customers. Recommendation systems learn a vector for every product and every customer, so that a customer's vector lands near the products they will like, as Chapter 12.12.6 shows.
• Text and images together. Models trained on photos with their captions put a photo and a description of it at the same point, so a text query can find an image. 12.6.2 introduces them.

A warning that comes with learned meaning

Embeddings learn whatever associations are in their training text, including the ones we do not want. Studies of word2vec, most famously a 2016 paper by Tolga Bolukbasi and colleagues titled "Man is to Computer Programmer as Woman is to Homemaker?", showed that occupation words carried gender associations from the news text the vectors were trained on. Any search or ranking system built on embeddings can inherit such patterns, for example by ranking some products or job listings differently for different groups. It is not a reason to avoid embeddings; it is a reason to measure outcomes by group, as Chapter 12.14.3 explains.

Recall

• One-hot encoding gives every pair of different words similarity zero and wastes huge sparse vectors. An embedding is a dense, learned vector that places similar meanings in similar directions, which makes semantic search possible.
• A vector is any list of numbers; an embedding is a vector learned to represent something. An embedding layer is a lookup table, equivalent to one-hot times a matrix.
• word2vec learns from "you shall know a word by the company it keeps": positive pairs from a sliding window, random negative pairs, and a dot-product classifier that pulls neighbours together and pushes random words apart. GloVe reaches similar vectors by fitting global co-occurrence counts.
• Word arithmetic (king − man + woman ≈ queen) is real but weaker than advertised. Static embeddings give one vector per word; contextual embeddings give one per occurrence and power modern models.
• Photos, sentences, products and customers can all be embedded; Cartwise finds duplicate listings with photo embeddings. Embeddings inherit biases from their data, so outcomes must be measured.

Self-test: Why is the dot product of any two different one-hot vectors zero? In the word2vec step, what would happen to the update if "invoice" had been a true neighbour too? Why does a static embedding serve the word "bank" badly?

Next: 12.6.2 Sentence Embeddings and Similarity — search needs one vector for a whole query and a whole product description; the next page shows how models are trained to produce good ones, how to measure closeness between them, and how to choose a model.