12.6.2 — Sentence Embeddings and Similarity

Search does not compare single words. It compares a whole query, "sneakers for jogging", with a whole product description, "Men's lightweight running shoes with cushioned soles for daily runs". For that, each piece of text needs one vector. The obvious idea is to take the word embeddings from 12.6.1 and average them.

It fails in ways that matter for a store. Average the words of "battery life is good" and "battery life is not good": the two averages differ only by one word's contribution out of five, so they come out almost identical, and a review search for complaints would return praise. Average "phone case for a laptop" and "laptop case for a phone": identical, because averaging throws away word order completely. An average of word vectors is a blurry summary of which topics are mentioned, not of what the sentence says.

This page covers how models are trained to produce good single vectors for whole texts, how to measure closeness between those vectors correctly, and how to choose an embedding model for a real system.

1. Sentence embedding models

A sentence embedding model takes a piece of text, a query, a product description, a paragraph of a policy document, and returns one vector for the whole thing. Inside, it is a transformer from Chapter 12.7: it reads every word in context, producing a contextual vector for each, and then combines them into one, usually by averaging (mean pooling) or by taking the vector at a special summary position.

That sounds like the averaging that just failed, but there is a crucial difference: the word vectors being averaged are contextual. The "good" in "not good" has already absorbed the "not" through attention before any averaging happens. Still, a general language model, trained only to predict missing words, does not automatically produce single vectors whose distances mean anything useful. Mean-pooling an off-the-shelf language model gives mediocre search. What makes a sentence embedding model good is the way it is trained afterwards, and that is the most important idea on this page.

2. Contrastive learning: pull matches together, push the rest apart

The training goal is simple to state: texts that mean the same thing, or that answer each other, should get vectors close together; unrelated texts should get vectors far apart. Training that way, by contrasting matching pairs with non-matching ones, is called contrastive learning. It is the word2vec idea from 12.6.1, scaled up from word pairs to text pairs.

You need training pairs that you know belong together. For product search, Cartwise has an excellent source: its own search logs. When a customer searched "sneakers for jogging" and then bought the running shoes, that (query, product) pair is a positive example. Public datasets supply others: questions paired with their answers, titles paired with their articles, a sentence paired with its paraphrase.

The cleverest part is where the negative examples come from. Training uses batches, so take a batch of, say, 3 pairs. Each query has one correct product, and the other products in the batch, which were bought after different queries, serve as its negatives for free. These are called in-batch negatives.

Contrastive training on a batch of three. Every query is compared with every product in the batch. The green diagonal holds the true pairs; every red cell is a free negative. Training pushes each row's diagonal value up relative to the others.

Each row of the table becomes a small classification question: of these three products, which one is mine? The model computes the cosine similarity of the query with every product, divides by a small temperature \tau (tau) to sharpen the differences, exactly the temperature of 12.2.6, and applies softmax and cross-entropy with the diagonal entry as the right answer. For a query q_i with true product p_i, the loss is

L_i = -\ln \frac{e^{\text{sim}(q_i, p_i)/\tau}}{\sum_j e^{\text{sim}(q_i, p_j)/\tau}}

Read it as "minus the log of the softmax probability given to the correct product". This loss has a name, InfoNCE, from a 2018 paper by Aäron van den Oord and colleagues. Work it through with the table's numbers and \tau = 0.1:

Query	Probabilities over the 3 products	Probability on the true one	Loss
"sneakers for jogging"	0.650, 0.027, 0.323	0.650	0.430
"1.5 litre kettle"	0.006, 0.968, 0.026	0.968	0.033
"phone charger"	0.340, 0.153, 0.507	0.507	0.679

The kettle query is already easy: its match scores 0.71 and nothing else comes close, so the loss is tiny. The other two are not. "Sneakers for jogging" scores 0.55 against the USB-C charger, nearly as high as its true match, and "phone charger" scores almost as high on the running shoes as on its own charger. Those rows have large losses, so their gradients are large, and training spends its effort exactly where the model is confused, pulling those true pairs together and those false pairs apart. Bigger batches give each query more negatives to be pushed away from, which is one reason contrastive training benefits from very large batches.

Two refinements matter in practice. Hard negatives: random products are usually easy to tell apart (nobody confuses shoes with kettles), so good training data deliberately includes near-misses, such as running shoes for a query asking for hiking boots, which teach the fine distinctions. Training on your own data: a general embedding model has never seen "GST invoice", Cartwise's internal product categories, or Indian English search habits. Fine-tuning it contrastively on a few thousand (query, clicked product) pairs from Cartwise's own logs usually improves search noticeably, and it is one of the highest-value, lowest-cost improvements available.

The same idea joins text and images

In 2021, OpenAI published CLIP (Contrastive Language–Image Pre-training): an image encoder and a text encoder trained together on about 400 million (image, caption) pairs collected from the internet, with exactly this loss. A photo of a red kettle and the text "a red electric kettle" land at nearly the same point. That makes searching images with words possible without any tags: embed every product photo once, embed the customer's text query, and find the nearest photos. It also makes "find products that look like this photo" possible, and Chapter 12.13.1 builds on it.

3. Measuring similarity, correctly

Once every text is a vector, search needs a number for "how close". 12.2.1 introduced the candidates; here is how they relate in practice.

Cosine similarity measures only the angle between two vectors, ignoring their lengths, from −1 to 1. For A = (1, 2, 3) and B = (2, 3, 4): A \cdot B = 2 + 6 + 12 = 20, \|A\| = \sqrt{14} \approx 3.74, \|B\| = \sqrt{29} \approx 5.39, so the cosine is 20 / (3.74 \times 5.39) \approx 0.993. For A and C = (-1, -2, 1): A \cdot C = -1 - 4 + 3 = -2, and with \|C\| = \sqrt{6} \approx 2.45, the cosine is -2 / (3.74 \times 2.45) \approx -0.218.

Dot product is the cosine multiplied by both lengths, so longer vectors score higher.

Euclidean distance is the straight-line distance between the two points.

Here is the fact that ties them together. If every vector has been normalised to length 1, then the squared Euclidean distance between two vectors is

\|\mathbf{a} - \mathbf{b}\|^2 = \|\mathbf{a}\|^2 + \|\mathbf{b}\|^2 - 2\,\mathbf{a}\cdot\mathbf{b} = 2 - 2\cos\theta

which you can check by expanding the left side as in 12.2.1. So with normalised vectors, all three measures put results in exactly the same order: a higher cosine is a higher dot product is a smaller distance. That is why production systems normalise every stored vector once, ahead of time, and then rank with a plain dot product, the cheapest of the three to compute.

There is one exception worth knowing. Some embedding models, especially in recommendation, are trained with an unnormalised dot product, and the length of a product's vector carries information, often how popular or generally appealing it is. Normalising those vectors throws that away. The rule is to use the similarity measure the model was trained with, which its documentation states.

import numpy as np

def cosine(a, b):
    a, b = np.asarray(a, float), np.asarray(b, float)
    na, nb = np.linalg.norm(a), np.linalg.norm(b)
    return 0.0 if na == 0 or nb == 0 else float(a @ b / (na * nb))   # (1)

print(round(cosine([1, 2, 3], [2, 3, 4]), 3))     # → 0.993
print(round(cosine([1, 2, 3], [-1, -2, 1]), 3))   # → -0.218

V = np.array([[1, 2, 3], [2, 3, 4], [-1, -2, 1]], float)
V /= np.linalg.norm(V, axis=1, keepdims=True)                         # (2) normalise once
print((V @ V[0]).round(3))                        # (3) → [ 1.     0.993 -0.218]

Line (1) computes cosine similarity, with a guard for an all-zero vector, which has no direction. Line (2) normalises every stored vector to length 1 in one step: axis=1 computes one length per row, and keepdims=True keeps the shape so the division lines up row by row. Line (3) scores every stored vector against the first with a single matrix-vector product and gets exactly the cosines, which is how a search over a million products is computed.

A similarity score is not a probability

A tempting mistake is to treat a cosine of 0.8 as "80 percent relevant" and set a fixed cutoff like "show results above 0.75". Similarity scores have no such meaning, and their typical range depends entirely on the model. With some models, nearly every pair of English sentences scores between 0.6 and 0.9, including unrelated ones, because the vectors all lie within a narrow cone of directions; with others, unrelated pairs score near 0. A cutoff of 0.75 might return everything with one model and nothing with another.

Sometimes you need a cutoff, for example to decide "no relevant product found, show a fallback". Choose that cutoff from data, never by feel. Label a few hundred (query, product) pairs as relevant or not. Look at the score each pair gets. Then pick the threshold that gives the precision and recall you want, exactly as for a classifier in 12.3.2. And redo it whenever the model changes.

4. Using an embedding model in practice

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/e5-small-v2")                  # (1)

products = ["passage: Men's lightweight running shoes, cushioned sole",
            "passage: Electric kettle 1.5 L, stainless steel",
            "passage: 20 W USB-C fast charger for phones"]
query = "query: sneakers for jogging"                                 # (2)

P = model.encode(products, normalize_embeddings=True)                 # (3) shape (3, 384)
q = model.encode(query, normalize_embeddings=True)
scores = P @ q                                                        # (4)
print(scores.argmax(), scores.round(2))   # → 0, with the shoes clearly highest (exact values vary)

sentence-transformers is a widely used open-source Python library for computing sentence embeddings; it wraps hundreds of pre-trained models behind the same encode call. Line (1) loads a small, free model from the E5 family, published by Microsoft researchers, which produces 384-number vectors and runs comfortably on an ordinary processor.

Line (2) shows a detail that silently ruins many systems. Many embedding models are asymmetric: they were trained with queries and documents marked differently, because a short query and a long product description are different kinds of text, and the model needs to know which is which. This family expects the prefixes query: and passage:. Leave them out and search still works, just noticeably worse, with no error to tell you. Every model's documentation says what it expects; read it.

Line (3) embeds all products and the query, normalised to length 1, so that line (4), a matrix-vector product, gives cosine similarities. The shoes score highest, although the query and the description share no words.

Choosing a model

There are hundreds of embedding models, free and paid. Four questions decide between them.

How good is it at your kind of task? The standard public comparison is the MTEB leaderboard, the Massive Text Embedding Benchmark, which scores models on dozens of tasks including retrieval, in many languages. Use it to shortlist, then measure the shortlist on a test set built from your own data, because leaderboard rank and performance on Cartwise's product catalogue are not the same thing. Chapter 12.9.5 builds such a test set.

Which languages? Cartwise customers search in English, Hindi, and a mixture of the two written in Latin letters ("running shoes sasta"). A model trained mostly on English handles that badly. Multilingual models exist and must be tested on real queries.

How many dimensions, and what does storage cost? Common sizes run from 384 to over 3,000. More dimensions can hold finer distinctions but cost memory and search time. One million products at 768 dimensions, stored as 4-byte floats, is 10^6 \times 768 \times 4 \approx 3.1 gigabytes. Two techniques cut that down:

• Matryoshka embeddings, named after Russian nesting dolls, are trained so that the most important information sits in the first dimensions. You can keep only the first 256 of 1,024 numbers and still have a usable embedding, trading a little quality for a quarter of the storage.
• Quantisation stores each number in fewer bits. At one byte per number instead of four, the million products take about 0.77 gigabytes; at one bit per number, keeping only the sign, about 96 megabytes, usually followed by re-scoring the top results with the full vectors. Chapter 12.6.3 covers how vector databases use this.

Hosted or self-hosted? Embedding APIs from model providers need no infrastructure and charge per piece of text; open models like the one above cost only your own compute, and keep your data in-house. At Cartwise's scale, a few million product texts embedded once and a few million queries a day, both are viable, and the choice usually comes down to quality on your test set and where the data is allowed to go.

Changing models means re-embedding everything

One rule has no exceptions. Vectors from different embedding models cannot be compared. Each model invents its own space; dimension 17 in one means nothing in particular in another, even when both have 384 dimensions. If you switch models and embed new queries with the new model while the product vectors still come from the old one, the dot products are meaningless, search returns nonsense, and nothing raises an error. So a model upgrade is a migration: re-embed every stored item with the new model, build a new index, and switch queries and index over at the same moment. The same applies when a hosted provider retires a model version. Keep the model name and version stored next to every vector, so you always know which space it lives in.

Recall

• Averaging word vectors loses negation and word order. A sentence embedding model is a transformer whose pooled output is one vector per text.
• Contrastive learning pulls matching pairs together and pushes others apart; in-batch negatives come free from the other pairs in a batch. The InfoNCE loss is softmax cross-entropy over a row of similarities divided by a temperature. Hard negatives and your own (query, clicked product) pairs improve it most. CLIP applies the same loss to images and captions.
• With normalised vectors, cosine, dot product and Euclidean distance rank identically (\|\mathbf{a}-\mathbf{b}\|^2 = 2 - 2\cos\theta), so systems normalise once and use the dot product, unless the model was trained with an unnormalised dot product.
• Similarity scores are not probabilities; choose any cutoff from labelled data, per model. Respect asymmetric models' query and passage prefixes.
• Shortlist with MTEB, then test on your own data; check languages; weigh dimensions against storage (Matryoshka truncation, quantisation). Vectors from different models are incomparable, so changing models means re-embedding everything.

Self-test: In the contrastive table, which query would contribute the most gradient, and why? Show that for unit vectors with cosine 0.9, the squared distance is 0.2. Your search returns poor results after a provider updates its embedding model, but only for new products. What happened?

Next: 12.6.3 Vector Search at Scale — comparing a query with three products is one line; comparing it with ten million, within the 100 milliseconds a search bar allows, needs indexes that find nearly the nearest vectors without checking them all.