Appearance
12.12.2 — Caching
When the Cartwise team sampled a week of "Ask Cartwise" questions, they found that a large share were the same few dozen questions in different words: "what is your return policy", "how do I return something", "return window for phones?", "can i send it back". Each one went through retrieval, reranking and a model call, taking about four seconds and costing about a cent, to produce essentially the same answer as the last thousand times. Meanwhile, the product-search pipeline re-embedded all 400,000 products every night, although on a typical night fewer than 3,000 had changed.
A cache stores the result of work so that the same work need not be done again. Caching is the oldest performance technique in computing, covered for ordinary systems in Chapter 7.6, and AI systems offer unusually large rewards for it, because the work being repeated is so expensive. They also offer an unusual risk, which Cartwise met in its first week of caching answers: a cache that returned "Your order 4471 is out for delivery" to a customer who had asked about order 4417. The two questions were 97% similar by meaning, and completely different in fact.
This page covers each layer of caching in an AI system, what it saves, and, just as carefully, when it returns the wrong thing.
1. The layers
Caches can sit at several points on a request's path, each reusing a different kind of work.
The figure's bottom line is the theme of this page. Caches on the right reuse an intermediate result: a processed prompt, a vector, a search result. They save part of the work and, used correctly, are always safe, because the model still writes a fresh answer from them. Caches on the left reuse the final answer. They save everything, and they are the only ones that can hand a customer an answer to someone else's question. The sections go from the safest to the riskiest.
2. Prompt and prefix caches
The provider's prompt cache was the largest saving in 12.10.10: when a request begins with exactly the same tokens as a recent one, the provider reuses its processed form, the KV cache of 12.7.6, and charges a small fraction of the input price. Everything said there applies: put the unchanging parts first (tools, then the system prompt, then stable documents), keep timestamps and per-request values at the end, and check cache_read_input_tokens to confirm it is working.
Self-hosted models get the same benefit from the serving engine. vLLM's automatic prefix caching keeps the KV-cache blocks of recent requests, using the paged memory of PagedAttention, and when a new request starts with the same tokens it reuses those blocks instead of recomputing them. For the ticket classifier, whose one-line system prompt is identical for every ticket, the saving is small; for a self-hosted RAG model whose 2,000-token instructions are identical every time, it removes most of the prefill work.
This layer is always safe, because the model still reads the new part of the request and writes a fresh answer. It should simply always be on.
3. Embedding and retrieval caches
Turning text into a vector, the embedding of 12.6.1, gives the same vector for the same text every time, so it can be cached indefinitely, provided the key includes everything that affects the vector:
python
import hashlib
def embedding_key(text: str, model: str, model_version: str) -> str:
return "emb:" + hashlib.sha256(f"{model}:{model_version}:{text}".encode()).hexdigest() # (1)
def embed_cached(texts: list[str]) -> list[list[float]]:
keys = [embedding_key(t, EMBED_MODEL, EMBED_VERSION) for t in texts]
found = cache.get_many(keys) # (2)
missing = [t for t, k in zip(texts, keys) if k not in found]
if missing:
fresh = embed_model.embed(missing) # (3)
cache.set_many({embedding_key(t, EMBED_MODEL, EMBED_VERSION): v
for t, v in zip(missing, fresh)})
found.update({embedding_key(t, EMBED_MODEL, EMBED_VERSION): v
for t, v in zip(missing, fresh)})
return [found[k] for k in keys]Line (1) builds the key from a fingerprint of the text and the model name and version, so that switching embedding models, which changes every vector, can never return a vector from the old model; mixing vectors from two models in one index makes search quietly wrong, as 12.6.3 explained. The fingerprint (a SHA-256 hash) turns any length of text into a fixed-size key. Line (2) looks up all the keys in one round trip. Line (3) embeds only the texts not already cached. For the nightly product job, keyed on each product's title and description, this turned 400,000 embeddings into about 3,000: only the products that changed.
Search results can be cached too, keyed on the rewritten query and the filters, such as the user's permissions and the language. Popular questions then skip retrieval. The cache must be cleared, or its entries expired, whenever the documents change; section 6 covers how.
4. Exact-match answer caches
The simplest answer cache stores the final answer under a key made from the request. It is appropriate when the same request should always get the same answer:
- Generating content for an unchanged item. The description for product 88231 depends only on the product's data and the prompt version. Key it on both, and regenerating is needed only when either changes.
- Classifying identical text. The same ticket text forwarded twice, or the same automatic email from a courier sent to a thousand customers, should get the same category.
- General questions with no personal content, normalised by lower-casing and removing extra spaces and punctuation, so that "What is your return policy?" and "what is your return policy" share an entry.
The key must contain everything that can change the answer: the model, the version of the prompt, the version of any documents used, the language, and for multi-tenant systems the tenant. Forgetting the prompt version is the classic bug: the team improves the prompt, and the cache keeps serving answers from the old one for days.
One side effect is worth knowing. A model's answers to the same question vary slightly from run to run. An exact-match cache freezes one of them, which makes the system more consistent: every customer asking the same thing gets the same wording. That is usually a benefit, provided the frozen answer was a good one.
5. Semantic caches
Most repeated questions are not repeated word for word. A semantic cache recognises questions with the same meaning: it embeds each new question, finds the most similar earlier question in a vector index of past questions, and if the similarity is above a threshold, returns that question's stored answer.
It is the most powerful answer cache and by far the most dangerous, because "similar in meaning" is not "has the same answer". The failures Cartwise found, and their causes:
| Question | Cached question | Similarity | Problem |
|---|---|---|---|
| Where is order 4417? | Where is order 4471? | 0.97 | different order |
| Can I return earphones? | Can I return opened earphones? | 0.95 | different rule |
| Return window for Plus members? | Return window? | 0.93 | different policy |
| Is it too late to return it? | (any) | n/a | depends on whose order |
Every row is the same underlying problem: an embedding captures what a question is about, not every detail that decides its answer. Order numbers, a single word such as "opened", a membership type, or simply who is asking all change the answer while barely moving the vector.
So a semantic cache is used only behind guards, each addressing one of those causes:
python
def semantic_lookup(question: str, ctx) -> str | None:
if ctx.mentions_personal_data or ctx.is_logged_in_order_question: # (1)
return None
entities = extract_entities(question) # order numbers, products, member types (2)
vector = embed_cached([question])[0]
hit = answer_index.nearest(vector, filters={"locale": ctx.locale,
"policy_version": POLICY_VERSION}) # (3)
if hit is None or hit.similarity < 0.93: # (4)
return None
if hit.entities != entities: # (5)
return None
return hit.answerLine (1) refuses to use the cache for anything personal: questions about a specific order, an account, or anything whose answer depends on who is asking. Those answers are not shared between customers, so they must never come from a shared cache. Line (2) extracts the details an embedding blurs: order numbers, product names, membership types, and words such as "opened" or "damaged" that change which rule applies. Line (3) searches only among cached answers in the same language and written under the current policy version, so a policy change makes every older entry unreachable at once.
Line (4) applies the similarity threshold. Line (5) requires the extracted details to match exactly, which is what stops "order 4417" from matching "order 4471" and "opened earphones" from matching "earphones". With these guards, Cartwise sent about 35% of "Ask Cartwise" questions through the semantic cache at all (the general, non-personal ones), and about 60% of those were hits. That meant roughly a fifth of all questions were answered in about 20 milliseconds instead of four seconds, at almost no cost.
Choosing the threshold
The threshold trades hits against mistakes, and the figure shows the shape of the trade.
Lowering the threshold catches more rephrasings, but also more questions that only look alike. There is no universal right value; it depends on the embedding model and the questions. It is found by measurement: take a few thousand real question pairs that the cache would match at various thresholds, have a model judge or a person decide whether the cached answer actually answers the new question, and choose the threshold where wrong answers become rare enough for your use. Then keep measuring in production, by sampling cache hits for the same review, because traffic changes.
A safer variant caches one step earlier. Instead of storing final answers, store the retrieved passages for each question. A semantic hit then skips retrieval, and the model still writes a fresh answer from the passages and the actual question. A look-alike question gets the right passages, if not perfectly targeted ones, and the model can still answer "opened earphones" differently from "earphones" because it reads the real words. This saves less than caching the answer, but it almost never produces a wrong one.
6. Keeping caches fresh
A cache entry is only correct while what it was computed from is unchanged. There are three ways to keep caches fresh, and real systems use all three.
Expiry (TTL). Each entry is kept for a fixed time and then discarded. It is simple and suits data that changes on its own schedule: an agent's cached order status might live for 60 seconds, a cached search result for an hour. The cost is a window of staleness: until expiry, an entry may be out of date.
Versions in the key. The prompt version, the policy version, the embedding model version and the document version are part of the key or the filter, as in the code above. A change of version makes old entries unreachable immediately, with no work at all. This is the strongest technique, and the reason the semantic lookup filters on POLICY_VERSION.
Purge on change. When a document changes, delete every entry that used it. This needs each cached answer stored with the IDs of the documents behind it, which a RAG system already has from its citations (12.9.4). When returns-2 is edited, the ingestion pipeline of 12.9.2 publishes an event, and every cached answer that cited returns-2 is deleted.
When Cartwise changed the electronics return window from 10 to 14 days, all three were in play: the new policy version made every old answer unreachable at once, the purge cleared the entries that cited the old document, and expiry would have caught anything missed within a day.
7. Measuring caches
Each layer is measured by three numbers: its hit rate (the share of lookups it answered), what each hit saves (tokens, money, milliseconds), and, for answer caches, its wrong-hit rate (the share of hits that gave a wrong answer), measured by sampling hits for review. A cache with a high hit rate and an unmeasured wrong-hit rate is a liability whose size you do not know.
Measured this way, Cartwise's layers ranked like this:
| Layer | Hit rate | Saving | Risk |
|---|---|---|---|
| Prompt cache | 90%+ of input tokens | largest, every request | none |
| Embedding cache | 99% nightly | most of the nightly job | none if versioned |
| Semantic answer cache | ~20% of all questions | whole requests | measured, ~0.3% of hits |
| Exact answer cache | ~5% of questions | whole requests | low if versioned |
| Tool result cache | ~40% of order lookups | API calls | staleness window |
The safe layers deliver the most dependable savings and should be on everywhere. The answer caches deliver the most dramatic savings per hit and need the guards, the versions and the measurement of this page to be worth having.
What the interviewer will push on
"How would you reduce the cost of an LLM chatbot?" A layered answer:
- The provider's prompt caching first. It is always safe, and often the biggest saving.
- Caching embeddings and retrieval results.
- An answer cache for general questions that are not about one person.
- Routing to cheaper models.
- Batching work that can wait.
The tell is ranking them by risk as well as saving.
"How does a semantic cache work, and what can go wrong?" Embed the question, find the nearest earlier question, return its answer above a similarity threshold. What goes wrong: questions about different entities or with one decisive word changed are highly similar but need different answers; personal questions must never be shared; stale answers after a policy change. Defences: no personal questions, exact match on extracted entities, version filters, measured thresholds, or caching retrieved passages instead of answers.
"How do you invalidate an AI cache?" Expiry for data that changes on its own schedule, versions in keys for prompts, policies and models, and purging entries by the document IDs they cited when a document changes.
"What must a cache key include?" Everything that can change the result: model and version, prompt version, document or policy version, language, tenant, and the normalised input. Forgetting the prompt version is the classic bug.
Something to volunteer: a cache that returns another customer's answer is a privacy incident, not just a wrong answer. That is why personal questions are excluded from shared answer caches entirely, not merely given a higher threshold.
Recall
- AI systems cache at several layers: answer caches (exact, semantic), retrieval caches (embeddings, search results), model-call caches (provider prompt cache, self-hosted prefix cache), and agent tool results.
- Intermediate caches are safe because the model still writes a fresh answer; answer caches save the most and are the only ones that can return a wrong answer.
- Prompt and prefix caches: always on; stable content first. Embedding cache keys include model and version. Search-result caches are keyed on query and filters and cleared on document change.
- Exact answer caches suit deterministic, non-personal requests; the key includes model, prompt version, document version, language and tenant.
- A semantic cache matches by meaning; similar questions can have different answers (different order numbers, one decisive word, membership type, who is asking). Guards: no personal questions, exact match on extracted entities, version filters, a measured threshold; or cache retrieved passages instead.
- Freshness: expiry, versions in keys, purge by cited document IDs. Measure hit rate, saving and, for answer caches, wrong-hit rate.
Self-test: Why did "Where is order 4417?" match "Where is order 4471?", and which line of semantic_lookup now prevents it? What must change in the cache when the embedding model is upgraded? Why is caching retrieved passages safer than caching final answers?
Next: 12.12.3 GPUs, Latency and Cost — for the models Cartwise runs itself, the next page covers the hardware: which GPU to choose, why generating text is limited by memory speed rather than arithmetic, how to plan capacity, and how latency is measured and budgeted.