What is an embedding retrieval system?

It is the machinery behind search-by-meaning: finding the items most relevant to a query when the two share no keywords at all — a photo that finds a product listing, a phrase that finds a video, a person who finds the people they might know. The trick is to turn every item and every query into an embedding — a learned point in a high-dimensional space, positioned so that things which mean similar things land near each other. Retrieval then reduces to a geometry problem: find the item points nearest the query point. The YouTube recommendations paper states this directly — once the model is trained, "the scoring problem reduces to a nearest neighbor search in the dot product space for which general purpose libraries can be used." A production system embeds its whole corpus offline, builds a specialized index over those points, and at request time embeds the query and asks the index for its nearest neighbors in a few milliseconds.

Why do embedding retrieval systems use two towers instead of one model?

Because the query and the corpus live on completely different time budgets. A two-tower model is two separate encoders — an item tower and a query tower — that map into the same embedding space, and, in the words of the two-tower retrieval paper, "the output of the model is the inner product of two embeddings." That split is the whole point: the item tower runs offline over the entire catalog, so the hundred-million item embeddings are computed once and frozen into an index. Only the query tower runs at request time, on one input, producing one vector. If a single model had to score the query against every item jointly, you would rerun the network a hundred million times per query; with two towers you run it once and let a nearest-neighbor index do the rest. The price is that the two towers can only interact through a dot product at the very end — no early cross-features — which is exactly why retrieval is a coarse first stage that a heavier ranking model refines.

What is the difference between exact and approximate nearest neighbor (ANN) search?

Exact search compares the query against every single vector and is guaranteed correct — FAISS notes that "the only index that can guarantee exact results is the IndexFlatL2 or IndexFlatIP" — but it is brute force: 100 million 256-dimensional vectors is 102 GB that every query must stream through, which at thousands of queries per second demands petabytes-per-second of memory bandwidth and hundreds of GPUs just to read the data. Approximate nearest neighbor (ANN) search trades a little accuracy for orders of magnitude less work: it looks at a small, cleverly chosen fraction of the vectors and accepts that it will occasionally miss a true neighbor. The quality it keeps is called recall — the fraction of the genuine top-k that it actually returned. Every ANN index exposes one dial that trades recall for speed: for IVF it is nprobe, for HNSW it is efSearch.

How do IVF and HNSW vector indexes differ?

They are two different ways to avoid scanning everything. IVF (an inverted file index) first clusters the corpus into cells around centroids — FAISS suggests on the order of "4*sqrt(N) to 16*sqrt(N)" cells — and at query time only searches the nprobe cells nearest the query, so "as a first approximation, this fraction is nprobe/nlist" of the database. Its failure mode is geometric: recall drops "when the cell of the nearest neighbor of a given query is not selected," i.e. a true neighbor sits just across a cell boundary you did not probe. HNSW builds a navigable small-world graph and greedily walks it toward the query in roughly logarithmic hops; its speed-accuracy dial is efSearch, and FAISS is blunt about the tradeoff — "if you have a lots of RAM or the dataset is small, HNSW is the best option, it is a very fast and accurate index." The catch is memory: HNSW stores the graph on top of the vectors, "(d * 4 + M * 2 * 4) bytes per vector," and it "does not support removing vectors from the index."

How do you keep a vector index fresh as new items arrive?

The main index is built offline and shipped as an immutable snapshot, which is what makes it cheap to serve — but a snapshot is stale the moment a new item appears, and for a corpus like YouTube, where "many hours of video are uploaded per second," staleness is the product problem. The standard answer is a two-tier read: a large, immutable base index rebuilt on a schedule, plus a small, mutable delta index that ingests fresh item embeddings continuously. A query searches both and merges the results, so a brand-new item is retrievable within seconds instead of waiting for the next full build. Periodically the delta is folded into a fresh base snapshot and the cycle restarts. This mirrors how log-structured storage keeps a big sorted file alongside a small in-memory buffer — the same shape, applied to vectors.

Design an Embedding Retrieval System

Search where the query and the item share no words at all — a photo finds the listing, a phrase finds the video, a person finds the people they might know. The retrieval shape under visual search, video search, similar listings, and People-You-May-Know, built from zero: what an embedding is (a learned point in space; nearness means similarity), why two towers instead of one model, a commit-first envelope on a 100M-item index, the honest deliberation between a brute-force GPU scan, IVF partitions, and an HNSW graph — with recall vs latency computed live on a real toy corpus — then the offline indexing pipeline, the online path, keeping the index fresh, training the towers on engagement with in-batch negatives, the multimodal shared space, and the failure sweep whose quietest box returns a confident 200 while silently retrieving nonsense.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of Design an Embedding Retrieval System — built to be understood, not skimmed.

Questions

What is an embedding retrieval system?: It is the machinery behind search-by-meaning: finding the items most relevant to a query when the two share no keywords at all — a photo that finds a product listing, a phrase that finds a video, a person who finds the people they might know. The trick is to turn every item and every query into an embedding — a learned point in a high-dimensional space, positioned so that things which mean similar things land near each other. Retrieval then reduces to a geometry problem: find the item points nearest the query point. The YouTube recommendations paper states this directly — once the model is trained, "the scoring problem reduces to a nearest neighbor search in the dot product space for which general purpose libraries can be used." A production system embeds its whole corpus offline, builds a specialized index over those points, and at request time embeds the query and asks the index for its nearest neighbors in a few milliseconds.
Why do embedding retrieval systems use two towers instead of one model?: Because the query and the corpus live on completely different time budgets. A two-tower model is two separate encoders — an item tower and a query tower — that map into the same embedding space, and, in the words of the two-tower retrieval paper, "the output of the model is the inner product of two embeddings." That split is the whole point: the item tower runs offline over the entire catalog, so the hundred-million item embeddings are computed once and frozen into an index. Only the query tower runs at request time, on one input, producing one vector. If a single model had to score the query against every item jointly, you would rerun the network a hundred million times per query; with two towers you run it once and let a nearest-neighbor index do the rest. The price is that the two towers can only interact through a dot product at the very end — no early cross-features — which is exactly why retrieval is a coarse first stage that a heavier ranking model refines.
What is the difference between exact and approximate nearest neighbor (ANN) search?: Exact search compares the query against every single vector and is guaranteed correct — FAISS notes that "the only index that can guarantee exact results is the IndexFlatL2 or IndexFlatIP" — but it is brute force: 100 million 256-dimensional vectors is 102 GB that every query must stream through, which at thousands of queries per second demands petabytes-per-second of memory bandwidth and hundreds of GPUs just to read the data. Approximate nearest neighbor (ANN) search trades a little accuracy for orders of magnitude less work: it looks at a small, cleverly chosen fraction of the vectors and accepts that it will occasionally miss a true neighbor. The quality it keeps is called recall — the fraction of the genuine top-k that it actually returned. Every ANN index exposes one dial that trades recall for speed: for IVF it is nprobe, for HNSW it is efSearch.
How do IVF and HNSW vector indexes differ?: They are two different ways to avoid scanning everything. IVF (an inverted file index) first clusters the corpus into cells around centroids — FAISS suggests on the order of "4*sqrt(N) to 16*sqrt(N)" cells — and at query time only searches the nprobe cells nearest the query, so "as a first approximation, this fraction is nprobe/nlist" of the database. Its failure mode is geometric: recall drops "when the cell of the nearest neighbor of a given query is not selected," i.e. a true neighbor sits just across a cell boundary you did not probe. HNSW builds a navigable small-world graph and greedily walks it toward the query in roughly logarithmic hops; its speed-accuracy dial is efSearch, and FAISS is blunt about the tradeoff — "if you have a lots of RAM or the dataset is small, HNSW is the best option, it is a very fast and accurate index." The catch is memory: HNSW stores the graph on top of the vectors, "(d * 4 + M * 2 * 4) bytes per vector," and it "does not support removing vectors from the index."
How do you keep a vector index fresh as new items arrive?: The main index is built offline and shipped as an immutable snapshot, which is what makes it cheap to serve — but a snapshot is stale the moment a new item appears, and for a corpus like YouTube, where "many hours of video are uploaded per second," staleness is the product problem. The standard answer is a two-tier read: a large, immutable base index rebuilt on a schedule, plus a small, mutable delta index that ingests fresh item embeddings continuously. A query searches both and merges the results, so a brand-new item is retrievable within seconds instead of waiting for the next full build. Periodically the delta is folded into a fresh base snapshot and the cycle restarts. This mirrors how log-structured storage keeps a big sorted file alongside a small in-memory buffer — the same shape, applied to vectors.

Design an Embedding Retrieval System

Questions

Related explainers