Why can’t you just score every item with your best model?

Because the arithmetic forbids it. A heavy relevance model that reads hundreds of features costs on the order of a few milliseconds per item — say 5 ms — and a request has a latency budget of roughly 200 ms. That means the best model can score about 40 items before the budget is gone, while the catalog holds millions. Scoring ten million items at 5 ms each is about fourteen hours of compute for a single request that owes an answer in a fifth of a second — off by a factor of a quarter-million, unfixable by a faster machine. So something far cheaper has to throw away 99.999% of the catalog before the good model ever runs. That forced move is why every large ranking system is a multi-stage funnel rather than one model and a sort. The Facebook ads paper frames the whole system the same way: "we would first build a cascade of classifiers of increasing computational cost."

What is the candidate generation, ranking, and re-ranking funnel?

It is the staged architecture that lets a slow, accurate model run under a tight latency budget. Candidate generation (also called retrieval or sourcing) reaches into the millions and cheaply returns a few thousand plausible items using pre-built structures — recent, popular, followed, and an approximate-nearest-neighbour embedding index — its job is recall, not precision. A light ranker then scores those few thousand with a cheap model in a fraction of a millisecond each and trims to a few dozen. A heavy ranker scores only those survivors with the expensive model that actually sets the order — a predicted click-through rate for ads, predicted engagement for a feed. Finally re-ranking reshapes the ordered list with business rules: diversity caps, freshness and fatigue, dedup, hard filters. Each stage looks at fewer items than the last and spends more per item, so the heavy model runs only where it fits the budget.

Why do ad click-prediction models need to be calibrated?

Because an ad auction ranks on expected revenue — the predicted click probability times the bid, pCTR × bid — so the probability has to be right in an absolute sense, not just in rank order. Calibration means a model that says "2%" is right on average when the true click rate of those items is 2%. The Facebook paper defines it as "the ratio of the average estimated CTR and empirical CTR" and calls it "essential to the success of online bidding and auction." An uncalibrated model loses real money: if it over-predicts one ad's click rate 2×, that ad's expected value inflates, it wins auction slots it should have lost to better-paying ads, and the platform collects less than it would have — while AUC, which "measures ranking quality without considering calibration," never sees the problem. When training down-samples negatives (real click rates are a few percent), the output probabilities are distorted and must be re-calibrated back to reality with q = p / (p + (1 − p) / w), or via an isotonic-regression calibration layer.

What is position bias in ranking training data?

Position bias is the fact that an item near the top of a list gets clicked more because of its position, not because it is better — so click logs are a biased signal for relevance. If you train naively on clicks, the model learns that whatever the old system happened to rank first is good, which means it learns its own past behaviour rather than true relevance. The unbiased-learning-to-rank literature states it directly: "the naïve approach of treating a click/no-click as a positive/negative relevance judgment is severely biased. In particular, the order of presentation has a strong influence on where users click." The effect is large — the top slot can draw several times the attention of the fifth — so equal items collect wildly unequal clicks. The fix is to log the propensity (how likely each item was to be shown at its position) and inverse-weight the training examples by it, so a click on a buried item counts for more than a click on one shown at the top.

How does harmful-content moderation reuse a ranking system?

It is the same architecture pointed at a different objective, with a different head. Candidate generation becomes "every piece of content posted," and the ranker becomes a classifier scoring each item's probability of breaking a rule. What changes is what you do with the score: a feed sorts by it, but moderation decides against a threshold — above it, act; below it, leave alone. That threshold is a precision/recall dial: raise it and you flag less but are more often right (higher precision, lower recall); lower it and you catch more harmful content but drown in false alarms. The part interviews forget is that a moderation system is a model plus a finite human-review queue: whatever the model flags but can't auto-action goes to reviewers, and if flagged-per-day exceeds what they can clear, harmful content sits live while the queue backs up. So the threshold is also a staffing decision — prevalence times volume versus review capacity — and only the highest-confidence scores should be auto-actioned.

Design a Ranking System

Ten million items, one person, two hundred milliseconds — you can never score the whole catalog with your best model, so every large ranker is a funnel. We compute the scoring budget first (a 5 ms/item heavy model against a 200 ms wall scores ~40 items, not ten million), then build the funnel live as a budget negotiation, walk candidate generation → light ranker → heavy ranker → re-ranking, and face the parts interviews skip: calibration (why an uncalibrated ad model loses real money), position bias in the training logs, offline-vs-online divergence, and the harmful-content moderation variant with its human-review queue — grounded in the published ad-CTR papers.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of Design a Ranking System — built to be understood, not skimmed.

Questions

Why can’t you just score every item with your best model?: Because the arithmetic forbids it. A heavy relevance model that reads hundreds of features costs on the order of a few milliseconds per item — say 5 ms — and a request has a latency budget of roughly 200 ms. That means the best model can score about 40 items before the budget is gone, while the catalog holds millions. Scoring ten million items at 5 ms each is about fourteen hours of compute for a single request that owes an answer in a fifth of a second — off by a factor of a quarter-million, unfixable by a faster machine. So something far cheaper has to throw away 99.999% of the catalog before the good model ever runs. That forced move is why every large ranking system is a multi-stage funnel rather than one model and a sort. The Facebook ads paper frames the whole system the same way: "we would first build a cascade of classifiers of increasing computational cost."
What is the candidate generation, ranking, and re-ranking funnel?: It is the staged architecture that lets a slow, accurate model run under a tight latency budget. Candidate generation (also called retrieval or sourcing) reaches into the millions and cheaply returns a few thousand plausible items using pre-built structures — recent, popular, followed, and an approximate-nearest-neighbour embedding index — its job is recall, not precision. A light ranker then scores those few thousand with a cheap model in a fraction of a millisecond each and trims to a few dozen. A heavy ranker scores only those survivors with the expensive model that actually sets the order — a predicted click-through rate for ads, predicted engagement for a feed. Finally re-ranking reshapes the ordered list with business rules: diversity caps, freshness and fatigue, dedup, hard filters. Each stage looks at fewer items than the last and spends more per item, so the heavy model runs only where it fits the budget.
Why do ad click-prediction models need to be calibrated?: Because an ad auction ranks on expected revenue — the predicted click probability times the bid, pCTR × bid — so the probability has to be right in an absolute sense, not just in rank order. Calibration means a model that says "2%" is right on average when the true click rate of those items is 2%. The Facebook paper defines it as "the ratio of the average estimated CTR and empirical CTR" and calls it "essential to the success of online bidding and auction." An uncalibrated model loses real money: if it over-predicts one ad's click rate 2×, that ad's expected value inflates, it wins auction slots it should have lost to better-paying ads, and the platform collects less than it would have — while AUC, which "measures ranking quality without considering calibration," never sees the problem. When training down-samples negatives (real click rates are a few percent), the output probabilities are distorted and must be re-calibrated back to reality with q = p / (p + (1 − p) / w), or via an isotonic-regression calibration layer.
What is position bias in ranking training data?: Position bias is the fact that an item near the top of a list gets clicked more because of its position, not because it is better — so click logs are a biased signal for relevance. If you train naively on clicks, the model learns that whatever the old system happened to rank first is good, which means it learns its own past behaviour rather than true relevance. The unbiased-learning-to-rank literature states it directly: "the naïve approach of treating a click/no-click as a positive/negative relevance judgment is severely biased. In particular, the order of presentation has a strong influence on where users click." The effect is large — the top slot can draw several times the attention of the fifth — so equal items collect wildly unequal clicks. The fix is to log the propensity (how likely each item was to be shown at its position) and inverse-weight the training examples by it, so a click on a buried item counts for more than a click on one shown at the top.
How does harmful-content moderation reuse a ranking system?: It is the same architecture pointed at a different objective, with a different head. Candidate generation becomes "every piece of content posted," and the ranker becomes a classifier scoring each item's probability of breaking a rule. What changes is what you do with the score: a feed sorts by it, but moderation decides against a threshold — above it, act; below it, leave alone. That threshold is a precision/recall dial: raise it and you flag less but are more often right (higher precision, lower recall); lower it and you catch more harmful content but drown in false alarms. The part interviews forget is that a moderation system is a model plus a finite human-review queue: whatever the model flags but can't auto-action goes to reviewers, and if flagged-per-day exceeds what they can clear, harmful content sits live while the queue backs up. So the threshold is also a staffing decision — prevalence times volume versus review capacity — and only the highest-confidence scores should be auto-actioned.

Design a Ranking System

Questions

Related explainers