What is a web crawler?

In the words of Olston and Najork’s survey, "a web crawler (also known as a robot or a spider) is a system for the bulk downloading of web pages" — the component that feeds a search engine’s index. The basic algorithm is simple: start from a set of seed URLs, download them, extract the hyperlinks they contain, and iteratively download the pages those links point to. The hard part is doing that at web scale — billions of pages — while staying polite to each site, avoiding traps and duplicates, and keeping the crawled copies reasonably fresh.

What is the URL frontier and why does it matter?

The frontier is the data structure holding every URL discovered but not yet downloaded — it decides what to crawl next, and it is where almost all the design lives. A naive first-in-first-out frontier gives a breadth-first crawl, but since most links on a page point back to the same site, that fires "a barrage of requests in short order" at one server, which "is considered ‘impolite,’ and may be construed as a denial-of-service attack." The real design is two stages: front queues that implement prioritisation (crawl important pages first) feeding back queues that implement politeness (one queue per host, drained by one worker, with a delay between fetches). The frontier balances "what’s worth crawling next" against "what am I allowed to crawl right now without hammering anyone."

How does a crawler stay polite to websites?

Two rules. First, it obeys robots.txt: "before attempting to crawl a site, a crawler should check whether the site supplies a /robots.txt file, and if so, adhere to its rules," caching that file so it isn’t re-fetched constantly. Second, it rate-limits itself per host — "most web crawlers obey a policy of not issuing multiple overlapping requests to the same server," and a conservative policy is to "delay subsequent requests by a multiple (say 10×) of the time it took to download the last page from that server." That per-host delay, enforced by giving each host its own back queue drained by a single worker, is what keeps a high-throughput crawler from accidentally taking a small site down.

How does a crawler avoid crawling the same page twice?

With two separate deduplication structures. A URL-seen test tracks every URL already discovered so links aren’t re-added to the frontier; because it only needs set membership, "there are multiple straightforward in-memory implementations… e.g., a hash table or Bloom filter," and each URL can be "compressed to a much smaller token, e.g., a 10-byte hash value," so it scales to billions. A content-seen test catches the different problem of the same content served under different URLs — it hashes the page body into a fingerprint and skips pages whose fingerprint it has already stored, since "there is a prevalence of duplicate and near-duplicate content on the web."

What is a spider trap and how do crawlers defend against it?

A spider trap is a site that "populates a large, possibly infinite URL space… with mechanically generated content" — the classic example is a calendar whose every month links to the next and previous month, "forming an unbounded chain of dynamically generated pages." Some traps are innocent; others are set up by spammers to inject content into the index. They’re dangerous because every generated URL is genuinely distinct, so the URL-seen dedup doesn’t catch them and the frontier floods. Defenses are heuristic rather than perfect: cap URL length and crawl depth, budget the number of pages fetched per site, and maintain manual blacklists of known-bad hosts.

Design a Web Crawler

The bulk downloader behind a search engine — built from zero. Why naive breadth-first crawling is rude, and how the URL frontier makes it polite: front queues that prioritise (by PageRank, traffic, freshness) feeding back queues that enforce one host per queue, one worker, a delay between fetches. Plus robots.txt and its cache, the two dedup structures (a bloom filter of seen URLs, a hash of seen content), DNS caching and the blocking-resolver trap, spider-trap defenses, and a freshness strategy — with a live frontier you can flood from one host to watch politeness throttle it while the others proceed.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Web Crawler — built to be understood, not skimmed.

Questions

What is a web crawler?: In the words of Olston and Najork’s survey, "a web crawler (also known as a robot or a spider) is a system for the bulk downloading of web pages" — the component that feeds a search engine’s index. The basic algorithm is simple: start from a set of seed URLs, download them, extract the hyperlinks they contain, and iteratively download the pages those links point to. The hard part is doing that at web scale — billions of pages — while staying polite to each site, avoiding traps and duplicates, and keeping the crawled copies reasonably fresh.
What is the URL frontier and why does it matter?: The frontier is the data structure holding every URL discovered but not yet downloaded — it decides what to crawl next, and it is where almost all the design lives. A naive first-in-first-out frontier gives a breadth-first crawl, but since most links on a page point back to the same site, that fires "a barrage of requests in short order" at one server, which "is considered ‘impolite,’ and may be construed as a denial-of-service attack." The real design is two stages: front queues that implement prioritisation (crawl important pages first) feeding back queues that implement politeness (one queue per host, drained by one worker, with a delay between fetches). The frontier balances "what’s worth crawling next" against "what am I allowed to crawl right now without hammering anyone."
How does a crawler stay polite to websites?: Two rules. First, it obeys robots.txt: "before attempting to crawl a site, a crawler should check whether the site supplies a /robots.txt file, and if so, adhere to its rules," caching that file so it isn’t re-fetched constantly. Second, it rate-limits itself per host — "most web crawlers obey a policy of not issuing multiple overlapping requests to the same server," and a conservative policy is to "delay subsequent requests by a multiple (say 10×) of the time it took to download the last page from that server." That per-host delay, enforced by giving each host its own back queue drained by a single worker, is what keeps a high-throughput crawler from accidentally taking a small site down.
How does a crawler avoid crawling the same page twice?: With two separate deduplication structures. A URL-seen test tracks every URL already discovered so links aren’t re-added to the frontier; because it only needs set membership, "there are multiple straightforward in-memory implementations… e.g., a hash table or Bloom filter," and each URL can be "compressed to a much smaller token, e.g., a 10-byte hash value," so it scales to billions. A content-seen test catches the different problem of the same content served under different URLs — it hashes the page body into a fingerprint and skips pages whose fingerprint it has already stored, since "there is a prevalence of duplicate and near-duplicate content on the web."
What is a spider trap and how do crawlers defend against it?: A spider trap is a site that "populates a large, possibly infinite URL space… with mechanically generated content" — the classic example is a calendar whose every month links to the next and previous month, "forming an unbounded chain of dynamically generated pages." Some traps are innocent; others are set up by spammers to inject content into the index. They’re dangerous because every generated URL is genuinely distinct, so the URL-seen dedup doesn’t catch them and the frontier floods. Defenses are heuristic rather than perfect: cap URL length and crawl depth, budget the number of pages fetched per site, and maintain manual blacklists of known-bad hosts.

Design a Web Crawler

Questions

Related explainers