Design a Web Crawler

A web crawler is deceptively approachable, "fetch pages, extract links, repeat" is a three-word description that fits in a tweet. The interview question is a trap for exactly that reason: the system looks trivial until you start asking the hard questions. How do you avoid crawling the same URL twice across a fleet of 500 workers? How do you avoid taking down small websites? How do you prioritise the 1% of URLs that account for 50% of the value?

The question tests three specific things that don't come up in most other system design problems:

Graph traversal at scale. The web is a directed graph with ~50 billion nodes and trillions of edges. Choosing BFS vs DFS isn't academic, it determines whether your crawl discovers important pages in hours or months.

Distributed coordination without a central bottleneck. A single seen-URL set that every worker writes to will become the choke point. Candidates who don't think about this usually hit it around the "scalability" section.

Being a good citizen. A crawler that ignores robots.txt and hammers domains will get IP-blocked and potentially create legal exposure. Politeness isn't a nice-to-have, it's architecturally required.

Functional requirements

Non-functional requirements

NFR reasoning

Before reaching for numbers, consider the shape of each dimension. Write throughput is pure fetch rate, easy to estimate but dominates bandwidth and storage planning. Storage compounds quickly: raw HTML at scale means petabytes, not terabytes. URL deduplication state is the unusual one here, the seen-URL set must fit in memory or we lose the ability to do fast deduplication. That constraint deserves its own estimate.

Component breakdown

URL Frontier Service, almost every scalability decision in this design traces back here. The frontier queues URLs waiting to be crawled, organises them into per-domain priority buckets, and enforces politeness delays so no single domain gets hammered. Unlike a simple FIFO, it must know which domains are rate-limited before dispatching the next URL.

Fetcher Pool, a fleet of HTTP worker processes that pull URLs from the frontier, issue the actual HTTP/S request, and stream raw HTML back for processing. Fetchers maintain their own DNS cache (DNS round-trips at 11k req/s would become the bottleneck) and a per-domain robots.txt cache with a 24h TTL.

Parser / Link Extractor, receives raw HTML, extracts all anchor href values, normalises them (resolve relative paths, strip fragments, canonicalise scheme and host), and computes a SimHash fingerprint of the body for near-duplicate detection. New URLs discovered here loop back to the frontier; the content hash goes to the metadata store.

Dedup Store, guards the frontier against re-enqueuing URLs already crawled. A distributed Bloom filter (backed by an in-memory store like Redis) handles the fast path; an exact hash set backstops the small fraction the filter passes through. Any URL must clear both gates before entering the frontier.

Object Store, holds raw HTML blobs, referenced by URL hash. A distributed blob store (Amazon S3 or Google Cloud Storage) handles the write throughput and durability that a relational database's BLOB columns cannot, raw HTML at 11,000 writes/second and 100KB per page demands it.

Metadata DB, records per-URL state: crawl status, last fetch timestamp, content hash, link count, HTTP response code, ETag, and computed importance score. The re-crawl Scheduler reads it to compute the next fetch time for each URL, making it the control plane for the entire crawl lifecycle.

Architectural rationale

Real-world comparison

Two questions dominate this section: in what order should URLs be fetched? and how do you know you've already seen a URL? They're related, the frontier algorithm determines how often URLs arrive at the Dedup Store, and the Dedup Store must keep up. At 11k QPS, that means sub-millisecond lookups; anything that falls through to a 10ms database read becomes the system's throughput ceiling.

Our choice for this system: BFS with priority scoring

Standard BFS discovers pages linked by many other pages (high-authority pages) early in the crawl. Adding a priority score, based on estimated page importance, change frequency, and link depth, turns the FIFO BFS queue into a best-first crawl that approximates what search engines actually do. A newly discovered URL gets a base priority; crawl history and inbound link count update it over time.

URL deduplication: two-tier approach

Tier 1 efficiently handles the vast majority of checks. Every URL the frontier wants to enqueue passes through Tier 1 first, if the Bloom filter says "seen", the URL is dropped immediately without hitting Redis. If the filter says "not seen", the URL proceeds to Tier 2 for an exact confirmation. The false positive case, where the Bloom filter incorrectly reports a genuinely new URL as already-seen, causes that URL to be silently skipped. At a 0.01% false positive rate this is acceptable crawl budget loss (roughly 1M missed URLs per day at 1B pages/day scale). Tier 2 is exact: a Redis HSET lookup on the normalised URL fingerprint, with 0% false positive rate. A Bloom filter never produces false negatives, so any URL that passes both tiers is definitively new.

A web crawler has two external-facing protocol surfaces: the HTTP protocol it speaks to target websites, and the internal API that external systems (search indexer, ML pipelines) use to query or drive the crawl. Both matter in an interview.

External: HTTP fetching protocol

Fetchers issue standard HTTP/1.1 or HTTP/2 GET requests. Key headers the crawler must send and respect:

// Outbound request headers (what we send)
GET /page HTTP/1.1
Host: example.com
User-Agent: MyCrawler/1.0 (+https://mysite.com/crawler)
Accept: text/html,application/xhtml+xml
Accept-Encoding: gzip, deflate, br
If-None-Match: "etag-from-last-crawl"       // conditional GET, avoid re-downloading if unchanged
If-Modified-Since: Tue, 01 Apr 2025 12:00:00 GMT

// Response handling
200 OK       → parse and store
301/302      → follow redirect (max 5 hops to prevent loops)
304 Not Modified → skip storage, update last-checked timestamp only
429 Too Many Requests → back off exponentially, respect Retry-After header
5xx          → retry with jitter, increment failure counter

Internal: Crawl Management API

The system exposes a REST API for operators and downstream consumers. Core endpoints:

POST /crawl/seeds, Submit seed URLs to bootstrap a crawl.

// Request
{
  "urls": ["https://example.com", "https://other.com"],
  "priority": "high",          // "high" | "normal" | "low"
  "crawl_depth": 3,            // max BFS depth from seed, null = unlimited
  "recrawl_ttl_hours": 24
}

// Response: 202 Accepted
{
  "job_id": "job_abc123",
  "queued_urls": 2,
  "estimated_start": "2026-04-01T12:00:05Z"
}

GET /pages/{url_hash}, Retrieve crawl metadata for a specific URL.

// Response: 200 OK
{
  "url": "https://example.com/page",
  "url_hash": "sha256:abc123...",
  "last_fetched_at": "2026-04-01T10:00:00Z",
  "http_status": 200,
  "content_hash": "md5:def456...",
  "simhash": "0xA3F2...",
  "content_length_bytes": 102400,
  "outbound_link_count": 47,
  "storage_key": "s3://crawl-bucket/pages/abc123.html.gz",
  "crawl_status": "indexed"      // "queued" | "fetching" | "indexed" | "failed"
}

The dominant path is the single-URL fetch cycle. Understanding each decision point, especially what happens on a cache hit, a redirect, or a robots.txt denial, is what separates an L3 answer from an L5 one. The key decision the NFR forces is conditional GET: the crawl freshness requirement (§2) demands we re-visit pages, but the storage NFR says we shouldn't re-store unchanged content. Conditional GET via ETags or If-Modified-Since resolves this.

There are three distinct entities in the crawl system. The differences in how each is used end up shaping the schema significantly, particularly the choice of key space and the need for certain index patterns.

Entities and access patterns

Two things stand out. First, dedup lookups and writes run at the same rate as fetches, 11,000/s, so the Dedup Store must handle that write rate without becoming the bottleneck. Second, the "next URLs to crawl" access pattern is the tricky one: it requires a range scan ordered by time, which Cassandra cannot serve efficiently from url_records, a WHERE next_crawl_at < now() query across all partitions is a full-cluster scatter-gather, not a range query. This forces a dedicated scheduling data structure, described below.

Schema

url_records, the metadata control plane (wide-column store like Apache Cassandra or AWS DynamoDB, sharded by url_hash):

-- url_records (Cassandra-style)
url_hash        TEXT  PRIMARY KEY    -- SHA-256 of normalised URL
url             TEXT                 -- full normalised URL
domain          TEXT                 -- extracted hostname, used for partition routing
status          TEXT                 -- queued | fetching | indexed | failed | skipped
http_status     INT                  -- last HTTP response code
last_fetched_at TIMESTAMP
next_crawl_at   TIMESTAMP            -- computed from TTL + importance
content_hash    TEXT                 -- MD5 of body bytes
simhash         BIGINT               -- 64-bit near-duplicate fingerprint
storage_key     TEXT                 -- S3 object key for raw HTML
importance      FLOAT                -- crawl priority score [0.0–1.0]
etag            TEXT                 -- from last response, used in conditional GET
fail_count      INT                  -- consecutive failures; triggers backoff

link_edges, the link graph (wide-column store, sharded by source_hash):

-- link_edges (correct Cassandra CQL, composite primary key)
CREATE TABLE link_edges (
  source_hash   TEXT,
  target_url    TEXT,           -- clustering column: range scan of all links from a page
  anchor_text   TEXT,
  discovered_at TIMESTAMP,
  PRIMARY KEY (source_hash, target_url)
);

scheduler_queue, the re-crawl scheduling index (a Redis sorted set, one per domain tier):

-- Redis sorted set: score = next_crawl_at Unix timestamp, member = url_hash
-- Allows efficient: ZRANGEBYSCORE scheduler:tier1 0 {now} LIMIT 0 500
-- The Scheduler reads a batch of due URLs, pushes them to the Frontier (Kafka), marks status=fetching in url_records
-- Separate sets per tier (tier1 = 1h TTL, tier2 = 24h, tier3 = 7–30 days)
ZADD scheduler:tier1 {next_crawl_at_unix} {url_hash}
ZRANGEBYSCORE scheduler:tier1 0 {now_unix} LIMIT 0 500   -- fetch next batch due
ZREM scheduler:tier1 {url_hash}                          -- remove after enqueue to Kafka

content blobs, raw HTML stored in an object store, keyed by url_hash:

-- Object store key pattern
s3://crawl-raw/{yyyy}/{mm}/{dd}/{url_hash}.html.gz

Caching in a web crawler works differently from most systems, the crawler is more consumer than server. Most systems design caches to serve many readers from few writes; this system writes constantly and caches to reduce the cost of its own outbound reads. The latency target is sustained crawl throughput of 11,000 req/s, not user-facing page load time. Every cache miss that falls through to a slower tier directly reduces how many pages per day the system can process.

Invalidation

DNS and robots.txt caches expire on TTL, no active invalidation. When a site updates its robots.txt, the cached version is stale for up to 24 hours; this is a known tradeoff (the alternative is fetching robots.txt on every request). The Bloom filter and URL hash set are write-only, there's no invalidation, by design. If a URL needs to be forcibly re-crawled (e.g. DMCA removal followed by re-addition), operators mark it eligible in Cassandra and the re-crawl Scheduler inserts a fresh entry into the Frontier, bypassing the Dedup Store check.

The single-machine design breaks at three specific points as load increases: the URL frontier becomes a coordination bottleneck when a single queue must enforce per-domain politeness across hundreds of workers; the seen-URL set outgrows available RAM and falls back to disk; and the re-crawl scheduler needs a data structure that can serve time-ordered range queries at 11k QPS, something a wide-column store's partitioning model can't provide. The production architecture addresses each with a targeted change.

Classic interview probes