Design a Chat System
like WhatsApp

500ms is the rough threshold below which a conversation feels live. Above it, users notice the lag and conversations feel sluggish. The p99 (not average) matters because a consistently fast average with a long tail still breaks the experience for 1 in 100 messages.

The ACK boundary matters enormously. If the server acknowledges receipt to the sender's device before persisting the message, a server crash between those two moments causes silent data loss, the sender thinks the message was sent, but it's gone. The server must persist durably before sending the ACK. Note that "no message loss after server ACK" is a steady-state guarantee: a simultaneous failure of two or more Cassandra replicas within the write commit window is a known, rare failure mode rather than a design shortcut to accept.

Strict global ordering across all conversations is impossibly expensive at this scale (it would require a distributed consensus round for every message). Causal ordering within a single conversation, where if message A happens before B in the conversation, every device sees A before B, is sufficient and much cheaper to achieve.

WebSocket connections require the server to maintain open file descriptors for the lifetime of the connection, sometimes hours. A stateless request-response model (HTTP) would require the client to poll every few hundred milliseconds, which at 500M DAU creates an enormous polling QPS and introduces the latency of a full polling cycle for every message.

Without a message queue, the chat server would write synchronously to Cassandra and then attempt delivery. If the chat server crashes after writing but before delivery, the recipient never gets the message (or gets it only after fetching history). Once a message is in Kafka, it will reach a consumer at least once, even if a chat server crashes partway through processing. The client_msg_id idempotency key at the application layer converts that at-least-once guarantee into exactly-once delivery at the recipient's device.

Messages have a very clear access pattern: you almost always read a conversation's recent messages in order. This maps perfectly to Cassandra's wide-column model: partition key = conversation_id, clustering key = message_id (time-ordered). A range scan retrieves the last N messages in a single sequential read. Cassandra also handles high write throughput naturally, its LSM-tree storage engine is write-optimized.

WebSocket connections are long-lived, so authentication cannot rely on per-request session cookies the way HTTP does. Sending a token as a URL query parameter (?token=...) is a common but dangerous pattern, query parameters appear in nginx access logs, CDN edge logs, and browser history, making the token semi-public. The correct production pattern is the first-message auth handshake: the WebSocket connection is established without credentials, and the client sends its JWT as the very first message frame before any application traffic. The server holds the connection in an authenticating state for up to 5 seconds; if no valid auth frame arrives, it closes the connection. This keeps tokens out of all log surfaces.

The JWT itself should be short-lived (15-minute expiry). Before it expires, the client fetches a new JWT via a REST call and sends it over the open WebSocket as a token.refresh frame, no reconnect needed. The server validates and updates the session's expiry in-place.

Without rate limiting, a compromised account or bot can flood conversations with thousands of messages per second, degrading the system for everyone. Enforce a per-user message rate limit at the chat server using a token bucket stored in an in-memory store (Redis): each user has a bucket with a capacity of N messages per minute (e.g. 60), refilled at 1/second. If the bucket is empty, the server responds with a rate_limited WebSocket frame and does not persist or forward the message.

Pre-signed S3 URLs allow clients to upload directly to object storage, which means the server must control what it signs for. Two mitigations: (1) enforce a max file size on the pre-signed URL (S3 supports content-length-range in the policy), so a client cannot upload a 10 GB file using a URL issued for a 25 MB limit; (2) limit pre-signed URL issuance per user per hour to prevent a bot from generating thousands of upload slots. Both checks happen in the REST handler before issuing the URL, and cost nothing in S3 bandwidth.

A single status column on the messages table fails for two reasons.

First, in a group of 500 members, "delivered" is per-recipient, one column can't represent 500 independent delivery states. You'd need a row per recipient anyway, which is exactly what message_receipts provides.

Second, updating a non-clustering column in Cassandra generates a tombstone on every cell write (Cassandra's MVCC model). At 100B messages/day with delivery + read events per message, the tombstone accumulation would continuously slow compaction. The receipt table avoids this entirely: each delivery event is an upsert of a new row keyed by recipient, no mutations, no tombstones.

The inbox is a list of conversations sorted by most-recent-message time. Every time a new message arrives in any conversation, that conversation needs to move to the top of the list.

In Cassandra, sort order is baked into the clustering key, you can't update it in place. Moving a conversation up the list means deleting the old row and inserting a new one with the updated timestamp. At 100B messages/day this is a continuous delete + insert churn, generating a tombstone on every operation. Tombstones don't get cleaned up immediately, they accumulate until compaction runs, and a heavy tombstone load degrades read performance across the partition.

A Redis sorted set handles this natively. ZADD inbox:{user_id} {ts} {conv_id} overwrites the score atomically if the key already exists, no delete, no tombstone. ZREVRANGE returns the top N conversations by recency in O(log N). The sorted list lives in Redis; per-conversation metadata and the read cursor (last_read_msg_id) stay in Cassandra where they belong.

At 500M DAU with 10% concurrently online, that's 50M active WebSocket connections. A single server can hold 50k–100k connections (limited by file descriptor limits and memory), so you need 500–1,000 chat servers minimum, and many more for redundancy and geographic distribution.

Scaling out is straightforward, add more chat servers and update the load balancer. The complexity is scaling in (or rolling deploys). You can't kill a chat server without disrupting all its active connections. The solution is graceful draining: the server stops accepting new connections, lets existing ones time out or explicitly closes them, and the clients automatically reconnect to surviving servers. This means your load balancer must support connection draining, and your client must handle reconnect with exponential backoff.

When Alice sends a message to a group of 100 people, the system must deliver it to 100 recipients. There are two fundamental strategies.

Fan-out on write: when the message is sent, immediately write a copy to each member's inbox (or enqueue a delivery event per member). Each member's delivery is handled independently. This is simple and keeps the read path fast (each member just reads their own inbox), but it's expensive for large groups, one message creates N writes and N delivery events.

Fan-out on read: store the message once, and let each member's client fetch it when they open the group conversation. Much cheaper on write, but requires a shared cursor per member (which messages have I seen?) and a more complex read path. Also means group members who open the conversation simultaneously all hit the same message record simultaneously, hot partition risk in Cassandra.

WhatsApp supports multiple linked devices (phone + web + desktop). Each device needs its own delivery state, a message delivered to your phone shouldn't automatically count as delivered to your laptop. The routing abstraction therefore operates at device granularity, not user granularity: each device registers its own session key (session:device:{device_id}) pointing to its chat server, and a companion set (user:devices:{user_id}) lists all of a user's currently active device IDs. This is the design already reflected in the §7 Redis schema.

When a message arrives for Bob, the system reads SMEMBERS user:devices:{bob_id} to get all his active device IDs, then for each device looks up its server address and routes the message there. Each device independently sends a delivery ACK. The "two-tick" indicator on the sender's side lights up when at least one of the recipient's devices acknowledges delivery. The "blue tick" lights up when any device marks it read, recorded as a new row in the message_receipts table (§7).

For offline sync: when a device comes online after being offline, it sends its last-seen message sequence number (last_read_msg_id from the user_conversations table) to the server. The server fetches all messages in the user's conversations with msg_id > last_read_msg_id from Cassandra and pushes them to the reconnecting device. This is the GET /messages/sync?since={seq} endpoint from §5b.

Cassandra partitions by conversation_id. A conversation with many active members, say, a group of 500 people who are all actively messaging, generates enormous write and read traffic to a single partition on a single Cassandra node. This is the "hot partition" problem: most traffic concentrates on one node, defeating the point of distributing across a cluster.

For most conversations this isn't a problem, traffic is distributed across millions of conversation_ids. But for extremely popular group conversations (100k+ member broadcast channels, which are a different beast from WhatsApp groups), you need a different partition strategy.

A single-region deployment adds hundreds of milliseconds of latency for users far from the data center, and a regional outage takes down all users globally. The L7/L8 answer requires multi-region deployment.

The key challenge: message ordering across regions. If Alice is in US-East and Bob is in EU-West, and both send messages to the same group conversation simultaneously, which one appears first? Without coordination, each region might apply them in a different order, and users see different conversation histories.

Solutions involve either routing all writes for a conversation to a single "home region" (the conversation leader), which adds cross-region latency for one party, or using a conflict-free replicated data type (CRDT) or vector clock scheme that allows concurrent writes and merges them deterministically. WhatsApp routes connections to the nearest region but uses a globally replicated database (built on top of custom Erlang infrastructure) to resolve ordering.

When a new user installs WhatsApp, the app identifies which of their phone contacts are already registered, without revealing to the server which contacts are not registered. Revealing non-matches would expose the user's full address book, including people who chose not to sign up.

The mechanism: the client hashes all contact phone numbers (e.g. SHA-256 of the E.164 normalised number) and sends only the hashes. The server checks each hash against a hash index of all registered phone numbers, and returns only the matching hashes. The client maps those back to names from its own contacts. The server never sees plaintext phone numbers for non-users; it only confirms existence for those in its own registry.

At scale, the hash lookup must be fast, a bloom filter pre-screens obvious misses before hitting the full index. Interviewers at L6+ sometimes ask this as a standalone question ("how do you implement contact discovery while preserving privacy?") so knowing the hash + bloom filter pattern is valuable on its own.

WhatsApp uses the Signal Protocol for E2EE. The fundamental property: the server can see that a message was sent from A to B, but cannot read its contents. The server holds encrypted ciphertext only.

This changes the server-side architecture significantly. The message body stored in Cassandra is ciphertext. The server cannot index, moderate, or search message content. The fan-out for group messages must account for encrypting the message for each recipient's device key separately, the sender's client does this, generating N encrypted payloads for N recipient devices. The server delivers the right ciphertext to the right device without understanding any of it.

The critical infrastructure piece is the key distribution server: each device registers its public key (identity key, signed prekeys, one-time prekeys) with the server. When a sender wants to message a new recipient for the first time, they fetch the recipient's prekeys from the server and establish a shared secret. The server never sees the shared secret, it only distributes public keys.