Design a Notification Service

The notification service prompt is deceptively broad. "Send notifications" sounds like a simple pub-sub problem. What makes it hard is the combination of heterogeneous delivery channels (push, email, SMS, in-app), wildly different latency tolerances across notification types, and the need to avoid both silent failures and duplicate deliveries at the same time.

The question is a proxy for four distinct skills: understanding how fan-out complexity grows with follower counts; reasoning about delivery guarantees and idempotency; designing for channel-specific rate limits imposed by third-party providers; and prioritising notification types so a marketing blast doesn't delay a password reset.

Notification systems span a huge design space. The first thing to do is nail down which of the four channel types are in scope, what notification categories exist, and whether "delivery" means at-most-once, at-least-once, or exactly-once. The answers change every major component.

Functional requirements

Before the NFRs, it's worth understanding how push delivery actually works, it shapes several of the constraints below.

Non-functional requirements

Notification systems have an unusual write profile: write volume (notification sends) can be many orders of magnitude larger than the number of triggering events, because a single social event (a post from a celebrity with 10M followers) fans out to millions of push notifications. Storage is dominated by the delivery log, not the notification metadata itself.

Component breakdown

Notification API is the single ingress point for all notification events. It validates the payload (notification type, target users, template ID), enforces per-caller rate limits, enriches the event with metadata (sending timestamp, idempotency key), and publishes to the appropriate priority queue topic. It does not fan out, that is the worker's job.

Priority queues are the architectural keystone. Three separate topics in a durable log (Apache Kafka or AWS SQS): transactional, social, and marketing. Each has independent consumer groups and independent scaling policies. This isolation ensures a 50M-message marketing blast cannot starve an OTP delivery, which directly satisfies the latency NFR from §2.

Fan-out workers consume from the priority queues. For each event, they look up the target user list, check preferences and quiet hours, select the right channel(s), and enqueue one send task per recipient per channel into the downstream dispatcher queue. They also write the initial record to the notification log.

Channel dispatchers are thin, stateless workers that handle one channel each: push (APNs + FCM), email (SendGrid or SES), and SMS (Twilio or Vonage). They implement channel-specific retry logic, third-party rate limit handling (respecting 429 responses), and payload formatting. Keeping them separated means a third-party provider outage only affects one channel, not the others.

Preferences DB stores per-user opt-in/out state, quiet hours, preferred channel order, and per-category preferences. It is a relational store (PostgreSQL) with a read-through distributed cache (Redis) in front of it, since preferences are read on every single notification send but change rarely.

Notification log is backed by a wide-column store (Apache Cassandra) split into two tables: notification_log is partitioned by user_id and clustered by sent_at, optimised for the inbox read ("give me all notifications for user X, newest first"). delivery_attempt is partitioned by notification_id and clustered by attempt_at, optimised for retry and support lookups ("what happened to notification Y?"). Splitting them lets each table's partition key match its dominant query, the core Cassandra data-modelling principle.

Architectural rationale

Real-world comparison

Fan-out is the defining algorithmic question for notification systems. When a user posts something, the system must decide: do we push that notification into each follower's inbox immediately (fan-out on write), or do we compute the notification list when each follower asks (fan-out on read)? Both have significant consequences across the entire design.

The right answer for production systems is a hybrid approach: fan-out on write for users with follower counts below a threshold (e.g. 10,000), and fan-out on read for high-follower accounts. The threshold is a configuration value, not a constant. Twitter's "celebrity problem" made this pattern famous, and it remains the standard solution.

async def fan_out(event: NotificationEvent):
    sender = get_user(event.sender_id)

    if sender.follower_count <= FANOUT_WRITE_THRESHOLD:
        # Fan-out on write: push to each follower's inbox
        follower_ids = get_follower_ids(event.sender_id)
        await batch_write_inbox(follower_ids, event)
    else:
        # Fan-out on read: store one record, resolve at query time
        await write_single_event(event)
        # Followers will see this via graph traversal on inbox load

    await log_delivery_attempt(event.notification_id, "fan_out_complete")

POST /v1/notifications

Triggers a notification. Returns immediately after enqueueing, does not wait for delivery.

// 400  invalid template_id or type mismatch
// 409  idempotency_key already used within 24 h (duplicate blocked)
// 429  rate limit exceeded, Retry-After header included

GET /v1/users/{user_id}/notifications

Returns the in-app inbox for a user, newest first. Cursor-paginated.

// ?limit=20           max items per page (default 20, max 100)
// ?cursor=<token>     opaque cursor from previous response
// ?unread_only=true   filter to unread notifications only

// ?limit=20           max items per page (default 20, max 100)
// ?cursor=<token>     opaque cursor from previous response
// ?unread_only=true   filter to unread notifications only

Additional endpoints

The dominant path in a notification system is the write path: an event arrives, fans out to recipients, checks their preferences, and dispatches per channel. The key architectural decision is where within the fan-out step to check preferences, in the fan-out worker before enqueuing per-channel tasks (saves provider API calls for opted-out users, but requires preference data in the worker) or in the dispatcher after enqueuing (cleaner separation but wastes queue capacity on sends that will be suppressed).

The preference check happens before channel dispatch, not after. This is the key decision point, driven directly by the NFR to respect quiet hours and opt-outs. Checking preferences in the fan-out worker (step 4) rather than the dispatcher (step 5) means we avoid making any provider API calls for opted-out users, which saves money and avoids consuming rate limit budget. The cost is that the fan-out worker must read preference data, which is why the preference cache (Redis, §8) is critical.

The notification system has three distinct entities with very different access patterns and storage characteristics. Understanding how each entity is queried determines every schema decision.

Access patterns

Two things stand out: preference reads happen on every single send and must be fast (driving the Redis cache), and the delivery log is almost pure-append with user-partitioned reads (driving the Cassandra choice).

device_tokens, PostgreSQL

Device tokens are the source-of-truth for push delivery. They are relational (one user can have many devices, each with one token per platform), low-volume compared to the notification log, and must survive Redis eviction. They live in PostgreSQL alongside preferences, with the Redis cache (§8) providing the 1-hour TTL read layer.

The notification service's caching needs fall into two tiers. The essential tier covers every single send regardless of channel, both caches are hit on every notification that goes through the fan-out worker. The extended tier covers push-specific details that only become relevant once you go deeper on channel dispatch.

Essential, hit on every send

Extended, push-specific L5+

Cache invalidation

When a user updates their preferences via the API, the system writes to PostgreSQL and immediately invalidates the Redis key (write-invalidate, not write-through). Write-invalidate is preferred here because the next read will always populate the cache with fresh data, and double-writes (to DB and cache) risk inconsistency if one fails. The 5-minute TTL provides a backstop for any invalidation misses.

At L5 and above, the interviewer expects you to go beyond the happy path and address the stress cases: what happens during a marketing blast, a celebrity event, or a third-party provider outage? The accordions below address the most common follow-up questions at each level, check the tag on each to see where it's expected.

Classic probes