Design a File Storage System

A file storage and sync system — Dropbox, Google Drive — looks like a managed folder. Underneath it spans chunked uploads, content-addressed deduplication, delta sync, offline-first conflict resolution, and a metadata layer that must track billions of file versions with sub-second query times.

Interviewers use this question to probe whether you think about data at rest (block storage, dedup) separately from data in motion (the sync protocol), and whether you understand how those two planes interact under failure conditions. Three tensions drive the core tradeoffs: bandwidth efficiency vs. simplicity, strong consistency vs. availability during partitions, and storage cost vs. query flexibility.

Two designs hide under this question: a consumer sync tool (Dropbox-like, mobile + desktop, strong sync) and an enterprise document platform (Google Drive-like, collaborative editing, sharing, search). The scope changes which problems are hard. Confirm scale (DAU, average file size, total storage) before assuming either.

Functional requirements

Non-functional requirements

NFR reasoning

File storage is a storage-bound problem, not a compute-bound one. Three numbers drive the cost model: total raw storage accumulated over time, the deduplication ratio that compresses it, and peak inbound bandwidth during business hours.

A file storage system has two fundamentally different data planes: the metadata plane, which tracks what files exist, who owns them, and what version each device has seen; and the block plane, which stores the actual file content as immutable, content-addressed chunks. Keeping these planes separate is what makes global deduplication and delta sync possible: metadata can be strongly consistent without imposing that cost on the block store, and blocks can be immutable and content-addressed without requiring the metadata layer to share those constraints.

The system splits into four functional clusters: a sync API tier that clients talk to, a metadata service backed by a sharded relational database, a block service backed by a content-addressable blob store, and a notification bus that pushes change events to subscribed client devices.

Component breakdown

CDN (content delivery network): caches file chunks at edge nodes globally. File reads are by far the highest-volume operations. Serving them from the nearest edge node reduces latency from hundreds of milliseconds (origin round-trip) to single-digit milliseconds for most users. The CDN stores chunks by their content hash: the same hash always resolves to the same bytes, so cached chunks never require invalidation.

API Gateway + Sync API servers: the stateless tier that authenticates requests, enforces rate limits, and routes to the correct downstream service. The Sync API is split from the download path: clients GET files via CDN-signed URLs, they PUT chunks directly to the Block Service's upload endpoints. The Sync API orchestrates the metadata transaction that records each completed upload.

Metadata Service: tracks every file record, folder tree, sharing permission, and version history entry. This is the most query-intensive service; every sync operation begins here to discover what changed. It runs on a sharded relational database — MySQL via Vitess (the sharding layer Dropbox itself used), or Google Cloud Spanner for globally replicated consistency — partitioned by user ID so that all queries for a given user's file tree hit the same shard. The relational model is chosen over document or wide-column stores because uploading a file must atomically create a file_versions row and update files.current_ver — a multi-row ACID transaction that eventually-consistent document stores cannot provide without application-level compensating logic.

Block Service: receives raw chunk uploads, computes SHA-256 fingerprints, checks the global dedup table, and either confirms the chunk already exists or writes it to the Chunk Store. This service is stateless and owns no data: all state lives in the hash lookup table and the Chunk Store.

Chunk Store: a durable, content-addressed object store (a managed object storage service like S3 or GCS) where each chunk is keyed by its SHA-256 hash. Objects are immutable once written. Because chunks are identified by content hash, two identical files uploaded by different users share the same physical chunk objects — deduplication is free at the storage layer.

Notification Bus: when a file upload completes, the Sync API publishes a change event to a durable message queue (such as Apache Kafka). A notification service fans out that event to all other devices for the same account connected via WebSocket or HTTP long-poll. Devices receive the event and know to pull the updated file manifest from the Metadata Service.

Cache: an in-memory store (Redis) sitting in front of the Metadata Service for hot read paths — active sessions, recently accessed file metadata, sharing permission checks. Because the Metadata Service owns all writes, it can push invalidation events after each commit without coordinating with other writers — there is no distributed invalidation problem.

Architectural rationale

Real-world comparison

The central algorithmic question in this design is: how do you split a file into chunks such that the same logical content produces the same chunks, across different file sizes and edit positions? The answer determines how effective delta sync and cross-user deduplication are in practice.

Our choice for this system is content-defined chunking (CDC) using a Rabin fingerprint sliding window. The algorithm slides a 48-byte window over the file's bytes and cuts a chunk boundary whenever the 32-bit rolling hash matches a target pattern. As a concrete example: requiring the lowest 13 bits to be zero yields an expected chunk size of 2^13 = 8 KB. For a 4 MB target, require the lowest 22 bits to match (2^22 ≈ 4 MB). Minimum and maximum chunk size bounds prevent degenerate tiny or huge chunks.

Deduplication via content addressing

After chunking, each chunk is identified by its SHA-256 fingerprint. Before uploading, the client sends the Block Service a list of chunk hashes. The Block Service checks each hash against the global chunk index. The Block Service skips any chunk whose hash is already in the index and notifies the client of which chunks to omit. Only the remaining chunks are uploaded. This achieves both intra-user dedup (re-uploading a file you already have) and inter-user dedup (uploading a file that someone else already uploaded).

The public API has two core operations: initiating a file upload (which returns upload URLs for missing chunks) and triggering a sync to discover what changed. Everything else — version restore, sharing, folder management — is important but does not change the core architecture.

POST /files/upload/init — initiate upload

The API gateway enforces a per-user token bucket on this endpoint: max 100 upload-init requests per minute, 10 concurrent upload sessions. Clients exceeding the limit receive 429 Too Many Requests with a Retry-After header.

// Request: client sends file metadata + list of chunk hashes
POST /files/upload/init
Authorization: Bearer <token>
Content-Type: application/json

{
  "path":        "/My Documents/report.pdf",
  "size_bytes":   52428800,           // 50 MB
  "modified_at":  "2026-04-10T14:32:00Z",
  "content_hash": "sha256:aef392...",  // whole-file hash
  "chunks": [
    { "index": 0, "hash": "sha256:c1a...", "size": 4194304 },
    { "index": 1, "hash": "sha256:d2b...", "size": 4194304 },
    { "index": 2, "hash": "sha256:e3c...", "size": 1048576 }
  ]
}

// Response: 200 OK — server identifies which chunks are missing
{
  "upload_session_id": "sess_abc123",
  "missing_chunks": [1, 2],           // chunk 0 already exists
  "upload_urls": {
    "1": "https://storage.example.com/chunks/d2b...?sig=...",
    "2": "https://storage.example.com/chunks/e3c...?sig=..."
  }
}

// Validation errors
400 Bad Request: chunk count > 10,000 or file > 5 GB
409 Conflict: path exists with same content_hash (client dedups early)\

POST /files/upload/complete — commit upload

// Request: after uploading all missing chunks, commit the transaction
POST /files/upload/complete
Content-Type: application/json

{
  "upload_session_id": "sess_abc123"
}

// Response: 201 Created
{
  "file_id":    "file_xyz789",
  "version_id": "ver_001",
  "url":        "https://cdn.example.com/files/file_xyz789?v=ver_001&sig=..."
}

// Error responses
404 Not Found: upload_session_id expired or unknown
422 Unprocessable: one or more missing chunks not yet uploaded

GET /sync/delta — pull changes since a cursor

// Request: client provides its last sync cursor
GET /sync/delta?cursor=eyJhY2N0IjoiMTIzIiwiY3Vyc29yIjoiMTcxMjcwMDAwMCJ9

// Response: list of changes since that cursor
{
  "changes": [
    {
      "type":       "modified",
      "file_id":    "file_xyz789",
      "path":       "/My Documents/report.pdf",
      "version_id": "ver_002",
      "size_bytes": 52457600,
      "chunks": [
        { "index": 0, "hash": "sha256:c1a..." },  // unchanged
        { "index": 1, "hash": "sha256:f4d..." },  // new chunk
        { "index": 2, "hash": "sha256:e3c..." }   // unchanged
      ]
    }
  ],
  "new_cursor": "eyJhY2N0IjoiMTIzIiwiY3Vyc29yIjoiMTcxMjcxMDAwMCJ9",
  "has_more":   false
}

Optional endpoints by level

POST /files/{id}/share — sharing endpoint

// Request: specify permission level, optional grantee, optional expiry
POST /files/{id}/share
Authorization: Bearer <token>
Content-Type: application/json

{
  "permission":    "view" | "edit",        // access level granted
  "grantee_id":   "usr_abc123",           // omit for a public link
  "expiry_seconds": 604800                // optional; 0 = no expiry
}

// Response
HTTP 201 Created
{
  "link_token":  "tok_xK9mPq2r",
  "share_url":   "https://app.example.com/s/tok_xK9mPq2r",
  "expires_at":  "2026-04-18T12:00:00Z", // null if no expiry
  "permission":  "view"
}

The link_token is stored in the file_shares table (§8). On access, the API verifies the token is not expired, increments an access counter for audit, and returns a presigned download URL. For grantee_id shares (direct user grants), the notification fan-out (§10) adds the grantee to the affected file's change events so they receive real-time sync updates for the shared file.

Search design note

File-name search can be served without a dedicated search index: a WHERE user_id = ? AND name LIKE 'prefix%' query on a (user_id, name) composite index in the metadata DB handles autocomplete efficiently at L3/L4. Full-text content search — searching inside document bodies — requires an async pipeline: upload completion triggers an event → text-extraction worker pulls the chunk bytes, extracts plaintext, and pushes to Elasticsearch → the index maps terms to file IDs. Clients query Elasticsearch directly via the API gateway. This pipeline is an L7/L8 concern; state the distinction clearly when the interviewer asks about GET /files/search.

The durability NFR (11 nines) and the upload bandwidth NFR (delta-only) from §2 drive every step in the upload flow, along with the quota check added in §8. The key question is: at what point does the server commit a file version, and what happens on device B when device A completes an upload?

Client-side early exit: before calling POST /upload/init, the client compares the whole-file content hash against its local version database. If the hash matches the last synced version, the upload is skipped entirely — zero network calls. This is the first and cheapest deduplication check, operating entirely on-device.

Resumable uploads for large files

For files larger than ~100 MB, network interruptions mid-upload are likely. The system implements the TUS resumable upload protocol (or an equivalent): the client first issues an upload initialization request, then uploads chunks in sequence. Each chunk upload is acknowledged. If the connection drops, the client queries the server for which offset was last committed and resumes from there. This directly addresses the upload bandwidth NFR from §2 — a 2 GB file with a dropped connection at 1.9 GB only re-uploads the last chunk, not the whole file.

Async thumbnail & preview generation

Thumbnail generation is a background pipeline, not a synchronous upload step. File browsers display 64×64 icons and hover previews independently of the full file — downloading the complete file just to render the list view would be impractical.

When the Sync API publishes the upload-complete event to the Notification Bus (step ⑨ above), a Thumbnail Worker subscribes as a second consumer on that same topic. For each image or document file, the worker:

1. Downloads the first chunk (or the full file for images) from the Block Store using the version's chunk list.
2. Generates two thumbnail variants: 64×64 px (file-list icon) and 256×256 px (hover preview).
3. Writes both variants to object storage under deterministic keys: thumbs/{file_id}/{version_id}/64.webp and thumbs/{file_id}/{version_id}/256.webp.
4. Updates the file_versions row with thumb_url_64 and thumb_url_256 columns, which the metadata API then returns to clients.

Because the thumbnail worker is an async consumer, it does not block the upload acknowledgement: the client gets its file_id + version_id immediately, and thumbnails appear in the UI once the worker finishes (typically within 1–3 seconds for images). For file types without a thumbnail (e.g., .zip), the worker is a no-op and the client falls back to a generic file-type icon.

§7.5 — Offline access: the local sync daemon L5+

The "offline access" functional requirement (§2) is satisfied by a persistent background daemon on the client device, not by the server. The daemon has three responsibilities: watching for local file changes, maintaining a local state database, and flushing queued work on reconnect.

1. File system watcher: the daemon registers with the OS-native change API — FSEvents on macOS, inotify on Linux, ReadDirectoryChangesW on Windows. When a watched file changes, the watcher fires an event within milliseconds. The daemon debounces rapid successive changes (e.g., an editor saving every keystroke) with a short idle delay (~500 ms) before treating the file as stable.
2. Local state database: a lightweight SQLite database tracks the last-synced version_id, content hash, and sync status (synced | pending_upload | pending_download) for every file in the sync folder. This is the ground truth for what the device believes is current. On app startup or reconnect, the daemon compares the local state DB against the server's GET /sync/delta response to identify any divergence.
3. Reconnect flush: when the device comes back online, the daemon walks its local state DB for any rows with pending_upload status — these are edits made while offline. Each is processed through the normal upload flow (§7). Offline edits discovered on reconnect are treated identically to concurrent device edits: the parent_version_id check at commit time detects conflicts, and the conflict-copy mechanism in §7 handles them — no special offline-conflict code path is needed.

Selective sync (marking only certain folders for offline access) works by limiting which paths the daemon watches and which entries it downloads to the local state DB. Files not marked for offline access are stubs: their metadata is local but their chunks are fetched on-demand when opened.

There are four categories of entities in this system: users + accounts, file metadata, chunk index, and sharing + permissions. Each has different access patterns, and those differences determine where each entity lives and how it is indexed.

Access patterns

All user-facing queries begin with a user_id filter, making user_id the natural sharding key: all records for a given user co-locate on one shard. The chunk hash lookup has no user context at all, which makes it a candidate for a separate, independently-scalable hash index.

Schema

Quota tracking

Storage quotas require two additions to the schema. A user_quotas table (co-located on the same shard as the user's files via user_id) tracks the current and limit values:

-- co-located with files shard via user_id
user_quotas (
  user_id        UUID PK  -- shard key,
  quota_limit_bytes  BIGINT   NOT NULL  -- e.g. 15 GB for free tier,
  quota_used_bytes   BIGINT   NOT NULL  DEFAULT 0,
  updated_at     TIMESTAMPTZ
)

quota_used_bytes is incremented atomically when a file version is committed and decremented on soft-delete. The Sync API reads this row at upload-init time and rejects the request with 507 Insufficient Storage if the upload would exceed the limit (see §7). The files table additionally carries deleted_at TIMESTAMPTZ and purge_after TIMESTAMPTZ columns to support the trash/GC pipeline: a daily background job hard-deletes rows where purge_after < NOW(), decrements quota_used_bytes, and decrements chunk_index.ref_count for orphaned chunks.

Version retention and GC lifecycle

The §2 NFR specifies a minimum 30-day version history. On each version creation, purge_after is set to created_at + retention_days (30 days for free tier, configurable per plan). The daily GC job runs DELETE FROM file_versions WHERE purge_after < NOW() and, for each removed version, atomically decrements chunk_index.ref_count for each chunk in the version's chunk list: UPDATE chunk_index SET ref_count = ref_count - 1 WHERE chunk_hash = ?. If the updated ref_count reaches 0, the GC then deletes the blob from the object store by its hash key (an S3/GCS DeleteObject call), and finally removes the chunk_index row. The object store has no ref_count column — it is a content-addressed blob store; all reference tracking lives in chunk_index. This two-step sequence (decrement in index → delete from object store if zero → delete index row) prevents races where a concurrent upload re-references a chunk between the decrement and the delete. This is distinct from the orphaned-chunk GC path (§11) which handles failed uploads — version-expiry GC is the planned lifecycle path for fully committed versions whose retention window has closed.

The caching architecture follows from two NFRs: metadata reads must complete in under 200 ms (§2), and file downloads must start within 2 seconds from the nearest edge (§2). These are satisfied by different cache layers operating at different points in the request path established in §4.

Cache invalidation

Chunk objects are immutable by design (content-addressed), so the CDN never needs invalidation for chunks: the same hash always returns the same bytes. Every metadata write triggers a cache invalidation event: after each successful commit, the Sync API publishes to a durable Kafka topic (the same cluster already in the design), and consumers clear the relevant Redis keys. Using a durable topic rather than Redis pub/sub ensures that a temporarily disconnected consumer does not permanently miss invalidation events. Redis pub/sub is fire-and-forget: it delivers nothing to subscribers that restart or lag. This keeps cache coherence bounded to the event propagation latency, typically under 100 ms within a region.

At 50M+ DAU the system faces four distinct scaling challenges: the chunk hash lookup table becomes a hot global bottleneck; metadata shards become unevenly loaded for power users; the sync notification layer must hold millions of persistent connections; and cross-region replication introduces consistency trade-offs.

The file storage question is calibrated by how deep you go on the sync protocol, chunking algorithm, and consistency model. An L3 answer covers the mechanics; an L7 answer owns the cost structure and consistency trade-offs.

files (
  file_id    UUID PK,
  user_id    UUID,
  path       TEXT,
  s3_url     TEXT,
  version    INT,
  updated_at TIMESTAMPTZ
)

Classic probes table