Design an Online Judge — System Design Interview Guide

An online judge is a deceptively tricky system design question. On the surface it sounds like a CRUD app: accept code, run it, return a result. The hard part is that running arbitrary, untrusted code is one of the most security-sensitive operations in software. Every architectural decision — from the execution environment to the queue topology to the storage layout — flows from that single constraint.

The question also spans a wide range of interesting sub-problems: secure sandboxing, resource fairness, queue-based decoupling, test case management, and leaderboard design. A strong answer demonstrates that you understand which of these is the primary constraint (isolation) and which are consequential (everything else).

Before designing anything, nail down scope. An online judge can mean a competitive programming platform (Codeforces), a hiring assessment tool (HackerRank), or a practice platform (LeetCode). The functional requirements are similar; the non-functional requirements diverge sharply on contest concurrency and plagiarism detection.

Functional requirements

Non-functional requirements

NFR reasoning

An online judge has two very different traffic profiles that must be estimated separately. Submission traffic is write-heavy, bursty, and computationally expensive — each submission triggers code execution that takes 1–5 seconds. Problem-browsing traffic is read-heavy, cheap, and highly cacheable — a user reads a problem statement many times before they submit once.

The most important output of this estimation isn't storage or QPS — it's worker count. Workers are the scarce resource, sized by Little's Law: at steady state, the number of items in the system equals the arrival rate multiplied by the service time (L = λW). This gives the minimum workers at 100% utilisation — but a system running at 100% utilisation has an unboundedly growing queue at any variance spike. Practical provisioning targets 70–80% utilisation, meaning you need the Little's Law result divided by 0.75. Getting this wrong means either under-provisioning (queue builds up, latency blows past 5 s) or over-provisioning (idle workers, wasted cost).

Component breakdown

CDN (content delivery network) serves static assets and — critically — problem statement HTML/JSON. Problem content is read-only after creation, so it can be cached at edge nodes indefinitely with a long TTL. This alone eliminates the majority of API server requests.

Load balancer distributes HTTPS traffic across API server instances. It terminates TLS, applies rate limiting per IP and per API key, and routes health checks. It does not route execution traffic — that goes through the queue.

API servers are stateless application nodes. They validate submissions (syntax check, size limit, language support), write submission records to the database, enqueue execution jobs, and serve read requests from cache. Being stateless means they scale horizontally without coordination.

Job queue (Kafka or SQS) is the critical decoupling point. The API enqueues a submission job and immediately returns an accepted response with a submission ID. The queue provides durability — if all workers crash, jobs are not lost. It also provides natural backpressure during contest spikes.

Execution workers are the production bottleneck. Each worker dequeues one job, boots a fresh sandbox environment, compiles and runs the submission against every test case, and writes the verdict. Workers are stateless; the number of active workers is the primary scaling lever for throughput.

PostgreSQL is the system of record for problems, submissions, users, and test case metadata. Relational schema is appropriate because the entities have well-defined relationships and ACID guarantees simplify submission status consistency.

Redis serves two distinct purposes: submission status polling (short TTL keys, updated by workers as test cases run) and leaderboard rankings (sorted sets, updated atomically per accepted submission).

Object storage (S3) holds test case input/output files. Test case files are immutable after creation, referenced by a deterministic key, and accessed at high frequency by workers. Object storage is cheaper and more scalable than PostgreSQL BLOB columns for this workload.

Architectural rationale

Real-world comparison

Four canonical approaches exist for executing untrusted code, each with a different isolation primitive and a different overhead cost. The choice between them is the most consequential decision in the execution layer:

Our choice for this system: Firecracker microVMs in production, Docker + seccomp in development and CI. Firecracker boots a dedicated Linux kernel per submission — there is no shared kernel surface between submissions or between submissions and the host. AWS Lambda, Fly.io, and Render all use this model for the same reason. gVisor reduces the host kernel attack surface but does not eliminate it: its Sentry process still calls a subset (~50) of real host kernel syscalls to handle I/O and memory, so a host kernel vulnerability in that call set is still reachable.

The performance cost of Firecracker is real but manageable. Pre-warming solves the cold start problem: the worker pool maintains paused Firecracker snapshots per language, each with the language runtime already initialised inside the guest. The 125 ms figure cited in Firecracker's benchmarks is kernel boot only — total cold start including Python or JVM runtime initialisation is 400–700 ms. With a warm snapshot that already has the runtime loaded, restore takes ~10–50 ms (depending on guest memory size), well within the 5 s budget. The warm pool must be maintained per language runtime — a Python snapshot cannot serve a C++ submission. At 210 peak worker slots across 4 languages, a conservative warm pool holds 50–60 slots per popular language, with the remainder allocated on contest demand.

// Resource limits config (per language)
{
  "python3":  { "cpu_ms": 5000, "wall_ms": 8000, "mem_mb": 256 },
  "java":     { "cpu_ms": 3000, "wall_ms": 6000, "mem_mb": 512 },
  "cpp":      { "cpu_ms": 1000, "wall_ms": 3000, "mem_mb": 256 },
  "javascript": { "cpu_ms": 4000, "wall_ms": 7000, "mem_mb": 256 }
}

The API surface for an online judge is small. Three endpoints carry most of the traffic: problem fetch, submission create, and submission status. The interesting design decisions are in validation, the response contract for async execution, and status polling.

POST /api/submissions — create a submission

// Request
POST /api/submissions
Authorization: Bearer <clerk_jwt>
Content-Type: application/json

{
  "problem_id": "two-sum",
  "language":   "python3",          // enum: cpp | java | python3 | javascript
  "source_code": "class Solution:\n    def twoSum(...",
  "contest_id":  "spring-2026"         // optional; null for practice
}

// Response — 202 Accepted (async; execution has not yet started)
{
  "submission_id": "sub_01HX8K...",
  "status":        "QUEUED",
  "poll_url":      "/api/submissions/sub_01HX8K...",
  "queued_at":     "2026-04-30T12:34:56Z"
}

Validation on submission create: source code size limit (64 KB), language must be in the supported set for the problem, rate limit enforced (5 submissions/minute per user per problem), contest window check if contest_id is set. Source code is stored verbatim; no sanitisation: the sandbox handles isolation. An Idempotency-Key request header (client-generated UUID) allows the API to return the existing submission record if the same request is retried within a 10-minute window, preventing duplicate executions from network retries or double-clicks.

GET /api/submissions/:id — poll for result

// Response while executing
{
  "submission_id": "sub_01HX8K...",
  "status":        "RUNNING",        // QUEUED | RUNNING | ACCEPTED | WRONG_ANSWER | TLE | MLE | RE | CE
  "tests_passed":  7,
  "tests_total":   20,
  "runtime_ms":    null,             // populated on completion
  "memory_kb":     null
}

// Response on completion (ACCEPTED)
{
  "submission_id": "sub_01HX8K...",
  "status":        "ACCEPTED",
  "tests_passed":  20,
  "tests_total":   20,
  "runtime_ms":    84,
  "memory_kb":     18432,
  "percentile_runtime": 92.4      // faster than 92.4% of accepted solutions
}

The dominant write path — user submits code, gets a verdict — is where the latency NFR (§2) and the isolation NFR collide. A synchronous execution model satisfies neither: it blocks connections during the 1–5 s execution window and makes isolation harder by coupling the API process to the sandbox. The queue-based design satisfies both at the cost of one tradeoff: the client must poll or subscribe for results rather than waiting on a synchronous response.

The critical tradeoff in this flow is between polling and WebSocket push for step ⑦. Polling is simpler: the client calls GET /submissions/:id every 500 ms; the API reads from Redis; no server-side connection state. WebSocket push is lower latency: the worker publishes a completion event to a Redis Pub/Sub channel, and the API server subscribed on behalf of the client's WebSocket connection forwards it immediately.

The polling design is strongly preferred at L3–L5. At L5–L6, the interviewer may probe whether you know that WebSocket connections require sticky routing (or Redis Pub/Sub fan-out) to work correctly behind a load balancer. Name that constraint and it becomes a strength, not a weakness.

// problem config (stored in PostgreSQL problems table)
{
  "comparator_type": "exact",        // exact | float_eps | special_judge
  "float_epsilon":   1e-6,            // used when comparator_type = float_eps
  "special_judge_s3_key": null        // path to comparator binary in S3
}

Before defining any columns, identify the entities and how they get accessed. The schema follows the access patterns, not the other way around.

Entities and access patterns

Two things stand out from this table. First, test cases are accessed in bulk by problem — all cases for a problem are fetched together by the worker, never individually. This means the test case files belong in object storage (S3) with a key pattern of problems/{problem_id}/cases/{case_id}.{in|out}, while metadata (case count, time limit per case) stays in PostgreSQL. Second, submissions are written at high volume but queried in two very different ways: point-lookup by ID (status polling) and range scan by user (submission history). Both access patterns need covering indexes.

Caching in an online judge operates at three distinct layers, each serving a different read pattern. The key insight is that these layers have completely different invalidation requirements: problem content almost never changes (long TTL), submission status changes frequently during execution (short TTL), and leaderboards change on every accepted verdict (near-real-time).

Scaling an online judge beyond a few thousand daily active users requires addressing three distinct bottlenecks: the execution worker pool (the primary constraint), the submission queue under contest burst load, and the database under high-write submission traffic. Each requires a different approach.

An online judge has several failure modes that don't exist in typical web services, because the execution layer introduces an entirely new class of failure: the sandboxed process. Understanding how each failure propagates — and where it's caught — is what separates an L5 answer from an L3 answer.

Three security concerns are specific to online judges and map directly to components already in the architecture: abuse of the submission pipeline, the problem setter trust model (who can create test cases and how are they validated?), and data integrity under adversarial conditions. L6+ interviewers probe all three explicitly.

Submission rate limiting and abuse prevention

Without rate limiting, a single malicious actor can flood the execution queue with thousands of submissions, consuming worker capacity and degrading fairness for other users. Rate limiting is enforced at two levels:

Classic probes — level-differentiated answers