System design interview: design a distributed job scheduler

1. Clarify the requirements

"Design a job scheduler" covers anything from a cron replacement to a workflow engine. Agree on the job types and the guarantee before anything else.

• Functional: schedule a one-off job to run at a given time ("send this reminder at 09:00 tomorrow"); schedule recurring jobs with a cron expression and a time zone; cancel, pause and resume jobs; retry failed runs; show the status and history of each job.
• Non-functional: an accepted job is never lost; a due job starts within a few seconds of its time; the system keeps running when any single node fails; one noisy tenant cannot starve the others.
• Out of scope unless asked: dependencies between jobs (a DAG or workflow engine), the business logic inside each job, sub-second precision.

2. Estimate the scale

These are assumed round numbers for a shared internal platform, chosen to make the arithmetic easy.

Two conclusions: storage is small, so the hard part is coordination, not volume; and the top-of-the-hour spike, not the average, sets the worker count. Smoothing that spike is worth a sentence later.

3. Define the API

POST /api/v1/jobs
  body:    { "type": "send-invoice", "payload": { "invoiceId": 8812 },
             "runAt": "2026-11-02T09:00:00Z", "idempotencyKey": "invoice-8812" }
     or:   { "type": "daily-digest", "cron": "0 9 * * MON-FRI",
             "timeZone": "Europe/London", "missedRuns": "run-once", "maxAttempts": 8 }
  returns: 201 { "jobId": "j_7hq2", "nextRunAt": "2026-11-02T09:00:00Z" }

GET    /api/v1/jobs/{jobId}           status, next run time, recent runs
DELETE /api/v1/jobs/{jobId}           cancel
POST   /api/v1/jobs/{jobId}/pause     and /resume

Internal, worker to scheduler:
POST /internal/leases                 { "workerId": "w-311", "types": ["send-invoice"], "max": 10 }
  returns: [{ "runId": "r_90", "runKey": "j_7hq2@2026-11-02T09:00Z", "attempt": 1,
             "fence": 5521, "leaseExpiresAt": "...", "payload": { ... } }]
POST /internal/leases/{runId}/heartbeat   extends the lease
POST /internal/leases/{runId}/complete    { "fence": 5521, "result": "succeeded" | "failed", "retryable": true }

4. Sketch the data model

The unique constraint on (job_id, scheduled_for) does a lot of quiet work: whatever goes wrong upstream, two schedulers cannot both create the 09:00 run of the same job.

5. Find due jobs on time

The central question is how the system notices that a job is due without scanning 500 million rows. There are three common answers.

A strong default combines them. The store is partitioned by a hash of job_id into, say, 1,024 partitions. Each scheduler node owns some partitions and, every 30 seconds, loads jobs due in the next two minutes into a timing wheel. A timing wheel is a ring of slots, one per second, where inserting a timer and advancing the clock are both constant time. When a slot fires, the scheduler creates the run row and pushes it onto a ready queue, then computes the job's next next_run_at. A job created for the next few seconds goes into the wheel directly as well as into the store.

Partitioning by job id rather than by time matters: partitioning by time would send all current work to one partition. See sharding strategies for the general rule.

6. The architecture

1. 1ScheduleThe API validates the job, computes the first next_run_at in UTC and writes the row. Nothing else happens until the job is nearly due.
2. 2FireThe owning scheduler's timer fires. It inserts the run row (a duplicate insert fails harmlessly on the unique key), enqueues the run and advances the job's next_run_at.
3. 3LeaseA worker takes the run with a lease of, say, 30 seconds and a fencing number, and heartbeats every 10 seconds while it works.
4. 4FinishThe worker reports success or failure with its fencing number. Failures are retried later or, after the last attempt, sent to the dead-letter queue.

7. Execute reliably

• Leases and visibility timeouts. A leased run is invisible to other workers until the lease expires. If the worker crashes, heartbeats stop, the lease expires and the run becomes available again. This is the same idea as a message queue's visibility timeout.
• Why at-least-once. A worker can be alive but paused (a long garbage-collection pause, a network partition) past its lease. Another worker then takes the run, and both may finish it. No scheduler can rule this out for jobs with outside side effects, so the run will sometimes execute twice.
• Idempotent handlers. Pass a stable run key (job_id plus scheduled_for) to the job, and have it record "run key done" in the same transaction as its effect, or pass the key as an idempotency key to the downstream API. Duplicates then do no harm: the effect happens once even if the code runs twice.
• Fencing. Each lease carries an increasing fencing number. The scheduler rejects a completion with an old number, and a downstream system that checks the number can refuse writes from a worker whose lease has been taken over.
• Retries with backoff. Delay = min(cap, base x 2^attempt), plus random jitter so a failed dependency is not hit by every retry at once. With a 10 second base and a 1 hour cap, the waits run 10 s, 20 s, 40 s and so on. Do not retry errors marked permanent.
• Dead-letter queue. After maxAttempts the run goes to a dead-letter queue with its last error, and an alert fires. Operators fix the cause and replay from there. Without this, a poison job retries forever.
• Fairness. Separate ready queues per tenant or job type, with per-tenant rate limits, stop one tenant's backlog from delaying everyone else.

8. What breaks, and what you do about it

9. Trade-offs to discuss

• Missed-run policy. If the platform is down for 20 minutes, a job that runs every 5 minutes misses 4 runs. Offer three choices per job: run every missed firing (catch-up, useful for billing periods), run once now (the usual default), or skip to the next scheduled time. Add a maximum lateness: a 09:00 digest delivered at 15:00 may be worse than none.
• Time zones and DST. Store 0 9 * * * with Europe/London, not as a UTC time, and compute each next run with a time zone library at the moment you schedule it, because time zone rules change. In the UK, clocks jump from 01:00 to 02:00 on the last Sunday of March, so local times between 01:00 and 01:59 do not exist that day; on the last Sunday of October they happen twice. Decide the rule: run a skipped time at the next valid instant, and run a repeated time once. Interval jobs ("every 15 minutes") should be computed in UTC so they are unaffected.
• Lease length. Short leases recover from crashes quickly but cause more duplicates and more heartbeat traffic; long leases do the opposite.
• At-least-once or at-most-once. Marking a run done before executing it gives at-most-once: no duplicates, but a crash loses the run. That fits work where a miss is cheap and a repeat is not, and nowhere else.
• Precision or cost. One-second precision needs in-memory timers; one-minute precision is fine with time buckets and polling alone.

10. How to talk through it in the interview

• Separate the job definition from each run early; most follow-ups are about runs.
• Name the guarantee, then show the three things that support it: leases, idempotency keys and fencing.
• Bring up DST and missed runs yourself. Few candidates do, and both are real production problems.

Practicing this design with Natively

Run the walk-through as a mock interview: ask a friend to play the interviewer on a call, or read the prompt aloud yourself. Natively transcribes the conversation as it happens and can suggest talking points for each question as it comes in; comparing them with what you said is a quick way to spot gaps such as fencing or the missed-run policy. If you draw the architecture, you can capture a screenshot of the diagram and have Natively explain it. After the session, the transcript and notes show where you ran long and what you skipped.