Design a Distributed Job Scheduler
Design a distributed job scheduler. Users submit jobs — one-time or recurring on a cron expression — with a payload; a job can be paused and resumed; due jobs run on a fleet of workers; failures are retried according to a per-job policy. The same run must never be launched twice, even when a dispatcher or a worker dies in the middle of its work.
Start by pinning down the requirements that change the design: one-time versus recurring and the cron precision, how many jobs fire in the busiest second, what happens to runs missed while a job was paused or the scheduler was down, whether two runs of the same job may overlap, and what exactly once means to the user. Then give the data model, the dispatch flow, how pause and resume work, the decisions you made and their alternatives, and how the system behaves when each component fails.
Hints
Hint 1
Make the job store the source of truth and the queue only a carrier of work: every component reads or writes the store, so a crash anywhere is recovered from its rows and nothing depends on the queue being complete.
Hint 2
The duplicate-launch guard is a unique key on (job_id, fire_time) in the runs table, claimed in the same transaction that advances next_fire_time; a second dispatcher's insert fails harmlessly. Workers add a lease with heartbeats so a dead worker's run can be retried.
Hint 3
Pause is a state change — running work finishes untouched and the dispatcher skips PAUSED jobs. Resume recomputes next_fire_time from now unless the user asked to catch up, and that is a policy you have to ask about.
Asked at
Sign in to talk to the interviewer — ask what's in scope, what scale to design for, or for a nudge when you're stuck. Conversations save to your account.