What is a DAG job scheduler?

A system that runs workflows defined as a DAG (directed acyclic graph) of tasks: it fires each workflow on a schedule (cron) or an external trigger, then runs each task only after all its upstream tasks succeed, retrying failures with backoff and respecting parallelism limits. Airflow, Dagster, and Prefect are examples. The hard part is not the graph — it is doing all of this without losing a run or double-running a task when a machine crashes.

How do you run more than one scheduler without double-scheduling tasks?

Make the durable state store — not the scheduler process — the source of truth, and have every scheduler claim work with a row-level lock. Airflow "supports running more than one scheduler concurrently" and coordinates purely through its metadata database: each scheduler runs SELECT ... FOR UPDATE SKIP LOCKED to grab a disjoint set of schedulable task rows, so two schedulers never pick up the same task, with no ZooKeeper or Raft required.

What is a zombie task and how do you handle it?

A zombie is a task whose worker died mid-run: the scheduler still sees it as "running" but no one is executing it. You detect it with heartbeats — the worker must renew a lease periodically; when heartbeats stop past a timeout, the scheduler marks the task failed and retries it as a new attempt. To stop the dead worker's late write from corrupting the retry, each attempt carries a fencing token (a monotonically increasing attempt number); a write tagged with a stale token is rejected.

How does a scheduler avoid double-running a job after a crash?

It cannot fully avoid running a task twice — a worker can finish work and die before reporting success, so the scheduler must retry. The honest guarantee is at-least-once dispatch plus idempotent task effects: deliver the task possibly twice, but write results keyed by (dag, run_id, task) so applying them twice has the same effect as once. Durable execution engines like Temporal push this further, persisting every completed step so a workflow "can pick up right where it left off."

What is the cron thundering-herd problem and how do you fix it?

Engineers overwhelmingly schedule jobs at round times — the top of the hour, midnight — so a huge fraction of DAGs fire at the same instant and the enqueue rate spikes 10x or more over the daily average. The fixes are to add jitter (offset each schedule by a small deterministic amount so the herd spreads out) and to shard the scheduling work across schedulers by DAG hash so no single scheduler owns the whole spike.

Design a DAG Job Scheduler

One scheduler process with in-memory state is a demo; the moment it crashes or you run two, jobs are lost or double-run. Build up to active-active schedulers over a durable store — SKIP LOCKED, heartbeats, fencing tokens, and idempotent retries — drawn and animated.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a DAG Job Scheduler — built to be understood, not skimmed.

Questions

What is a DAG job scheduler?: A system that runs workflows defined as a DAG (directed acyclic graph) of tasks: it fires each workflow on a schedule (cron) or an external trigger, then runs each task only after all its upstream tasks succeed, retrying failures with backoff and respecting parallelism limits. Airflow, Dagster, and Prefect are examples. The hard part is not the graph — it is doing all of this without losing a run or double-running a task when a machine crashes.
How do you run more than one scheduler without double-scheduling tasks?: Make the durable state store — not the scheduler process — the source of truth, and have every scheduler claim work with a row-level lock. Airflow "supports running more than one scheduler concurrently" and coordinates purely through its metadata database: each scheduler runs SELECT ... FOR UPDATE SKIP LOCKED to grab a disjoint set of schedulable task rows, so two schedulers never pick up the same task, with no ZooKeeper or Raft required.
What is a zombie task and how do you handle it?: A zombie is a task whose worker died mid-run: the scheduler still sees it as "running" but no one is executing it. You detect it with heartbeats — the worker must renew a lease periodically; when heartbeats stop past a timeout, the scheduler marks the task failed and retries it as a new attempt. To stop the dead worker's late write from corrupting the retry, each attempt carries a fencing token (a monotonically increasing attempt number); a write tagged with a stale token is rejected.
How does a scheduler avoid double-running a job after a crash?: It cannot fully avoid running a task twice — a worker can finish work and die before reporting success, so the scheduler must retry. The honest guarantee is at-least-once dispatch plus idempotent task effects: deliver the task possibly twice, but write results keyed by (dag, run_id, task) so applying them twice has the same effect as once. Durable execution engines like Temporal push this further, persisting every completed step so a workflow "can pick up right where it left off."
What is the cron thundering-herd problem and how do you fix it?: Engineers overwhelmingly schedule jobs at round times — the top of the hour, midnight — so a huge fraction of DAGs fire at the same instant and the enqueue rate spikes 10x or more over the daily average. The fixes are to add jitter (offset each schedule by a small deterministic amount so the herd spreads out) and to shard the scheduling work across schedulers by DAG hash so no single scheduler owns the whole spike.

Design a DAG Job Scheduler

Questions

Related explainers