Distributed Job Scheduler System Design (Airflow / Temporal / Celery Style)

Designing a distributed, highly available job scheduler capable of handling repetitive, immediate, and future-scheduled tasks at scale (targeting 10,000 concurrent jobs per second with sub-2-second execution latency).

1. Why Do We Need a Distributed Job Scheduler?

A distributed job scheduler is an infrastructure service that allows users and applications to execute repetitive or one-off tasks at specified points in time.

• The Status Quo Problem: In monolithic systems, developers often rely on basic OS-level cron utilities (crontab). However, single-node cron breaks down in modern cloud environments because it lacks horizontal scalability, fault tolerance, historical audit trails, dynamic rescheduling, and real-time execution monitoring.
• Enterprise Use Cases: Database cleanup scripts, periodic report generation, asynchronous webhook deliveries, and automated billing cycles all require a bulletproof distributed scheduling engine.

2. Core Mental Model & Analogy

Real-Life Analogy: The Restaurant Kitchen & Expediter

Analogy: Think of a distributed job scheduler like a busy restaurant kitchen.

• Customers (Users) place orders for dishes (Jobs) that need to be served immediately, later tonight, or every Friday at 6 PM.
• The Order Desk / Front Desk (Job Service & DB) records all orders and organizes them into a schedule.
• The Head Chef / Watcher Service constantly scans the master order board, looking at what needs to be cooked in the next 5 minutes, and places them onto the Prep Line Queue (Kafka).
• Line Cooks / Worker Nodes (Executor Service) pull tickets off the queue, cook the dishes, and report progress back to the tracking board in real-time. If a cook drops a plate or burns out (Server Failure), a manager spots the stale timer and reassigns the ticket to another cook.

3. Functional and Non-Functional Requirements

A. Functional Requirements

1. Job Creation & Scheduling: Users can schedule jobs to run immediately, at a specific future timestamp, or as recurring cron expressions.
2. Real-Time Monitoring Dashboard: Users can view the live status, execution history, and logs of their jobs via a monitoring portal.
3. Job Lifecycle Management: Users can cancel pending/running jobs, update job metadata, or manually trigger (run now) jobs on-demand.

B. Non-Functional Requirements

• Scale: Must support peak loads of 10,000 concurrent jobs per second.
• CAP Theorem Trade-Off (AP over CP): We prioritize Availability (A) and Eventual Consistency. If a job creation takes 1–2 seconds to reflect in the monitoring dashboard, it is acceptable; the scheduling and execution pipeline must never go down.
• At-Least-Once Execution: Jobs must guarantee execution. If a node crashes, the job must be safely retried.
• Low Latency Execution: The scheduler must pick up and execute jobs within 2 seconds of their scheduled timestamp.

4. Core Entities and Database Schema

To persist metadata and execution states reliably, we utilize a relational database (PostgreSQL) optimized with targeted indexing and partitioning.

Table 1: jobs (Master Job Metadata)

Stores the definition and schedule configuration of every registered job.

Table 2: job_runs (Execution Audit & Status History)

Tracks individual execution instances (attempts) of jobs.

5. Public REST API Design

The API layer is cleanly separated into write/mutation operations (Job Service) and read/search operations (Search Service).

6. High-Level Architecture Flow

7. Low-Level Design & Deep Dive: The Execution & Polling Engine

A naive approach where worker nodes query the database every second creates severe database locks and performance degradation. To achieve high scalability, we introduce a decoupled Watcher Service, Kafka Event Streaming, and Redis Caching.

A. The Watcher & Sliding Window Polling Pattern

Instead of scanning the database every single second, we implement a Watcher Service that operates on a sliding window:

1. The Watcher Service queries the PostgreSQL database every 20 seconds.
2. It fetches all jobs scheduled to run within the next 5 minutes.
3. It pushes these upcoming jobs into a temporary Kafka Run Queue.
4. It updates the job_runs table status to QUEUED.

B. Race Condition Prevention: The 20-Second Gap Problem

The Problem: If the Watcher Service polls at 5:00:00, and a user schedules a job at 5:00:10 to run immediately, the watcher will miss it for the current window, causing it to execute a day late. The Solution: Any job created with job_type = IMMEDIATE or scheduled within a 1-minute threshold bypasses the Watcher Service entirely and is pushed straight into the Kafka broker for instant execution.

8. Failure Handling, Retries, and Dead Letter Queues (DLQ)

Handling component crashes and execution failures gracefully is critical to satisfying our at-least-once execution non-functional requirement.

Scenario 1: Job Execution Failure (Script Error)

Scenario 2: Worker Node Hard Crash (Server Death)

If an execution worker crashes mid-job, it cannot send a completion notification, leaving the job stuck in the RUNNING state indefinitely.

• Solution: The Watcher Service performs a secondary check. It scans the job_runs table for jobs where status = RUNNING and modified_time has not updated in over 15 seconds (heartbeat timeout).
• The Watcher assumes the worker node died, automatically flags the run as failed, increments the attempt count, and re-triggers the job into the Kafka queue.

9. Real-Time Job Cancellation Flow

To cancel a job that is currently executing on a worker node:

1. User sends a DELETE /api/v1/jobs/{job_id} request.
2. The Job Service writes a cancellation flag into a Redis Cluster with a Time-To-Live (TTL).
3. While the worker node executes a long-running job, a background thread polls the local Redis cache every 10 seconds.
4. If a cancellation key is detected, the worker immediately aborts the running process.

10. Database Optimization & Scaling Strategies

Because the system experiences high read/write volumes, PostgreSQL must be tuned:

• Partitioning: The jobs table is partitioned by schedule_time so the Watcher Service only scans relevant time-range partitions instead of locking the entire table.
• Indexing:
  • Primary index on job_id for lightning-fast CRUD lookups.
  • Composite index on (status, modified_time) for the Watcher Service's failure-detection queries.

11. Architectural Alternatives for Kafka

While Kafka is exceptional for throughput, standard Kafka queues are FIFO (First-In-First-Out). If a high-priority immediate job enters behind 100 queued jobs, it waits for the queue to clear.

Summary and Key Takeaways

• Distributed Job Schedulers require a decoupling of write APIs, database polling watchers, streaming brokers, and execution workers.
• The Watcher Service Sliding Window reduces database load by polling in bulk (e.g., every 20 seconds for the next 5 minutes) rather than hammering the DB every second.
• Immediate Job Bypass: Real-time or near-immediate jobs bypass the database watcher and push directly to Kafka for sub-2-second latency.
• Heartbeat Monitoring: Orphaned jobs caused by dead worker nodes are recovered automatically by monitoring stale modified_time timestamps.
• At-Least-Once Guarantees: Managed via multi-attempt retry topics, dead letter queues (DLQ), and persistent execution audit logs (job_runs).
• Availability over Consistency (AP): Embracing eventual consistency ensures the scheduling and execution pipeline remains operational even under peak network partitions.