Home/Projects/Scalable Job Queue Scheduler
Prototype
Distributed SystemsInfrastructureBackend Engineering

Scalable Job Queue Scheduler

Building a job queue from scratch exposes the design challenges that Celery and BullMQ solve but don't explain: at-least-once vs. exactly-once delivery tradeoffs, visibility timeout management, priority queue data structures under load, and worker heartbeat protocols for detecting silent failures.

recruiter perspective

Built a distributed task queue with at-least-once delivery and priority scheduling

  • ›Redis-backed priority queue with visibility timeout for safe at-least-once execution
  • ›Worker health monitoring with automatic re-queue on worker failure
  • ›FastAPI interface for job enqueueing, status polling, and cancellation
View repo
Hiring Summary

A distributed job queue scheduler implementing priority-aware task dispatch, at-least-once delivery via visibility timeouts, and worker health monitoring. Built to understand implementation-level design challenges in production task queue systems.

Understanding how job queues actually work — visibility timeouts, priority ordering, retry backoff, worker coordination — requires building one from scratch rather than using an opaque framework.

Challenges Addressed
  • At-least-once delivery: jobs must not be silently dropped if a worker crashes mid-execution
  • Priority scheduling: high-priority jobs must preempt low-priority work without starvation
  • Worker health: silent worker failures must be detected and jobs re-queued automatically
  • API must be simple enough to enqueue jobs with one call
◆Scalable Job Queue SchedulerDISTRIBUTED SYSTEMS/visual-placeholder

Redis-backed queue with sorted sets for priority ordering. Visibility timeout: worker atomically dequeues and writes a processing lock with TTL; if TTL expires before completion, the job becomes visible again. PostgreSQL persists job history and dead-letter queue. Worker heartbeat: workers write a TTL key; coordinator re-queues jobs whose worker key has expired.

Stack
PythonRedisFastAPIPostgreSQLDocker
Decision
Visibility timeout over message acknowledgment
Explicit ack requires workers to be alive to ack. Visibility timeout makes the job invisible while processing and automatically re-enqueues if the lock expires — at-least-once delivery without worker coordination.
Decision
Redis sorted set for priority queue
A sorted set with score = (priority * large_constant - timestamp) naturally orders by priority with timestamp as tiebreaker. O(log n) enqueue and dequeue; no heap maintenance code.
Tradeoff
At-least-once vs. exactly-once delivery
Chose: At-least-once with idempotency key support
Exactly-once requires distributed transactions, which are expensive. At-least-once with idempotency keys achieves safe execution without the coordination overhead.
01Distributed worker pool with consistent hashing for queue partitioning
02Cron-style scheduled job support
03Real-time worker dashboard for queue depth and throughput monitoring