Scalable Job Queue Scheduler
Building a job queue from scratch exposes the design challenges that Celery and BullMQ solve but don't explain: at-least-once vs. exactly-once delivery tradeoffs, visibility timeout management, priority queue data structures under load, and worker heartbeat protocols for detecting silent failures.
Built a distributed task queue with at-least-once delivery and priority scheduling
- ›Redis-backed priority queue with visibility timeout for safe at-least-once execution
- ›Worker health monitoring with automatic re-queue on worker failure
- ›FastAPI interface for job enqueueing, status polling, and cancellation
A distributed job queue scheduler implementing priority-aware task dispatch, at-least-once delivery via visibility timeouts, and worker health monitoring. Built to understand implementation-level design challenges in production task queue systems.
Understanding how job queues actually work — visibility timeouts, priority ordering, retry backoff, worker coordination — requires building one from scratch rather than using an opaque framework.
- At-least-once delivery: jobs must not be silently dropped if a worker crashes mid-execution
- Priority scheduling: high-priority jobs must preempt low-priority work without starvation
- Worker health: silent worker failures must be detected and jobs re-queued automatically
- API must be simple enough to enqueue jobs with one call
Redis-backed queue with sorted sets for priority ordering. Visibility timeout: worker atomically dequeues and writes a processing lock with TTL; if TTL expires before completion, the job becomes visible again. PostgreSQL persists job history and dead-letter queue. Worker heartbeat: workers write a TTL key; coordinator re-queues jobs whose worker key has expired.