The queue that ate its worker

The pitch for an unbounded queue is irresistible: nothing is ever rejected. Traffic spikes, the queue absorbs it, the workers catch up later, nobody sees an error. It feels like resilience. Mostly it is a machine for converting a fast visible failure into a slow invisible one.

What actually happens

When arrival rate passes service rate, depth grows without limit and so does latency. Nothing errors. Every request is accepted, every job is eventually processed, and every user quietly experiences a system that looks fine while handing back answers that are twenty minutes stale. Your dashboards are green. Your error rate is zero. Your product is broken.

Then the queue itself becomes the resource under pressure. Memory climbs, the broker slows, and the component you added for stability turns into the thing that falls over.

Put a wall up and choose what hits it

Setting a maximum depth forces the question you have been avoiding: when you cannot keep up, what gives? There is no universal answer, but there are only about four:

• Reject new work and say so — honest, and lets the caller retry sensibly.
• Drop the oldest — correct when work is only valuable while fresh.
• Drop by priority — correct when some work genuinely matters more.
• Slow the producer down — correct when the producer is yours to slow.

All four beat the fifth option, which is to accept everything and hope. That is precisely what an unbounded queue picks for you, silently, on your behalf.

pythonMAX_DEPTH = 10_000

async def submit(job: Job) -> None:
    if await queue.depth() >= MAX_DEPTH:
        # fail fast and loudly, while it is still one caller's problem
        raise Overloaded(retry_after=30)
    await queue.put(job)

Throughput is a flattering metric

It looks healthy right up to the moment it does not, because a saturated system still processes jobs at full speed — it is simply losing ground. Depth and oldest-message age tell the truth far earlier. If depth has trended upward for an hour you are already in an incident; nobody has paged you yet.

A bounded queue tells you the truth immediately. An unbounded one tells you the same truth later, after it has become expensive.