Concurrency limits and queue depth under load

The 429 you get at the worst possible time

If you’ve run anything past a demo, you’ve seen it: a burst of traffic hits your app, the provider starts returning 429s, your retry logic kicks in and makes things worse, and now your p99 latency looks like a hockey stick. This isn’t a bug in the model. It’s what happens when concurrency limits and queue depth collide with real traffic patterns, and most teams don’t think about it until it’s already on fire.

This piece is about the mechanics: what a concurrency limit actually caps, why queues form even when you’re “under” your limit, and the handful of patterns that keep a system stable instead of spiraling.

What a concurrency limit is actually capping

People often conflate two different things: rate limits and concurrency limits.

A rate limit caps how much you can send over a time window, usually expressed as requests per minute (RPM) and tokens per minute (TPM). You can burst a little, but average out over the window and you’ll get throttled.

A concurrency limit caps how many requests can be in flight at the same time, regardless of how fast you’re sending them. This matters a lot for LLM calls specifically because generation is slow. A single completion can take anywhere from a few hundred milliseconds to tens of seconds depending on output length, so “in flight” isn’t a brief window, it’s a real chunk of wall clock time where that slot is occupied.

Most providers enforce both. You can be well under your TPM budget and still get throttled because you have too many long-running generations open at once. This is the part that surprises people coming from typical REST API work, where a request is usually done in under a second and concurrency rarely becomes the binding constraint.

There’s also a client-side concurrency limit that people forget about: your own connection pool, your own worker count, your own event loop. If you’re making synchronous calls from a thread pool sized for a normal web app, you’ll bottleneck yourself before you ever touch the provider’s ceiling.

Why queue depth spikes even when you’re “within limits”

This is the part that trips people up: you can be under your rate limit on average and still build up a queue, because averages hide bursts.

The useful mental model here is Little’s Law: the number of requests waiting in a system equals the arrival rate multiplied by the average time each request spends in the system. Say your app normally sends 20 requests per second and each one takes about 2 seconds end to end. That’s roughly 40 requests in flight at steady state. Nothing dramatic.

Now say a marketing push, a cron job, or a retry storm doubles your arrival rate for 90 seconds. If your concurrency limit stays fixed, the extra requests don’t get rejected outright, they queue. And because each request is still taking 2 seconds (or longer, if the provider is also under load and generation slows down), that queue doesn’t drain instantly when the burst ends. It drains at whatever rate your concurrency ceiling allows, which means the tail of that burst is still working through the system minutes after the burst itself stopped.

This is why queue depth, not raw request count, is usually the metric that predicts an outage. A system can handle a high steady rate fine and still fall over from a short burst, because the burst pushes queue depth past the point where new requests start timing out before they’re even served.

What happens when nobody’s watching the queue

Left unmanaged, this plays out in a predictable sequence:

Requests start timing out client-side before the provider even finishes them, because your own timeout is shorter than how long the request is actually queued plus processing.

Retry logic fires, and if it’s naive (no backoff, no jitter, retrying every failed request immediately) it adds more load to a system that’s already saturated. This is a retry storm: the thing meant to add resilience is what actually takes the system down.

429s and 5xxs get mixed together, and if your code treats them the same way, you end up retrying a hard rate-limit rejection at the same aggressive pace as a transient network blip.

Downstream systems back up too. If your LLM calls sit behind a queue that feeds a database write or a webhook, that backpressure propagates. A slow LLM tier can end up starving unrelated parts of your app that share a worker pool.

None of this is exotic. It’s the same failure mode you’d see with any slow, rate-limited downstream dependency. The LLM part just makes it worse because request duration is longer and more variable than most APIs people are used to designing around.

Patterns that actually hold up

Bound your own concurrency explicitly. Don’t rely on the provider to be your only guardrail. Use a semaphore, a worker pool, or a token bucket sized comfortably under your known ceiling, and treat that as a hard cap in your own code. If you know your account is limited to a given number of concurrent generations, cap your client at something below that, not at the exact number, because a few slots need headroom for retries and for slightly-slow-to-close connections.

Separate rate limit errors from everything else. A 429 with a Retry-After header is telling you exactly how long to wait. Honor it. Don’t apply generic exponential backoff on top of a value the provider already gave you, and don’t retry a 429 as fast as you’d retry a dropped connection.

Backoff needs jitter, not just exponential growth. If ten workers all get throttled at the same moment and all back off on the same fixed schedule, they’ll all retry at the same moment too, which just recreates the burst. Add randomness to the wait time so retries spread out instead of re-synchronizing.

Set client-side timeouts longer than your median latency but shorter than your patience. If your typical completion takes 3 seconds, a 5-second timeout will trigger constantly under any load at all, and you’ll be retrying healthy requests. Give yourself real headroom, then alert on queue depth and timeout rate as separate signals, not just on raw error count.

Use a circuit breaker for sustained failure, not just a retry loop. If failures cross a threshold over a short window, stop sending new requests for a cooldown period instead of letting every worker independently hammer a saturated endpoint. This protects both your own system and the provider’s, and it means you fail fast and predictably instead of failing slow and expensively.

Batch where the workload allows it. If you’re processing something like a backlog of documents rather than serving live user requests, you don’t need max concurrency, you need steady throughput. Running fewer concurrent requests with a queue you control gives you predictable cost and lets you avoid touching rate limits at all, at the cost of total wall-clock time. That’s a real tradeoff, not a free win: lower concurrency means the job takes longer to finish, full stop.

The tradeoff nobody likes stating plainly

There is no setting that maximizes throughput, minimizes latency, and minimizes cost at the same time. Higher concurrency gets you more done per minute but pushes you toward rate limits and makes failures more correlated. Lower concurrency is gentler and more predictable but slower. Aggressive retries paper over transient failures but amplify real ones. The right answer depends on whether you’re serving live user traffic, where latency matters more than efficiency, or running a batch job, where the opposite is often true.

The honest takeaway is that concurrency limits and queue depth aren’t a config value you set once. They’re a live property of your traffic pattern, and the systems that hold up under load are the ones that measure queue depth directly instead of inferring health from error rates alone.

If you’re building or shipping with these tools regularly, we cover this kind of production reality on AI Tool Gazette, without the vendor spin.