What to do when your LLM provider goes down
Your model provider will go down. Not might. Will. If you’ve had anything running against a hosted LLM API for more than a couple of months, you’ve already eaten at least one incident where requests started timing out, returning 500s, or silently queuing for minutes at a time. The question isn’t whether this happens. It’s whether your app degrades gracefully or falls over completely, and whether you find out from your monitoring or from an angry user.
This is a guide to the actual mechanics of surviving that, written from the side of someone who pays the API bill and has to explain the downtime to whoever is upstream of me.
Why single-provider apps break first
Most side projects and a lot of production apps call one provider’s SDK directly. openai.chat.completions.create() or the equivalent, no wrapper, no fallback, sometimes no retry logic at all. This works fine until the provider has a bad ten minutes, and then every request your app makes either hangs until your own timeout fires or comes back with an error your code was never written to handle.
The failure mode isn’t usually “the whole API is dead.” It’s messier: elevated latency on one region, degraded availability for one model tier, rate limiting that kicks in harder than usual because everyone else is also retrying into the same endpoint. Your app sees a mix of slow responses, outright errors, and requests that succeed but take four times as long as normal. If your code treats “the API call didn’t return in 2 seconds” as an edge case instead of the normal operating condition it sometimes is, you’ll find out the hard way.
What fallback actually means at the API layer
Fallback isn’t a single feature. It’s three separate mechanisms stacked on top of each other, and most outages are survived (or not) based on which of the three you actually built:
Retries with backoff. A single failed request gets retried, but not immediately and not forever. A common pattern is a short delay before the first retry, then doubling the delay each subsequent attempt, capped at two or three tries total. Retrying immediately in a tight loop is how you turn a provider having a bad moment into you getting rate limited on top of it, so the backoff matters as much as the retry itself.
Timeouts that match your use case. A chat UI where a user is staring at a spinner should time out in a few seconds. A background job summarizing a document overnight can tolerate thirty. Using the same timeout value everywhere, especially the SDK default, means you’re either killing slow-but-fine requests too early or letting a hung request block a queue worker for minutes.
Provider fallback. When retries against provider A keep failing, the request goes to provider B instead. This is the part people mean when they say “fallback” and it’s the hardest of the three to build, because it requires your app to not actually care which provider answered the prompt.
The part that actually breaks: prompt and output parity
Here’s the piece that trips people up when they first try to add a second provider. Switching providers isn’t a config change if your prompts were tuned against one model’s quirks. System prompt handling differs. Function calling and structured output formats differ enough that a schema that reliably produces valid JSON on one provider can produce malformed JSON on another. Token limits differ. The tone and verbosity of the response differs even at similar temperature settings.
If your fallback path has never actually been exercised against real traffic, you don’t have a fallback. You have an assumption. The only way to know your secondary provider produces usable output for your actual prompts is to route a slice of live traffic to it on a normal day, compare outputs, and fix the prompt or parsing logic before you need it during an incident.
Routing layers: what they do and what they cost you
There’s a category of open source tooling built specifically for this, LiteLLM and OpenRouter being the two most commonly used. Mechanically, they sit between your app and the providers, exposing one API shape and translating your request into whatever each backend actually expects. You configure a primary and one or more fallbacks, and the router handles the retry-then-switch logic instead of you writing it by hand.
That’s genuinely useful and it saves you from reinventing retry and backoff logic badly. But it’s not free of tradeoffs, and I haven’t benchmarked either against the other so I won’t rank them:
- You’ve added a hop. Every request now goes through the router’s infrastructure before it reaches the provider, which adds latency and a new dependency that can itself go down.
- Aggregators that proxy requests generally add their own margin on top of provider pricing. Whether that’s worth it depends entirely on your volume and how much engineering time you’d otherwise spend maintaining your own routing code.
- You lose some control over exact model versions and provider-specific parameters that the router’s abstraction layer doesn’t expose.
None of that makes them a bad choice. It means “add a router” isn’t automatically the right answer for every setup, and you should read what each one actually does before assuming it solves your specific failure mode.
Caching and idempotency buy you the first five minutes
A lot of outages are short. If you can serve a cached response for a repeated or similar query, you don’t need a live model call at all for that slice of traffic. This matters most for anything RAG-adjacent, where the same question gets asked repeatedly and the underlying documents haven’t changed. Caching won’t help a genuinely novel prompt, but it takes real load off your fallback path exactly when that path is most likely to also be stressed.
Idempotency matters on the write side. If a retried request can double-charge a user, double-send an email, or double-write a database row because your first attempt actually succeeded but the response got lost in transit, that’s a bug your retry logic just exposed, not caused. Every action your LLM output triggers downstream needs to be safe to run twice.
Degraded mode is a product decision, not just an engineering one
When your primary and fallback are both struggling, someone has to decide what the user sees. Options include falling back to a smaller or faster model with a quality tradeoff you’re willing to disclose, queuing the request and telling the user it’ll take longer, or returning a plain “try again shortly” instead of pretending everything is normal. Whichever you pick, decide it in advance. Deciding it live, during an incident, under pressure, is how you end up either serving garbage output silently or taking the whole feature offline when a partial answer would have been fine.
The checklist worth having before you need it
Set timeouts per use case, not one global default. Add retry with backoff, capped at two or three attempts. Actually test your fallback provider against real prompts on a normal day, not just during the outage. Cache aggressively for repeated queries. Make every downstream write idempotent. Decide your degraded-mode UX ahead of time. And check the provider’s status page as part of your incident process, since it tells you whether you’re looking at a five-minute blip or something that’s going to eat your afternoon.
None of this requires switching providers permanently or picking a “winner.” It requires treating outages as a normal operating condition instead of an exception, because that’s what they are.
For more breakdowns like this on the tools, models, and pipelines people actually run in production, head back to the AI Tool Gazette homepage.