Caching model responses without serving stale answers
If you’re running anything with real traffic on top of an LLM API, you’ve already done the math. The same handful of questions get asked over and over, and paying full price to regenerate an identical answer every time is money you’re lighting on fire. So you cache. Then, a week later, a user reports that your assistant confidently told them a price, a status, or a fact that was true three days ago and isn’t true now. The actual problem with LLM response caching is whether you can do it without quietly turning your app into a machine that serves yesterday’s truth with today’s confidence. Deciding to cache is the easy half.
What caching actually means for an LLM call
There are really three different things people lump under “LLM caching,” and mixing them up is where most of the trouble starts.
The first is exact-match response caching: you hash the full request (prompt, model, temperature, any tool definitions) and store the response keyed on that hash. Next time the same hash comes in, you skip the API call entirely. This is the cheapest form of caching and also the safest, because it only ever fires on a literal repeat of the same input.
The second is prefix or prompt caching, which is a provider-level feature rather than something you build yourself. Anthropic and OpenAI both let you mark a chunk of a prompt (a long system prompt, a big set of tool definitions, a document you’re grounding on) as reusable across calls. The provider keeps the internal state for that prefix around for a short window and skips recomputing it on the next request that shares it. This caches the expensive part of the setup work rather than the answer, and it’s billed at a reduced rate compared to a fresh read of the same tokens. It’s a latency and cost optimization, not a staleness risk, because the model still generates a fresh response every time.
The third is semantic caching: instead of matching on an exact hash, you embed the incoming query, search a vector store for a similar past query, and if the similarity score clears some threshold, you return the cached answer for the old query instead of running the model again. This is where staleness actually lives, and it’s worth being honest about why.
Where staleness sneaks in
Exact-match caching is safe against staleness in exactly one sense: it never serves an answer to a question it wasn’t literally asked before. The risk there is temporal drift, not semantic drift. If you cache “what’s the current status of order #4471” and don’t expire that entry, you’ll serve a two-day-old status forever, because nothing about the request changed even though the underlying data did.
Semantic caching adds a second failure mode on top of that. Two questions can be close enough in embedding space to trigger a cache hit while meaning genuinely different things at different points in time. “What’s your refund policy” asked in January and “what’s your refund policy” asked in July might embed almost identically, and if the policy changed in March, the cache doesn’t know that. The cache is matching on wording similarity, not on whether the world described by that wording has moved. This is the part that catches teams off guard, because a semantic cache hit feels like a correct answer. The model returns a previously-generated, fully confident response to a question that resembles the one it was asked before, with no hedge and no flag on the uncertainty.
RAG pipelines compound this. If you’re caching based on the user’s query alone and not on which documents got retrieved for it, you can serve a cached answer that was grounded in a document version that’s since been edited or deleted. The query looks the same. The knowledge base underneath it doesn’t.
Building a cache that knows when it’s wrong
The fix is to tie the cache’s validity to something that actually tracks the data, not just the question. “Don’t cache” is the wrong lesson to take from this.
Start with TTLs that match the volatility of what you’re answering. A cached response about a company’s founding year can live for months. A cached response about inventory, pricing, or account status should live for minutes, if you cache it at all. This sounds obvious written down, but it’s the step most people skip because a single global TTL is easier to configure than per-category ones.
Better than a blind TTL is invalidation tied to a write. If your app has a database row for the order, the price, or the policy doc, hash that row (or its updated_at timestamp) into the cache key. When the underlying record changes, the key changes, and the old cache entry becomes unreachable without you having to remember to purge it on a schedule. This is more work upfront, but it means staleness is bounded by “did the source data change,” not by “did we guess the right expiry window.”
For RAG specifically, key the cache on the retrieved chunk IDs and their content hashes, not just the user’s query text. If the retriever pulls a different or updated set of chunks, that’s a different cache key even if the query is byte-for-byte identical. It also gives you a debugging trail: when someone reports a wrong answer, you can check whether it came from a cache hit and, if so, which document version it was grounded in.
For semantic caches, raise the similarity threshold higher than feels natural at first, and separately, cap how long a semantic cache entry lives even if nothing seems to have changed, since you often don’t have a clean signal for “the underlying fact moved.” A stale-while-revalidate pattern helps here: serve the cached answer immediately, but kick off a background regeneration, and swap the cache entry once the fresh one lands. Users get the speed of a cache hit most of the time, and the window during which a wrong answer can be served is bounded to one regeneration cycle instead of open-ended.
Don’t cache non-deterministic output as if it were fixed
One thing that’s easy to miss: if your temperature is above zero, “the same prompt” has a distribution of acceptable answers rather than one correct cached one. Caching under those conditions means the first response you happen to generate becomes the permanent answer for every future identical request, which quietly removes whatever variability you were paying for by sampling at temperature > 0 in the first place. If you want caching, either pin temperature to 0 for the cached path, or accept that you’re caching one sample out of many and treat it accordingly, especially for anything creative or open-ended where a single frozen response will start to feel repetitive to repeat users.
What to log and watch
None of this works if you can’t tell, after the fact, whether a bad answer came from the model or from the cache. Log the cache key, whether the request was a hit or miss, and the age of the entry that got served, alongside the response itself. When someone flags a wrong answer, the first question should be answerable in one query: was this generated fresh, or was it a semantic hit on a three-week-old entry that’s aged past its usefulness. Track your hit rate too, as a risk metric. A cache with a 90% hit rate on a knowledge base that updates daily is a cache that’s mostly serving yesterday’s data with a fresh coat of paint.
Caching model responses is worth doing. The API bill doesn’t lie, and repeat questions are real. But the goal is a cache whose invalidation logic is as carefully built as the retrieval or generation logic it’s sitting in front of, not a maximum hit rate. Match the cache key to the thing that can actually change, not just to the words the user typed.
If you’re working through this kind of infrastructure decision on your own stack, we cover more of these practical AI engineering tradeoffs at AI Tool Gazette.