LLM cost attribution: what each customer actually costs you in tokens

You get the monthly invoice from your model provider. It’s a single number. Meanwhile you have a few hundred customers on a flat-rate plan, and you have no idea if customer A costs you three cents a month or thirty dollars. That gap is where margin quietly disappears, and it’s why llm cost attribution has become a real line item on the engineering roadmap instead of a nice-to-have.

I’ve run into this on every product I’ve shipped that wraps an LLM behind a subscription. The provider dashboard tells you total spend. It does not tell you who’s driving it. Fixing that requires instrumenting your own request path, not waiting for a vendor to hand you a report.

Why the total bill doesn’t help you

Total spend divided by active customers gives you an average, and averages hide the thing you actually need to see. If you have 500 customers and one of them is running your product in a loop against a large document set, that one account can account for a third of your token spend while paying the same $29/month as everyone else. Averages tell you the business is fine. Per-customer data tells you the business is fine except for the seventeen accounts quietly running at a loss.

This matters more with LLM costs than with most other infrastructure costs because usage is so lopsided. A database query costs roughly the same no matter who runs it. A prompt with a 40,000-token document attached costs an order of magnitude more than a prompt with a two-sentence question, and both can come from customers on the identical pricing tier.

What you actually need to capture

The unit you’re attributing is the individual API call, not the customer session and not the day. Every response from an LLM provider’s chat completion endpoint includes a usage object with input tokens, output tokens, and increasingly a breakdown of cached versus uncached input tokens. That object is the ground truth. Anything you build has to originate there, not from estimating token counts yourself with a tokenizer guess, because your estimate and the provider’s billing meter will drift apart over time as models and tokenizers change.

The minimum row you need in a ledger:

customer or tenant ID
request ID (for dedup and debugging)
model identifier, the specific model string, because you will route across models
input tokens, output tokens, cached input tokens if the provider reports them separately
computed cost in USD based on that model’s published per-token rate at call time
timestamp
the endpoint or feature that triggered the call

That last field matters more than people expect. Without it, you can tell that a customer is expensive, but not why. Is it their onboarding flow re-summarizing the same document on every page load? Is it a support chatbot that never truncates history and resends the whole conversation on every turn? You can’t fix what you can’t localize to a feature.

The streaming problem

If you’re streaming responses back to the UI, which most chat-style products do, the usage object doesn’t arrive until the stream closes. You have to accumulate the write in a finally block or equivalent, not inline with each chunk. This sounds obvious once you say it, but it’s the single most common reason cost ledgers end up missing rows: someone wires up logging on the happy path of a non-streaming call, ships streaming later, and the usage capture silently stops firing for the new code path. If your ledger’s row count is meaningfully lower than your actual request count, check this first.

Retries and partial failures still cost money

A request that times out on your side after the provider has already generated and sent tokens still burned those tokens. If your retry logic just re-fires the same prompt without checking whether the first attempt got billed, you can double-count a customer’s real usage, or worse, undercount it if you only log on success and the successful retry looks like the only attempt. Log the attempt, not just the outcome. Tie retries to the same request ID with an attempt number so you can reconcile against the provider’s own billing export later, because that export is the only source you can fully trust when your own numbers look off.

Prompt caching changes what “input tokens” means

Anthropic and OpenAI both offer some form of prompt caching, where a repeated prefix (a system prompt, a long document you reuse across turns) gets billed at a fraction of the normal input rate on subsequent calls. If your ledger only stores a single “input_tokens” number, you’ll systematically overstate cost for any customer whose usage pattern benefits from caching, like someone chatting repeatedly against the same uploaded document. Store cached and uncached input tokens as separate fields and apply the correct rate to each. A customer with a high cache hit rate might have a huge raw token count and a small actual bill, and if you only look at token count you’ll misjudge them as expensive when they’re actually one of your cheaper accounts to serve.

Agentic loops multiply the unit you’re counting

If your product does tool calling, or runs a multi-step agent loop to answer one user question, a single customer action can trigger five, ten, or more separate LLM calls. Each of those calls has its own usage object and needs its own ledger row, tagged with a shared “conversation” or “action” ID so you can roll them up. If you only log the final response to the user, you will miss the intermediate tool-calling turns entirely, and those turns often carry the largest input token counts because they include growing conversation history and tool output on every step. This is usually where the biggest gap shows up between what people assume a feature costs and what it actually costs, because the visible answer is short but the reasoning and tool-use path behind it wasn’t cheap.

Model routing means one price doesn’t apply to all rows

If you route some requests to a smaller, cheaper model and others to a larger one based on task complexity, your cost-per-row calculation has to look up the rate for the specific model used on that call, not a single blended rate for “the AI feature.” Store the model string on every row and keep a small rate table you update when a provider changes pricing, rather than hardcoding a dollar figure into the logging code. Providers change prices without much notice, and a stale hardcoded rate will make your dashboards wrong in a way that’s hard to notice because the numbers still look plausible.

What you do with the data once you have it

Once the ledger is real, the useful queries are simple: total cost by customer over the last 30 days, sorted descending, next to their subscription tier. That single view tells you which accounts are running at negative margin and by how much. It also catches bugs fast, an account with a token count ten times its usual pattern is very often a retry loop or an unbounded conversation history, not a customer suddenly getting ten times more value. I’ve caught more runaway-loop bugs from a spike on a per-customer cost dashboard than from any application log.

You don’t need a fancy analytics stack for this. A single append-only table and a daily rollup query gets you most of the value. The work is in getting the capture right at the source, streaming responses, retries, caching, and multi-call agent loops, so that the numbers in the table actually reflect what you were billed.

If you’re building or shipping with LLM APIs and want more of this kind of grounded, no-nonsense breakdown of how these tools actually behave under real usage, check out the rest of what we cover at AI Tool Gazette.