How to log LLM prompts without storing customer data

The problem nobody flags until the audit

You ship an LLM feature. It breaks in a way you can’t reproduce locally. Your instinct, correctly, is to pull the logs and see what the model actually saw and said. The problem is that “what the model actually saw” usually includes a customer’s name, their account number, a support ticket they pasted in full, or a chunk of a contract your RAG pipeline retrieved and stuffed into the context window. Now that’s sitting in a log file, probably in plaintext, probably with a retention policy nobody wrote down, probably readable by anyone with access to your logging dashboard.

I’ve run into this on every RAG project I’ve touched. The pattern is always the same: logging gets added late, as a debugging aid, by whoever is fighting the fire that week. It works. It stays. Six months later someone in legal asks where customer data lives and the honest answer includes “also our Datadog index.”

This is an engineering problem with a real fix, and the fix is cheaper than dealing with it after the fact.

What you actually need logs for

Before deciding what to strip, be clear on why you’re logging at all. In practice it’s usually one of four things. Debugging a specific bad response means you need to see the exact prompt and completion for one request. Tracking drift or regressions means you want to know if output quality changed after a prompt edit or a model swap. Cost and latency monitoring covers token counts, time to first token, model used, and retry counts. Eval and fine-tuning data collection means you want a corpus of real traffic to build a test set from.

Only the first and last actually require the raw text. The other two run fine on metadata alone. A lot of teams log full payloads by default because it’s the path of least resistance in whatever framework they’re using, not because the use case demands it. That default is the thing to change.

Log the shape, not the substance

For latency and cost tracking, log a structured event per request that never touches the content:

request ID, timestamp, model name, and version
input token count, output token count
latency, retry count, HTTP status
which prompt template or system prompt version was used (by ID, not text)
which retrieval sources were hit, by document ID, not by retrieved text

That’s enough to catch a cost spike, a latency regression after a model swap, or a jump in token usage after someone edited a prompt template. None of it requires storing what the customer typed or what the model retrieved on their behalf. This is the log stream that should have a long retention window, because it’s cheap and it’s not sensitive.

Redact before it leaves the process, not after

The mistake I see most often is redacting logs after they’re written, as a cleanup job that scrubs an index periodically. That means the raw data existed on disk, even briefly, and it means every downstream consumer of that log (search index, backup, replication to a second region) had a window where it held real customer data. Redaction has to happen in the request path, before the write, not as a batch job that runs later.

Concretely that means:

Run a PII scrubber (regex for emails, phone numbers, card numbers, plus a named-entity pass for names and addresses) on the prompt and completion before the logging call, not on the stored copy.
For structured fields you know the shape of, like account IDs or order numbers, replace them with a deterministic hash so you can still correlate two log lines that mention the same account without storing the account number itself.
Truncate or drop retrieved context entirely from the debug log by default. Log the document IDs that were retrieved, not their contents. If you need to see the actual retrieved text for a specific failure, pull it live from the vector store using the IDs and the request timestamp, not from the log.

None of this is exotic. It’s the same pattern you’d use for scrubbing credit card numbers out of application logs, applied to a new kind of payload.

Two-tier storage: metadata forever, payloads briefly

The setup that’s worked for me is splitting logging into two stores with different lifetimes.

The metadata store (token counts, latency, model, template version, hashed correlation IDs) has no sensitive content in it, so it can live for a year or more in whatever observability stack you already run. It’s what powers your dashboards and cost alerts.

The payload store, if you keep one at all, holds the redacted prompt and completion text, tied to a request ID, with a short TTL. Seven to fourteen days is usually enough to debug something a customer reports, and short enough that “how long do you retain conversation content” has a boring answer instead of an open-ended one. Access to this store should be scoped to on-call engineers, not the whole team, and it should sit in its own encrypted table or bucket rather than mixed into your general log index where every service’s logs already have broad read access.

If you genuinely need a corpus for eval or fine-tuning, that’s a separate, deliberate export with its own consent and retention story. It shouldn’t be “whatever happens to still be in the debug logs.”

Client-side versus server-side redaction

If you’re calling a third-party model API, you have a choice about where redaction happens. Doing it client-side, before the prompt ever leaves your infrastructure, means the provider never sees the raw customer data either, which matters if your contract with them doesn’t cover certain data categories or if you’re operating somewhere with data residency rules. Doing it only in your own logging layer, after the call, means the raw text still went over the wire to the provider even if your logs are clean.

Anthropic and OpenAI both state they don’t use API traffic to train models by default, and that’s a real and useful commitment, but it’s about training, not about their own request logs, which they retain for some period for abuse monitoring. That’s their retention, not yours. It doesn’t change what you do on your side. If you have specific commitments to customers about where their data goes, redact before the API call, not just before your own log write.

Proxy-based logging tools change where redaction lives

A chunk of the LLM observability tooling that’s shown up in the last couple of years, things like Langfuse or Helicone, work by sitting as a proxy or SDK wrapper between your app and the model API. That’s convenient for capturing token counts and latency automatically, but it also means the proxy sees the full prompt and completion before you’ve had a chance to redact anything, unless the tool has an explicit redaction or masking step you configure. If you adopt one of these, check whether redaction happens before their capture step or after, and whether their own storage has a retention setting you control. I’m not ranking these tools or telling you which one to buy. I’m telling you to read the data flow diagram before you wire one into a path that touches customer text, because the convenience of auto-instrumentation is exactly what causes teams to skip the redaction step they’d have added manually.

The checklist that actually matters

If you do nothing else: separate metadata logging from content logging, redact content before the write instead of after, put a short TTL on anything that holds real prompt or completion text, and restrict who can read that store. Log document IDs instead of retrieved text in your RAG pipeline. And when you add a new logging tool to the stack, trace where the payload sits before it’s masked, because that’s the step everyone forgets under deadline pressure.

None of this is hard. It’s just not the default behavior of the frameworks and dashboards you’ll reach for first, so it has to be a deliberate decision someone makes and writes down.

If you want more of this kind of practical, no-hype breakdown of what actually happens under the hood of AI tools, check out more from AI Tool Gazette.