← Back

LLM Tracing Without Logging Every Sensitive Prompt

Trace LLM requests across retrieval, generation, and tools while limiting sensitive data collection. Design useful spans, safe attributes, and actionable failure views.

LLM tracing should help you explain why a request was slow, wrong, or expensive without turning your observability platform into a second copy of every private conversation. Start with operations and outcomes. Add content capture only when there is a justified need and an appropriate access policy.

Suppose a user reports that a support assistant took 18 seconds and returned an outdated answer. A full prompt dump may still leave you guessing. A trace that shows retrieval version, selected document IDs, tool retries, and stage timings gives you a direct investigation path.

Model the request as a tree of work

OpenTelemetry describes traces as records of a request's path through an application, built from spans representing units of work. For an LLM application, useful spans often include request handling, retrieval, reranking, generation, and tool execution.

Keep one root span for the user-visible operation. Add child spans around meaningful boundaries that have their own latency and failure behavior. Avoid a single span named AI that conceals all the work inside it.

An illustrative trace could look like this:

answer-support-question
  authorize-request
  retrieve-passages
  select-context
  generate-answer
    provider-attempt-1
    provider-attempt-2
  validate-citations

This structure is a design sketch. The exact instrumentation depends on your framework and SDK. Preserve parent-child relationships across asynchronous work so retries and background tasks remain connected to the originating request.

Capture identifiers before content

Useful attributes include application version, retrieval configuration, model identifier, prompt-template version, token counts when available, finish reason, retry count, and a categorized error. These often explain a regression without storing raw user text.

For retrieval, capture stable document IDs and versions where policy allows. Record candidate count, delivered passage count, and whether context was truncated. Avoid putting every document ID into a metric label; high-cardinality detail belongs in appropriately controlled trace data, not an unbounded metric dimension.

The OpenTelemetry GenAI conventions repository is the current home linked from the older GenAI documentation pages. Check the version your instrumentation implements before copying attribute names from a tutorial. Conventions and instrumentation support can evolve independently.

Separate your stable application fields from provider-specific fields. A provider migration should not erase the meaning of retrieval_config_version or tool_outcome in your operational dashboards.

Redact before export where possible

OpenTelemetry's sensitive-data guidance discusses controlling collected data and processing sensitive attributes. Apply minimization at the earliest practical point. Deleting a field after it already reached several collectors is a weaker posture than never emitting it.

Prefer structured allowlists to broad “log everything except secrets” rules. Credentials can appear in unusual fields, and private information is broader than recognizable API-key patterns. A support ticket body may be sensitive even when no regular expression flags it.

If content capture is necessary for a restricted debugging session, define who can enable it, which requests are included, where the data goes, and when it expires. Keep that mode distinct from normal production telemetry.

Hashing identifiers can reduce direct exposure, but it is not automatically anonymization. Stable hashes can still enable correlation, and low-entropy inputs may be guessable. Choose the representation based on the actual access and privacy requirements.

Make retries and failures visible

A request that succeeds on its third attempt should not look identical to one that succeeds immediately. Record attempts as separate spans or equivalent events, with failure categories and elapsed time.

Distinguish provider errors, application validation failures, authorization denials, and intentional abstention. A safe refusal due to missing evidence is not necessarily a system outage. An invalid tool result is not the same as a user cancellation.

For streaming, separate time to first output from time to completion. Users experience the initial wait differently from a long but steadily streaming answer. If the application buffers output for validation, measure the user-visible first byte as well as the provider's first token.

Record cancellation propagation. A disconnected client should not leave an expensive generation or tool loop running indefinitely unless the product intentionally supports background completion.

Connect traces to quality evaluation

A trace explains execution; it does not independently establish answer quality. Attach an evaluation case ID for test traffic or a controlled feedback reference for production investigations. Keep human review outcomes separate from raw transport status.

For the outdated-answer example, inspect the corpus version and selected document versions first. If retrieval chose the old policy, investigate indexing or ranking. If it chose the new policy but the answer contradicted it, inspect generation and validation. The same user symptom can emerge from different stages.

Build dashboards around actionable slices: errors by operation, latency by model and request class, retry rate, context truncation rate, and denied tool calls. Avoid a dashboard full of token totals that never identifies a next step.

Test telemetry as part of the product

Create synthetic requests with deliberately recognizable dummy secrets, controlled tool failures, and a retry. Confirm that sensitive markers do not appear in exported telemetry and that the expected spans remain connected. These are recommended tests, not tests run for this article.

Check access controls and retention on the observability backend too. A careful emitter cannot compensate for unrestricted access to a debug store that contains private payloads.

The metric cardinality guide covers bounded aggregate dimensions; traces provide a separate investigation path. Connect trace outcomes to the agent evaluation dataset so observed failures can become repeatable regression cases.