CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 3 · Lesson 9/25

    Telemetry and instrumentation for Claude agents

    Telemetry and Instrumentation

    22 min read
    4% of exam
    9 sources
    Published 19 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Structure agent telemetry as one trace per task, with a span for each model request and each tool use, so a slow run can be read turn by turn
    • Record token usage as separate input, output, cache-read and cache-write counts, and price each count at its own rate without counting the same tokens twice
    • Use cache diagnostics to find where a prompt prefix diverged when cache reads fall to zero
    • Trace reasoning, tool arguments, tool results and citations back to the run that produced them
    • Choose content-capture settings that keep traces useful under zero data retention and stricter data-handling rules

    Key concept

    Span tree per task — A trace records one unit of work, such as one ticket or one chat request, from start to finish. Each model request and each tool use becomes a child span inside it, carrying its own timing, status, token counts and IDs. Because every step has a place in that tree, you can tell which tool call belonged to which turn and where the time went.

    1.Distributed traces for agent loops and tool calls

    An agent that makes eight model calls and four tool calls per task can't be debugged from one log line per API call. Those lines carry timestamps but no parent. When a ticket takes forty seconds, nothing ties a given tool execution to the turn that requested it, or says whether the time went to the model or to the tool. A trace fixes this by giving each task a root span and nesting every step under it. The Claude API and the Anthropic SDK emit these spans with OpenTelemetry, using attributes from the GenAI semantic conventions, so any platform that accepts OTLP can show them without extra setup.

    The tool-runner trace shows the shape. The tool runner is an SDK helper that runs the model-and-tool loop for you: it sends each request, runs your tools and sends their results back to Claude. Custom tools run in your own code, through the runner, while server tools such as web search run on the API. Your application opens an answer_question span and calls the tool runner inside it, which emits anthropic.messages.tool_runner. Each request the runner sends gets an anthropic.messages.create client span from the SDK and a server span from the API. Under the server span, the API adds a span for every model request and every server tool it runs, such as web search. When the model asks for a custom tool, the request ends and the runner executes the tool as an anthropic.custom_tool_use get_weather span. Then a second anthropic.messages.create carries the tool result back. Reading the tree top to bottom gives you the loop in order, with each step timed.

    The Claude Agent SDK and Claude Code emit their own span set. claude_code.interaction wraps one turn of the agent loop. claude_code.llm_request wraps each Claude API call and records model name, latency and token counts. claude_code.tool wraps each tool invocation, with child spans for the permission wait and for the execution itself. That split separates a tool that ran slowly from a run that sat waiting for a human to approve the tool. Traces are in beta and need CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 on top of OTEL_TRACES_EXPORTER. Metrics and log events don't need the beta flag.

    Agent SDK telemetry settings: one exporter per signal, plus the beta flag that traces requirepython
    OTEL_ENV = {
        "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
        # Required for traces, which are in beta. Metrics and log events do not need this.
        "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
        # Choose an exporter per signal. Use otlp for the SDK; see the Note below.
        "OTEL_TRACES_EXPORTER": "otlp",
        "OTEL_METRICS_EXPORTER": "otlp",
        "OTEL_LOGS_EXPORTER": "otlp",
        # Standard OTLP transport configuration.
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4318",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token",
    }

    For production, set these variables in the process environment, so every query() call picks them up with no code change. Pass them per call through env only when agents in the same process need different settings. In Python, env is merged on top of the inherited environment. In TypeScript it replaces the inherited environment entirely, so you have to spread process.env first or lose PATH and your API key.

    The other common fixes don't restore the structure. Adding prompt token counts to each log line flags large calls but still doesn't link a tool call to its turn. Logging full bodies forces engineers to reconstruct the order by hand and copies content you may not be allowed to keep. A sampled latency histogram shows that the tail is slow, never which step inside a ticket made it slow.

    Checkpoint 1 of 9· Fill the gap

    This Agent SDK configuration has to export traces as well as metrics and log events. Which variable goes in the blank?

    OTEL_ENV = {
        "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
        # Required for traces, which are in beta. Metrics and log events do not need this.
        " ? ": "1",
        # Choose an exporter per signal. Use otlp for the SDK; see the Note below.
        "OTEL_TRACES_EXPORTER": "otlp",
        "OTEL_METRICS_EXPORTER": "otlp",
        "OTEL_LOGS_EXPORTER": "otlp",
        # Standard OTLP transport configuration.
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4318",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token",
    }

    Checkpoint 2 of 9· Exam question

    Juniper Logistics runs a customer support agent in production that handles shipment-exception tickets. Each ticket triggers 6–12 model calls plus several tool calls, such as carrier lookups and refund checks. The logging pipeline writes one timestamped line per API call. Engineers have already added input sizes to those lines, yet slow tickets show no pattern, and the tool executions between model calls leave no entries at all. Reading a ticket's lines, nobody can tell which tool call belonged to which turn or where the time went. Compliance forbids storing customer message content beyond the existing retention window, and the dashboard team already tracks aggregate latency percentiles but cannot link them to individual tickets. Which change best fixes this?

    Sources12

    2.Token counts and cost that add up

    Input token counts already include the cache. On the spans, gen_ai.usage.input_tokens covers cache reads and writes, and the cache portions are also reported as separate attributes. Add cache_read to input_tokens and those tokens are counted twice. Multiply the total by one blended input price and you get the other error: cache reads cost far less than fresh input. On Claude Opus 5.5 and Sonnet 5.5, cache hits are billed at 0.05x the base input price, and most other models use 0.1x. A workload that keeps re-reading a large cached prompt is therefore overstated badly by a single price.

    Token usage attributes on request and model-request spans
    AttributeWhat it counts
    gen_ai.usage.input_tokensAll input tokens, cache reads and writes included
    gen_ai.usage.output_tokensOutput tokens
    gen_ai.usage.cache_read.input_tokensTokens read from the prompt cache
    gen_ai.usage.cache_write.input_tokensTokens written to the prompt cache
    anthropic.usage.cache_creation.ephemeral_5m_input_tokensTokens written to the 5-minute cache (Messages API spans only)
    anthropic.usage.cache_creation.ephemeral_1h_input_tokensTokens written to the 1-hour cache (Messages API spans only)

    The correct cost metric records four counts per request: uncached input, output, cache read and cache write. It prices each at its own per-model rate, because Anthropic rates usage at standard per-model, per-feature rates. Report cache writes by duration too, since the 5-minute and 1-hour caches are counted separately. Pricing cache reads at zero isn't a shortcut. They cost little, but they aren't free. Scaling a blended price by the hit rate is no better, because it still treats every token as having one price.

    Uncached input is not a span attribute; derive it. Subtract gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens from gen_ai.usage.input_tokens. A response's own usage object follows a different convention: its input_tokens is already the uncached part, reported beside cache_read_input_tokens and cache_creation_input_tokens. In the cache-diagnostics response shown later, input_tokens of 42 next to 41,850 cache-creation tokens means 42 fresh tokens plus 41,850 written, and the span's gen_ai.usage.input_tokens for that request would be their sum.

    Token usage appears on several spans of the same request: anthropic.messages.create, anthropic.model_request and anthropic.compaction. Summing it across every span inflates spend. Pick one span type per request as the source of truth. For billing, use the API's own span: the SERVER-kind anthropic.messages.create span, emitted under the claude-platform service name. Its counts are what your organization is billed for, even when the request fails or is cancelled. The SDK emits a CLIENT-kind span with the same name, and that one holds what the response reported, which can be lower or missing for a failed call.

    Attribution needs dimensions as well as counts. Every API span carries anthropic.workspace.id. If you set user.id in the W3C baggage header, it shows up on the request's API span and on every span beneath it. Use an opaque key: the API drops any user.id that contains an email address. In the Agent SDK, OTEL_RESOURCE_ATTRIBUTES can tag spans and metrics with values such as tenant.id, service.version and deployment.environment. Which of these dimensions belong on dashboards and alerts is covered in production monitoring and alerting.

    Checkpoint 3 of 9· Check yourself

    A finance team finds that the trace-based cost dashboard is higher than the invoice. It is also missing usage for requests that failed or were cancelled. Which source for per-request token usage fixes both problems?

    Checkpoint 4 of 9· Exam question

    Cobalt Insurance runs a claims-summarization service in production. Every request carries the same large system prompt, which is cached, plus a short claim-specific message, and the model returns a generated summary. A finance dashboard fed by the service's telemetry multiplies each request's total token count by a single blended price. The monthly figure it shows overstates the invoice by about 30% on this workload. Finance notes that the gap widens when the cache hit rate rises and also shifts between output-heavy and input-heavy days, so a one-time correction factor has not held. The usage data returned with each response already breaks tokens into every billed category, but the pipeline stores only the sum. What instrumentation change corrects the cost metric?

    Sources3456

    3.Cache-performance diagnostics

    A cache miss is silent by default. The cache only helps when the start of the prompt is byte-for-byte identical to a recent request. A timestamp interpolated into the system prompt, a reordered tool list or an edit to an earlier message breaks the match. The only symptom is usage.cache_read_input_tokens dropping to zero, with nothing to say what changed.

    The prefix is marked by cache breakpoints. A breakpoint is a cache_control marker on a content block, and it caches everything up to and including that block, in the order tools, system, then messages. With automatic caching the breakpoint sits on the last cacheable block of the request and moves forward as the conversation grows. Changing any block at or before a breakpoint changes the prefix, so the next request misses.

    Cache diagnostics names the divergence point. Add a diagnostics object to every request: previous_message_id: null on the first turn to opt in, then the previous response's id on each later turn. The API stores a fingerprint of each opted-in request, compares the new request against it, and reports where the two first differ: model, system prompt, tools or message history. Fingerprints hold only hashes and token-count estimates, never raw prompt content, so turning diagnostics on adds no content-exposure concern. In a streaming response, the field arrives on the message_start event.

    A response whose cache_miss_reason blames the system prompt for 41,850 missed tokensjson
    {
      "id": "msg_01Xyz...",
      "type": "message",
      "role": "assistant",
      "content": [{ "type": "text", "text": "..." }],
      "usage": {
        "input_tokens": 42,
        "cache_read_input_tokens": 0,
        "cache_creation_input_tokens": 41850,
        "output_tokens": 210
      },
      "diagnostics": {
        "cache_miss_reason": {
          "type": "system_changed",
          "cache_missed_input_tokens": 41850
        }
      }
    }
    Reading the diagnostics field
    ValueMeaningWhat to do
    nullNot opted in, first turn, or nothing diverged before the last breakpointNothing to fix structurally
    {"cache_miss_reason": null}Comparison still running when the response was serializedTreat as inconclusive; check the next turn
    *_changed type, e.g. system_changedFirst point where the request structure divergedFix that part of the prompt
    previous_message_not_found or unavailableNo comparison was producedCheck the id you carried forward

    Diagnostics compare request structure, whether or not the cache actually hit. Read them together with cache_read_input_tokens. A *_changed reason on a turn whose cache reads also fell to zero points to the cause, and the cache_missed_input_tokens figure tells you how much the regression cost.

    Checkpoint 5 of 9· Match them up

    You opted a multi-turn agent into cache diagnostics, and cache reads keep dropping to zero. Match each diagnostics value on a response to the correct next step.

    Tap a term, then the definition that fits it.

    Sources73

    4.Tracing reasoning, tool use and grounding

    When an answer is wrong, the question is what the model saw, what it thought and what it called. Message attributes on spans are JSON in the GenAI conventions' message format, with each Anthropic content block mapped to one part. text stays text. tool_use and server_tool_use become tool_call, and MCP calls also record the server name. Tool results, including server tool results such as web search, become tool_call_response. A thinking block becomes a reasoning part without its signature, so extended-thinking content can be read next to the tool calls it led to. Managed Agents is a Claude API service in which the agent runs in a session on the API side: your application sends events to the session through the Sessions API and the API runs each agent turn, where with the Messages API you send every request yourself. Managed Agents spans don't include thinking content.

    Tool arguments and results live on the tool span itself: arguments in gen_ai.tool.call.arguments, results in anthropic.tool_result.content. A request span holds the system prompt in gen_ai.system_instructions, the input in gen_ai.input.messages and the response in gen_ai.output.messages. All of these are content attributes, so they exist only when content capture is on. For Managed Agents, group spans by gen_ai.conversation.id and sort by start time to read one thread as a transcript. Group by anthropic.session.id to see the whole session across traces.

    Grounding traceability comes from the response itself. When search results are passed with citations enabled, each citation carries search_result_index, start_block_index, end_block_index and cited_text, which is the full text of the cited blocks. Whole blocks are the smallest citable unit. Split source content into smaller text blocks if you need to tie a claim to a specific passage instead of a whole document. Logging these citation fields next to the response gives you a claim-to-source record without capturing the full prompt.

    Checkpoint 6 of 9· Check yourself

    To avoid storing prompts, your team exports traces in metadata-only mode. A user disputes an answer, and an engineer opens the tool span for the search the agent ran. What will the engineer find, and how do you keep a record of which claim came from which source?

    Sources518

    5.Structured log events

    Log events are records of discrete things that happened, kept separate from spans and metrics. The Agent SDK emits a structured record for each prompt, API request, API error and tool result, and you enable them with OTEL_LOGS_EXPORTER. Metrics carry counters for tokens, cost, sessions and tool decisions. Traces carry the timing tree.

    The events worth building on are these:

    • tool_decision and tool_result: each tool call, whether it was allowed or denied, and its result
    • mcp_server_connection: MCP server connections and connection failures
    • permission_mode_changed: changes to the agent's permission mode
    • API request and API error events, one per call to the Claude API

    Tool failures need these events in particular. A tool that returns an error result to the model, which then retries, never produces an HTTP error, so a dashboard built on status codes stays green while the agent loops. Counting tool_result events by outcome shows the failure your HTTP monitoring can't see. Every request span also carries anthropic.request.id, the value of the request-id response header, which is the identifier to quote when escalating a specific call. How to cluster and sample these records across many runs is covered in trace analysis at scale.

    Checkpoint 7 of 9· Match them up

    Status-code dashboards for a production Agent SDK deployment show no problems, but several incidents need explaining. Match each log event to the incident it lets you diagnose.

    Tap a term, then the definition that fits it.

    Sources25

    6.Content capture under privacy constraints

    Each destination and each SDK client has a content mode. Metadata only (metadata_only) is the default. It keeps identifiers, timing, status, token counts and text you choose, such as agent and tool names and MCP server names. Full content (content) adds prompts and responses. That lets you read failed conversations and build evals from real traffic. With zero data retention, Messages API content is held only in memory until it is delivered. Managed Agents isn't eligible for zero data retention, and its retention doesn't change. An organization with a HIPAA, FERPA or K-12 data-handling restriction gets metadata-only export from destinations whatever the setting, but SDK spans aren't affected.

    A client rule that prompts and tool outputs must never leave the VPC is stricter than zero data retention, and it rules out Full content to any outside backend. Metadata mode still supports debugging. A span tree with tool names, status, latency, model, token counts and anthropic.request.id shows which step failed, how long it took and which call to escalate. The alternatives each fail the rule. Role-based access on a vendor backend controls who reads the content, but the content has already left. A regex pass for names and emails misses everything else in a prompt. Turning tracing off in production throws away the debugging that metadata mode keeps.

    Claude Code and Agent SDK content flags, all disabled by default
    VariableWhat it adds
    OTEL_LOG_USER_PROMPTSUser prompt text on prompt events and the interaction span
    OTEL_LOG_ASSISTANT_RESPONSESAssistant response text; falls back to OTEL_LOG_USER_PROMPTS when unset
    OTEL_LOG_TOOL_DETAILSTool input arguments such as Bash commands and file paths, plus real MCP, skill and plugin names
    OTEL_LOG_TOOL_CONTENTA tool.output span event with file contents and Bash output; requires tracing
    OTEL_LOG_RAW_API_BODIESFull Messages API request and response JSON as log events, with extended-thinking content redacted

    OTEL_LOG_RAW_API_BODIES covers the most. Its bodies include the entire conversation history, and turning it on implies consent to everything the three narrower flags would reveal. Set it to file:<dir> to write untruncated bodies to disk and keep only a body_ref pointer in the event.

    Content capture also has size limits. On the API side, Full content holds up to 0.75 MB per span by default, which is enough for a 200k-token context of English text. When content exceeds the limit, the system prompt and earliest messages are dropped first and anthropic.content.truncated is set to true. Raise the limit with traces.max_content_bytes in the SDK if your backend accepts larger payloads, or lower it if a proxy rejects big requests. Claude Code truncates content attributes at 60 KB by default (CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH).

    Checkpoint 8 of 9· Match them up

    Before any Agent SDK content flag is turned on in production, a security review needs to know what each one would expose. Match each flag to the content it adds to telemetry.

    Tap a term, then the definition that fits it.

    Checkpoint 9 of 9· Exam question

    Basalt Health runs a production claims-triage agent on Claude for a regulated hospital client. The contract requires zero data retention, and the client's security team forbids prompts and tool outputs from leaving the client's VPC, including to the shared third-party tracing service Basalt Health already uses elsewhere. Engineers have no staging data that resembles production, and past staging replays with synthetic inputs did not reproduce the data-dependent tool failures seen in the field. On-call engineers still need to work out where and how a failing agent run went wrong, such as which tool call failed, how long it took, and how large its result was. What is the best telemetry design?

    Sources19

    Sources

    (9)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “The tool runner runs get_weather (shown as the anthropic.custom_tool_use get_weather span).”
      ↩︎ Distributed traces for agent loops and tool calls
      “the tool runner, the SDK helper that runs your tools and sends their results back to Claude”
      ↩︎ Distributed traces for agent loops and tool calls
      “in which your application sends a message to a session and runs a custom tool”
      ↩︎ Tracing reasoning, tool use and grounding
      “The arguments are in gen_ai.tool.call.arguments, and the result is in anthropic.tool_result.content.”
      ↩︎ Tracing reasoning, tool use and grounding
      “Spans include identifiers, timing, status, token counts, and text that you choose”
      ↩︎ Content capture under privacy constraints
      “destinations export only metadata regardless of their content mode.”
      ↩︎ Content capture under privacy constraints
      “With Full content, each span holds up to 0.75 MB of content by default.”
      ↩︎ Content capture under privacy constraints
      “A span is one step in a trace, such as a model request or a tool use.”
      ↩︎ Key concept
      “For Messages API requests, the API holds the exported content only in memory until it delivers it to the destination.”
      ↩︎ Exam trap 1
      “Each request has its token usage on several of its spans.”
      ↩︎ Exam trap 3
      “With zero data retention, you can still use Full content.”
      ↩︎ Prediction
    2. 2.
      “claude_code.tool: wraps each tool invocation, with child spans for the permission wait (claude_code.tool.blocked_on_user) and the execution itself (claude_code.tool.execution).”
      ↩︎ Distributed traces for agent loops and tool calls
      “Structured records for each prompt, API request, API error, and tool result”
      ↩︎ Structured log events
      “tool_decision and tool_result: each tool call, whether it was allowed or denied, and its result”
      ↩︎ Structured log events
      “In TypeScript, env replaces the inherited environment entirely, so include ...process.env in the object you pass.”
      ↩︎ Exam trap 4
    3. 3.
      “Cache hits and refreshes on Claude Opus 5.5 and Claude Sonnet 5.5 are priced at 0.05x the base input price.”
      ↩︎ Token counts and cost that add up
      “Marking a block with cache_control writes exactly one cache entry: a hash of the prefix ending at that block.”
      ↩︎ Cache-performance diagnostics
    4. 4.
      “Token usage rated in USD at standard per-model, per-feature rates”
      ↩︎ Token counts and cost that add up
    5. 5.
      “the token counts are the tokens that your organization is billed for, even when the request fails or the application cancels it.”
      ↩︎ Token counts and cost that add up
      “The input tokens, cache reads and writes included.”
      ↩︎ Token counts and cost that add up
      “To protect user privacy, the API drops any user.id that contains one.”
      ↩︎ Token counts and cost that add up
      “Each Anthropic content block becomes one GenAI conventions message part”
      ↩︎ Tracing reasoning, tool use and grounding
      “The value of the request-id response header.”
      ↩︎ Structured log events
      “The input tokens, cache reads and writes included.”
      ↩︎ Exam trap 2
      “Attributes tagged with Content are only on spans in content mode, from the SDK and the API alike.”
      ↩︎ Checkpoint
    6. 6.
      “If cache_read_input_tokens is small next to input_tokens plus cache_creation_input_tokens”
      ↩︎ Token counts and cost that add up
    7. 7.
      “Without cache diagnostics, the only signal is usage.cache_read_input_tokens dropping to zero, with no indication of what changed.”
      ↩︎ Cache-performance diagnostics
      “Fingerprints contain only hashes and token-count estimates (never raw prompt content)”
      ↩︎ Cache-performance diagnostics
      “The comparison is about request structure, independent of whether the cache actually hit.”
      ↩︎ Cache-performance diagnostics
      “For *_changed types this identifies the first divergence point; previous_message_not_found and unavailable are cases where no comparison was produced.”
      ↩︎ Checkpoint
    8. 8.
      “The text block is the minimal citable unit: Claude cites whole blocks, not substrings within a block.”
      ↩︎ Tracing reasoning, tool use and grounding
    9. 9.
      “Enabling this implies consent to everything OTEL_LOG_USER_PROMPTS, OTEL_LOG_TOOL_DETAILS, and OTEL_LOG_TOOL_CONTENT would reveal”
      ↩︎ Content capture under privacy constraints

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.