CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 3 · Lesson 16/39

    Instrumentation gaps in agentic systems: finding the signal that would have fired

    Instrumentation and Monitoring Gaps

    16 min read
    2.63% of exam
    5 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Explain why transport-level signals such as status codes, latency and token spend miss failures where a tool returns a successful but useless result
    • Choose the per-tool, per-agent monitoring strategy that would have surfaced a failure that went undetected in a multi-agent architecture
    • Enable metrics, log events and traces for the Agent SDK and Claude Code, including the beta flag that traces require
    • Choose the content gate that exposes tool commands and paths without recording prompt text
    • Pin every installation to one central OTLP collector through managed settings

    Key concept

    Instrumentation gap — A failure that no collected signal reacts to, because every metric being watched measures whether a call happened and how fast, not what came back. To close the gap, find the signal the failure would actually change and monitor that one.

    1.Why a scaled agent can fail without raising an alarm

    Most agent dashboards watch the transport layer: did the API call succeed, how long did it take, how many tokens did it cost. Each claude_code.llm_request span carries exactly these attributes: success, status_code, an error_class such as api_timeout or server_overload, duration_ms (wall-clock time including retries) and ttft_ms. These are the right signals for overloads, timeouts and retry storms, which production reliability controls are built to handle. None of them tells you whether a tool's answer was any use.

    A transcript and an outcome are different things. A flight-booking agent can say "Your flight has been booked" while no reservation exists in the database. Monitoring that checks only the shape of a transcript (calls were made, calls returned) has the same blind spot. A tool that quietly degrades produces a transcript that looks normal, and replies that look fine to the customer, for as long as nobody measures what the tool returned.

    Scale makes this worse. With one agent and one tool, a person reading a few sessions might notice the empty searches. With an orchestrator delegating to six subagents, each with its own tools, one tool's failure is a small fraction of total traffic, so aggregate dashboards stay flat.

    What each family of Claude Code signals reacts to, and what it can't see
    SignalReacts toBlind to
    success, status_code, error_class on claude_code.llm_requestFailed API requests, timeouts, server overloadA successful request that was fed an empty or wrong tool result
    duration_ms, ttft_msSlow requests and retriesWrong answers delivered at normal speed
    claude_code.token.usageCost spikes, unusual token consumptionFailures that leave spend roughly unchanged
    tool_result events, tool success rates, error patterns by tool typePer-tool behaviour: what each tool did and returnedWhether the result was correct, unless tool content or details are captured

    Checkpoint 1 of 8· Match them up

    Each failure below happens in a scaled multi-agent deployment. Match each one to the Claude Code signal that would change when it happens.

    Tap a term, then the definition that fits it.

    Sources12

    2.Selecting the monitor that would have surfaced the failure

    Start from the failure and work backwards. Ask which measurable quantity actually changed, then check whether anything was watching it. In the empty-search case, latency was normal, every call returned HTTP 200 and spend didn't move. The one thing that changed was the content of one tool's results: they went from populated to empty. The monitor that would have caught it tracks that quantity per tool, such as an empty-result rate, and alerts when it departs from that tool's own normal level.

    Claude Code's monitoring guidance lists per-tool views as analysis targets alongside cost spikes: most frequently used tools, tool success rates, average tool execution times, and error patterns by tool type. Managed Agents, a beta Claude API feature in which agents run as stateful sessions that you inspect in the Claude Console, and which is separate from the Agent SDK and Claude Code, has the same per-tool view in the session viewer's Tools tab, which shows each configured tool along with call counts, failures, and median duration. To judge whether a result was empty, you need the result itself. In the Agent SDK, OTEL_LOG_TOOL_CONTENT=1 adds a tool.output span event with file contents, Bash output, and whatever MCP tools, WebFetch and WebSearch return. It requires tracing to be enabled.

    Attribution makes a per-tool signal usable at scale. On each model request span, agent_id identifies the subagent or teammate that issued it (it's absent on the main session), parent_agent_id identifies the agent that spawned it, and query_source_safe names the issuing subsystem. Token metrics can be broken down by agent.name, so spend can be attributed to specific subagent types. Within one process, the Agent SDK can give each agent its own service identity through per-call env options. In the snippet below, the comment line stands in for the exporter setup: CLAUDE_CODE_ENABLE_TELEMETRY=1, an OTEL_*_EXPORTER selector for each signal, and the OTLP protocol, endpoint and headers.

    Per-agent service name and version as OpenTelemetry resource attributes (Agent SDK, Python)python
    options = ClaudeAgentOptions(
        env={
            # ... exporter configuration from the Enable telemetry export example ...
            "OTEL_SERVICE_NAME": "support-triage-agent",
            "OTEL_RESOURCE_ATTRIBUTES": "service.version=1.4.0,deployment.environment=production",
        },
    )

    Signals tied to the cause help, but they only fire when the failure shows up at their layer. Managed Agents sends a vault_credential.refresh_failed webhook when MCP OAuth credentials cannot refresh, and mcp_server_connection events record server connection failures. Use them to catch a credential that breaks the connection itself. A tool whose backing index silently returns nothing behind a healthy connection still needs the result-level monitor. Webhooks can also be missed, which is why the guidance is to also list sessions on a schedule.

    Checkpoint 2 of 8· Check yourself

    After an index migration, the document-lookup MCP tool used by one of six subagents starts returning no matches. The MCP connection stays up, every call returns HTTP 200 with normal latency, and spend does not change. Which monitor would have caught this?

    Checkpoint 3 of 8· Exam question

    A support platform runs an orchestrator that delegates to six subagents, each calling its own tools. For three weeks the retrieval subagent's search tool returned empty results because its index credential had expired. Every API call returned HTTP 200, latency stayed normal, and the agent answered from general model knowledge, so customers got plausible but ungrounded replies. Which monitoring strategy would have surfaced this failure?

    Sources1345

    3.Enabling metrics, log events and traces

    The Agent SDK exports three OpenTelemetry signals, and each one answers different questions. Metrics are counters for tokens, cost, sessions, lines of code and tool decisions. Log events are structured records for each prompt, API request, API error and tool result. Traces are spans for each interaction, model request, tool call and hook. Each has its own exporter variable. Traces are in beta and also need CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1. Setting OTEL_TRACES_EXPORTER alone produces no spans, and with no spans, OTEL_LOG_TOOL_CONTENT has nothing to attach the tool output to.

    Turning on all three signals with OTLP export from the Agent SDKpython
    OTEL_ENV = {
        "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
        # Required for traces, which are in beta. Metrics and log events do not need this.
        "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
        # Choose an exporter per signal. Use otlp for the SDK; see the Note below.
        "OTEL_TRACES_EXPORTER": "otlp",
        "OTEL_METRICS_EXPORTER": "otlp",
        "OTEL_LOGS_EXPORTER": "otlp",
        # Standard OTLP transport configuration.
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4318",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token",
    }

    The sample selects otlp for all three signals, which sends them to the collector named by OTEL_EXPORTER_OTLP_ENDPOINT. Other selector values exist (console, and none to disable a signal), but the SDK sample uses otlp throughout. There are two places to set these variables. For production, set them in the process environment (shell, container or orchestrator) before the application starts, and every query() call picks them up with no code change. Use per-call env when agents in the same process need different telemetry settings, for example a different service name per subagent. The two SDKs differ here: in Python, env is merged on top of the inherited environment, but in TypeScript it replaces the environment entirely. A TypeScript agent that passes only the telemetry variables loses PATH and its API key unless it spreads ...process.env first.

    Checkpoint 4 of 8· Fill the gap

    A team's collector receives metrics and log events from their Agent SDK service, but no spans show up, so OTEL_LOG_TOOL_CONTENT never attaches any tool.output. Which variable fills the blank in this configuration?

    OTEL_ENV = {
        "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
        # Required for traces, which are in beta. Metrics and log events do not need this.
        " ? ": "1",
        # Choose an exporter per signal. Use otlp for the SDK; see the Note below.
        "OTEL_TRACES_EXPORTER": "otlp",
        "OTEL_METRICS_EXPORTER": "otlp",
        "OTEL_LOGS_EXPORTER": "otlp",
        # Standard OTLP transport configuration.
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4318",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token",
    }

    Sources41

    4.Content gates: exposing tool commands without exposing prompts

    By default, telemetry records that things happened, not what they contained. The user_prompt attribute on interaction spans reads <REDACTED> unless its gate is set. A tool_result event shows that a Bash tool ran, but not which command it ran. Each kind of content has its own opt-in variable, so a team can expose exactly the content it needs for an investigation and nothing more.

    Content gates and what each one adds to telemetry
    VariableAddsNote
    OTEL_LOG_USER_PROMPTS=1Prompt text on claude_code.user_prompt events and the claude_code.interaction spanHas no effect on tool arguments
    OTEL_LOG_TOOL_DETAILS=1Tool input arguments (file paths, shell commands, search patterns) on tool_result events; real agent, skill, plugin and MCP server names on cost and token metricsThe gate for which command ran
    OTEL_LOG_TOOL_CONTENT=1tool.output span event with file contents, Bash output and MCP, WebFetch and WebSearch results, truncated at 60 KB by defaultRequires tracing to be enabled
    OTEL_LOG_RAW_API_BODIESFull Messages API request and response JSON, including the entire conversation historyImplies consent to everything the other three reveal

    Raw API bodies are the wrong tool for a no-prompts rule. They contain the entire conversation history, so the prompt text is already collected at capture time. Filtering it out later doesn't undo that. Enabling raw bodies counts as consent to everything the other three gates would reveal. For questions like "which command ran" and "which file was touched", set OTEL_LOG_TOOL_DETAILS=1 and leave OTEL_LOG_USER_PROMPTS unset. For MCP calls, tool details also replace the redacted literal mcp_tool with the real server and tool names.

    Checkpoint 5 of 8· Check yourself

    A security lead has a plan for a team bound by a no-prompt-text retention policy. They want to set OTEL_LOG_RAW_API_BODIES=1 and add a collector rule that deletes prompt text before anything is stored. They argue this keeps the policy while giving incident reviewers every tool call in full. Why is this the wrong choice?

    Checkpoint 6 of 8· Exam question

    A compliance team requires that Claude Code telemetry never store prompt text. In an incident review, the security team found that tool-result events showed a shell tool had run, but not which command or file paths were involved. The platform owner must close this gap before the next audit and keep compliance's sign-off. Which change should the architect propose?

    Sources14

    5.Pinning every installation to one collector

    A gap can also come from telemetry that leaves the machine but lands in the wrong place. If each developer chooses their own exporter destination, some installations will report somewhere the security team never looks. In Claude Code, setting OTEL_EXPORTER_OTLP_ENDPOINT in managed settings removes every per-signal endpoint a developer has set. Developers then can't point one signal at a different collector, and administrators don't need to set each per-signal endpoint variable separately. A setting committed to a repository or written by an onboarding script doesn't give that guarantee. It's an ordinary file that developers can edit, so the override only comes from the managed setting.

    Once telemetry reaches one place, put identity on it. OTEL_RESOURCE_ATTRIBUTES can carry enduser.id and tenant.id, and the security events then support the questions that matter after an incident: tool_decision records whether each call was allowed or denied and where the decision came from, permission_mode_changed records escalations, and mcp_server_connection records connections and connection failures. For tracing one specific incident through these records, see root-cause isolation from logs and traces.

    Checkpoint 7 of 8· Match them up

    All installations now report to one central collector. During a post-incident review, the security team has four questions. Match each record or attribute to the question it answers.

    Tap a term, then the definition that fits it.

    Checkpoint 8 of 8· Exam question

    A security architect wants every Claude Code installation in the company to send telemetry to one central OTLP collector, and wants developers unable to redirect it with their own per-signal endpoints or credentials. Which configuration achieves this?

    Sources14

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Short error class token when the request failed, such as api_timeout or server_overload.”
      ↩︎ Why a scaled agent can fail without raising an alarm
      “Wall-clock duration including retries”
      ↩︎ Why a scaled agent can fail without raising an alarm
      “Identifier of the subagent or teammate that issued the request. Absent on the main session”
      ↩︎ Selecting the monitor that would have surfaced the failure
      “Attributing spend to specific skills, plugins, or subagent types via the skill.name, plugin.name, and agent.name attributes”
      ↩︎ Selecting the monitor that would have surfaced the failure
      “Traces exporter types, comma-separated. Use none to disable”
      ↩︎ Enabling metrics, log events and traces
      “Prompt text. Value is <REDACTED> unless the gate is set”
      ↩︎ Content gates: exposing tool commands without exposing prompts
      “Claude Code removes every developer-set per-signal endpoint”
      ↩︎ Pinning every installation to one collector
      “Tool call allowed or denied, and where the decision came from, such as configuration, a hook, or the user | tool_decision”
      ↩︎ Checkpoint
    2. 2.
      “The outcome is the final state in the environment at the end of the trial.”
      ↩︎ Why a scaled agent can fail without raising an alarm
      “The outcome is the final state in the environment at the end of the trial.”
      ↩︎ Key concept
      “The outcome is the final state in the environment at the end of the trial.”
      ↩︎ Exam trap 1
    3. 3.
      “along with call counts, failures, and median duration”
      ↩︎ Selecting the monitor that would have surfaced the failure
      “Inspect a session in the Claude Console, read its token usage and list cost, and debug unexpected agent behavior.”
      ↩︎ Selecting the monitor that would have surfaced the failure
    4. 4.
      “A tool.output span event on claude_code.tool with file contents, Bash output, and what MCP tools, WebFetch, and WebSearch return”
      ↩︎ Selecting the monitor that would have surfaced the failure
      “OTEL_TRACES_EXPORTER plus CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1”
      ↩︎ Enabling metrics, log events and traces
      “This is the recommended approach for production deployments.”
      ↩︎ Enabling metrics, log events and traces
      “In TypeScript, env replaces the inherited environment entirely, so include ...process.env in the object you pass.”
      ↩︎ Enabling metrics, log events and traces
      “Bodies include the entire conversation history and have extended-thinking content redacted.”
      ↩︎ Content gates: exposing tool commands without exposing prompts
      “Enabling this implies consent to everything the three variables above would reveal”
      ↩︎ Content gates: exposing tool commands without exposing prompts
      “tool_decision and tool_result: each tool call, whether it was allowed or denied, and its result”
      ↩︎ Pinning every installation to one collector
      “OTEL_TRACES_EXPORTER plus CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1”
      ↩︎ Exam trap 2
      “Tool input arguments such as file paths, shell commands, and search patterns on claude_code.tool_result events”
      ↩︎ Exam trap 3
      “Enabling this implies consent to everything the three variables above would reveal”
      ↩︎ Exam trap 4
      “tool_decision and tool_result: each tool call, whether it was allowed or denied, and its result”
      ↩︎ Checkpoint
      “Tool input arguments such as file paths, shell commands, and search patterns on claude_code.tool_result events”
      ↩︎ Prediction
      “Bodies include the entire conversation history and have extended-thinking content redacted. Enabling this implies consent to everything the three variables above would reveal”
      ↩︎ Checkpoint
    5. 5.
      “MCP OAuth credentials that cannot refresh”
      ↩︎ Selecting the monitor that would have surfaced the failure
      “You can miss a webhook delivery, so also list your sessions on a schedule and check the idle ones.”
      ↩︎ Selecting the monitor that would have surfaced the failure

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.